Method and device for identifying advertisement in audio, equipment and medium

Abstract

The invention relates to a method, a device, equipment and a medium for identifying advertisements in audio, and belongs to the technical field of advertisement identification, the method comprises the following steps: constructing an advertisement sample library in which advertisement samples are stored; to-be-recognized audio is acquired and preprocessed; extracting features of the preprocessed to-be-recognized audio to obtain a frame-level feature sequence; extracting a subsequence in the frame-level feature sequence by using a sliding window, and carrying out feature comparison on the subsequence and an advertisement sample in a sample library to obtain a comparison score; and performing hit judgment and time positioning according to the comparison score to obtain an identification result of the advertisement in the to-be-identified audio. According to the invention, the advertisement fragment in the to-be-identified audio is accurately identified.

Description

Technical Field

[0001] This invention belongs to the field of advertising recognition technology, and specifically relates to a method, apparatus, device, and medium for recognizing advertisements in audio. Background Technology

[0002] Advertising plays an increasingly important role in daily information dissemination. With the acceleration of commercialization, advertisements appear more and more frequently on multiple media platforms such as television, radio, and the internet, subtly influencing people's lifestyles. However, advertising can also affect the user experience, especially when users are listening to audio content; sudden advertisements can cause user annoyance. Therefore, it is necessary to identify advertising segments in audio in a timely and effective manner so that users can be aware of them in advance or skip them. In current technologies, when identifying advertisements in audio, a simple comparison is usually made between advertising samples in a sample library and the audio to be identified, which has technical problems that cannot guarantee the accuracy of the identification results. Summary of the Invention

[0003] Based on the above analysis, the embodiments of the present invention aim to provide a method, apparatus, device, and medium for identifying advertisements in audio, in order to solve the technical problem that the accuracy of the identification results cannot be guaranteed in the prior art.

[0004] The objective of this invention is achieved as follows:

[0005] A first aspect of the present invention provides a method for identifying advertisements in audio, comprising:

[0006] Construct an advertising sample library, which stores advertising samples;

[0007] Acquire the audio to be recognized and perform preprocessing;

[0008] Features of the preprocessed audio to be identified are extracted to obtain a frame-level feature sequence;

[0009] A subsequence is extracted from the frame-level feature sequence using a sliding window, and the subsequence is compared with the feature of an advertisement sample in the sample library to obtain a comparison score.

[0010] Based on the comparison score, a hit determination and time location are performed to obtain the recognition result of the advertisement in the audio to be identified.

[0011] Furthermore, the construction of the advertising sample library includes: collecting original audio; resampling, denoising, and normalizing the original audio; dividing the processed original audio into three categories—short, medium, and long—according to audio duration; setting a first sliding window, a second sliding window, and a third sliding window of different specifications for the short, medium, and long audio categories; using the three sliding windows to extract feature sequences of the processed original audio from the three categories of audio respectively, as advertising samples; and dividing all advertising samples into a first feature set, a second feature set, and a third feature set according to the corresponding audio category.

[0012] Furthermore, the subsequences in the frame-level feature sequence are extracted using a sliding window, including: using a first sliding window to extract several first subsequences from the frame-level feature sequence with the same length as each advertisement sample in the first feature set; using a second sliding window to extract several second subsequences from the frame-level feature sequence with the same length as each advertisement sample in the second feature set; and using a third sliding window to extract several third subsequences from the frame-level feature sequence with the same length as each advertisement sample in the third feature set.

[0013] Further, the step of comparing the subsequence with the advertising samples in the sample library includes: sequentially calculating the similarity between the feature vector of each advertising sample in the first feature set and the feature vector in the corresponding first subsequence, and taking the average value as the comparison score between the current first subsequence and the current advertising sample; sequentially calculating the similarity between the feature vector of each advertising sample in the second feature set and the feature vector in the corresponding second subsequence, and taking the average value as the comparison score between the current second subsequence and the current advertising sample; sequentially calculating the similarity between the feature vector of each advertising sample in the third feature set and the feature vector in the corresponding third subsequence, and taking the average value as the comparison score between the current third subsequence and the current advertising sample.

[0014] Furthermore, the similarity is cosine similarity or Euclidean distance.

[0015] Furthermore, the step of determining the hit and locating the time based on the comparison score includes: if the comparison score is greater than a preset threshold, then marking the current subsequence as a candidate hit region of the current advertising sample; if the current advertising sample has multiple candidate hit regions, taking the subsequence with the highest comparison score as the hit segment of the current advertising sample; performing time mapping on the hit segment to return the start and end time period of the hit segment in the audio to be identified; and using the advertising samples corresponding to all hit segments in the audio to be identified and the start and end time periods as the identification result.

[0016] Furthermore, it also includes: dynamically expanding the time boundary of the hit segment, extending the hit segment forward and backward with a fixed step size, and recalculating the comparison score. If the comparison score increases, the extension continues until the score decreases, so as to update the hit segment.

[0017] A second aspect of the present invention provides an apparatus for identifying advertisements in audio, comprising:

[0018] The sample library construction module is used to construct an advertising sample library, which stores advertising samples;

[0019] The acquisition module is used to acquire the audio to be recognized and perform preprocessing.

[0020] The feature extraction module is used to extract features from the preprocessed audio to be recognized, and obtain a frame-level feature sequence.

[0021] The feature comparison module is used to extract subsequences from the frame-level feature sequence using a sliding window, and compare the subsequences with the advertising samples in the sample library to obtain a comparison score.

[0022] The recognition module is used to determine the hit score and locate the time based on the comparison score, so as to obtain the recognition result of the advertisement in the audio to be recognized.

[0023] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the method for identifying advertisements in audio as described in any embodiment.

[0024] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for identifying advertisements in audio as described in any embodiment.

[0025] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0026] The method for identifying advertisements in audio provided by the present invention processes the audio to be identified into a frame-level feature sequence, uses a sliding window to extract a subsequence of the frame-level feature sequence, compares the subsequence with the advertisement samples in the sample library, and then combines hit determination and time positioning to accurately identify the advertisement segments in the audio to be identified. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings.

[0028] Figure 1 This is a flowchart of the method for identifying advertisements in audio provided in Embodiment 1 of the present invention;

[0029] Figure 2 This is a schematic diagram of the device for identifying advertisements in audio provided in Embodiment 2 of the present invention;

[0030] Figure 3 This is a schematic diagram of the electronic device architecture provided in Embodiment 3 of the present invention. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be noted that, unless otherwise specified, the implementation methods and features in the implementation methods in this disclosure can be combined, separated, interchanged, and / or rearranged. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1

[0033] A specific embodiment of the present invention, such as Figure 1 As shown, a method for identifying advertisements in audio is disclosed, including the following steps:

[0034] S1. Construct an advertising sample library, which stores advertising samples;

[0035] S2. Acquire the audio to be recognized and perform preprocessing;

[0036] S3. Extract the features of the preprocessed audio to be recognized to obtain a frame-level feature sequence;

[0037] S4. Use a sliding window to extract subsequences from the frame-level feature sequence, and compare the subsequences with the advertising samples in the sample library to obtain a comparison score.

[0038] S5. Based on the comparison score, a hit determination and time location are performed to obtain the recognition result of the advertisement in the audio to be identified.

[0039] Compared with existing technologies, the method for identifying advertisements in audio provided in this embodiment processes the audio to be identified into a frame-level feature sequence, uses a sliding window to extract a subsequence of the frame-level feature sequence, compares the subsequence with the advertisement samples in the sample library, and then combines hit determination and time positioning to accurately identify the advertisement segments in the audio to be identified.

[0040] In this embodiment, the construction of the advertising sample library in step S1 includes:

[0041] S101. Collect the original audio and perform resampling, noise reduction and normalization on the original audio.

[0042] S102. The processed original audio is divided into three categories: short, medium, and long, according to the audio duration.

[0043] S103. Set different specifications for the first sliding window, the second sliding window, and the third sliding window for short, medium, and long audio categories;

[0044] S104. Use three sliding windows to extract the feature sequences of the processed original audio from the three types of audio respectively, and use them as advertising samples. Divide all advertising samples into the first feature set, the second feature set and the third feature set according to the corresponding audio classification.

[0045] For example, the original audio can come from advertising segments from radio, podcasts, audio platforms, etc., forming the original audio. Each advertising segment is standardized, including resampling to a uniform sampling rate, removing background noise, and amplitude normalization. Then, all the original audio segments are divided into three categories according to the audio duration T, such as short ads (T<30s), medium ads (30s≤T<60s), and long ads (T≥60s). A sliding window is set for each type of audio, and the corresponding feature sequence is extracted. Thus, each audio segment is extracted as a feature sequence as an advertising sample. Then, according to the corresponding audio classification rules, all advertising samples are divided into a first feature set, a second feature set, and a third feature set, and then stored in the advertising sample library.

[0046] In this embodiment, step S2, the preprocessing of the audio to be identified, is similar to the processing of the original audio described above.

[0047] Specifically, both the feature extraction of the original audio in step S1 and the feature extraction of the audio to be identified in step S3 can be performed using Mel spectrum. First, the audio signal is converted into frequency domain features that can describe the audio content, such as short-time Fourier transform. Then, the samples are vectorized using statistical methods, such as based on MFCC (Mel frequency cepstral coefficients), to generate low-dimensional feature vectors. The feature vectors are then combined to form a feature sequence.

[0048] In this embodiment, step S4, which involves using a sliding window to extract subsequences from the frame-level feature sequence, includes:

[0049] S401. Using a first sliding window, extract several first subsequences from the frame-level feature sequence, each with a length equal to that of each advertisement sample in the first feature set;

[0050] S402. Using a second sliding window, extract several second subsequences from the frame-level feature sequence, each with the same length as the advertisement sample in the second feature set.

[0051] S403. Using a third sliding window, extract several third subsequences from the frame-level feature sequence that are equal in length to each advertisement sample in the third feature set.

[0052] Specifically, the sizes of the first, second, and third sliding windows remain unchanged, but the step size can be set according to the actual situation. Assume the first feature set S1 includes three advertising samples, denoted as S1 = {O1, O2, O3}, which respectively include 4, 5, and 6 feature vectors, denoted as... The frame-level feature sequence consists of 20 feature vectors, represented as Z = [z1, z2, ..., z2]. 20 Using the first sliding window, several first subsequences with the same length as O1, O2, and O3 are extracted from the frame-level feature sequence Z. The extraction of the second and third subsequences is the same as that of the first subsequences, and will not be described again.

[0053] In this embodiment, step S4, which involves comparing the subsequence with the advertising samples in the sample library, includes:

[0054] S404. Calculate the similarity between the feature vector of each advertisement sample in the first feature set and the feature vector in the corresponding first subsequence in turn, and take the average value as the comparison score between the current first subsequence and the current advertisement sample.

[0055] Specifically, compared with advertising samples The corresponding first subsequence has 17 elements, represented as [z k ,z k+1 ,z k+2 ,z k+3 Given that 1 ≤ k ≤ 17, calculate the feature vector z in each first subsequence. k and z k+1 and z k+2 and z k+3 and The similarity between the four similarities is calculated, and the average of the four similarities is taken as the comparison score between the first subsequence and the advertising sample O1 under the current k value; similarly, the comparison scores of other advertising samples in the first feature set are calculated in the same way.

[0056] S405. Calculate the similarity between the feature vector of each advertisement sample in the second feature set and the feature vector in the corresponding second subsequence in turn, and take the average value as the comparison score between the current second subsequence and the current advertisement sample.

[0057] S406. Calculate the similarity between the feature vector of each advertisement sample in the third feature set and the feature vector in the corresponding third subsequence in turn, and take the average value as the comparison score between the current third subsequence and the current advertisement sample.

[0058] Specifically, the alignment scores of the advertising samples and subsequences in the second and third feature sets are based on the same principle as in step S404.

[0059] In this embodiment, the similarity is cosine similarity or Euclidean distance.

[0060] In this embodiment, step S5, which involves determining the hit score and locating the time based on the comparison score, includes:

[0061] S501. If the comparison score is greater than the preset threshold, the current subsequence is marked as the candidate hit region of the current advertising sample.

[0062] S502. If there are multiple candidate hit regions in the current ad sample, take the subsequence with the highest alignment score as the hit segment of the current ad sample.

[0063] S503. Perform time mapping on the hit segment and return the start and end time periods of the hit segment in the audio to be identified;

[0064] S504. The recognition result is the advertising samples corresponding to all hit segments in the audio to be recognized and the start and end time periods.

[0065] Specifically, the preset threshold in step S501 can be flexibly set according to the actual situation. If the current advertising sample has no candidate hit area, it means that there is no advertising segment in the audio to be identified that matches the current advertising sample. If the current advertising sample has a unique candidate hit area, it means that there is an advertising segment in the audio to be identified that matches the current advertising sample. If the current advertising sample has multiple candidate hit areas, it means that there are multiple advertising segments in the audio to be identified that match the current advertising samples. The most matching advertising segment is found through step S502. Then, based on the start and end index of the hit segment in the frame-level feature sequence, combined with the sampling rate and frame shift, the start and end time of the segment in the original audio is mapped. All matching advertising sample IDs and the start and end time periods in the audio to be identified are combined to form the final identification result.

[0066] The method for identifying advertisements in audio provided in this embodiment first constructs a classified and stored advertisement sample library, then processes the frame-level feature sequence of the audio to be identified into subsequences that match the scale of each advertisement sample in the three feature sets, and then finds the subsequence that best matches the advertisement sample through the hit determination rule, and obtains the recognition result after time mapping, thus realizing the accurate identification of advertisement segments of various durations in the audio.

[0067] In some embodiments, it also includes:

[0068] S6. Dynamically expand the time boundary of the hit segment by extending the hit segment forward and backward with a fixed step size, and recalculate the comparison score. If the comparison score increases, continue to extend until the score decreases, so as to update the hit segment.

[0069] Example 2

[0070] This embodiment provides a device for identifying advertisements in audio, such as... Figure 2 As shown, it includes:

[0071] The sample library construction module is used to construct an advertising sample library, which stores advertising samples;

[0072] The acquisition module is used to acquire the audio to be recognized and perform preprocessing.

[0073] The feature extraction module is used to extract features from the preprocessed audio to be recognized, and obtain a frame-level feature sequence.

[0074] The feature comparison module is used to extract subsequences from the frame-level feature sequence using a sliding window, and compare the subsequences with the advertising samples in the sample library to obtain a comparison score.

[0075] The recognition module is used to determine the hit score and locate the time based on the comparison score, so as to obtain the recognition result of the advertisement in the audio to be recognized.

[0076] Example 3

[0077] This embodiment provides an electronic device, such as... Figure 3 As shown, it includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method for identifying advertisements in audio as described in any of the above embodiments.

[0078] Example 4

[0079] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, it implements the method for identifying advertisements in audio as described in any of the above embodiments.

[0080] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0081] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0082] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0083] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for identifying advertisements in audio, characterized in that, include: Construct an advertising sample library, which stores advertising samples; Acquire the audio to be recognized and perform preprocessing; Features of the preprocessed audio to be identified are extracted to obtain a frame-level feature sequence; A subsequence is extracted from the frame-level feature sequence using a sliding window, and the subsequence is compared with the feature of an advertisement sample in the sample library to obtain a comparison score. Based on the comparison score, a hit determination and time location are performed to obtain the recognition result of the advertisement in the audio to be identified.

2. The method for identifying advertisements in audio according to claim 1, characterized in that, The construction of the advertising sample library includes: Collect the original audio, and then resample, denoise, and normalize it. The processed raw audio is divided into three categories according to audio duration: short, medium, and long. Different sizes of first, second, and third sliding windows are set for short, medium, and long audio audio. Three sliding windows were used to extract feature sequences of the processed original audio from three types of audio, which were then used as advertising samples. All advertising samples were divided into the first feature set, the second feature set, and the third feature set according to their corresponding audio classification.

3. The method for identifying advertisements in audio according to claim 2, characterized in that, Extracting subsequences from the frame-level feature sequence using a sliding window includes: The first sliding window is used to extract several first subsequences of equal length to each advertisement sample in the first feature set from the frame-level feature sequence. The second sliding window is used to extract several second subsequences of equal length to each advertisement sample in the second feature set from the frame-level feature sequence. A third sliding window is used to extract several third subsequences of equal length to each advertisement sample in the third feature set from the frame-level feature sequence.

4. The method for identifying advertisements in audio according to claim 3, characterized in that, The step of comparing the subsequence with the advertising samples in the sample library includes: Calculate the similarity between the feature vector of each advertisement sample in the first feature set and the feature vector in the corresponding first subsequence in turn, and take the average value as the comparison score between the current first subsequence and the current advertisement sample; Calculate the similarity between the feature vector of each advertisement sample in the second feature set and the feature vector in the corresponding second subsequence in turn, and take the average value as the comparison score between the current second subsequence and the current advertisement sample; Calculate the similarity between the feature vector of each advertisement sample in the third feature set and the feature vector in the corresponding third subsequence in turn, and take the average value as the comparison score between the current third subsequence and the current advertisement sample.

5. The method for identifying advertisements in audio according to claim 4, characterized in that, The similarity is cosine similarity or Euclidean distance.

6. The method for identifying advertisements in audio according to claim 1, characterized in that, The process of determining hit rate and locating time based on comparison scores includes: If the comparison score is greater than the preset threshold, the current subsequence is marked as a candidate hit region of the current advertisement sample; If the current ad sample has multiple candidate hit regions, the subsequence with the highest alignment score is taken as the hit segment of the current ad sample; Perform time mapping on the hit segment to return the start and end time periods of the hit segment in the audio to be identified; The recognition result is defined as the advertising samples corresponding to all hit segments in the audio to be identified, along with the start and end time periods.

7. The method for identifying advertisements in audio according to claim 6, characterized in that, Also includes: The time boundary of the hit segment is dynamically expanded by extending the hit segment forward and backward with a fixed step size, and the comparison score is recalculated. If the comparison score increases, the extension continues until the score decreases, so as to update the hit segment.

8. A device for identifying advertisements in audio, characterized in that, The device includes: The sample library construction module is used to construct an advertising sample library, which stores advertising samples; The acquisition module is used to acquire the audio to be recognized and perform preprocessing. The feature extraction module is used to extract features from the preprocessed audio to be recognized, and obtain a frame-level feature sequence. The feature comparison module is used to extract subsequences from the frame-level feature sequence using a sliding window, and compare the subsequences with the advertising samples in the sample library to obtain a comparison score. The recognition module is used to determine the hit score and locate the time based on the comparison score, so as to obtain the recognition result of the advertisement in the audio to be recognized.

9. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method for identifying advertisements in audio as described in any one of claims 1-7.

10. A storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for identifying advertisements in audio as described in any one of claims 1-7.

Images