A method, apparatus, device, and medium for identifying advertisements in audio

Abstract

The application relates to a method, device, equipment and medium for identifying advertisements in audio, belonging to the technical field of advertisement identification, and the method comprises the following steps: constructing an advertisement sample library, wherein the advertisement sample library stores advertisement samples; obtaining audio to be identified and performing pretreatment; extracting features of the pretreated audio to be identified to obtain a frame-level feature sequence; extracting a subsequence in the frame-level feature sequence by using a sliding window, and performing feature comparison between the subsequence and the advertisement samples in the sample library to obtain comparison scores; and performing hit determination and time positioning according to the comparison scores to obtain an identification result of the advertisements in the audio to be identified. The application realizes accurate identification of advertisement segments in the audio to be identified.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of advertisement recognition, and in particular relates to a method, device, equipment and medium for recognizing advertisements in audio. BACKGROUND

[0002] Advertisements play an increasingly important role in daily information transmission. With the acceleration of commercialization, advertisements are increasingly frequently presented on media platforms such as television, radio, the Internet and the like, and subtly influence people's lifestyles. However, advertisements also affect the user experience, especially when the user is listening to audio content, the sudden advertisements can cause the user to feel repulsed. Therefore, it is necessary to timely and effectively recognize the advertisement segments in the audio so as to facilitate the user to know in advance or skip. In the prior art, when recognizing advertisements in audio, the advertisement samples in the sample library are usually simply compared with the audio to be recognized, and there is a technical problem that the accuracy of the recognition result cannot be guaranteed. SUMMARY

[0003] In view of the above analysis, the embodiments of the present application aim to provide a method, device, equipment and medium for recognizing advertisements in audio, so as to solve the technical problem that the accuracy of the recognition result cannot be guaranteed in the prior art.

[0004] The purpose of the present application is achieved as follows:

[0005] The first aspect of the present application provides a method for recognizing advertisements in audio, comprising:

[0006] constructing an advertisement sample library, wherein the advertisement sample library stores advertisement samples;

[0007] obtaining audio to be recognized and performing preprocessing;

[0008] extracting features of the preprocessed audio to be recognized to obtain a frame-level feature sequence;

[0009] extracting a subsequence in the frame-level feature sequence by using a sliding window, and performing feature comparison between the subsequence and the advertisement samples in the sample library to obtain a comparison score;

[0010] performing hit determination and time positioning according to the comparison score to obtain a recognition result of the advertisements in the audio to be recognized.

[0011] Further, the constructing the advertisement sample library comprises: collecting original audio, resampling, denoising and normalizing the original audio; dividing the processed original audio into three categories of short, medium and long according to audio duration; setting different specifications of first, second and third sliding windows for the three categories of audio; using the three sliding windows to respectively extract feature sequences of the processed original audio in the three categories of audio as advertisement samples; and dividing all the advertisement samples into first, second and third feature sets according to the corresponding audio categories.

[0012] Further, the sliding extraction of the sub-sequences in the frame-level feature sequence using the sliding window comprises: using the first sliding window to slide and extract a plurality of first sub-sequences equal in length to each advertisement sample in the first feature set from the frame-level feature sequence; using the second sliding window to slide and extract a plurality of second sub-sequences equal in length to each advertisement sample in the second feature set from the frame-level feature sequence; and using the third sliding window to slide and extract a plurality of third sub-sequences equal in length to each advertisement sample in the third feature set from the frame-level feature sequence.

[0013] Further, the feature comparison of the sub-sequences with the advertisement samples in the sample library comprises: sequentially calculating the similarity between the feature vector of each advertisement sample in the first feature set and the feature vector in the corresponding first sub-sequence, and taking the average value as the comparison score of the current first sub-sequence and the current advertisement sample; sequentially calculating the similarity between the feature vector of each advertisement sample in the second feature set and the feature vector in the corresponding second sub-sequence, and taking the average value as the comparison score of the current second sub-sequence and the current advertisement sample; and sequentially calculating the similarity between the feature vector of each advertisement sample in the third feature set and the feature vector in the corresponding third sub-sequence, and taking the average value as the comparison score of the current third sub-sequence and the current advertisement sample.

[0014] Further, the similarity is cosine similarity or Euclidean distance.

[0015] Further, the hit determination and time positioning according to the comparison score comprises: if the comparison score is greater than a preset threshold, marking the current sub-sequence as a candidate hit region of the current advertisement sample; if there are multiple candidate hit regions for the current advertisement sample, taking the sub-sequence with the maximum comparison score as the hit segment of the current advertisement sample; performing time mapping on the hit segment to return the start and end time period of the hit segment in the audio to be recognized; and taking the advertisement sample corresponding to all hit segments in the audio to be recognized and the start and end time period as the recognition result.

[0016] Further, the time boundary of the hit segment is dynamically extended, the hit segment is extended forward and backward with a fixed step, and the alignment score is recalculated, if the alignment score is improved, the extension is continued until the score decreases, so as to update the hit segment.

[0017] The second aspect of the present application provides a device for identifying an advertisement in audio, comprising:

[0018] A sample library construction module is configured to construct an advertisement sample library, wherein the advertisement sample library stores advertisement samples.

[0019] An acquisition module is configured to acquire audio to be identified and perform preprocessing.

[0020] A feature extraction module is configured to extract features of the preprocessed audio to be identified to obtain a frame-level feature sequence.

[0021] A feature alignment module is configured to extract a sub-sequence in the frame-level feature sequence by using a sliding window, and perform feature alignment between the sub-sequence and the advertisement samples in the sample library to obtain an alignment score.

[0022] An identification module is configured to perform hit determination and time positioning according to the alignment score to obtain an identification result of the advertisement in the audio to be identified.

[0023] The third aspect of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to implement the method for identifying an advertisement in audio according to any one of the embodiments.

[0024] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method for identifying an advertisement in audio according to any one of the embodiments.

[0025] Compared with the prior art, the present application can achieve at least one of the following beneficial effects:

[0026] The method for identifying an advertisement in audio provided by the present application can accurately identify the advertisement segment in the audio to be identified by processing the audio to be identified into a frame-level feature sequence, extracting a sub-sequence of the frame-level feature sequence by using a sliding window, performing feature alignment between the sub-sequence and the advertisement samples in the sample library, and combining hit determination and time positioning. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present specification or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present specification, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0028] Figure 1 A flowchart of a method for identifying an advertisement in audio provided for Embodiment 1 of the present application;

[0029] Figure 2 A schematic diagram of an apparatus for identifying an advertisement in audio provided for Embodiment 2 of the present application;

[0030] Figure 3 A schematic diagram of an electronic device architecture provided for Embodiment 3 of the present application. DETAILED DESCRIPTION

[0031] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. It should be noted that the embodiments in the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged without conflict. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0032] Embodiment 1

[0033] One specific embodiment of the present application, as shown in Figure 1 discloses a method for identifying an advertisement in audio, comprising the following steps:

[0034] S1, constructing an advertisement sample library, wherein the advertisement sample library stores advertisement samples;

[0035] S2, obtaining audio to be identified and performing preprocessing;

[0036] S3, extracting features of the preprocessed audio to be identified to obtain a frame-level feature sequence;

[0037] S4, extracting a sub-sequence in the frame-level feature sequence using a sliding window, and performing feature comparison between the sub-sequence and the advertisement samples in the sample library to obtain a comparison score;

[0038] S5, performing hit determination and time positioning according to the comparison score to obtain an identification result of the advertisement in the audio to be identified.

[0039] Compared with the prior art, the method for identifying advertisements in audio provided by the embodiment can accurately identify the advertisement segments in the audio to be identified by processing the audio to be identified into a frame-level feature sequence, using a sliding window to intercept a sub-sequence of the frame-level feature sequence, comparing the sub-sequence with advertisement samples in a sample library, and combining hit determination and time positioning.

[0040] In the embodiment, the construction of the advertisement sample library in step S1 includes:

[0041] S101, collecting original audio, and performing resampling, denoising and normalization processing on the original audio;

[0042] S102, dividing the processed original audio into three categories of short, medium and long according to audio duration;

[0043] S103, setting different specifications of the first sliding window, the second sliding window and the third sliding window for the three categories of audio;

[0044] S104, using the three kinds of sliding windows to respectively extract the feature sequences of the processed original audio in the three categories of audio as advertisement samples, and dividing all the advertisement samples into a first feature set, a second feature set and a third feature set according to the corresponding audio classification.

[0045] For example, the original audio can come from advertisement segments of radio, podcast, audio platform, etc. to form original audio. Each advertisement is standardized, including resampling to a unified sampling rate, removing background noise, amplitude normalization processing, etc. Then all original audios are divided into three categories according to audio duration T, such as short advertisement (T<30s), medium advertisement (30s≤T<60s) and long advertisement (T≥60s). A sliding window is set for each type of audio, and then the corresponding feature sequence is extracted. Thus, each audio is extracted as a feature sequence as an advertisement sample, and all the advertisement samples are divided into a first feature set, a second feature set and a third feature set according to the corresponding audio classification rule, and then stored in the advertisement sample library.

[0046] In the embodiment, the preprocessing of the audio to be identified in step S2 is similar to the processing of the original audio described above.

[0047] Specifically, the feature extraction of the original audio in step S1 and the feature extraction of the audio to be identified in step S3 can both use the Mel spectrum. First, the audio signal is converted into frequency domain features that can describe the audio content, such as short-time Fourier transform. Further, sample vectorization is performed by statistical methods, such as MFCC (Mel Frequency Cepstral Coefficient), to generate low-dimensional feature vectors, which are combined into feature sequences.

[0048] In this embodiment, the step S4 uses a sliding window to extract a subsequence from the frame-level feature sequence, comprising:

[0049] S401, using a first sliding window to extract a plurality of first subsequences equal to the length of each advertisement sample in the first feature set from the frame-level feature sequence;

[0050] S402, using a second sliding window to extract a plurality of second subsequences equal to the length of each advertisement sample in the second feature set from the frame-level feature sequence;

[0051] S403, using a third sliding window to extract a plurality of third subsequences equal to the length of each advertisement sample in the third feature set from the frame-level feature sequence.

[0052] Specifically, the first, second and third sliding windows used have a fixed size, but the step size can be set according to actual conditions. Assuming that the first feature set S1 includes three advertisement samples, denoted as S1 = {O1, O2, O3}, and including 4, 5 and 6 feature vectors respectively, denoted as The frame-level feature sequence consists of 20 feature vectors, denoted as Z = [z1, z2,..., z 20 ], and the first sliding window is used to extract a plurality of first subsequences equal to the length of O1, O2 and O3 from the frame-level feature sequence Z. The extraction of the second and third subsequences is the same as the extraction of the first subsequences, and will not be described again.

[0053] In this embodiment, the step S4 compares the subsequence with the advertisement sample in the sample library, comprising:

[0054] S404, calculating the similarity between the feature vectors of each advertisement sample in the first feature set and the feature vectors in the corresponding first subsequence, and taking the average value as the comparison score of the current first subsequence and the current advertisement sample;

[0055] Specifically, the advertisement sample corresponding to the first subsequence has 17, denoted as [z k , z k+1 , z k+2 , z k+3 ], 1 ≤ k ≤ 17, and the feature vectors z k in each first subsequence are calculated. z k+1 , z k+2 , z k+3 , the average value of the four similarities is taken as the comparison score of the first sub-sequence and the advertisement sample O1 under the current k value; similarly, the comparison scores of other advertisement samples in the first feature set are calculated in the same way.

[0056] S405, the similarity between the feature vector of each advertisement sample in the second feature set and the corresponding feature vector in the second sub-sequence is calculated in sequence, and the average value is taken as the comparison score of the current second sub-sequence and the current advertisement sample.

[0057] S406, the similarity between the feature vector of each advertisement sample in the third feature set and the corresponding feature vector in the third sub-sequence is calculated in sequence, and the average value is taken as the comparison score of the current third sub-sequence and the current advertisement sample.

[0058] Specifically, the comparison scores of the advertisement samples and the sub-sequences in the second feature set and the third feature set are the same as the principle of step S404.

[0059] In this embodiment, the similarity is cosine similarity or Euclidean distance.

[0060] In this embodiment, the hit determination and time positioning according to the comparison score in step S5 include:

[0061] S501, if the comparison score is greater than a preset threshold, the current sub-sequence is marked as a candidate hit region of the current advertisement sample;

[0062] S502, if there are multiple candidate hit regions for the current advertisement sample, the sub-sequence with the maximum comparison score is taken as the hit segment of the current advertisement sample;

[0063] S503, time mapping is performed on the hit segment, and the start and end time period of the hit segment in the audio to be recognized is returned;

[0064] S504, the advertisement sample corresponding to all hit segments in the audio to be recognized and the start and end time period are taken as the recognition result.

[0065] Specifically, the preset threshold in step S501 can be flexibly set according to actual conditions. If there is no candidate hit region for the current advertisement sample, it indicates that there is no advertisement segment in the audio to be recognized that matches the current advertisement sample. If there is only one candidate hit region for the current advertisement sample, it indicates that there is an advertisement segment in the audio to be recognized that matches the current advertisement sample. If there are multiple candidate hit regions for the current advertisement sample, it indicates that there are multiple advertisement segments in the audio to be recognized that match the multiple current advertisement samples. The most matched advertisement segment is found through step S502. Then, according to the start and end indexes of the hit segment in the frame-level feature sequence, combined with the sampling rate and frame shift, the start and end times of the segment in the original audio are mapped, and all matched advertisement sample IDs and the start and end time periods in the audio to be recognized are combined to form the final recognition result.

[0066] The method for identifying advertisements in audio provided in this embodiment first constructs a classified and stored advertisement sample library, then processes the frame-level feature sequence of the audio to be recognized into sub-sequences that match the dimensions of each advertisement sample in the three feature sets, then finds the most matched sub-sequence to the advertisement sample through a hit determination rule, and obtains a recognition result after time mapping, thereby accurately identifying various length advertisement segments in audio.

[0067] In some embodiments, the method further comprises:

[0068] S6, dynamically expanding the time boundary of the hit segment, extending the hit segment forward and backward with a fixed step, and recalculating the comparison score. If the comparison score improves, continue to extend until the score decreases, to update the hit segment.

[0069] Embodiment 2

[0070] This embodiment provides a device for identifying advertisements in audio, as shown in Figure 2 The device comprises:

[0071] A sample library construction module is configured to construct an advertisement sample library, and the advertisement sample library stores advertisement samples.

[0072] An acquisition module is configured to acquire an audio to be recognized and perform preprocessing.

[0073] A feature extraction module is configured to extract features of the preprocessed audio to be recognized, to obtain a frame-level feature sequence.

[0074] A feature comparison module is configured to extract a sub-sequence in the frame-level feature sequence using a sliding window, and compare the sub-sequence with the advertisement samples in the sample library to obtain a comparison score.

[0075] An identification module is configured to determine a hit and locate a time according to the comparison score, to obtain an identification result of an advertisement in the audio to be recognized.

[0076] Embodiment 3

[0077] This embodiment provides an electronic device, such as Figure 3 As shown, comprising a memory and a processor, the memory stores a computer program, the computer program is executed by the processor to implement the method of identifying an advertisement in audio as claimed in any one of the above embodiments.

[0078] Embodiment 4

[0079] This embodiment provides a computer readable storage medium, which stores a computer program, the program is executed by a processor to implement the method of identifying an advertisement in audio as claimed in any one of the above embodiments.

[0080] The computer readable storage medium includes permanent and non-permanent, removable and non-removable media, which can be realized by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. According to the definition in this paper, computer readable medium does not include transitory computer readable medium, such as modulated data signal and carrier wave.

[0081] The skilled person should also further realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in the above description in general terms. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0082] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and

[0083] The above detailed description describes the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for identifying advertisements in audio, characterized in that, include: Construct an advertising sample library, which stores advertising samples; The construction of the advertising sample library includes: collecting raw audio; resampling, denoising, and normalizing the raw audio; dividing the processed raw audio into three categories: short, medium, and long, according to audio duration; setting different sizes of first, second, and third sliding windows for the three categories of short, medium, and long audio; using the three sliding windows to extract feature sequences of the processed raw audio from the three categories of audio respectively, as advertising samples; and dividing all advertising samples into a first feature set, a second feature set, and a third feature set according to the corresponding audio classification. Acquire the audio to be recognized and perform preprocessing; Features of the preprocessed audio to be identified are extracted to obtain a frame-level feature sequence; A subsequence is extracted from the frame-level feature sequence using a sliding window, and the subsequence is compared with the feature of an advertisement sample in the sample library to obtain a comparison score. Based on the comparison score, a hit determination and time location are performed to obtain the recognition result of the advertisement in the audio to be identified.

2. The method for identifying advertisements in audio according to claim 1, characterized in that, Extracting subsequences from the frame-level feature sequence using a sliding window includes: The first sliding window is used to extract several first subsequences of equal length to each advertisement sample in the first feature set from the frame-level feature sequence. The second sliding window is used to extract several second subsequences of equal length to each advertisement sample in the second feature set from the frame-level feature sequence. Using a third sliding window, extract several third subsequences of equal length to each advertisement sample in the third feature set from the frame-level feature sequence.

3. The method for identifying advertisements in audio according to claim 2, characterized in that, The step of comparing the subsequence with the advertising samples in the sample library includes: Calculate the similarity between the feature vector of each advertisement sample in the first feature set and the feature vector in the corresponding first subsequence in turn, and take the average value as the comparison score between the current first subsequence and the current advertisement sample; Calculate the similarity between the feature vector of each advertisement sample in the second feature set and the feature vector in the corresponding second subsequence in turn, and take the average value as the comparison score between the current second subsequence and the current advertisement sample; Calculate the similarity between the feature vector of each advertisement sample in the third feature set and the feature vector in the corresponding third subsequence in turn, and take the average value as the comparison score between the current third subsequence and the current advertisement sample.

4. The method for identifying advertisements in audio according to claim 3, characterized in that, The similarity is cosine similarity or Euclidean distance.

5. The method for identifying advertisements in audio according to claim 1, characterized in that, The process of determining hit rate and locating time based on comparison scores includes: If the comparison score is greater than the preset threshold, the current subsequence is marked as a candidate hit region of the current advertisement sample; If the current ad sample has multiple candidate hit regions, the subsequence with the highest alignment score is taken as the hit segment of the current ad sample; Perform time mapping on the hit segment to return the start and end time periods of the hit segment in the audio to be identified; The recognition result is defined as the advertising samples corresponding to all hit segments in the audio to be identified, along with the start and end time periods.

6. The method for identifying advertisements in audio according to claim 5, characterized in that, Also includes: The time boundary of the hit segment is dynamically expanded by extending the hit segment forward and backward with a fixed step size, and the comparison score is recalculated. If the comparison score increases, the extension continues until the score decreases, so as to update the hit segment.

7. A device for identifying advertisements in audio, characterized in that, The device includes: A sample library construction module is used to construct an advertising sample library, which stores advertising samples. The construction of the advertising sample library includes: collecting raw audio; resampling, denoising, and normalizing the raw audio; classifying the processed raw audio into short, medium, and long categories based on audio duration; setting different sizes of first, second, and third sliding windows for the short, medium, and long audio categories; using the three sliding windows to extract feature sequences from the processed raw audio in the three categories respectively, as advertising samples; and classifying all advertising samples into a first feature set, a second feature set, and a third feature set according to their corresponding audio categories. The acquisition module is used to acquire the audio to be recognized and perform preprocessing. The feature extraction module is used to extract features from the preprocessed audio to be recognized, and obtain a frame-level feature sequence. The feature comparison module is used to extract subsequences from the frame-level feature sequence using a sliding window, and compare the subsequences with the advertising samples in the sample library to obtain a comparison score. The recognition module is used to determine the hit score and locate the time based on the comparison score, so as to obtain the recognition result of the advertisement in the audio to be recognized.

8. An electronic device, characterized in that, The device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method for identifying advertisements in audio as described in any one of claims 1-6.

9. A storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for identifying advertisements in audio as described in any one of claims 1-6.

Images