A method and system for intelligent monitoring of television advertisements

Abstract

This invention discloses an intelligent monitoring method and system for television advertisements, primarily relating to the field of television advertisement monitoring technology. It includes: extracting and storing keyframe features from all advertisement videos to construct an advertisement library; real-time decoding of television channel video streams, extracting image features and matching them with the advertisement library; if the same advertisement matches three different keyframes within 20 seconds, a coarse match is considered successful; for advertisements with successful coarse matches, extracting recorded video and performing frame-level alignment with the advertisement video, verifying the advertisement playback result through image similarity and audio similarity. The beneficial effect of this invention is that it enables rapid comparison of advertisements while improving the accuracy of the comparison.

Description

Technical Field

[0001] This invention relates to the field of television advertising monitoring technology, specifically an intelligent monitoring method and system for television advertisements. Background Technology

[0002] With the development of the media landscape, television remains an indispensable platform for brand promotion and product marketing. Therefore, accurate and efficient monitoring, identification, and analysis of public service announcements and commercial advertisements broadcast on television channels have significant social and market value.

[0003] Currently, most automated advertising monitoring relies on audio feature comparison, such as the solutions disclosed in patent documents CN102799605A, CN109712642A, and CN104254002A, which can identify advertisements in audio broadcasts. However, in the television field, the visual portion of the advertisement is also very important, and audio recognition alone cannot meet the requirements. CN113191661B uses histogram statistics and perceptual hash values ​​of images for comparison, assuming that the advertisement is just an image. If used in television video advertising scenarios, the accuracy will be significantly reduced. CN107426585A is applied to television advertising videos and combines image and audio features during recognition, but it only uses the first three frames and the last three frames. This means that if the beginning and end of the advertisement are incomplete, it is difficult to identify. In addition, it trains an SVM model for classification, which requires retraining if new advertisements are added, which is cumbersome.

[0004] Therefore, there is an urgent need for an intelligent monitoring method and system for television advertising to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent monitoring method and system for television advertisements, which can quickly compare advertisements while improving the accuracy of the comparison.

[0006] To achieve the above objectives, the present invention employs the following technical solution:

[0007] On the one hand, an intelligent monitoring method for television advertisements is provided, including the following steps:

[0008] Step S1: Extract and store the keyframe features of all advertising videos to build an advertising library;

[0009] Step S2: Decode the TV channel video stream in real time, extract image features and match them with the advertising database. If the same advertisement matches 3 different keyframes within 20 seconds, the coarse match is considered successful.

[0010] Step S3: For ads that have successfully matched coarsely, extract the recorded video and perform frame-level alignment with the ad video, and verify the ad playback results through image similarity and audio similarity.

[0011] Preferably, in step S1, extracting the keyframe features of all advertising videos specifically involves:

[0012] Extract the first 20 seconds of the advertisement video and evenly extract 30 frames of images.

[0013] Calculate the grayscale histogram of each frame and select the 15 frames with the smallest total number of pixels with 10 consecutive grayscale values ​​as keyframes.

[0014] For each keyframe, calculate the perceptual hash value and differential hash value respectively.

[0015] Preferably, the calculation steps for the perceived hash value are as follows:

[0016] Scale the image to 32×32 pixels and convert it to grayscale;

[0017] Apply the discrete cosine transform and take the 8×8 coefficient matrix in the upper left corner;

[0018] Compare the coefficients with the mean to generate a 64-bit binary hash value.

[0019] Preferably, the calculation steps for the differential hash value are as follows:

[0020] Scale the image to 9×8 pixels and convert it to grayscale;

[0021] Compare the grayscale values ​​of adjacent pixels row by row to generate a 64-bit binary hash value.

[0022] Preferably, the coarse matching includes:

[0023] Maintain a matching queue for each advertisement, recording the keyframe index and decoding time within 20 seconds;

[0024] Calculate the perceptual hash value Hamming distance and differential hash value Hamming distance between channel images and keyframes in the ad library;

[0025] If the perceptual hash distance is less than 6 and the differential hash distance is less than 20, then record the matching result.

[0026] Preferably, the fine comparison includes:

[0027] Alignment position in recorded video is located based on the minimum index keyframe of the matching.

[0028] The similarity between keyframes and recorded frames is calculated using the SSIM algorithm. If the maximum similarity is greater than 0.7, the alignment is considered successful.

[0029] After alignment, the images and audio are compared frame by frame.

[0030] Preferably, the audio similarity calculation includes:

[0031] Extract MFCC features from the two audio streams;

[0032] Calculate the Pearson correlation coefficient from the time-series vector of the L-order cepstral coefficients;

[0033] If the mean correlation coefficient is less than the threshold, then time misalignment recalculation is performed.

[0034] Preferably, false alarm filtering is also included:

[0035] If, in a fine comparison, there are more than 50% of inconsistent frames within 2 consecutive seconds and the audio similarity is less than 0.5, then the coarse match is determined to be a false alarm.

[0036] On the other hand, a monitoring system based on an intelligent monitoring method for television advertising is provided, comprising:

[0037] Ad library building module: used to extract and store keyframe features from ad videos;

[0038] Real-time matching module: decodes the channel video stream, executes coarse matching logic, and triggers fine matching;

[0039] Fine-grained comparison module: performs frame-level alignment and multimodal verification on ads that have successfully matched coarsely;

[0040] Results output module: Generates reports on ad playback time, completeness, and accuracy.

[0041] Preferably, the real-time matching module further includes:

[0042] Video slicing recording unit: Stores channel video streams in 10-second increments;

[0043] Dynamic queue management unit: Maintains a matching record queue for each advertisement with a 20-second sliding window.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. This invention provides an intelligent monitoring method for television advertisements. First, keyframe features are extracted from all advertisement videos to construct an advertisement library. Then, the video stream of the television channel is decoded in real time, image features are extracted, and matched with the keyframe features in the advertisement library. If three keyframes of an advertisement are matched within 20 seconds, the advertisement is considered to have a coarse match. If the coarse match is successful, the recorded video and the advertisement video are compared in detail to obtain the comparison result.

[0046] 2. This invention improves the accuracy of ad recognition with minimal efficiency loss by using a cascaded mode that combines coarse matching with fine matching, and the final recognition results are more accurate and richer. Attached Figure Description

[0047] Figure 1 This is a flowchart of the method of the present invention;

[0048] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0049] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined in this application.

[0050] In this invention, terms such as "upper," "lower," "left," "right," "front," "back," "vertical," "horizontal," "side," and "bottom" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used only to facilitate the description of the structural relationships of the various components or elements of this invention and do not specifically refer to any component or element in this invention. They should not be construed as limiting the invention.

[0051] Example:

[0052] like Figure 1 As shown, this embodiment provides an intelligent monitoring method for television advertisements, including the following steps:

[0053] Step S1: Extract and store the keyframe features of all advertising videos to build an advertising library;

[0054] Step S2: Decode the TV channel video stream in real time, extract image features and match them with the advertising database. If the same advertisement matches 3 different keyframes within 20 seconds, the coarse match is considered successful.

[0055] Step S3: For ads that have successfully matched coarsely, extract the recorded video and perform frame-level alignment with the ad video, and verify the ad playback results through image similarity and audio similarity.

[0056] Specifically, step S1 is as follows:

[0057] First, obtain the advertisement video. If the video is longer than 20 seconds, select the first 20 seconds of the video, and evenly extract 30 images from the video and save them to a list.

[0058] Then, calculate the grayscale histogram for each image in the list. The grayscale histogram is actually an array of length 256, where the index corresponds to the grayscale value and the element value is the number of pixels in the image that are equal to that grayscale value. Calculate the maximum total number of pixels for 10 consecutive grayscale values, sort them according to the total number, and select the 15 with the smallest number as the keyframes.

[0059] The perceptual hash value and differential hash value of each keyframe are calculated using perceptual hashing and differential hashing algorithms, which constitute the keyframe features of an advertisement video. Therefore, the hash value features of an advertisement form the advertisement library.

[0060] Step S2, specifically:

[0061] First, each ad has a queue that records the decoding time of the matched images and the keyframe index, and only retains the results from the most recent 20 seconds.

[0062] The system retrieves the video stream and saves video slices. Simultaneously, it decodes the video stream to obtain images. Every few frames, one image is extracted, and its perceptual hash value and differential hash value are calculated. These are then compared with relevant feature values ​​in the ad library to calculate the Hamming distance. If one distance is less than 6 and the other is less than 20, a match is considered successful. Multiple ad keyframes may be matched; the image decoding time and keyframe index are recorded in the corresponding ad queue.

[0063] For an ad that matches a keyframe, check if there are 3 different keyframe indices in its queue. If so, the coarse match is considered successful; otherwise, continue matching.

[0064] The Hamming distance is calculated as follows:

[0065] The two hash values ​​are XORed bitwise, and the number of 1s in the result (i.e. the number of different bits) is counted. Therefore, the smaller the distance, the higher the image similarity.

[0066] Step S3 is as follows:

[0067] First, based on the decoding time of the matched image, the approximate location of the video where the advertisement is recorded is determined and extracted.

[0068] Then, the smallest index keyframe that is matched is selected, the recorded video is decoded to obtain an image, and the similarity between the decoded image and the keyframe is calculated using the SSIM algorithm to locate the position of the keyframe in the recorded video. In addition, the position of the keyframe is known in the advertisement video, so the recorded video and the advertisement video can be accurately aligned.

[0069] After alignment, images are compared frame by frame using the SSIM algorithm, and audio is compared using Mel-Cepstral Coefficient and Pearson Correlation Coefficient. If the number of inconsistent images exceeds a certain limit within two seconds, and the audio similarity is below a threshold, the coarsely matched advertisement is considered a false alarm, and no further comparison is performed.

[0070] The SSIM algorithm, specifically:

[0071] The Structural Similarity Index (SSIM) algorithm calculates the similarity between two images. SSIM not only technically reflects the pixel differences between images but also reflects human visual perception, providing a comprehensive evaluation of factors such as brightness, contrast, and structure. The SSIM calculation formula is as follows:

[0072]

[0073] in:

[0074] and These are the average pixel values ​​for images x and y, respectively;

[0075] and denoted as the pixel variances of images x and y, respectively;

[0076] The covariance between the x and y values ​​of the image;

[0077] and These are two constants used to stabilize the denominator, which may be close to zero.

[0078] The specific steps are as follows:

[0079] (1) To reduce the amount of computation, the two images are scaled up by a certain ratio;

[0080] (2) Convert the image to YUV format and extract the Y component to obtain a two-dimensional matrix;

[0081] (3) Use an 11x11 sliding window to slide on the Y component, calculate the SSIM value, and then take the average as a similarity measure between the two images.

[0082] The audio similarity calculation is as follows:

[0083] Audio similarity is calculated based on Mel-frequency cepstral coefficients (MFCC) features and Pearson correlation coefficients. The specific steps are as follows:

[0084] First, the audio data is preprocessed by dividing it into short time frames, each with a duration of 20-40 milliseconds;

[0085] Applying a Fast Fourier Transform (FFT) to each frame of audio data converts the time-domain signal into a frequency-domain signal;

[0086] In the frequency domain, the audio signal is passed through a set of Mel filters, and the logarithmic energy of the output of each filter bank is calculated.

[0087] Substituting the aforementioned logarithmic energy into the Discrete Cosine Transform (DCT), the cepstral coefficients of order L are obtained;

[0088] By combining the cepstral coefficients of the same order from all short-time audio frames, L temporal feature vectors are obtained.

[0089] Smooth the L time-series feature vectors, calculate the Pearson correlation coefficient, and take the mean as the final similarity.

[0090] The audio from the two video streams may have some synchronization differences with the video, causing a delay between the two sets of audio data, which will affect the calculation of audio similarity. If the audio similarity is less than a threshold, the Pearson correlation coefficient is calculated in a staggered manner; if it is greater than the threshold, the audio is still considered to be consistent.

[0091] Based on the above method, this embodiment will further illustrate the method using the advertising monitoring of a certain TV station as an example:

[0092] 1. Based on the advertisement video provided by the advertiser, extract the first 20 seconds of the video (or the entire video if it is less than 20 seconds). Evenly extract 30 frames of images. Calculate the grayscale histogram for each frame, then calculate the maximum value of 10 consecutive grayscale pixel values. Sort the images according to this value and select the 15 smallest images as keyframes. Calculate the perceptual hash value and differential hash value for each keyframe. Combine the keyframe image, the keyframe's position in the advertisement video, the perceptual hash value, and the differential hash value to form feature data. Sort all keyframe feature data according to their position in the advertisement video and place them in a list. Store this list and the advertisement video in the advertisement database.

[0093] 2. At the start of the ad monitoring program, a queue is created for each ad to record the decoding time of the matched image and the keyframe index, retaining only the results from the most recent 20 seconds. Then, a video stream from a specific station is retrieved, recording short segments of 10 seconds each, which are then decoded into images. Every three frames are extracted, and the perceptual hash value and differential hash value are calculated. For each ad, every feature data in the data list is traversed, calculating the Hamming distance between perceptual hash values ​​and the Hamming distance between differential hash values. If one is less than 6 and the other less than 20, a match is considered successful. If multiple matches are found, the one with the smallest Hamming distance is used, and the image decoding time and the keyframe index are stored in the ad's queue. The ad's queue is then analyzed; if three different keyframe indices exist in the queue, a coarse match is considered successful.

[0094] 3. If the coarse match is successful, proceed to the fine comparison procedure.

[0095] First, based on the image decoding time corresponding to the smallest index keyframe and its position in the ad video, the position of the ad's starting frame in the recorded video is roughly located, and 5 seconds of video are extracted before and after it.

[0096] Then, the smallest index keyframe that was matched is selected, and the recorded video is decoded to obtain an image. The similarity between the decoded image and the keyframe is calculated using the SSIM algorithm. The frame with the highest similarity is selected. If the maximum similarity is greater than 0.7, the match is considered successful; otherwise, the match is considered unsuccessful. If the match fails, the next keyframe is selected, and the above process is repeated. If all matches fail, the coarse matching result is considered a false alarm, and the fine comparison program is exited. If the match is successful, the position of the keyframe in the recorded video can be located. In addition, the position of the keyframe is known in the advertisement video, thus achieving precise alignment between the recorded video and the advertisement video.

[0097] After alignment, images are compared frame by frame using the SSIM algorithm, and audio is compared using Mel-Cepstral Coefficient and Pearson Correlation Coefficient. If more than 50% of the images are inconsistent within two seconds, and the audio similarity is less than 0.5, the coarse-matched ad is considered a false alarm, and the fine-tuning process is terminated. Otherwise, the coarse-match result is considered correct, and the comparison of images and audio continues. The comparison results can be analyzed to determine the ad's start and end times, completeness, and accuracy.

[0098] like Figure 2 As shown, this embodiment also provides an intelligent monitoring system for television advertisements, including:

[0099] Ad library building module: used to extract and store keyframe features from ad videos;

[0100] Real-time matching module: decodes the channel video stream, executes coarse matching logic, and triggers fine matching;

[0101] Fine-grained comparison module: performs frame-level alignment and multimodal verification on ads that have successfully matched coarsely;

[0102] Results output module: Generates reports on ad playback time, completeness, and accuracy.

[0103] The real-time matching module also includes:

[0104] Video slicing recording unit: Stores channel video streams in 10-second increments;

[0105] Dynamic queue management unit: Maintains a matching record queue for each advertisement with a 20-second sliding window.

[0106] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An intelligent monitoring method for television advertisements, characterized in that, Includes the following steps: Step S1: Extract and store the keyframe features of all advertising videos to build an advertising library; Step S2: Decode the TV channel video stream in real time, obtain images, extract image features and match them with the advertising database. If the same advertisement matches 3 different keyframes within 20 seconds, the coarse match is considered successful. Step S3: For ads that have successfully matched coarsely, proceed to fine comparison and verify the ad playback results through image similarity and audio similarity. In step S1, keyframe features of all advertising videos are extracted, specifically as follows: Extract the first 20 seconds of the advertisement video and evenly extract 30 frames of images. Calculate the grayscale histogram of each frame image, obtain the maximum value of the total number of pixels with 10 consecutive grayscale values ​​in each frame image, sort the 30 extracted frames based on the maximum value, and select the 15 frames with the smallest values ​​as keyframes. For each keyframe, calculate the perceptual hash value and differential hash value separately; The calculation steps for the perceptual hash value are as follows: Scale the image to 32×32 pixels and convert it to grayscale; Apply the discrete cosine transform and take the 8×8 coefficient matrix in the upper left corner; Compare the coefficients and mean of the 8×8 coefficient matrix to generate a 64-bit binary hash value; The steps for calculating the differential hash value are as follows: Scale the image to 9×8 pixels and convert it to grayscale; Compare the grayscale values ​​of adjacent pixels row by row to generate a 64-bit binary hash value; In step S2, the coarse matching includes: Maintain a matching queue for each advertisement, recording the keyframe index and decoding time within 20 seconds; Calculate the perceptual hash value Hamming distance and differential hash value Hamming distance between channel images and keyframes in the ad library; If the perceptual hash distance is less than 6 and the differential hash distance is less than 20, then record the matching result; In step S3, the detailed comparison includes: The alignment position in the recorded video is located based on the minimum index keyframe of the matching, the recorded video is extracted and decoded into an image, and the recorded video is specifically recorded as a video slice by pulling the video stream of the TV channel. The similarity between the minimum index keyframe and the decoded image is calculated using the SSIM algorithm. If the maximum similarity is greater than 0.7, the alignment is considered successful. After alignment, image similarity is calculated by comparing images frame by frame, and audio similarity is calculated by comparing audio. Step S2 also includes: false alarm filtering, specifically: If, in a fine comparison, more than 50% of the images are inconsistent within 2 consecutive seconds and the audio similarity is less than 0.5, then the coarse matching is determined to be a false alarm, and the false alarm information is filtered and deleted. Audio similarity calculation includes: The audio data is preprocessed by dividing it into short time frames, each with a duration of 20-40 milliseconds; Applying a Fast Fourier Transform (FFT) to each frame of audio data converts the time-domain signal into a frequency-domain signal; In the frequency domain, the audio signal is passed through a set of Mel filters, and the logarithmic energy of the output of each filter bank is calculated. Substituting the aforementioned logarithmic energy into the Discrete Cosine Transform (DCT), the cepstral coefficients of order L are obtained; By combining the cepstral coefficients of the same order from all short-time audio frames, L temporal feature vectors are obtained. Smooth the L time-series feature vectors respectively, calculate the Pearson correlation coefficient, and take the mean as the final similarity; If the mean Pearson correlation coefficient is less than the threshold, then time misalignment recalculation is performed.

2. An intelligent monitoring system for television advertisements, used to implement the intelligent monitoring method for television advertisements as described in claim 1, characterized in that, include: Ad library building module: used to extract and store keyframe features from ad videos; Real-time matching module: decodes TV channel video streams, executes coarse matching logic, and triggers fine matching; Fine-grained processing module: For ads that are successfully matched in the coarse phase, a fine-grained comparison is performed, and the ad playback results are verified through image similarity and audio similarity. Results output module: Generates reports on ad playback time, completeness, and accuracy.

3. The intelligent monitoring system for television advertising according to claim 2, characterized in that, The real-time matching module also includes: Video slicing and recording unit: Stores TV channel video streams in 10-second increments; Dynamic queue management unit: Maintains a matching record queue for each advertisement with a 20-second sliding window.

Images