Television advertisement intelligent monitoring method and system

Abstract

The invention discloses an intelligent broadcast monitoring method and system for television advertisements, and mainly relates to the technical field of television advertisement broadcast monitoring. Comprising the steps of extracting and storing key frame features of all advertisement videos, and constructing an advertisement library; decoding a television channel video stream in real time, extracting picture features to be matched with the advertisement library, and if the same advertisement is matched with three different key frames within 20 seconds, judging that rough matching is successful; and for the advertisement which is successfully roughly matched, intercepting the recorded video, carrying out frame-level alignment on the recorded video and the advertisement video, and verifying an advertisement playing result through image similarity and audio similarity. The beneficial effects of the invention are that the method can achieve the quick comparison of advertisements, and improves the comparison accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of television advertisement monitoring, and in particular to an intelligent monitoring method and system for television advertisements. Background Art

[0002] As the media landscape evolves, television remains an indispensable and crucial platform for brand and product promotion. Therefore, accurately and efficiently monitoring, identifying, and analyzing public service announcements and commercials broadcast on television channels holds significant social and market value.

[0003] Currently, most automated advertising monitoring relies on audio feature comparison, such as the solutions disclosed in patent documents such as CN102799605A, CN109712642A, and CN104254002A, which can identify advertisements in sound broadcasts. However, in the field of television, the picture part of the advertisement is also very important, and only identifying the audio cannot meet the needs. CN113191661B uses the histogram statistics and perceptual hash value of the image for comparison. Its default advertisement is a picture. If it is used in a TV video advertising scenario, the accuracy will be greatly reduced. The application scenario of CN107426585A is TV advertising video. It also combines image features and audio features during recognition, but it only uses the first three frames and the last three frames. That is to say, if the beginning and end of the advertisement are incomplete, it is difficult to recognize it. In addition, it trains the SVM model for classification. If new advertisements are added, it needs to be retrained, which is relatively cumbersome.

[0004] Therefore, there is an urgent need for an intelligent monitoring method and system for television advertisements to solve the above problems. Summary of the Invention

[0005] The object of the present invention is to provide an intelligent monitoring method and system for television advertisements, which can realize rapid comparison of advertisements while improving the accuracy of comparison.

[0006] To achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions: In one aspect, a method for intelligently monitoring television commercials is provided, comprising the following steps: Step S1: Extract and store key frame features of all advertising videos to build an advertising library; Step S2: Decode the TV channel video stream in real time, extract image features and match them with the advertisement library. If the same advertisement is matched to three different key frames within 20 seconds, the rough match is determined to be successful. Step S3: For advertisements that have been roughly matched successfully, the recorded video is captured and aligned with the advertisement video at the frame level, and the advertisement playback result is verified through image similarity and audio similarity.

[0007] Preferably, in step S1, key frame features of all advertising videos are extracted, specifically: Intercept the first 20 seconds of the ad video and evenly extract 30 frames; Calculate the grayscale histogram of each frame and select the 15 frames with the smallest total number of grayscale pixels in 10 consecutive frames as key frames; The perceptual hash value and differential hash value are calculated for each key frame.

[0008] Preferably, the steps for calculating the perceptual hash value are: Scale the image to 32×32 pixels and grayscale; Apply the discrete cosine transform and take the upper left 8×8 coefficient matrix; Compare the coefficients to the mean and generate a 64-bit binary hash value.

[0009] Preferably, the steps for calculating the differential hash value are: Resize the image to 9×8 pixels and grayscale; Compare the grayscale values ​​of adjacent pixels row by row to generate a 64-bit binary hash value.

[0010] Preferably, the rough matching includes: Maintain a matching queue for each ad, recording the key frame index and decoding time within 20 seconds; Calculate the perceptual hash value Hamming distance and differential hash value Hamming distance between the channel image and the key frames of the ad library; If the perceived hash distance is less than 6 and the differential hash distance is less than 20, the matching result is recorded.

[0011] Preferably, the detailed comparison includes: Locate the alignment position in the recorded video based on the matched minimum index keyframe; The similarity between the key frame and the recorded frame is calculated using the SSIM algorithm. If the maximum similarity is greater than 0.7, the alignment is considered successful. After alignment, compare the image and audio frame by frame.

[0012] Preferably, the audio similarity calculation includes: Extract MFCC features of two-channel audio; Calculate the Pearson correlation coefficient for the L-order cepstral coefficient time series vector; If the mean value of the correlation coefficient is less than the threshold, a time-shifted recalculation is performed.

[0013] Preferably, false positive filtering is also included: If the image inconsistency frames are greater than 50% and the audio similarity is less than 0.5 within 2 consecutive seconds in the fine comparison, the coarse match is considered a false alarm.

[0014] On the other hand, a monitoring system based on an intelligent monitoring method for television advertisements is provided, comprising: Ad library building module: used to extract and store key frame features of advertising videos; Real-time matching module: decodes channel video streams, executes coarse matching logic, and triggers fine matching; Fine matching module: performs frame-level alignment and multimodal verification on ads that have been roughly matched. Result output module: Generates advertisement playback time, completeness and correctness reports.

[0015] Preferably, the real-time matching module further includes: Video slice recording unit: stores channel video stream in 10-second units; Dynamic Queue Management Unit: Maintains a matching record queue with a 20-second sliding window for each ad.

[0016] Compared with the prior art, the beneficial effects of the present invention are: 1. The present invention provides an intelligent monitoring method for television commercials. First, key frame features are extracted from all advertising videos to build an advertising library. Then, the TV channel video stream is decoded in real time, image features are extracted, and matched with the key frame features in the advertising library. Within 20 seconds, if three key frames of an advertisement are matched, it is considered that the advertisement has been roughly matched. If the rough match is successful, the recorded video and the advertisement video are finely compared to obtain a comparison result.

[0017] 2. The present invention improves the accuracy of advertisement recognition by combining coarse matching with fine matching in a cascade mode without much loss in efficiency, and the final recognition results are more accurate and rich. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the method of the present invention; Figure 2 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0019] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0020] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention, and should not be understood as limiting the present invention.

[0021] Example: like Figure 1 As shown, this embodiment provides an intelligent monitoring method for television advertisements, comprising the following steps: Step S1: Extract and store key frame features of all advertising videos to build an advertising library; Step S2: Decode the TV channel video stream in real time, extract image features and match them with the advertisement library. If the same advertisement is matched to three different key frames within 20 seconds, the rough match is determined to be successful. Step S3: For advertisements that have been roughly matched successfully, the recorded video is captured and aligned with the advertisement video at the frame level, and the advertisement playback result is verified through image similarity and audio similarity.

[0022] Wherein, step S1 is specifically as follows: First, get the ad video. If the video is longer than 20 seconds, select the first 20 seconds of the video, evenly extract 30 images from the video, and save them in a list. Then, calculate the grayscale histogram for each image in the list. The grayscale histogram is actually an array of length 256, where the index corresponds to the grayscale value and the element value is the number of points on the image that equals the grayscale value. Calculate the total number of pixels with the maximum grayscale value of 10 consecutive times, sort them according to the total number, and select the smallest 15, which are the key frames. The perceptual hash value and differential hash value of each key frame are calculated by the perceptual hash algorithm and differential hash algorithm, which are the key frame features of an advertisement video. Therefore, the hash value features of the advertisement constitute the advertisement library.

[0023] Step S2 is specifically as follows: First, each ad has a queue that records the matching image decoding time and keyframe index, and only retains the results of the most recent 20 seconds; Pull the video stream and save the video slices; decode the video stream to obtain images, extract a perceptual hash value and a differential hash value every few frames, and calculate the Hamming distance between each and the relevant feature values ​​in the ad library. If one distance is less than 6 and the other is less than 20, the match is considered successful. It may match multiple key frames of the ad, and the image decoding time and key frame index are recorded in the corresponding ad queue. For ads that match keyframes, check whether there are three different keyframe indexes in the queue. If so, the rough match is considered successful. Otherwise, continue matching. Calculate the Hamming distance, specifically: The two hash values ​​are XORed bit by bit, and the number of 1s in the result (i.e. the number of different bits) is counted. Therefore, the smaller the distance, the higher the image similarity.

[0024] Step S3 is specifically as follows: First, based on the matching image decoding time, the ad's location in the recorded video is roughly located and captured. Then, the matching minimum index keyframe is selected, the recorded video is decoded to obtain an image, and the similarity between the decoded image and the keyframe is calculated using the SSIM algorithm to locate the position of the keyframe in the recorded video. In addition, the position of the keyframe is known in the advertisement video, so the recorded video and the advertisement video can be accurately aligned. After alignment, images are compared frame by frame using the SSIM algorithm, and audio is compared using Mel-frequency cepstral coefficients and Pearson correlation coefficients. If the number of inconsistent images exceeds a certain number within two seconds, and the audio similarity falls below a threshold, the roughly matched ad is considered a false positive and no further comparison is performed. The SSIM algorithm is as follows: The Structural Similarity Index (SSIM) algorithm calculates the similarity between two images. SSIM not only technically reflects the pixel differences between images, but also reflects the human eye's perception of an image. It is a comprehensive assessment of factors such as brightness, contrast, and structure. The SSIM calculation formula is as follows:

[0025] in: and are the average values ​​of pixels in image x and y respectively; and are the pixel variances of images x and y respectively; is the covariance between images x and y; and are two constants used to stabilize the denominator which may be close to zero; The specific steps are as follows: (1) In order to reduce the amount of calculation, the two images are scaled according to a certain ratio; (2) Convert the image to YUV format and extract the Y component to obtain a two-dimensional matrix; (3) Use an 11x11 sliding window to slide on the Y component, calculate the SSIM value, and then take the average as the similarity measure of the two images; Audio similarity calculation, specifically: Audio similarity calculation is performed based on Mel-frequency cepstral coefficient (MFCC) features and Pearson correlation coefficient. The specific steps are as follows: First, the audio data is preprocessed and divided into short time frames, with each frame lasting 20-40 milliseconds. Apply Fast Fourier Transform (FFT) to each frame of audio data to convert the time domain signal into a frequency domain signal; In the frequency domain, the audio signal is passed through a set of Mel filters and the logarithmic energy of each filter bank output is calculated; Substitute the above logarithmic energy into the discrete cosine transform DCT to obtain the L-order cepstral coefficient; The cepstral coefficients of the same order of all short-time audio frames are combined to obtain L time series feature vectors.

[0026] Smooth the L time series feature vectors separately, calculate the Pearson correlation coefficient, and take the average as the final similarity.

[0027] The audio of the two video streams may be slightly out of sync with the video, resulting in some delay between the two sets of audio data, which can affect the calculation of audio similarity. If the audio similarity is less than the threshold, the Pearson correlation coefficient is calculated in an offset manner. If it is greater than the threshold, the audio is also considered consistent.

[0028] According to the above method, this embodiment takes the advertising monitoring of a certain TV station as an example to further illustrate: 1. Based on the advertising video provided by the advertiser, extract the first 20 seconds of video (or the entire video if it is less than 20 seconds), evenly sample 30 frames, calculate the grayscale histogram for each frame, and then calculate the maximum value of 10 consecutive grayscale pixel values. Sorted by this value, select the first 15 smallest images as key frames; then calculate the perceptual hash value and differential hash value for each key frame, and form feature data by combining the key frame image, the key frame's position in the advertising video, the perceptual hash value, and the differential hash value; sort the feature data of all key frames according to their position in the advertising video and put them into a list in order; store this list and the advertising video in the advertising library.

[0029] 2. At the start of the ad monitoring program, a queue is created for each ad. The matching image decoding time and keyframe index are recorded, and only the results for the most recent 20 seconds are retained. A video stream is then pulled from a specific channel, recorded as 10-second segments, and decoded into images. Every three frames are extracted, and the perceptual hash value and differential hash value are calculated. For each ad, each feature data in the data list is iterated over, and the Hamming distance between the perceptual hash values ​​and the Hamming distance between the differential hash values ​​are calculated. If one is less than 6 and another is less than 20, the match is considered successful. If multiple matches are found, the image decoding time and keyframe index of the one with the smallest Hamming distance are used, and the result is stored in the ad's queue. The ad's queue is then analyzed, and if three different keyframe indices are found in the queue, a coarse match is considered successful.

[0030] 3. If the rough match is successful, enter the fine comparison program.

[0031] First, based on the image decoding time corresponding to the matched minimum index key frame and its position in the advertising video, the position of the recorded video where the advertising start frame is located is roughly located, and an additional 5 seconds of video is captured forward and backward.

[0032] The matching keyframe with the lowest index is then selected, and the recorded video is decoded to generate an image. The SSIM algorithm is used to calculate the similarity between the decoded image and the keyframe, and the frame with the highest similarity is obtained. If the maximum similarity is greater than 0.7, the match is considered successful; otherwise, the match is considered unsuccessful. If the match fails, the subsequent keyframe is selected and the above process is repeated. If all matches fail, the coarse match result is considered a false positive, and the fine comparison process is exited. If the match is successful, the keyframe's position in the recorded video can be located. Furthermore, the keyframe's position is known in the commercial video, allowing for precise alignment of the recorded and commercial videos.

[0033] After alignment, the images are compared frame by frame using the SSIM algorithm, both forward and backward. The audio is compared using Mel-frequency cepstral coefficients and Pearson correlation coefficients. If the discrepancies between the images exceed 50% within two seconds, and the audio similarity falls below 0.5, the rough match is considered a false positive, and the fine-tuning process is terminated. Otherwise, the rough match is considered correct, and image and audio comparisons continue. The comparison results can be used to analyze the ad's start and end time, completeness, and accuracy.

[0034] like Figure 2 As shown, this embodiment also provides an intelligent monitoring system for television advertisements, including: Ad library building module: used to extract and store key frame features of advertising videos; Real-time matching module: decodes channel video streams, executes coarse matching logic, and triggers fine matching; Fine matching module: performs frame-level alignment and multimodal verification on ads that have been roughly matched. Result output module: Generates advertisement playback time, completeness and correctness reports.

[0035] The real-time matching module also includes: Video slice recording unit: stores channel video stream in 10-second units; Dynamic Queue Management Unit: Maintains a matching record queue with a 20-second sliding window for each ad.

[0036] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An intelligent monitoring method for television advertisements, characterized in that: The following steps are involved: Step S1: Extract and store key frame features of all advertising videos to build an advertising library; Step S2: Decode the TV channel video stream in real time, extract image features and match them with the advertisement library. If the same advertisement is matched to three different key frames within 20 seconds, the rough match is determined to be successful. Step S3: For advertisements that have been roughly matched successfully, the recorded video is captured and aligned with the advertisement video at the frame level, and the advertisement playback result is verified through image similarity and audio similarity.

2. The intelligent monitoring method for television advertisements according to claim 1, characterized in that: In step S1, key frame features of all advertisement videos are extracted, specifically: Intercept the first 20 seconds of the ad video and evenly extract 30 frames; Calculate the grayscale histogram of each frame and select the 15 frames with the smallest total number of grayscale pixels in 10 consecutive frames as key frames; The perceptual hash value and differential hash value are calculated for each key frame.

3. The intelligent monitoring method for television advertisements according to claim 2, characterized in that: The calculation steps of the perceptual hash value are: Scale the image to 32×32 pixels and grayscale; Apply the discrete cosine transform and take the upper left 8×8 coefficient matrix; Compare the coefficients to the mean and generate a 64-bit binary hash value.

4. The intelligent monitoring method for television advertisements according to claim 2, characterized in that: The steps for calculating the differential hash value are: Resize the image to 9×8 pixels and grayscale; Compare the grayscale values ​​of adjacent pixels row by row to generate a 64-bit binary hash value.

5. The intelligent monitoring method for television advertisements according to claim 1, characterized in that: The rough matching includes: Maintain a matching queue for each ad, recording the key frame index and decoding time within 20 seconds; Calculate the perceptual hash value Hamming distance and differential hash value Hamming distance between the channel image and the key frames of the ad library; If the perceived hash distance is less than 6 and the differential hash distance is less than 20, the matching result is recorded.

6. The intelligent monitoring method for television advertisements according to claim 1, characterized in that: The detailed comparison includes: Locate the alignment position in the recorded video based on the matched minimum index keyframe; The similarity between the key frame and the recorded frame is calculated using the SSIM algorithm. If the maximum similarity is greater than 0.7, the alignment is considered successful. After alignment, compare the image and audio frame by frame.

7. The intelligent monitoring method for television advertisements according to claim 6, characterized in that: Step S2 also includes: false alarm filtering, specifically: If the number of inconsistent image frames within 2 consecutive seconds is greater than 50% and the audio similarity is less than 0.5 in the fine comparison, the coarse match is judged as a false alarm and the false alarm information is filtered and deleted.

8. The intelligent monitoring method for television advertisements according to claim 6, characterized in that: Audio similarity calculation includes: Extract MFCC features of two-channel audio; Calculate the Pearson correlation coefficient for the L-order cepstral coefficient time series vector; If the mean value of the correlation coefficient is less than the threshold, time shift recalculation is performed.

9. An intelligent monitoring system for television advertisements, characterized in that: include: Ad library building module: used to extract and store key frame features of advertising videos; Real-time matching module: decodes channel video streams, executes coarse matching logic, and triggers fine matching; Fine matching module: performs frame-level alignment and multimodal verification on ads that have been roughly matched successfully; Result output module: Generates advertisement playback time, completeness and correctness reports.

10. The intelligent monitoring system for television advertisements according to claim 9, characterized in that: The real-time matching module also includes: Video slice recording unit: stores channel video stream in 10-second units; Dynamic Queue Management Unit: Maintains a matching record queue with a 20-second sliding window for each ad.

Images