Traditional media advertisement separation and identification method based on multi-modal large model

Abstract

The application discloses a traditional media advertisement separation and identification method based on a multi-modal large model, which comprises the following steps: S1, extracting advertisement audio and video signals in television and radio and preprocessing the same; S2, extracting audio features of television and radio from the preprocessed audio and video signals, finding out a subset of the longest same segment from continuous video through feature extraction and feature retrieval, and recording the start and end time points; S3, building a traditional media advertisement knowledge base by using past artificial experience data, judging whether the segment is an advertisement or not by using a large language model and enhanced retrieval, and extracting the text boundary of the advertisement; and S4, revising the start and end boundaries of the advertisement by using VAD mute detection and speaker synchronization. In the television and radio feature extraction process, the neural network and attention mechanism are used, the problem of long time span of traditional media audio is effectively solved, the timing features of the audio can be better extracted, and the accuracy of advertisement matching is improved.

Description

Technical Field

[0001] This invention belongs to the field of advertising monitoring technology, and in particular relates to a method for separating and identifying traditional media advertisements based on a multimodal large model, which is used to automatically and efficiently separate and identify advertising content from traditional media programs such as television and radio. Background Technology

[0002] As an important indicator of economic development, advertising plays a crucial role in shaping public opinion. Monitoring traditional media advertising, especially television and radio advertising, is a vital part of advertising regulation. In the traditional media advertising monitoring process, extracting advertising content clips from continuously broadcast television and radio programs is the most time-consuming and critical task.

[0003] Currently, the process primarily relies on manual viewing of programs and marking the start and end times of advertisements to create individual ad versions. While the repetitive nature of ad placements is utilized, using audio feature searches to identify duplicate ads, the initial creation and marking of new ads still depend on manual operation. This not only consumes a significant amount of manpower and time but is also prone to human error.

[0004] While existing technologies for ad extraction utilize audio features (such as audio fingerprints and MFCC), visual features (such as specific colors and layouts in images), and specific neural network encoders / decoders or Transformers for feature extraction and matching, they still cannot escape the reliance on manual initial labeling of new ads. These technologies are inefficient when dealing with massive amounts of television and radio content, making it difficult to meet the need for rapid and accurate ad monitoring. Summary of the Invention

[0005] This invention aims to provide a traditional media advertising separation and identification method based on a multimodal large model. By integrating information from multiple modalities and leveraging the powerful retrieval and judgment capabilities of the large model, it automatically extracts advertising content and advertising boundaries from continuous television and radio broadcasts, significantly improving the efficiency of traditional media advertising separation, reducing labor costs, and enhancing the accuracy and timeliness of advertising monitoring.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following solution:

[0007] Traditional media advertising separation and identification methods based on multimodal large models include the following steps:

[0008] Step S1: Multimodal preprocessing: Extract the audio and video signals of advertisements from television and radio, and preprocess them;

[0009] Step S2: Temporal Feature Enhancement: Extract audio features from the preprocessed audio and video signals of television and radio, find the longest identical subset of segments from the continuous video through feature extraction and feature retrieval, and record the beginning and end of that time point;

[0010] Step S3: Large Model Retrieval and Judgment: Build a traditional media advertising knowledge base using past human experience data, and use a large language model and enhanced retrieval methods to determine whether the fragment is an advertisement and extract the text boundaries of the advertisement;

[0011] Step S4: Boundary Optimization: Revise the beginning and end boundaries of the advertisement using VAD silence detection and speaker synchronization.

[0012] Further optimization, step S1 specifically includes:

[0013] Step S1.1: Use FFMPEG to extract the audio and video signals of advertisements from television and radio broadcasts;

[0014] Step S1.2: Use the Wiener filtering algorithm to perform noise reduction on the audio signal:

[0015] Step S1.3: Use the VAD algorithm to detect speech activity and silence periods in the audio; use the ASR model to transcribe the speech into text, record the start and end times of the text, identify speech segments of the same speaker, slice the audio and video files at equal intervals, and generate text segments with timestamps.

[0016] Further optimization, step S2 specifically includes:

[0017] Step S2.1: Perform a short-time Fourier transform on the preprocessed audio signal to obtain the time-frequency representation X(t,f), and convert the time-frequency signal into the Mel spectrum M(t,f);

[0018] Step S2.2: Convert the Mel spectrum signal into the input format of a convolutional neural network (CNN), and use the CNN for audio feature processing to obtain the feature matrix F. cnn Specifically, the Mel spectrum is first normalized to have a mean of 0 and a standard deviation of 1 to speed up model training. Next, a CNN model is constructed, defining the structure and parameters of convolutional layers, pooling layers, and fully connected layers. Then, the CNN model is trained using training data, and the Mel spectrum to be processed is input into the trained model to obtain the audio feature matrix.

[0019] Step S2.3: Add Transformer positional encoding (PE) to the feature matrix to obtain audio features with positional encoding. Then, perform multi-head self-attention calculation, followed by feedforward neural network calculation, residual and layer normalization calculations to obtain audio features F with temporal dependencies. trans .

[0020] The multi-head attention computation specifically involves the following steps: First, calculating attention scores: For input audio features with positional encoding, multiply them by three different weight matrices to obtain query, key, and value matrices. Then, calculate the dot product of the query and key matrices to obtain the attention score matrix. Second, applying the Softmax function: Apply the Softmax function to the attention score matrix to transform the scores into a probability distribution, obtaining the attention weight matrix. Third, calculating the weighted sum: Multiply the attention weight matrix by the value matrix to obtain the weighted sum matrix, which is the output of the multi-head attention. Fourth, merging the multi-head attention: Concatenate the outputs of multiple attention heads and then transform them through a linear layer to obtain the final multi-head attention output.

[0021] Specifically, the feedforward neural network calculation involves inputting the output of the multi-head attention into the feedforward neural network, processing it through two fully connected layers and a nonlinear activation function (such as ReLU), and obtaining the output of the feedforward neural network.

[0022] The output of the multi-head attention network is then added to the output of the feedforward neural network to obtain the output of the residual connection. The output of the residual connection is then subjected to layer normalization to obtain the final temporal feature matrix.

[0023] Step S2.4: Perform nearest neighbor search on the audio features using the vector retrieval engine, calculate similarity using Euclidean distance, and obtain continuous similar subsets through dynamic programming to determine the time range of similar segments [t]. n ,t n+m ].

[0024] The audio features are used to perform nearest neighbor search in the vector retrieval engine, specifically including:

[0025] 1) Vector storage: Audio feature vectors are stored in Milvus, with each vector corresponding to an audio segment.

[0026] 2) Query Vector: Input the audio feature vector to be queried into Milvus.

[0027] 3) Nearest Neighbor Search: Milvus uses the Approximate Nearest Neighbor Search (ANN) algorithm to quickly find the vector most similar to the query vector in a high-dimensional vector space, i.e., the similar segment.

[0028] 4) Similarity calculation: Calculate the Euclidean distance between the query vector and the found similar segment vectors as a measure of similarity.

[0029] Using dynamic programming, we can find continuous similar subsets and determine the temporal start point of similar segments, specifically including:

[0030] 1) Define the state: Define a state array to record the maximum length of a continuous similar subset ending with each similar segment.

[0031] 2) State transition equation: For each similar segment, traverse all the preceding similar segments. If the similarity between the current segment and a preceding segment meets the threshold requirement, then update the maximum length of the current segment in the state array.

[0032] 3) Backtracking: Based on the state array, find the longest continuous similar subset and determine its starting point in time.

[0033] Further optimization involves selecting ResNet-50 in the improved CNN to process the audio features of television broadcasts. The input to ResNet-50 is a single-channel Mel spectrum.

[0034] Further optimization, step S3, the large model retrieval and judgment, specifically includes:

[0035] Step S3.1: According to the time range [t] n ,t n+m Extract the transcribed text from the preprocessing stage and concatenate the time [t]. n ,t n+m Features T within the range i ;

[0036] Step S3.2: Create an advertising feature sample library A and calculate feature T. i Compared with ad sample A in the advertising database j The similarity between S(T) i A j The cosine similarity is used for nearest neighbor search. If the similarity exceeds a set threshold, the segment is initially determined to be an advertisement.

[0037] Step S3.3: Design a Prompt using a large language model to extract advertising content from the transcribed text. If adjacent advertising segments have consecutive high confidence, they are merged.

[0038] Further optimization involves step S3.2, which involves creating an advertising feature sample library, specifically including:

[0039] Step S3.2.1: Data Collection: Initially, collect historical monitoring data on traditional media advertising. This data includes manually annotated audio and video clips of advertisements, along with corresponding text scripts, ad type, and placement time information. For example, collect television and radio advertising materials from the past few years, including commercials, subtitle text, and records of ad placement times. In addition to your own historical monitoring data, supplementary data can be obtained from industry reports, advertising databases, and relevant research institutions to enrich the knowledge base.

[0040] Step S3.2.2: Data preprocessing:

[0041] The collected data is cleaned to remove noise, duplicate data, and incorrect annotations. For example, the text script is checked for garbled characters, typos, and other issues, and then corrected.

[0042] The advertising segments are annotated in detail, including the start and end times of the advertisement, the type of product or service advertised, and the theme of the advertisement, so as to provide accurate labels for subsequent model training and retrieval.

[0043] The cleaned and labeled data is then structured and stored in a database or file system. Relational databases (such as MySQL) or non-relational databases (such as MongoDB) can be used to store the data for easy management and retrieval.

[0044] Step S3.2.3: Knowledge Base Construction:

[0045] Features are extracted from advertising data, including audio features (such as MFCC, spectral centroid, etc.), video features (such as color histogram, motion vector, etc.), and text features (such as keywords, word frequency, etc.). These features will be used for subsequent retrieval and judgment.

[0046] Index the data in the knowledge base to enable fast retrieval. Techniques such as inverted indexes and hash indexes can be used to improve retrieval efficiency.

[0047] Further optimization involves merging adjacent ad segments with consecutive high confidence levels in step S3.3, specifically including:

[0048] Step S3.3.1: Define the confidence level and threshold:

[0049] When the large language model determines that a segment is an advertisement, it outputs the corresponding confidence score (e.g., the range of 0-1, with higher values ​​indicating a higher likelihood of an advertisement). For example, if the threshold is set to 0.8, only segments with a confidence score ≥ 0.8 will be retained.

[0050] Check the time sequence and confidence level of adjacent segments. If segment A (time t1-t2, confidence level 0.85) and segment B (time t2-t3, confidence level 0.88) are consecutive in time and both exceed the threshold, they are judged as consecutive high-confidence advertising segments.

[0051] Step S3.3.2: Merge the transcribed text of consecutive segments, with the new boundary being the start time of the first segment and the end time of the last segment, and splice the text content into a complete advertisement text.

[0052] If fragment features (such as semantic vectors and keyword features) have been extracted in the early stage, the features are merged by weighted average (such as by confidence weight) or splicing method to form the comprehensive features of the merged advertisement.

[0053] Output results: The merged ad text, time boundaries, and high confidence markers are output.

[0054] By merging these segments, the integrity of ad recognition is improved, preventing ad fragmentation caused by audio segmentation and recognition errors, and restoring the true boundaries of the ads. At the same time, reducing the number of fragmented ad segments provides more structured input for downstream tasks such as ad statistics and compliance review, optimizing subsequent processing efficiency.

[0055] Further optimization involves extracting the text boundaries of the advertisement, specifically including:

[0056] 1) Boundary positioning based on timestamps:

[0057] Time alignment: If the audio segment has been transcribed from speech and each word or sentence has a corresponding timestamp, the start and end positions of the advertisement text are located based on the start and end times of the audio segment identified as an advertisement.

[0058] Boundary adjustment: When locating text boundaries, some adjustments need to be made, such as removing irrelevant words or noise before and after the advertisement, to ensure the accuracy of the boundaries.

[0059] 2) Semantic-based boundary optimization:

[0060] Semantic analysis: Perform semantic analysis on the located advertising text to determine whether the beginning and end of the text conform to the semantic characteristics of the advertisement. For example, advertisements usually contain promotional information about products or services, promotional activities, etc. If the beginning or end of the text does not contain this information, the boundaries can be further adjusted.

[0061] Contextual understanding: Optimize text boundaries by combining contextual information of the ad text, such as preceding and following sentences and paragraphs. For example, if the ad text ends with an incomplete sentence, the next sentence can be consulted to determine the correct boundaries.

[0062] By following the steps above, we can use large language models and enhanced retrieval to determine whether an audio segment is an advertisement and extract the text boundaries of the advertisement.

[0063] Further optimization, specifically step S4 boundary optimization, includes: during the revision process, by fine-tuning the start and end times, selecting the nearest silent time point to obtain the final advertising time.

[0064] Step S4.1: Use the VAD algorithm to detect audio frame by frame, mark the silent intervals, and use the beginning and end points of the silent intervals as candidate positions for the advertisement boundary. For example, if the detected silent interval is [t1, t2], the advertisement may start at t1 or end at t2.

[0065] Step S4.2: Use a speaker segmentation algorithm (such as pyannote.audio) to divide the audio into segments with different speakers and generate speaker change points. For example, an advertisement may be broadcast by a specific announcer whose voice differs from that of the hosts of the preceding and following programs.

[0066] If the endpoint of the silent interval coincides with or is close to the speaker change point (e.g., time difference < 0.5 seconds), then that endpoint is more likely to be the advertising boundary.

[0067] If the speaker switches between different roles before and after being muted (e.g., from a program host to an advertising announcer), the muting point is the start of the advertisement; conversely, the muting point is the end of the advertisement.

[0068] Step S4.3: Combine the VAD silence interval and speaker change point to determine the final boundary. For example: if the starting point t1 of the silence interval [t1, t2] is accompanied by a speaker change, then the advertisement start boundary is revised to t1. If the speaker switches back to the program content after the ending point t2 of the silence interval, then the advertisement end boundary is revised to t2.

[0069] Compared with the prior art, the beneficial effects of the present invention are:

[0070] 1. The method described in this invention utilizes neural networks and attention mechanisms in the process of extracting features from television and radio broadcasts, effectively solving the problem of long audio time spans in traditional media, and can better extract the temporal features of audio, thereby improving the accuracy of ad matching.

[0071] 2. The method described in this invention fully considers the characteristics of traditional media advertising broadcasts and comprehensively utilizes technologies such as nearest neighbor search, dynamic programming, knowledge base retrieval, large language model judgment, and silence detection to achieve higher accuracy in advertising extraction time boundary identification, greatly reducing manual intervention and errors.

[0072] 3. The method described in this invention automates the separation and identification of advertisements, significantly improves the efficiency of traditional media advertising monitoring, can quickly process a large amount of television and radio program content, promptly detect and monitor advertisements, and provide strong support for advertising supervision. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of a traditional media advertising separation and recognition method based on a multimodal large model. Detailed Implementation

[0074] The present invention will be further described below with reference to the embodiments, but it should not be construed as the scope of the present invention being limited to the following embodiments.

[0075] Example 1:

[0076] like Figure 1 As shown, a traditional media advertising separation and identification method based on a multimodal large model includes the following steps:

[0077] Step S1: Multimodal preprocessing: Extract the audio and video signals of advertisements from television and radio, and preprocess them. Specifically, this includes:

[0078] Step S1.1: Use FFMPEG to extract the audio and video signals of advertisements from television and radio broadcasts;

[0079] Step S1.2: Use the Wiener filtering algorithm to perform noise reduction on the audio signal:

[0080]

[0081] in, Let |H(r)| represent the estimated value of the audio signal at the r-th frequency component in the frequency domain, |H(r)| represent the amplitude response of the signal at the r-th frequency component, Y(r) represent the noisy signal received at the r-th frequency component, and λ represent the power spectrum of the noise.

[0082] Step S1.3: Use the VAD algorithm to detect speech activity and silence periods in the audio; use the ASR model to transcribe the speech into text, record the start and end times of the text, identify speech segments of the same speaker, slice the audio and video files at equal intervals, and generate text segments with timestamps.

[0083] Step S2: Temporal Feature Enhancement: Extract audio features from the preprocessed audio and video signals of television and radio, find the longest identical subset of segments from the continuous video through feature extraction and feature retrieval, and record the beginning and end of that time point.

[0084] Specifically, it includes:

[0085] Step S2.1: Perform a short-time Fourier transform on the preprocessed audio signal to obtain the time-frequency representation X(t,f).

[0086]

[0087] The time-frequency signal is then converted into a Mel spectrum M(t,f).

[0088]

[0089] Where x(n) represents the value of the preprocessed audio signal at the nth sampling point, representing the discrete sample values ​​of the audio signal. w(nt) represents the window function (such as the Hanning window or Hamming window), used to frame the audio signal. t represents the time position of the analysis, using a sliding window function to extract audio segments from different time periods. H g (f) represents the frequency response function of the g-th filter in the Mel filter bank, used to filter the time-frequency signal X(t,f) and extract frequency components that conform to the Mel scale. G represents the total number of filters in the Mel filter bank, which determines the dimension (i.e., the number of features) of the final Mel spectrum.

[0090] Step S2.2: Convert the Mel spectrum signal into the input format of a convolutional neural network (CNN), and use the CNN for audio feature processing to obtain the feature matrix F. cnn Specifically, the Mel spectrum is first normalized to have a mean of 0 and a standard deviation of 1 to speed up model training. Next, a CNN model is constructed, defining the structure and parameters of convolutional layers, pooling layers, and fully connected layers. Then, the CNN model is trained using training data, and the Mel spectrum to be processed is input into the trained model to obtain the audio feature matrix F. cnn .

[0091] In this embodiment, CNN is used for audio feature processing of television broadcasts, and ResNet-50 is selected here. Since the ResNet pre-trained model expects RGB images (224×224×3), but the Mel spectrum is a single channel (T×F×1), the Mel spectrum is copied three times to become (T×F×3) to adapt to ResNet.

[0092] The input Mel spectrum M(t,f) is processed by ResNet to output a 512-dimensional high-dimensional feature, which includes spectral patterns and background sound features.

[0093] F cnn =ResNet(M′);

[0094] C l+1 (t,f)=ReLU(W*C l (t,f)+b);

[0095] Where M′ represents the Mel spectrum data input to the ResNet network; C l (t,f) represents the feature map of the l-th layer in the convolutional neural network, with dimensions of time t and frequency f; W represents the convolution kernel weight matrix; B represents the bias term, which adds an offset to the convolution result to increase the expressive power of the model; ReLU represents the rectified linear activation function, which modifies the convolution result W*C. l (t,f)+b represents a nonlinear transformation, with the formula ReLU(x)=max(0,x), used to introduce nonlinear characteristics and enhance the network's ability to learn complex patterns. C l+1 (t,f) represents the feature map output to the (l+1)th layer after the l-th convolution and activation operations, which incorporates new spectral features for further processing by subsequent network layers.

[0096] Step S2.3: Add Transformer positional encoding (PE) to the feature matrix to obtain audio features with positional encoding. Then, perform multi-head self-attention calculation, followed by feedforward neural network calculation, residual and layer normalization calculations to obtain audio features F with temporal dependencies. trans .

[0097] Among them, adding position encoding to the Transformer is as follows:

[0098] PE(t,2i)=sin(t / 10000 2i / D )

[0099] PE(t,2i+1)=cos(t / 10000 2i / D );

[0100] Where T is the time step, D is the feature dimension, and i is the dimension index value.

[0101] Finally, the positional encoding is added to the audio feature matrix.

[0102] F input =F cnn +PE.

[0103] The multi-head attention calculation is specifically as follows: First, the attention score is calculated: For the input audio features with positional encoding, they are first multiplied by three different weight matrices to obtain three matrices: Query, Key, and Value.

[0104] Q = W Q F input K = W K F input V = W V Finput

[0105] Where Q, K, and V represent the query matrix, key matrix, and value matrix, respectively; W Q W K W V These represent the trainable parameter matrices, respectively;

[0106] Then, the dot product of the query matrix and the key matrix is ​​calculated to obtain the attention score matrix.

[0107]

[0108] Where, d k is the scaling factor, and B represents the attention score matrix.

[0109] Calculate the attention weights and normalize them using Softmax so that the sum of the weights is 1. Multiply V by the attention score to obtain the aggregated information.

[0110] Attention(Q,K,V)=softmax(A)V

[0111] Finally, multi-head attention MHSA (F) is performed. input )calculate.

[0112] MHSA(F input = Concat(head1,...,head) h W O ;

[0113] Each Transformer layer is followed by a feedforward neural network, which enhances the model's expressive power.

[0114] FFN(x)=max(0,xW1+b1)W2+b2;

[0115] Among them, head h This represents the output of the h-th attention head. OThe projection matrix (W1, b1) performs a linear transformation on the concatenated multi-head attention output (Concat(head1,…,headh)), integrating multi-head information to output the final feature representation. x represents the input to the feedforward neural network; W1 and b1 represent the weight matrix and bias term of the first-layer linear transformation, performing a linear transformation (xW1+b1) on the input x to expand the feature dimension or extract preliminary abstract features. max(0,·) represents the ReLU activation function, introducing non-linearity, filtering out negative values ​​in the linear transformation result, and enhancing the model's ability to express complex patterns. W2 and b2 represent the weight matrix and bias term of the second-layer linear transformation, further transforming the activated features (max(0,xW1+b1)W2+b2), compressing the feature dimension or generating the final feature representation, thus improving the model's expressive power.

[0116] We employ residual connections and layer normalization. We use LayerNorm to normalize features, making training more stable. We also use residual connections to prevent gradient vanishing during the solution process.

[0117] F trans =LayerNorm(MHSA(F input )+F input )

[0118] F trans =LayerNorm(FFN(F input )+F input ).

[0119] Step S2.4: Perform nearest neighbor search (ANN) on the audio features in the vector retrieval engine, calculate similarity using Euclidean distance, and obtain continuous similar subsets through dynamic programming to determine the time range of similar segments [t]. n ,t n+m Specifically:

[0120]

[0121] Similarity calculation uses Euclidean distance d(F) i ,F j ):

[0122] d(F i ,F j )=||F i -F j ||

[0123] F i ,F j This represents the feature vector of an audio segment, where i,j are the segment indices, and NN(F) i ) represents the nearest neighbor function, used to find the nearest neighbor with respect to feature F. i Most similar feature Fj The output makes the similarity sim(F) i ,F j The largest F j The index j.

[0124] Set a similarity threshold θ, if sim(F i ,F j If ) > θ, then we consider a i and a j They may belong to the same advertisement. Let S be the longest subset of advertisements, S = {a...} k ,a k+1 ,...,a k+m};

[0125] Among them, a i and a j Let i represent an audio segment, where i and j are segment indices; k is the index of the starting segment in the subset S of advertisements; and m represents the offset of the number of segments in the subset.

[0126] Seeking the target in M i,i+1 =1, calculate using dynamic programming. M i,i+1 =1 indicates an indicator matrix element.

[0127]

[0128] After calculation, the longest ad segment is max(dp[i]), and the ad duration is [t]. k ,t k+m ].

[0129] Step S3: Large-Scale Model Retrieval and Judgment: A traditional media advertising knowledge base is built using past human experience data. A large language model and enhanced retrieval methods are employed to determine whether a segment is an advertisement and to extract the text boundaries of the advertisement. Specifically, this includes:

[0130] Step S3.1: According to the time range [t] n ,t n+m Extract the transcribed text from the preprocessing stage and concatenate the time [t]. n ,t n+m Features T within the range i ;

[0131] Step S3.2: Create an advertising feature sample library A and calculate audio features T. i Compared with ad sample A in the advertising database j The similarity between S(T) i A j ),

[0132]

[0133] Among them, F Ti ,F Aj The features to be detected are T i And advertising sample A j eigenvectors.

[0134] Cosine similarity is used for nearest neighbor search.

[0135]

[0136] If S(T) i A * If ) > θ (threshold), then T i It might be an advertisement.

[0137] Step S3.3: Design a Prompt using a large language model to extract advertising content from the transcribed text. If adjacent advertising segments have consecutive high confidence, they are merged.

[0138] If adjacent advertising segments (T) i ,T i+1 ,...T i+k If the confidence level is consistently high, then the data will be merged.

[0139]

[0140] T final This represents the final advertisement segment after the merger. ad (T i ) indicates advertising segment T i The confidence level of the advertisement. τ is the confidence threshold.

[0141] Step S4: Boundary Optimization: Revise the beginning and end boundaries of the advertisement using VAD silence detection and speaker synchronization.

[0142] Boundary revision process. Further fine-tuning of start and end times using VAD silence detection:

[0143] (t start ,t end ) = VAD(T final );

[0144] Finally, after fine-tuning (t) start ,t end Within the specified time frame, the final advertising content will be determined.

[0145] Application examples:

[0146] Separate the TV commercials for a certain TV series.

[0147] Step S1: Input data: One episode of a TV series is 60 minutes long, including 3 commercial breaks; the resolution is 1920×1080, and the audio sampling rate is 44.1kHz.

[0148] Step S2, Preprocessing results: The audio file obtained is 210MB in size; 360 10-second segments are generated, and 42 silent intervals are detected by VAD.

[0149] Step S3, Feature Enhancement Output: Mel spectrum matrix: 360×40×1; ResNet output features: 360×512; Transformer temporal features: 360×512.

[0150] Step S4, Large Model Retrieval: Find the longest matching segment with a similarity of 0.92; perform LLM verification with a confidence level of 0.96, and determine it to be an advertisement.

[0151] Step S5, Boundary Optimization: VAD fine-tuning, starting time from 00:15:30 → 00:15:30.5, final output ad timestamp: [00:15:30.5-00:16:15.5].

[0152] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A traditional media advertising separation and identification method based on a multimodal large model, characterized in that, Includes the following steps: Step S1: Multimodal preprocessing: Extract the audio and video signals of advertisements from television and radio, and preprocess them; Step S2: Temporal feature enhancement: Extracting audio features from preprocessed audio and video signals (TV and radio), identifying the longest identical subset of segments from continuous video using feature extraction and retrieval methods, and recording the beginning and end timestamps; specifically including: Step S2.1: Perform a short-time Fourier transform on the preprocessed audio signal to obtain the time-frequency representation X(t,f), and convert the time-frequency signal into the Mel spectrum M(t,f); Step S2.2: Convert the Mel spectrum signal into the input format of a convolutional neural network (CNN), and use the CNN for audio feature processing to obtain the feature matrix F. cnn ; Step S2.3: Add Transformer positional encoding (PE) to the feature matrix to obtain audio features with positional encoding. Then, perform multi-head self-attention calculation, followed by feedforward neural network calculation, residual and layer normalization calculations to obtain audio features F with temporal dependencies. trans ; Step S2.4: Perform nearest neighbor search on the audio features using the vector retrieval engine, calculate similarity using Euclidean distance, and obtain continuous similar subsets through dynamic programming to determine the time start of similar segments [t]. n ,t n+m Specifically: ； Similarity calculation uses Euclidean distance d(F) i ,F j ): ； F i ,F j Let i and j represent the feature vectors of the audio segments, respectively, where i and j are the segment indices; NN(F i ) represents the nearest neighbor function, used to find the nearest neighbor with respect to feature F. i Most similar feature F j ; sim(F i ,F j ) represents F i ,F j The similarity between them; The output represents the similarity sim(F) i ,F j The largest F j index j; F represents i ,F j The Euclidean distance between two eigenvectors; Set similarity threshold ,if Then it is considered that a i and a j They belong to the same advertisement; let S be the longest subset of advertisements. ; where a i and a j This represents an audio segment, where i and j are segment indices; k is the index of the starting segment in the subset S of advertisements; and m represents the offset of the number of segments in the subset. Seeking the target ,in Dynamic programming is used for calculation; M i,i+1 =1 indicates an indicator matrix element; ; Calculations show that the longest advertisement segment is... The advertising time is ; The dynamic programming state variable is the length of the current consecutive ad segment when the i-th audio segment ends. The start time of the advertisement. This is the end time of the advertisement; Step S3: Large-Scale Model Retrieval and Judgment: A traditional media advertising knowledge base is built using past human experience data. A large language model and augmented retrieval methods are employed to determine whether a segment is an advertisement and to extract the text boundaries of the advertisement. Specifically, this includes: Step S3.1: According to the time range [t] n ,t n+m Extract the transcribed text from the preprocessing stage and concatenate the time [t]. n ,t n+m Features T within the range i ; Step S3.2: Create an advertising feature sample library A and calculate feature T. i Compared with ad sample A in the advertising database j The similarity between S(T) i A j The cosine similarity is used for nearest neighbor search. If the similarity exceeds a set threshold, the segment is initially determined to be an advertisement. Step S3.3: Design a Prompt using a large language model to extract advertising content from the transcribed text. If adjacent advertising segments have consecutive high confidence levels, they are merged. Specifically, this includes: Step S3.3.1: Define the confidence level and threshold: When the large language model determines that a segment is an advertisement, it outputs the corresponding confidence score; Check the temporal order and confidence level of adjacent segments. If different segments are consecutive in time and their confidence levels all exceed the threshold, they are judged as consecutive high-confidence advertising segments. Step S3.3.2: Merge the transcribed text of consecutive segments, with the new boundary being the start time of the first segment and the end time of the last segment, and splice the text content into the complete advertisement text; If segment features have been extracted in the early stage, the features are merged by weighted average or splicing method to form the comprehensive features of the merged advertisement; Output the merged ad text, time boundaries, and high-confidence markers; Step S4: Boundary Optimization: Utilizing VAD silence detection and speaker synchronization, revise the beginning and end boundaries of the advertisement; during the revision process, fine-tune the start and end times, selecting the nearest silence time point to obtain the final advertisement duration, specifically including: Step S4.1: Use the VAD algorithm to detect audio frame by frame, mark the silent intervals, and use the beginning and end points of the silent intervals as candidate positions for the advertisement boundary; Step S4.2: Use a speaker segmentation algorithm to divide the audio into segments of different speakers and generate speaker change points; If the endpoint of the silent zone coincides with or is close to the speaker change point, then that endpoint is more likely to be the advertising boundary. If the speaker switches between being muted and not muted, the start point of the muting is the start of the advertisement; conversely, the end point of the muting is the end of the advertisement. Step S4.3: Combine the VAD silence interval and speaker change point to determine the final boundary.

2. The traditional media advertising separation and identification method based on a multimodal large model according to claim 1, characterized in that, Step S1 specifically includes: Step S1.1: Use FFMPEG to extract the audio and video signals of advertisements from television and radio broadcasts; Step S1.2: Use the Wiener filtering algorithm to perform noise reduction on the audio signal: Step S1.3: Use the VAD algorithm to detect speech activity and silence periods in the audio; use the ASR model to transcribe the speech into text, record the start and end times of the text, identify speech segments of the same speaker, slice the audio and video files at equal intervals, and generate text segments with timestamps.

3. The traditional media advertising separation and identification method based on a multimodal large model according to claim 1, characterized in that, In step S2.2, ResNet-50 is selected in the improved CNN to process the audio features of television broadcasts. The input of ResNet-50 is a single-channel Mel spectrum.

4. The traditional media advertising separation and identification method based on a multimodal large model according to claim 1, characterized in that, Step S3.2, which involves creating an advertising feature sample library, specifically includes: Step S3.2.1: Data collection: In the early stage, historical data of traditional media advertising is collected. This data includes manually annotated advertising audio and video clips, as well as corresponding text scripts, advertising type and placement time information; Step S3.2.2: Data preprocessing: The collected data is cleaned to remove noise, duplicate data, and incorrect labels; The advertising segments are annotated in detail, including the start and end times of the advertisement, the type of product or service advertised, and the theme of the advertisement, so as to provide accurate labels for subsequent model training and retrieval. The cleaned and labeled data is then structured and stored in a database or file system. Step S3.2.3: Knowledge Base Construction: Extract features from advertising data, including audio features, video features, and text features; Index the data in the knowledge base for quick retrieval.

Images