Advertisement video key frame identification method, system, device and storage medium

Abstract

The application relates to the technical field of image recognition, in particular to an advertisement video key frame identification method, system and device and a storage medium, which comprises the following steps: obtaining an original advertisement video and deconstructing the original advertisement video to obtain a candidate frame set; inputting the candidate frame set into a multimodal large model to perform reasoning to extract a visual semantic feature vector and a marketing state distribution vector; using the visual semantic feature vector to represent semantic correlation and introducing a bias correction term according to the marketing state distribution vector, and combining visual saliency score to calculate a single-frame recommendation score; using the single-frame recommendation score and inter-frame visual similarity to construct a kernel matrix, performing a determinant point process to filter to obtain a key frame set; obtaining click conversion data of the key frame set, calculating a feedback reward, and performing online updating on parameters of the multimodal large model based on the feedback reward through a policy gradient algorithm. The application can ensure that the identified key frame is based on accurate expression of a selling point of a product rather than pure visual brightness.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method, system, device and storage medium for keyframe recognition of advertising videos. Background Technology

[0002] In the mobile internet and digital marketing technology fields, video advertising has become a core medium connecting brands and consumers. Keyframes in advertising videos, as static summaries of the video content, are often used as video covers or dynamic creative materials, and their quality directly determines the click-through rate and conversion rate of the advertisement. Efficiently and accurately extracting keyframes that can attract user attention and stimulate purchasing desire from massive amounts of video data is of great significance for optimizing advertising effectiveness and improving advertisers' ROI.

[0003] Existing video keyframe extraction technologies have undergone several stages of development. Early solutions were mostly based on low-level visual features, such as detecting shot boundaries by calculating pixel differences, histogram changes, or optical flow between adjacent frames, and selecting the middle frame of the shot as the keyframe. These methods have low computational costs, but due to a lack of semantic understanding of the video content, they are prone to selecting blurry or information-poor transitional frames. With the popularization of deep learning technology, methods based on clustering analysis and aesthetic evaluation have gradually become mainstream. These methods use convolutional neural networks to extract feature vectors from video frames, use clustering algorithms to select representative frames, or use aesthetic scoring models to select high-quality frames with clear composition and rich colors. Currently, a more commonly used approach is video summarization technology based on deep semantic matching. It typically uses a dual-tower structure to encode the video frame sequence and text description separately, aiming to maximize the semantic similarity between visual features and text features, thereby selecting the frames that best fit the video theme.

[0004] However, existing technologies rely excessively on general semantic matching and visual features, lacking the ability to differentiate between the deeper marketing logic and actual conversion drivers of advertisements. On the one hand, while general models can identify objects or people in an image, they cannot understand the unique narrative structure and marketing intent of an advertisement. For example, in a cosmetics advertisement that includes a display of the pain point of bare skin, the product usage process, and the final makeup effect, an existing model might mistakenly select an image showing skin imperfections as the cover image based solely on visual impact or surface semantic matching. While this selection may be semantically accurate, it could potentially trigger user aversion and reduce click-through rates. On the other hand, existing models are easily influenced by explicit biases in the data, tending to award high scores to images with high saturation, high brightness, or exquisite composition. However, in advertising scenarios, actual conversion effects are often driven by the information density of the image, such as clear price tags and promotional copy, or emotional resonance, such as specific facial expressions, rather than simply visual aesthetics. In summary, existing technologies struggle to effectively decouple visually salient features from actual conversion value. This results in keyframes that, while visually appealing, often fail to convey effective product information or stimulate purchase desire, leading to high-quality keyframes but low actual conversion rates. Summary of the Invention

[0005] This application provides a method, system, device, and storage medium for keyframe recognition in advertising videos, ensuring that the recognized keyframes are based on their accurate expression of the product's selling points rather than simply visual appeal. This application provides the following technical solutions:

[0006] In a first aspect, this application provides a method for identifying keyframes in an advertising video, the method comprising:

[0007] The original advertising video is acquired and subjected to multimodal data deconstruction and adaptive temporal pyramid sampling to construct a candidate frame set;

[0008] The candidate frame set is input into a multimodal large model configured with a low-rank adaptive module and marketing status prompts, and inference is performed to extract visual semantic feature vectors and marketing status distribution vectors.

[0009] The visual semantic feature vector is used to represent semantic relevance, and a bias correction term is introduced based on the marketing state distribution vector. The single-frame recommendation score is calculated by combining the visual saliency score.

[0010] A kernel matrix is ​​constructed using the single-frame recommendation score and inter-frame visual similarity, and a matrix multiplication optimization process is performed to filter out the key frame set from the candidate frame set.

[0011] The click conversion data of the keyframe set is obtained, the feedback reward is calculated using the click conversion data, and the parameters of the multimodal large model are updated online based on the feedback reward using a policy gradient algorithm.

[0012] In one specific implementation scheme, the step of acquiring the original advertising video and performing multimodal data deconstruction and adaptive temporal pyramid sampling to construct a candidate frame set includes:

[0013] Receive the input raw advertising video file Original advertising video file Contains the original frame sequence Audio stream and metadata ,in, Indicates the total number of original frames;

[0014] An adaptive temporal pyramid sampling strategy is used to filter the original frame sequence, and the perceptual hash difference between two adjacent frames in the original frame sequence is calculated. A dynamic sampling rate function is constructed based on this perceptual hash difference. The calculation formula is: ,in, and All are preset adjustment coefficients; the original frame sequence is non-uniformly sampled according to the calculated dynamic sampling rate to construct a candidate frame set. ,in This represents the total number of candidate frames after sampling;

[0015] Using automatic speech recognition models to analyze audio streams Transcription is performed to generate a text sequence containing timestamp information. Optical character recognition (OCR) technology is used to extract the superimposed text information on each frame in the candidate frame set to obtain the corresponding OCR text. .

[0016] In one specific implementation, the step of inputting the candidate frame set into a multimodal large model configured with a low-rank adaptive module and marketing status cues includes:

[0017] The first Frame candidate frames Image segmentation Each image patch is then processed through a linear projection layer. Each image patch is mapped to a visual token sequence. ,in Represents category labels used to characterize the global semantics of an image;

[0018] Will be with candidate frames Corresponding OCR text and text sequences Character-level concatenation is performed according to a preset order to form a complete text description. A text segmenter is then used to segment and vectorize the text description, generating a text token sequence composed of multiple text tags. ;

[0019] Introducing learnable temporal location vectors The formula for calculating this time-series position vector is: ,in, This represents the sinusoidal position encoding function;

[0020] Aggregate the generated vectors to generate the first vector. Composite input vector of frames The calculation method is as follows ,in This indicates a splicing operation.

[0021] In one specific implementation, the inference process to extract the visual semantic feature vector and the marketing state distribution vector includes:

[0022] The multimodal large model includes a visual encoder, which employs a Transformer-based network architecture; the self-attention mechanism within the Transformer encoding layer includes querying the projection matrix. and value projection matrix The original weight matrix in the pre-trained model Keeping unchanged, in querying the projection matrix AND-value projection matrix Along the computational path, a trainable low-rank adaptive module is connected in parallel. The low-rank adaptive module is composed of a dimensionality reduction matrix. With increasing dimension matrix Composed of series connection;

[0023] During the forward propagation computation, for the input hidden layer state... Calculate the corrected output The calculation formula is as follows:

[0024] ;

[0025] in, Scaling factor For constant hyperparameters, Let be the rank of the low-rank matrix;

[0026] composite input vector The input is fed into the Transformer encoding layer, where it undergoes iterative computation through multiple layers of self-attention mechanisms to extract class labels. The corresponding final hidden layer state is defined as a visual semantic feature vector containing marketing intent. .

[0027] In one specific implementation, the inference process to extract the visual semantic feature vector and the marketing state distribution vector further includes:

[0028] The multimodal large model includes a text encoder, with a pre-set AIDA marketing status cue vector set at the input of the text encoder. The four vectors correspond to the four stages of attention, interest, desire, and action in the marketing funnel model; for the first vector in the candidate frame set... Frame, obtain its corresponding text token sequence , text token sequence Each of the four marketing state vectors is concatenated to construct four parallel prompt inputs, denoted as . ;

[0029] Label visual categories The mapping is converted into a query vector, which will then pass through a text encoder. Encoded prompt input Mapped to key vectors and value vectors, the first... The frame belongs to the marketing stage. probability score As shown below:

[0030] ;

[0031] in, Visual category tags With text encoder Encoded prompt input Cosine similarity between them;

[0032] Output marketing state distribution vector .

[0033] In one specific implementation scheme, the step of using the visual semantic feature vector to represent semantic relevance, introducing a bias correction term based on the marketing state distribution vector, and calculating the single-frame recommendation score by combining the visual saliency score includes:

[0034] Using a lightweight saliency detection network For the Frame candidate frames Perform feature extraction and output visual saliency score. ;

[0035] Obtain the textual description of the selling points of the current advertised product, and map it into a feature vector of selling points using a text encoder. ;

[0036] Computing visual semantic feature vectors With selling point feature vector The semantic transformation score is obtained by calculating the dot product similarity between the two pairs of strings. , ;

[0037] The ratings are fused and calculated using a counterfactual correction formula, and a single-frame recommendation score is defined. The calculation formula is as follows:

[0038] ;

[0039] in, The debiasing coefficients are in constant form. Use the Sigmoid activation function; As a dynamic visual deviation term, the marketing stage to which the current frame belongs is determined based on the marketing state distribution vector, and the deviation benchmark value corresponding to that specific stage is retrieved from preset historical statistical data as... .

[0040] In one specific implementation scheme, the step of constructing a kernel matrix using the single-frame recommendation score and inter-frame visual similarity, performing a matrix-based point process diversity combination optimization, and selecting a keyframe set from the candidate frame set includes:

[0041] Definition of the first Quality metrics of candidate frames , , For the first The single-frame recommendation score of the frame; defining the first frame Frame candidate frame and the first Visual similarity measurement between candidate frames , and For the first Frame candidate frame and the first Visual semantic feature vectors of candidate frames;

[0042] Construction dimension kernel matrix nuclear matrix Each element in As shown below:

[0043] ;

[0044] in, For the first Quality metrics for candidate frames; based on kernel matrix The process involves performing a variety of combinatorial optimizations to select the optimal subset from the candidate frame set. The process is divided into establishing the optimization objective and performing approximate solutions.

[0045] The selection objective is defined as finding a subset. Select a subset from the candidate frame set probability Proportional to the submatrix corresponding to that subset The determinant value is given by the formula: ;

[0046] in, Representing a subset The submatrix formed by the rows and columns corresponding to the index. Represents the determinant of a submatrix;

[0047] The keyframe set is initialized to empty. In each iteration, all unselected candidate frames are traversed, and the frame that maximizes the logarithmic increment of the determinant of the submatrix when added to the current set is selected. This process is repeated until the number of selected frames reaches a preset threshold. To obtain the keyframe set .

[0048] Secondly, this application provides an advertising video keyframe recognition system, which adopts the following technical solution:

[0049] An advertising video keyframe recognition system, comprising:

[0050] The video deconstruction module is used to acquire the original advertising video and perform multimodal data deconstruction and adaptive temporal pyramid sampling to construct a candidate frame set;

[0051] The vector extraction module is used to input the candidate frame set into a multimodal large model configured with a low-rank adaptive module and marketing status prompts, and perform inference to extract visual semantic feature vectors and marketing status distribution vectors.

[0052] The scoring module is used to represent semantic relevance using the visual semantic feature vector, introduce a bias correction term based on the marketing state distribution vector, and calculate the single-frame recommendation score by combining the visual saliency score.

[0053] The keyframe filtering module is used to construct a kernel matrix using the single-frame recommendation score and inter-frame visual similarity, perform matrix multiplication optimization processing, and filter the keyframe set from the candidate frame set.

[0054] The strategy update module is used to acquire click conversion data of the keyframe set, calculate feedback rewards using the click conversion data, and update the parameters of the multimodal large model online based on the feedback rewards using a strategy gradient algorithm.

[0055] Thirdly, this application provides an electronic device, the device including a processor and a memory; the memory stores a program, the program being loaded and executed by the processor to implement an advertising video keyframe recognition method as described in the first aspect.

[0056] Fourthly, this application provides a computer-readable storage medium storing a program that, when executed by a processor, is used to implement an advertising video keyframe recognition method as described in the first aspect.

[0057] This application first constructs a candidate frame set by performing multimodal data deconstruction and adaptive temporal pyramid sampling on the original advertising video. This set is then input into a multimodal large model configured with a low-rank adaptive module and marketing status cues to extract visual semantic feature vectors and marketing status distribution vectors. Subsequently, the visual semantic feature vectors are used to represent semantic relevance, and a bias correction term is introduced based on the marketing status distribution vector. A single-frame recommendation score is calculated by combining the visual saliency score with the visual similarity score. Then, a kernel matrix is ​​constructed using this score and the inter-frame visual similarity, and a matrix-determinant point process diversity combination optimization process is performed to filter out the key frame set. Finally, the click conversion data is used to calculate the feedback reward, and the model parameters are updated online using a policy gradient algorithm. By introducing a bias correction term using the marketing state distribution vector and combining visual semantic features with saliency scores, the scoring bias caused by relying solely on visual appeal can be effectively corrected, ensuring that the single-frame recommendation score accurately reflects the semantic relevance and true marketing value of the image. By constructing a kernel matrix using recommendation scores and visual similarity and performing deterministic point processing, dual constraints of quality and diversity are achieved during the selection process, effectively eliminating visually repetitive content and ensuring that the output keyframe set has both high scores and low redundancy. Through feedback rewards based on click-conversion data and a strategy gradient algorithm, an online update mechanism for model parameters is established, enabling the model to iterate itself based on actual business results, thereby achieving adaptive optimization in response to dynamic market feedback.

[0058] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings. Attached Figure Description

[0059] Figure 1 This is a flowchart illustrating the advertising video keyframe recognition method in the embodiments of this application.

[0060] Figure 2 This is a schematic diagram of the overall process of the advertising video keyframe recognition method in the embodiments of this application.

[0061] Figure 3 This is a structural block diagram of the advertising video keyframe recognition system in the embodiments of this application.

[0062] Figure 4 This is a block diagram of an electronic device for identifying keyframes in advertising videos, as described in this application. Detailed Implementation

[0063] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate this application, but are not intended to limit the scope of this application.

[0064] Optionally, this application uses the advertising video keyframe recognition method provided in various embodiments as an example for description in an electronic device. The electronic device is a terminal or a server. The terminal can be a computer, tablet computer, etc. This embodiment does not limit the type of electronic device.

[0065] Reference Figure 1 This is a flowchart illustrating an embodiment of an advertising video keyframe recognition method provided in this application. The method includes at least the following steps:

[0066] Step S101: Obtain the original advertising video and perform multimodal data deconstruction and adaptive temporal pyramid sampling on it to construct a candidate frame set.

[0067] In step S101, the input original advertising video file is first received. Original advertising video file Contains the original frame sequence Audio stream and metadata ,in, Represents the total number of raw frames, metadata This refers to descriptive information related to the content of the advertisement video, such as the product title and product category. (This refers to the original advertisement video file.) Perform multimodal deconstruction operations to process visual stream, audio stream, and text stream data separately.

[0068] For processing visual stream data, given that the original frame sequences typically have a high frame rate, directly processing the entire dataset would result in computational redundancy. Therefore, an adaptive temporal pyramid sampling strategy is employed to filter the original frame sequences. Specifically, the perceptual hash difference between adjacent frames in the original frame sequence is calculated. A dynamic sampling rate function is constructed based on this perceptual hash difference. The calculation formula is: ,in, and All are preset adjustment coefficients. This dynamic sampling rate function increases the sampling rate when the perceived hash difference between the images is large, and decreases the sampling rate when the images are relatively still. The original frame sequence is non-uniformly sampled based on the calculated dynamic sampling rate to construct a candidate frame set. ,in This application uses the total number of candidate frames after sampling. Represents the first in the candidate frame set Frame, in which .

[0069] For processing audio and text stream data, an automatic speech recognition model is used to process the audio stream. Transcription is performed to generate a text sequence containing timestamp information. Simultaneously, optical character recognition (OCR) technology is used to extract the superimposed text information from each frame in the candidate frame set, resulting in the corresponding OCR text. Through the above operations, the multimodal data parsing and structured preparation of the original advertising video were completed.

[0070] Step S102: Input the candidate frame set into the multimodal large model configured with low-rank adaptive module and marketing status prompt, and perform inference to extract visual semantic feature vector and marketing status distribution vector.

[0071] In step S102, in order for the multimodal large model to simultaneously perceive visual content, temporal location, and marketing intent, the data in the candidate frame set needs to be standardized and embedded to construct a temporally enhanced multimodal input vector for input to the multimodal large model.

[0072] For the first candidate frame in the candidate frame set frame Construct its composite input vector The specific process is as follows:

[0073] First, a visual embedding operation is performed to embed candidate frames. Image segmentation Each image patch is then processed through a linear projection layer. Each image patch is mapped to a visual token sequence. ,in This represents the category label used to characterize the global semantics of the image. Next, a text embedding operation is performed. The metadata obtained in step S101 is then compared with the candidate frame. Corresponding OCR text and the text sequence generated in step S101 Character-level concatenation is performed according to a preset order to form a complete text description. Then, a text segmenter is used to segment and vectorize the text description, generating a text token sequence composed of multiple text tags. Next, perform an absolute temporal embedding operation. To enable the multimodal large model to recognize the absolute position of the candidate frame in the original ad video, a learnable temporal position vector is introduced. The formula for calculating this time-series position vector is: ,in, This represents the sinusoidal positional encoding function. Finally, the generated vectors are aggregated to generate the first... Composite input vector of frames The calculation method is as follows ,in This indicates a splicing operation. Through the above steps, the construction of multimodal input data for candidate frames is completed.

[0074] Before performing the specific feature extraction, we first need to clarify the structure and construction method of the multimodal large model used in this step. The multimodal large model in this application is not a general pre-trained model, but a special model that has been structurally adapted for advertising and marketing scenarios. The multimodal large model consists of two parts: a visual encoder and a text encoder, which are arranged in a dual-tower structure.

[0075] The visual encoder adopts a Transformer-based network architecture, with the Transformer encoding layer as its core computational unit. To adapt to advertising business while retaining general visual capabilities, this application incorporates a low-rank adaptive (LoRA) module into the self-attention mechanism within the Transformer encoding layer.

[0076] Specifically, the self-attention mechanism relies on the linear projection of pre-trained weights during computation, including a query projection matrix used to determine the attention focus. and the value projection matrix used to extract feature content The original weight matrix in the pre-trained model Set to frozen state, weight matrix The dimension is This ensures that its parameters remain constant during training to maintain the ability to extract general visual features. (In querying the projection matrix...) AND-value projection matrix Along the computational path, a trainable low-rank adaptive module is connected in parallel. The low-rank adaptive module consists of a dimensionality reduction matrix. With increasing dimension matrix Concatenated. Among them, the dimension reduction matrix... The dimension is Upgraded dimensional matrix The dimension is ,in Denotes the rank of a low-rank matrix, and the rank... Much smaller than the original weight matrix Dimensions and To ensure that the model's behavior in the initial training phase is consistent with the pre-trained model, the dimensionality reduction matrix is ​​adjusted. Initialize using a Gaussian distribution for the up-dimensional matrix. Use all-zero initialization.

[0077] During the forward propagation computation, for the input hidden layer state... Calculate the corrected output The calculation formula is as follows:

[0078] ;

[0079] in, Scaling factor For constant hyperparameters, by dividing by rank To reduce the impact of the model optimization process on Sensitivity to value selection. The first term of the formula generates a basic feature vector based on general pre-trained knowledge, and the second term reduces the dimensionality of the input to rank. The bottleneck structure of the low-dimensional space is then raised back to the original space, generating a learnable incremental feature vector. Adding these two terms is geometrically equivalent to superimposing a directionally controllable offset onto the original basic feature vector. For the query projection matrix, as the parameters of the dimensionality reduction and raising matrices iterate during training, this incremental feature vector gradually evolves into an indicator vector pointing to a specific semantic region of the advertisement. Superimposing this indicator vector changes the orientation of the query vector in the feature space, increasing the dot product value with the key vectors representing key elements of the advertisement, such as product identifiers and promotional text. According to the operational principle of the self-attention mechanism, the increased dot product value shifts the attention weight distribution, thus forcing the model to shift its focus from general visually salient regions to specific regions with high marketing value.

[0080] Based on the above architecture, for the first candidate frame in the candidate frame set... frame Perform visual feature extraction. Combine the input vector... The input is fed into stacked Transformer encoding layers. During the iterative computation through multiple layers of self-attention mechanisms, thanks to the redirection of attention weights by the aforementioned low-rank adaptive module, the category labeling... It can adaptively capture and aggregate feature information highly relevant to advertising and marketing from visual token sequences while suppressing background noise interference. Category tags are extracted as the data flows through the final layer of the visual encoder. The corresponding final hidden layer state is defined as a visual semantic feature vector containing marketing intent. .

[0081] After completing the low-rank adaptive transformation of the multimodal large-model visual encoder, the text encoder is also based on the Transformer architecture. This application pre-sets a set of learnable AIDA marketing state cue vectors at the input of the text encoder as soft cues to guide the text encoder in understanding the marketing funnel structure of the advertisement. This step aims to transform the AIDA marketing funnel theory into a high-dimensional vector representation that the model can understand through soft cue technology, and to calculate the probability distribution of the current candidate frame belonging to each marketing stage.

[0082] Specifically, first, construct and initialize the AIDA marketing state vector set. Define the state vector set. The four vectors in this set correspond to the four stages of the marketing funnel model: Attention, Interest, Desire, and Action. Each element in the set... All are of dimension Instead of fixed discrete text characters, learnable continuous vectors are used. These vectors serve as soft cues updated with network parameters, responsible for representing the abstract semantics of each marketing stage in the feature space. Secondly, parallel cue inputs are constructed. For the first... Frame, obtain its corresponding text token sequence , text token sequence Each of the four marketing state vectors is concatenated to construct four parallel prompt inputs, denoted as . , Next, cross-attention interaction and probability scoring are performed. Utilizing the cross-attention mechanism of a multimodal large model, the visual category label in the composite input vector is calculated. The above four input prompts The degree of semantic matching between them. Specifically, the model labels the visual categories. The mapping is converted into a query vector, which will then pass through a text encoder. Encoded prompt input Mapped to key and value vectors, the cosine similarity between the query vector and the key vector is calculated to measure the relevance of the visual content of the current screen to the semantics of a specific marketing stage. Based on this similarity, the [missing information - likely a specific marketing stage] is calculated. The frame belongs to the marketing stage. probability score As shown below:

[0083] ;

[0084] in, Visual category tags With text encoder Encoded prompt input The cosine similarity between them is calculated by summing the exponents of the results for all four marketing stages in the denominator, and is used to perform Softmax normalization.

[0085] Finally, a marketing state distribution vector is generated. Through the above calculations, the model outputs a 4-dimensional marketing state distribution vector. This vector quantitatively characterizes the first The marketing role of frames in the advertising narrative structure, such as whether they are primarily used to attract user attention or to stimulate user desire by showcasing product details.

[0086] Furthermore, as a preferred embodiment, the aforementioned multimodal large model was trained and fine-tuned using a historical advertising video dataset before being put into use, in order to determine the parameter values ​​of the dimension reduction matrix and the dimension increase matrix in the aforementioned low-rank adaptive module, as well as the value of the AIDA marketing status prompt vector set.

[0087] Specifically, this application utilizes historical advertising video datasets and minimizes the multi-task joint loss function. This completes the iterative update and determination of the above parameters. The multi-task joint loss function consists of semantic alignment loss. Losses related to marketing stage classification It consists of two parts, and the specific construction and calculation process is as follows:

[0088] First, construct the semantic alignment loss function. To ensure semantic consistency between visual content and product text descriptions in the feature space, an Information Noise Contrast Estimation Loss (InfoNCE Loss) is used as a constraint. This is applied to a feature space containing... Batch of video-text pairs, semantic alignment loss function The expression is as follows:

[0089] ;

[0090] in, This indicates the total number of sample pairs in the current training batch. and All are sample indices, with values ​​ranging from 1 to... , Indicates the first The normalized visual feature vectors obtained by extracting video samples through a visual encoder. Indicates the first The normalized text feature vectors obtained by extracting the correct product text corresponding to each video sample using a text encoder. Forming positive sample pairs Indicates the first in the current batch The text feature vector corresponding to each sample To form negative sample pairs, The temperature coefficient is a preset constant hyperparameter used to adjust the model's sensitivity to distinguishing negative samples. This is achieved by minimizing... This drives the model parameter updates, increasing the dot product value of matched positive sample pairs in the feature space and decreasing the dot product value of unmatched negative sample pairs.

[0091] Secondly, construct the classification loss function for the marketing stage. To enable the model to identify AIDA marketing stages, the true marketing stage labels of the samples are used as supervision signals, and the cross-entropy loss function is employed for constraint. Marketing stage classification loss function. The expression is as follows:

[0092] ;

[0093] in, Indicates the first The real label corresponding to the frame is represented as a one-hot vector, that is, when the first frame... The frame truly belongs to the first Each stage ,otherwise .

[0094] Finally, the overall optimization objective is defined and the parameters are updated. The weighted sum of the two losses yields the overall optimization objective function as follows:

[0095] ;

[0096] in, For the parameter set of the low-rank adaptive module, This is a task balancing hyperparameter used to adjust the weight of classification tasks in the total loss during the marketing phase. During backpropagation using historical data, only the parameter set of the low-rank adaptive module is considered. The gradients of the (i.e., the reduced-dimensional matrix and the increased-dimensional matrix) and the marketing status cue vector set are calculated and their values ​​are updated, while the original weights of the pre-trained model remain completely frozen. Through several rounds of iterative training, when the overall optimization objective function converges to a preset threshold, training is stopped and the above parameters are locked, thus completing the parameter determination of the multimodal large model.

[0097] Step S103: Use visual semantic feature vectors to represent semantic relevance, introduce a bias correction term based on the marketing state distribution vector, and calculate the single-frame recommendation score by combining the visual saliency score.

[0098] In step S103, a bias-reduction scoring mechanism is constructed to address the common visual salience bias problem in advertising video editing by decoupling the visual appeal of the image from the substantive value of the content. This mechanism uses mathematical calculations to eliminate inflated scores caused solely by visual appeal, thereby selecting high-quality shots that accurately convey product information.

[0099] Specifically, firstly, the system quantifies the superficial visual appeal of the current frame. This metric represents the degree of instinctive stimulation the image provides to the user's visual senses, and is considered a distraction that needs to be controlled. The system utilizes an existing lightweight saliency detection network. For the first The system extracts features from each frame, analyzes the color vibrancy, brightness contrast, and texture complexity of the image, and outputs a normalized numerical value, namely the visual saliency score. A higher score indicates that the image is more likely to attract attention, but this does not necessarily mean that the image contains substantial product information.

[0100] Secondly, quantify the semantic relevance of the current frame to the core selling points of the product. This metric represents the true driving force behind the visual's ability to promote user conversion. This is achieved by utilizing the visual semantic feature vector containing marketing intent output in step S102. As input, the text description of the selling points of the current advertised product (such as ultra-long battery life or whitening and spot removal) is obtained, and then mapped into a selling point feature vector by a text encoder. Calculate visual semantic feature vectors With selling point feature vector The semantic transformation score is obtained by calculating the dot product similarity between the two pairs of strings. ,Right now This score quantitatively characterizes the semantic relevance between visual content and textual selling points, reflecting whether the content of the current frame accurately showcases the core functions of the product.

[0101] Finally, the counterfactual correction formula is used to fuse the two scores to eliminate baseline bias caused by visual salience. During this process, the marketing state distribution vector output in step S102... Used to determine the degree of deviation correction at different marketing stages. Defines the final single-frame recommendation score. The calculation formula is as follows:

[0102] ;

[0103] in, The debiasing coefficient is a constant used to control the penalty weight for visually saliency factors; The Sigmoid activation function is used to map visual saliency scores to a nonlinear space to simulate the saturation effect perceived by the human eye. This is a dynamic visual bias term. The value of this term is not fixed, but depends on the marketing state distribution vector output in step S102. The marketing stage (such as attention, interest, desire, or action) of the current frame is determined based on the marketing state distribution vector, and the deviation benchmark value corresponding to that specific stage is retrieved from the preset historical statistics. .

[0104] Using this calculation formula, the system subtracts a correction term, composed of visual salience and deviation baseline values, from the semantic conversion score. This reduces the weight of visually elaborate but content-empty frames and increases the weight of visually simple but accurately highlighting selling points frames, ultimately generating a single-frame recommendation score. It can accurately reflect the degree to which the frame contributes to the advertising conversion goal.

[0105] Step S104: Construct a kernel matrix using the single-frame recommendation score and inter-frame visual similarity, perform matrix multiplication optimization processing, and select the key frame set from the candidate frame set.

[0106] In step S104, this step aims to address the redundancy problem in the screening results that may result from simply sorting by score. Since a high score for a single frame does not directly equate to the overall performance of the frame combination, in order to avoid the final selected keyframe set containing a large number of shots with highly repetitive visual content (such as multiple extremely similar product close-ups), this application introduces a Determinantal Point Process (DPP) to optimize the diversity of combinations of the candidate frame set.

[0107] Specifically, firstly, based on the calculation results of the preceding steps, the basic metrics used to construct the kernel matrix are defined. On one hand, the first... Frame quality metrics This metric is directly derived from the single-frame recommendation score calculated in step S103. That is, the corresponding number Frames To ensure the numerical values ​​are non-negative and to enhance the discriminative power, an exponential function is used for mapping, and the calculation formula is as follows: On the other hand, defining the first Frame and the Visual similarity measurement between frames This metric is based on the visual semantic feature vector output in step S102. and Calculate cosine similarity to quantitatively characterize the degree of overlap between two frames at the visual semantic level.

[0108] Secondly, using the aforementioned quality metric and visual similarity metric, a dimension is constructed as follows: positive semidefinite kernel matrix nuclear matrix Each element in It integrates information on quality and diversity, and its definition formula is as follows:

[0109] ;

[0110] in, For the first The quality metric for candidate frames is finally based on the kernel matrix. The process involves performing diversity combinatorial optimization to select the optimal subset from the candidate frame set. This process is divided into two parts: establishing the optimization objective and performing approximate solutions. At the objective establishment level, based on the mathematical principles of determinant point processes, the selection objective is defined as finding a subset. (i.e., potential keyframe combinations), select a subset from the candidate frame set. probability Proportional to the submatrix corresponding to that subset The determinant value is given by the formula: .in, Indicates that only a subset is retained. The submatrix formed by the rows and columns corresponding to the indices. In geometric space, the determinant of this submatrix is... The value of corresponds to the volume of the parallel polyhedron spanned by all eigenvectors in the subset. The magnitude of this volume is determined by the magnitude of the vectors (corresponding to the quality of the frame) and the angle between the vectors (corresponding to the dissimilarity between frames). Therefore, maximizing this determinant value mathematically drives the algorithm to find combinations of frames with large magnitudes (high quality) and orthogonality (large differences), thus fundamentally solving the redundancy problem.

[0111] At the approximate solution execution level, since traversing all subsets to find the global optimum is computationally infeasible, a greedy algorithm is used for approximate solution. Specifically, the keyframe set is initialized to empty. In each iteration, all unselected candidate frames are traversed, and the frame added to the current set that maximizes the logarithmic increment of the submatrix determinant is calculated. This frame is then selected. This process is repeated until the number of selected frames reaches a preset threshold. This yields the final set of keyframes. .

[0112] Step S105: Obtain click conversion data of the keyframe set, calculate feedback reward using the click conversion data, and update the parameters of the multimodal large model online based on the feedback reward using the policy gradient algorithm.

[0113] In step S105, the aim is to construct an online reinforcement learning loop based on real business feedback. By feeding back user behavior data after actual deployment to the model, the multimodal large model can continuously adapt to real-time visual preference changes in different vertical industries (such as gaming, beauty, or e-commerce), thereby achieving self-evolution of the model.

[0114] Specifically, firstly, the set of keyframes selected in step S104 is deployed to a real advertising platform. During the deployment period, the number of impressions and clicks for each frame on the user's end are tracked and recorded in real time. The acquired click-conversion data includes, but is not limited to, whether the keyframe was clicked by a user, the conversion rate after the click, and the click-through rate (CTR) after the impression. This real data constitutes an objective standard for evaluating the quality of the model's selection. Secondly, a discretized reward function is constructed using the collected click-conversion data to quantify the business value of the current keyframe. The system calculates a feedback reward for each deployed keyframe. The calculation rules are as follows:

[0115] ;

[0116] In this formula, a positive reward of 1 is given when a user clicks on a keyframe, encouraging the model to use similar visual features in the future through positive feedback; a negative reward is given when the user does not click. ,in, This is the penalty coefficient for non-clicks, usually set to a small positive number. The purpose of introducing this coefficient is to balance the natural imbalance between positive and negative samples in advertising scenarios (i.e., more exposures, fewer clicks), and to prevent the model from falling into a conservative strategy due to excessive negative feedback.

[0117] Finally, the model parameters are updated using the policy gradient algorithm based on feedback rewards. In this process, the multimodal large model in step S102 is treated as a policy network in reinforcement learning. To ensure computational efficiency and maintain the model's generalizability, this step only applies to the parameters of the low-rank adaptive (LoRA) module in the model. Update the model while keeping the pre-trained weights of the base model frozen. Output keyframes. Rather than the feedback rewards received after the campaign Bind and construct trajectory data The trajectory data records a single action chosen by the model and its corresponding environmental feedback. To accurately evaluate the performance of the current keyframe, it is necessary to estimate the dominance function. Advantage function This is used to quantitatively represent the increase in click-through rate (CTR) achieved by a keyframe relative to the average level. The reward received for a given keyframe. When it is above the historical average, the corresponding advantage function A positive value indicates that the frame has positive service gain; conversely, a negative value indicates a negative value. The gradient direction for parameter updates is calculated using the Proximity Policy Optimization (PPO) algorithm. The gradient calculation formula is as follows:

[0118] ;

[0119] in, This represents the set of parameters for the low-rank adaptive module to be updated. This represents the objective function for optimization, i.e., the model's optimization with the current parameters. The expected cumulative reward that can be obtained below; Indicates the parameter The partial derivative (gradient) is the symbol used to determine the gradient in order to maximize the objective function. ,parameter The direction that should be adjusted; This represents the trajectory data collected. Trajectory distribution of samples in the middle. This represents the mathematical expectation, i.e., the expectation from the distribution. The average value is calculated on a batch of sampled data to ensure the stability of the gradient estimation; This represents the output probability of a multimodal large model. Specifically, it refers to the probability of output given video content. Under these conditions, the model selects specific keyframes. The probability of; This represents the advantage function of the estimation, which acts as a scalar weight in the gradient direction.

[0120] The gradient obtained based on the above calculation The parameters of the low-rank adaptive (LoRA) module are updated by adjusting the parameter values ​​along the gradient direction, enabling the model to generate high-advantage values ​​with a higher probability when faced with similar video content in the future. Keyframes are used to enable online evolution for specific vertical industries.

[0121] In summary, combining Figure 2 This application proposes a method for keyframe recognition in advertising videos. The method first performs multimodal deconstruction and adaptive temporal pyramid sampling on the original advertising video. The sampling rate is dynamically adjusted by calculating the perceptual hash difference between adjacent frames to construct a candidate frame set. The candidate frame set is then input into a multimodal large model configured with a low-rank adaptive module and marketing status cues. Through inference, visual semantic feature vectors containing marketing intent and marketing status distribution vectors representing the narrative roles in the advertisement are extracted. Based on this, the visual semantic feature vectors represent semantic relevance, and a dynamic bias correction term is introduced based on the marketing status distribution vector. A causal bias correction formula is constructed by combining visual saliency scores to calculate the single-frame recommendation score. Next, a kernel matrix is ​​constructed based on the single-frame recommendation score and inter-frame visual similarity. A matrix optimization process is performed using a matrix-determinant point process diversity combination, and the geometric volume maximization principle is used to filter the candidate frame set to obtain the keyframe set. Finally, an online reinforcement learning loop is established to obtain real click-through conversion data for the keyframe set and calculate feedback rewards. A proximal policy optimization algorithm is used to update the parameters of the low-rank adaptive module online, enabling the model to self-evolve.

[0122] First, to address the issue of inflated scores due to mere visual saliency, this application constructs a scoring mechanism based on counterfactual correction. It uses causal inference principles to define and eliminate visual saliency as interference, while dynamically adjusting the deviation benchmark for different marketing stages using a marketing state distribution vector. This ensures that the final high-scoring keyframes are driven by semantic relevance highly matched to the product's selling points, rather than simply by visual appeal. Second, to address the visual redundancy issue in the selection results, this application introduces a determinant point process to optimize the candidate frame set. By maximizing the determinant of the kernel matrix composed of quality and similarity metrics, the algorithm mathematically finds a combination of feature vectors with large modulus lengths and nearly orthogonal angles between them. This maximizes the visual diversity of the set while ensuring high quality for individual frames. Finally, to address the difficulty of general models adapting to specific vertical industry preferences, this application uses a policy gradient algorithm to feed back real business click feedback to the model, fine-tuning only the low-rank adaptive module. This allows the model to continuously learn and capture specific visual features that trigger user click behavior in specific industries (such as gaming or beauty), achieving adaptive evolution of the model to dynamic market preferences.

[0123] Furthermore, addressing the technical problem of existing technologies failing to decouple visual saliency from actual conversion value, resulting in high-quality but inefficient keyframes, this application achieves the following technical benefits: By constructing a scoring formula based on causal inference, visual saliency can be defined as an interference factor and mathematically eliminated using dynamic bias terms, thereby forcibly achieving deep decoupling between visual stimuli and the semantic value of the product during the scoring stage. This mechanism ensures that the final selected keyframes are based on their accurate expression of the product's selling points rather than simply visual appeal, effectively avoiding the selection of empty images. Simultaneously, supplemented by online strategy updates based on real business feedback, the model's misunderstanding of high-conversion features is further corrected, ensuring that the selected keyframes not only possess visual quality but also effectively stimulate users' desire to purchase, significantly improving the actual click-through rate of advertising videos.

[0124] Figure 3 This is a structural block diagram of an advertising video keyframe recognition system provided in one embodiment of this application. The system includes at least the following modules:

[0125] The video deconstruction module is used to acquire the original advertising video and perform multimodal data deconstruction and adaptive temporal pyramid sampling to construct a candidate frame set;

[0126] The vector extraction module is used to input the candidate frame set into a multimodal large model configured with a low-rank adaptive module and marketing status prompts, and perform inference to extract visual semantic feature vectors and marketing status distribution vectors.

[0127] The scoring module is used to represent semantic relevance using visual semantic feature vectors, introduce a bias correction term based on the marketing state distribution vector, and calculate the single-frame recommendation score by combining the visual saliency score.

[0128] The keyframe filtering module is used to construct a kernel matrix using the single-frame recommendation score and inter-frame visual similarity, perform a matrix-based multivariate combination optimization process, and filter the candidate frame set to obtain the keyframe set.

[0129] The strategy update module is used to acquire click conversion data from the keyframe set, calculate feedback rewards using the click conversion data, and update the parameters of the multimodal large model online based on the feedback rewards using a strategy gradient algorithm.

[0130] For relevant details, please refer to the above method implementation examples.

[0131] Figure 4 This is a block diagram of an electronic device provided in one embodiment of this application. The device includes at least a processor 401 and a memory 402.

[0132] Processor 401 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0133] The memory 402 may include one or more computer-readable storage media, which may be non-transitory. The memory 402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 402 are used to store at least one instruction, which is executed by the processor 401 to implement the advertising video keyframe recognition method provided in the method embodiments of this application.

[0134] In some embodiments, the electronic device may also optionally include: a peripheral device interface and at least one peripheral device. The processor 401, memory 402, and peripheral device interface can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface via a bus, signal line, or circuit board. Indicatively, peripheral devices include, but are not limited to: radio frequency circuits, touch displays, audio circuits, and power supplies.

[0135] Of course, electronic devices may also include fewer or more components, and this embodiment does not limit this.

[0136] Optionally, this application also provides a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the advertising video keyframe recognition method of the above method embodiments.

[0137] Optionally, this application also provides a computer product including a computer-readable storage medium storing a program that is loaded and executed by a processor to implement the advertising video keyframe recognition method of the above method embodiments.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for keyframe recognition in advertising videos, characterized in that, The method includes: The original advertising video is acquired and subjected to multimodal data deconstruction and adaptive temporal pyramid sampling to construct a candidate frame set; The candidate frame set is input into a multimodal large model configured with a low-rank adaptive module and marketing status prompts, and inference is performed to extract visual semantic feature vectors and marketing status distribution vectors. The visual semantic feature vector is used to represent semantic relevance, and a bias correction term is introduced based on the marketing state distribution vector. The single-frame recommendation score is calculated by combining the visual saliency score. The process of using the visual semantic feature vector to represent semantic relevance, introducing a bias correction term based on the marketing state distribution vector, and calculating the single-frame recommendation score in conjunction with the visual saliency score includes: Using a lightweight saliency detection network For the first Frame candidate frames Perform feature extraction and output visual saliency scores. ; Obtain the textual description of the selling points of the current advertised product, and map it into a feature vector of selling points using a text encoder. ; Computing visual semantic feature vectors With selling point feature vector The semantic transformation score is obtained by calculating the dot product similarity between the two pairs of strings. , ; The ratings are fused and calculated using a counterfactual correction formula, and a single-frame recommendation score is defined. The calculation formula is as follows: ； in, The debiasing coefficients are in constant form. Use the Sigmoid activation function; As a dynamic visual deviation term, the marketing stage to which the current frame belongs is determined based on the marketing state distribution vector, and the deviation benchmark value corresponding to that specific stage is retrieved from preset historical statistical data as... ; A kernel matrix is ​​constructed using the single-frame recommendation score and inter-frame visual similarity, and a matrix multiplication optimization process is performed to filter out the key frame set from the candidate frame set. The click conversion data of the keyframe set is obtained, the feedback reward is calculated using the click conversion data, and the parameters of the multimodal large model are updated online based on the feedback reward using a policy gradient algorithm.

2. The advertising video keyframe recognition method according to claim 1, characterized in that, The process of acquiring the original advertising video, performing multimodal data deconstruction and adaptive temporal pyramid sampling to construct a candidate frame set includes: Receive the input raw advertising video file Original advertising video file Contains the original frame sequence Audio stream and metadata ,in, Indicates the total number of original frames; An adaptive temporal pyramid sampling strategy is used to filter the original frame sequence, and the perceptual hash difference between two adjacent frames in the original frame sequence is calculated. A dynamic sampling rate function is constructed based on this perceptual hash difference. The calculation formula is: ,in, and All are preset adjustment coefficients; the original frame sequence is non-uniformly sampled according to the calculated dynamic sampling rate to construct a candidate frame set. ,in This represents the total number of candidate frames after sampling; Using automatic speech recognition models to analyze audio streams Transcription is performed to generate a text sequence containing timestamp information. Optical character recognition (OCR) technology is used to extract the superimposed text information on each frame in the candidate frame set to obtain the corresponding OCR text. .

3. The advertising video keyframe recognition method according to claim 2, characterized in that, The step of inputting the candidate frame set into the multimodal large model configured with a low-rank adaptive module and marketing status cues includes: The first Frame candidate frames Image segmentation Each image patch is then processed through a linear projection layer. Each image patch is mapped to a visual token sequence. ,in Represents category labels used to characterize the global semantics of an image; Will be with candidate frames Corresponding OCR text and text sequences Character-level concatenation is performed according to a preset order to form a complete text description. Then, a text segmenter is used to segment and vectorize the text description, generating a text token sequence composed of multiple text tags. ; Introducing learnable temporal location vectors The formula for calculating this time-series position vector is: ,in, This represents the sinusoidal position encoding function; Aggregate the generated vectors to generate the first vector. Composite input vector of frames The calculation method is as follows ,in This indicates a splicing operation.

4. The advertising video keyframe recognition method according to claim 3, characterized in that, The inference process to extract visual semantic feature vectors and marketing status distribution vectors includes: The multimodal large model includes a visual encoder, which employs a Transformer-based network architecture; the self-attention mechanism within the Transformer encoding layer includes querying the projection matrix. and value projection matrix The original weight matrix in the pre-trained model Keeping unchanged, in querying the projection matrix AND-value projection matrix Along the computational path, a trainable low-rank adaptive module is connected in parallel. The low-rank adaptive module is composed of a dimensionality reduction matrix. With increasing dimension matrix Composed of series connection; During the forward propagation computation, for the input hidden layer state... Calculate the corrected output The calculation formula is as follows: ； in, Scaling factor For constant hyperparameters, Let be the rank of the low-rank matrix; composite input vector The input is fed into the Transformer encoding layer, where it undergoes iterative computation through multiple layers of self-attention mechanisms to extract class labels. The corresponding final hidden layer state is defined as a visual semantic feature vector containing marketing intent. .

5. The advertising video keyframe recognition method according to claim 4, characterized in that, The inference process for extracting the visual semantic feature vector and marketing state distribution vector also includes: The multimodal large model includes a text encoder, with a pre-set AIDA marketing status cue vector set at the input of the text encoder. The four vectors correspond to the four stages of attention, interest, desire, and action in the marketing funnel model; for the first vector in the candidate frame set... Frame, obtain its corresponding text token sequence , text token sequence Each of the four marketing state vectors is concatenated to construct four parallel prompt inputs, denoted as . ; Label visual categories The mapping is converted into a query vector, which will then pass through a text encoder. Encoded prompt input Mapped to key vectors and value vectors, the first... The frame belongs to the marketing stage. probability score As shown below: ； in, Visual category tags With text encoder Encoded prompt input Cosine similarity between them; Output marketing state distribution vector .

6. The advertising video keyframe recognition method according to claim 1, characterized in that, The step of constructing a kernel matrix using the single-frame recommendation score and inter-frame visual similarity, performing determinant-based point-based diversity combination optimization processing, and selecting the keyframe set from the candidate frame set includes: Definition of the first Quality metrics of candidate frames , , For the first The single-frame recommendation score of the frame; defining the first frame Frame candidate frame and the first Visual similarity measurement between candidate frames , and For the first Frame candidate frame and the first Visual semantic feature vectors of candidate frames; Construction dimension kernel matrix nuclear matrix Each element in As shown below: ； in, For the first Quality metrics for candidate frames; based on kernel matrix The process involves performing a variety of combinatorial optimizations to select the optimal subset from the candidate frame set. The process is divided into establishing the optimization objective and performing approximate solutions. The selection objective is defined as finding a subset. Select a subset from the candidate frame set probability Proportional to the submatrix corresponding to that subset The determinant value is given by the formula: ; in, Representing a subset The submatrix formed by the rows and columns corresponding to the index. Represents the determinant of a submatrix; The keyframe set is initialized to empty. In each iteration, all unselected candidate frames are traversed, and the frame that maximizes the logarithmic increment of the determinant of the submatrix when added to the current set is selected. This process is repeated until the number of selected frames reaches a preset threshold. To obtain the keyframe set .

7. A keyframe recognition system for advertising videos, characterized in that, include: The video deconstruction module is used to acquire the original advertising video and perform multimodal data deconstruction and adaptive temporal pyramid sampling to construct a candidate frame set; The vector extraction module is used to input the candidate frame set into a multimodal large model configured with a low-rank adaptive module and marketing status prompts, and perform inference to extract visual semantic feature vectors and marketing status distribution vectors. The scoring module is used to represent semantic relevance using the visual semantic feature vector, introduce a bias correction term based on the marketing state distribution vector, and calculate the single-frame recommendation score by combining the visual saliency score. The process of using the visual semantic feature vector to represent semantic relevance, introducing a bias correction term based on the marketing state distribution vector, and calculating the single-frame recommendation score in conjunction with the visual saliency score includes: Using a lightweight saliency detection network For the first Frame candidate frames Perform feature extraction and output visual saliency scores. ; Obtain the textual description of the selling points of the current advertised product, and map it into a feature vector of selling points using a text encoder. ; Computing visual semantic feature vectors With selling point feature vector The semantic transformation score is obtained by calculating the dot product similarity between the two pairs of strings. , ; The ratings are fused and calculated using a counterfactual correction formula, and a single-frame recommendation score is defined. The calculation formula is as follows: ； in, The debiasing coefficients are in constant form. Use the Sigmoid activation function; As a dynamic visual deviation term, the marketing stage to which the current frame belongs is determined based on the marketing state distribution vector, and the deviation benchmark value corresponding to that specific stage is retrieved from preset historical statistical data as... ; The keyframe filtering module is used to construct a kernel matrix using the single-frame recommendation score and inter-frame visual similarity, perform matrix multiplication optimization processing, and filter the keyframe set from the candidate frame set. The strategy update module is used to acquire click conversion data of the keyframe set, calculate feedback rewards using the click conversion data, and update the parameters of the multimodal large model online based on the feedback rewards using a strategy gradient algorithm.

8. An electronic device, characterized in that, The device includes a processor and a memory; the memory stores a program that is loaded and executed by the processor to implement an advertising video keyframe recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a program that, when executed by a processor, is used to implement an advertising video keyframe recognition method as described in any one of claims 1 to 6.

Images