Video language understanding method and device based on semantic saliency probe

Abstract

The invention discloses a video language understanding method and device based on a semantic saliency probe, and the method comprises the steps: firstly, receiving a video sequence, carrying out the feature extraction frame by frame, converting an original image frame into a low-dimensional semantic feature vector, and obtaining a frame-level feature sequence; then, inputting the frame-level feature sequence and the text semantic vector into a semantic significance probe, calculating a frame importance score of each frame feature, and outputting frame weight distribution; selecting the first K frames with the highest frame importance score for aggregation to obtain key frame representation; and finally, taking the key frame representation as condition input, combining the language features, and generating video natural language description in an autoregression mode. According to the method, continuous frame importance distribution is output through the semantic significance probe, the defect that traditional hard screening cannot be differentiated is overcome, and integrated optimization of key frame selection and description generation is achieved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically, to a video language understanding method and apparatus based on semantic saliency probes. Background Technology

[0002] With the explosive growth of surveillance videos, social media videos, and multimedia data, automatic video understanding and description generation has become an important direction in artificial intelligence research and applications. Existing technologies typically rely on convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformer models based on self-attention mechanisms in deep learning to model video frame sequences and then generate natural language descriptions.

[0003] In practical applications (such as intelligent security, event retrieval, and video summarization), keyframe selection is a core step in video understanding. However, most frames in a video are redundant, with only a small number of keyframes containing information that is crucial for semantic description. Therefore, efficiently selecting keyframes becomes a challenge for video language understanding models.

[0004] Currently, video frame weighting is mainly based on cross-modal attention mechanisms. After the video encoder outputs frame features, a cross-modal attention mechanism is introduced to calculate the correlation between frame features and language features, and a weighted average is used to generate the overall representation of the video, which is then input into the language decoder to generate a description. Although this method can capture the correspondence between video frames and language semantics to some extent, it still has the following shortcomings: Lack of sparsity: Attention mechanisms typically produce a smooth distribution, with almost all frames being assigned a certain weight, making it difficult to explicitly highlight the few truly key frames. Poor interpretability: The generated weights cannot be directly mapped to a limited number of keyframes, which is not conducive to providing interpretable evidence in scenarios such as monitoring and evidence collection; Insufficient semantic drive: It relies on attention mechanisms for self-learning, lacks explicit modeling of factors such as saliency and target, and is easily affected by redundant background frames, causing the generated descriptions to deviate from the actual events. Summary of the Invention

[0005] In view of this, the present invention provides a video language understanding method and apparatus based on semantic saliency probes, which solves the problems of poor interpretability and insufficient semantic drive in existing video language understanding methods.

[0006] To achieve the above objectives, the following solution is proposed: A video language understanding method based on semantically saliency probes includes: The video sequence is received and features are extracted frame by frame. The original image frames are replaced with low-dimensional semantic feature vectors to obtain a frame-level feature sequence. The frame-level feature sequence and text semantic vector are input into the semantic saliency probe to calculate the frame importance score of each frame feature and output the frame weight distribution. The top K frames with the highest frame importance scores are selected and aggregated to obtain the keyframe representation; Using keyframe representations as conditional inputs and combining language features, a video natural language description is generated through an autoregressive approach.

[0007] Preferably, the text semantic vector is obtained by encoding based on the text description or semantic prompts provided during the training phase.

[0008] Preferably, a convolutional neural network or a visual Transformer is used to extract features from each frame of the video sequence.

[0009] Preferably, the process of calculating the frame importance score of each frame feature and outputting the frame weight distribution includes: Calculate the semantic similarity score, saliency score, and targetability score for each frame. Standardize each score; By fusing the scores, the frame importance score of each frame feature is obtained; The frame weight distribution is obtained by Softmax normalization.

[0010] Preferably, the formula for calculating the significance score is: ; ; ; in, For the intra-frame patch feature set, d is the feature dimension, and N is the total number of features. This is a mapping function.

[0011] Preferably, the formula for calculating the target score is: ; in, For learnable target detection vectors, This is the Sigmoid function. A video language understanding device based on a semantically salient probe includes: The video feature extraction module receives the video sequence and extracts features frame by frame, converting the original image frames into low-dimensional semantic feature vectors to obtain a frame-level feature sequence. The weight calculation module takes the frame-level feature sequence and text semantic vector as input to the semantic saliency probe, calculates the frame importance score of each frame feature, and outputs the frame weight distribution. The keyframe extraction module selects the top K frames with the highest frame importance scores and aggregates them to obtain the keyframe representation; The video language generation module uses keyframe representations as input and combines them with language features to generate natural language descriptions of the video through an autoregressive approach.

[0012] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This invention provides a video language understanding method based on a semantically saliency probe. The method receives a video sequence and extracts features frame by frame, converting the original image frames into low-dimensional semantic feature vectors to obtain a frame-level feature sequence. This frame-level feature sequence and text semantic vectors are input into a semantically saliency probe to calculate the frame importance score of each frame's features, outputting a frame weight distribution. The top K frames with the highest frame importance scores are selected and aggregated to obtain keyframe representations. Using these keyframe representations as conditional inputs, combined with language features, a video natural language description is generated through an autoregressive approach. This invention outputs a continuous frame importance distribution through the semantically saliency probe, avoiding the non-differentiable drawbacks of traditional hard-screening, and achieving integrated optimization of keyframe selection and description generation.

[0013] This invention introduces a universal cue template and optional slot-filling semantic cue, enabling frame importance assessment to rely not only on visual saliency and target relevance but also on linguistic semantic guidance, thereby more accurately highlighting key frames relevant to the target event. Using a unified cue template ensures consistent input format during training and inference, guaranteeing the stability of frame importance judgments in practical applications and reducing semantic shift and distribution drift issues. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0015] Figure 1 A flowchart of a video language understanding method based on a semantically salient probe provided in an embodiment of the present invention; Figure 2 This is an architecture diagram of a video language understanding method based on semantically salient probes provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the semantically significant probe weight distribution calculation process provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a video language understanding device based on a semantically salient probe, provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] First, combined Figure 1-2 This invention introduces a video language understanding method based on a semantically saliency probe, as provided in an embodiment of the present invention. Figure 1-2 As shown, the method includes: Step S01: Receive the video sequence and extract features frame by frame, replacing the original image frames with low-dimensional semantic feature vectors.

[0018] Specifically, the input video sequence is received and fed frame by frame into a video encoder. The video encoder extracts features from each frame of the input video sequence, converting the original image frames into low-dimensional semantic feature vectors, thus obtaining a frame-level feature sequence. The video encoder can be a convolutional neural network (CNN), a visual Transformer (ViT), etc. The video encoder takes video frames as input. Output frame features ,in, Input the image for frame t. Let be the feature vector of frame t. , H represents height, W represents width, and C represents channel. Let be a real number, and d be the feature dimension.

[0019] Step S02: Input the frame-level feature sequence and text semantic vector into the semantic saliency probe, and calculate the frame importance score of each frame feature.

[0020] Specifically, during the training phase, the text description or semantic cues corresponding to the video are input into the text encoder. The text encoder encodes the input text, mapping it to a semantic vector to provide semantic guidance for frame importance scoring. The text encoder can use BERT or Transformer Encoder. The text encoder takes a text sequence S as input. Output the text semantic vector z, where, .

[0021] Considering the generalization ability of the trained model during inference, a unified text input needs to be designed to be converted into semantic vectors. Slot-filling can be used to write all prompts in a uniform structure, ensuring strict consistency between training and inference. During the training phase, half of the data uses existing sentence labels to analyze and fill templates using LLM (Large Language Model), while the other half uses templates with "NONE" filled in. Correspondingly, during the inference phase, when there are user prompts, such as "What is that man doing?", some information can be extracted to fill the template, such as "subject = man". When there are no existing sentence prompts during inference, the "NONE" template continues to be used. The advantage of this approach is that most samples maintain consistency using a general template, while a small portion of samples use captions to fill slots and enhance semantic alignment. This allows the probe to learn to "automatically find keyframes without prompts" and also performs better under linguistic constraints. The slot-filling template is set as follows: [CLS] Task = Keyframe Selection; Scene = [SCENE]; Subject = [SUBJECT]; Action = [ACTION]; Object = [OBJECT]; Location = [LOCATION]; Timeline = [TIMEHINT]; Context = [CONTEXT]; Focus Factors = [FOCUS]; Ignore element = [NEGATIVE]; Output = Select the frame most relevant to the semantics described above [SEP] The slotted template is converted into a text semantic vector using a text encoder.

[0022] Furthermore, the training phase of the semantically salient probe is differentiable, and the inference phase can directly output a set of keyframes, taking into account both sparsity and interpretability.

[0023] The video frame feature sequence and the text semantic vector are input together into the semantically salient probe. For example... Figure 3 As shown, the semantic saliency probe calculates the frame importance score for each frame using a multi-factor scoring mechanism (semantic similarity, saliency, and targetability). The scores are then normalized, and the frame weight distribution is output to form a frame-level importance representation. By comprehensively considering semantic similarity, saliency, and targetability, the frame selection results are ensured to be both semantically relevant and highlight visually salient targets, providing a basis for subsequent keyframe selection. The semantic saliency probe is input to a video frame feature sequence. Given a text semantic vector z and a video frame set T (total number of video frames), output the frame weight distribution. The fusion feature representation, wherein the frame weight distribution satisfies The process of calculating the frame importance score for each frame feature is as follows: (1) Calculate the semantic similarity score of each frame. Semantic similarity score is used to measure the degree of semantic relevance between a frame and the text.

[0024]

[0025] in, The semantic similarity score. These are the learnable projection matrices for video frames and text semantics, respectively.

[0026] (2) Calculate the saliency score for each frame. The saliency score is used to measure how well a frame is visually focused.

[0027] ; ; ; in, The significance score is... The larger the value, the more focused the attention on the frame, and the more likely it is to be a semantically salient frame. For the intra-frame patch feature set, This is a mapping function used to generate patch attention distributions (e.g., lightweight MLP or ViT attention heads). Let be the attention distribution entropy.

[0028] (3) Calculate the target score for each frame. The target score is used to measure whether a frame contains a describable target or action.

[0029]

[0030] in, For target scores, For learnable target detection vectors, This is the Sigmoid function, with an output range of [0,1].

[0031] (4) Fraction fusion and normalization First, standardize the subfractions: ,in, , , These are the standardized semantic similarity score, saliency score, and target score, respectively. , These are the mean and standard deviation of the semantic similarity scores, respectively. , These are the mean and standard deviation of the significance scores, respectively. , These are the mean and standard deviation of the target scores, respectively.

[0032] Then, the scores are combined to obtain the frame importance score for each frame feature. The frame importance score is as follows:

[0033] in, These are the fusion coefficients, used to balance the influence of the three types of factors. This is a bias term.

[0034] Finally, the frame weight distribution is obtained through Softmax normalization. The normalization process is as follows:

[0035] in, This is a temperature parameter used to control the smoothness of the weight distribution.

[0036] Step S03: Select the top K frames with the highest frame importance scores for aggregation.

[0037] Specifically, based on the importance distribution of the probe output, during the training phase, frame weight distribution is used to weight and aggregate all frame features to obtain a global video representation. This process is fully differentiable, ensuring end-to-end training. During the inference phase, Top-K frames are selected based on frame scores to form a sparse set of keyframes, and these selected keyframes are aggregated to obtain a keyframe representation. This achieves training-inference consistency, ensures end-to-end trainability of the model, and outputs interpretable keyframe results. The specific algorithm for keyframe selection and aggregation is as follows: (1) Input and output Input: Frame feature sequence and weight distribution .

[0038] Output: Keyframe set and keyframe representation u, (Training / inference for use by the decoder) (reasoning).

[0039] (2) Training phase: soft selection and differentiable aggregation During the training phase, no discrete filtering is performed. Instead, the frame features are directly weighted and summed using softmax weights to ensure end-to-end differentiability, resulting in a global representation of the video, i.e., the keyframe representation during the inference phase. ; ; Where τ>0 represents temperature; training can gradually anneal from τ0 to a smaller value to make the distribution sharper and approximate hard selection.

[0040] To avoid "full flattening" or "over-concentration", sparsity regularization needs to be added: ; in, This is the perturbation term, used for logarithmic values ​​within parentheses that are 0. It can be set to 0.001.

[0041] Combined with the existing autoregressive loss function: ; in, The vocabulary set has been generated for history.

[0042] The loss function during the training phase is: .

[0043] During training, only the global representation of the video is used to drive the decoder and alignment loss; The gradient can be fed back to the scorer and encoder, maintaining differentiability.

[0044] (3) Reasoning stage: Hard selection (Top-K) Output a sparse set of keyframes during inference. This, along with the corresponding aggregate representation, namely the keyframe representation u, improves efficiency and interpretability.

[0045] Top-K selection: , , Where K is the frame budget (e.g., K=3) 5) To maintain amplitude consistency, the frame features are averaged to obtain the keyframe representation u.

[0046] Step S04: Using keyframe representations as conditional inputs and combining language features, a natural language description of the video is generated through an autoregressive approach.

[0047] Specifically, after acquiring the keyframe representation, it is used as a conditional input and combined with language features to generate a natural language description of the video through an autoregressive approach, achieving end-to-end video language understanding. The keyframe representation u is input into the cross-modal decoder as a generation condition, and the decoder can be a Transformer Decoder model.

[0048] The decoder combines historical generated words Generates complete video natural language description sequences autoregressively. The final output is a natural language description text and a corresponding set of keyframes, achieving a unification of video understanding and semantic interpretation. This invention not only generates natural language descriptions but also directly outputs keyframe locations as evidence, facilitating applications in monitoring, evidence collection, and other scenarios. For example, when applied to intelligent video surveillance, in actual deployment, the system takes long-term video streams captured in real-time by surveillance cameras as input. First, a video encoder extracts frame-by-frame features. Then, a semantic saliency probe module calculates the importance score of each frame, automatically selecting a small number of keyframes and aggregating them into a global video representation. Subsequently, a cross-modal decoder generates natural language descriptions based on this keyframe representation for real-time output of video event semantics. This embodiment can automatically generate descriptive statements such as "someone fell" or "suspicious person entered a restricted area" without manual annotation or monitoring, while providing corresponding keyframes as evidence. This not only improves the intelligence level of the monitoring system but also has strong interpretability and practical value.

[0049] This invention, through its embodiment, outputs a continuous distribution of frame importance using a semantically salient probe, avoiding the non-differentiability drawbacks of traditional hard screening and achieving integrated optimization of keyframe selection and description generation. By introducing multi-factor scoring such as semantic similarity, saliency, and targetability, the model can more accurately align language and video content.

[0050] This invention introduces a universal cue template and optional slot-filling semantic cuees, enabling frame importance assessment to rely not only on visual saliency and target relevance but also on linguistic semantic guidance, thereby more accurately highlighting keyframes relevant to the target event. Maintaining a consistent input format during training and inference ensures the stability of the model's frame importance judgments in practical applications, reducing semantic shift and distribution drift issues.

[0051] The following describes the video language understanding device based on semantic saliency probes provided in the embodiments of the present invention. The video language understanding device based on semantic saliency probes described below and the video language understanding method based on semantic saliency probes described above can be referred to and correspond to each other.

[0052] First, combine Figure 4 This paper introduces a video language understanding device based on semantic saliency probes, such as... Figure 4 As shown, the video language understanding device based on semantically salient probes may include: The video feature extraction module 100 receives the video sequence and extracts features frame by frame, converting the original image frames into low-dimensional semantic feature vectors to obtain a frame-level feature sequence. The weight calculation module 200 inputs the frame-level feature sequence and text semantic vector into the semantic saliency probe, calculates the frame importance score of each frame feature, and outputs the frame weight distribution. The keyframe extraction module 300 selects the top K frames with the highest frame importance scores and aggregates them to obtain the keyframe representation; The video language generation module 400 uses keyframe representations as input and combines language features to generate natural language descriptions of the video through an autoregressive approach.

[0053] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0054] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0055] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video language understanding method based on semantic saliency probes, characterized in that, include: The video sequence is received and features are extracted frame by frame. The original image frames are replaced with low-dimensional semantic feature vectors to obtain a frame-level feature sequence. The frame-level feature sequence and text semantic vector are input into the semantic saliency probe to calculate the frame importance score of each frame feature and output the frame weight distribution. The top K frames with the highest frame importance scores are selected and aggregated to obtain the keyframe representation; Using keyframe representations as conditional inputs and combining language features, a video natural language description is generated through an autoregressive approach.

2. The video language understanding method based on semantically salient probes according to claim 1, characterized in that, The text semantic vector is obtained by encoding based on the text description or semantic cues provided during the training phase.

3. The video language understanding method based on semantically salient probes according to claim 1, characterized in that, Convolutional neural networks or visual Transformers are used to extract features from each frame of the video sequence.

4. The video language understanding method based on semantically saliency probes according to claim 1, characterized in that, The process of calculating the frame importance score of each frame feature and outputting the frame weight distribution includes: Calculate the semantic similarity score, saliency score, and targetability score for each frame. Standardize each score; By fusing the scores, the frame importance score of each frame feature is obtained; The frame weight distribution is obtained by Softmax normalization.

5. The video language understanding method based on semantically salient probes according to claim 4, characterized in that, The formula for calculating the significance score is: ； ； ； in, For the intra-frame patch feature set, , d It is the feature dimension. N For the total number of features, This is a mapping function.

6. The video language understanding method based on semantically saliency probes according to claim 4, characterized in that, The formula for calculating the target score is: ； in, For learnable target detection vectors, This is the Sigmoid function.

7. A video language understanding device based on a semantically salient probe, characterized in that, include: The video feature extraction module receives the video sequence and extracts features frame by frame, converting the original image frames into low-dimensional semantic feature vectors to obtain a frame-level feature sequence. The weight calculation module takes the frame-level feature sequence and text semantic vector as input to the semantic saliency probe, calculates the frame importance score of each frame feature, and outputs the frame weight distribution. The keyframe extraction module selects the top K frames with the highest frame importance scores and aggregates them to obtain the keyframe representation; The video language generation module uses keyframe representations as input and combines them with language features to generate natural language descriptions of the video through an autoregressive approach.

Images