Context semantic modulation method and system for video subtitle generation

Abstract

The invention provides a context semantic modulation method for video subtitle generation, and belongs to the field of video subtitle generation. The method aims at solving the problems that in an existing context semantic modulation method, the semantic alignment effect is poor, and key information is prone to being ignored. According to the method, the context information of each output token is integrated, and the dynamic attention weight between the tokens is further learned, so that fine-grained semantic alignment between the to-be-generated token and the generated token is realized; according to the method, the semantic association can be more effectively modeled, and the semantic weight of the key information can be adaptively adjusted, so that the representation of important features is enhanced, and the key information is prevented from being ignored.

Description

Technical Field

[0001] This invention relates to the field of video caption generation technology, and more specifically, to a contextual semantic modulation method and system for video caption generation. Background Technology

[0002] Video description tasks aim to automatically generate natural language descriptions for input videos, and are widely used in scenarios such as multimodal understanding and intelligent retrieval. Early methods often used CNNs to extract video frame features and combined them with sequence models such as RNNs and LSTMs to generate text sequences. However, limited by language modeling capabilities and the scope of time series modeling, these methods struggle to accurately represent complex events and long-term dependent information.

[0003] In recent years, the introduction of the Transformer architecture has significantly promoted the development of video description technology. A series of self-attention-based video description models have been proposed, widely applying mechanisms such as multimodal input (images, actions, semantic labels), cross-modal alignment, and hierarchical decoding, achieving significant performance improvements. However, the Transformer architecture in existing video models mostly uses first-order attention computation, only capturing the direct correspondence between current video features and text features, ignoring the fine-grained semantic information that may exist between attention weights, and outputting vague video subtitles, usually using general terms to describe, such as using "people" to describe "man" or "woman".

[0004] To overcome the limitations of first-order attention representation, Wang et al. proposed the Attention-in-Attention (AiA) module. This structure introduces an "attention within attention" mechanism on top of the original attention mechanism. Specifically, it identifies and suppresses isolated erroneous correlations on the original attention map while enhancing correct correlations with local support. The residuals are dynamically adjusted based on the context, making the corrected attention weights more aligned with practical needs, thereby capturing more complex higher-order semantic relationships. The AiA module has demonstrated good performance in tasks such as image classification and object tracking, indicating its potential to enhance the expressive power of attention maps. However, it still suffers from poor contextual semantic alignment and a tendency to overlook key information. Summary of the Invention

[0005] The technical problem to be solved by this invention is:

[0006] To address the issues of poor semantic alignment and the tendency to overlook key information in existing context semantic modulation methods.

[0007] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0008] This invention provides a contextual semantic modulation method for video subtitle generation, comprising the following steps:

[0009] S100. Calculate the correlation between the observed partial sequence and the reference statement representation through similarity score matrix and scaling processing;

[0010] S200. Apply a bias to the attention weights using the semantic context of the current decoder to obtain the modulated correlation matrix.

[0011] S300: The correlation matrix modulated in step S200 is used to capture the correlation between features through a fully connected layer and a normalization layer;

[0012] S400 uses the decoder's positional encoding and then passes through a fully connected layer to obtain the final observed partial sequence and reference statement representation. After normalization calculation by Softmax and multiplication by the modulation correlation matrix, the final context-dependent text correlation matrix is ​​obtained.

[0013] Further, in step S100, the obtained correlation matrix between the observed partial sequence and the reference statement is as follows:

[0014]

[0015] in, This indicates the relevance between the observed partial sequence and the reference statement. ; and This represents the mapped text features, i.e., the observed partial sequence and the reference sentence representation, where l is the prediction length and dm is the batch size.

[0016] Further, in step S200, the following are included:

[0017] The decoder state at the current time t Convert to a modulation coefficient :

[0018]

[0019] in, It is a sigmoid activation function, and the output range is controlled within (0,1) to ensure the stability of the modulation coefficient; It is a state Learnable linear projection weights For state The bias term;

[0020] Modulated correlation matrix Represented as:

[0021]

[0022] Further, in step S300, the following are included:

[0023] To achieve consistent relevance between generated tokens and tokens in the reference sentence, thereby enhancing the appropriate relevance of relevant token pairs and suppressing erroneous relevance of irrelevant token pairs, the resulting modulated relevance matrix... The correlation between features is captured through fully connected layers and normalization layers:

[0024]

[0025] in, This represents the computation of the fully connected layer. , represents the correlation matrix output after the fully connected layer and layer normalization, d represents the output feature size after the fully connected layer, and LayerNorm represents the layer normalization process.

[0026] Further, in step S400, the following is included:

[0027] Use the decoder's positional encoding To help understand the relative order of tokens, a fully connected layer is used again to enhance the learning of correlations, resulting in the observed partial sequence and the related items represented by the reference statement. and reference statement as follows:

[0028]

[0029] The two relevant terms are normalized using Softmax and then multiplied with the modulation correlation matrix to obtain the context-dependent text correlation matrix. :

[0030]

[0031] A context semantic modulation system for video caption generation, the system having program modules corresponding to the steps described above, which execute the steps in the aforementioned context semantic modulation method for video caption generation when running.

[0032] A computer-readable storage medium storing a computer program configured to, when invoked by a processor, implement steps of a contextual semantic modulation method for video caption generation.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] This invention provides a contextual semantic modulation method for video subtitle generation. By integrating the contextual information of each output token, it further learns the dynamic attention weights between tokens, thereby achieving fine-grained semantic alignment between the token to be generated and the generated tokens. This method can not only model semantic associations more effectively, but also adaptively adjust the semantic weights of key information, thereby enhancing the representation of important features and avoiding the omission of key information. Attached Figure Description

[0035] Figure 1 This is a framework diagram of a contextual semantic modulation method for video subtitle generation in an embodiment of the present invention. Detailed Implementation

[0036] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0037] Specific Implementation Plan 1: Combining Figure 1 As shown, the present invention provides a contextual semantic modulation method for video subtitle generation, comprising the following steps:

[0038] The S100 attention-enhanced contextual semantic modulation (ACSM) framework differs significantly from the traditional Transformer attention module. The traditional Transformer attention module directly takes a query vector and a set of key-value pairs as input, generating a weighted sum of values ​​as output by calculating attention weights. In contrast, ACSM in the self-attention module integrates the contextual information of each output token and further learns dynamic attention weights between tokens, thereby achieving fine-grained semantic alignment between the token to be generated and the already generated tokens. This method not only models semantic associations more effectively but also adaptively adjusts the semantic weights of key information, thereby enhancing the representation of important features and preventing key information from being ignored.

[0039] Prior to the ACSM component, the masked language encoder will have observed partial sequences. and reference statement Mapped into the attention module of the decoder, and respectively using and This represents the mapped text features (the observed partial sequence and the reference sentence representation), where l is the prediction length and dm is the batch size.

[0040] The ACSM component measures the relevance between the observed partial sequence and the reference statement representation through a similarity score matrix and scaling processing:

[0041]

[0042] in, This indicates the relevance between the observed partial sequence and the reference statement. ;

[0043] S200. After obtaining this relevance, in order to dynamically adjust the emphasis of similarity, a bias is added to the attention weights using the semantic context of the current decoder.

[0044] First, the state of the decoder at the current time t. Convert to a modulation coefficient :

[0045]

[0046] in, It is a sigmoid activation function, and the output range is controlled within (0,1) to ensure the stability of the modulation coefficient; It is a state Learnable linear projection weights For state The bias term;

[0047] The final modulated correlation matrix Represented as:

[0048]

[0049] S300. Next, in order to seek the relevance consistency between the generated tokens and the tokens in the reference sentence, so as to enhance the appropriate relevance of relevant token pairs and suppress the erroneous relevance of irrelevant token pairs, the resulting modulated relevance matrix is... The correlation between features is captured through fully connected layers and normalization layers:

[0050]

[0051] in, This represents the computation of the fully connected layer. , represents the correlation matrix output after the fully connected layer and layer normalization, d represents the output feature size after the fully connected layer, and LayerNorm represents the layer normalization process;

[0052] S400. To ensure the structural correctness of the generated descriptive sentences, this invention adopts the positional encoding of the decoder. To help understand the relative order of tokens, a fully connected layer is used again to enhance the learning of correlations, resulting in the observed partial sequence and the related items represented by the reference statement. and reference statement as follows:

[0053]

[0054] The two relevant terms are normalized using Softmax and then multiplied with the modulation correlation matrix to obtain the context-dependent text correlation matrix. :

[0055]

[0056] The text-visual feature correlation matrix is ​​obtained using the ACSM component in the internal attention mechanism. This allows for further adjustment of the weight distribution between visual and textual features, increasing the likelihood that key information will be retained.

[0057] Specific Implementation Scheme 2: The present invention provides a contextual semantic modulation system for video subtitle generation. The system has a program module corresponding to the above steps, and executes the steps in the above-mentioned contextual semantic modulation method for video subtitle generation when running.

[0058] The other combinations and connections in this implementation scheme are the same as in Specific Implementation Scheme 1.

[0059] Specific Implementation Scheme 3: The present invention provides a computer-readable storage medium storing a computer program configured to implement, when called by a processor, the steps of a context semantic modulation method for video subtitle generation.

[0060] The other combinations and connections in this implementation scheme are the same as in Specific Implementation Scheme 1.

[0061] experiment

[0062] Dataset settings

[0063] For video description generation tasks, the mainstream datasets currently available are MSVD, MSRVTT, and VATEX. Considering experimental costs and the comprehensiveness and fairness of evaluation methods, this experiment selected MSVD and MSRVTT as datasets and used four commonly used evaluation metrics for scoring: BLEU-4 (B4) bilingual evaluation alternative, ROUGE-L (R) recall-oriented summarization evaluation aid, METEOR (M) explicit ranking translation evaluation metric, and CIDEr (C) consensus-based image description evaluation metric.

[0064] The MSVD dataset is one of the earliest and most popular benchmark datasets for video captioning, consisting of 1970 video clips from various aspects of life. Each video clip is accompanied by several sentence annotations of varying lengths. Consistent with existing work, this experiment uses the first 1200 video clips as training data, 100 video clips as validation data, and the remaining 670 video clips as test data. The MSRVTT dataset is another popular benchmark dataset for video captioning, consisting of 10000 video clips, each with 20 sentence annotations of varying lengths. Compared to MSVD, the sentence annotations in MSRVTT are richer and more diverse. Therefore, this experiment chooses MSRVTT as the main dataset to examine the ability of the model of this invention to generate accurate and diverse descriptive sentences. Consistent with the NACF model, this experiment uses the first 6513 video clips as training data, 497 video clips as validation data, and the remaining 2990 video clips as test data.

[0065] The attention heads were set to 32, with a dimension of 256. The feature size was determined by the input tensor size. This experiment used a 16GB Nvidia RTX 2080 Ti GPU on an Ubuntu 20.04 system, employing Python version 3.7 and Torch version 1.6.0 for model training and testing. During model training, the batch size was set to 64, and the training epoch was set to 50.

[0066] As shown in Tables 1 and 2 below, the baseline model (Method 1) achieved a CIDEr score of 86.3 for MSVD and 50.3 for MSRVTT without introducing any components. When ACSM (Method 2) was enabled individually, the dataset metrics showed a steady improvement. For example, the introduction of ACSM on MSVD increased the CIDEr from 86.3 to 90.5 (+4.2); ACSM also improved the CIDEr on MSRVTT. This indicates that the attention map correction achieved by ACSM through semantic modulation contributes more significantly to the generation quality, especially in the semantic coherence (CIDEr) and diversity (METEOR) metrics.

[0067] Table 1 Ablation Experiment Results on MSVD Dataset

[0068]

[0069] Table 2 Ablation Experiment Results on MSRVTT Dataset

[0070]

[0071] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A contextual semantic modulation method for video subtitle generation, characterized in that, Includes the following steps: S100. Calculate the correlation between the observed partial sequence and the reference statement representation through similarity score matrix and scaling processing; S200. Apply a bias to the attention weights using the semantic context of the current decoder to obtain the modulated correlation matrix. S300: The correlation matrix modulated in step S200 is used to capture the correlation between features through a fully connected layer and a normalization layer; S400 uses the decoder's positional encoding and then passes through a fully connected layer to obtain the final observed partial sequence and reference statement representation. After normalization calculation by Softmax and multiplication by the modulation correlation matrix, the final context-dependent text correlation matrix is ​​obtained.

2. The contextual semantic modulation method for video subtitle generation according to claim 1, characterized in that: In step S100, the correlation matrix obtained between the observed partial sequence and the reference statement is as follows: in, This indicates the relevance between the observed partial sequence and the reference statement. ; and This represents the mapped text features, i.e., the observed partial sequence and the reference sentence representation, where l is the prediction length and dm is the batch size.

3. The contextual semantic modulation method for video subtitle generation according to claim 2, characterized in that: Step S200 includes, The decoder state at the current time t Convert to a modulation coefficient : in, It is a sigmoid activation function, and the output range is controlled within (0,1) to ensure the stability of the modulation coefficient; It is a state Learnable linear projection weights For state The bias term; Modulated correlation matrix Represented as:

4. The contextual semantic modulation method for video subtitle generation according to claim 3, characterized in that: Step S300 includes, To achieve consistent relevance between generated tokens and tokens in the reference sentence, thereby enhancing the appropriate relevance of relevant token pairs and suppressing erroneous relevance of irrelevant token pairs, the resulting modulated relevance matrix... The correlation between features is captured through fully connected layers and normalization layers: in, This represents the computation of the fully connected layer. , represents the correlation matrix output after the fully connected layer and layer normalization, d represents the output feature size after the fully connected layer, and LayerNorm represents the layer normalization process.

5. A contextual semantic modulation method for video subtitle generation according to claim 4, characterized in that: Step S400 includes, Use the decoder's positional encoding To help understand the relative order of tokens, a fully connected layer is used again to enhance the learning of correlations, resulting in the observed partial sequence and the related items represented by the reference statement. and reference statement as follows: The two relevant terms are normalized using Softmax and then multiplied with the modulation correlation matrix to obtain the context-dependent text correlation matrix. :

6. A contextual semantic modulation system for video subtitle generation, characterized in that: The system has a program module corresponding to the steps described in any one of claims 1-5, and executes the steps in the above-described context semantic modulation method for video subtitle generation when running.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to, when invoked by a processor, implement the steps of any one of claims 1-5, a context semantic modulation method for video subtitle generation.

Images