Cross-modal structure alignment method and system for video subtitle generation

Abstract

The invention provides a cross-modal structure alignment method and system for video subtitle generation, and belongs to the technical field of video subtitle generation. The method aims at solving the problems that in an existing attention mechanism, structural compatibility in a multi-modal or language generation scene is not considered, and noise is generated in the modal fusion process. Structural compatibility in a multi-modal or language generation scene is considered, feature loss after text features and visual features are fused can be reduced, noise generated by modal fusion is reduced, natural mismatch of cross-modal fusion in a semantic mapping space is relieved, and the method is suitable for being popularized and applied. Therefore, the modeling capability of the model on the high-order semantic relationship is improved, and the negative influence generated after multi-modal fusion is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video subtitle generation, in particular to a cross-modal structure alignment method and system for video subtitle generation. BACKGROUND

[0002] Attention mechanism, as one of the key modules in deep learning, is widely used in visual understanding and natural language generation. Since the introduction of Transformer, the standard self-attention mechanism has established the relevance within the sequence through the query-key dot-product, and has become the core structure in the task of video description. Subsequently, researchers have proposed various variants to improve its expressive power, such as multi-head attention, sparse attention, bilinear cross-attention, and multi-scale attention. These methods have enhanced the feature selection ability of the model to some extent, but most of them still remain at the level of first-order dependency modeling.

[0003] To break through the limitations of first-order attention expression, Wang et al. proposed the Attention-in-Attention (AiA) module, which has shown good performance in image classification, target tracking and other tasks, indicating its potential to enhance the expressive power of attention maps. However, the original AiA module design is mainly aimed at single-modal image features, and does not consider the structural compatibility in multi-modal or language generation scenarios. When using multi-modal input to realize video subtitle generation, modal noise is inevitably generated in the process, leading to object recognition errors or inaccurate labels, as shown in FIG. 1. Figure 1 As shown in FIG. 1, both the video screenshot and the reference caption indicate "a eye makeup", but the output caption can only output "something". SUMMARY

[0004] The technical problem to be solved by the present application is:

[0005] To solve the problem that the existing attention mechanism does not consider the structural compatibility in multi-modal or language generation scenarios, and that noise is generated in the modal fusion process, causing object recognition errors or inaccurate labels.

[0006] The technical solution adopted by the present application to solve the above technical problem is:

[0007] The present application provides a cross-modal structure alignment method for video subtitle generation, comprising the following steps:

[0008] S100, based on the context-aware semantic result, introducing a guide vector extracted from the visual feature level to guide the structure adaptation of the visual feature and the context-aware semantic result, and obtaining a text-related enhanced matrix;

[0009] S200, generating a modal-independent guidance vector through a nonlinear activation function;

[0010] S300, performing affine transformation on the text-related enhanced matrix generated in step S100 by using the guidance vector generated in step S200, projecting the visual feature tensor into a structure alignment space through linear transformation, realizing collaborative representation of visual-text features through matrix multiplication, and finally obtaining bidirectional self-attention weights.

[0011] Further, in step S100, for the text-visual feature correlation matrix , in order to reduce feature loss after the text feature is fused with the visual feature, the correlation matrix is subjected to feature enhancement to obtain a text-related enhanced matrix :

[0012]

[0013] wherein, is the text-visual feature correlation matrix obtained in the internal attention, and dm is the batch size; represents the calculation of the full connection layer.

[0014] Further, in step S200, when cross-modal alignment is performed, the visual feature tensor output by the encoder is projected into a structure alignment space through linear transformation to obtain , and the text-related enhanced matrix is subjected to structure optimization to realize collaborative representation between modalities, wherein K represents the sequence length of the video frame.

[0015] The modal-independent guidance vector generated through the nonlinear activation function is:

[0016]

[0017] wherein, is the visual feature tensor , wv is the learnable linear projection weight of the visual feature tensor , and bv is the bias term of the visual feature tensor . is the sigmoid activation function.

[0018] Further, in step S300, the guidance vector obtained in step S200 is used to adjust the text-related enhanced matrix through affine transformation to obtain , and the visual feature tensor of the encoder is projected into a structure alignment space through linear transformation to obtain , and then collaborative representation of visual-text features is realized through matrix multiplication :

[0019]

[0020] Finally, the collaborative representation of the obtained visual-textual features Dimension recovery is performed through linear transformation to obtain the final bidirectional self-attention weight :

[0021]

[0022] wherein, is the observed part sequence; is the source sentence representation.

[0023] A cross-modal structure alignment system for video caption generation has program modules corresponding to the above steps, and runs to perform the steps of the above cross-modal structure alignment method for video caption generation.

[0024] A computer-readable storage medium stores a computer program, and the computer program is configured to realize the steps of a cross-modal structure alignment method for video caption generation when called by a processor.

[0025] Compared with the prior art, the beneficial effects of the present application are:

[0026] The cross-modal structure alignment method and system for video caption generation of the present application considers the structural compatibility in the multi-modal or language generation scene, can reduce the feature loss after the fusion of text features and visual features, reduce the noise generated by modal fusion, alleviate the natural mismatch of cross-modal fusion in the semantic mapping space, and further improve the modeling ability of the model for high-order semantic relationships and reduce the negative impact after multi-modal fusion. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is a fuzzy generated video caption example;

[0028] Figure 2 is a framework diagram of the CSACE module in the bidirectional self-attention mechanism in the embodiment of the present application, wherein (a) is a framework diagram of the self-attention mechanism, and (b) is a framework diagram of the internal attention mechanism.

[0029] Figure 3 is a framework diagram of the ACSM model in the embodiment of the present application;

[0030] Figure 4 is a flowchart of the CSACE module in the embodiment of the present application. DETAILED DESCRIPTION

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] Specific Implementation Plan 1: Combining Figures 2 to 4 As shown, this invention provides a cross-modal structure alignment method for video subtitle generation, comprising the following steps:

[0033] S100. In order to alleviate the natural mismatch in the semantic mapping space of cross-modal fusion, the cross-modal structure alignment (CSA) model is based on context-aware semantic results. By introducing a guiding vector extracted from the visual feature level, it guides the visual features and context-aware semantic results to perform structural adaptation.

[0034] After obtaining the text-visual feature correlation matrix Subsequently, to reduce feature loss after fusing text features with visual features, it is necessary to further enhance the correlation matrix while maintaining its size to obtain the text correlation enhancement matrix. :

[0035]

[0036] in, The text-visual feature correlation matrix obtained in internal attention. dm is the batch size. This represents the computation of the fully connected layer;

[0037] S200. During cross-modal alignment, the visual feature tensor output by the encoder. Text-related enhancement matrix Intermodal cooperative representation needs to be achieved through structural optimization, where K represents the sequence length of video frames; to this end, the CSA model generates modality-independent guiding vectors through a nonlinear activation function. :

[0038]

[0039] in, It is a visual feature tensor Learnable linear projection weights For visual feature tensors The bias term; It is a sigmoid activation function, and the output range is controlled within (0,1) to ensure the stability of the modulation coefficient;

[0040] S300. Using this guiding vector, adjust the text relevance enhancement matrix through affine transformation. get and the encoder's visual feature tensor By linear transformation projection to structure alignment space , the CSA model realizes the collaborative representation of visual-text features through matrix multiplication :

[0041]

[0042] Finally, the obtained CSA component result is dimensionally restored by linear transformation to obtain the final bidirectional self-attention weight added to the CSA CE model :

[0043]

[0044] wherein, is the observed part of the sequence; is the source sentence representation.

[0045] Preferably, in order to obtain the text-related enhancement matrix , the following method can be used:

[0046] The traditional Transformer attention module directly takes the query vector (query) and a set of key-value pairs (key-value pairs) as input, and outputs the weighted sum of the generated value by calculating the attention weight; in contrast, in the self-attention module of the bidirectional self-attention mechanism, the ACSM model further learns the dynamic attention weight between tokens by integrating the context information of each output token, thereby realizing fine-grained semantic alignment between the generated token and the generated token; This method not only can more effectively model semantic association, but also can adaptively adjust the semantic weight of key information, thereby enhancing the representation of important features and avoiding the neglect of key information;

[0047] Before the ACSM model, the mask language encoder maps the observed part of the sequence and the reference sentence representation to the attention module of the decoder, and respectively uses and to represent the mapped text features (observed part of the sequence and reference sentence representation), wherein l is the prediction length, and dm is the batch size (batch size);

[0048] The ACSM model measures the correlation between the observed part of the sequence and the reference sentence representation through the similarity score matrix and scaling processing:

[0049]

[0050] wherein, denotes the relevance between the observed partial sequence and the reference sentence representation, ;

[0051] After obtaining the relevance, in order to dynamically adjust the emphasis of similarity, the attention weight is biased using the semantic context of the current decoder; first, the state of the current time t decoder is converted into a modulation coefficient :

[0052]

[0053] wherein, is a sigmoid activation function, the output range is controlled in (0, 1), and the modulation coefficient is stable; is a state linear projection weight that can be learned, is a bias term for the state ;

[0054] The final relevance matrix after modulation is represented as:

[0055]

[0056] Next, in order to seek the consistency of the relevance between the generated token and the token in the reference sentence, the appropriate relevance of the relevant token pair is enhanced and the error relevance of the irrelevant token pair is inhibited, so that the obtained modulation relevance matrix captures the association between features through a fully connected layer and a normalization layer:

[0057]

[0058] wherein, denotes the calculation of the fully connected layer, denotes the relevance matrix output after the fully connected layer and the layer normalization, d denotes the output feature size after the fully connected layer, and LayerNorm denotes the layer normalization processing;

[0059] In order to ensure the structural correctness of the generated description sentence, the position encoding of the decoder is followed to help understand the relative order of the token, and the relevance between the observed partial sequence and the reference sentence representation is obtained again through a fully connected layer to strengthen the learning of the relevance between the observed partial sequence and the reference sentence representation. and the reference sentence representation is as follows:

[0060]

[0061] The two obtained correlation terms are normalized by Softmax and multiplied by the modulation correlation matrix to obtain a context-related text correlation matrix :

[0062]

[0063] The ACSM module is used in the internal attention (including the attention-enhanced context semantic modulation ACSM model and the cross-modal structure alignment CSA model) to obtain a text-visual feature correlation matrix The weight distribution between the visual features and the text features can be further adjusted, so that the possibility of retaining key information is increased.

[0064] Specific implementation scheme two: the application discloses a cross-modal structure alignment system for video subtitle generation, which has program modules corresponding to the above steps, and executes the steps of the above cross-modal structure alignment method for video subtitle generation.

[0065] Other combinations and connection relationships of the embodiment are the same as those of the specific implementation scheme one.

[0066] Specific implementation scheme two: the application discloses a computer readable storage medium, which stores a computer program configured to realize the steps of a cross-modal structure alignment method for video subtitle generation when called by a processor.

[0067] Other combinations and connection relationships of the embodiment are the same as those of the specific implementation scheme one.

[0068] Experiment

[0069] Dataset setting

[0070] For the video description generation task, the current mainstream datasets are MSVD, MSRVTT and VATEX. Considering the experimental cost and the comprehensiveness and fairness of the evaluation method, MSVD and MSRVTT are selected as the datasets in this experiment, and BLEU-4 (B4) bilingual evaluation is used as the evaluation tool, ROUGE-L (R) is used as the recall-oriented summary evaluation tool, METEOR (M) is used as the explicit ranking translation evaluation index, and CIDEr (C) is used as the consensus-based image description evaluation four commonly used evaluation indexes are used for scoring.

[0071] The MSVD dataset is one of the earliest popular video captioning benchmark datasets, which consists of 1970 video clips from different fields of life. Each video clip is annotated with several sentences of different lengths. In line with existing work, the first 1200 video clips are selected as training data, 100 video clips are selected as validation data, and the remaining 670 video clips are selected as test data. The MSRVTT dataset is another popular video captioning benchmark dataset, which consists of 10000 video clips, each of which is annotated with 20 sentences of different lengths. Compared with MSVD, the sentence annotations in MSRVTT are more abundant and diverse. Therefore, the MSRVTT is selected as the main dataset in this experiment to test the ability of the model to generate accurate and diverse descriptive sentences. In line with the NACF model, the first 6513 video clips are selected as training data, 497 video clips are selected as validation data, and the remaining 2990 video clips are selected as test data.

[0072] Experimental setup

[0073] The number of attention heads is set to 32, and the dimension is set to 256. The feature size is determined by the input tensor size. In this experiment, a 16GB GPU Nvidia RTX 2080 Ti is used to train and test the model under the Ubuntu 20.04 system, using Python version 3.7 and torch version 1.6.0. During model training, the training batch size is set to 64, and the training period is set to 50.

[0074] Ablation experiments

[0075] To illustrate the effectiveness of the designed CSACE module, including the attention-enhanced context semantic modulation component ACSM and the cross-modal structure alignment component CSA, we conducted an ablation experiment on its components. As shown in Tables 1 and 2, as we increase the designed CSACE module, the performance of the model improves compared to the baseline model. When the baseline model (Method 1) does not introduce any components, the CIDEr score of MSVD is 86.3, and the CIDEr score of MSRVTT is 50.3. When ACSM (Method 2) and CSA (Method 3) are enabled separately, the indicators of both datasets show stable improvement. For example, the introduction of ACSM on MSVD improves the CIDEr from 86.3 to 90.5 (+4.2), and CSA brings a gain of 0.9; the improvement of ACSM on CIDEr (51.4 vs. 50.3) on MSRVTT is also better than CSA (50.9 vs. 50.3). This indicates that the attention map correction achieved by semantic modulation is more significant to the generation quality, especially in terms of semantic coherence (CIDEr) and diversity (METEOR).

[0076] The CSACE module achieves optimal performance in both experiments. Taking MSVD as an example, its CIDEr (91.7) surpasses the results of using ACSM or CSA components alone, and is also best in other indicators (such as BLEU-4, ROUGE-L). This phenomenon verifies the complementarity of ACSM and CSA: ACSM enhances the relevance of generated content through hidden state driven semantic modulation, while CSA reduces fusion noise through cross-modal alignment, both of which optimize the accuracy of the attention mechanism. It is worth noting that the addition of CSA on MSRVTT has a slight improvement in CIDEr (51.5 vs. 51.4), but still has obvious gains in METEOR (29.5 vs. 29.2) and ROUGE-L (62.7 vs. 62.4), indicating that the robustness of CSA to low-frequency semantics (such as abstract concepts) may be more advantageous.

[0077] Table 1. Ablation experiment results of MSVD dataset

[0078]

[0079] Table 2. Ablation experiment results of MSRVTT dataset

[0080]

[0081] Experiments further reveal the adaptability of components under different data scales. Although the absolute increase in indicators of MSRVTT (large scale) is smaller than that of MSVD, the relative trend is consistent, indicating that the improvement of CSACE is universal. For example, ACSM brings a 2.2% improvement in CIDEr on MSRVTT (from 50.3 to 51.4), while MSVD is 4.9% (from 86.3 to 90.5), the difference may be due to the suppression ability of large-scale data itself to noise, but the components can still further tap the performance potential.

[0082] Although the present disclosure is as above, the protection scope of the present disclosure is not limited to this. The person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present disclosure, and these changes and modifications will fall within the protection scope of the present disclosure.

Claims

1. A cross-modal structure alignment method for video subtitle generation, characterized in that, Includes the following steps: S100. Based on the context-aware semantic results, a guiding vector extracted from the visual feature level is introduced to guide the visual features and the context-aware semantic results to perform structural adaptation, thereby obtaining a text-related enhancement matrix. S200: Generate mode-independent guiding vectors through a nonlinear activation function; S300: Using the guiding vector generated in step S200, the text relevance enhancement matrix generated in step S100 is subjected to an affine transformation. The visual feature tensor is projected onto the structure alignment space through a linear transformation. The visual-text features are then represented collaboratively through matrix multiplication, and finally, bidirectional self-attention weights are obtained.

2. The cross-modal structure alignment method for video subtitle generation according to claim 1, characterized in that: In step S100, for the text-visual feature correlation matrix To reduce feature loss after fusing text features with visual features, feature enhancement is performed on the correlation matrix to obtain a text correlation enhancement matrix. : in, The text-visual feature correlation matrix obtained in internal attention. dm is the batch size; This represents the computation of the fully connected layer.

3. The cross-modal structure alignment method for video subtitle generation according to claim 2, characterized in that: In step S200, during cross-modal alignment, the visual feature tensor output by the encoder... Text-related enhancement matrix Intermodal collaborative representation is achieved through structural optimization, where K represents the sequence length of video frames; Generating mode-independent guiding vectors using a nonlinear activation function for: in, It is a visual feature tensor Learnable linear projection weights For visual feature tensors The bias term; It is the sigmoid activation function.

4. The cross-modal structure alignment method for video subtitle generation according to claim 3, characterized in that: In step S300, the text relevance enhancement matrix is ​​adjusted by affine transformation using the guiding vector obtained in step S200. get and the encoder's visual feature tensor The result is obtained by projecting a linear transformation onto the structural alignment space. Then, the co-representation of visual and textual features is achieved through matrix multiplication. : Finally, the obtained co-representation of visual-text features will be... Dimensionality recovery is performed through linear transformation to obtain the final bidirectional self-attention weights. : in, This refers to the observed portion of the sequence; This represents the source statement.

5. A cross-modal structure alignment system for video subtitle generation, characterized in that: The system has a program module corresponding to the steps of any one of claims 1-4 above, and executes the steps in the above-described method for cross-modal structure alignment for video subtitle generation when running.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to, when invoked by a processor, implement the steps of the cross-modal structure alignment method for video subtitle generation according to any one of claims 1-4.

Images