Long video text description method based on self-evolving perturbation mechanism
By introducing a self-evolutionary perturbation mechanism in the long video text description method, adaptively adjusting the attention weight distribution, and directly extracting key information from untrimmed long videos, the poor correlation between paragraphs and video content and calculation overhead in the existing methods is solved. The big problem is to achieve efficient and accurate text generation, and is suitable for fields such as intelligent video understanding.
Patent Information
- Application Number
- CN202410672644.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-28
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2044-05-28
AI Technical Summary
The existing long video text description methods have poor correlation with video content when generating paragraphs and high computing overhead, which limits the real-time nature of practical applications such as intelligent monitoring.
Using a method based on the self-evolving perturbation mechanism, video clip features are extracted through 3DCNN, combined with transformer encoder and decoder, intra- and inter-layer perturbation mechanisms are introduced, attention weight distribution is adaptively adjusted, and long-range and short-range dependencies of video are captured, directly Generate descriptions from untrimmed long videos.
It improves the correlation between generated text and video, reduces computational overhead, meets real-time processing requirements, and generates more accurate and coherent text descriptions, which is significantly better than the performance of existing methods on multiple data sets.
Smart Images
Figure CN118397515B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a long video-based text description method. Background Art
[0002] The long video text description task aims to generate a coherent multi-sentence description for an untrimmed video containing multiple events. Since this technology has extremely high application value in the fields of intelligent monitoring, abnormal pedestrian behavior detection, personnel search and rescue, etc., it has attracted widespread attention and research in the industry. However, long videos usually contain multiple events, which requires the model to detect the time in the video and generate an appropriate text description for each event. This brings great difficulties to the text description of long videos.
[0003] The existing two-stage method first detects events in the video and then generates descriptions for each event. However, the accuracy of the event detection model and the high computational overhead of the model restrict its practical application in intelligent surveillance. In recent years, the one-stage method has become popular because it omits the event detection stage and directly generates descriptions from long, untrimmed videos. Although this method reduces the computational cost, the paragraphs generated by the current one-stage method are poorly related to the video content, which introduces new problems for the practical application of this task. Summary of the invention
[0004] In order to overcome the shortcomings of the prior art, the present invention provides a long video text description method based on a self-evolution perturbation mechanism, which realizes the requirements of adaptive extraction of key information and real-time processing.
[0005] The technical solution adopted by the present invention to solve the technical problem comprises the following steps:
[0006] Step 1: A long video is first divided into several non-overlapping video clips. For each video clip, a feature map is extracted after a pre-trained 3DCNN convolutional neural network. ,in Indicates the number of video clips, represents the features extracted from the first video clip, represents the features extracted from the second video clip, represents the features extracted from the Lth video clip;
[0007] Step 2: Use the transformer encoder to encode the feature map of the extracted video clips; the feature map obtained from each video clip is input into the transformer encoder as a token, and the encoded video features are obtained as follows:
[0008]
[0009] in represents a feed-forward neural network, Represents a multi-head attention mechanism;
[0010] Step 3: In the decoder part, a self-evolving perturbation mechanism is introduced, including intra-layer perturbation and inter-layer perturbation. The core idea is to adaptively adjust the attention weight distribution through learnable perturbation terms when calculating the self-attention of each layer of the decoder, so that it pays more attention to the video clips that are important to the current generated words. At the same time, the difference in feature receptive fields at different levels of the decoder is used to promote the fusion of features at different levels through the self-distillation loss function to capture the long-range and short-range dependencies of the video. The synergy of the two self-evolving perturbations enables the decoder to adaptively mine the key information of the video and generate a description text that is more in line with the video content.
[0011] Step 4: The purpose of adding perturbations is to identify key frames in long videos in a self-evolving manner, so the removal of more important video clips should have a greater impact on the final loss. The perturbation generator is updated using a negative MLE loss. The perturbation loss for:
[0012]
[0013] in, represents the loss function of the video paragraph description, represents the parameters of the perturbation generator, To balance the hyperparameters of the regularization term and the perturbation loss, The regularization term is used to stabilize the training process;
[0014] Step 5: Get the perturbation term of the decoder at the tth time step After that, the disturbance term Guide the original attention weight distribution Adjustment to better capture key frames in long videos;
[0015] Step 6: After performing perturbations within the layer in step 4, use the information differences learned by different layers of the transformer decoder to perform self-evolving perturbations between layers. Since the shallow layers of the transformer tend to capture short-range dependencies, while the deep layers can model longer-range dependencies, the model’s ability to model long-range and short-range relationships in long videos is enhanced through information interaction and fusion between layers.
[0016] Step 7: Finally, add the self-distillation loss to the original cross entropy loss to get the total training target for:
[0017]
[0018] in, is the cross entropy loss, is the loss from distillation, To balance the weight coefficients of the two loss terms; by jointly optimizing the two loss functions, the model can adaptively mine key information in long videos while generating accurate and coherent text descriptions, thereby improving the relevance of the generated text to the video.
[0019] In step 3, for the features generated by the transformer encoder, self-evolving perturbations are introduced in the self-attention calculation of each layer of the decoder. Specifically, when calculating the attention weight, a learnable perturbator is first generated:
[0020]
[0021] in represents the sigmoid function, represents dot product, Q and K represent the query vector query and key vector key in the attention calculation respectively. The query vector query represents the feature vector of the word currently being processed, and the key vector key represents the feature vector of each video clip output by the encoder. represents the dimensions of the query vector and key vector; the perturbation term With the original attention weight The shape is the same, and its element value is mapped to between 0 and 1 through the sigmoid function. The perturbation term is fused with the original attention weight in the following way:
[0022]
[0023] in, represents the dot product between elements, represents the initial weight of uniform distribution, the perturbation term The larger the value of The closer the fused weights are to the original weights, the closer the fused weights are to the uniform distribution. Through the fusion operation, the perturbation term adaptively adjusts the attention weight distribution of each time step. After training and optimization, the perturbation term can learn to selectively focus on the more important video clips according to the needs of the current word generation, giving them higher weights. The fused attention weights Then participate in the subsequent attention aggregation calculation; obtain the aggregated video features; and use them to guide the generation of the current word.
[0024] In step 5, the present invention designs three fusion methods to fuse the disturbance term and the original attention weight, which are suitable for different scenarios; the three fusion methods are selected according to specific task requirements and data characteristics, or the results of different methods are integrated:
[0025] Step 5.1 Linear combination: Linearly combine the perturbation term and the original attention weight to obtain the adjusted attention weight:
[0026]
[0027] in, is a hyperparameter that controls the influence of the disturbance term. is the disturbance term, when The closer it is to 1, the smaller the impact of the disturbance term is, and the closer the adjusted weight is to the original weight; conversely, the greater the impact of the disturbance term is, the greater the difference between the adjusted weight and the original weight is. is the new attention weight after linear combination;
[0028] Step 5.2. Gating selection mechanism: Use the gating unit to control the fusion ratio of the perturbation term and the original weight:
[0029]
[0030] in, is the parameter matrix of the gating unit, Represents vector concatenation; is the gate signal matrix,
[0031] To gate the new attention weights after fusion, the gating unit adaptively determines the fusion ratio of the perturbation term and the original weight according to their values, thereby adjusting the attention weight distribution more flexibly;
[0032] Step 5.3. Progressive Convergence: As training progresses, gradually increase the impact of the disturbance term so that the model can first be trained stably and then gradually adapt to the disturbance:
[0033]
[0034]
[0035] in, is the weight affected by the disturbance term, step represents the current training step number, is a preset threshold value, which is determined according to the specific task and data set, and is set to 1 / 10 to 1 / 5 of the total number of training steps; T is the total number of training steps. As the training progresses, the weight of the disturbance term gradually increases from 0 to 1, so that the model can smoothly adapt to the impact of the disturbance.
[0036] In practical applications, the fusion method is selected according to the following principles, and one of them can be selected:
[0037] 1. When a decoder model with more than 3 layers is required, choose progressive fusion to make model training more stable;
[0038] 2. When the task requires the accuracy to reach the required index, select the gate selection mechanism and adaptively adjust the fusion ratio;
[0039] 3. When the task requires the inference speed of computing efficiency to reach more than 20 characters per second, choose linear combination, which is simple and efficient in calculation.
[0040] Fusion attention weights It is then used for subsequent attention aggregation to obtain adjusted video features to guide the final word generation.
[0041] In step 6, a self-distillation loss is introduced in the last layer of the decoder to realize inter-layer perturbation. Assume that the decoder has N layers, and the output of the Nth layer is regarded as the "teacher". The output of each layer is regarded as a "student", which promotes information fusion and complementarity between different layers by minimizing the difference between students and teachers.
[0042] Self-distillation losses The calculation formula is:
[0043]
[0044] in, represents the output of the decoder layer n at the tth time step, represents the output of the last layer N of the decoder at the tth time step, T is the sequence length, represents KL divergence, which is used to measure the difference between two distributions;
[0045] When training the model, the self-distillation loss is minimized by the gradient descent algorithm. , prompting the shallow output of the decoder to align with the deep output, so that the shallow layer can also learn certain long-range dependency information; at the same time, it also encourages the deep output to contain more local information to make full use of the complementarity of different layers; during back propagation, the self-distillation loss provides additional supervision signals for each layer, helping the model to better integrate information from different layers, thereby realizing self-evolution perturbations between layers.
[0046] The beneficial effect of the present invention is that a novel framework based on self-evolution perturbation is proposed to solve the problem that the text generated by the current one-stage video description framework is not highly correlated with the video. Compared with the existing methods, the present invention has the following outstanding advantages and beneficial effects:
[0047] 1. By introducing a self-evolving perturbation mechanism in the transformer encoder, the key information in the long video is effectively captured. Different from the traditional attention mechanism, the present invention uses a learnable perturbation term to adaptively adjust the attention weight distribution, so that the model can more accurately identify the most important video clips for the current generated words, thereby extracting more relevant and key video information.
[0048] 2. Get rid of the dependence on the event extraction model and greatly reduce the computational overhead. The traditional two-stage framework needs to first detect the key events in the video through a separate event extraction model, and then describe each event, which not only increases the computational complexity, but also limits the real-time performance of the model. The present invention adopts an end-to-end one-stage framework, and directly extracts key information from the original video through a self-evolving perturbation mechanism, without the need for prior event extraction, thereby greatly improving the processing efficiency and meeting the needs of real-time processing.
[0049] 3. By introducing inter-layer self-evolving perturbations in the transformer decoder, the model's ability to model long-range and short-range dependencies in long videos is enhanced. Unlike previous methods that only focus on local information or only focus on global information, this paper uses the characteristics of different layers of the transformer decoder to promote information interaction and fusion between different layers through self-distillation loss. This enables the model to take into account both local details and global semantics, and generate more accurate, coherent, and more relevant text descriptions to the video.
[0050] 4. The overall framework is simple and effective, easy to implement and expand. The self-evolving perturbation mechanism of the present invention can be seamlessly integrated into the existing transformer encoding and decoding framework. Only minor modifications to the attention calculation process and the introduction of additional loss terms are required to achieve adaptive extraction and fusion of key information. This concise design makes the present invention easy to implement and promote, and can be easily combined with other technical modules to further improve performance.
[0051] 5. Experimental results on multiple datasets show that the present invention significantly outperforms existing one-stage and two-stage video description methods, and achieves new optimal performance in terms of the correlation and coherence between the generated text and the video. This proves the effectiveness of the present invention in capturing key video information and modeling long-term and short-term dependencies, and provides a new and efficient solution for long video description tasks.
[0052] In summary, the present invention effectively solves the key problems faced by the one-stage video description framework through the self-evolution perturbation mechanism, while improving the quality of generated text, taking into account both computational efficiency and real-time requirements. These advantages make the present invention have broad application prospects in multiple fields such as intelligent video understanding, video summarization, and video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] Figure 1 It is a framework diagram of the long video text description model based on the adaptive perturbation mechanism of the present invention. DETAILED DESCRIPTION
[0054] The present invention is further described below in conjunction with the accompanying drawings and embodiments.
[0055] The present invention provides a long video text description method based on a self-evolution perturbation mechanism.
[0056] The main modules of the technical solution of the present invention are as follows: the system includes three parts, the first part is the process of extracting video clip features, the second part is to find the initial video key frame candidate based on the intra-layer self-evolution perturbation mechanism, and the third part is to capture the coarse-grained and fine-grained information of the long video based on the inter-layer self-evolution perturbation mechanism, and finally realize the text description of the long video. In the first part, the convolutional neural network of 3DCNN is used to extract features of each video clip, and then the extracted initial video clip features are encoded using the transformer encoder to obtain the initial weight distribution of each video clip. In the second part, the distribution of the weight of the video clip in each layer of the transformer is perturbed, and the importance of each clip to the generation of the current word is judged by observing the change of the output loss, and the initial video key frame candidate is obtained. In the third part, the information between the layers of the transformer decoder is self-distilled, and the characteristics of different layers of the transformer capturing different granularity information are used to realize the integration of long-range and short-range information of the long video, and obtain the key frames in the long video, and finally realize the generation of text descriptions more relevant to the long video without the need for event monitoring.
[0057] The following is a specific embodiment of the present invention.
[0058] The present invention provides a long video text description method based on an adaptive perturbation mechanism, and the specific process is as follows:
[0059] 1. Extraction of video features
[0060] Given a long video in a natural scene, the video is first segmented into several non-overlapping segments, and then input into the 3DCNN convolutional neural network to obtain video features. , L represents the number of video clips.
[0061] 2. Encoding of video features
[0062] After extracting the video segment features, we encode the features through the transformer encoder. We set the transformer's latent space dimension to 512. We input the position-encoded video features into the transformer to obtain the fused features. .
[0063] 3. Training of the Perturbation Generator
[0064] The encoded features In order to capture the importance of video clips between decoder layers, we use a perturbation function to judge the importance of video clips in the transformer. First, we calculate the similarity in the normal attention mechanism, and then use the sigmoid function to get a perturbation coefficient. , combining the perturbation coefficient with the original attention similarity to obtain the similarity after perturbation. Then we update the parameters in the perturbator through negative MLE loss to find the inter-layer video clips that generate better video clips for the current text description, that is, the key video clips.
[0065] 4. Using perturbators to guide generation
[0066] After the previous stage, we obtained the perturbation coefficient of the perturbator, and we use this coefficient to guide the learning of the attention process. First, the coefficient generated by the perturbator is combined with the similarity in the attention mechanism to obtain the similarity after perturbation, and then the perturbed similarity is combined with the original similarity using the three combination methods we designed (linear combination, gated fusion, and progressive fusion). The combined model can better utilize the video clip features that are more relevant to the current generated words.
[0067] 5. Interlayer self-evolution disturbance generation
[0068] In the previous section, we used the perturbator to adjust the weights of the video clips in the layer. Then, we used the different features learned by different layers in the transformer decoder. The deep layers are more inclined to long-distance dependencies, while the shallow layers are more inclined to short-distance dependencies. We self-distilled the outputs (logits) of different layers of the transformer encoder to enable the model to better integrate long-range and short-range information, so that the generated description is more relevant to the video.
[0069] 6. Environment Configuration
[0070] During training, the Adam optimizer is used and the initial learning rate is set to For feature extraction, we use Resnet-200 to extract 2D features and I3D to extract RGB+Flow features. Both are pre-trained on the kinetics dataset. The feature dimension is 4096, the maximum length of the video description is limited to 150 words, the vocabulary size of the text model is 12046, the hidden dimension in the transformer model is 512, and the number of heads is 8. Use pytorch to implement the entire framework.
Claims
1. A long video text description method based on self-evolution perturbation mechanism, characterized by The steps include: Step 1: A long video is first divided into several non-overlapping video clips. For each video clip, a feature map is extracted after the pre-trained 3DCNN convolutional neural network. in Indicates the number of video clips, represents the features extracted from the first video clip, represents the features extracted from the second video clip, Indicates Features extracted from video clips; Step 2: Use the transformer encoder to encode the feature map of the extracted video clips; the feature map obtained from each video clip is input into the transformer encoder as a token, and the encoded video features are obtained as follows: Among them, FFN represents feedforward neural network, and MultiHead represents multi-head attention mechanism; Step 3: A self-evolving perturbation mechanism is introduced in the transformer decoder, including intra-layer perturbations and inter-layer perturbations. When calculating the self-attention of each layer of the decoder, the attention weight distribution is adaptively adjusted through the learnable perturbation term, so that it pays more attention to the video clips that are important to the current generated words. At the same time, the difference in feature receptive fields at different levels of the decoder is used to promote the fusion of features at different levels through the self-distillation loss function, so as to capture the long-range and short-range dependencies of the video. Step 4: The purpose of adding perturbations is to identify key frames in long videos in a self-evolving way. Therefore, after more important video clips are removed, the impact on the final loss is greater. The perturbation generator is updated using a negative MLE loss. The perturbation loss for: in, represents the loss function of the video paragraph description, θ p represents the parameters of the perturbation generator, λ is a hyperparameter that balances the regularization term and the perturbation loss, and λ|1-p t |2The regular term is used to stabilize the training process; Step 5: Get the perturbation term p of the transformer decoder at the tth time step t After that, the disturbance term p t Guide the original attention weight α t Adjustment to better capture key frames in long videos; Step 6: After performing intra-layer perturbations in step 4, perform inter-layer perturbations using the information differences learned by different layers of the transformer decoder. Since the shallow layers of the transformer tend to capture short-range dependencies, while the deep layers can model longer-range dependencies, the model’s ability to model long-range and short-range relationships in long videos is enhanced through information interaction and fusion between layers. In step 6, a self-distillation loss is introduced in the last layer of the decoder to achieve inter-layer perturbation. Assume that the decoder has N layers in total, and the output of the Nth layer is regarded as the "teacher", and the output of the previous 1 to N-1 layers is regarded as the "student". By minimizing the difference between the student and the teacher, the information fusion and complementarity between different layers are promoted; Self-distillation losses The calculation formula is: in, represents the output of the decoder layer n at the tth time step, represents the output of the last layer N of the decoder at the tth time step, T is the sequence length, KL(·||·) represents the KL divergence, which is used to measure the difference between two distributions; When training the model, the self-distillation loss is minimized by the gradient descent algorithm. It encourages the shallow output of the decoder to be aligned with the deep output, so that the shallow layer can also learn certain long-range dependency information; at the same time, it also encourages the deep output to contain more local information to make full use of the complementarity of different layers; during back propagation, the self-distillation loss provides additional supervision signals for each layer, helping the model to better integrate information from different layers, thereby achieving self-evolution perturbations between layers; Step 7: Finally, add the self-distillation loss to the original cross entropy loss to get the total training target L cap for: in, is the cross entropy loss, is the self-distillation loss, and λ is the weight coefficient for balancing the two loss terms. By jointly optimizing the two loss functions, the model can adaptively mine key information in long videos while generating accurate and coherent text descriptions, thereby improving the relevance of the generated text to the video.
2. The long video text description method based on the self-evolution perturbation mechanism according to claim 1 is characterized by: In step 3, for the features generated by the transformer encoder, self-evolving perturbations are introduced in the self-attention calculation of each layer of the transformer decoder. The specific steps are as follows: when calculating the attention weight, a learnable perturbator is first generated: where σ(·) represents the sigmoid function, represents the dot product, Q and K represent the query vector query and key vector key in the attention calculation respectively. The query vector query represents the feature vector of the word currently being processed, and the key vector key represents the feature vector of each video clip output by the encoder. k represents the dimensions of the query vector and key vector; the perturbation term p t With the original attention weight α t The shape is the same, and its element value is mapped to between 0 and 1 through the sigmoid function. The perturbation term is fused with the original attention weight in the following way: Among them, ⊙ represents the dot product between elements, α0 represents the initial weight of uniform distribution, and the perturbation term p t The larger the value of The closer it is to the original attention weight; conversely, the closer the fused attention weight is to a uniform distribution; through the fusion operation, the perturbation term adaptively adjusts the attention weight distribution of each time step; after training and optimization, the perturbation term can learn to selectively focus on the video clips that are more important to it according to the needs of the current word generation, giving them higher weights; the fused attention weight Then participate in the subsequent attention aggregation calculation; obtain the aggregated video features; and use them to guide the generation of the current word.
3. The long video text description method based on the self-evolution perturbation mechanism according to claim 1 is characterized by: In step 5, three fusion methods are designed to fuse the perturbation term and the original attention weight, which are suitable for different scenarios. The three fusion methods are selected according to specific task requirements and data characteristics, or the results of different methods are integrated: Step 5.1 Linear combination: Linearly combine the perturbation term and the original attention weight to obtain the adjusted attention weight: Among them, ζ is a hyperparameter that controls the influence of the disturbance term, and p t is the disturbance term, when p t The closer it is to 1, the smaller the impact of the disturbance term is, and the closer the adjusted weight is to the original weight; conversely, the greater the impact of the disturbance term is, the greater the difference between the adjusted weight and the original weight is. is the new attention weight after linear combination; Step 5.
2. Gating selection mechanism: Use the gating unit to control the fusion ratio of the perturbation term and the original attention weight: g t =Sigmoid(Wg·[p t ;α t ]) Among them, W g is the parameter matrix of the gated unit, [·,·] represents vector concatenation; g t is the gate signal matrix, To gate the new attention weight after fusion, the gating unit adaptively determines the fusion ratio of the perturbation term and the original attention weight according to their values, thereby adjusting the attention weight distribution more flexibly; Step 5.
3. Progressive Convergence: As training progresses, gradually increase the impact of the disturbance term so that the model can first be trained stably and then gradually adapt to the disturbance: Among them, p is the weight of the disturbance term, step represents the current number of training steps, and τ is a preset threshold. The threshold value depends on the specific task and data set and is set to 1 / 10 to 1 / 5 of the total number of training steps. As the training progresses, the weight of the disturbance term gradually increases from 0 to 1, so that the model can smoothly adapt to the impact of the disturbance.
4. The long video text description method based on the self-evolution perturbation mechanism according to claim 3 is characterized in that: in practical application, the fusion method is selected according to the following principles, and only one of them can be selected: 4.
1. When a decoder model with more than 3 layers is required, choose progressive fusion to make model training more stable; 4.
2. When the task requires the accuracy to reach the required index, select the gate selection mechanism and adaptively adjust the fusion ratio; 4.
3. When the task requires the inference speed of computing efficiency to reach more than 20 characters per second, choose linear combination, which is simple and efficient in calculation; New attention weight after fusion It is then used for subsequent attention aggregation to obtain adjusted video features to guide the final word generation.