Video representation fusion method in service supervision scene

Abstract

The invention discloses a video representation fusion method in a service supervision scene, and the method comprises the following steps: 1, selecting training data, selecting a Panda70M data set for a short video, and selecting the capture data of Charades, Charades Ego, Activity Net and Valley for a long video; step 2, firstly obtaining a feature cube, then creating a frame representation fusion module, and through the frame representation fusion module, using a static attention mask to make each Query only need to pay attention to a fixed local token in the feature cube, or using a dynamic attention mask to make the position of a sub-cube concerned by each Query change as required; and step 3, on the basis of Q-Former, creating three loss functions based on the training data selected in the step 1, and carrying out joint training to complete video representation fusion. According to the video representation fusion method in the service supervision scene, video representation fusion can be simply and effectively achieved through setting of the first step, the second step and the third step.

Description

Technical Field

[0001] The present invention relates to a video processing method, and more specifically to a video representation fusion method in a service supervision scenario. Background Art

[0002] In the service supervision scenario, video is a common data form, and service supervision requires the model to be able to understand the video content and identify illegal content from the video. This requires a good video representation encoding method to convert the semantic content contained in the video as completely and efficiently as possible into a matrix representation form that the model can understand.

[0003] At present, most of the current video representation coding methods are based on image encoders (such as CLIP). For example, in Video-ChatGPT, video frames are first uniformly extracted from the video, and then the extracted video frames are encoded separately using the image encoder to obtain the representation of each video frame. However, since there are many extracted video frames, if the representations of all frames are directly spliced, the representation will be very long, which poses a huge challenge for the model to understand the video content. One of the simplest methods is to use a simple pooling layer (such as a maximum pooling layer or an average pooling layer) to compress the representation length of multiple video frames to an acceptable level. Video-ChatGPT uses this method. In addition, InternVideo2 directly uses an existing video encoder. Video-LLaMA uses the Q-Former proposed by BLIP to fuse video frame representations. Video-LLaMA2 further improves the video frame representation fusion method. Video-LLaMA2 no longer uses Q-Fomrer to fuse multiple video frame representations into a video representation, but instead uses a series of trainable CNN (convolutional neural network) modules to fuse multiple video frame representations. Similarly, the CNN module here is also trained end-to-end together with the language model and cannot be used as a video encoder alone.

[0004] However, the above methods have the following problems: * Simple and direct splicing. The method is simple, but the obtained video representation is very long and not efficient.

[0005] * Video frame representation fusion method based on pooling strategy. The advantage is that it does not require training, but the effect depends on the choice of pooling strategy, and the upper limit of the effect of the method based on pooling strategy is low.

[0006] * Representation fusion method based on trainable intermediate layers. For example, Video-LLaMA and Video-LLaMA2 adopt this method. Unfortunately, Video-LLaMA and Video-LLaMA2 are trained end-to-end together with the language model. They cannot be separated and trained and used as independent video encoders, and their usage scenarios are relatively limited.

[0007] The second is the video encoder-based method. For example, UMT and InternVideo2 use video encoders to directly encode videos into representations without any additional operations. However, training such a video encoder requires a lot of resources, including a large amount of video text data, a large number of graphics card servers, and a lot of time. For example, InternVideo2 used 1 million pieces of data and took 1 month to train on 256 A100 GPUs. Summary of the invention

[0008] In view of the shortcomings of the prior art, the present invention aims to provide an efficient video frame representation fusion method based on a picture encoder, which has the following characteristics compared with the existing methods: * The training process is simple and efficient, requiring less video text data and can be completed on a small number of GPUs.

[0009] * Trained as a separate module rather than end-to-end with the language model, and can be used as a standalone video encoder.

[0010] To achieve the above object, the present invention provides the following technical solution: a video representation fusion method in a service supervision scenario, characterized in that it includes the following steps: Step 1: Select training data. For shorter videos, select the Panda70M dataset. For longer videos, select Charades, CharadesEgo, ActivityNet, and Valley caption data. Step 2: First, obtain the feature cube, and then create a frame representation fusion module. Through the frame representation fusion module, use a static attention mask so that each query only needs to focus on a fixed local token in the feature cube, or use a dynamic attention mask so that the position of the sub-cube that each query focuses on changes as needed; Step three: Based on Q-Former, three loss functions are created based on the training data selected in step one for joint training to complete the training of the video representation fusion module. Then, the trained video representation fusion module is used to represent and fuse the videos in the service supervision scenario.

[0011] As a further improvement of the present invention, the specific method of obtaining the feature cube in step 2 is: first preprocess the video, evenly extract frames of the video data and resize them to a uniform scale, and then send the video frames to the picture encoder for encoding to obtain the representation of each video frame.

[0012] As a further improvement of the present invention, the representation fusion module constructed in the step 2 is a multi-layer transformer model, which includes self-attention and cross-attention mechanisms. During the fusion process, Q learnable queries are used as input, and feature cubes are used as keys and values. Through the cross-attention layer, the frame representations are gradually fused into the Q learnable queries, and the self-attention between the queries is used to complete the mutual fusion of the frame representations.

[0013] As a further improvement of the present invention, the static attention mask used in the step 2 is a mask cube with a dimension of (S*T*T), specifically, the feature cube with a shape of (M*N*N) is divided into many sub-cubes, each sub-cube has a size of (S*T*T), a total of (M / S * N / T * N / T) = Q sub-cubes, and each learnable Query focuses on one sub-cube.

[0014] As a further improvement of the present invention, the dynamic attention mask used in the step 2 is as follows: the Query of the transformer cross attention layer is output to a linear layer, and the hidden state dimension of the Query is mapped from H to 3, which respectively represent the offset of the sub-cube that the current Query focuses on in the three dimensions of the feature cube. Then, when the next layer of transformer performs cross attention, the focused sub-cube will move according to this offset, thereby realizing a dynamic attention mask and allowing the model to select the position of the sub-cube as needed.

[0015] As a further improvement of the present invention, the specific method of performing joint training in step three is as follows: first, Q-Former is converted into the video data domain, corresponding to VTC, VTM and VTG respectively, and then VTC, VTM and VTG are calculated in turn.

[0016] As a further improvement of the present invention, the method of calculating VTC in step three is: first, the text description of the video is passed through the word segmentation and embedding layers, and then input into the representation fusion module, but no cross-attention calculation is performed. The token corresponding to the text CLS header is taken as the text representation, and then the cosine similarity is calculated using the text representation and the Q tokens of the video representation respectively. Finally, the largest one is taken as the final similarity, and then the cross-entropy loss is calculated.

[0017] As a further improvement of the present invention, the calculation method of VTM in step three is: first, Q learnable queries and text tokens are sent to the representation fusion module together, and self-attention and cross-attention calculations are performed at the same time. Then, the Q tokens of the output corresponding to the learnable query are taken as the result, the average value is calculated, and then a linear layer is passed to perform a binary classification operation and calculate the cross entropy loss.

[0018] As a further improvement of the present invention, the calculation method of VTG in step 3 is as follows: first, Q learnable queries and text tokens are sent to the representation fusion module together, and self-attention and cross-attention calculations are performed at the same time, so that the model performs the next token prediction task and calculates the cross entropy loss; Among them, causal attention is used when performing self-attention and cross-attention calculations.

[0019] The beneficial effect of the present invention is that through the setting of step one, appropriate training data can be effectively selected according to the length of the video, which is more targeted. Then, through the setting of step two, the video data can be effectively preprocessed, and frame representation fusion is performed after preprocessing to achieve efficient feature extraction and reduce feature redundancy. Then, the model's understanding and reasoning ability of the video is improved through static and dynamic attention masks. Compared with the methods in the prior art, the training process is simple and efficient, requires less video text data, and can be completed on a small number of GPUs. It can be trained as a separate module instead of end-to-end training with the language model, and can be used as a separate video encoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a model architecture used in the video representation fusion method in a service supervision scenario of the present invention; Figure 2 Schematic diagram designed for attention mask; Figure 3 Schematic diagram of the MoE feedforward neural network. DETAILED DESCRIPTION

[0021] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0022] Reference Figures 1 to 3 As shown, a video representation fusion method in a service supervision scenario of this embodiment includes the following three steps: selection of training data, frame representation fusion module architecture, and construction of training loss function.

[0023] Selection of training data Since video data contains a lot of redundant information, a common video processing method is to evenly extract a fixed number of video frames. If the video is too long, the number of extracted video frames is too small to reflect all the information contained in the video. If the video is too short, the extracted video frames are too homogeneous, which is not conducive to the understanding of long videos. Therefore, when selecting training data, various video lengths should be balanced.

[0024] For shorter videos, we chose the Panda70M dataset. The feature of this dataset is that the average video length is 3-4 seconds, but the advantage is that the number is relatively large and the data is relatively diverse, which is conducive to the generalization of the model. For longer videos, we chose Charades, CharadesEgo, ActivityNet and Valley caption data. The average video length of these datasets is about 10 seconds, but the number is relatively small. In addition, considering that there are many changes in the front and back scenes in the video data, and the scenes where events occur continuously, we additionally introduced the KineticsGEB dataset. The feature of this dataset is that there are continuous scenes with front and back associations, which is conducive to the model to better learn the dynamic information in the video.

[0025] In addition, due to the difficulty in obtaining video data, we also selected image datasets such as CC3M and VisualGenome to add to the training data. By leveraging the diversity of image data, we can achieve modal complementarity and improve the generalization ability of the model.

[0026] Frame representation fusion module architecture Get feature cube First, we need to preprocess the video, evenly extract frames from the video data and resize them to a uniform scale, and then send the video frames to the image encoder for encoding to obtain the representation of each video frame. Assuming that the representation shape of each video frame is (N*N), the hidden state dimension is H, and the number of randomly extracted video frames is M, then the representation we finally obtain is combined into M*N*N*H. The dimension of the hidden state is not considered here, so the dimension of the representation is M*N*N. From the shape point of view, it is a cube, which we call a feature cube.

[0027] Frame representation fusion module However, the dimension of each frame of the feature cube (that is, the dimension where M is located) is independent of each other, and the length of M*N*N is too long to be used directly. Therefore, a representation fusion module is needed to fuse the representation of the frame into the representation of the video. The frame representation fusion module refers to Q-Former and uses BERT as the basic framework. BERT is an encoder-only transformer model. In order to enable the model to focus on the video frame representation through cross-attention, a cross-attention layer is introduced in the middle of the BERT layer. And set Q learnable queries as input.

[0028] In summary, our frame representation fusion module is a multi-layer transformer model that includes self-attention and cross-attention mechanisms. Q learnable queries are used as input, and feature cubes are used as keys and values. Through the cross-attention layer, the frame representations are gradually fused into the Q learnable queries, and the self-attention between queries is used to complete the mutual fusion of frame representations.

[0029] Because our training data contains video caption data and image caption data, and has three modes: video, image, and text. The frame representation fusion module needs to process data of these three modes at the same time, so referring to the MOE (mixed expert model) architecture, we modified the FFN (feedforward network) of the transformer block. The FFN was modified from one MLP to four MLPs, and each MLP is an expert. The data calculations of the three modes of video, image, and text are handled by their respective experts (video expert, image expert, text expert). At the same time, there is a shared expert that can process data of all modes.

[0030] Static attention mask design At the cross-attention level, additional changes are required. If full-attention cross-attention is used, each learnable query needs to focus on M*N*N tokens, which is very difficult for the model to learn. Therefore, we have designed a set of attention mask mechanisms specifically for this purpose, so that each query only needs to focus on local tokens in the feature cube, reducing the amount of calculation and learning difficulty. Specifically, we set a mask cube with a dimension of (S*T*T). Each query only needs to focus on (S*T*T) tokens, which is smaller than M*N*N, and each token focuses on different (S*T*T) tokens. This is equivalent to dividing the feature cube with a shape of (M*N*N) into many sub-cubes, each sub-cube is of size (S*T*T), a total of (M / S * N / T *N / T) = Q sub-cubes, and each learnable query focuses on one sub-cube. S is actually divided in the M dimension, that is, the dimension of different video frames, and T is divided in the N dimension, that is, within a single video frame. In this way, each Query can focus on the local information of the same position represented by multiple video frames, which is conducive to the model to simultaneously extract the temporal information and spatial information contained in the video.

[0031] In addition, since the representation of the video frame obtained after passing through the image encoder has introduced spatial position information through position embedding, but there is no additional temporal position information between the representations of each video frame to help the model distinguish the temporal relationship between the previous and next frames, so here we introduce an additional learnable time embedding and add it to the representation of the video frame. The vocabulary size of the time embedding is M, corresponding to the M uniformly extracted video frames.

[0032] Dynamic attention mask design The attention mask mechanism mentioned above can effectively shorten the length of tokens that each learnable query needs to pay attention to, while making full use of the spatiotemporal information of the video feature cube. However, the position of the token that each query pays attention to is actually static and unchanged. For example, the first query will pay attention to S*T*T tokens in the upper left rear of the feature cube, then it will always pay attention to these tokens unchanged. If there is a complete entity in the video that is divided into different sub-cubes, this static method is obviously not conducive to the model extracting relevant information about this entity. Therefore, a dynamic attention mask mechanism is proposed here, that is, the position of the sub-cube that each query pays attention to changes on demand.

[0033] Referring to the design of Deformable Attention, the Query of the transformer cross-attention layer is output to a linear layer, and the hidden state dimension of the Query is mapped from H to 3, which respectively represent the offset of the sub-cube that the current Query focuses on in the three dimensions of the feature cube. Then, when the next layer of transformer performs cross-attention, the focused sub-cube will move according to this offset, thereby realizing a dynamic attention mask and allowing the model to select the position of the sub-cube as needed. Of course, since the offset obtained by the previous layer does not exist when the first cross-attention is calculated, the setting corresponding to the dynamic attention mask is preset. The offsets of subsequent sub-cubes are offset based on the position when the previous cross-attention is calculated.

[0034] Construction of training loss function Also based on Q-Former, Q-Former training uses three loss functions (ITC, ITM, ITG) for joint training. We convert them to the video data domain, corresponding to (VTC, VTM, VTG) respectively.

[0035] When calculating VTC, for video data, the input of the representation fusion module is only Q learnable queries, and self-attention and cross-attention calculations are performed at the same time, and the video representation of length Q is output. After the text description of the video passes through the word segmentation and embedding layers, it is also input to the representation fusion module, but no cross-attention calculation is performed, and the token corresponding to the text CLS header is taken as the text representation. The cosine similarity is calculated using the text representation and the Q tokens of the video representation. The largest one is taken as the final similarity, and then the cross entropy loss is calculated.

[0036] When calculating VTM, Q learnable queries and text tokens are sent to the representation fusion module together, and self-attention and cross-attention calculations are performed at the same time. The Q tokens corresponding to the learnable query are taken as the result, the average is calculated, and then passed through a linear layer for binary classification (matching and mismatching options), and the cross entropy loss is calculated.

[0037] When calculating VTG, like VTM, Q learnable queries and text tokens are sent to the representation fusion module together, and self-attention and cross-attention calculations are performed at the same time. The difference is that the self-attention of VTC and VTM is bidirectional attention, while VTG uses causal attention to let the model perform the next token prediction task (next token prediction). The cross entropy loss is also calculated.

[0038] In summary, the above three steps: selection of training data, frame representation fusion module architecture, and training loss function can simply and effectively realize the video representation fusion in the service supervision scenario, and the training process is simple and efficient.

[0039] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A video representation fusion method in a service supervision scenario, characterized by: The steps include: Step 1: Select training data. For shorter videos, select the Panda70M dataset. For longer videos, select Charades, CharadesEgo, ActivityNet, and Valley caption data. Step 2: First, obtain the feature cube, and then create a frame representation fusion module. Through the frame representation fusion module, use a static attention mask so that each query only needs to focus on a fixed local token in the feature cube, or use a dynamic attention mask so that the position of the sub-cube that each query focuses on changes as needed; Step three: Based on Q-Former, three loss functions are created based on the training data selected in step one for joint training to complete the training of the video representation fusion module. Then, the trained video representation fusion module is used to represent and fuse the videos in the service supervision scenario.

2. The video representation fusion method in the service supervision scenario according to claim 1 is characterized by: The specific method of obtaining the feature cube in step 2 is: first preprocess the video, evenly extract frames of the video data and resize them to a uniform scale, and then send the video frames to the image encoder for encoding to obtain the representation of each video frame.

3. The video representation fusion method in the service supervision scenario according to claim 2 is characterized in that: The representation fusion module constructed in the step 2 is a multi-layer transformer model, which includes self-attention and cross-attention mechanisms. During the fusion process, Q learnable queries are used as input, and feature cubes are used as keys and values. Through the cross-attention layer, the frame representations are gradually fused into the Q learnable queries, and the mutual fusion of frame representations is completed through the self-attention between queries.

4. The video representation fusion method in the service supervision scenario according to claim 3 is characterized by: The static attention mask used in step 2 is a mask cube with a dimension of (S*T*T). Specifically, the feature cube with a shape of (M*N*N) is divided into many sub-cubes. The size of each sub-cube is (S*T*T), with a total of (M / S * N / T * N / T) = Q sub-cubes. Each learnable query focuses on one sub-cube.

5. The video representation fusion method in the service supervision scenario according to claim 4 is characterized in that: The dynamic attention mask used in the step 2 is as follows: the Query of the transformer cross-attention layer is output to a linear layer, and the hidden state dimension of the Query is mapped from H to 3, which respectively represent the offset of the sub-cube that the current Query focuses on in the three dimensions of the feature cube. Then, when the next layer of transformer performs cross-attention, the focused sub-cube will move according to this offset, thereby realizing a dynamic attention mask and allowing the model to select the position of the sub-cube as needed.

6. The video representation fusion method in the service supervision scenario according to claim 5 is characterized by: The specific method of performing joint training in step three is as follows: first, convert Q-Former into the video data domain, corresponding to VTC, VTM and VTG respectively, and then calculate VTC, VTM and VTG in turn.

7. The video representation fusion method in the service supervision scenario according to claim 6 is characterized by: The method for calculating VTC in step three is: first, the text description of the video is passed through the word segmentation and embedding layers, and then input into the representation fusion module, but without cross attention calculation, the token corresponding to the text CLS head is taken as the text representation, and then the cosine similarity between the text representation and the Q tokens of the video representation is calculated respectively, and finally the largest one is taken as the final similarity, and then the cross entropy loss is calculated.

8. The video representation fusion method in the service supervision scenario according to claim 7 is characterized by: The calculation method of VTM in step 3 is: first, Q learnable queries and text tokens are sent to the representation fusion module together, and self-attention and cross-attention calculations are performed at the same time. Then, the Q tokens of the output corresponding to the learnable query are taken as the result, the average value is calculated, and then a linear layer is passed to perform a binary classification operation and calculate the cross entropy loss.

9. The video representation fusion method in the service supervision scenario according to claim 8 is characterized in that: The calculation method of VTG in step 3 is as follows: first, Q learnable queries and text tokens are sent to the representation fusion module together, and self-attention and cross-attention calculations are performed at the same time, so that the model performs the next token prediction task and calculates the cross entropy loss; Among them, VTG adopts causal attention, and VTC and VTM adopt bidirectional attention.

Images