Video representation fusion method in service supervision scenario

Abstract

The present invention discloses a video representation fusion method in a service supervision scenario, comprising the following steps: Step 1, selecting training data, selecting the Panda70M dataset for shorter videos, and selecting the caption data of Charades, CharadesEgo, ActivityNet, and Valley for longer videos; Step 2, first obtaining a feature cube, and then creating a frame representation fusion module, using a static attention mask through the frame representation fusion module to allow each query to only focus on a fixed local token in the feature cube, or using a dynamic attention mask to allow the position of the sub-cube focused on by each query to change as needed; Step 3, based on Q-Former, creating three loss functions based on the training data selected in Step 1 for joint training to complete video representation fusion. The video representation fusion method in the service supervision scenario of the present invention can simply and effectively achieve the fusion of video representations through the settings of Steps 1 to 3.

Description

Technical Field

[0001] The present invention relates to a video processing method, and more particularly to a video representation fusion method in a service supervision scenario. Background Art

[0002] Video is a common data format in service regulation scenarios. Service regulation requires models to understand video content and identify illegal content. This requires a robust video representation encoding method that efficiently and completely converts the semantic content of a video into a matrix representation that the model can understand.

[0003] Most current video representation encoding methods are based on image encoders (such as CLIP). For example, in Video-ChatGPT, frames are first uniformly extracted from the video. These frames are then encoded individually using an image encoder to generate a representation for each frame. However, since there are many extracted frames, simply concatenating the representations of all frames would result in a very long representation, posing a significant challenge for the model to understand the video content. A simple approach is to use a simple pooling layer (such as max pooling or average pooling) to compress the representations of multiple frames to an acceptable length. Video-ChatGPT employs this approach, while InternVideo2 directly uses an off-the-shelf video encoder. Video-LLaMA utilizes the Q-Former proposed by BLIP to fuse video frame representations. Video-LLaMA2 further improves this method. Instead of using the Q-Former to fuse multiple frame representations into a single video representation, Video-LLaMA2 employs a series of trainable convolutional neural network (CNN) modules to fuse the representations of multiple frames. Similarly, the CNN module here is also trained end-to-end together with the language model and cannot be used as a video encoder alone.

[0004] However, the above methods have the following problems:

[0005] * Simple direct splicing. This method is simple, but the obtained video representation is very long and not efficient.

[0006] * A video frame representation fusion method based on a pooling strategy. This method has the advantage of not requiring training, but its effectiveness is highly dependent on the choice of pooling strategy, and its effectiveness is also limited.

[0007] * Representation fusion methods based on trainable intermediate layers. Examples include Video-LLaMA and Video-LLaMA2. Unfortunately, both Video-LLaMA and Video-LLaMA2 are trained end-to-end alongside the language model. They cannot be separated and trained as standalone video encoders, limiting their use cases.

[0008] The second approach is based on video encoders. For example, UMT and InternVideo2 use video encoders that can directly encode videos into representations without requiring additional processing. However, training such video encoders requires significant resources, including large amounts of video and text data, numerous graphics card servers, and a significant amount of time. For example, InternVideo2 used one million pieces of data and took a month to train on 256 A100 GPUs. Summary of the Invention

[0009] In view of the shortcomings of the existing technology, the present invention aims to provide an efficient video frame representation fusion method based on a picture encoder. Compared with the existing methods, the present invention has the following main features:

[0010] * The training process is simple and efficient, requiring less video text data and can be completed on a small number of GPUs.

[0011] * Trained as a separate module rather than end-to-end with the language model, and can be used as a standalone video encoder.

[0012] To achieve the above objectives, the present invention provides the following technical solution: a video representation fusion method in a service supervision scenario, characterized by comprising the following steps:

[0013] Step 1: Select training data. For shorter videos, choose the Panda70M dataset. For longer videos, choose Charades, CharadesEgo, ActivityNet, and Valley caption data.

[0014] Step 2: First, obtain the feature cube and then create a frame representation fusion module. The frame representation fusion module uses a static attention mask to ensure that each query only focuses on a fixed local token in the feature cube, or uses a dynamic attention mask to allow the position of the sub-cube that each query focuses on to change as needed.

[0015] In step three, based on Q-Former, three loss functions are created based on the training data selected in step one for joint training to complete the training of the video representation fusion module. The trained video representation fusion module is then used to perform representation fusion on the videos in the service supervision scenario.

[0016] As a further improvement of the present invention, the specific method of obtaining the feature cube in step 2 is: first, preprocess the video, evenly extract frames from the video data and resize it to a uniform scale, and then send the video frames to the image encoder for encoding to obtain the representation of each video frame.

[0017] As a further improvement of the present invention, the representation fusion module constructed in step 2 is a multi-layer transformer model, which includes self-attention and cross-attention mechanisms. During the fusion process, Q learnable queries are used as input, and feature cubes are used as keys and values. Through the cross-attention layer, the frame representations are gradually fused into the Q learnable queries, and the self-attention between the queries is used to complete the mutual fusion of the frame representations.

[0018] As a further improvement of the present invention, the static attention mask used in step 2 is a mask cube with a dimension of (S*T*T). Specifically, the feature cube with a shape of (M*N*N) is divided into many sub-cubes. The size of each sub-cube is (S*T*T), with a total of (M / S * N / T * N / T) = Q sub-cubes. Each learnable query focuses on one sub-cube.

[0019] As a further improvement of the present invention, the dynamic attention mask used in step 2 is as follows: the query output of the transformer cross-attention layer is fed into a linear layer, and the hidden state dimension of the query is mapped from H to 3, which respectively represent the offset of the sub-cube focused on by the current query in the three dimensions of the feature cube. Then, when the next layer of transformer performs cross-attention, the focused sub-cube will move according to this offset, thereby realizing a dynamic attention mask and allowing the model to select the position of the sub-cube as needed.

[0020] As a further improvement of the present invention, the specific method of performing joint training in step three is as follows: first, the Q-Former is converted into the video data domain, corresponding to VTC, VTM and VTG respectively, and then VTC, VTM and VTG are calculated in sequence.

[0021] As a further improvement of the present invention, the method for calculating VTC in step three is as follows: first, the text description of the video is passed through the word segmentation and embedding layers, and then input into the representation fusion module, but no cross-attention calculation is performed. The token corresponding to the text CLS header is taken as the text representation, and then the cosine similarity between the text representation and the Q tokens of the video representation is calculated respectively. Finally, the largest one is taken as the final similarity, and then the cross-entropy loss is calculated.

[0022] As a further improvement of the present invention, the calculation method of VTM in step three is as follows: first, Q learnable queries and text tokens are sent to the representation fusion module together, and self-attention and cross-attention calculations are performed simultaneously. Then, the Q tokens corresponding to the learnable query are output as the result, the average value is calculated, and then a linear layer is passed to perform a binary classification operation and calculate the cross entropy loss.

[0023] As a further improvement of the present invention, the calculation method of VTG in step 3 is as follows: first, Q learnable queries and text tokens are sent to the representation fusion module together, and self-attention and cross-attention calculations are performed simultaneously. The model is then asked to perform the next token prediction task and calculate the cross-entropy loss.

[0024] Among them, causal attention is used when performing self-attention and cross-attention calculations.

[0025] The beneficial effect of the present invention is that, through the setting of step one, appropriate training data can be effectively selected according to the length of the video, which is more targeted. Then, through the setting of step two, the video data can be effectively preprocessed, and frame representation fusion can be performed after preprocessing to achieve efficient feature extraction and reduce feature redundancy. Then, static and dynamic attention masks are used to improve the model's understanding and reasoning ability of the video. Compared with the methods in the prior art, the training process is simple and efficient, requires less video text data, and can be completed on a small number of GPUs. It can be trained as a separate module instead of end-to-end training with the language model, and can be used as a separate video encoder. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Schematic diagram of the model architecture used in the video representation fusion method in the service supervision scenario of the present invention;

[0027] Figure 2 Schematic diagram designed for attention mask;

[0028] Figure 3 Schematic diagram of the MoE feedforward neural network. DETAILED DESCRIPTION

[0029] The present invention will be further described below with reference to the embodiments shown in the accompanying drawings.

[0030] Reference Figures 1 to 3 As shown, a video representation fusion method in a service supervision scenario of this embodiment includes the following three steps: selection of training data, frame representation fusion module architecture, and construction of training loss function.

[0031] Selection of training data

[0032] Because video data contains a large amount of redundant information, a common video processing method is to uniformly extract a fixed number of video frames. If the video is too long, the number of extracted frames will be too small to fully capture the video's information. If the video is too short, the extracted frames will be too homogeneous, hindering the understanding of long videos. Therefore, when selecting training data, it is important to choose a balanced range of video lengths.

[0033] For shorter videos, we chose the Panda70M dataset. This dataset features an average video length of 3-4 seconds, but also offers a large and diverse dataset, which facilitates model generalization. For longer videos, we selected caption data from Charades, CharadesEgo, ActivityNet, and Valley. These datasets have an average video length of approximately 10 seconds, but are relatively small in number. Furthermore, given the high frequency of scene changes and sequential events in video data, we also included the KineticsGEB dataset. This dataset features continuous scenes with sequential connections, which helps the model better learn dynamic information in videos.

[0034] In addition, due to the difficulty in obtaining video data, we also selected image datasets such as CC3M and VisualGenome to add to the training data. By leveraging the diversity of image data, we can achieve modal complementarity and improve the generalization ability of the model.

[0035] Frame representation fusion module architecture

[0036] Get feature cube

[0037] First, the video needs to be preprocessed. The video data is uniformly frame-sampled and resized to a uniform scale. Then, each video frame is sent to the image encoder for encoding to obtain a representation for each frame. Assuming that the representation shape of each video frame is (N*N), the hidden state dimension is H, and the number of randomly sampled video frames is M, the resulting representation is M*N*N*H. Ignoring the hidden state dimension, the representation dimension is M*N*N. Its shape is a cube, which we call a feature cube.

[0038] Frame representation fusion module

[0039] However, the feature cube dimensions for each frame (that is, the dimension where M resides) are independent, and the length of M*N*N is too long to be directly used. Therefore, a representation fusion module is needed to fuse the frame representations into a video representation. This frame representation fusion module is modeled after the Q-Former and uses BERT as its basic framework. BERT is an encoder-only transformer model. To enable the model to focus on video frame representations through cross-attention, a cross-attention layer is introduced between the BERT layers. Q learnable queries are then set as input.

[0040] In summary, our frame representation fusion module is a multi-layer Transformer model that incorporates self-attention and cross-attention mechanisms. It takes Q learnable queries as input, with feature cubes as keys and values. Through the cross-attention layer, it gradually fuses the frame representations into the Q learnable queries. Self-attention between queries further completes the mutual fusion of the frame representations.

[0041] Because our training data includes both video and image caption data, spanning three modalities: video, image, and text, the frame representation fusion module needs to process data from all three modalities simultaneously. Therefore, we modified the FFN (feedforward network) in the transformer block, referencing the Mixture of Experts (MOE) architecture. The FFN was modified from a single MLP to four, each representing an expert. Data computation for the three modalities (video, image, and text) is handled by their respective experts (video expert, image expert, and text expert). A shared expert handles data from all modalities.

[0042] Static attention mask design

[0043] At the cross-attention level, additional changes are required. If full cross-attention is used, each learnable query would need to focus on M*N*N tokens, which is very difficult for the model to learn. Therefore, we designed an attention mask mechanism specifically for this purpose. This allows each query to focus only on a subset of tokens in the feature cube, reducing computational effort and learning difficulty. Specifically, we set up a mask cube of dimension (S*T*T). Each query only needs to focus on (S*T*T) tokens, which is smaller than M*N*N, and each token focuses on a different (S*T*T) tokens. This is equivalent to splitting the (M*N*N) feature cube into many sub-cubes, each of size (S*T*T), for a total of (M / S * N / T *N / T) = Q sub-cubes, with each learnable query focusing on one sub-cube. S is actually segmented along the M dimension, that is, the dimension of different video frames, and T is segmented along the N dimension, that is, within a single video frame. This allows each query to focus on local information at the same location represented by multiple video frames, which helps the model simultaneously extract both temporal and spatial information contained in the video.

[0044] Furthermore, since the representations obtained after the video frame encoder already incorporate spatial position information through position embedding, the representations of each video frame lack additional temporal position information to help the model distinguish the temporal relationship between frames. Therefore, a learnable time embedding is added to the video frame representation. The vocabulary size of the time embedding is M, corresponding to the M uniformly sampled video frames.

[0045] Dynamic attention mask design

[0046] The attention mask mechanism mentioned above is already very effective in shortening the number of tokens that each learnable query must focus on, while fully leveraging the spatiotemporal information of the video feature cube. However, the positions of the tokens focused on by each query are static and unchanging. For example, if the first query focuses on S*T*T tokens in the upper left rear corner of the feature cube, it will remain focused on these tokens. If a complete entity in the video is divided into different sub-cubes, this static approach will clearly hinder the model from extracting relevant information about this entity. Therefore, a dynamic attention mask mechanism is proposed, where the position of the sub-cube focused on by each query changes as needed.

[0047] Referring to the design of Deformable Attention, the query output of the transformer cross-attention layer is fed into a linear layer, and the hidden state dimension of the query is mapped from H to 3, representing the offset of the sub-cube that the current query focuses on in the three dimensions of the feature cube. Then, when the next transformer layer performs cross-attention, the focused sub-cube will move according to this offset, thereby achieving a dynamic attention mask, allowing the model to select the position of the sub-cube as needed. Of course, since the offset calculated by the previous layer does not exist when the first cross-attention is calculated, the setting corresponding to the dynamic attention mask is preset. The offsets of subsequent sub-cubes are all offset based on the position during the previous cross-attention calculation.

[0048] Construction of training loss function

[0049] Also based on Q-Former, Q-Former training uses three loss functions (ITC, ITM, and ITG) for joint training. We translate these to the video data domain, corresponding to (VTC, VTM, and VTG).

[0050] When calculating VTC, for video data, the representation fusion module inputs only Q learnable queries, performs both self-attention and cross-attention calculations, and outputs a video representation of length Q. The video's text description, after undergoing word segmentation and embedding, is also input into the representation fusion module, but cross-attention calculations are not performed. Instead, the token corresponding to the text CLS header is used as the text representation. The cosine similarity is calculated between the text representation and the Q tokens of the video representation. The largest similarity is taken as the final similarity, and then the cross-entropy loss is calculated.

[0051] When computing the VTM, Q learnable query and text tokens are fed into the representation fusion module, where both self-attention and cross-attention are performed. The Q tokens corresponding to the learnable query are output as the result, the average is calculated, and then the result is passed through a linear layer for binary classification (matching or mismatching), and the cross-entropy loss is calculated.

[0052] When calculating VTG, like VTM, Q learnable query and text tokens are fed into the representation fusion module, where both self-attention and cross-attention are calculated. The difference is that VTC and VTM use bidirectional self-attention, while VTG uses causal attention, allowing the model to perform the next token prediction task. Cross-entropy loss is also calculated.

[0053] In summary, the above three steps: selection of training data, frame representation fusion module architecture, and training loss function can be used to simply and effectively implement video representation fusion in service supervision scenarios, and the training process is simple and efficient.

[0054] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A video representation fusion method for a service supervision scenario, characterized by: The steps include: Step 1: Select training data. For shorter videos, choose the Panda70M dataset. For longer videos, choose Charades, CharadesEgo, ActivityNet, and Valley caption data. Step 2: First, obtain the feature cube and then create a frame representation fusion module. The frame representation fusion module uses a static attention mask to ensure that each query only focuses on a fixed local token in the feature cube, or uses a dynamic attention mask to allow the position of the sub-cube that each query focuses on to change as needed. Step 3: Based on the Q-Former, three loss functions are created based on the training data selected in step 1 for joint training to complete the training of the video representation fusion module. Then, the trained video representation fusion module is used to represent and fuse the video in the service supervision scenario. The specific method of obtaining the feature cube in step 2 is as follows: first, the video is preprocessed, the video data is evenly framed and resized to a uniform scale, and then the video frames are sent to the image encoder for encoding to obtain the representation of each video frame. The representation fusion module constructed in step 2 is a multi-layer transformer model, which contains self-attention and cross-attention mechanisms. During the fusion process, Q learnable queries are used as input and feature cubes are used as keys and values. Through the cross-attention layer, the frame representations are gradually fused into the Q learnable queries, and the self-attention between queries is used to complete the mutual fusion of the frame representations. The static attention mask used in step 2 is a mask cube with a dimension of (S*T*T), specifically, the feature cube with a shape of (M*N*N) is divided into many sub-cubes, each sub-cube is of size (S*T*T), and a total of (M / S * N / T * N / T) = Q sub-cubes, and each learnable Query focuses on one sub-cube; the dynamic attention mask used in the second step is as follows: the Query output of the transformer cross-attention layer is fed into a linear layer, and the hidden state dimension of the Query is mapped from H to 3, which respectively represent the offset of the sub-cube focused by the current Query in the three dimensions of the feature cube. Then, when the next layer of transformer performs cross-attention, the focused sub-cube will move according to this offset, thereby realizing a dynamic attention mask and allowing the model to select the position of the sub-cube as needed.

2. The video representation fusion method in the service supervision scenario according to claim 1 is characterized by: The specific method of performing joint training in step three is as follows: first, convert the Q-Former into the video data domain, corresponding to VTC, VTM and VTG respectively, and then calculate VTC, VTM and VTG in sequence.

3. The video representation fusion method in the service supervision scenario according to claim 2 is characterized by: The method for calculating VTC in step 3 is as follows: first, the text description of the video is passed through the word segmentation and embedding layers, and then input into the representation fusion module, but without performing cross-attention calculation. The token corresponding to the text CLS header is taken as the text representation. Then, the cosine similarity between the text representation and the Q tokens of the video representation is calculated respectively. Finally, the largest one is taken as the final similarity, and then the cross-entropy loss is calculated.

4. The video representation fusion method in the service supervision scenario according to claim 3 is characterized by: The calculation method of VTM in step 3 is as follows: first, Q learnable queries and text tokens are sent to the representation fusion module together, and self-attention and cross-attention calculations are performed simultaneously. Then, the Q tokens corresponding to the learnable query are output as the result, the average value is calculated, and then a linear layer is passed to perform a binary classification operation and calculate the cross entropy loss.

5. The video representation fusion method in the service supervision scenario according to claim 4 is characterized by: The calculation method for VTG in step 3 is as follows: first, Q learnable queries and text tokens are sent to the representation fusion module, and self-attention and cross-attention calculations are performed simultaneously. Then, the model performs the next token prediction task and calculates the cross-entropy loss. Among them, VTG adopts causal attention, and VTC and VTM adopt bidirectional attention.

Images