A method for training a video understanding multi-task processing model based on a large language model

Abstract

The application relates to a video understanding multi-task processing model training method based on a large language model, and the method comprises the following steps: S1, acquiring video data, inputting the video data into a double-branch feature extraction part of a multi-task processing model, and outputting video features; S2, acquiring prompt words corresponding to a video understanding task; S3, inputting the video features and the prompt words into a decoder part of the multi-task processing model, and outputting a video understanding task result. Compared with the prior art, the application has the advantages of improving the universality of a video understanding multi-task decoder and reducing training time.

Description

Technical Field

[0001] This invention relates to video decoders, and more particularly to a training method for a video understanding multi-task processing model based on a large language model. Background Technology

[0002] Video understanding is a crucial task in computer vision, involving the analysis and comprehension of video content. Unlike static images, videos also contain dynamic information over time, increasing the complexity of the analysis. Video understanding has wide applications in practical fields such as surveillance systems, robotics, and wearable devices. In the past, handling multiple video understanding tasks typically required customizing specialized models for each task, which not only increased design and computational costs but also limited the sharing of data and knowledge across different tasks.

[0003] In recent years, Large-Scale Language Models (LLMs) have demonstrated powerful capabilities in natural language processing tasks, learning sequence modeling and task processing abilities on large-scale corpora through pre-training. This has inspired the application of the advantages of LLMs to video understanding tasks, aiming to develop a general-purpose multi-task decoder for video understanding capable of uniformly handling multiple tasks. Existing implementation schemes include:

[0004] 1. VideoChat1 / 2: It uses a video transformer and a query transformer (Q-Former) to compress video tokens and fine-tunes the model through multi-stage training. It can handle tasks such as video question answering and video caption generation, but it has limitations in handling long videos and multi-task tasks.

[0005] 2. ST-LLM: It uses video transformers and query transformers for video feature encoding and fine-tunes the model through three-stage training. It also faces the challenges of long video understanding and multi-task processing.

[0006] 3. HawkEye: It uses a vision transformer and an image query transformer to encode video frames and then performs temporal modeling through a video query transformer. It has some effect on global content understanding, but its dynamic compression module is difficult to optimize and it is insufficient in preserving long spatiotemporal context.

[0007] In summary, the existing chat-centric video large language model (Video-LLM) has the following limitations:

[0008] 1. Limitations in task support: These models often focus only on short videos and support only one or two tasks, such as video question answering or video caption generation.

[0009] 2. Optimization challenges of dynamic compression techniques: The dynamic compression techniques used in existing methods make optimization difficult, leading to insufficient multimodal alignment, especially when dealing with long spatiotemporal contexts.

[0010] 3. Large training data requirements: The entire training process of these models requires a large amount of training data and multiple (three to four) stages of training strategies, which increases the consumption of computing resources and time. Summary of the Invention

[0011] The purpose of this invention is to provide a training method for a video understanding multi-task processing model based on a large language model, which aims to improve the versatility of the video understanding multi-task decoder while reducing training time.

[0012] The objective of this invention can be achieved through the following technical solutions:

[0013] A training method for a video understanding multi-task processing model based on a large language model is proposed. The method involves two-stage training and includes the following steps:

[0014] A multi-task processing network is constructed, which includes a dual-branch feature extraction network containing a visual encoder, a task decoder, and a multilayer perceptron (MLP). A visual language dataset is obtained, and the multi-task processing network is pre-trained based on the visual language dataset to obtain the pre-trained MLP parameters.

[0015] Obtain the video instruction fine-tuning dataset, input the video instruction fine-tuning dataset into the multi-task processing network for training, use the trained two-branch feature extraction model as the two-branch feature extraction part of the multi-task processing model, and use the trained task decoder and multilayer perceptron (MLP) as the decoder part of the multi-task processing model.

[0016] After training, the specific steps for video understanding based on the multi-task processing model are as follows:

[0017] Acquire video data, input the video data into the dual-branch feature extraction part of the multi-task processing model, and output video features;

[0018] Obtain the prompt words corresponding to the video comprehension task;

[0019] The video features and prompts are input into the decoder part of the multi-task processing model, and the video understanding task results are output.

[0020] Furthermore, the specific steps for inputting the video instruction fine-tuning dataset into the multi-task processing network for training are as follows:

[0021] Image instruction fine-tuning data is acquired, and the question-and-answer data in the image instruction fine-tuning data is organized into a single-turn or multi-turn dialogue format. At the same time, traditional task conversion into question-and-answer templates is acquired to form question-and-answer instruction fine-tuning data. The image instruction fine-tuning data and question-and-answer instruction fine-tuning data are merged to obtain a video instruction fine-tuning dataset. The video instruction fine-tuning dataset is input into the dual-branch feature extraction network, task decoder, and multilayer perceptron (MLP) of the multi-task processing network for simultaneous training.

[0022] Furthermore, the specific steps for pre-training the multi-task processing network based on the visual language dataset are as follows:

[0023] A visual language dataset is obtained, which consists of multiple images and image description pairs. The visual language dataset is input into a multi-task processing network. At this time, the parameters of the branch feature extraction network and the task decoder are frozen. A multilayer perceptron (MLP) is trained to align the semantics of vision and language. The parameters of the pre-trained MLP are saved.

[0024] Furthermore, the task decoder is a large language model.

[0025] Furthermore, the specific steps for inputting video data into the dual-branch feature extraction part of the multi-task processing model and outputting video features are as follows:

[0026] A fixed number of frames are uniformly sampled from the video data to obtain a global spatiotemporal representation. The global spatiotemporal representation is input into the video thumbnail branch of the dual-branch feature extraction part. The global spatiotemporal representation is input into the visual encoder of the video thumbnail branch. The image semantic features output by the visual encoder of the video thumbnail branch are subjected to 3D adaptive pooling operation to obtain the video thumbnail spatiotemporal sequence.

[0027] Multiple frames are sampled from the video data at a fixed frame rate. These multiple frames are input into the high-definition temporal branch of the dual-branch feature extraction part. The high-definition temporal branch visual encoder outputs frame-level features. The frame-level features are compressed into temporal high-definition features through spatiotemporal adaptive pooling operation in the compression space.

[0028] The video features are obtained by concatenating the video's thumbnail spatiotemporal sequence and time-series high-definition features.

[0029] Furthermore, the video thumbnail spatiotemporal sequence is a global spatial-temporal representation independent of the video duration.

[0030] Furthermore, the visual encoder is built based on a Transformer network.

[0031] Furthermore, the time-series high-definition feature is a one-dimensional time-series feature.

[0032] Furthermore, the specific steps for obtaining the prompt words corresponding to the video comprehension task are as follows:

[0033] The video understanding task is transformed into a unified text generation question, generating prompts in the form of textual question-and-answer format.

[0034] Furthermore, the video understanding task includes handling video issues, generating subtitles, or detecting events.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) The two-stage training of this invention introduces instruction fine-tuning training. Through instruction fine-tuning on the dataset, a general decoding capability in various tasks is achieved. This strategy transforms all video understanding tasks into a unified text generation format similar to visual question answering (VQA), supporting a wide range of applications from dense video captioning to video boundary localization, summarization, event segmentation, and video question answering.

[0037] (2) The feature extraction part of the present invention uses a dual-branch sampling strategy to reduce the length of the input token while preserving the global spatiotemporal information of the video, thereby improving the efficiency and performance of processing long videos. Attached Figure Description

[0038] Figure 1 This is a flowchart of the present invention;

[0039] Figure 2 This is a flowchart of the dual-branch feature extraction process of the present invention. Detailed Implementation

[0040] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0041] This invention proposes a training method for a multi-task video understanding processing model based on a large language model (LLM). The aim is to achieve efficient processing of various video understanding tasks by combining a large-scale language model (LLM) with an innovative dual-branch sampling strategy. A flowchart illustrating the practical application of this method is shown below. Figure 1 As shown, the specific objective of this invention is:

[0042] 1. Support more tasks: By transforming various tasks into a unified text generation problem, the model can handle more types of tasks and improve its task coverage.

[0043] 2. Unified Model Architecture: Provides a unified model architecture that can handle multiple video understanding tasks within a single framework, reducing design and training costs and improving the model's adaptability and flexibility.

[0044] 3. Efficient static compression method: By adopting a dual-branch sampling strategy, the length of the input token is reduced while preserving the global spatiotemporal information of the video, thereby improving the efficiency and performance of processing long videos.

[0045] 4. Simplified training process: The training method adopts a two-stage approach of visual language alignment pre-training and instruction fine-tuning, which makes the process simpler and more efficient, reducing training time and computational resource consumption.

[0046] The core of this invention lies in:

[0047] 1. Unified task processing framework: By transforming various video understanding tasks into a unified text generation problem, multi-task processing is achieved, reducing the need for task-specific models and improving the flexibility and generalization ability of task processing.

[0048] 2. Dual-branch sampling strategy: While maintaining performance, reduce the number of tokens input to a large language model, thereby reducing training and inference overhead.

[0049] Video thumbnail branch: Samples a fixed number of frames evenly from the video to generate a global spatiotemporal representation, capturing the overall structure and content of the video.

[0050] High-definition temporal branch: Samples video at a fixed frame rate, extracts detailed temporal features, and preserves the dynamic information of the video sequence.

[0051] 3. Progressive alignment and instruction fine-tuning from image to video: A two-stage training method is adopted. First, the alignment of vision and text is learned through visual language pre-training. Then, instruction fine-tuning is performed on sequential datasets of images and videos to improve the model's instruction following ability and performance in various tasks.

[0052] The specific steps of the training method of the present invention are as follows:

[0053] A multi-task processing network is constructed, which includes a dual-branch feature extraction network containing a visual encoder, a task decoder, and a multilayer perceptron (MLP). A visual language dataset is obtained, and the multi-task processing network is pre-trained based on the visual language dataset to obtain the pre-trained MLP parameters.

[0054] Obtain the video instruction fine-tuning dataset, input the video instruction fine-tuning dataset into the multi-task processing network for training, use the trained two-branch feature extraction model as the two-branch feature extraction part of the multi-task processing model, and use the trained task decoder and multilayer perceptron (MLP) as the decoder part of the multi-task processing model.

[0055] After training, the specific steps for video understanding based on the multi-task processing model are as follows:

[0056] Acquire video data, input the video data into the dual-branch feature extraction part of the multi-task processing model, and output video features;

[0057] Obtain the prompt words corresponding to the video comprehension task;

[0058] The video features and prompts are input into the decoder part of the multi-task processing model, and the video understanding task results are output.

[0059] The model includes the following modules:

[0060] Frequency encoder: Employs a vision transformer to encode the input video, generating a video feature representation. The vision transformer can extract spatial features from video frames, capturing detailed information from each frame to lay the foundation for subsequent temporal modeling.

[0061] Task Decoder: Utilizes a large language model (such as GPT-4) as the task decoder. This decoder combines video features with task cues (such as questions, captions, etc.) to generate corresponding text output. By leveraging the powerful reasoning capabilities of the language model, the model can handle various tasks such as video question answering, caption generation, and event detection within a unified framework.

[0062] Text generation problem transformation: All video understanding tasks are transformed into a unified text generation problem. For example, video question answering can be transformed into generating descriptive answers, and video event detection can be transformed into a text generation task describing the start and end times of events. In this way, the processing of different tasks can be unified into a single decoder, avoiding the need to use dedicated models for different tasks.

[0063] A two-branch network includes:

[0064] Video Thumbnail Branch:

[0065] A global spatiotemporal representation is generated by uniformly sampling a fixed number of frames from the video (16 frames from the entire video). First, 16 frames are uniformly sampled from the entire video. Then, each frame is input into a visual encoder to extract frame-level image semantic features. A 3D adaptive pooling operation is used to downsample the video features to a size of 4×12×12, generating a global spatiotemporal representation that is independent of the video duration.

[0066] HD-Temporal Branch:

[0067] Multiple frames are sampled from the video at a fixed frame rate (e.g., 1 frame per second), and each frame is then input into a visual encoder to extract frame-level image semantic features. The resulting frame-level features are compressed into one-dimensional temporal features through a spatiotemporal adaptive pooling operation. The final one-dimensional temporal features are the high-definition temporal features.

[0068] This branch preserves the dynamic information of the video sequence, ensuring that the model can capture the complete temporal structure in the video.

[0069] This two-branch strategy not only reduces the number of input tokens and computational complexity, but also improves the efficiency and performance of the model in processing long videos.

[0070] In the above steps, examples of converting video understanding tasks into unified text generation questions are shown in Table 1, which provides prompts for converting tasks into question-and-answer templates (including question templates and answer templates).

[0071] Table 1 shows examples of tasks transformed into a unified text generation problem.

[0072]

[0073]

[0074] This invention employs a two-stage training method to improve the model's performance on various tasks:

[0075] Visual-Language Alignment Pre-training: Pre-training is performed on a carefully selected visual-language dataset to enable the model to learn the alignment relationships between visuals and text. During pre-training, the model is jointly trained to align visual features and corresponding text descriptions, enhancing its understanding of multimodal information. The main steps are as follows: 1. First, prepare a concept-balanced dataset of 558,000 images and image description pairs, and organize the data into a single-turn or multi-turn dialogue format. 1. Indicates image placeholders. 2. Employs an MLP connection approach, freezing the visual encoder ViT and the large language model LLM. The visual tokens of the 576 individual images following ViT are input into the large language model to train the MLP for aligning visual and linguistic semantics. 3. After training, only the MLP parameters are saved to save storage space.

[0076] Progressive instruction fine-tuning: First, instruction fine-tuning is performed on an image dataset, then further fine-tuning is done on a self-defined video instruction fine-tuning dataset. This dataset contains instructions for various video understanding tasks, and the model is fine-tuned using these instructions to improve performance across multiple tasks. The main steps are as follows: 1. Prepare 665,000 image instruction fine-tuning data points, organizing the question-and-answer data into single-turn or multi-turn dialogue formats. Additionally, traditional tasks are converted into question-and-answer templates and merged with traditional question-and-answer instruction fine-tuning data to construct a new video instruction fine-tuning dataset. 2. First, load the visual encoder ViT, the large language model LLM, and the pre-trained MLP. During the training phase, all three components are trained simultaneously.

[0077] By using visual language alignment pre-training and instruction fine-tuning, this invention not only enhances the model's understanding of multimodal information, but also improves its generalization ability on various video understanding tasks.

[0078] The key point of this invention is:

[0079] Universal Task Decoder: VideoLLM-D uses a large language model (LLM) as a universal task decoder, directly outputting the final results for each task without requiring a task-specific header. This innovation eliminates the need to design a separate custom decoder for each task.

[0080] Multi-task instruction tuning: VideoLLM-D achieves general decoding capabilities across various tasks through instruction fine-tuning on the VideoLLM-D-IT dataset. This strategy transforms all video understanding tasks into a unified text generation format similar to visual question answering (VQA), supporting a wide range of applications from dense video captioning to video boundary localization, summarization, event segmentation, and video question answering.

[0081] Multi-task optimization and inference capabilities: By designing a general decoder and introducing specialized datasets (such as the VideoLLM-D-ZS-IT and VideoLLM-D-IT instruction datasets), VideoLLM-D is endowed with multi-task optimization and inference capabilities, enabling it to handle a variety of video understanding tasks within a unified instruction-driven framework.

[0082] Based on the above key points, the present invention achieves the following effects:

[0083] 1. Support for more tasks: By transforming multiple tasks into a unified text generation problem, the model expands the types of tasks it can handle and improves the task coverage.

[0084] 2. Unified model architecture: Reduces design and training costs and improves the adaptability and flexibility of the model.

[0085] 3. Efficient spatiotemporal context processing: The dual-branch sampling strategy improves the efficiency and performance of processing long videos.

[0086] 4. Simplified training process: A simpler and more efficient training method is adopted, reducing training time and computational resource consumption.

[0087] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A training method for a video understanding multi-task processing model based on a large language model, characterized in that, The method involves two-stage training, specifically including the following steps: A multi-task processing network is constructed, which includes a dual-branch feature extraction network with a visual encoder, a task decoder, and a multilayer perceptron (MLP). A visual language dataset is obtained, and the multi-task processing network is pre-trained based on the visual language dataset to obtain the pre-trained MLP parameters. Obtain the video instruction fine-tuning dataset, input the video instruction fine-tuning dataset into the multi-task processing network for training, use the trained two-branch feature extraction model as the two-branch feature extraction part of the multi-task processing model, and use the trained task decoder and multilayer perceptron (MLP) as the decoder part of the multi-task processing model. After training, the specific steps for video understanding based on the multi-task processing model are as follows: Acquire video data, input the video data into the dual-branch feature extraction part of the multi-task processing model, and output video features; Obtain the prompt words corresponding to the video comprehension task; The video features and prompts are input into the decoder part of the multi-task processing model, and the video understanding task results are output. The specific steps for inputting video data into the dual-branch feature extraction part of the multi-task processing model and outputting video features are as follows: A fixed number of frames are uniformly sampled from the video data to obtain a global spatiotemporal representation. The global spatiotemporal representation is input into the video thumbnail branch of the dual-branch feature extraction part. The global spatiotemporal representation is input into the visual encoder of the video thumbnail branch. The image semantic features output by the visual encoder of the video thumbnail branch are subjected to 3D adaptive pooling operation to obtain the video thumbnail spatiotemporal sequence. Multiple frames are sampled from the video data at a fixed frame rate. These multiple frames are input into the high-definition temporal branch of the dual-branch feature extraction part. The high-definition temporal branch visual encoder outputs frame-level features. The frame-level features are compressed into temporal high-definition features through spatiotemporal adaptive pooling operation in the compression space. The video features are obtained by concatenating the video's thumbnail spatiotemporal sequence and time-series high-definition features.

2. The method for training a video understanding multi-task processing model based on a large language model according to claim 1, characterized in that, The specific steps for inputting the video instruction fine-tuning dataset into the multi-task processing network for training are as follows: Image instruction fine-tuning data is acquired, and the question-and-answer data in the image instruction fine-tuning data is organized into a single-turn or multi-turn dialogue format. At the same time, traditional task conversion into question-and-answer templates is acquired to form question-and-answer instruction fine-tuning data. The image instruction fine-tuning data and question-and-answer instruction fine-tuning data are merged to obtain a video instruction fine-tuning dataset. The video instruction fine-tuning dataset is input into the dual-branch feature extraction network, task decoder, and multilayer perceptron (MLP) of the multi-task processing network for simultaneous training.

3. The method for training a video understanding multi-task processing model based on a large language model according to claim 2, characterized in that, The specific steps for pre-training the multi-task processing network based on the visual language dataset are as follows: A visual language dataset is obtained, which consists of multiple images and image description pairs. The visual language dataset is input into a multi-task processing network. At this time, the parameters of the branch feature extraction network and the task decoder are frozen. A multilayer perceptron (MLP) is trained to align the semantics of vision and language. The parameters of the pre-trained MLP are saved.

4. The method for training a video understanding multi-task processing model based on a large language model according to claim 3, characterized in that, The task decoder is a large language model.

5. The method for training a video understanding multi-task processing model based on a large language model according to claim 1, characterized in that, The video thumbnail spatiotemporal sequence is a global spatial-temporal representation that is independent of the video duration.

6. The method for training a video understanding multi-task processing model based on a large language model according to claim 1, characterized in that, The visual encoder is built based on a Transformer network.

7. The method for training a video understanding multi-task processing model based on a large language model according to claim 1, characterized in that, The time-series high-definition feature is a one-dimensional time-series feature.

8. The method for training a video understanding multi-task processing model based on a large language model according to claim 1, characterized in that, The specific steps to obtain the prompt words corresponding to the video comprehension task are as follows: The video understanding task is transformed into a unified text generation question, generating prompts in the form of textual question-and-answer format.

9. The method for training a video understanding multi-task processing model based on a large language model according to claim 1, characterized in that, The video understanding task involves handling video issues, generating subtitles, or detecting events.

Images