Video understanding multi-task processing model training method based on large language model

Abstract

The invention relates to a video understanding multi-task processing model training method based on a large language model, and the method comprises the following steps: S1, obtaining video data, inputting the video data into a double-branch feature extraction part of a multi-task processing model, and outputting video features; s2, obtaining a prompt word corresponding to the video understanding task; and S3, inputting the video features and the cue words into a decoder part of the multi-task processing model, and outputting a video understanding task result. Compared with the prior art, the method has the advantages that the universality of the video understanding multi-task decoder is improved, the training time is shortened, and the like.

Description

Technical Field

[0001] The present invention relates to a video decoder, and in particular to a video understanding multi-task processing model training method based on a large language model. Background Art

[0002] Video understanding is an important task in the field of computer vision, which involves analyzing and understanding video content. Unlike static images, videos also contain dynamic information in the time dimension, which increases the complexity of analysis. Video understanding is very common in practical applications, such as surveillance systems, robots, and wearable devices. In the past, processing multiple video understanding tasks usually required customizing specialized models for each task, which not only increased the design and computational costs, but also limited the sharing of data and knowledge between different tasks.

[0003] In recent years, large-scale language models (LLMs) have shown powerful capabilities in natural language processing tasks, learning sequence modeling and task processing capabilities on large-scale corpora through pre-training. This inspired the introduction of the advantages of LLMs into video understanding tasks to develop a general video understanding multi-task decoder that can uniformly handle multiple tasks. Implementation solutions of existing technologies:

[0004] 1. VideoChat1 / 2: It uses Video Transformer and Query Transformer to compress video tokens and fine-tune the model through multi-stage training. It can handle tasks such as video question answering and video subtitle generation, but it has limitations in processing long videos and multi-tasks.

[0005] 2.ST-LLM: It uses video transformer and query transformer for video feature encoding and fine-tunes the model through three-stage training. It also faces the challenges of long video understanding and multi-task processing.

[0006] 3. HawkEye: Uses Vision Transformer and Image Query Transformer to encode video frames, and then performs temporal modeling through Video Query Transformer. It has a certain effect on global content understanding, but its dynamic compression module is difficult to optimize and has deficiencies in long-term spatiotemporal context retention.

[0007] In summary, the existing chat-centric Video Large Language Model (Video-LLM) has the following limitations:

[0008] 1. Limitations of task support: These models tend to focus only on short videos and only support one or two tasks, such as video question answering or video caption generation.

[0009] 2. Optimization challenges of dynamic compression techniques: The dynamic compression techniques used in existing methods make optimization difficult, resulting in insufficient multimodal alignment, especially when dealing with long spatiotemporal contexts.

[0010] 3. Large demand for training data: The entire training process of these models requires a large amount of training data and multi-stage (three to four) training strategies, which increases the consumption of computing resources and time. Summary of the invention

[0011] The purpose of the present invention is to provide a video understanding multi-task processing model training method based on a large language model in order to improve the versatility of the video understanding multi-task decoder while reducing the training time.

[0012] The purpose of the present invention can be achieved by the following technical solutions:

[0013] A video understanding multi-task processing model training method based on a large language model, the method is a two-stage training, specifically comprising the following steps:

[0014] Constructing a multi-task processing network, the multi-task processing network includes a dual-branch feature extraction network including a visual encoder, a task decoder and a multi-layer perceptron MLP, obtaining a visual language dataset, pre-training the multi-task processing network based on the visual language dataset, and obtaining pre-trained multi-layer perceptron MLP parameters;

[0015] Obtain a video instruction fine-tuning dataset, input the video instruction fine-tuning dataset into the multi-task processing network for training, use the trained dual-branch feature extraction model as the dual-branch feature extraction part of the multi-task processing model, and use the trained task decoder and multi-layer perceptron MLP as the decoder part of the multi-task processing model;

[0016] After training is completed, the specific steps for performing video understanding tasks based on the multi-task processing model are as follows:

[0017] Obtain video data, input the video data into the dual-branch feature extraction part of the multi-task processing model, and output video features;

[0018] Get the prompt words corresponding to the video understanding task;

[0019] The video features and prompt words are input into the decoder part of the multi-task processing model, and the video understanding task results are output.

[0020] Furthermore, the specific steps of inputting the video instruction fine-tuning dataset into the multi-task processing network for training are:

[0021] Image instruction fine-tuning data is obtained, and question and answer data in the image instruction fine-tuning data is organized into the format of a single round or multi-round dialogue. At the same time, traditional tasks are obtained and converted into question and answer templates to form question and answer instruction fine-tuning data. The image instruction fine-tuning data and the question and answer instruction fine-tuning data are merged to obtain a video instruction fine-tuning data set. The video instruction fine-tuning data set is input into a dual-branch feature extraction network, a task decoder, and a multi-layer perceptron MLP of a multi-task processing network for simultaneous training.

[0022] Furthermore, the specific steps of pre-training the multi-task processing network based on the visual language dataset are:

[0023] A visual language dataset is obtained, wherein the visual language dataset consists of multiple image and image description pairs. The visual language dataset is input into a multi-task processing network, at which time the parameters of the branch feature extraction network and the task decoder are frozen, a multi-layer perceptron MLP is trained, the semantics of vision and language are aligned, and the parameters of the pre-trained multi-layer perceptron MLP are saved.

[0024] Furthermore, the task decoder is a large language model.

[0025] Furthermore, the specific steps of inputting the video data into the dual-branch feature extraction part of the multi-task processing model and outputting the video features are:

[0026] Uniformly sampling a fixed number of frames from the video data to obtain a global spatiotemporal representation, the global spatiotemporal representation is input into a video thumbnail branch in a dual-branch feature extraction part, the global spatiotemporal representation is input into a visual encoder of the video thumbnail branch, a 3D adaptive pooling operation is performed on the image semantic features output by the visual encoder of the video thumbnail branch, and a video thumbnail spatiotemporal sequence is obtained;

[0027] A plurality of frames are sampled from the video data at a fixed frame rate, the plurality of frames are input into the high-definition temporal branch in the dual-branch feature extraction part, the plurality of frames are input into the visual encoder of the high-definition temporal branch, the visual encoder of the high-definition temporal branch outputs frame-level features, and the frame-level features are compressed into temporal high-definition features through a spatiotemporal adaptive pooling operation in a compressed space;

[0028] The video thumbnail spatiotemporal sequence and the temporal high-definition features are spliced ​​to obtain video features.

[0029] Furthermore, the video thumbnail spatiotemporal sequence is a global space-time representation that is independent of the video duration.

[0030] Furthermore, the visual encoder is constructed based on a Transformer network.

[0031] Furthermore, the time series high-definition feature is a one-dimensional time series feature.

[0032] Furthermore, the specific steps for obtaining the prompt words corresponding to the video understanding task are:

[0033] The video understanding task is converted into a unified text generation problem, generating prompt words in the form of textual question and answer.

[0034] Furthermore, the video understanding task is to process video questions, generate subtitles or detect events.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) The two-stage training of the present invention introduces instruction fine-tuning training, and through instruction fine-tuning on the dataset, universal decoding capabilities in various tasks are achieved. This strategy transforms all video understanding tasks into a unified text generation format similar to visual question answering (VQA), supporting a wide range of applications from dense video captioning to video boundary location, summarization, event segmentation, and video question answering.

[0037] (2) The feature extraction part of the present invention uses a dual-branch sampling strategy to reduce the input token length while retaining the global spatiotemporal information of the video, thereby improving the efficiency and performance of processing long videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of the present invention;

[0039] Figure 2 This is a dual-branch feature extraction flow chart of the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0041] The present invention proposes a video understanding multi-task processing model training method based on a large language model, aiming to achieve efficient processing of multiple video understanding tasks by combining a large language model (LLM) and an innovative dual-branch sampling strategy. The flowchart of the actual application of the method of the present invention is as follows Figure 1 As shown, the specific objectives of the present invention are:

[0042] 1. Support more tasks: By converting multiple tasks into a unified text generation problem, the types of tasks that the model can handle are expanded and the task coverage is improved.

[0043] 2. Unified model architecture: Provides a unified model architecture that can handle multiple video understanding tasks within a single framework, reducing design and training costs and improving the adaptability and flexibility of the model.

[0044] 3. Efficient static compression method: Through the dual-branch sampling strategy, while retaining the global spatiotemporal information of the video, the input token length is reduced, improving the efficiency and performance of processing long videos.

[0045] 4. Simplified training process: The two-stage training method of visual language alignment pre-training and instruction fine-tuning is adopted, which makes the process more concise and efficient, reducing the training time and consumption of computing resources.

[0046] The core of the present invention is:

[0047] 1. Unified task processing framework: By converting various video understanding tasks into a unified text generation problem, multi-task processing is achieved, the need for task-specific models is reduced, and the flexibility and generalization ability of task processing are improved.

[0048] 2. Dual-branch sampling strategy: Reduce the number of tokens fed into large language models while maintaining performance, thereby reducing training and inference overhead.

[0049] Video thumbnail branch: uniformly samples a fixed number of frames from the video to generate a global spatiotemporal representation that captures the overall structure and content of the video.

[0050] HD timing branch: samples the video at a fixed frame rate, extracts detailed timing features, and retains the dynamic information of the video sequence.

[0051] 3. Progressive alignment of images to videos and fine-tuning of instructions: A two-stage training approach is adopted. First, the alignment of vision and text is learned through visual language pre-training, and then instructions are fine-tuned on sequential datasets of images and videos to improve the model's instruction following ability and performance in multiple tasks.

[0052] The specific steps of the training method of the present invention are:

[0053] Constructing a multi-task processing network, the multi-task processing network includes a dual-branch feature extraction network including a visual encoder, a task decoder and a multi-layer perceptron MLP, obtaining a visual language dataset, pre-training the multi-task processing network based on the visual language dataset, and obtaining pre-trained multi-layer perceptron MLP parameters;

[0054] Obtain a video instruction fine-tuning dataset, input the video instruction fine-tuning dataset into the multi-task processing network for training, use the trained dual-branch feature extraction model as the dual-branch feature extraction part of the multi-task processing model, and use the trained task decoder and multi-layer perceptron MLP as the decoder part of the multi-task processing model;

[0055] After training is completed, the specific steps for performing video understanding tasks based on the multi-task processing model are as follows:

[0056] Obtain video data, input the video data into the dual-branch feature extraction part of the multi-task processing model, and output video features;

[0057] Get the prompt words corresponding to the video understanding task;

[0058] The video features and prompt words are input into the decoder part of the multi-task processing model, and the video understanding task results are output.

[0059] The model includes the following modules:

[0060] Video encoder: Use the Vision Transformer to encode the input video and generate a video feature representation. The Vision Transformer can extract the spatial features of the video frame and lay the foundation for subsequent temporal modeling by capturing the detailed information of each frame.

[0061] Task decoder: Use a large language model (such as GPT-4) as a task decoder. The decoder combines video features with task prompts (such as questions, subtitle prompts, etc.) to generate corresponding text output. By leveraging the powerful reasoning ability of the language model, the model is able to handle multiple tasks such as video question answering, subtitle generation, event detection, etc. in a unified framework.

[0062] Text generation problem conversion: All video understanding tasks are converted into a unified text generation problem. For example, the video question answering task can be converted into a task of generating descriptive answers, and the video event detection can be converted into a text generation task that describes the start and end time of the event. In this way, the processing of different tasks can be unified into the same decoder, avoiding the use of specialized models for different tasks.

[0063] The dual-branch network includes:

[0064] Video Thumbnail Branch:

[0065] A fixed number of frames (16 frames for the whole video) are uniformly sampled from the video to generate a global spatiotemporal representation. First, 16 frames are uniformly sampled from the whole video, and then each frame is input into the visual encoder to extract frame-level image semantic features. A 3D adaptive pooling operation is used to downsample the video features to a size of 4×12×12 to generate a global spatiotemporal representation that is independent of the video duration.

[0066] HD-Temporal Branch:

[0067] Multiple frames are sampled from the video at a fixed frame rate (e.g., 1 frame per second), and then each frame is input into the visual coder to extract frame-level image semantic features. The obtained frame-level features are compressed into one-dimensional temporal features through the spatiotemporal adaptive pooling operation of the compression space. The final one-dimensional temporal features are high-definition temporal features.

[0068] This branch preserves the dynamic information of the video sequence, ensuring that the model can capture the complete temporal structure in the video.

[0069] This dual-branch strategy not only reduces the number of input tokens and computational complexity, but also improves the efficiency and performance of the model in processing long videos.

[0070] In the above steps, an example of converting the video understanding task into a unified text generation problem is shown in Table 1, which converts the task into prompt words of a question-and-answer template (including a question template and an answer template).

[0071] Table 1. Examples of tasks converted into unified text generation problems

[0072]

[0073]

[0074] This paper adopts a two-stage training method to improve the performance of the model on multiple tasks:

[0075] Visual-language alignment pre-training: Pre-training is performed on carefully selected visual-language datasets to enable the model to learn the alignment relationship between vision and text. During the pre-training process, the model performs joint training alignment through visual features and corresponding text descriptions to enhance the understanding of multimodal information. It is mainly divided into the following steps: 1. First, prepare a conceptually balanced dataset of 558,000 images and image description pairs, organize the data into a single-round or multi-round dialogue format, and use Represents an image placeholder. 2. Use MLP connection to freeze the visual encoder ViT and the large language model LLM. Input the visual tokens of the 576 single images after ViT into the large language model, and train MLP to align the semantics of vision and language. 3. After training, only the MLP parameters are saved to save storage space.

[0076] Progressive instruction fine-tuning: First, fine-tune the instructions on the image dataset, and then continue to fine-tune on the self-defined video instruction fine-tuning dataset. This dataset contains instructions for various video understanding tasks. The model is fine-tuned through these instructions to improve performance on multiple tasks. It is mainly divided into the following steps: 1. Prepare 665,000 image instruction fine-tuning data, and organize the question and answer data into a single-round or multi-round dialogue format. In addition, convert traditional tasks into question and answer templates, and merge them with traditional question and answer instruction fine-tuning data to construct a new video instruction fine-tuning dataset. 2. First load the visual encoder ViT, the large language model LLM, and the pre-trained MLP. During the training phase, train the three components at the same time.

[0077] Through visual-language alignment pre-training and instruction fine-tuning, the present invention not only enhances the model's understanding of multimodal information, but also improves its generalization ability on various video understanding tasks.

[0078] The key points of the present invention are:

[0079] Universal Task Decoder: VideoLLM-D uses a large language model (LLM) as a universal task decoder, without the need for a task-specific header, and directly outputs the final result for each task. This innovation eliminates the need to design a separate custom decoder for each task.

[0080] Multi-task instruction tuning: VideoLLM-D achieves universal decoding capabilities in various tasks through instruction fine-tuning on the VideoLLM-D-IT dataset. This strategy transforms all video understanding tasks into a unified text generation format similar to visual question answering (VQA), supporting a wide range of applications from dense video captioning to video boundary localization, summarization, event segmentation, and video question answering.

[0081] Multi-task optimization and reasoning capabilities: By designing a general decoder and introducing specialized datasets (such as VideoLLM-D-ZS-IT and VideoLLM-D-IT instruction datasets), VideoLLM-D is endowed with multi-task optimization and reasoning capabilities, enabling it to handle multiple video understanding tasks in a unified instruction-driven framework.

[0082] Based on the above key points, the present invention achieves the following effects:

[0083] 1. Support more tasks: By converting multiple tasks into a unified text generation problem, the types of tasks that the model can handle are expanded and the task coverage is improved.

[0084] 2. Unified model architecture: reduces design and training costs and improves the adaptability and flexibility of the model.

[0085] 3. Efficient spatiotemporal context processing: Through the dual-branch sampling strategy, the efficiency and performance of processing long videos are improved.

[0086] 4. Simplified training process: A simpler and more efficient training method is adopted to reduce the training time and consumption of computing resources.

[0087] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A video understanding multi-task processing model training method based on a large language model, characterized in that: The method is a two-stage training, which specifically includes the following steps: Constructing a multi-task processing network, the multi-task processing network includes a dual-branch feature extraction network including a visual encoder, a task decoder and a multi-layer perceptron MLP, obtaining a visual language dataset, pre-training the multi-task processing network based on the visual language dataset, and obtaining pre-trained multi-layer perceptron MLP parameters; Obtain a video instruction fine-tuning dataset, input the video instruction fine-tuning dataset into the multi-task processing network for training, use the trained dual-branch feature extraction model as the dual-branch feature extraction part of the multi-task processing model, and use the trained task decoder and multi-layer perceptron MLP as the decoder part of the multi-task processing model; After training is completed, the specific steps for performing video understanding tasks based on the multi-task processing model are as follows: Obtain video data, input the video data into the dual-branch feature extraction part of the multi-task processing model, and output video features; Get the prompt words corresponding to the video understanding task; The video features and prompt words are input into the decoder part of the multi-task processing model, and the video understanding task results are output.

2. According to the method for training a video understanding multi-task processing model based on a large language model in claim 1, it is characterized in that: The specific steps of inputting the video instruction fine-tuning dataset into the multi-task processing network for training are: Image instruction fine-tuning data is obtained, and question and answer data in the image instruction fine-tuning data is organized into the format of a single round or multi-round dialogue. At the same time, traditional tasks are obtained and converted into question and answer templates to form question and answer instruction fine-tuning data. The image instruction fine-tuning data and the question and answer instruction fine-tuning data are merged to obtain a video instruction fine-tuning data set. The video instruction fine-tuning data set is input into a dual-branch feature extraction network, a task decoder, and a multi-layer perceptron MLP of a multi-task processing network for simultaneous training.

3. The video understanding multi-task processing model training method based on a large language model according to claim 2 is characterized in that: The specific steps of pre-training the multi-task processing network based on the visual language dataset are: A visual language dataset is obtained, wherein the visual language dataset consists of multiple image and image description pairs. The visual language dataset is input into a multi-task processing network, at which time the parameters of the branch feature extraction network and the task decoder are frozen, a multi-layer perceptron MLP is trained, the semantics of vision and language are aligned, and the parameters of the pre-trained multi-layer perceptron MLP are saved.

4. The video understanding multi-task processing model training method based on a large language model according to claim 3 is characterized in that: The task decoder is a large language model.

5. The video understanding multi-task processing model training method based on a large language model according to claim 1, characterized in that: The specific steps of inputting the video data into the dual-branch feature extraction part of the multi-task processing model and outputting the video features are as follows: Uniformly sampling a fixed number of frames from the video data to obtain a global spatiotemporal representation, the global spatiotemporal representation is input into a video thumbnail branch in a dual-branch feature extraction part, the global spatiotemporal representation is input into a visual encoder of the video thumbnail branch, a 3D adaptive pooling operation is performed on the image semantic features output by the visual encoder of the video thumbnail branch, and a video thumbnail spatiotemporal sequence is obtained; A plurality of frames are sampled from the video data at a fixed frame rate, the plurality of frames are input into the high-definition temporal branch in the dual-branch feature extraction part, the plurality of frames are input into the visual encoder of the high-definition temporal branch, the visual encoder of the high-definition temporal branch outputs frame-level features, and the frame-level features are compressed into temporal high-definition features through a spatiotemporal adaptive pooling operation in a compressed space; The video thumbnail spatiotemporal sequence and the temporal high-definition features are spliced ​​to obtain video features.

6. The video understanding multi-task processing model training method based on a large language model according to claim 5, characterized in that: The video thumbnail spatiotemporal sequence is a global space-time representation that is independent of the video duration.

7. The video understanding multi-task processing model training method based on a large language model according to claim 5, characterized in that: The visual encoder is built based on the Transformer network.

8. The video understanding multi-task processing model training method based on a large language model according to claim 5, characterized in that: The time series high-definition feature is a one-dimensional time series feature.

9. The video understanding multi-task processing model training method based on a large language model according to claim 1, characterized in that: The specific steps to obtain the prompt words corresponding to the video understanding task are: The video understanding task is converted into a unified text generation problem, generating prompt words in the form of textual question and answer.

10. The video understanding multi-task processing model training method based on a large language model according to claim 1, characterized in that: The video understanding task is to process video questions, generate subtitles or detect events.

Images