A dense video detection method based on a large-scale language model

Abstract

The application discloses a dense video detection method based on a large-scale language model, which comprises the following steps: collecting an existing image caption dataset and a video caption dataset, and constructing image-text pairs and frame image sequence-text pairs; building a multi-modal large language model framework, performing distributed training through a DeepSpeed tool, and fine-tuning the multi-modal large language model by using the image-text pairs and the frame image sequence-text pairs; collecting an existing dense video dataset, uniformly sampling frames of the dense video to obtain frame image sequences, and filling and truncating events of the dense video; and constructing a model input paradigm, and performing supervised fine-tuning on the fine-tuned multi-modal large language model to realize dense video detection. The application combines dense video detection with a multi-modal large language model, and fully utilizes text information of image caption and video caption tasks and visual and text understanding capabilities of the multi-modal large language model through transfer learning.

Description

Technical Field

[0001] This invention belongs to the field of dense video detection technology, and relates to a dense video detection method based on a large-scale language model. Background Technology

[0002] Dense video detection aims to locate all events in a video in time. It can be effectively applied to online videos, helping users quickly locate video segments of interest without watching the entire video. This aids in understanding longer videos and is a highly significant task.

[0003] Traditional dense video detection tasks can be broken down into two sub-tasks: event localization and segment-level video caption recognition. Event localization predicts the temporal boundaries of each event and identifies video captions based on the located time, typically requiring two different underlying model architectures. This two-stage approach not only demands more manual component design, but the effectiveness of segment-level video captioning also heavily relies on the accuracy of event localization. Because event localization and segment-level video recognition are performed separately, the potential interaction between the two is lacking, reducing the correlation between event localization and event content understanding, and also affecting the accuracy of localization. Therefore, a simpler and more effective end-to-end dense video detection method has emerged.

[0004] End-to-end dense video detection tasks perform event localization and segment-level video caption recognition simultaneously. A common approach is to construct two parallel prediction heads—an event localization head and a video caption recognition head. Frame image features extracted from the visual encoder are fused with learnable event query features in the model to obtain intermediate features, which are then directly decoded into a set of events with temporal locations and corresponding captions. Learnable event query features are typically of fixed length. To adapt to videos of varying lengths, an additional event counter is usually stacked on top of the decoder to further predict the final number of events. End-to-end dense video detection has a simpler model structure and effectively addresses the lack of potential correlation between event localization and segment-level video caption recognition in two-stage tasks.

[0005] Large-scale language models (LLMs) are currently a hot topic in natural language processing, demonstrating remarkable capabilities in complex tasks such as text understanding and reasoning. Numerous domain-specific models are built upon LLMs. By utilizing pre-trained large-scale language models, pre-trained image encoders, and additional learnable adapters, supervised fine-tuning on multimodal corpora can yield multimodal large-scale language models (MLLMs). Multimodal large-scale language models not only inherit the powerful text content recognition capabilities of their foundations but also gain a deeper understanding of image modal content, performing complex multimodal tasks (such as image understanding and image question answering). Recent research in the multimodal field has demonstrated that fine-tuning a wide range of downstream tasks based on multimodal large-scale language models can yield remarkable results, such as visual question answering and visual commonsense reasoning. End-to-end intensive video detection based on multimodal large-scale language models can effectively leverage the visual content understanding capabilities of large-scale language models, enhancing the accuracy of event localization.

[0006] Dense video detection tasks require models to understand complex video content. Traditional dense video detection models have few parameters, simple structures, and insufficient training data, resulting in incomplete understanding and inaccurate segmentation of videos. Multimodal large language models, based on large-scale language models, have demonstrated powerful visual understanding capabilities, providing a new solution for dense video detection tasks. However, the insufficient data volume in dense video detection tasks makes it impossible to train multimodal large language models. Existing solutions involve extracting audio from the video, transcribing it into text, and performing data augmentation. However, this approach suffers from complex operations, limited effective content in the transcribed text, and discrepancies between the audio and video content. Summary of the Invention

[0007] The purpose of this invention is to address the problems of simple model structure, inaccurate segmentation, and insufficient visual understanding in dense video detection, and to propose a dense video detection method based on a large-scale language model.

[0008] This invention provides a dense video detection method based on a large-scale language model, comprising:

[0009] Step 1: Collect existing image caption datasets and construct image-text pairs; collect existing video caption datasets and construct frame image sequence-text pairs;

[0010] Step 2: Build a multimodal large language model framework and perform distributed training using the DeepSpeed ​​tool, and fine-tune the multimodal large language model using image-text pairs and frame image sequence-text pairs;

[0011] Step 3: Collect existing dense video datasets, uniformly sample the dense videos to obtain frame image sequences, and pad and truncate the events in the dense videos;

[0012] Step 4: Construct the model input paradigm, including event query prefixes, frame image sequence features, and text sequence features, and perform supervised fine-tuning of the multimodal large language model after fine-tuning in Step 2 to achieve dense video detection.

[0013] Furthermore, step 1 specifically includes:

[0014] Step 1.1: Collect publicly available datasets for image captioning tasks and construct image-text pairs;

[0015] Step 1.2: Collect video text pairs for the video captioning task, uniformly sample the video to obtain a frame image sequence, and construct frame image sequence-text pairs.

[0016] Furthermore, step 2 specifically includes:

[0017] Step 2.1: Build a multimodal large language model framework, including an image encoder, a multimodal adapter, and a large-scale language model;

[0018] Step 2.2: Build a distributed training environment using the DeepSpeed ​​tool, and use Zero-3 data parallelism for training;

[0019] Step 2.3: Freeze the image encoder parameters and fine-tune the multimodal adapter and large language model using full parameters of image-text pairs and frame image sequence-text pairs.

[0020] Furthermore, in step 2.1, the image encoder uses the ViT model from the pre-trained CLIP model, the multimodal adapter uses the MLP model, and the large-scale language model uses the Llama2-7B model. The image is encoded in a single modality by the ViT model, and then the image features are mapped to the text representation space of the LLM by the multimodal adapter. The ViT model is frozen during training, and multimodal alignment is performed between the multimodal adapter and the LLM.

[0021] Furthermore, step 3 specifically includes:

[0022] Step 3.1: Collect public datasets for dense video detection tasks, uniformly sample dense videos to obtain frame image sequences, and align the event localization time with the frame sequences;

[0023] Step 3.2: Count the number of events in each video in the dense video dataset, select the maximum number of events N, and fill the number of events in videos with less than N with {0,0}, where {0,0} means that the start frame number and end frame number of the filled event are both 0. Perform event truncation operation on videos with more than the maximum number of events N, and delete the redundant events.

[0024] Furthermore, in step 3.1, 100 frames of images are uniformly sampled from the video, and videos with fewer than 100 frames are deleted.

[0025] Furthermore, the selection of the maximum number of events N in step 3.2 is based on the proportion of videos in the dense video dataset with an event number less than or equal to N, rounded down to 90%. The difficulty of the event localization task lies in the prediction of the event start time and end time, which requires converting the prediction of the event start time and end time into the prediction of the event start frame number and end frame number.

[0026] Furthermore, step 4 specifically involves:

[0027] Step 4.1: At the start of training, the frame image sequence is processed by an image encoder to obtain frame image sequence features; the text corresponding to each event is processed by the embedding layer of a large-scale language model to obtain text sequence features;

[0028] Step 4.2: Randomly initialize the event query prefix. The event query prefix is ​​represented by a set of learnable vectors. The event query prefix is ​​concatenated with the frame image sequence features and text sequence features obtained in Step 4.1 to form the model input paradigm. The input paradigm is then input into the multimodal large language model.

[0029] Step 4.3: Predict the start and end frame numbers of events one-to-one using learnable event query prefixes, transforming event localization into an ensemble prediction task, and calculate the loss for model training.

[0030] Furthermore, in the model input paradigm of step 4.2, the frame image sequence is segmented using the special character [SEP]; the text sequence is segmented using the special character [event].

[0031] Furthermore, the expression for the Loss function of the model in step 4.3 is:

[0032] Loss = event ce +event giou +caption ce

[0033] Where Loss represents the model's loss function, and event... ce The cross-entropy loss represents the predicted start and end frame numbers of the event compared to the actual start and end frame numbers; event giou The GIOU loss represents the difference between the predicted and actual event ranges; caption ce The cross-entropy loss represents the difference between the predicted and actual text sequences; the actual start and end frame numbers of the events in the video are represented as... The actual text sequence in the video is represented as [x1, x2…x n ],in 1≤n≤N, where C represents the number of images in the frame image sequence, N represents the number of events in the video, s represents the start frame, and e represents the end frame; the length of the event query prefix feature is 2N. After passing through the large language model, it is mapped to the frame representation space by the event localization header to obtain the predicted probability corresponding to frame C+1. Where 0 ≤ k ≤ C;

[0034] The difference between the predicted start and end frame numbers of the event and the actual start and end frame numbers of the event is calculated using the cross-entropy loss formula:

[0035]

[0036] Among them, u k This represents the one-hot representation of frame number i; This represents the probability that event n starts at frame number k. This represents the probability that the termination frame number of event n is k.

[0037] The predicted event range and the actual event range are constrained using the following formula:

[0038]

[0039] Where A represents the predicted range of events, B represents the actual range of events, and D represents the minimum range that can contain both A and B;

[0040] The difference between the predicted text sequence and the true text sequence is calculated using the cross-entropy loss formula:

[0041]

[0042] Where L represents the text length, p(y nh |x n ,y n Let x represent the probability of the caption for event n when predicting the h-th word. n It is real text, y n It is the predicted text, y nh It is the h-th word of the predicted text, y n,＜h It refers to the first h-1 words of the predicted text.

[0043] The dense video detection method based on a large-scale language model of the present invention has at least the following beneficial effects:

[0044] 1. This invention utilizes datasets from image captioning and video captioning tasks to supervise the fine-tuning of a multimodal large language model, enabling the model to understand the alignment relationships of image-text pairs and frame image sequence-text pairs, and effectively alleviating the problem of insufficient dense video data.

[0045] 2. This invention combines dense video detection tasks with a multimodal large language model, effectively utilizing the visual understanding capabilities of the multimodal large language model, and demonstrating the feasibility of integrating complex tasks such as dense video detection into a large-scale pre-trained model. Attached Figure Description

[0046] Figure 1 This is a flowchart of a dense video detection method based on a large-scale language model according to the present invention;

[0047] Figure 2 This is a schematic diagram of the established multimodal large language model;

[0048] Figure 3 This is a schematic diagram of the input paradigm of a multimodal large language model. Detailed Implementation

[0049] like Figure 1 As shown, a dense video detection method based on a large-scale language model according to the present invention includes the following steps:

[0050] Step 1: Collect existing image caption datasets and construct image-text pairs; collect existing video caption datasets and construct frame image sequence-text pairs. Specifically, Step 1 involves:

[0051] Step 1.1: Collect publicly available datasets for image captioning tasks and construct image-text pairs;

[0052] Step 1.2: Collect video text pairs for the video captioning task, uniformly sample the video to obtain a frame image sequence, and construct frame image sequence-text pairs.

[0053] This invention employs a multimodal large language model based on a large-scale language model. Due to the large number of parameters and complex structure of language models, supervised fine-tuning with a large amount of multimodal data is usually required to achieve satisfactory multimodal alignment results. However, the amount of dense video detection data is insufficient, making it difficult to train the multimodal large language model effectively.

[0054] The dense video detection task comprises two sub-tasks: event localization and video segment recognition. The event localization task can be viewed as the model needing to understand the visual content within a sequence of frame images, while the video segment recognition task can be viewed as the model needing to possess multimodal alignment capabilities. Therefore, the detection method of this invention first uses a video caption dataset to enhance the multimodal large language model's ability to understand frame image sequences, and then uses an image caption dataset to enhance the multimodal large language model's multimodal alignment capabilities.

[0055] Step 2: Build a multimodal large language model framework and perform distributed training using the DeepSpeed ​​tool. Fine-tune the multimodal large language model using image-text pairs and frame image sequence-text pairs. Specifically, Step 2 involves:

[0056] Step 2.1: Build a multimodal large language model framework, including an image encoder, a multimodal adapter, and a large-scale language model, such as... Figure 2 As shown.

[0057] In specific implementation, in step 2.1, the image encoder uses the ViT model from the pre-trained CLIP model, the multimodal adapter uses the MLP model, and the large-scale language model uses the Llama2-7B model. The image is encoded in a single modality by the ViT model, and then the image features are mapped to the text representation space of the LLM by the multimodal adapter. The ViT model is frozen during training, and multimodal alignment is performed between the multimodal adapter and the LLM.

[0058] Step 2.2: Build a distributed training environment using the DeepSpeed ​​tool, and use Zero-3 data parallelism for training;

[0059] Multimodal large language models have a large number of parameters, making training difficult and requiring advanced equipment. Since fine-tuning of all parameters is necessary during training, the detection method in this invention uses DeepSpeed ​​for distributed training to reduce memory pressure and employs Zero-3 data parallelism for training.

[0060] Step 2.3: Freeze the image encoder parameters and fine-tune the multimodal adapter and large language model using full parameters of image-text pairs and frame image sequence-text pairs.

[0061] Step 3: Collect existing dense video datasets, uniformly sample the dense videos to obtain frame image sequences, and pad and truncate the events in the dense videos. Specifically, Step 3 involves:

[0062] Step 3.1: Collect public datasets for dense video detection tasks, uniformly sample dense videos to obtain frame image sequences, and align the event localization time with the frame sequence.

[0063] In specific implementation, step 3.1 involves uniformly sampling 100 frames of images from the video and deleting videos with fewer than 100 frames.

[0064] Step 3.2: Count the number of events in each video in the dense video dataset, select the maximum number of events N, and fill the number of events in videos with less than N with {0,0}, where {0,0} means that the start frame number and end frame number of the filled event are both 0. Perform event truncation operation on videos with more than the maximum number of events N, and delete the redundant events.

[0065] In practice, the maximum number of events N is chosen based on the percentage of videos in the dense video dataset with an event count less than or equal to N, rounded down to 90%. The difficulty of the event localization task lies in predicting the start and end times of events, which requires converting the prediction of the start and end times of events into the prediction of the start and end frame numbers of events.

[0066] Step 4: Construct the model input paradigm, including event query prefixes, frame image sequence features, and text sequence features. Supervised fine-tuning of the multimodal large language model fine-tuned in Step 2 is then performed to achieve dense video detection. Specifically, Step 4 involves:

[0067] Step 4.1: At the start of training, the frame image sequence is processed by an image encoder to obtain frame image sequence features; the text corresponding to each event is processed by the embedding layer of a large-scale language model to obtain text sequence features.

[0068] Step 4.2: Randomly initialize the event query prefix. The event query prefix is ​​represented by a set of learnable vectors. The event query prefix is ​​concatenated with the frame image sequence features and text sequence features obtained in Step 4.1 to form the model input paradigm. The input paradigm is then input into the multimodal large language model.

[0069] In specific implementation, in the model input paradigm of step 4.2, the frame image sequence is segmented using the special character [SEP]; the text sequence is segmented using [event]. For example... Figure 3 The diagram shown is a schematic of the input paradigm.

[0070] Step 4.3: Predict the start and end frame numbers of events one-to-one using learnable event query prefixes, transforming event localization into an ensemble prediction task, and calculate the loss for model training.

[0071] In practice, the learnable event query prefix is ​​randomly initialized before training. After the model outputs, it integrates multimodal information from frame image sequences and text sequences. After passing through the event localization detection head, it is mapped one-to-one to the start and end frame numbers of the event to obtain the predicted event. The text sequence is then processed by the model to obtain the predicted text sequence.

[0072] The expression for the Loss function of the model in step 4.3 is as follows:

[0073] Loss = event ce +event giou +caption ce

[0074] Where Loss represents the model's loss function, and event... ce The cross-entropy loss represents the predicted start and end frame numbers of the event compared to the actual start and end frame numbers; event giou The GIOU loss represents the difference between the predicted and actual event ranges; caption ce The cross-entropy loss represents the difference between the predicted and actual text sequences; the actual start and end frame numbers of the events in the video are represented as... The actual text sequence in the video is represented as [x1, x2…x n ],in 1≤n≤N, where C represents the number of images in the frame image sequence, N represents the number of events in the video, s represents the start frame, and e represents the end frame; the length of the event query prefix feature is 2N. After passing through the large language model, it is mapped to the frame representation space by the event localization header to obtain the predicted probability corresponding to frame C+1. Where 0 ≤ k ≤ C.

[0075] The difference between the predicted start and end frame numbers of the event and the actual start and end frame numbers of the event is calculated using the cross-entropy loss formula:

[0076]

[0077] Among them, u k This represents the one-hot representation of frame number i; This represents the probability that event n starts at frame number k. This represents the probability that the termination frame number of event n is k.

[0078] The predicted event range and the actual event range are constrained using the following formula:

[0079]

[0080] Where A represents the predicted range of events, B represents the actual range of events, and D represents the minimum range that can contain both A and B;

[0081] The difference between the predicted text sequence and the true text sequence is calculated using the cross-entropy loss formula:

[0082]

[0083] Where L represents the text length, p(y nh |x n ,y n Let x represent the probability of the caption for event n when predicting the h-th word. n It is real text, y n It is the predicted text, y nh It is the h-th word of the predicted text, y n,＜h It refers to the first h-1 words of the predicted text.

[0084] This invention's detection method constructs a paradigm combining dense video detection and a large language model within the framework of a multimodal large language model. This addresses the weakness of traditional dense video detection models in understanding visual information, providing a new solution for dense video detection. Furthermore, during the training of the multimodal large language model, supervised fine-tuning is performed using publicly available data from both image and video captioning tasks through transfer learning. The model can learn not only the interaction information between images and text but also the interaction relationships between image sequences and text in videos, effectively alleviating the problem of insufficient data for dense video detection tasks. This demonstrates the feasibility of integrating complex tasks such as dense video detection into large-scale pre-trained models.

[0085] The above description is only a preferred embodiment of the present invention and is not intended to limit the ideas of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for dense video detection based on a large-scale language model, characterized in that, include: Step 1: Collect existing image caption datasets and construct image-text pairs; Collect existing video caption datasets and construct frame image sequence-text pairs; Step 2: Build a multimodal large language model framework and perform distributed training using the DeepSpeed ​​tool, and fine-tune the multimodal large language model using image-text pairs and frame image sequence-text pairs; Step 3: Collect existing dense video datasets, uniformly sample the dense videos to obtain frame image sequences, and pad and truncate the events in the dense videos; Step 4: Construct the model input paradigm, including event query prefixes, frame image sequence features, and text sequence features, and perform supervised fine-tuning of the multimodal large language model after fine-tuning in Step 2 to achieve dense video detection; Step 4 specifically involves: Step 4.1: At the start of training, the frame image sequence is processed by an image encoder to obtain frame image sequence features; the text corresponding to each event is processed by the embedding layer of a large-scale language model to obtain text sequence features; Step 4.2: Randomly initialize the event query prefix. The event query prefix is ​​represented by a set of learnable vectors. The event query prefix is ​​concatenated with the frame image sequence features and text sequence features obtained in Step 4.1 to form the model input paradigm. The input paradigm is then input into the multimodal large language model. Step 4.3: Predict the start and end frame numbers of events one-to-one using the learnable event query prefix, transforming event localization into an ensemble prediction task, and calculate the model training loss; the expression for the model's loss function is: Where Loss represents the model's loss function. This represents the cross-entropy loss between the predicted start and end frame numbers of the event and the actual start and end frame numbers of the event. The GIOU loss represents the difference between the predicted event range and the actual event range. The cross-entropy loss represents the difference between the predicted and actual text sequences. The actual start and end frame numbers of the events in the video are represented as follows: The actual text sequence in the video is represented as ,in , , C represents the number of images in the frame image sequence, N represents the number of events in the video, s represents the start frame, and e represents the end frame. The length of the event query prefix feature is 2N. After passing through the large language model, it is mapped to the frame representation space by the event localization header to obtain the predicted probability corresponding to frame C+1. ,in ; The difference between the predicted start and end frame numbers of the event and the actual start and end frame numbers of the event is calculated using the cross-entropy loss formula: wherein, onehot(i) denotes the onehot representation of frame number i; P(k | n) denotes the probability that the start frame number of event n is k, P(k | n) denotes the probability that the end frame number of event n is k, The predicted event range and the actual event range are constrained using the following formula: Where A represents the predicted range of events, B represents the actual range of events, and D represents the minimum range that can contain both A and B; The difference between the predicted text sequence and the true text sequence is calculated using the cross-entropy loss formula: Where L represents the text length. Let x represent the probability of the caption for event n when predicting the h-th word. n It is real text, y n It is the predicted text, y nh It is the h-th word of the predicted text. It is the first h-1 words of the predicted text.

2. The dense video detection method based on a large-scale language model of claim 1, wherein, Step 1 specifically involves: Step 1.1: Collect publicly available datasets for image captioning tasks and construct image-text pairs; Step 1.2: Collect video text pairs for the video captioning task, uniformly sample the video to obtain a frame image sequence, and construct frame image sequence-text pairs.

3. The dense video detection method based on a large-scale language model of claim 1, wherein, Step 2 specifically involves: Step 2.1: Build a multimodal large language model framework, including an image encoder, a multimodal adapter, and a large-scale language model; Step 2.2: Build a distributed training environment using the DeepSpeed ​​tool, and use Zero-3 data parallelism for training; Step 2.3: Freeze the image encoder parameters and fine-tune the multimodal adapter and large language model using full parameters of image-text pairs and frame image sequence-text pairs.

4. The dense video detection method based on a large-scale language model of claim 3, wherein, In step 2.1, the image encoder uses the ViT model from the pre-trained CLIP model, the multimodal adapter uses the MLP model, and the large-scale language model uses the Llama2-7B model. The image is encoded in a single modality by the ViT model, and then the image features are mapped to the text representation space of the LLM by the multimodal adapter. The ViT model is frozen during training, and multimodal alignment is performed between the multimodal adapter and the LLM.

5. The dense video detection method based on a large-scale language model of claim 1, wherein, Step 3 specifically involves: Step 3.1: Collect public datasets for dense video detection tasks, uniformly sample dense videos to obtain frame image sequences, and align the event localization time with the frame sequences; Step 3.2: Count the number of events in each video in the dense video dataset, select the maximum number of events N, and fill the number of events in videos with less than N with {0,0}, where {0,0} means that the start frame number and end frame number of the filled event are both 0. Perform event truncation operation on videos with more than the maximum number of events N, and delete the redundant events.

6. The dense video detection method based on a large-scale language model of claim 5, wherein, In step 3.1, 100 frames of images are uniformly sampled from the video, and videos with fewer than 100 frames are deleted.

7. The dense video detection method based on a large-scale language model of claim 5, wherein, The selection of the maximum number of events N in step 3.2 is based on the proportion of videos in the dense video dataset with an event number less than or equal to N, rounded down to 90%. The difficulty of the event localization task lies in the prediction of the event start time and end time, which requires converting the prediction of the event start time and end time into the prediction of the event start frame number and end frame number.

8. The dense video detection method based on a large-scale language model as described in claim 1, characterized in that, In step 4.2, the frame image sequence in the model input paradigm is segmented using the special character [SEP]; the text sequence is segmented using the special character [event].

Images