A video understanding method and system based on large language model

Abstract

The present invention relates to the field of video processing technology, and more specifically to a large language model-based video understanding method and system. The method comprises obtaining video question-and-answer examples to form a video question-and-answer dataset, preprocessing the video question-and-answer dataset to obtain vector representation features of the video question-and-answer dataset, reorganizing an existing text-image model into a video feature extraction model, processing the vector representation features using the video feature extraction model, obtaining vector features of video analysis results, and analyzing the vector features of the video analysis results using a large language model to obtain answers to the video question-and-answer examples. The present invention performs video understanding question-and-answering through data preprocessing, model training, and a frozen large language model, thereby improving answer accuracy and saving training computing power and data resources.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video understanding method and system based on a large language model. Background Art

[0002] With advances in multimedia technology, the generation and consumption of video content has exploded, making the development of video understanding technology increasingly crucial. Video understanding technology can analyze and identify key information and behavioral patterns within video content, improving the user viewing experience while also enhancing the efficiency of content retrieval and management. In areas such as security monitoring, traffic management, and medical analysis, the application of video understanding technology can promptly detect and prevent potential problems.

[0003] In the existing technology, video understanding question-answering systems mainly fall into two categories: small-scale multimodal models and those based on large language models. Small-scale multimodal models use pre-trained encoders trained on small datasets to solve specialized video understanding problems. However, video question-answering systems based on small-scale multimodal models lack general and extensive language knowledge and have poor text comprehension capabilities. Their performance and generalization are poor, especially in open-ended scenarios. Video understanding systems based on large models introduce a video encoder before the input of a frozen large language model to complete open-ended video question-answering. Leveraging the large number of parameters and the deep language and knowledge comprehension capabilities accumulated from massive training data, these systems have achieved remarkable results in the field of video understanding. However, video understanding systems based on large language models currently face the significant demand for video data and computing resources during the pre-training phase. Furthermore, the scale and scene coverage of video data are generally more limited than those of image data, which can exacerbate the discrepancy between video pre-training datasets and downstream task scenarios, thereby impacting transferability. Therefore, developing a computationally and data-efficient large language model video understanding system is particularly important. Summary of the Invention

[0004] In order to make up for the shortcomings of the existing technology, the present invention proposes an efficient video understanding method and system based on the reuse of large model spatiotemporal parameters, which can migrate the original text-image large model to the video understanding field, thereby efficiently applying the large language model in the video understanding system and efficiently completing video training, saving a lot of video resources and computing power.

[0005] To achieve the above objectives, in a first aspect, the present invention adopts a technical solution: an efficient video understanding method based on large model spatiotemporal parameter reuse, comprising:

[0006] S1: Obtain video question answering examples to form a video question answering dataset;

[0007] S2: Preprocess the video question answering dataset to obtain vector representation features of the video question answering dataset;

[0008] S3: Reorganize the existing text-image model into a video feature extraction model, use the video feature extraction model to process the vector representation features, and obtain the vector features of the video analysis results;

[0009] S4: Use the large language model to analyze the vector features of the video analysis results and obtain the answer to the corresponding video question and answer example.

[0010] Preferably, the method of preprocessing the video question-answering dataset in S2 includes: filtering and segmenting sentences in the text portion of the video question-answering dataset using a word segmenter of a Bert model, removing special symbols, and obtaining processed text;

[0011] Preferably, the method of preprocessing the video question and answer dataset in S2 includes: using the ffmpeg video processing program to extract key frames of the video on a time-series average basis for the video part of the video question and answer dataset to obtain corresponding pictures.

[0012] Preferably, the method of preprocessing the video question answering dataset in S2 includes: using bert-tokenizer and ViT-G / 14 to convert the obtained processed text and corresponding pictures into vector representations, and obtaining the vector representation features of the corresponding video question answering dataset.

[0013] Preferably, S3 includes the following steps:

[0014] S31: After each spatial self-attention network of the text-image model, a temporal self-attention network with the same structure as the spatial self-attention network is initialized, and the parameters of the spatial self-attention network in the text-image model are reused into the parameters of the self-attention network on the video temporal sequence;

[0015] S32: Temporal and spatial modeling are alternately stacked and repeated several times to achieve efficient fusion of text and video multimodal information. After parameter reuse, the basic unit stacking of the original text-image large model, spatial self-attention network-spatial cross-attention network, is transformed into a basic unit stacking of temporal self-attention network-spatial self-attention network-spatial cross-attention network.

[0016] S33: When processing data streams, a channel transformation strategy is used to merge the batch and temporal channels of the video vector when entering the spatial attention network, and merge the batch and spatial length channels of the video vector when entering the temporal attention network to complete the temporal and spatial interactions in the image vector and obtain the vector features representing the video analysis results;

[0017] S34: Train the reorganized video feature extraction model. During training, add a low-rank matrix with the same input and output dimensions as the original parameter matrix to each parameter of the attention network. Keep the original model parameters frozen during the training phase and only adjust the low-rank matrices.

[0018] In a second aspect, the present invention provides an efficient video understanding system based on large-model spatiotemporal parameter reuse, the system comprising:

[0019] The acquisition unit is used to acquire video question-and-answer examples to form a video question-and-answer dataset. The acquisition unit can be a client, through which the user uploads the video to be analyzed and the video comprehension questions to the server. When in use, the user's uploaded video and question text, as well as the user's voice and speaker information, are collected and uploaded to the server.

[0020] A dataset preprocessing module, configured to preprocess the video question answering dataset to obtain vector representation features of the video question answering dataset;

[0021] Video feature extraction module: used to reorganize the existing text-image model into a video feature extraction model, use the video feature extraction model to process vector representation features, and obtain vector features of video analysis results;

[0022] The large language analysis module is used to analyze the vector features of the video analysis results and obtain the answers to the corresponding video question and answer examples.

[0023] In a third aspect, the present invention provides an efficient video understanding device based on large-scale model spatiotemporal parameter reuse, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented:

[0024] Obtain video question-answering examples to form a video question-answering dataset;

[0025] Preprocessing the video question answering dataset to obtain vector representation features of the video question answering dataset;

[0026] Reorganize the existing text-image model into a video feature extraction model, use the video feature extraction model to process vector representation features, and obtain vector features of video analysis results;

[0027] Use a large language model to analyze the vector features of the video analysis results and obtain the answers to the corresponding video question and answer examples.

[0028] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0029] Obtain video question-answering examples to form a video question-answering dataset;

[0030] Preprocessing the video question answering dataset to obtain vector representation features of the video question answering dataset;

[0031] Reorganize the existing text-image model into a video feature extraction model, use the video feature extraction model to process vector representation features, and obtain vector features of video analysis results;

[0032] Use a large language model to analyze the vector features of the video analysis results and obtain the answers to the corresponding video question and answer examples.

[0033] The beneficial effects of the present invention are as follows:

[0034] Video understanding question and answering is performed through data preprocessing, model training and frozen large language models, which can improve the accuracy of answers and save training computing power and data resources. Video understanding systems based on small-scale neural networks lack large-scale natural language knowledge and have low accuracy. Existing video understanding systems based on large-scale language models have high training costs and large data requirements. Compared with other large video understanding models, the present invention reuses the parameters of the text-image large model and can quickly complete time series modeling, thereby saving data and computing power resources for time series modeling. Moreover, the reused text-image large model covers a wide range of image scenes and has good visual text alignment effect, thereby improving the scene coverage and multimodal fusion effect in video understanding, and further improving the generalization during fine-tuning of downstream tasks. In addition, the spatial network and temporal network cyclic stacking structure of the present invention can better complete the efficient fusion of text and video multimodal information at different levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0036] Figure 1 is a flow chart of the method of the present invention;

[0037] Figure 2 is a flow chart of the system of the present invention;

[0038] Figure 3 Schematic diagram of the extraction network in the video feature extraction model of the present invention;

[0039] Figure 4 It is a data flow diagram of the network extraction in the video feature extraction model of the present invention;

[0040] Figure 5 This is an explanatory diagram of the low-rank matrix fine-tuning parameters in the present invention. DETAILED DESCRIPTION

[0041] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0042] like Figures 1 to 5 As shown, the present invention provides a video understanding method based on a large language model, including:

[0043] S1: Obtain video question answering examples to form a video question answering dataset;

[0044] S2: Preprocess the video question answering dataset to obtain vector representation features of the video question answering dataset;

[0045] S3: reorganizing the existing text-image model into a video feature extraction model, using the video feature extraction model to process vector representation features to obtain vector features of video analysis results;

[0046] S4: Use the large language model to analyze the vector features of the video analysis results and obtain the answer to the corresponding video question and answer example.

[0047] Specifically, the method of preprocessing the video question answering dataset in S2 includes: filtering and segmenting sentences in the text portion of the video question answering dataset using a word segmenter of the Bert model, removing special symbols, and obtaining processed text;

[0048] Specifically, the method of preprocessing the video question and answer dataset in S2 includes: using the ffmpeg video processing program to extract key frames of the video on a time-series average basis for the video part of the video question and answer dataset to obtain corresponding pictures.

[0049] Specifically, the method of preprocessing the video question answering dataset in S2 includes: using bert-tokenizer and ViT-G / 14 to convert the obtained processed text and corresponding pictures into vector representations, and obtaining the vector representation features of the corresponding video question answering dataset.

[0050] Specifically, S3 includes the following steps:

[0051] S31: After each spatial self-attention network of the text-image model, a temporal self-attention network with the same structure as the spatial self-attention network is initialized, and the spatial self-attention network parameters in the text-image model are reused into the self-attention network parameters on the video temporal sequence to achieve efficient fine-tuning.

[0052] S32: Time and space modeling are alternately stacked and repeated several times to achieve efficient fusion of text and video multimodal information. After parameter reuse, the basic unit stacking of spatial self-attention network-spatial cross-attention network of the original text-image large model is transformed into the basic unit stacking of temporal self-attention network-spatial self-attention network-spatial cross-attention network several times, which can achieve efficient fusion of text and video multimodal information.

[0053] S33: When processing data streams, use Figure 3-4 The channel transformation strategy shown in the figure merges the batch and temporal channels of the video vector when entering the spatial attention network, and merges the batch and spatial length channels of the video vector when entering the temporal attention network to complete the temporal interaction and spatial interaction in the image vector, and obtain the vector features representing the video analysis results;

[0054] S34: Train the reorganized video feature extraction model. During training, add a low-rank matrix with the same input and output dimensions as the original parameter matrix to each parameter of the attention network. Keep the original model parameters frozen during the training phase and only adjust each low-rank matrix. This can greatly reduce the learnable parameters of the large model.

[0055] Specifically, during the training process, we train the video feature extraction model reorganized above. The training data is the data collected in S1. By freezing the parameters of the large language model, spatial self-attention network, temporal self-attention network and spatial cross-attention network, a large number of training parameters are saved. Then, low-rank matrix parameters are added to the attention query layer and attention value layer of each attention network and fine-tuned. The role of low-rank matrix parameters is shown in the figure below. Figure 3 As shown;

[0056] In terms of data flow, such as Figure 1-2 As shown, the present invention first uses a text encoder and an image encoder to obtain text and image features. On this basis, the model query vector and the text representation vector are first input into the spatial self-attention network for spatial interaction, and then Figure 4 The channel transformation in the spatial attention network merges the batch and temporal channels of the video vector, and merges the batch and spatial length channels of the video vector when entering the temporal attention network to complete the temporal interaction and spatial interaction in the image vector. The present invention uses the temporal spatial attention network to perform temporal attention interaction in the model query vector, and finally accepts the visual representation vector of the video frame and the model query vector in the spatial cross attention network, completing the image feature interaction in space through channel transformation.

[0057] During training, the parameter reuse strategy employed by this invention allows for direct fine-tuning training on downstream tasks, bypassing the video pre-training phase. This strategy reduces the number of adjustable parameters, significantly conserving resources for fine-tuning training. Furthermore, existing methods' video pre-training phases can reduce generalization to downstream tasks due to issues such as limited video data and poor alignment with text. This invention, by using image models to complete video tasks, improves transferability across downstream task scenarios and problems.

[0058] In a second aspect, the present invention provides an efficient video understanding system based on large-model spatiotemporal parameter reuse, the system comprising:

[0059] The acquisition unit is used to obtain video question-and-answer examples to form a video question-and-answer dataset. The acquisition unit can be a client, through which users upload the video to be analyzed and questions about video comprehension to the server. During use, the user's uploaded video and question text, as well as the user's voice and speaker information, are collected and uploaded to the server. Simultaneously, the client receives the server's answers to the video questions and provides them to the user in text form. Before model training, users can upload related videos and text question-and-answer pairs through the client to help the server collect data for training.

[0060] A dataset preprocessing module, configured to preprocess the video question answering dataset to obtain vector representation features of the video question answering dataset;

[0061] Video feature extraction module: used to reorganize the existing text-image model into a video feature extraction model, use the video feature extraction model to process vector representation features, and obtain vector features of video analysis results;

[0062] The large language analysis module is used to analyze the vector features of the video analysis results and obtain the answers to the corresponding video question and answer examples.

[0063] In a third aspect, the present invention provides an efficient video understanding device based on large-scale model spatiotemporal parameter reuse, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented:

[0064] Obtain video question-answering examples to form a video question-answering dataset;

[0065] Preprocessing the video question answering dataset to obtain vector representation features of the video question answering dataset;

[0066] Reorganize the existing text-image model into a video feature extraction model, use the video feature extraction model to process vector representation features, and obtain vector features of video analysis results;

[0067] Use a large language model to analyze the vector features of the video analysis results and obtain the answers to the corresponding video question and answer examples.

[0068] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0069] Obtain video question-answering examples to form a video question-answering dataset;

[0070] Preprocessing the video question answering dataset to obtain vector representation features of the video question answering dataset;

[0071] Reorganize the existing text-image model into a video feature extraction model, use the video feature extraction model to process vector representation features, and obtain vector features of video analysis results;

[0072] Use a large language model to analyze the vector features of the video analysis results and obtain the answers to the corresponding video question and answer examples.

[0073] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An efficient video understanding method based on large model spatiotemporal parameter reuse, characterized in that: The method is applied to an efficient video understanding system based on large-model spatiotemporal parameter reuse, and the method comprises: S1: Obtain video question answering examples to form a video question answering dataset; S2: Preprocess the video question answering dataset to obtain vector representation features of the video question answering dataset; S3: Reorganize the existing text-image model into a video feature extraction model, use the video feature extraction model to process the vector representation features, and obtain the vector features of the video analysis results; wherein S3 includes the following steps: S31: Based on each spatial self-attention network of the text-image model, initialize a temporal self-attention network with the same structure as the spatial self-attention network, and reuse the spatial self-attention network parameters in the text-image model with the self-attention network parameters on the video temporal sequence; S32: Temporal and spatial modeling are alternately stacked and repeated several times to achieve efficient fusion of text and video multimodal information. After parameter reuse, the basic unit stacking of the original text-image large model, spatial self-attention network-spatial cross-attention network, is transformed into a basic unit stacking of temporal self-attention network-spatial self-attention network-spatial cross-attention network. S33: When processing data streams, a channel transformation strategy is used to merge the batch and temporal channels of the video vector when entering the spatial attention network, and merge the batch and spatial length channels of the video vector when entering the temporal attention network to complete the temporal and spatial interactions in the image vector and obtain the vector features representing the video analysis results; S34: training the reorganized video feature extraction model. During training, a low-rank matrix with the same input and output dimensions as the original parameter matrix is ​​added to each parameter of the attention network. During the training phase, the original model parameters are frozen, and only the low-rank matrices are adjusted. S4: Use the large language model to analyze the vector features of the video analysis results and obtain the answer to the corresponding video question and answer example.

2. The efficient video understanding method based on large model spatiotemporal parameter reuse according to claim 1, characterized in that: The method for preprocessing the video question and answer dataset in S2 includes: for the text part of the video question and answer dataset, using the Bert model word segmenter to filter and segment the sentences, removing special symbols, and obtaining processed text.

3. The efficient video understanding method based on large model spatiotemporal parameter reuse as claimed in claim 2, characterized in that: The method of preprocessing the video question answering dataset in S2 includes: using the ffmpeg video processing program to extract key frames of the video on a time-series average basis for the video part of the video question answering dataset to obtain corresponding pictures.

4. The efficient video understanding method based on large model spatiotemporal parameter reuse as claimed in claim 3, characterized in that: The method for preprocessing the video question answering dataset in S2 includes: using bert-tokenizer and ViT-G / 14 to convert the obtained processed text and corresponding pictures into vector representations, and obtaining the vector representation features of the corresponding video question answering dataset.

5. An efficient video understanding system based on large-scale model spatiotemporal parameter reuse, characterized by: The system includes: An acquisition unit, used to acquire video question-answering examples to form a video question-answering dataset; A dataset preprocessing module, configured to preprocess the video question answering dataset to obtain vector representation features of the video question answering dataset; Video feature extraction module: used to reorganize the existing text-image model into a video feature extraction model, use the video feature extraction model to process vector representation features, and obtain vector features of video analysis results; wherein, the video feature extraction module, when used, includes the following sub-steps: According to each spatial self-attention network of the text-image model, a temporal self-attention network with the same structure as the spatial self-attention network is initialized, and the spatial self-attention network parameters in the text-image model are reused into the self-attention network parameters on the video temporal sequence. The temporal and spatial modeling are alternately stacked and repeated several times to achieve efficient fusion of text and video multimodal information. After parameter reuse, the basic unit stacking of the spatial self-attention network-spatial cross-attention network of the original text-image model is transformed into the basic unit stacking of the temporal self-attention network-spatial self-attention network-spatial cross-attention network. When processing the data stream, a channel transformation strategy is used to merge the batch and temporal channel of the video vector when entering the spatial attention network, and merge the batch and spatial length channel of the video vector when entering the temporal attention network to complete the temporal interaction and spatial interaction in the image vector, and obtain the video analysis result representation vector feature. The reorganized video feature extraction model is trained. During training, a low-rank matrix with the same input and output dimensions as the original parameter matrix is ​​added to each parameter of the attention network. During the training phase, the original model parameters are kept frozen and only the low-rank matrices are adjusted. The large language analysis module is used to analyze the vector features of the video analysis results and obtain the answers to the corresponding video question and answer examples.

6. An efficient video understanding device based on large-scale model spatiotemporal parameter reuse, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the following steps are implemented: Obtain video question-answering examples to form a video question-answering dataset; Preprocessing the video question answering dataset to obtain vector representation features of the video question answering dataset; The existing text-image model is reorganized into a video feature extraction model, and the video feature extraction model is used to process the vector representation features to obtain the vector features of the video analysis results. The implementation of this step includes the following sub-steps: According to each spatial self-attention network of the text-image model, a temporal self-attention network with the same structure as the spatial self-attention network is initialized, and the spatial self-attention network parameters in the text-image model are reused into the self-attention network parameters on the video temporal sequence. The temporal and spatial modeling are alternately stacked and repeated several times to achieve efficient fusion of text and video multimodal information. After parameter reuse, the basic unit stacking of the spatial self-attention network-spatial cross-attention network of the original text-image model is transformed into the basic unit stacking of the temporal self-attention network-spatial self-attention network-spatial cross-attention network. When processing the data stream, a channel transformation strategy is used to merge the batch and temporal channel of the video vector when entering the spatial attention network, and merge the batch and spatial length channel of the video vector when entering the temporal attention network to complete the temporal interaction and spatial interaction in the image vector, and obtain the video analysis result representation vector feature. The reorganized video feature extraction model is trained. During training, a low-rank matrix with the same input and output dimensions as the original parameter matrix is ​​added to each parameter of the attention network. During the training phase, the original model parameters are kept frozen and only the low-rank matrices are adjusted. Use a large language model to analyze the vector features of the video analysis results and obtain the answers to the corresponding video question and answer examples.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the following steps are performed: Obtain video question-answering examples to form a video question-answering dataset; Preprocessing the video question answering dataset to obtain vector representation features of the video question answering dataset; The existing text-image model is reorganized into a video feature extraction model, and the video feature extraction model is used to process the vector representation features to obtain the vector features of the video analysis results. The implementation of this step includes the following sub-steps: According to each spatial self-attention network of the text-image model, a temporal self-attention network with the same structure as the spatial self-attention network is initialized, and the spatial self-attention network parameters in the text-image model are reused into the self-attention network parameters on the video temporal sequence. The temporal and spatial modeling are alternately stacked and repeated several times to achieve efficient fusion of text and video multimodal information. After parameter reuse, the basic unit stacking of the spatial self-attention network-spatial cross-attention network of the original text-image model is transformed into the basic unit stacking of the temporal self-attention network-spatial self-attention network-spatial cross-attention network. When processing the data stream, a channel transformation strategy is used to merge the batch and temporal channel of the video vector when entering the spatial attention network, and merge the batch and spatial length channel of the video vector when entering the temporal attention network to complete the temporal interaction and spatial interaction in the image vector, and obtain the video analysis result representation vector feature. The reorganized video feature extraction model is trained. During training, a low-rank matrix with the same input and output dimensions as the original parameter matrix is ​​added to each parameter of the attention network. During the training phase, the original model parameters are kept frozen and only the low-rank matrices are adjusted. Use a large language model to analyze the vector features of the video analysis results and obtain the answers to the corresponding video question and answer examples.

Images