Video understanding large model optimization and evaluation method, system and device and storage medium

Abstract

The invention discloses a video understanding large model optimization and evaluation method, system and device and a storage medium, which are in one-to-one correspondence, and in the scheme, a new connector structure is designed to improve the global time sequence understanding capability, and the new connector structure comprises a space-time downsampler, a local bidirectional Mama structure and a linear layer. The time-space downsampler can reduce the token storage overhead; meanwhile, the local bidirectional Mama structure makes up for the problem that the receptive field is limited on one hand, and on the other hand can model intra-frame features and inter-frame features at the same time; in addition, training of the connector is low in cost, a three-stage progressive training strategy is used, and catastrophic forgetting can be resisted. Moreover, a semi-automatic data generation pipeline is constructed, and global time sequence understanding data is provided based on the pipeline, so that the evaluation vacancy of the existing reference field in the capability is made up.

Description

Technical Field

[0001] The present invention relates to the technical field of video understanding large model optimization, and in particular to a video understanding large model optimization and evaluation method, system, device and storage medium. Background Art

[0002] The video understanding big model is currently a key research direction in the field of video understanding. It mainly consists of three parts: visual encoder, connector and big language model. The core idea of ​​this design is to map video frames to language feature space to achieve extensive understanding with the power of big language model. After a video is input, the model first uniformly samples a fixed number of frames. Then, the visual encoder extracts the features of these frames. In order to adapt to different sampling strategies, the visual encoder is often an image encoder, that is, the visual branch of CLIP (Contrastive Language–Image Pre-training). Then, the connector maps all visual features to the language feature space. In addition, the connector also plays the role of modeling the temporal relationship between frames. There are three common connector structures, namely Q-Former (Transformer structure, Transformer is a neural network based on attention mechanism), multi-layer perceptron and convolutional structure. Q-Former uses a small number of query tokens to capture the temporal information and visual information in the video. Its advantage is that the receptive field is global and the number of tokens after mapping is small, but its disadvantage is that it cannot maintain the token order. Multi-layer perceptron is the opposite. It maintains the token order but brings serious token overhead. The convolution structure is a compromise between the two, which is usually sandwiched by two layers of two-dimensional convolution and a single layer of three-dimensional convolution. In addition, Q-Former and convolution structures also require additional linear layers to assist feature mapping. The large language model directly uses the Vicuna series or the Mistral series. Vicuna is an open source large language model based on LLaMA (large language model), and Mistral is an efficient open source large language model developed by Mistral AI.

[0003] In order to evaluate the capabilities of large models for video understanding, benchmark datasets have also received strong attention. Existing datasets are mainly used to evaluate general video understanding capabilities, that is, within one benchmark, different categories of evaluation are included. Based on the type of understanding, there are actions, targets, locations, scenes, quantities, attributes, postures, roles, cognition, etc., and each category can be further divided into subcategories. Based on the length of the video, there are short videos, medium videos, and long videos. Based on the video category, it can be divided into movies and TV, science, sports, art, life, etc. In terms of evaluation format, existing datasets are mainly closed-ended questions and answers or open-ended questions and answers, which are relatively in line with the capabilities of large language models.

[0004] However, existing benchmarks focus on video-level understanding or local event-level understanding and rarely pay attention to global event-level understanding. Global event-level understanding is defined as global temporal understanding, which can be specifically interpreted as the ability of the model to understand all key events in the video. This ability relies on temporal modeling and event recognition, is a basic ability, and is also the key to stimulating the general ability of the model. In addition, closed-ended question answering has the problem of shortcut learning, that is, the model can execute exclusion strategies through language clues and common sense clues, which makes it difficult to truly measure the ability of the model. Open-ended question answering often relies on large commercial models such as GPT (Generative Pre-trained Transformer) for evaluation, and its reliability is questionable.

[0005] In view of this, the present invention is proposed. Summary of the invention

[0006] The purpose of the present invention is to provide a video understanding large model optimization and evaluation method, system, device and storage medium, which can improve the global temporal understanding ability of the video understanding large model, and at the same time construct evaluation benchmark data for global temporal understanding, which can better evaluate the fine-grained understanding ability of the video understanding large model for the entire video.

[0007] The objective of the present invention is achieved through the following technical solutions: A video understanding large model optimization and evaluation method, comprising: Construct a connector including a spatiotemporal downsampler, a local bidirectional Mamba structure and a linear layer connected in sequence, and form a large video understanding model with a visual encoder and a large language model; wherein a residual structure is also used between the spatiotemporal downsampler and the local bidirectional Mamba structure, and Mamba is a selective state space model; The video understanding large model is trained progressively in three stages; in the first stage, the local bidirectional Mamba structure is trained using video-text pairs and image-text pairs in the form of descriptions; in the second stage, the entire connector is trained using video-text pairs in the form of descriptions; in the third stage, the connector and the large language model are trained using video-text pairs in the form of multiple choices and descriptions; Use a semi-automated data generation pipeline to build a global time series understanding dataset; A specific instruction fine-tuning training set is constructed using a global temporal understanding dataset, and the large video understanding model after three-stage progressive training is fine-tuned. The global temporal understanding dataset is then used to evaluate the performance of the fine-tuned large video understanding model.

[0008] A video understanding large model optimization and evaluation system, comprising: A model construction unit is used to construct a connector including a spatiotemporal downsampler, a local bidirectional Mamba structure and a linear layer connected in sequence, and to form a large video understanding model with a visual encoder and a large language model; wherein a residual structure is also used between the spatiotemporal downsampler and the local bidirectional Mamba structure, and Mamba is a selective state space model; A three-stage progressive training unit is used to perform three-stage progressive training on the video understanding large model; in the first stage, the local bidirectional Mamba structure is trained using the video text pairs and image text pairs in the form of description; in the second stage, the entire connector is trained using the video text pairs in the form of description; in the third stage, the connector and the large language model are trained using the video text pairs in the form of multiple choice and description; A data construction unit, which uses a semi-automated data generation pipeline to build a global time series understanding dataset for evaluation and training; The fine-tuning and evaluation unit is used to construct a specific instruction fine-tuning training set using a global temporal understanding dataset, and to perform fine-tuning training on the video understanding large model after three-stage progressive training, and then use the global temporal understanding dataset to perform performance evaluation on the fine-tuned video understanding large model.

[0009] A processing device, comprising: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0010] A readable storage medium stores a computer program, which implements the above method when the computer program is executed by a processor.

[0011] It can be seen from the technical solution provided by the present invention that a new connector structure is designed to improve the global temporal understanding ability, which includes a spatiotemporal downsampler, a local bidirectional Mamba (selective state space model) structure and a linear layer. The spatiotemporal downsampler can reduce the token storage overhead; at the same time, the local bidirectional Mamba structure, on the one hand, makes up for the problem of limited receptive field, and on the other hand, it can simultaneously model intra-frame features and inter-frame features; in addition, the training of the connector is low-cost, and a three-stage progressive training strategy is used to combat catastrophic forgetting. In addition, since the existing benchmark datasets for large video understanding models mainly focus on video-level understanding and local event-level understanding, video-level understanding is coarse-grained and local event-level understanding cannot well measure the model's fine-grained understanding ability of the entire video, which is a basic ability for the model. Therefore, the present invention constructs a semi-automatic data generation pipeline and proposes a global temporal understanding benchmark based on this pipeline to make up for the evaluation gap in this ability in the existing benchmark field. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A flowchart of a video understanding large model optimization and evaluation method provided by an embodiment of the present invention; Figure 2 A schematic diagram of the overall framework of a large video understanding model provided by an embodiment of the present invention; Figure 3 A schematic diagram of a semi-automatic data generation pipeline provided by an embodiment of the present invention; Figure 4 An exemplary schematic diagram of a qualitative analysis result provided by an embodiment of the present invention; Figure 5 Another exemplary schematic diagram of a qualitative analysis result provided by an embodiment of the present invention; Figure 6 A schematic diagram of a video understanding large model optimization and evaluation system provided by an embodiment of the present invention; Figure 7 A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0015] First, the terms that may be used in this article are explained as follows: The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0016] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0017] The following is a detailed description of a video understanding large model optimization and evaluation method, system, device and storage medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in this field. If no specific conditions are specified in the embodiments of the present invention, the conventional conditions in the field or the conditions recommended by the manufacturer shall be followed. The instruments used in the embodiments of the present invention, for which the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0018] Embodiment 1 The embodiment of the present invention provides a method for optimizing and evaluating a large video understanding model. Figure 1 As shown, it mainly includes the following steps: Step 1: Build a connector that includes sequentially connected spatiotemporal downsamplers, local bidirectional Mamba (selective state space model) structures, and linear layers, and form a large video understanding model with the visual encoder and large language model.

[0019] Preferably, the spatiotemporal downsampler includes a three-part convolution structure, wherein the first and third parts are two-dimensional convolutions for spatial modeling within a frame; and the second part is a three-dimensional convolution for temporal modeling between frames.

[0020] In the embodiment of the present invention, the local bidirectional Mamba structure integrates two scanning modes, namely, local forward scanning and local reverse scanning within a frame.

[0021] Step 2: Perform three-stage progressive training on the video understanding large model.

[0022] In an embodiment of the present invention, in the first stage, video-text pairs and image-text pairs in description form are used to train local bidirectional Mamba structures; in the second stage, video-text pairs in description form are used to train the entire connector; in the third stage, video-text pairs in multiple-choice form and description form are used to train the connector and the large language model.

[0023] In addition, a residual structure is used between the spatiotemporal downsampler and the local bidirectional Mamba structure to further combat the forgetting problem.

[0024] Step 3: Use a semi-automated data generation pipeline to build a global time series understanding dataset.

[0025] Step 4: Use the global temporal understanding dataset to construct a specific instruction fine-tuning training set, and fine-tune the video understanding model after the three-stage progressive training, and then use the global temporal understanding dataset to evaluate the performance of the fine-tuned video understanding model.

[0026] It should be noted that the serial numbers of the above steps are only used to mark different steps and do not represent the execution order between the steps. The specific execution order of the steps can be determined by a person skilled in the art according to the content of the steps; for example, step 1 and step 2 should be executed in sequence, and step 4 should be executed on the basis of step 2 and step 3, but there is no strict logical order between step 3 and step 1 and step 2. Therefore, the execution order of step 3 and step 1 and step 2 can be determined according to actual conditions.

[0027] In the above scheme provided by the embodiment of the present invention, the data generation pipeline can be used to realize semi-automatic benchmark production. In industrial scenarios, it can generate data sets at low cost, and the global temporal understanding data set can be applied to the ability assessment of the video understanding large model to guide model optimization; the connector proposed by the present invention can be applied to the general video understanding large model, and the video understanding large model can be deployed on the server to complete the video understanding task. In addition, the progressive training strategy proposed by the present invention can be used to continue the training stage, effectively alleviating the overhead problem and forgetting problem in industrial model training.

[0028] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0029] 1. Model structure plan.

[0030] In general, the video understanding model includes: visual encoder, connector and large language model. In order to further improve the global temporal understanding ability of the video understanding model, this paper proposes a new connector structure, such as Figure 2 As shown in the middle part, the connector consists of three parts, namely the spatiotemporal downsampler, the local bidirectional Mamba and the linear layer.

[0031] The spatiotemporal downsampler consists of a three-part (e.g., three-layer) convolutional structure arranged in sequence, where the first and third layers are two-dimensional convolutions for spatial modeling within a frame, and the second layer is a three-dimensional convolution for temporal modeling between frames. Each convolution can reduce the number of tokens to alleviate memory overhead.

[0032] Local bidirectional Mamba is based on the Mamba structure, which combines two scanning methods: local forward scanning and local reverse scanning. These two scanning methods are for intra-frame; between frames, they are still unidirectional. These two scanning methods effectively maintain the token order while modeling image features. Figure 2 As shown, the local bidirectional Mamba structure includes: three linear units (responsible for feature mapping), a local forward scanning module, a local reverse scanning module and an activation layer; the input of the local bidirectional Mamba structure enters the first linear unit D1 and the second linear unit D2, and the output of the first linear unit D1 is divided into two paths through the activation layer; the output of the second linear unit D2 is respectively subjected to the local forward scanning module and the local reverse scanning module for global timing modeling, and the outputs of the two scanning methods are separately combined with one path of the activation layer (that is, combined in a multiplication manner, using the symbol ), and then enter the third linear unit D3 after fusion. The output of the third linear unit D3 enters the scanning module at the end to output the scanning mode; the above two scanning modules each adjust the token order according to a specific scanning method, and then use one-dimensional convolution, nonlinear activation and state space model to capture all temporal features. After adjustment, the features obtained by the two scanning methods are fused in an average manner (using the symbol express); Figure 2 The dotted box on the right shows the internal structure principle of the two scanning modules. is the activation function, and the same activation function as the activation layer can be used. For example, the SiLU (smoothed linear unit) function can be selected. The dotted box below shows the meaning of adjusting the order of tokens in the two scanning modes. The token here is a feature unit, which is a common term in the art and will not be elaborated in the present invention.

[0033] The linear layer is used to further map the features into the language feature space.

[0034] Considering that both the visual encoder and the large language model can be implemented using existing solutions, they will not be described in detail.

[0035] 3. Model training plan.

[0036] Since there are available pre-trained parameters for modules other than Mamba, it is feasible to train Mamba to unify the model. In order to combat the catastrophic forgetting problem that exists in the process of embedding Mamba, the present invention adopts a three-stage progressive training strategy. In the first stage, the present invention uses video-text pairs and image-text pairs in description form to only train local bidirectional Mamba to achieve feature alignment. In the second stage, the present invention uses video-text pairs in description form to simultaneously train the entire connector to achieve secondary alignment of features. In the third stage, the present invention uses video-text pairs in multiple-choice form and description form to train the connector and language model to achieve overall fusion. In addition, a residual structure is used between the spatiotemporal downsampler and the local bidirectional Mamba structure to further combat the forgetting problem. Figure 3 The α in is the proportional coefficient of the local bidirectional Mamba structure output during residual connection. The residual structure uses trainable parameters to adjust the degree of fusion between features.

[0037] The video-text pairs and image-text pairs in descriptive form, and video-texts in multiple-choice and descriptive forms involved in this part of the training are all existing data forms and can be collected in conventional ways, which will not be elaborated in the present invention.

[0038] 3. Global time series understanding dataset construction and evaluation scheme.

[0039] As mentioned before, the existing benchmark datasets for large video understanding models mainly focus on video-level understanding and local event-level understanding. Video-level understanding is coarse-grained, while local event-level understanding cannot well measure the model's fine-grained understanding ability of the entire video, which is a basic ability for the model. Therefore, the present invention constructs a semi-automatic data generation pipeline and proposes a global temporal understanding benchmark based on this pipeline to fill the gap in the evaluation of this ability in the existing benchmark field.

[0040] Specifically, the preferred implementation of the present invention using a semi-automated data generation pipeline to construct a global time series understanding dataset is as follows: 1. Build a semi-automated data generation pipeline that includes a temporal cue elimination pipeline and an illusion generation pipeline, such as Figure 3 As shown, an example of the overall framework of this semi-automated data generation pipeline is presented.

[0041] 2. Collect video data with video event sequences to form an original data set and input it into a semi-automated data generation pipeline. Each sample in the original data set contains a video and a video event sequence; the video event sequence records the event content in the corresponding video. For example, the event content of a video is ["cooking", "eating", "washing dishes"]. The above recorded event content is the label of the video, that is, the video event sequence. Each element in the video event sequence is a real event.

[0042] For example, the training set and the test set (if not available, the validation set) of five datasets with video event sequence labels, namely ActivityNet, ViTT, YouCook2, COIN and VidChapters, can be integrated to form the original dataset.

[0043] 3. The temporal cue elimination pipeline eliminates semantic cues from each video event sequence in the original data set, then filters out video event sequences containing common sense cues, and outputs video event sequences that do not contain common sense cues and have undergone semantic cue elimination.

[0044] Preferably, the temporal clue elimination pipeline includes two parts: semantic clue elimination and common sense clue elimination.

[0045] (3.1) The semantic clue elimination part mainly focuses on language clues with potential temporal logic, such as target reference, temporal words, etc. The implementation process of this part includes: the external large language model can be guided by prompt words to replace and delete phrases to eliminate language clues.

[0046] (3.2) The common sense clue elimination part mainly focuses on common event sequences. The implementation process of this part includes: firstly, shuffling the video event sequence after semantic clues are eliminated, and then guiding the external large language model to sort the shuffled video event sequence through prompt words; if the sorting is correct, it indicates that the corresponding video event sequence is a common sense sequence, and the corresponding video event sequence is eliminated. In order to ensure thoroughness, this process is repeated until no samples are eliminated in the data set.

[0047] 4. The video event sequence output by the timing cue elimination pipeline enters the hallucination generation pipeline to generate hallucination events.

[0048] Preferably, the hallucination generation pipeline includes three parts: hallucination generation, structural clue elimination and subject consistency correction.

[0049] (4.1) The hallucination generation part includes four steps. The first two steps refer to identifying the video event sequence output by the timing cue elimination pipeline to obtain the corresponding themes and roles; then combining the themes and roles to generate hallucination events that can be inserted into the corresponding video event sequence. This process covers the last two steps, that is, the first step generates related hallucination events, and the second step requires that the generated hallucination insertion can be inserted into the event sequence to further constrain the correlation between the hallucination and the original event sequence.

[0050] (4.2) The structural clue elimination part determines whether the sentence structure of the hallucination event generated by the hallucination generation part is consistent with the sentence structure of the corresponding video event sequence output by the timing clue elimination pipeline. If not, modification is performed.

[0051] (4.3) The theme consistency correction part determines whether the theme of the hallucination event generated by the hallucination generation part is consistent with the corresponding video event sequence. If not, it is modified.

[0052] After the above three parts are processed in sequence, the hallucination event output by the hallucination generation part is obtained.

[0053] 5. Integrate and shuffle the video event sequence output by the timing cue elimination pipeline and the hallucination events generated by the hallucination generation pipeline to obtain a rearranged data set containing hallucination events, and then construct a global timing understanding data set.

[0054] In the embodiment of the present invention, the global temporal understanding dataset includes the above three parts of data, namely: (1) the video event sequence output by the temporal clue elimination pipeline (formed as a temporal dataset after being shuffled); (2) the hallucination events output by the hallucination generation part (formed as a hallucination dataset after being inserted into the video event sequence); (3) the rearranged dataset containing the hallucination events (formed after the hallucination events are inserted into the video sequence and shuffled). The question-answering form of the rearranged dataset containing hallucinations and the temporal dataset is sorting, for example: 1→2→3, while the question-answering form of the hallucination dataset is enumeration, for example: 1&3 (1 and 3).

[0055] In this step, a global temporal understanding dataset is constructed, in which each test sample includes a video, a processed video event sequence, and a corresponding instruction. The processed video event sequence is any of the three parts of data mentioned above. The instruction requires the video understanding model to be evaluated to generate a specific form of answer based on the video and event sequence, for example, the answer form is "1→2→3" or "1&3". This is a semi-open answer form that can strike a balance between easing shortcut learning and evaluation convenience.

[0056] It should be noted that, for ease of understanding, Figure 3Examples of video event sequences, hallucination events, and integrated and disrupted event sequences are provided. In practical applications, the specific content of each type of event needs to be determined according to the actual situation, and the present invention does not limit it. At the same time, the present invention is applicable to multiple types of languages. Figure 2 The text is provided in English. In actual application, the user can also adjust the language of the input text according to actual needs.

[0057] The above global time series understanding dataset can use the time series dataset and the hallucination dataset to evaluate the time series modeling and event recognition capabilities respectively. For the time series dataset, the instruction requires the model to sort the disrupted event sequence. For the hallucination dataset, the instruction requires to select hallucinations from the event sequence mixed with hallucinations. The event rearrangement dataset containing hallucinations can evaluate these two capabilities at the same time, and its corresponding instruction requires the model to sort the real events.

[0058] In addition, since the existing models cannot generate a specific form of answer based on the sequence introduction and instructions, small-scale instruction fine-tuning is necessary. In order to avoid passing preference knowledge to the model during the training process, the present invention combines 3 rules to generate a specific instruction fine-tuning training set. Each data in the specific instruction fine-tuning training set is a question-answer pair, including training samples and corresponding answers. The training samples include videos, processed and rule-constrained video event sequences, and corresponding instructions. The processed and rule-constrained video event sequences refer to video event sequences processed using set rule constraints. Similarly, instructions are used to guide the video understanding model to generate a specified form of answer; the video event sequences processed using set rule constraints include: 1) Each video event sequence contains at least 2 real events (i.e., events in the corresponding video event sequence in the original data set); 2) A set proportion (e.g., 50%) of the question-answer pairs contain hallucination events; 3) When hallucination events are included, the position of the hallucination events is random. It should be noted that in the training set, the sorted data set shares the processing method of "shuffling after adding hallucinations".

[0059] In the implementation process, since the insertion of hallucination events and the scrambling of event sequences can be obtained through the code, the insertion position and the scrambling method can be recorded in advance, that is, the answer is known, so that the corresponding event sequence can be composed of question-answer pairs, which are used to supervise the output of the model during training, so as to calculate the loss function and fine-tune the model. The fine-tuning training scheme involved can be implemented with reference to conventional technology, and the present invention will not elaborate on it.

[0060] During the evaluation process, the test sample is input and the answer is output to form a set of question-answer pairs. The question-answer pairs generated in the re-ranked dataset and the time series dataset containing hallucinations are called sorted question-answer pairs, and the question-answer pairs generated in the hallucination dataset are called hallucinated question-answer pairs.

[0061] For sorted question-answer pairs, you can choose the edit distance as the evaluation metric. For hallucinated question-answer pairs, you can choose IOU (Intersection over Union) as the evaluation metric, and the formula is: ; Among them, P is the predicted sequence number set, G is the real sequence number set, It is a symbol for calculating the number of elements in a collection.

[0062] 4. Description of the performance of the solution of the present invention.

[0063] Existing benchmark datasets for large video understanding models mainly focus on video-level understanding and local event-level understanding. Video-level understanding is coarse-grained, while local event-level understanding cannot well measure the model's ability to understand the entire video in a fine-grained manner, which is a basic capability for the model. Therefore, the present invention first constructs a semi-automatic data generation pipeline and proposes a global temporal understanding benchmark based on this pipeline to fill the gap in the evaluation of this capability in the existing benchmark field.

[0064] In order to improve the performance of the model in global temporal understanding, this paper proposes a new connector structure, which consists of a spatiotemporal downsampler, a local bidirectional Mamba and a linear layer. The spatiotemporal downsampler can reduce the token storage overhead, while the local bidirectional Mamba can make up for the problem of limited receptive field on the one hand, and can model both intra-frame features and inter-frame features on the other hand. In addition, the training of the connector is low-cost. Based on the existing pre-training parameters, the present invention proposes a three-stage progressive training strategy to combat catastrophic forgetting.

[0065] The global temporal understanding benchmark evaluates the performance of existing mainstream video understanding large models, including: VideoLLaMA2, VideoChat2, VideoLLaMA, and MovieChat; among them, VideoLLaMA is an audio and video language model that aims to give the large language model the ability to understand video and audio; VideoLLaMA2 is a multimodal video large language model that aims to improve video understanding and audio understanding capabilities; VideoChat2 is a multimodal video understanding model that aims to achieve deep understanding and conversational interaction of video content by combining video-based models and large language models; MovieChat is an innovative long video understanding framework that combines visual models and large language models to efficiently process long video content of more than 10,000 frames; considering that the above models are all existing models, they are not elaborated. During the evaluation process, the edit distance was used as the evaluation indicator. The evaluation results of VideoLLaMA2 were 30.70%, VideoChat2 were 42.53%, VideoLLaMA were 26.34%, and MovieChat were 24.89%. The above evaluation shows that the existing large video understanding models do not have strong global temporal understanding capabilities, which provides more guidance for future research. The baseline model with the connector structure proposed in the present invention achieved 55.98% on this benchmark, an increase of 25.29% compared to the baseline model VideoLLaMA2. This shows that the structure can well improve the global understanding ability. In addition, the connector has generalization capabilities. The optimized model achieved 56.82% and 45.00% on the general video understanding benchmarks MVBench and VideoMME, respectively, an increase of 3.42% and 1.00% compared to the baseline model. Figure 4 and Figure 5 Some qualitative analysis results are shown. Each example contains a video, a question, VideoChat2, VideoLLaMA2, and the answer of the optimized model of the present invention. The example also contains the result of scoring using GPT4o-mini; GPT4o-mini is a mini version of GPT-4o, and GPT-4o is the full-featured GPT-4 (fourth-generation generative pre-trained transformer) model; similarly, Figure 4 and Figure 5 The text example in English is also used in the text. Compared with VideoChat2 and VideoLLaMA2, the present invention can answer more video content and more accurately, where the solid line marks the text that hits the video content, and the dotted line marks the hallucination text. This shows that the optimization scheme of the present invention has a benefit in global temporal understanding.

[0066] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0067] Embodiment 2 The present invention also provides a video understanding large model optimization system, which is mainly used to implement the method provided in the above embodiment, such as Figure 6 As shown, the system mainly includes: A model construction unit is used to construct a connector including a spatiotemporal downsampler, a local bidirectional Mamba structure and a linear layer connected in sequence, and to form a large video understanding model with a visual encoder and a large language model; wherein a residual structure is also used between the spatiotemporal downsampler and the local bidirectional Mamba structure, and Mamba is a selective state space model; A three-stage progressive training unit is used to perform three-stage progressive training on the video understanding large model; in the first stage, the local bidirectional Mamba structure is trained using the video text pairs and image text pairs in the form of description; in the second stage, the entire connector is trained using the video text pairs in the form of description; in the third stage, the connector and the large language model are trained using the video text pairs in the form of multiple choice and description; A data construction unit, which uses a semi-automated data generation pipeline to build a global time series understanding dataset for evaluation and training; The fine-tuning and evaluation unit is used to construct a specific instruction fine-tuning training set using a global temporal understanding dataset, and to perform fine-tuning training on the video understanding large model after three-stage progressive training, and then use the global temporal understanding dataset to perform performance evaluation on the fine-tuned video understanding large model.

[0068] Considering that the main technical details involved in the system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0069] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0070] Embodiment 3 The present invention also provides a processing device, such as Figure 7 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0071] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0072] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example: The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.; The output device may be a display terminal; The memory may be a random access memory (RAM) or a non-volatile memory such as a disk storage.

[0073] Embodiment 4 The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0074] In the embodiment of the present invention, the readable storage medium as a computer-readable storage medium can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0075] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.

Claims

1. A video understanding large model optimization and evaluation method, characterized in that: include: Construct a connector including a spatiotemporal downsampler, a local bidirectional Mamba structure and a linear layer connected in sequence, and form a large video understanding model with a visual encoder and a large language model; wherein a residual structure is also used between the spatiotemporal downsampler and the local bidirectional Mamba structure, and Mamba is a selective state space model; The video understanding large model is trained progressively in three stages; in the first stage, the local bidirectional Mamba structure is trained using video-text pairs and image-text pairs in the form of descriptions; in the second stage, the entire connector is trained using video-text pairs in the form of descriptions; in the third stage, the connector and the large language model are trained using video-text pairs in the form of multiple choices and descriptions; Use a semi-automated data generation pipeline to build a global time series understanding dataset; A specific instruction fine-tuning training set is constructed using a global temporal understanding dataset, and the large video understanding model after three-stage progressive training is fine-tuned. The global temporal understanding dataset is then used to evaluate the performance of the fine-tuned large video understanding model.

2. A video understanding large model optimization and evaluation method according to claim 1, characterized in that: The spatiotemporal downsampler includes a three-part convolution structure, wherein the first and third parts are two-dimensional convolutions for spatial modeling within a frame; and the second part is a three-dimensional convolution for temporal modeling between frames.

3. A video understanding large model optimization and evaluation method according to claim 1, characterized in that: The local bidirectional Mamba structure integrates two scanning modes, namely, local forward scanning and local reverse scanning within a frame; the local bidirectional Mamba structure includes: three linear units, a local forward scanning module, a local reverse scanning module and an activation layer; the input of the local bidirectional Mamba structure enters the first linear unit and the second linear unit, and the output of the first linear unit is divided into two paths through the activation layer; the output of the second linear unit is respectively subjected to global timing modeling by the local forward scanning module and the local reverse scanning module, and the outputs of the two scanning modes are separately combined with one path of the activation layer, and then fused to enter the third linear unit, and the output of the third linear unit enters the tail scanning module to output the scanning mode.

4. A video understanding large model optimization and evaluation method according to claim 1, characterized in that: The semi-automated data generation pipeline used to construct the global time series understanding dataset includes: Build a semi-automated data generation pipeline that includes a temporal cue removal pipeline and an illusion generation pipeline; Video data with video event sequences are collected to form a raw data set, and input into a semi-automatic data generation pipeline, where each sample in the raw data set includes a video and a video event sequence; The temporal cue elimination pipeline eliminates semantic cues from each video event sequence in the original data set, and then filters out the video event sequence containing common sense cues, and the output video event sequence is a video event sequence that does not contain common sense cues and has undergone semantic cue elimination; The video event sequence output by the timing clue elimination pipeline enters the hallucination generation pipeline to generate hallucination events; The video event sequence output by the timing cue elimination pipeline is integrated and shuffled with the hallucination events generated by the hallucination generation pipeline to construct a global timing understanding dataset, that is, a rearranged dataset containing hallucination events.

5. A video understanding large model optimization and evaluation method according to claim 4, characterized in that: The temporal clue elimination pipeline includes two parts: semantic clue elimination and common sense clue elimination; The semantic clue elimination part uses prompt words to guide the external large language model to replace and delete phrases and eliminate language clues; In the common sense clue elimination part, the video event sequence after the semantic clues are eliminated is first shuffled, and the external large language model is guided by prompt words to sort the shuffled video event sequence; if the sorting is correct, it indicates that the corresponding video event sequence is a common sense sequence, and the corresponding video event sequence is eliminated.

6. A video understanding large model optimization and evaluation method according to claim 4, characterized in that: The hallucination generation pipeline includes three parts: hallucination generation, structural clue elimination and subject consistency correction; The hallucination generation part identifies the video event sequence output by the timing clue elimination pipeline, obtains the corresponding theme and role, and then generates a hallucination event that can be inserted into the corresponding video event sequence by combining the theme and role; The structural clue elimination part determines whether the sentence structure of the hallucination event generated by the hallucination generation part is consistent with the sentence structure of the corresponding video event sequence output by the timing clue elimination pipeline, and if not, makes a modification; The theme consistency correction part determines whether the theme of the hallucination event generated by the hallucination generation part is consistent with the corresponding video event sequence. If not, it is modified.

7. A video understanding large model optimization and evaluation method according to claim 4, characterized in that: Each data in the specific instruction fine-tuning training set is a question-answer pair, including a training sample and a corresponding answer, the training sample includes a video, a processed and rule-constrained video event sequence, and a corresponding instruction, the processed and rule-constrained video event sequence refers to a video event sequence processed using set rule constraints, the processed video event sequence is data in the global temporal understanding dataset, and the instruction is used to guide the video understanding large model to generate a specified form of answer; The video event sequence processed by the set rule constraints includes: Each video event sequence contains at least two real events, where the real events refer to events in the corresponding video event sequence in the original data set; A set proportion of question-answer pairs included hallucination events; Also, when hallucination events are included, the locations of the hallucination events are random.

8. A video understanding large model optimization and evaluation system, characterized in that: include: A model construction unit is used to construct a connector including a spatiotemporal downsampler, a local bidirectional Mamba structure and a linear layer connected in sequence, and to form a large video understanding model with a visual encoder and a large language model; wherein a residual structure is also used between the spatiotemporal downsampler and the local bidirectional Mamba structure, and Mamba is a selective state space model; A three-stage progressive training unit is used to perform three-stage progressive training on the video understanding large model; in the first stage, the local bidirectional Mamba structure is trained using the video text pairs and image text pairs in the form of description; in the second stage, the entire connector is trained using the video text pairs in the form of description; in the third stage, the connector and the large language model are trained using the video text pairs in the form of multiple choice and description; A data construction unit, which uses a semi-automated data generation pipeline to build a global time series understanding dataset for evaluation and training; The fine-tuning and evaluation unit is used to construct a specific instruction fine-tuning training set using a global temporal understanding dataset, and to perform fine-tuning training on the video understanding large model after three-stage progressive training, and then use the global temporal understanding dataset to perform performance evaluation on the fine-tuned video understanding large model.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images