Multimodal video dataset optimization and prediction processing method

Abstract

The application discloses a kind of multimodal video dataset optimization and prediction processing method, including the following steps: step one, using the strategy processing video dataset of multiple maps to one text;Step two, each video is identified;Step three, using natural language processing tool, in-depth analysis is carried out to video related text, and key information is extracted therefrom;Step four, after automatically carrying out text analysis in step three, preliminary verification of video-text matching is carried out;Step five, after being automatically processed, it is submitted to artificial for auditing;Step six, the optimized video dataset is used to train multimodal prediction model.The multimodal video dataset optimization and prediction processing method of the application, by the setting of step one to step six, can improve data processing efficiency and prediction accuracy by integrating advanced multimodal learning technology.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and natural language processing, and further to methods for optimizing and predicting multimodal video datasets. Background Technology

[0002] Multimodal Large Language Models (MLLMs) are large language models that can accept multimodal inputs, such as images and language. Important works in MLLM include LLaVA, which combines CLIP and Vicuna, using a simple linear adapter and fine-tuning on multimodal instruction-based data. MiniGPT-4 employs a more complex QFormer architecture to extract condensed visual token features as input to the large language model. InstructBLIP further augments the training data with 13 diverse tasks, including video data, to improve the instruction-following ability of MLLMs. mPLUG-Owl corrects alignment errors and enhances its multi-turn dialogue capabilities. Otter uses a cross-attention architecture for more nuanced visual-language fusion. Lynx is capable of inferring video-level sequences, employing prefix tuning for instruction tuning, providing a more efficient alternative to the cross-attention format.

[0003] Video prediction is a significant research area in computer vision, particularly in future prediction based on video input. For example, the Robotic Pushing dataset contains 59,000 interactions involving robot pushing actions. This dataset achieves accurate video predictions by using the robot's impending action as a conditional factor. Another example is research on human activity prediction based on surveillance camera video. The CLEVRER dataset uses a 3D engine to simulate the motion of physical objects and predict collision effects based on trajectory paths. The SUTD-TrafficQA dataset explores real-world scenes from a first-person perspective, assessing the likelihood of car accidents in a driver's field of vision. In the field of autonomous driving, the nuScenes dataset utilizes urban traffic data to evaluate the predictability of trajectories between pedestrians and vehicles in busy urban traffic environments.

[0004] In summary, benchmarks like RAVEN, ActivityNet-QA, STAR, and Seed-Bench set the standard for specific reasoning tasks. RAVEN challenges models to perform relational and analogical reasoning on abstract images, emphasizing combinatorial reasoning. ActivityNet-QA targets video content, using crowdsourced QA pairs to test concise reasoning. STAR focuses on contextual reasoning in real-world videos, using hypergraphs to represent dynamic situations. Seed-Bench evaluates the generative understanding capabilities of MLLMs, containing 19K multiple-choice questions across 12 evaluation dimensions in image and video modalities. Our work differs from these structured, typically multiple-choice-based approaches by introducing a novel benchmark that integrates extended visual sequences and long queries to generate open-ended answers. This approach encourages models to process complex information and produce context-sensitive responses, extending the scope of MLLM evaluation beyond traditional paradigms.

[0005] Despite their impressive multimodal understanding capabilities, MLLMs also face challenges in evaluation due to their generative nature, resulting in low data processing efficiency and predictive accuracy. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention aims to provide an efficient method for structuring video datasets. This method is suitable for constructing human activity prediction tasks based on complex video datasets (such as Charades), and improves data processing efficiency and prediction accuracy by integrating advanced multimodal learning techniques.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a method for optimizing and predicting multimodal video datasets, characterized by comprising the following steps:

[0008] Step 1: Process the video dataset using a multi-image-to-text strategy;

[0009] Step two: Identify key segments containing important behaviors in each video to select video segments;

[0010] Step 3: Use natural language processing tools to perform in-depth analysis of the video-related text, extract key information, and match it with the behavior of the video segments selected in Step 2. Automatic text analysis is performed. Step 4: After the automatic text analysis in Step 3, perform preliminary verification of video-text matching to ensure that each selected video segment and its corresponding text description are consistent in content and context.

[0011] Step 5: After automatic processing, the video is submitted to a human reviewer to assess the video's coherence, the accuracy of action sequences, and the completeness of the text, and to make adjustments and optimizations accordingly.

[0012] Step 6: Use the optimized video dataset to train the multimodal prediction model.

[0013] As a further improvement to the present invention, the specific steps of automatic text analysis using natural language processing tools in step three are as follows:

[0014] Step 31: Segment the last sentence, ensuring that it contains only a single predicate in its structure;

[0015] Step 32: Find the text and video clips to be predicted.

[0016] As a further improvement of the present invention, the specific method for dividing the last sentence in step three and ensuring that the structure has only a single predicate is as follows: According to the punctuation, locate the last clause and preprocess it: If there is a conjunction before it, delete the conjunction; if there is no subject, inherit the subject of the previous sentence to ensure a single predicate structure; if there is only one predicate, it proves that the last sentence is complete, and continue processing; if there are two or more predicate verbs, it means that the sentence still needs to be divided into clauses.

[0017] As a further improvement to the present invention, the specific steps for finding the predicted text and video segments in step three-two are as follows:

[0018] Step 1: Locate the last noun at the end of the sentence and use a natural language processing tool to analyze the subordinate verb of that noun;

[0019] Step 2: The verb anchored to the last noun is used as the automatically analyzed prediction content;

[0020] Step 3: Use the "verb-noun" pair to find the verb phrase with the closest cosine similarity and its code in the action dictionary of the dataset;

[0021] Step 4: Check if the similar verb phrase codes found in the action category annotation of this data itself exist. If they do, then this is considered the action segment found. Take the start time of the corresponding annotation of this action segment as the cutoff point, remove the later video segments, and keep the earlier video segments as the vision information to be sent to the contestant model. At the same time, remove the last verb noun segment we found from the text as context information.

[0022] The beneficial effects of this invention are:

[0023] Efficiency Improvement: By combining automated processing with expert review, data processing efficiency has been significantly improved.

[0024] Enhanced Accuracy: Precise video segment selection and text matching ensure high accuracy of the prediction model. Multimodal Fusion: Innovative image-text separation strategies effectively fuse visual and textual information, supporting complex multimodal learning.

[0025] Application prospects: This technical solution can be widely used in fields such as artificial intelligence, machine learning, and video analysis, and is especially suitable for scenarios that require efficient and accurate processing and prediction of complex video data. Attached Figure Description

[0026] Figure 1 The differences and handling of the Charades and ActivityNet dataset formats;

[0027] Figure 2 A schematic diagram illustrating the entire process of transforming the Charades dataset into a prediction task.

[0028] Figure 3 This is a schematic diagram for example demonstration. Detailed Implementation

[0029] The present invention will now be described in further detail with reference to the embodiments shown in the accompanying drawings. The human activity prediction task in this embodiment is explained in detail below:

[0030] Definition: In human-centered activity tasks, each data point captures a snapshot of real-world human activity. The model's task is to use common-sense reasoning to predict subsequent human behavior or activities.

[0031] Challenges: First, human activities occur within environmental or social contexts, which significantly influence subsequent behavior, requiring models to skillfully integrate these contextual variables. Second, the complexity of human behavior, guided by unwritten norms and cultural codes, presents a considerable challenge to integrating common-sense reasoning into the models.

[0032] Dataset Construction: Two datasets are available for this task: ActivityNet Captions for general daily activities and Charades for specific indoor behaviors. Charades presents a challenge due to its complex annotations and overlapping time frames. We address this by selecting the last verb-noun pair in the annotations for prediction. A 260-video subset of Charades was rigorously evaluated by domain experts to ensure dataset quality and consistency.

[0033] The basic information, challenges, and characteristics of the Charades dataset are as follows:

[0034] 1. Unlike the ActivityNet dataset, where each image has a corresponding original caption and is a one-to-one image-text match, the Charades dataset only has an overall description for each video. To address this, we adopt an image-text separation strategy. Instead of finding or designing text descriptions for every context image, we directly match multiple images to one text, only requiring a one-to-one correspondence between the predicted image and text.

[0035] 2. Charades has numerous video segment classification labels for each video, with overlapping and mixed start and end times. Each video also has a script and description, which are the annotator's description of the entire video. Identifying and extracting the predicted segments from each dataset is challenging. Our strategy during the modification process was to try to find the last action in the text annotation and ensure that this action has a corresponding labeled video segment.

[0036] Reference Figures 1 to 3 As shown, this embodiment presents a multimodal video dataset optimization and prediction processing method, aiming to realize the structured processing and prediction model for video datasets of the present invention, particularly in the application scenario of human behavior prediction. The implementation method includes the application of image-text separation strategy, accurate selection and processing of video segments, automatic text analysis and matching, verification and optimization of video-text matching, and final review and adjustment by an expert team, specifically including the following steps:

[0037] Image-text separation strategy application: When processing the Charades video dataset, a multi-image-to-one-text strategy is adopted. This approach avoids the pursuit of a direct one-to-one correspondence between each video segment and the text, instead focusing on achieving accurate image-text matching during the prediction stage.

[0038] Video segment selection: For each video, identify key segments containing important actions. This process includes identifying the last action in the text annotation and ensuring that this action has a clear corresponding segment in the video.

[0039] Automated text analysis: Using natural language processing tools, such as Spacy, to perform in-depth analysis of video-related text. Key information, such as action descriptions, is extracted to facilitate matching with behaviors within video clips.

[0040] Video-Text Matching Validation: After automatic analysis, a preliminary verification of video-text matching is performed. This ensures that each selected video segment and its corresponding text description are consistent in content and context.

[0041] Expert Team Review: After automated processing, the video is submitted to a review team composed of multiple domain experts for final inspection. The experts will evaluate the video's coherence, the accuracy of action segments, and the completeness of the text, and make necessary adjustments.

[0042] Model training and evaluation: The optimized video dataset is used to train the multimodal prediction model, and then its performance is tested and evaluated in specific evaluation scenarios, such as human behavior prediction.

[0043] Through the above steps, the present invention can effectively improve the processing efficiency and prediction accuracy of video datasets, especially in complex multimodal learning environments.

[0044] Specifically, such as Figures 1 to 2 As shown, the specific implementation process of the automatic text analysis and expert team review mentioned above is as follows:

[0045] In the automatic processing stage, the spacy tool is used to process the text content.

[0046] Step 1: Segment the last sentence, ensuring that it contains only a single predicate. The method to ensure that the sentence contains only a single predicate is as follows:

[0047] (a) Locate and preprocess the last clause based on punctuation:

[0048] 1. If there is a conjunction (and, and then, etc.) before the word, delete the conjunction.

[0049] 2. If there is no subject, then the subject of the previous sentence is adopted.

[0050] (ii) Guaranteeing a single predicate structure:

[0051] 1. If there is only one predicate, it proves that the last sentence is complete, and we can continue processing.

[0052] 2. If there are two or more predicate verbs, it means the sentence needs to be divided into clauses.

[0053] Step 2: Find the ground truth (gt) for prediction (including text and video clips).

[0054] 1. Find the last noun at the end of the sentence and use Spyay to analyze the subordinating verb of that noun.

[0055] 2. The verb anchored to the last noun is used as the automatically analyzed prediction content.

[0056] 3. Use this "verb-noun" pair to find the verb phrase with the closest cosine similarity in the action dictionary of the dataset, along with its code.

[0057] 4. The similar verb phrase codes found should exist in the action category annotations of this data itself. If they exist, then this is considered the action segment we found. We use the start time of the corresponding annotation of this action segment as the cutoff point, remove the later video segments, and keep the earlier video segments as the vision information for the contestant model. At the same time, we extract the last verb-noun segment we found from the text as context information.

[0058] During the expert review phase, after completing the aforementioned modifications, three experts conducted a comprehensive evaluation and correction of the 260 test data points. The evaluation criteria are as follows:

[0059] Visual assessment:

[0060] 1. The actions in this video should be as fluid and logical as possible.

[0061] 2. The extracted video clips are accurately located. (If the predicted action appears in the context, the clips are manually deleted.)

[0062] Text-based evaluation:

[0063] 1. The context should ideally be a complete sentence. If there are any extraneous conjunctions or nouns, delete them manually.

[0064] 2. During the processing, experts found that in some videos, the crowdsourced actors laughed during filming, making the direct descriptions of the scenes illogical. Therefore, the context of these videos was adjusted and the predicted content was modified.

[0065] Finally, the test data, optimized by experts, can be transferred to the model for training. The specific application of this method to the Charades dataset is shown in the table below:

[0066] Experimental results of mainstream models on the human activity prediction task of the Charades dataset

[0067]

[0068] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for optimizing and predicting multimodal video datasets, characterized in that: Includes the following steps: Step 1: Process the video dataset using a multi-image-to-text strategy; Step two: Identify key segments containing important behaviors in each video to select video segments; Step 3: Use natural language processing tools to perform in-depth analysis of the video-related text, extract key information, and match it with the behavior of the video segments selected in Step 2 to automatically perform text analysis; Step four: After the automatic text analysis in step three, perform preliminary verification of video-text matching to ensure that each selected video segment and its corresponding text description are consistent in content and context. Step 5: After automatic processing, the video is submitted to a human reviewer to assess the video's coherence, the accuracy of action sequences, and the completeness of the text, and to make adjustments and optimizations accordingly. Step six involves using the optimized video dataset to train a multimodal prediction model; the specific steps for automatic text analysis using natural language processing tools in step three are as follows: Step 31: Segment the last sentence, ensuring that it contains only a single predicate in its structure; Step 32: Find the text and video clips to be predicted; The specific steps for finding the predicted text and video clips in step three, part two are as follows: Step 1: Locate the last noun at the end of the sentence and use a natural language processing tool to analyze the subordinate verb of that noun; Step 2: The verb anchored to the last noun is used as the automatically analyzed prediction content; Step 3: Use the "verb-noun" pair to find the verb phrase with the closest cosine similarity and its code in the action dictionary of the dataset; Step 4: Check if the similar verb phrase codes found in the action classification annotation of this data itself exist. If they do, it is considered that this is the action segment found. Take the start time of the corresponding annotation of this action segment as the cutoff point, remove the later video, and keep the earlier video segment as the vision information to be sent to the contestant model. At the same time, remove the last verb noun segment found in the text as context information.

2. The method for optimizing and predicting multimodal video datasets according to claim 1, characterized in that: The specific method for separating the last sentence in step three and ensuring that the structure contains only a single predicate is as follows: Based on the punctuation, locate the last clause and preprocess it: If there is a conjunction before it, delete the conjunction; if there is no subject, inherit the subject of the previous sentence to ensure a single predicate structure; if there is only one predicate, it proves that the last sentence is complete, and continue processing; if there are two or more predicate verbs, it means that the sentence still needs to be divided into clauses.

Images