Time sequence enhanced video question and answer method and system

Abstract

The invention relates to a time sequence enhanced video question answering method and system, and relates to the field of time sequence video question answering. Comprising the steps of 1, constructing multi-dimensional time sequence instruction data; defining and defining five time sequence dimensions, establishing a data collection and screening process, and ensuring that the data preparation work of each time sequence dimension is complete; 2, multi-task time sequence instruction fine adjustment is carried out; constructing a plurality of time sequence auxiliary tasks, and finely adjusting the video question-answering model by using the collected data so as to enhance the time sequence question-answering capability of the model; 3, performing multi-dimensional time sequence question and answer evaluation; and for five time sequence dimensions, additionally collecting and constructing an evaluation data set, and evaluating the time sequence question-answering capability of the video question-answering model. A specific data collection and construction method is designed for each dimension, and a data basis is provided for enhancing the time sequence question and answer ability of the video question and answer model. The limitation of data capacity is broken through, and the time sequence question-answering capability of the video question-answering model can be effectively improved in multiple dimensions.

Description

Technical Field

[0001] The present invention relates to the field of time-series video question answering, and in particular to a time-series enhanced video question answering method and system. Background Art

[0002] Video question answering aims to correctly answer questions about the content of a video. Unlike image question answering that processes static images, video data contains complex temporal dynamic characteristics. Therefore, the perception and understanding of video content in the temporal dimension constitutes a key technical challenge facing video question answering.

[0003] In order to improve the performance of video question answering models on temporal question answering, adapter-based methods aim to design a new type of adapter to significantly reduce the number of video tokens. This type of method selectively downsamples the visual tokens in the video frames by introducing spatiotemporal convolution or attention mechanisms in the adapter, thereby increasing the number of video frames that can be processed. At the same time, by introducing strategies such as long-context training and multi-image training, training strategy-based methods significantly improve the ability of video question answering models to process long videos. In addition, data-based methods construct time-sensitive datasets and enhance the model's perception and understanding of temporal dynamics by fine-tuning the video question answering model on these datasets.

[0004] In addition, in order to comprehensively evaluate the temporal question answering capabilities of video question answering models, it is urgent to establish a robust and unbiased evaluation benchmark. Early evaluation benchmarks mainly focused on the static visual perception and understanding capabilities of the model, while ignoring the importance of the dynamic temporal dimension. To make up for this deficiency, a number of evaluation benchmarks specifically for temporal question answering have emerged recently. These benchmarks introduce diverse temporal tasks such as action recognition and temporal prediction, and cover a wider range of evaluation scenarios.

[0005] Although the above research work has made some progress in the field of time-series video question answering, it still faces the following problems: (1) Explicit modeling of the temporal relationship between video frames is ignored. Simply increasing the number of input video frames cannot fundamentally improve the model's ability to perceive and understand temporal dynamics. Previous studies have shown that existing video question-answering models perform poorly in tasks that require a high degree of temporal perception and understanding. These models rely more on prior knowledge or statistical biases in the pre-training process to complete temporal question-answering tasks, rather than truly understanding the temporal dependencies between video frames.

[0006] (2) Insufficient coverage of the temporal dimension. Existing benchmarks fail to fully cover all key temporal dimensions in temporal question answering tasks. Although current benchmarks have designed a variety of task forms, in terms of the temporal question answering capabilities required to actually complete these tasks, most tasks only involve two or three temporal dimensions. In addition, existing benchmarks generally have statistical biases in data distribution, which allows video question answering models to rely on statistical priors for evaluation, which cannot accurately reflect the model's true performance in temporal question answering. Summary of the invention

[0007] In order to address the deficiencies of the prior art, the present invention provides a time-enhanced video question-answering method, which enhances the multi-dimensional time-series question-answering capability of the video question-answering model while providing an objective and unbiased video time-series question-answering evaluation benchmark.

[0008] The invention also provides a time sequence enhanced video question-answering system.

[0009] In order to achieve the above object, the present invention adopts the following technical solution: In a first aspect, the present invention provides a time-enhanced video question-answering method.

[0010] A time-order enhanced video question answering method, comprising: Step 1: Multi-dimensional sequential instruction data construction; Clarify and define the five time series dimensions, establish a data collection and screening process, and ensure that the data preparation for each time series dimension is complete; Among them, data collection and screening include the following specific steps: Design specialized tasks for each temporal dimension; Collect open-source video datasets with rich temporal dynamics, exclude videos that do not meet the duration requirements, and automatically generate questions and answers based on temporal annotations, where the answers include open-ended answers and multiple-choice answers; Sampling generated questions and answers; Step 2: Fine-tune multi-task timing instructions; Construct multiple time-series auxiliary tasks and use the collected data to fine-tune the video question-answering model to enhance the model's time-series question-answering capabilities; Among them, multiple time series auxiliary tasks are constructed, including: Collect video instruction fine-tuning data, use the open source pre-trained video question-answering model to filter the data containing time-series questions and answers, and construct auxiliary tasks based on the remaining data; Construct a video frame index prediction task, randomly extract a frame from the original video frame sequence and place it at the front of the sequence, and use the video question answering model to predict the original position index of the video frame; Construct an instruction video question-answering task, splice multiple videos, and indicate the target video in natural language. The video question-answering model will answer questions based on the target video. Step 3: Multi-dimensional sequential question answering evaluation; For the five temporal dimensions, additional evaluation datasets are collected and constructed to evaluate the temporal question answering capabilities of the video question answering model.

[0011] Preferably, according to the present invention, five temporal dimensions are clarified and defined, including: summarizing the video temporal question-answering capability into five dimensions: temporal dynamics, temporal reasoning, duration, temporal positioning and temporal sequence.

[0012] Preferably, according to the present invention, a dedicated task is designed for each time series dimension, including: In terms of temporal dynamics, the task form is designed to answer the direction of movement of a specified object in the video; the specified object is described in natural language; the direction of movement is represented by direction nouns, including four categories: "upper left", "lower left", "upper right" and "lower right"; For temporal reasoning, the task format is designed to answer the possible actions that may occur next based on the existing content of the video, where the actions are represented by gerund phrases; Regarding duration, the task format is designed to answer the duration of a specified action behavior, where the action behavior is specified in natural language; the duration is divided into three categories: long, medium, and short, which are respectively expressed as "longer than two-thirds of the video", "between one-third and two-thirds of the video", and "shorter than one-third of the video"; For time positioning, the task format is designed to answer the occurrence time of a specified action behavior, where the action behavior is specified in natural language form, and the occurrence time is expressed in three categories: "located in the front of the video", "located in the middle of the video", and "located in the back of the video"; For the time sequence, the task format is designed to answer the correct order in which three specified actions occur, where each action is represented by a gerund phrase.

[0013] Preferably, according to the present invention, an open source video dataset with rich temporal dynamics is collected, videos whose duration does not meet the requirements are excluded, and questions and answers are automatically generated according to temporal annotations, including: Collect videos and temporal annotations from datasets containing rich temporal dynamics; exclude videos whose duration does not meet the requirements; Generate open-ended question-answer pairs, manually construct several open-ended question samples, use ChatGPT to generate several different open-ended questions for each temporal dimension based on the constructed open-ended question samples and task description, and generate open-ended answers based on the video temporal annotations; Generate multiple-choice question-answer pairs, manually construct several multiple-choice question samples, use ChatGPT to generate several different multiple-choice questions for each temporal dimension based on the constructed multiple-choice question samples and task description, and generate one correct option and three incorrect options based on the video temporal annotation.

[0014] Preferably, according to the present invention, the generated questions and answers are sampled; including: For the three temporal dimensions of time dynamics, duration, and time location, which have fixed answer options, the number of different options is balanced through sampling to keep the number of options consistent; For the two temporal dimensions of temporal reasoning and time order, which have no fixed answer options, the action category with the least number is used, and the number of action categories is balanced through sampling to avoid long-tail distribution; The number of open-ended question-answer pairs and multiple-choice question-answer pairs was balanced through sampling.

[0015] Preferably, according to the present invention, video instruction fine-tuning data is collected, and the data containing time-series questions and answers is filtered using an open source pre-trained video question-answering model; including: Construct input instructions for the pre-trained video question answering model; The constructed input instructions and videos are input into the pre-trained video question-answering model, and each instruction has a corresponding video. According to the response of the pre-trained video question-answering model, the video question-answer pairs that do not contain temporal content are filtered out.

[0016] Preferably, according to the present invention, constructing a video frame index prediction task comprises: For video question-answer pairs that do not contain temporal content, a frame is randomly selected from the original video frame sequence and placed at the front of the video frame sequence; Construct instructions, use ChatGPT to generate 10 instructions based on manually constructed examples, and randomly sample one of the instructions for each question-answer pair; Construct the answer.

[0017] Preferably, according to the present invention, constructing an instruction video question answering task includes: For video question-answer pairs that do not contain temporal content, a video is randomly extracted from the dataset and randomly spliced ​​to the front or back of the original video; Construct instructions, use ChatGPT to generate 10 instructions based on manually constructed examples, and randomly sample one of the instructions for each question-answer pair; Directly use the answers from the original video as the answers for the spliced ​​video.

[0018] Preferably, according to the present invention, the three parts of data of the mixed constructed multi-dimensional timing instruction data, the constructed multi-task timing instruction and the remaining video instruction fine-tuning data, i.e., the video question-answering pair data containing timing content, are used to fine-tune the video question-answering model; including: Input the instruction text into the encoding layer to obtain the text semantic features ; Input the video frame sequence into the visual encoder to obtain visual features , and use the visual adapter to map the visual features into the text semantic space to obtain the visual semantic features aligned with the text semantic features ; Semantic features of text and visual semantic features After concatenation, the input is pre-trained into a large-scale language model, and the model output is supervised and fine-tuned using cross entropy loss.

[0019] Preferably, according to the present invention, for five time series dimensions, additional evaluation data sets are collected and constructed to evaluate the time series question answering capability of the video question answering model; including: Construct a multi-dimensional time series question answering evaluation dataset; including: Use similar steps as above to construct the evaluation dataset; Use three open source pre-trained video question-answering models, Qwen2-VL, LLaVA-OneVision, and MiniCPM-V2.6, as filters, input randomly sampled single-frame video content and constructed question-answer pairs, and judge whether the filters can correctly answer the questions based on the answers output by the filters. If at least two filters can correctly answer the questions, it is determined that the video question-answer pair has a single-frame deviation; Remove question-answer pairs with single-frame deviations from the dataset and rebalance the number of different answer candidates through sampling to obtain the final evaluation dataset; Input the video and question-answer pairs in the constructed evaluation dataset into the video question-answering model and obtain the answers. Compare whether the answers output by the video question-answering model are the same as the real answers. Count the proportion of questions correctly answered by the video question-answering model to the total number of questions.

[0020] In a second aspect, the present invention provides a time-enhanced video question-answering system.

[0021] A time-order enhanced video question answering system, comprising: The multi-dimensional time series instruction data construction unit is configured to: clarify and define five time series dimensions, establish data collection and screening processes, and ensure that data preparation for each time series dimension is complete; A multi-task timing instruction fine-tuning unit is configured to: construct multiple timing auxiliary tasks and use the collected data to fine-tune the video question answering model to enhance the timing question answering capability of the model; The multi-dimensional temporal question answering evaluation unit is configured to collect and construct additional evaluation datasets for five temporal dimensions to evaluate the temporal question answering capabilities of the video question answering model.

[0022] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention innovatively proposes a timing-enhanced video question-answering method and system, including multi-dimensional timing instruction data construction, multi-task timing instruction fine-tuning and multi-dimensional timing question-answering evaluation, which makes up for the shortcomings of the existing technology in timing question-answering.

[0023] 2. The multi-dimensional time series instruction data construction part of the present invention summarizes the five time series dimensions involved in time series question and answer, and designs specific data collection and construction methods for each dimension, providing a data basis for enhancing the time series question and answer capabilities of the video question and answer model.

[0024] 3. The multi-task timing instruction fine-tuning part of the present invention integrates multiple timing auxiliary tasks that do not require additional supervisory information, breaking through the limitation of data capacity, and can effectively improve the timing question answering ability of the video question answering model in multiple dimensions.

[0025] 4. The multi-dimensional temporal question-answering evaluation part of the present invention collects a large amount of video data and constructs unbiased question-answering pairs, providing an objective and comprehensive evaluation benchmark for evaluating the temporal question-answering capability of the video question-answering model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0027] Figure 1 It is a flow chart of the multi-dimensional time sequence instruction data construction method of the present invention; Figure 2 A schematic diagram of a multi-task timing instruction fine-tuning method of the present invention; Figure 3 Schematic diagram of the video question-answering model of the present invention; Figure 4 Schematic diagram of the multi-dimensional time-series question-answering evaluation method of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0030] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0031] Example 1 A time-order enhanced video question answering method, comprising: Step 1: Multi-dimensional sequential instruction data construction; Figure 1 As shown: Clarify and define the five time series dimensions, establish a data collection and screening process, and ensure that the data preparation for each time series dimension is complete; Among them, data collection and screening include the following specific steps: Design specialized tasks for each time series dimension; these tasks rely on the time series question-answering capabilities of the corresponding dimension to complete; Collect open-source video datasets with rich temporal dynamics, exclude videos that do not meet the duration requirements, and automatically generate questions and answers based on temporal annotations, where the answers include open-ended answers and multiple-choice answers; Sampling the generated questions and answers to balance the data distribution and eliminate the long tail distribution; Step 2: Fine-tune multi-task timing instructions; Construct multiple time-series auxiliary tasks and use the collected data to fine-tune the video question-answering model to enhance the model's time-series question-answering capabilities; Among them, multiple time series auxiliary tasks are constructed, including: Collect video instruction fine-tuning data, use the open source pre-trained video question-answering model to filter the data containing time-series questions and answers, and construct auxiliary tasks based on the remaining data; Construct a video frame index prediction task, randomly extract a frame from the original video frame sequence and place it at the front of the sequence, and use the video question answering model to predict the original position index of the video frame; Construct an instruction video question-answering task, splice multiple videos, and indicate the target video in natural language. The video question-answering model will answer questions based on the target video. Step 3: Multi-dimensional sequential question answering evaluation; For the five temporal dimensions, additional evaluation datasets are collected and constructed to evaluate the temporal question answering capabilities of the video question answering model.

[0032] Example 2 The difference between the time-series enhanced video question-answering method according to Embodiment 1 is that: Clarify and define five temporal dimensions, including: summarizing video temporal question-answering capabilities into five dimensions: temporal dynamics, temporal reasoning, duration, temporal positioning, and temporal order.

[0033] Design specialized tasks for each time series dimension, including: The temporal dynamic dimension concerns the dynamic changes of the content in the video over time. In view of temporal dynamics, the task form is designed to answer the movement direction of a specified object in the video. The specified object is described in natural language, such as "red car". The movement direction is expressed by direction nouns, including four categories: "upper left", "lower left", "upper right" and "lower right". Questions in the temporal reasoning dimension focus on events that will happen in the future. For temporal reasoning, the task format is designed to answer the possible actions that may occur next based on the existing content in the video, where the actions are expressed by gerund phrases; such as "pick up the basketball."

[0034] Questions in the duration dimension focus on the duration of an event. For duration, the task format is designed to answer the duration of a specified action, where the action is specified in natural language; for example, "a boy crossed the road", the duration is divided into three categories: long, medium, and short, which are respectively expressed as "longer than two-thirds of the video", "between one-third and two-thirds of the video", and "shorter than one-third of the video"; The time location dimension focuses on the time point when a specific event occurs in the video. For time location, the task form is designed to answer the occurrence time of a specified action behavior, where the action behavior is specified in natural language and the occurrence time is expressed in three categories: "located in the front of the video", "located in the middle of the video" and "located in the back of the video". Questions in the time order dimension focus on the order in which events occur. For the time order, the task format is designed to answer the correct order in which three specified actions occur, where each action is represented by a gerund phrase, such as "pick up the box, open the box, and take out the toy."

[0035] Collect open-source video datasets with rich temporal dynamics, exclude videos that do not meet the duration requirements, and automatically generate questions and answers based on temporal annotations, including: Videos and temporal annotations are collected from datasets containing rich temporal dynamics, such as ActivityNet Caption, VidOR, Ego4d Goal-Step, and Charades; videos with unsatisfactory duration are excluded; ActivityNet Caption, a large-scale video description dataset, consists of videos from various scenes, each with a corresponding natural language description, which usually involves the actions, events, or scenes shown in the video; VidOR, consists of multiple video clips, each with dynamically changing objects, and provides detailed annotation information for each video, including object category, location, timestamp, etc.; Ego4d Goal-Step, a first-person immersive video dataset, mainly focuses on first-person videos captured by wearable cameras (such as smart glasses); Charades, a multi-label dataset for video action recognition, with a special focus on human activity recognition in daily life.

[0036] Generate open-ended question-answer pairs, manually construct a few (small) open-ended question samples, use ChatGPT to generate a few (10) different open-ended questions for each temporal dimension based on the constructed open-ended question samples and task description, and generate open-ended answers based on the video temporal annotations; such as "Question: Can you determine the direction in which <child> moves relative to the video screen? Answer: Bottom left". ChatGPT is an artificial intelligence question-answering system based on a large-scale language model, which is specifically used to process and generate natural language text; Generate multiple-choice question-answer pairs, manually construct several (small) multiple-choice question samples, use ChatGPT based on the constructed multiple-choice question samples and task description, generate several (10) different multiple-choice questions for each temporal dimension, and generate one correct option and three incorrect options based on the video temporal annotation. For example, "Question: Where does the part <She combs her hair to increase the volume of her hair> appear in the video? Options: (a) in the middle of the video (b) at the front of the video (c) at the back of the video. Answer: (a)".

[0037] The generated questions and answers are sampled to balance the data distribution and eliminate long-tail distribution and single-frame bias. Including: For the three temporal dimensions with fixed answer options, namely time dynamics, duration and time location, the number of different options is balanced through sampling to keep the number of options consistent; for example, they are 450, 670 and 890 respectively, and all are sampled as 450.

[0038] For the two temporal dimensions of temporal reasoning and time order, which have no fixed answer options, the action category with the least number is used, and the number of action categories is balanced through sampling to avoid long-tail distribution; Specifically, if the frequency of some answer options in the data set is low, the oversampling method is used, that is, the number of low-frequency answer options is increased by duplicating existing question-answer pairs; if the frequency of some answer options is high, the downsampling method is used to randomly delete some question-answer pairs with high-frequency answer options to achieve the purpose of balancing the number of options.

[0039] The number of open-ended question-answer pairs and multiple-choice question-answer pairs was balanced through sampling.

[0040] Construct multiple time-series auxiliary tasks and use the collected data to fine-tune the video question answering model to enhance the model's time-series question answering capabilities, such as Figure 2 shown.

[0041] Collect video instruction fine-tuning datasets, such as Video-ChatGPT, which is a multimodal dataset that combines video and question-answering, and aims to enable the model to interact with users in natural language based on watching video content and conduct more complex video question-answering; use open source pre-trained video question-answering models, such as Qwen2.5-72B, which is an open source large-scale language model. "72B" refers to the number of parameters of the model, which is 7.2 billion. Filter the data in the video instruction fine-tuning dataset; including: Construct the input instructions of the pre-trained video question-answering model; the format is "Analyze the following conversation between the human and the assistant to determine whether the conversation reflects the understanding of the video content in the time dimension. If so, please reply <yes>. If not, please reply <no>. The conversation content is as follows: {human: [question], assistant: [answer]}", where "[question]" and "[answer]" are placeholders and need to be replaced with the question and answer content corresponding to the video in the instruction fine-tuning dataset.

[0042] The constructed input instructions and videos are input into the pre-trained video question-answering model, and each instruction has a corresponding video. According to the response of the pre-trained video question-answering model, the video question-answer pairs that do not contain temporal content are filtered out.

[0043] Construct a video frame index prediction task; including: For video question-answer pairs that do not contain temporal content, a frame is randomly selected from the original video frame sequence and placed at the front of the video frame sequence; Construct instructions and use ChatGPT to generate 10 instructions based on manually constructed examples. For each question-answer pair, randomly sample one of the instructions. The final instruction format is such as "The first frame in the sequence is a frame randomly sampled from the video, and the remaining video frames are arranged in the original order. Please predict the position of the first frame in the original video frame sequence, and then perform the following task. Question: [question]". Among them, "[question]" is a placeholder and needs to be replaced with the question content corresponding to the video.

[0044] Construct an answer. The format is "(2, 3).[answer]". Among them, "(2, 3)" represents the original position index of the sampled frame, and "[answer]" is a placeholder and needs to be replaced with the answer content corresponding to the video.

[0045] Construct instruction video question answering task; including: For video question-answer pairs that do not contain temporal content, a video is randomly extracted from the dataset, namely Video-ChatGPT, and randomly spliced ​​to the front or back of the original video; Construct instructions and use ChatGPT to generate 10 instructions based on manually constructed examples. For each question-answer pair, randomly sample one of the instructions; the final instruction format is such as "This is the splicing of two videos on the timeline. Please perform the following tasks based on the [index]th video. Question: [question]". Among them, "[index]" is the position index placeholder of the original video, and "[question]" is the placeholder corresponding to the original video content.

[0046] Directly use the answers from the original video as the answers for the spliced ​​video.

[0047] like Figure 3 As shown, the mixed constructed multi-dimensional timing instruction data, the constructed multi-task timing instruction and the remaining video instruction fine-tuning data, i.e., the video question-answering pair data containing timing content, fine-tune the video question-answering model; including: Input the instruction text into the encoding layer to obtain the text semantic features ; Input the video frame sequence into the visual encoder to obtain visual features , and use the visual adapter to map the visual features into the text semantic space to obtain the visual semantic features aligned with the text semantic features ; Semantic features of text and visual semantic features After concatenation, the input is pre-trained into a large-scale language model, and the model output is supervised and fine-tuned using cross entropy loss.

[0048] For the five time series dimensions, additional evaluation datasets are collected and constructed to evaluate the time series question answering capabilities of the video question answering model; Figure 4 As shown, including: Construct a multi-dimensional time series question answering evaluation dataset; including: The evaluation dataset is constructed using steps similar to the above; the difference is that only multiple-choice question-answer pairs are constructed, and different data sources are selected to verify the out-of-domain generalization ability.

[0049] Use three open source pre-trained video question-answering models, Qwen2-VL, LLaVA-OneVision, and MiniCPM-V2.6, as filters, input randomly sampled single-frame video content and constructed question-answer pairs, and judge whether the filters can correctly answer the questions based on the answers output by the filters. If at least two filters can correctly answer the questions, it is determined that the video question-answer pair has a single-frame deviation; Remove question-answer pairs with single-frame deviations from the dataset and rebalance the number of different answer candidates through sampling to obtain the final evaluation dataset; Input the video and question-answer pairs in the constructed evaluation dataset into the video question-answering model and obtain the answers. Compare whether the answers output by the video question-answering model are the same as the real answers. Count the proportion of questions correctly answered by the video question-answering model to the total number of questions.

[0050] Example 3 A time-order enhanced video question answering system, comprising: The multi-dimensional time series instruction data construction unit is configured to: clarify and define five time series dimensions, establish data collection and screening processes, and ensure that data preparation for each time series dimension is complete; A multi-task timing instruction fine-tuning unit is configured to: construct multiple timing auxiliary tasks and use the collected data to fine-tune the video question answering model to enhance the timing question answering capability of the model; The multi-dimensional temporal question answering evaluation unit is configured to collect and construct additional evaluation datasets for five temporal dimensions to evaluate the temporal question answering capabilities of the video question answering model.

[0051] It is understandable that the above-mentioned units can be separately or completely combined into one or several other units to constitute, or some of the units can be further divided into multiple functionally smaller units to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the video behavior segment candidate set generation system can also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

Claims

1. A time-enhanced video question-answering method, characterized in that: include: Step 1: Multi-dimensional sequential instruction data construction; Clarify and define the five time series dimensions, establish a data collection and screening process, and ensure that the data preparation for each time series dimension is complete; Among them, data collection and screening include the following specific steps: Design specialized tasks for each temporal dimension; Collect open-source video datasets with rich temporal dynamics, exclude videos that do not meet the duration requirements, and automatically generate questions and answers based on temporal annotations, where the answers include open-ended answers and multiple-choice answers; Sampling generated questions and answers; Step 2: Fine-tune multi-task timing instructions; Construct multiple time-series auxiliary tasks and use the collected data to fine-tune the video question-answering model to enhance the model's time-series question-answering capabilities; Among them, multiple time series auxiliary tasks are constructed, including: Collect video instruction fine-tuning data, use the open source pre-trained video question-answering model to filter the data containing time-series questions and answers, and construct auxiliary tasks based on the remaining data; Construct a video frame index prediction task, randomly extract a frame from the original video frame sequence and place it at the front of the sequence, and use the video question answering model to predict the original position index of the video frame; Construct an instruction video question-answering task, splice multiple videos, and indicate the target video in natural language. The video question-answering model will answer questions based on the target video. Step 3: Multi-dimensional sequential question answering evaluation; For the five temporal dimensions, additional evaluation datasets are collected and constructed to evaluate the temporal question answering capabilities of the video question answering model.

2. The method for video question answering with time sequence enhancement according to claim 1, characterized in that: Clarify and define five temporal dimensions, including: summarizing video temporal question-answering capabilities into five dimensions: temporal dynamics, temporal reasoning, duration, temporal positioning, and temporal order.

3. The method for video question answering with time sequence enhancement according to claim 1, characterized in that: Design specialized tasks for each time series dimension, including: In view of temporal dynamics, the task form is designed to answer the movement direction of a specified object in the video; the specified object is described in natural language; the movement direction is represented by a direction noun, including four categories: "upper left", "lower left", "upper right" and "lower right"; For temporal reasoning, the task format is designed to answer the possible actions that may occur next based on the existing content of the video, where the actions are represented by gerund phrases; Regarding duration, the task format is designed to answer the duration of a specified action behavior, where the action behavior is specified in natural language; the duration is divided into three categories: long, medium, and short, which are respectively expressed as "longer than two-thirds of the video", "between one-third and two-thirds of the video", and "shorter than one-third of the video"; For time positioning, the task format is designed to answer the occurrence time of a specified action behavior, where the action behavior is specified in natural language form, and the occurrence time is expressed in three categories: "located in the front of the video", "located in the middle of the video", and "located in the back of the video"; For the time sequence, the task format is designed to answer the correct order in which three specified actions occur, where each action is represented by a gerund phrase.

4. The time-enhanced video question-answering method according to claim 1, characterized in that: Collect open-source video datasets with rich temporal dynamics, exclude videos that do not meet the duration requirements, and automatically generate questions and answers based on temporal annotations, including: Collect videos and temporal annotations from datasets containing rich temporal dynamics; exclude videos whose duration does not meet the requirements; Generate open-ended question-answer pairs, manually construct several open-ended question samples, use ChatGPT to generate several different open-ended questions for each temporal dimension based on the constructed open-ended question samples and task description, and generate open-ended answers based on the video temporal annotations; Generate multiple-choice question-answer pairs, manually construct several multiple-choice question samples, use ChatGPT to generate several different multiple-choice questions for each temporal dimension based on the constructed multiple-choice question samples and task description, and generate one correct option and three incorrect options based on the video temporal annotation.

5. The time-order enhanced video question-answering method according to claim 1, characterized in that: Sample generated questions and answers; including: For the three temporal dimensions of time dynamics, duration, and time location, which have fixed answer options, the number of different options is balanced through sampling to keep the number of options consistent; For the two temporal dimensions of temporal reasoning and time order, which have no fixed answer options, the action category with the least number is used, and the number of action categories is balanced through sampling to avoid long-tail distribution; The number of open-ended question-answer pairs and multiple-choice question-answer pairs was balanced through sampling.

6. The time-order enhanced video question-answering method according to claim 1, characterized in that: Collect video instruction fine-tuning data and use the open source pre-trained video question answering model to filter the data containing time-series question answering; including: Construct input instructions for the pre-trained video question answering model; The constructed input instructions and videos are input into the pre-trained video question-answering model, and each instruction has a corresponding video. According to the response of the pre-trained video question-answering model, the video question-answer pairs that do not contain temporal content are filtered out.

7. The time-order enhanced video question-answering method according to claim 1, characterized in that: Construct a video frame index prediction task; including: For video question-answer pairs that do not contain temporal content, a frame is randomly selected from the original video frame sequence and placed at the front of the video frame sequence; Construct instructions, use ChatGPT to generate 10 instructions based on manually constructed examples, and randomly sample one of the instructions for each question-answer pair; Construct the answer.

8. The time-order enhanced video question-answering method according to claim 1, characterized in that: Construct instruction video question answering task; including: For video question-answer pairs that do not contain temporal content, a video is randomly extracted from the dataset and randomly spliced ​​to the front or back of the original video; Construct instructions, use ChatGPT to generate 10 instructions based on manually constructed examples, and randomly sample one of the instructions for each question-answer pair; Directly use the answers from the original video as the answers for the spliced ​​video.

9. A time-order enhanced video question-answering method according to any one of claims 1 to 8, characterized in that: The three parts of data, namely, the video question-answering pair data containing the time sequence content, are mixed with the constructed multi-dimensional time sequence instruction data, the constructed multi-task time sequence instruction data, and the remaining video instruction fine-tuning data, to fine-tune the video question-answering model; include: Input the instruction text into the encoding layer to obtain the text semantic features ; Input the video frame sequence into the visual encoder to obtain visual features , and use the visual adapter to map the visual features into the text semantic space to obtain the visual semantic features aligned with the text semantic features ; Semantic features of text and visual semantic features After concatenation, the pre-trained large-scale language model is input and the model output is supervised and fine-tuned using cross entropy loss; For the five time series dimensions, additional evaluation datasets are collected and constructed to evaluate the time series question answering capabilities of the video question answering model. These include: Construct a multi-dimensional time series question answering evaluation dataset; including: Use similar steps as above to construct the evaluation dataset; Use three open source pre-trained video question-answering models, Qwen2-VL, LLaVA-OneVision, and MiniCPM-V2.6, as filters, input randomly sampled single-frame video content and constructed question-answer pairs, and judge whether the filters can correctly answer the questions based on the answers output by the filters. If at least two filters can correctly answer the questions, it is determined that the video question-answer pair has a single-frame deviation; Remove question-answer pairs with single-frame deviations from the dataset and rebalance the number of different answer candidates through sampling to obtain the final evaluation dataset; Input the video and question-answer pairs in the constructed evaluation dataset into the video question-answering model and obtain the answers. Compare whether the answers output by the video question-answering model are the same as the real answers. Count the proportion of questions correctly answered by the video question-answering model to the total number of questions.

10. A time-enhanced video question-answering system, characterized in that: include: The multi-dimensional time series instruction data construction unit is configured to: clarify and define five time series dimensions, establish data collection and screening processes, and ensure that data preparation for each time series dimension is complete; A multi-task timing instruction fine-tuning unit is configured to: construct multiple timing auxiliary tasks and use the collected data to fine-tune the video question answering model to enhance the timing question answering capability of the model; The multi-dimensional temporal question answering evaluation unit is configured to collect and construct additional evaluation datasets for five temporal dimensions to evaluate the temporal question answering capabilities of the video question answering model.

Images