Method and system for automatically constructing long video question and answer data set

Abstract

The invention provides a method and system for automatically constructing a long video question and answer data set, and the method comprises the steps: processing a to-be-processed video, and obtaining a video tree; extracting each sub-tree in the video tree, and respectively inputting the sub-trees into a pre-configured video understanding model to obtain understanding data corresponding to each sub-tree; and according to all the understanding data, generating a question and answer data set corresponding to the to-be-processed video. According to the method and system for automatically constructing the long video question and answer data set, the large-scale and long-time-span high-quality question and answer data set is automatically generated through a machine.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for automatically constructing a long video question-answering data set. Background Art

[0002] With the rapid development of artificial intelligence technology, chatbots have emerged; chatbots are computer programs that communicate through dialogue or text. They can simulate human dialogue and pass the Turing test. Chatbots can be used for practical purposes, such as customer service or information acquisition. Some chatbots are equipped with natural language processing systems, but most simple systems only capture the input keywords and then find the most appropriate response sentence from the database. As the leader among chatbots, video chatbots are end-to-end chatbots that can read pictures and understand videos. The technology is relatively complex; especially in the training stage, the first technical problem that needs to be solved is to generate a question-and-answer dataset that can be used to train a neural network model after video understanding.

[0003] Most existing methods (such as VideoChat) use large language models (LLM, such as GPT-4) to automatically build video datasets, but they mainly focus on short videos. Due to the limited input length of LLM, the dense text description of long videos can easily lead to too much information, making it difficult for the model to effectively understand, and thus unable to generate high-quality, long-span question-answering data. Summary of the invention

[0004] One of the objectives of the present invention is to provide a method and system for automatically constructing a long video question-answering dataset to solve the above-mentioned technical problems.

[0005] An embodiment of the present invention provides a method for automatically constructing a long video question-answering dataset, including:

[0006] Process the video to be processed and obtain the video tree;

[0007] Extract each subtree in the video tree, input them into the pre-configured video understanding model respectively, and obtain the understanding data corresponding to each subtree;

[0008] Based on all the understanding data, generate the question-answering dataset corresponding to the video to be processed.

[0009] Preferably, the steps of constructing the video tree are as follows:

[0010] Densely sample the video to be processed to obtain multiple frames of images;

[0011] Provide subtitles for each frame of the picture;

[0012] According to the subtitle description, summarize it into an event description;

[0013] Aggregate event descriptions to form video clip descriptions;

[0014] Each video clip description is regarded as a subtree in the video tree, the event description corresponding to the video clip is regarded as the lower node of the subtree, and the subtitle description corresponding to the event description is regarded as the lower node.

[0015] Preferably, the frequency of dense sampling is 1FPS.

[0016] Preferably, the video understanding model comprises a large language model.

[0017] Preferably, the method for automatically constructing a long video question answering dataset further includes:

[0018] Conduct security verification on the acquisition device and transmission path of the video to be processed;

[0019] When the security verification fails, the construction process of the video tree is analyzed according to the pre-configured first analysis rule to determine the first true evaluation data; the correlation between the event descriptions between each adjacent subtree in the video tree is analyzed to determine the second true evaluation data; the correlation between the subtitle descriptions between each adjacent event description is analyzed to determine the third true evaluation data; the first true evaluation data, the second true evaluation data, and the third true evaluation data are comprehensively analyzed to obtain the verification value of the true evaluation verification;

[0020] When the number of question-and-answer datasets that have passed security verification does not meet the requirements of model training, question-and-answer datasets that have not passed security are selected as question-and-answer datasets for model training in descending order of verification values.

[0021] The present invention also provides a system for automatically constructing a long video question and answer dataset, comprising: a processing module, an understanding module and a generation module; wherein the processing module processes the video to be processed to obtain a video tree; the understanding module extracts each subtree in the video tree, respectively inputs them into a pre-configured video understanding model, and obtains understanding data corresponding to each subtree; the generation module generates a question and answer dataset corresponding to the video to be processed based on all the understanding data.

[0022] Preferably, the processing module includes: a sampling unit, a subtitle description unit, an event aggregation unit, an event aggregation unit and a construction unit; wherein the sampling unit densely samples the video to be processed to obtain multiple frames of images; the subtitle description unit performs subtitle description on each frame of the image; the event aggregation unit aggregates the event descriptions into event descriptions based on the subtitle descriptions; the event aggregation unit aggregates the event descriptions to form a video clip description; the construction unit describes each video clip as a subtree in the video tree, the event description corresponding to the video clip as a lower-level node of the subtree, and the subtitle description corresponding to the event description as a lower-level node.

[0023] Preferably, the frequency of dense sampling is 1FPS.

[0024] Preferably, the video understanding model comprises a large language model.

[0025] Preferably, the system for automatically constructing a long video question and answer dataset also includes: a security verification module, a real evaluation module and a data selection module; wherein the security verification module performs security verification on the acquisition device and transmission path of the video to be processed; when the security verification fails, the real evaluation module analyzes the construction process of the video tree according to the pre-configured first analysis rule to determine the first real evaluation data; the real evaluation module analyzes the correlation between the event descriptions between each adjacent subtree in the video tree to determine the second real evaluation data; the real evaluation module analyzes the correlation between the subtitle descriptions between each adjacent event description to determine the third real evaluation data; the real evaluation module comprehensively analyzes the first real evaluation data, the second real evaluation data, and the third real evaluation data to obtain a verification value for the real evaluation verification; when the number of question and answer datasets that have passed the security verification does not meet the requirements of model training, the data selection module selects the question and answer datasets that have not passed the security according to the verification values ​​in descending order as the question and answer datasets for model training.

[0026] The present invention has the following advantages:

[0027] 1. The video tree can accommodate videos of almost infinite length, and simplify, summarize, and compress the core content of the video in the video tree, and describe the video content (segments, events, frames) and their corresponding clear time annotations at different time scales.

[0028] 2. When generating Long-Form QA Generat i on video question-answering data, the video content input to LLM is a subtree of the video tree, such as segment 1 (event 1, ..., event E), segment 2 (event 1, ..., event E). This ensures that it does not exceed the input length limit of LLMs, and at the same time, it can obtain the core description of the video content over a long time scale, so as to understand, filter, summarize and reason about this information, and finally generate high-quality question-answering data that is closely aligned with the video content and has a long time span.

[0029] 3. Due to the good structuring of the video tree and the flexibility of video content representation, video question and answer generation can become very diverse and scalable. It can not only be applied to video question and answer data generation, but also extended to automatic data generation in the fields of multi-image joint understanding, long audio understanding, and multi-modal joint understanding, while having the advantages mentioned above.

[0030] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0031] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0033] Figure 1 A schematic diagram of a method for automatically constructing a long video question-answering dataset in an embodiment of the present invention;

[0034] Figure 2 A structural diagram of a video tree in an embodiment of the present invention;

[0035] Figure 3 A schematic diagram of a method for automatically constructing a long video question-answering dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0037] The embodiment of the present invention provides a method for automatically constructing a long video question answering dataset. Figure 1 As shown, including:

[0038] Step 1: Process the video to be processed and obtain the video tree;

[0039] Step 2: extract each subtree in the video tree, input them into the pre-configured video understanding model respectively, and obtain the understanding data corresponding to each subtree;

[0040] Step 3: Based on all the understanding data, generate a question-answering dataset corresponding to the video to be processed.

[0041] The steps of constructing the video tree are as follows: densely sample the video to be processed to obtain multiple frames; describe each frame with subtitles; summarize the event description based on the subtitle description; aggregate the event description to form a video clip description; take each video clip description as a subtree in the video tree, the event description corresponding to the video clip as the lower node of the subtree, and the subtitle description corresponding to the event description as the next node. The frequency of dense sampling is 1FPS. The video understanding model includes a large language model.

[0042] The working principle and beneficial effects of the above technical solution are:

[0043] Using powerful basic models of each modality to structure information from modalities such as images, videos, and voices into text, and then using LLM's powerful text processing capabilities to understand, filter, summarize and reason about this information, generating multimodal data is already a relatively mature existing technology, but there is still very little work in the field of automatically generating video question-answering datasets, and only short videos, relatively simple, and low-quality video question-answering data can be generated. The main reason for this is that the input length of LLM is limited (generally 8k tokens), and the dense text description of long videos input will make the information too lengthy, resulting in LLM's inability to process and understand the core content of the video, and thus unable to generate high-quality question-answering data with a long time span. That is, most existing methods (such as VideoChat) use large language models (LLM, such as GPT-4) to automatically construct video datasets, but they are mainly focused on short videos. Due to the limited input length of LLM, the dense text description of long videos easily leads to too much information, making it difficult for the model to effectively understand, and thus unable to generate high-quality, long-span question-answering data.

[0044] The core of the present invention is to structure the video content into a tree step by step, such as Figure 2As shown. First, the video is densely sampled (1fps), and then the latest visual language model Vi s ion Language Model ls, VLMs, is used to caption each frame of the picture. LLM is used to aggregate them into event descriptions. Its time range is divided by LLM according to the relevance of the video content represented by the subtitle description. Finally, the video is divided into multiple events, and each event contains a series of frames of related events. Going up, the event description is further aggregated into video segment descriptions. Finally, the video is divided into multiple segments, each of which contains a series of related events. The video content is structured as a tree as shown in the figure below. The root node is the video tree, and the leaf nodes correspond to the video content descriptions at each level (segment, event, frame). After the video tree is generated, when performing the Long-Form QA generation for video question and answer data, the video content input to the LLM is a subtree of the video tree, such as segment 1 (event 1, ..., event E), segment 2 (event 1, ..., event E); the core description of the video content over a long time scale can be obtained, so as to understand, filter, summarize and infer this information, and finally generate high-quality question and answer data that is closely aligned with the video content and has a long time span. Specific operations may include: statistics of the understanding data (question and answer data) output by the LLM corresponding to each subtree are input into a pre-configured LLM, and the LLM understands, filters, summarizes and infers this information on the question and answer data; and obtains the question and answer data corresponding to the video to be processed. Taking an actual video as an example, the video has the following video clips: a dog playing, a dog and a person playing interactively, a dog and two people playing interactively, and two people playing interactively; the events corresponding to a dog playing include: a dog running around, a dog pushing a ball, etc.; the subtitle descriptions of the pictures include: there is a dog in the picture, there is a dog in the picture and the dog is pushing a ball, etc.

[0045] In one embodiment, the method for automatically constructing a long video question answering dataset further includes:

[0046] Before executing step 1, the length of the video to be processed is verified to verify whether it is a long video; that is, whether its length exceeds 8k token; step 1 is executed only when the verification is passed, otherwise it is not executed.

[0047] LLM summarizes subtitle descriptions by first calculating the similarity between subtitle descriptions corresponding to adjacent images. When the similarity is greater than a preset similarity threshold, they are regarded as the same event. Then, the subtitle descriptions of the images corresponding to the same event are summarized and reasoned to form an event description. For the aggregation operation of event descriptions, the relevance (association value) of adjacent events is mainly judged by a pre-configured event association judgment library. When the association value is greater than a preset association threshold, the adjacent events are regarded as the same video clip. Among them, the event association judgment library analyzes the relevance as follows: based on the pre-configured keywords in the keyword extraction library corresponding to the event association judgment library, keywords are extracted from the event description, and a keyword set is constructed based on the extracted keywords; the keyword set is matched with the standard word set in the event association judgment library, and the association value corresponding to the matched standard word set is retrieved;

[0048] When aggregating events to generate video segments, there may be a situation where the video segment exceeds the length limit. To avoid exceeding the limit, the middle part with the lowest correlation value in each adjacent event in the video segment can be segmented until each segment meets the length limit.

[0049] In one embodiment, the method for automatically constructing a long video question answering dataset further includes:

[0050] Conduct security verification on the acquisition device and transmission path of the video to be processed;

[0051] When the security verification fails, the construction process of the video tree is analyzed according to the pre-configured first analysis rule to determine the first true evaluation data; the correlation between the event descriptions between each adjacent subtree in the video tree is analyzed to determine the second true evaluation data; the correlation between the subtitle descriptions between each adjacent event description is analyzed to determine the third true evaluation data; the first true evaluation data, the second true evaluation data, and the third true evaluation data are comprehensively analyzed to obtain the verification value of the true evaluation verification;

[0052] When the number of question-and-answer datasets that have passed security verification does not meet the needs of model training, select question-and-answer datasets that have not passed security as question-and-answer datasets for model training based on the verification value from large to small. For example: it is necessary to extract question-and-answer datasets corresponding to N (any value from 200 to 1000) videos as data for model training; if the number of videos that have passed security verification is NM (M is also any value from 200 to 1000, and N is greater than M), then it is necessary to extract question-and-answer datasets corresponding to M videos that have not passed security verification, and extract them in order from large to small based on the verification value when extracting;

[0053] Among them, security verification includes: first, determine whether the acquiring device is a device registered in the system (the device identification number can be used to query the device whitelist configured in the system. If it is, the verification passes, otherwise it fails), and secondly, verify the transmission path to ensure whether the data is secure; the verification of the transmission path is to extract features of each node on the transmission path and the path itself, and the extracted features include: the number of nodes, the pre-configured security value of each node, etc.; then use the extracted features as index items to retrieve the corresponding security value from the pre-configured security analysis library; when the security value is greater than the preset security threshold, the verification passes; otherwise it fails.

[0054] If the security verification fails, it means that the video is not directly from the real space, and there may be processing operations such as editing. How to analyze the video itself and indirectly determine whether it has been processed? During the analysis process, the first ratio between the number of pictures ignored in the construction of the video tree and the total number of pictures, and the second ratio between the number of events ignored and the total number of events are queried in the first analysis table to determine the first score value of the first real evaluation data; wherein, the first ratio and the second ratio in the first analysis table are associated with the first score value; the first ratio and the second ratio are output correspondingly when the summary event and the aggregation event process are output respectively; the representation of the correlation is averaged, and then the second analysis table is queried based on the average value to obtain the second score value of the second real evaluation data; wherein, the average value in the second analysis table is correspondingly associated with the second score value and is pre-configured; the correlation between the subtitle descriptions of each adjacent event description, that is, the similarity between the two, the similarity is averaged, and then the third analysis table is queried to obtain the third score value of the third real evaluation data; the verification value is the sum of the first score value, the second score value and the third score value.

[0055] The present invention also provides a system for automatically constructing a long video question-answering dataset. Figure 3 As shown, it includes: a processing module 1, an understanding module 2 and a generation module 3; wherein, the processing module 1 processes the video to be processed to obtain a video tree; the understanding module 2 extracts each subtree in the video tree, and inputs them into a pre-configured video understanding model respectively to obtain the understanding data corresponding to each subtree; the generation module 3 generates a question and answer data set corresponding to the video to be processed based on all the understanding data.

[0056] Among them, the processing module includes: a sampling unit, a subtitle description unit, an event aggregation unit, an event aggregation unit and a construction unit; wherein the sampling unit densely samples the video to be processed to obtain multiple frames of pictures; the subtitle description unit describes each frame of the picture with subtitles; the event aggregation unit aggregates the event descriptions into event descriptions based on the subtitle descriptions; the event aggregation unit aggregates the event descriptions to form a video clip description; the construction unit describes each video clip as a subtree in the video tree, the event description corresponding to the video clip as a lower-level node of the subtree, and the subtitle description corresponding to the event description as a lower-level node.

[0057] Among them, the frequency of dense sampling is 1FPS.

[0058] Among them, the video understanding model includes a large language model.

[0059] In one embodiment, a system for automatically constructing a long video question-and-answer dataset also includes: a security verification module, a real evaluation module, and a data selection module; wherein the security verification module performs security verification on the acquisition device and transmission path of the video to be processed; when the security verification fails, the real evaluation module analyzes the construction process of the video tree according to a pre-configured first analysis rule to determine the first real evaluation data; the real evaluation module analyzes the correlation between the event descriptions between each adjacent subtree in the video tree to determine the second real evaluation data; the real evaluation module analyzes the correlation between the subtitle descriptions between each adjacent event description to determine the third real evaluation data; the real evaluation module comprehensively analyzes the first real evaluation data, the second real evaluation data, and the third real evaluation data to obtain a verification value for the real evaluation verification; when the number of question-and-answer datasets that have passed the security verification does not meet the requirements of the model training, the data selection module selects the question-and-answer datasets that have not passed the security verification in descending order according to the verification values ​​as the question-and-answer datasets for model training.

[0060] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for automatically constructing a long video question-answering dataset, characterized in that: include: Process the video to be processed and obtain the video tree; Extract each subtree in the video tree, input them into the pre-configured video understanding model respectively, and obtain the understanding data corresponding to each subtree; Based on all the understanding data, generate the question-answering dataset corresponding to the video to be processed.

2. The method for automatically constructing a long video question-answering dataset as claimed in claim 1, characterized in that: The steps to construct the video tree are as follows: Densely sample the video to be processed to obtain multiple frames of images; Provide subtitles for each frame of the picture; According to the subtitle description, summarize it into an event description; Aggregate event descriptions to form video clip descriptions; Each video clip description is regarded as a subtree in the video tree, the event description corresponding to the video clip is regarded as the lower node of the subtree, and the subtitle description corresponding to the event description is regarded as the lower node.

3. The method for automatically constructing a long video question-answering dataset as claimed in claim 2, characterized in that: The frequency of dense sampling is 1FPS.

4. The method for automatically constructing a long video question-answering dataset according to claim 1, characterized in that: Video understanding models include large language models.

5. The method for automatically constructing a long video question-answering dataset as claimed in claim 1, characterized in that: Also includes: Conduct security verification on the acquisition device and transmission path of the video to be processed; When the security verification fails, analyzing the video tree construction process according to a pre-configured first analysis rule to determine first true evaluation data; Analyze the correlation between event descriptions between adjacent subtrees in the video tree to determine second true evaluation data; analyze the correlation between subtitle descriptions between adjacent event descriptions to determine third true evaluation data; Comprehensively analyzing the first real evaluation data, the second real evaluation data, and the third real evaluation data to obtain a verification value of the real evaluation verification; When the number of question-and-answer datasets that have passed security verification does not meet the requirements of model training, question-and-answer datasets that have not passed security are selected as question-and-answer datasets for model training in descending order of verification values.

6. A system for automatically constructing a long video question-answering dataset, characterized in that: include: Processing module, understanding module and generation module; the processing module processes the video to be processed to obtain the video tree; the understanding module extracts each subtree in the video tree, inputs them into the pre-configured video understanding model respectively, and obtains the understanding data corresponding to each subtree; the generation module generates the question and answer data set corresponding to the video to be processed based on all the understanding data.

7. The system for automatically constructing a long video question-answering dataset as claimed in claim 6, characterized in that: The processing module includes: a sampling unit, a subtitle description unit, an event aggregation unit, an event aggregation unit and a construction unit; wherein the sampling unit densely samples the video to be processed to obtain multiple frames of images; the subtitle description unit describes each frame of the image in subtitles; the event aggregation unit aggregates the subtitle description into an event description based on the subtitle description; the event aggregation unit aggregates the event description to form a video segment description; the construction unit describes each video segment as a subtree in the video tree, the event description corresponding to the video segment as a lower-level node of the subtree, and the subtitle description corresponding to the event description as a lower-level node.

8. The system for automatically constructing a long video question-answering dataset as claimed in claim 7, characterized in that: The frequency of dense sampling is 1FPS.

9. The system for automatically constructing a long video question-answering dataset as claimed in claim 6, characterized in that: Video understanding models include large language models.

10. The system for automatically constructing a long video question-answering dataset according to claim 6, characterized in that: Also includes: A security verification module, a true evaluation module and a data selection module; wherein the security verification module performs security verification on the acquisition device and transmission path of the video to be processed; when the security verification fails, the true evaluation module analyzes the construction process of the video tree according to the pre-configured first analysis rule to determine the first true evaluation data; the true evaluation module analyzes the correlation between the event descriptions between each adjacent subtree in the video tree to determine the second true evaluation data; the true evaluation module analyzes the correlation between the subtitle descriptions between each adjacent event description to determine the third true evaluation data; the true evaluation module comprehensively analyzes the first true evaluation data, the second true evaluation data and the third true evaluation data to obtain the verification value of the true evaluation verification; when the number of question and answer data sets that have passed the security verification does not meet the requirements of the model training, the data selection module selects the question and answer data sets that have not passed the security according to the verification values ​​in descending order as the question and answer data sets for model training.

Images