Video question answering method and system, task analysis method and model training method

By decomposing complex problems into atomic tasks and using atomic task analysis library for analysis, combined with the comprehensive reasoning of video question-and-answer model, the problem of inaccurate answers in the existing technology in complex VideoQA scenarios is solved, and the in-depth analysis of video content is achieved. The accuracy of the answer is improved.

CN118917416APending Publication Date: 2024-11-08ALIPAY (HANGZHOU) INFORMATION TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202410922451.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing video Q&A technology is difficult to provide accurate answers in complex VideoQA scenarios, and cannot meet users' needs for in-depth analysis of video content.

Method used

By decomposing the questions to be answered into candidate atomic video question-and-answer tasks, and using the atomic task analysis library to analyze each task separately, generate atomic video analysis answers, and then the target video question-and-answer model performs comprehensive reasoning to output the final answer.

Benefits of technology

It significantly reduces the difficulty and complexity of video analysis, and can analyze video content more accurately and in-depth, and is suitable for complex VideoQA scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917416A_ABST
    Figure CN118917416A_ABST
Patent Text Reader

Abstract

One or more embodiments of the invention provide a video question and answer method and system, a task analysis method and a model training method, and the video question and answer method comprises the steps: inputting a to-be-analyzed video, a to-be-answered question and a candidate atomic video question and answer task set into a target video question and answer model, guiding the target video question-answering model to decompose the to-be-answered question into a to-be-answered atomic video question-answering task set based on the to-be-analyzed video; for the to-be-analyzed video, respectively determining an atomic video analysis answer corresponding to each candidate atomic video question and answer task in the to-be-answered atomic video question and answer task set; and inputting the determined atomic video analysis answer to the target video question-answering model, so that the target video question-answering model outputs a final video analysis answer corresponding to the question to be answered according to the obtained atomic video analysis answer.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of video question-answering technology, and in particular, to a video question-answering method, system, task parsing method, and model training method. Background Art

[0002] Video Question Answering (VideoQA) is an interdisciplinary research direction involving multiple fields such as computer vision, natural language processing, and multimodal information fusion. It aims to enable machine learning models to understand video content and answer questions raised by humans in natural language.

[0003] In the related art, the video question-answering model obtained through training can understand the questions raised by users, analyze the video content accordingly, and then output the corresponding answers. However, the methods in the related art can only provide satisfactory answers in the scenario of simple VideoQA. The so-called simple VideoQA means that the questions raised by users are relatively simple and superficial, and only a relatively superficial analysis of the video is required - for example, only by observing a single video frame - to get an accurate answer.

[0004] However, in reality, it is often not a simple VideoQA scenario, but more about complex VideoQA scenarios, that is, the questions raised by users have a certain depth, and in-depth analysis of the video is required to get a satisfactory answer. However, related technologies cannot meet the user needs in complex VideoQA scenarios. Summary of the invention

[0005] In view of this, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of one or more embodiments of this specification, a video question-answering method is proposed, including:

[0007] Inputting a video to be parsed, a question to be answered, and a set of candidate atomic video question-answering tasks into a target video question-answering model, and guiding the target video question-answering model to decompose the question to be answered into a set of atomic video question-answering tasks to be answered based on the video to be parsed, wherein the set of atomic video question-answering tasks to be answered includes at least one candidate atomic video question-answering task from the set of candidate atomic video question-answering tasks;

[0008] For the video to be parsed, respectively determine the atomic video parsing answer corresponding to each candidate atomic video question-answering task in the set of atomic video question-answering tasks to be answered;

[0009] The determined atomic video parsing answer is input into the target video question-answering model, so that the target video question-answering model outputs a final video parsing answer corresponding to the question to be answered based on the obtained atomic video parsing answer.

[0010] According to a second aspect of one or more embodiments of the present specification, an atomic task parsing method for video question answering is proposed, which is applied to an atomic task parsing library corresponding to any candidate atomic video question answering task in a candidate atomic video question answering task set; wherein each candidate atomic video question answering task in the candidate atomic video question answering task set has a corresponding atomic task parsing library; the method comprises:

[0011] A parsing library call request is received, wherein the parsing library call request is used to instruct the corresponding atomic task parsing library to parse the video to be parsed to complete any of the candidate atomic video question-answering tasks; wherein the parsing library call request is initiated when the target video question-answering model decomposes the question to be answered into a set of atomic video question-answering tasks to be answered based on the video to be parsed, and the set of atomic video question-answering tasks to be answered includes any of the candidate atomic video question-answering tasks;

[0012] Inputting the video to be parsed into a predefined video question answering model, and guiding the predefined video question answering model to provide an output corresponding to any candidate atomic video question answering task based on the video to be parsed;

[0013] According to the output of the predefined video question-answering model, the atomic video parsing answer corresponding to any of the candidate atomic video question-answering tasks is returned, so that the target video question-answering model outputs a final video parsing answer corresponding to the question to be answered based on the obtained atomic video parsing answer.

[0014] According to a third aspect of one or more embodiments of this specification, a video question-answering collaborative system is proposed, including:

[0015] An atomic task parsing library corresponding to each candidate atomic video question-answering task in the candidate atomic video question-answering task set, for parsing the provided video through a predefined video question-answering model to complete the corresponding atomic video question-answering task;

[0016] Model proxy service for:

[0017] Inputting a to-be-analyzed video, a to-be-answered question, and the candidate atomic video question-answering task set into a target video question-answering model, and guiding the target video question-answering model to decompose the to-be-answered question into an atomic video question-answering task set based on the to-be-analyzed video, wherein the atomic video question-answering task set includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set;

[0018] Based on the set of atomic video question-answering tasks to be answered, a matching atomic task parsing library is called to obtain corresponding atomic video parsing answers; and,

[0019] The determined atomic video parsing answer is input into the target video question-answering model, so that the target video question-answering model outputs a final video parsing answer corresponding to the question to be answered based on the obtained atomic video parsing answer.

[0020] According to a fourth aspect of one or more embodiments of this specification, a video question-answering system is proposed, including:

[0021] Target video question answering model;

[0022] A video question and answer collaborative system as described in the third aspect.

[0023] According to a fifth aspect of one or more embodiments of this specification, a method for training a video question answering model is proposed, including:

[0024] Obtain a training set, wherein each training sample in the training set includes a sample video, a sample question, and a sample atomic video question-answering task set to be answered, wherein the sample atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task from a candidate atomic video question-answering task set;

[0025] Input the training set and the candidate atomic video question-answering task set into the target video question-answering model, and obtain the actual atomic video question-answering task set to be answered outputted by the target video question-answering model for each training sample, wherein the actual atomic video question-answering task set to be answered is obtained by decomposing the sample questions in the corresponding training samples based on the sample videos in the corresponding training samples by the target video question-answering model, and the actual atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set;

[0026] The target video question answering model is iteratively updated according to the difference between the actual set of atomic video question answering tasks to be answered and the sample set of atomic video question answering tasks to be answered.

[0027] According to a sixth aspect of one or more embodiments of this specification, a method for training a video question answering model is proposed, including:

[0028] Obtaining a training set, wherein each training sample in the training set includes a sample video, a sample question, and a sample final video parsing answer;

[0029] Inputting the training set and the candidate atomic video question-answering task set into the target video question-answering model, and obtaining the to-be-answered atomic video question-answering task set outputted by the model for each training sample, wherein the to-be-answered atomic video question-answering task set is obtained by decomposing the sample questions in the corresponding training samples based on the sample videos in the corresponding training samples by the target video question-answering model, and the to-be-answered atomic video question-answering task set contains at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set;

[0030] For each sample video in each training sample, the atomic video parsing answers corresponding to each candidate atomic video question-answering task in the corresponding set of atomic video question-answering tasks to be answered are determined respectively, and the determined atomic video parsing answers are input into the target video question-answering model, so that the target video question-answering model outputs the actual final video parsing answers corresponding to the sample questions in the corresponding training sample;

[0031] The target video question-answering model is iteratively updated according to a predefined loss function; wherein the loss function includes an answer difference term, and the answer difference term is the difference between the actual final video parsing answer and the sample final video parsing answer.

[0032] According to the seventh aspect of one or more embodiments of this specification, an electronic device is proposed, comprising: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in the first aspect, the second aspect, the fifth aspect or the sixth aspect by running the executable instructions.

[0033] According to the eighth aspect of one or more embodiments of this specification, a computer-readable storage medium is proposed, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in the first aspect, the second aspect, the fifth aspect or the sixth aspect are implemented.

[0034] According to a ninth aspect of one or more embodiments of the present specification, a computer program product is proposed, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method described in the first aspect, the second aspect, the fifth aspect or the sixth aspect.

[0035] It can be seen from the above embodiments that this specification, by defining candidate atomic video question-answering tasks, can guide the target video question-answering model to decompose the originally more complex questions to be answered into relatively simple candidate atomic video question-answering tasks, which is equivalent to disassembling the multiple dimensions that the questions to be answered face at the same time, so that when the videos to be analyzed are further analyzed for each candidate atomic video question-answering task, compared with directly analyzing the videos to be analyzed based on the questions to be answered, the difficulty and complexity of video analysis can be significantly reduced, and the video content contained in the video to be analyzed can be analyzed more accurately and deeply. Furthermore, the target video question-answering model performs comprehensive reasoning based on the answers to the atomic video analyses, which can make full use of the video content analyzed by each candidate atomic video question-answering task from its corresponding dimensions to obtain more accurate answers, and is therefore better suited for complex VideoQA scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of a video frame included in a video for video question answering provided by an exemplary embodiment.

[0037] Figure 2 It is a schematic diagram of the architecture of a video question-answering system provided by an exemplary embodiment.

[0038] Figure 3 It is a flowchart of a video question-and-answer method provided by an exemplary embodiment.

[0039] Figure 4 A schematic diagram of a path obtained through path planning provided by an exemplary embodiment.

[0040] Figure 5 It is a flowchart of an atomic task parsing method for video question and answering provided by an exemplary embodiment.

[0041] Figure 6 It is a schematic diagram of the architecture of a video question-answering collaborative system provided by an exemplary embodiment.

[0042] Figure 7 It is a flowchart of a method for training a video question-answering model provided by an exemplary embodiment.

[0043] Figure 8 It is a flowchart of another method for training a video question-answering model provided by an exemplary embodiment.

[0044] Fig. 9 It is a structural schematic diagram of a device provided by an exemplary embodiment.

[0045] Fig.10 It is a block diagram of a video question-and-answer device provided by an exemplary embodiment.

[0046] Fig.11 It is a block diagram of an atomic task parsing device for video question and answering provided by an exemplary embodiment.

[0047] Fig.12 It is a block diagram of a training device for a video question-answering model provided by an exemplary embodiment.

[0048] Fig.13 It is a block diagram of another training device for a video question-answering model provided by an exemplary embodiment. DETAILED DESCRIPTION

[0049] As mentioned above, the relevant technology can only provide satisfactory answers through the video question answering model in the simple VideoQA scenario. When the questions raised by users are more complex, that is, in the complex VideoQA scenario, the answers provided by the video question answering model are often of poor quality.

[0050] For example, in a VideoQA scenario, Figure 1 A video frame in the video in question is shown. When the question provided by the user is relatively simple, for example, the question is "What types of objects are contained in the video? (A) Ship (B) Car (C) Sailboat (D) Airplane", then the video question answering model's video parsing task only involves video object recognition, which only requires a simple analysis such as Figure 1 The video frame shown in the figure can be parsed to identify the type of object in the red box, and then give the answer "(C) sailboat". When the user's question is more complex, such as "What action is the person in the video doing? (A) Blowing (B) Sailing (C) Swimming (D) Fighting", the video question answering model only uses the following example to answer the question: Figure 1 The video frame shown, or other video frames contained in the video, generally have a lower probability of giving the correct answer.

[0051] To this end, this specification provides a video question-answering solution, which, in the process of performing VideoQA through a target video question-answering model, does not only rely on the video parsing capability of the target video question-answering model itself, but also through reasonable task decomposition and scheduling, enables the target video question-answering model to better adapt to complex VideoQA scenarios and provide satisfactory answers.

[0052] The technical solution of this specification will be described in detail below with reference to the accompanying drawings.

[0053] Figure 2 FIG. 1 is a schematic diagram of the architecture of a video question-answering system provided by an exemplary embodiment. Figure 2 As shown, the system may include a target video question answering model 21, a model proxy service 22, an atomic task parsing library 23-24, etc.

[0054] The target video question-answering model 21 can be deployed on a physical server of an independent host, or can be deployed on a virtual server carried by a host cluster. The target video question-answering model 21 can be any type of model for implementing VideoQA, and this specification does not limit this. For example, the target video question-answering model 21 can be a large multimodal model (Large Multimodal Models, referred to as LMM or LMMs). Through deep learning technology, especially by training large-scale neural network models (for example, containing millions to billions of parameters), LMM can not only accurately understand the questions asked by users in natural language, but also capture and express very complex multimodal relationships, and then can process and fuse data from different sensory inputs, such as text, images, audio and video, making it relatively more suitable for VideoQA scenarios.

[0055] The model proxy service 22 is a service proposed in this specification for complex VideoQA scenarios. It can enhance the performance of the target video question-answering model 21 in complex VideoQA scenarios by reasonably scheduling the target video question-answering model 21 and even the atomic task parsing library 23-24. For example, the model proxy service 22 can guide the target video question-answering model 21 to decompose the question to be answered, such as decomposing it into one or more candidate atomic video question-answering tasks, and then schedule the atomic task parsing library 23-24 to process each candidate atomic video question-answering task respectively, and then guide the target video question-answering model 21 to perform comprehensive reasoning based on the atomic video parsing answers corresponding to each candidate atomic video question-answering task, so as to obtain the final video parsing answer.

[0056] The atomic task parsing libraries 23-24, etc. correspond one-to-one to the pre-established candidate atomic video question-and-answer tasks. Each atomic task parsing library can be deployed on a physical server of an independent host, or can be deployed on a virtual server carried by a host cluster. Among them, different atomic task parsing libraries can be deployed in the same physical server or virtual server, or in different physical servers or virtual servers, and this specification does not limit this. Under the scheduling of the model proxy service 22, the atomic task parsing libraries 23-24, etc. can execute the corresponding candidate atomic video question-and-answer tasks and generate corresponding atomic video parsing answers. As mentioned above, these atomic video parsing answers can be used for comprehensive reasoning of the target video question-and-answer model 21 to obtain the final video parsing answer.

[0057] Regarding the target video question-answering model 21, the model proxy service 22, the atomic task parsing library 23-24, etc. in the video question-answering solution of this specification, the functions implemented by each and the coordination logic between them can be referred to as follows: Figure 3-5The embodiments shown and the related description.

[0058] Figure 3 FIG. 1 is a flow chart of a video question-answering method provided by an exemplary embodiment. Figure 3 As shown, this method can be applied to Figure 2 The model proxy service 22 shown includes the following steps:

[0059] Step 302: input the video to be analyzed, the question to be answered, and the set of candidate atomic video question and answer tasks into the target video question and answer model, and guide the target video question and answer model to decompose the question to be answered into the set of atomic video question and answer tasks to be answered based on the video to be analyzed, wherein the set of atomic video question and answer tasks to be answered includes at least one candidate atomic video question and answer task from the set of candidate atomic video question and answer tasks.

[0060] In the VideoQA scenario, the target video question-answering model needs to parse the video to answer questions raised by users or other objects based on the parsing results, and output the corresponding answers. The above-mentioned video to be parsed is the video that needs to be parsed by the target video question-answering model, and the question to be answered is the question that needs to be answered by the target video question-answering model based on the parsing results of the video to be parsed. For example, Figure 1 In the embodiment shown, Figure 1 It can be a certain video frame of the video to be analyzed, and the question to be answered can be "What action is taken by the person in the video? (A) Blowing (B) Sailing (C) Swimming (D) Fighting" as mentioned above.

[0061] In the above embodiment, in addition to the stem “What action is performed by the person in the video?”, the question to be answered also includes the options “(A) Blowing (B) Sailing (C) Swimming (D) Fighting”. However, those skilled in the art should understand that in other embodiments, the question to be answered may only include the stem but not necessarily the options.

[0062] The candidate atomic video question-answering task set includes one or more candidate atomic video question-answering tasks. These candidate atomic video question-answering tasks may be obtained in at least one of the following ways: established based on experience, obtained by analyzing and counting historical questions in complex VideoQA scenarios in advance, etc. The so-called "atoms" indicate that these candidate atomic video question-answering tasks are basic and indivisible video question-answering tasks in this specification, and these video question-answering tasks can be reused multiple times.

[0063] Exemplarily, the candidate atomic video question-answering tasks included in the candidate atomic video question-answering task set may include at least one of the following types:

[0064] Video Object Detection, which is used to identify specific objects in videos;

[0065] Video Object Tracking, which is used to track objects in video sequences;

[0066] Video Action Recognition, which is used to identify the actions of objects in videos;

[0067] Video Face Detection, which is used to detect and recognize the face of an object in a video and perform sentiment analysis;

[0068] Video Classification, which is used to classify videos into preset categories;

[0069] Video Segmentation, which is used to divide a video into different parts based on scenes or objects;

[0070] Video Captioning is a task to generate text descriptions for videos to explain the content of the videos.

[0071] The video summarization task is to extract video key frames or key segments and combine them into a video summary.

[0072] It should be noted that: the candidate atomic video question-answering tasks or their sets may be different depending on the understanding of the video question-answering tasks, or the analysis and statistical methods used. In fact, those skilled in the art can determine the candidate atomic video question-answering tasks to be established based on actual conditions, and this specification does not limit this.

[0073] When guiding the target video question-answering model to decompose the question to be answered, it is necessary to provide the target video question-answering model with corresponding prompts, which can be called question decomposition prompts here, that is, prompts used to achieve question decomposition, so that the target video question-answering model can decompose the question to be answered into one or more candidate atomic video question-answering tasks.

[0074] Still with the above Figure 1Taking the illustrated embodiment as an example, the prompt words here may include, for example: "You need to gradually decompose a complex problem about the video into given atomic tasks, which will be used to solve this complex problem." The video in the prompt words is the video to be analyzed as described above, the complex problem is the problem to be solved as described above, and the atomic task is the candidate atomic video question-answering task as described above.

[0075] Of course, the candidate atomic video question-answering task set and the above-mentioned prompt words can be independently input into the target video question-answering model. It is only necessary to correctly guide the target video question-answering model to select the "atomic task" in its prompt words from the independently input candidate atomic video question-answering task set. Alternatively, the candidate atomic video question-answering task set can also be integrated into the prompt words to facilitate the target video question-answering model to understand. For example, the prompt words may include: "You need to gradually decompose a complex problem about video into the following atomic tasks: video object recognition task, used to identify specific objects in the video; video object tracking task, used to track objects in the video sequence... Video summary generation task, used to extract video key frames or key segments, and combine them into a video summary. You need to select one or more atomic tasks to solve this complex problem."

[0076] Under the guidance of the above-mentioned prompt words, the target video question-answering model can decompose the question to be answered into one or more candidate atomic video question-answering tasks by analyzing the video to be analyzed. Although the technical solution of this specification can handle complex problems in complex VideoQA scenarios, the technical solution can also deal with simple problems in simple VideoQA scenarios. For example, for simple questions to be answered, the set of atomic video question tasks to be answered generated by the target video question-answering model may only contain one candidate atomic video question-answering task, while for complex questions to be answered, the set of atomic video question tasks to be answered generated by the target video question-answering model may contain multiple candidate atomic video question-answering tasks at the same time.

[0077] In the aforementioned Figure 1 In the illustrated embodiment, it is assumed that after the target video question answering model analyzes the video to be analyzed, the question to be answered is decomposed into multiple candidate atomic video question answering tasks, including:

[0078] Atomic task ①: video action recognition task;

[0079] Atomic task ②: video object tracking task.

[0080] Of course, if the pre-established set of candidate atomic video question-answering tasks is different, the decomposition results may also change accordingly, which will not be elaborated here.

[0081] Step 304: for the video to be parsed, determine the atomic video parsing answer corresponding to each candidate atomic video question-answering task in the set of atomic video question-answering tasks to be answered.

[0082] By decomposing the questions to be answered into the set of atomic video question-answering tasks to be answered as described above, the complex video scene is disassembled, so that the execution process of each candidate atomic video question-answering task is greatly reduced in difficulty and complexity compared to directly answering the questions to be answered. Therefore, when processing each candidate atomic video question-answering task, it is possible to ensure that an accurate atomic video parsing answer is obtained, which helps to further obtain an accurate final video parsing answer based on this.

[0083] For each candidate atomic video question-answering task in the set of atomic video question-answering tasks to be answered, a video question-answering solution in the related art can be used for processing. For example, these candidate atomic video question-answering tasks can be processed by a video question-answering model, similar to the processing process in a simple VideoQA scenario in the related art.

[0084] In one embodiment, the above-mentioned target video question-answering model can still be used to process each candidate atomic video question-answering task. At this time, the above-mentioned video to be analyzed can be re-input, or it can be stated that the previously input video to be analyzed is still used. In addition, it is necessary to provide the target video question-answering model with prompt words corresponding to the candidate atomic video question-answering task to guide the target video question-answering model to answer the corresponding candidate atomic video question-answering task.

[0085] In another embodiment, other video question answering models different from the target video question answering model can be used to process each candidate atomic video question answering task. Similarly, the video to be parsed needs to be input to the other video question answering model, and the prompt words corresponding to the candidate atomic video question answering task need to be provided to the target video question answering model to guide the target video question answering model to answer the corresponding candidate atomic video question answering task.

[0086] In the case of using other video question-answering models for answering, the relevant process can be implemented by calling a pre-established atomic task parsing library. For each pre-established candidate atomic video question-answering task, a corresponding atomic task parsing library can be established one by one. For example, in the aforementioned embodiment, if 8 candidate atomic video question-answering tasks are pre-established, 8 corresponding atomic task parsing libraries can be established accordingly. Correspondingly, the atomic task parsing libraries that match each candidate atomic video question-answering task in the set of atomic video question-answering tasks to be answered can be called, and the atomic task parsing library is used to parse the provided video through other video question-answering models to complete the corresponding candidate atomic video question-answering task; and, obtain the atomic video parsing answer returned by the atomic task parsing library for the video to be parsed.

[0087] The atomic task parsing library can be regarded as a "black box". When a candidate atomic video question-answering task needs to be processed, it only needs to match the corresponding atomic task parsing library, call the atomic task parsing library and provide it with the video to be parsed, and then the corresponding atomic video parsing answer can be obtained from the atomic task parsing library without paying attention to the details such as its internal connection with the video question-answering model, the guidance of the video question-answering model, and the processing of the model output, which greatly simplifies the process of obtaining the atomic video parsing answer and reduces the complexity of scheduling.

[0088] In one embodiment, the atomic task parsing library can expose an API interface for calling it, such as VideoAPI. Correspondingly, the atomic task parsing library can also be called VideoAPI Library. After obtaining the set of atomic video question-and-answer tasks to be answered as described above, the VideoAPI corresponding to each candidate atomic video question-and-answer task contained in the set can be matched and determined, and the corresponding VideoAPI Library can be called accordingly.

[0089] Based on a relatively broad understanding, any software program, software and hardware module or hardware structure that can answer the candidate atomic video question-answering task can be used as an atomic task parsing library. Of course, illustratively, this specification can provide an embodiment of an atomic task parsing library. For example, the LangChain architecture can be used to implement each atomic task parsing library. Among them, the Wrappers (wrapper) component provided by LangChain enables the atomic task parsing library to be connected to other video question-answering models as described above, such as GPT-4 or other large language models (LargeLanguage Models, LLM) in the relevant technology, or other types of models. Through the Prompt Templates (prompt word template) component provided by LangChain, prompt words for corresponding candidate atomic video question-answering tasks can be created to guide the video question-answering model connected to the Wrappers to parse the video to be parsed, thereby answering the corresponding candidate atomic video question-answering task.

[0090] It should be pointed out that: without interference, the output of the target video question and answer model or other video question and answer models for the candidate atomic video question and answer tasks may be in a non-natural language form, which will be detrimental to the subsequent comprehensive reasoning based on the atomic video parsing answers. Therefore, for the above situation, the video parsing prompt words that need to be input to the target video question and answer model or other video question and answer models can be optimized. The role of the video parsing prompt words was originally only to guide the target video question and answer model or other video question and answer models to parse the video to be parsed to complete the corresponding candidate atomic video question and answer tasks, and output the corresponding atomic video parsing answers. After optimization, it can further guide the target video question and answer model or other video question and answer models to convert their own output into natural language as the corresponding atomic video parsing answer. Of course, the above-mentioned non-natural language output can also be converted into natural language in other ways, and this specification does not limit this.

[0091] Continuing with the foregoing embodiment, assuming that the problem to be solved is decomposed into atomic task ①: video action recognition task, atomic task ②: video object tracking task as mentioned above, the VideoAPILibrary① corresponding to the video action recognition task can be called to process the atomic task ①, and the VideoAPI Library② corresponding to the video object tracking task can be called to process the atomic task ②, and the corresponding atomic video parsing answers can be obtained respectively.

[0092] For example, assume that the atomic video parsing answer ① output by VideoAPI Library① for atomic task ① is: "People like water activities in the water, confidence: 0.85"; and the atomic video parsing answer ② output by VideoAPI Library② for atomic task ② is "frame id: 20 object id: 3, confidence: 0.51, position: (499, 278, 513, 289); frame id: 20 object id: 2, confidence: 0.52, position: (84, 253, 96, 278);...; frame id: 60 object id: 5, confidence: 0.68, position: (525, 272, 553, 327)", which indicates that people are continuously engaging in water activities.

[0093] Step 306: input the determined atomic video parsing answer to the target video question-answering model, so that the target video question-answering model outputs a final video parsing answer corresponding to the question to be answered based on the obtained atomic video parsing answer.

[0094] By decomposing the question to be answered into candidate atomic video question-answering tasks, answering each candidate atomic video question-answering task is equivalent to deeply analyzing the video to be analyzed from different dimensions, and the atomic video analysis answers are used to describe the analysis results in each dimension. Compared with directly answering the question to be answered, the above process can analyze the video content more deeply from various dimensions, so that the atomic video analysis answers contain more effective information. Therefore, by providing the atomic video analysis answers to the target video question-answering model, the target video question-answering model performs comprehensive reasoning based on these atomic video analysis answers. Compared with directly answering the question to be answered, the effective information that can be obtained and adopted in its reasoning process is relatively more, so the final video analysis answer generated is relatively more accurate, so it can achieve better performance in complex VideoQA scenarios.

[0095] If the question to be answered is previously decomposed into multiple candidate atomic video question answering tasks, multiple atomic video parsing answers will be obtained here. Although the target video question answering model can analyze the relationship between these atomic video parsing answers and achieve comprehensive reasoning, in order to further improve the accuracy of its reasoning, the comprehensive reasoning process of the target video question answering model can also be guided.

[0096] For example, the target video question-answering model can be guided to perform path planning (Path Planning) for each received atomic video analysis answer; wherein the final video analysis answer is obtained by the target video question-answering model through comprehensive reasoning on each received atomic video analysis answer based on the planned path.

[0097] Among them, the idea of ​​Chain of Thought (CoT) can be used to realize the above path planning. Of course, in order to realize this guidance, it is necessary to provide corresponding prompt words to the target video question and answer model. Generally speaking, a complete Prompt containing CoT often consists of three parts: instructions, rationales, and examples. Among them, instructions are used to describe the problem and inform the output format of the model. Rationale refers to the intermediate reasoning process of CoT, which can include solutions to the problem, intermediate reasoning steps, and any external knowledge related to the problem. Examples refer to the basic format of providing input-output pairs to the model in a few-sample manner. Each example includes: questions, reasoning processes, and answers. Then, the above-mentioned atomic video parsing answers can be used as external knowledge or independent model inputs such as the above, and passed to the target video question and answer model.

[0098] Each atomic video analysis answer can be used as a thought in path planning. For example, in the aforementioned embodiment, the atomic video analysis answer ① can be used as thought ①, and the atomic video analysis answer ② can be used as thought ②. Then, after logically combining thought ① and thought ②, comprehensive reasoning is achieved. There are usually two types of logical combinations here: series combination and parallel combination.

[0099] Therefore, the path planned by the target video question-answering model may include: at least one serial sub-path for performing serial comprehensive reasoning on the corresponding atomic video analysis answer; and / or, at least one parallel sub-path for performing parallel comprehensive reasoning on the corresponding atomic video analysis answer.

[0100] Assuming that the serial sub-path includes thought-a, thought-b and thought-c connected in series, the target video question-answering model needs to first analyze thought-a, and combine the analysis result with thought-b for further analysis, and then combine the result of the further analysis with thought-c for analysis, so as to obtain the analysis result corresponding to the serial sub-path.

[0101] Assuming that the parallel subpath includes parallel thought-d and thought-e, the target video question-answering model can analyze thought-d and thought-e in combination to obtain the analysis result corresponding to the parallel subpath.

[0102] If the path contains both serial subpaths and parallel subpaths, such as Figure 4As shown, thought-a, thought-b and thought-c form a serial sub-path, and the serial sub-path further forms a parallel sub-path with thought-d and thought-e. Then the target video question-answering model can: first analyze the serial sub-path composed of thought-a, thought-b and thought-c as described above, obtain the analysis result corresponding to the serial sub-path, and then comprehensively analyze the analysis result with thought-d and thought-e to obtain the final analysis result.

[0103] Of course, when guiding the target video question-answering model to perform path planning, it is necessary to provide it with corresponding prompt words. For example, the prompt words can be: "There are two possible path planning methods: serial and parallel. Please design a possible path that includes all outputs to obtain the final answer to the question." Among them, the so-called "output" is the atomic video parsing answer in this manual, that is, guiding the target video question-answering model to perform path planning for all atomic video parsing answers.

[0104] Assume that in the embodiment including thought ① and thought ② as described above, after the target video question-answering model performs path planning, it obtains a parallel path consisting of thought ① and thought ②. And, the target video question-answering model performs comprehensive reasoning based on the parallel path, for example: based on thought ①, the target video question-answering model analyzes and determines the behavior of people engaged in water activities; and, based on thought ②, the target video question-answering model confirms that people are always active in the water throughout the video. By analyzing these two items in parallel, it can be determined that people are engaged in water activities, and this activity runs through the entire video, thereby inferring that the main activity may be sailing.

[0105] In summary, this specification, by defining candidate atomic video question-answering tasks, can guide the target video question-answering model to decompose the originally more complex questions to be answered into relatively simple candidate atomic video question-answering tasks, which is equivalent to disassembling the multiple dimensions that the questions to be answered face at the same time, so that when the videos to be analyzed are further analyzed for each candidate atomic video question-answering task, compared with directly analyzing the videos to be analyzed based on the questions to be answered, the difficulty and complexity of video analysis can be significantly reduced, and the video content contained in the video to be analyzed can be analyzed more accurately and deeply. Furthermore, the target video question-answering model performs comprehensive reasoning based on the answers to the atomic video analyses, and can make full use of the video content analyzed by each candidate atomic video question-answering task from its corresponding dimensions to obtain more accurate answers, so it can be better applied to complex VideoQA scenarios.

[0106] Figure 5 FIG. 1 is a flowchart of an atomic task parsing method for video question answering provided by an exemplary embodiment. Figure 5 As shown, this method can be applied to the atomic task parsing library corresponding to any candidate atomic video question answering task in the candidate atomic video question answering task set, such as Figure 2 The atomic task parsing library 23, the atomic task parsing library 24 or other atomic task parsing libraries shown include the following steps:

[0107] Step 502, receiving a parsing library call request, wherein the parsing library call request is used to instruct the corresponding atomic task parsing library to parse the video to be parsed to complete any of the candidate atomic video question and answer tasks; wherein, when the target video question and answer model decomposes the question to be answered into a set of atomic video question and answer tasks to be answered based on the video to be parsed, and the set of atomic video question and answer tasks to be answered includes any of the candidate atomic video question and answer tasks, the parsing library call request is initiated.

[0108] Step 504: input the video to be analyzed into a predefined video question-answering model, and guide the predefined video question-answering model to provide an output corresponding to any candidate atomic video question-answering task based on the video to be analyzed.

[0109] Step 506, based on the output of the predefined video question-answering model, returns the atomic video parsing answer corresponding to any of the candidate atomic video question-answering tasks, so that the target video question-answering model outputs the final video parsing answer corresponding to the question to be answered based on the obtained atomic video parsing answer.

[0110] Those skilled in the art will understand that: Figure 5 The embodiment shown in the figure describes the video question-answering solution of this specification from the perspective of the atomic task parsing library. Figure 3 In the embodiment shown, the video question-answering solution of this specification has been described in detail from the perspective of the model proxy service, and the contents involving the atomic task parsing library or related to the atomic task parsing library are applicable to the present invention. Figure 5 Therefore, you can refer to the above content to understand Figure 5 The illustrated embodiments and other contents related to the atomic task parsing library will not be described in detail here.

[0111] Figure 6 FIG. 1 is a schematic diagram of the architecture of a video question-answering collaborative system provided by an exemplary embodiment. Figure 6 As shown in Figure 2The video question answering system shown reduces the target video question answering model. In other words, the video question answering collaborative system includes a model proxy service 602 and an atomic task parsing library 604.

[0112] There may be one or more atomic task parsing libraries 604. In fact, the atomic task parsing library 604 corresponds one-to-one to each candidate atomic video question-answering task in the candidate atomic video question-answering task set, and is used to parse the provided video through a predefined video question-answering model to complete the corresponding atomic video question-answering task.

[0113] The model proxy service 602 is used to:

[0114] Inputting a to-be-analyzed video, a to-be-answered question, and the candidate atomic video question-answering task set into a target video question-answering model, and guiding the target video question-answering model to decompose the to-be-answered question into an atomic video question-answering task set based on the to-be-analyzed video, wherein the atomic video question-answering task set includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set;

[0115] Based on the set of atomic video question-answering tasks to be answered, a matching atomic task parsing library 604 is called to obtain corresponding atomic video parsing answers; and

[0116] The determined atomic video parsing answer is input into the target video question-answering model, so that the target video question-answering model outputs a final video parsing answer corresponding to the question to be answered based on the obtained atomic video parsing answer.

[0117] Optionally, the model proxy service 602 is further used for:

[0118] Guiding the target video question-answering model to perform path planning for each received atomic video parsing answer;

[0119] Among them, the final video analysis answer is obtained by the target video question-answering model through comprehensive reasoning of the received atomic video analysis answers based on the planned path.

[0120] Of course, regarding the model proxy service 602 and the atomic task parsing library 604, in fact, reference can be made to the aforementioned Figure 3 and Figure 5 The embodiments described separately will not be described in detail here.

[0121] It should be noted that if the demander develops and provides video Q&A services to the outside world, the demander can develop Figure 2 The video question-and-answer system shown in the figure can provide video question-and-answer services to the outside world through the deployed video question-and-answer system. Alternatively, if the demand side does not have the development capability, the supply side can Figure 2 The video question-and-answer system shown is developed as a whole product and provided to the demand side for deployment, and then the video question-and-answer service is provided to the outside through the deployed video question-and-answer system. Alternatively, some demand sides may have developed a certain video question-and-answer model by themselves or obtained it from other channels, or the service provided by the video question-and-answer model, and they hope to use the video question-and-answer model as the target video question-and-answer model, but do not want to use the video question-and-answer model as shown in the figure. Figure 2 The video question answering model used in the video question answering system shown in FIG. Figure 6 The video question-answering collaborative system shown is developed as an integral product and provided to the demand side for deployment. The video question-answering collaborative system can connect to the target video question-answering model that the demand side wants to adopt and enhance the performance of the target video question-answering model in complex VideoQA scenarios.

[0122] It can be seen that by providing Figure 6 The video question and answer collaborative system shown makes the selection of the target video question and answer model more flexible, facilitates operations such as upgrading or replacing the target video question and answer model, and ensures that the process will not affect the video question and answer collaborative system. For example, there is no need to redevelop the video question and answer collaborative system, and it has extremely strong flexibility.

[0123] Based on the above, this specification can guide the target video question-answering model to decompose the questions to be answered through the model proxy service, so as to process each candidate atomic video question-answering task obtained by the decomposition. Therefore, the accurate decomposition of the questions to be answered is very important for the final performance of the video question-answering. To this end, this specification provides a method for training a video question-answering model, which can assist in the training of the target video question-answering model to improve its ability to decompose the questions to be answered.

[0124] Figure 7 FIG. 1 is a flowchart of a method for training a video question answering model provided by an exemplary embodiment. Figure 7 As shown, the method may include the following steps:

[0125] Step 702, obtaining a training set, wherein each training sample in the training set includes a sample video, a sample question, and a sample atomic video question-answering task set to be answered, wherein the sample atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set.

[0126] The sample atomic video question answering task set to be answered is obtained by decomposing the sample questions based on the sample video, which is equivalent to the label data of the training sample. The candidate atomic video question answering task set, as described in the above embodiment, is a set of pre-established candidate atomic video question answering tasks, which will not be repeated here.

[0127] There are many ways to obtain a training set, and this specification does not limit this. For example, the above training set can be formed based on MVBench, a benchmark test for evaluating the video understanding ability of multimodal models. MVBench contains 20 video tasks that cannot be efficiently solved by a single frame alone, each task contains 200 video questions and answers, that is, a total of 4000 video questions and answers. By inputting the videos involved in these video questions and answers, the description of the video questions and answers, and the definition of "complex video questions and answers" into a large language model such as GPT-3.5 or similar, and guiding it to determine whether the current video question and answer belongs to a complex video question and answer. Finally, the training set in this specification contains 2325 training samples obtained based on the above method, and these training samples are all complex video questions and answers, that is, video questions and answers in complex VideoQA scenarios.

[0128] Of course, in addition to generating a training set, the above method can also be used to generate a validation set to verify the ability of the trained target video question-answering model to decompose the questions to be answered. Alternatively, the above method can also be used to generate a test set to test the ability of any video question-answering model to decompose the questions to be answered.

[0129] Step 704, input the training set and the candidate atomic video question and answer task set into the target video question and answer model, and obtain the actual atomic video question and answer task set to be answered outputted for each training sample, wherein the actual atomic video question and answer task set to be answered is obtained by decomposing the sample questions in the corresponding training samples based on the sample videos in the corresponding training samples by the target video question and answer model, and the actual atomic video question and answer task set to be answered contains at least one candidate atomic video question and answer task from the candidate atomic video question and answer task set.

[0130] Step 706: iteratively update the target video question answering model according to the difference between the actual set of atomic video question answering tasks to be answered and the sample set of atomic video question answering tasks to be answered.

[0131] By reasonably designing the loss function used in the training process and calculating the loss value, the parameters of the target video question answering model can be optimized. After several rounds of iterative training, if the loss value meets the preset requirements or reaches the predefined upper limit of the training rounds, the training is terminated.

[0132] It should be noted that: for the process of parsing sample videos and decomposing sample questions by the target video question-answering model, reference can be made to the process of parsing the video to be analyzed to decompose the question to be answered in the aforementioned embodiment, which will not be repeated here.

[0133] In addition to the decomposition of the questions to be answered, as mentioned above, the target video question answering model also needs to perform comprehensive reasoning based on each atomic video parsing answer to obtain the final video parsing answer. Therefore, in addition to training the target video question answering model's ability to decompose the questions to be answered, other capabilities of the target video question answering model can also be trained to further optimize the performance of the target video question answering model in complex VideoQA scenarios. Figure 8 Provide explanation.

[0134] Figure 8 FIG. 1 is a flowchart of another video question answering model training method provided by an exemplary embodiment. Figure 8 As shown, the method may include the following steps:

[0135] Step 802: Obtain a training set, where each training sample in the training set includes a sample video, a sample question, and a sample final video analysis answer.

[0136] Step 804, input the training set and the candidate atomic video question and answer task set into the target video question and answer model, and obtain the actual atomic video question and answer task set to be answered outputted for each training sample, wherein the actual atomic video question and answer task set to be answered is obtained by decomposing the sample questions in the corresponding training samples based on the sample videos in the corresponding training samples by the target video question and answer model, and the actual atomic video question and answer task set to be answered contains at least one candidate atomic video question and answer task from the candidate atomic video question and answer task set.

[0137] Step 806, for each sample video in the training sample, determine the atomic video parsing answers corresponding to each candidate atomic video question-answering task in the corresponding actual set of atomic video question-answering tasks to be answered, and input the determined atomic video parsing answers to the target video question-answering model, so that the target video question-answering model outputs the actual final video parsing answers corresponding to the sample questions in the corresponding training sample.

[0138] Similar to the aforementioned embodiment, here, the atomic task parsing library that matches each candidate atomic video question-answering task in the actual set of atomic video question-answering tasks to be answered can be called, and the atomic task parsing library is used to parse the provided video through a predefined video question-answering model to complete the corresponding candidate atomic video question-answering task; and, obtain the atomic video parsing answer returned by the atomic task parsing library for the corresponding sample video. Among them, the predefined video question-answering model adopted by the atomic task parsing library can be, for example, GPT-4; or, as mentioned above, the predefined video question model can be the target video question-answering model itself, because the candidate atomic video question-answering task itself is an atomic task after decomposition, and the complexity has been greatly reduced, so the target video question-answering model can also handle it smoothly. Of course, the atomic video parsing answer can also be obtained by other means, and this specification does not limit this.

[0139] Similar to the above embodiment, the target video question answering model can be guided to perform path planning for each received atomic video analysis answer; wherein the actual final video analysis answer is obtained by the target video question answering model performing comprehensive reasoning on each received atomic video analysis answer based on the planned path. Of course, comprehensive reasoning can also be achieved in other ways.

[0140] Regarding the process of processing the candidate atomic video question-and-answer tasks in the actual set of atomic video question-and-answer tasks to be answered, please refer to the description of processing the candidate atomic video question-and-answer tasks in the aforementioned embodiments; and regarding the process of comprehensive reasoning on the atomic video analysis answers, please refer to the relevant descriptions in the aforementioned embodiments, which will not be repeated here.

[0141] Step 808, iteratively update the target video question-answering model according to a predefined loss function; wherein the loss function includes an answer difference term, which is the difference between the actual final video analysis answer and the sample final video analysis answer.

[0142] and Figure 7 Compared with the training scheme shown in FIG. 1 , the training scheme of this embodiment runs through the entire process of the target video question answering model performing video question answering, so that it not only participates in the decomposition operation of the sample question, but also gives the actual final video parsing answer for the sample question. Therefore, by setting up the answer difference term in the loss function, the target video question answering model can be iteratively optimized based on the difference in the final video parsing answer.

[0143] In addition to constraining the final video parsing answer through the answer difference term, this embodiment can also add Figure 7The constraints implemented based on the perspective of task decomposition in the training sample can also include: a set of sample atomic video question-answering tasks to be answered, wherein the set of sample atomic video question-answering tasks to be answered contains at least one candidate atomic video question-answering task; and the loss function can also include: a task difference term, wherein the task difference term is the difference between the actual set of atomic video question-answering tasks to be answered and the set of sample atomic video question-answering tasks to be answered.

[0144] Then, by performing multi-angle constraints on both the problem decomposition and the final video analysis answer, the loss value can be measured more comprehensively and meticulously, and the training process of the target video question answering model can be further optimized, so that the trained target video question answering model can achieve better performance in complex VideoQA scenarios.

[0145] Fig. 9 is a schematic structural diagram of a device provided by an exemplary embodiment. Fig. 9 At the hardware level, the device includes a processor 902, an internal bus 904, a network interface 906, a memory 908, and a non-volatile memory 910, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 902 reading the corresponding computer program from the non-volatile memory 910 into the memory 908 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0146] Please refer to Fig.10 The video question-answering device can be applied to Fig. 9 The device shown in the figure is used to implement the technical solution of this specification. The video question-answering device may include:

[0147] A question decomposition guiding unit, used for inputting a to-be-analyzed video, a to-be-answered question and a candidate atomic video question-answering task set into a target video question-answering model, and guiding the target video question-answering model to decompose the to-be-answered question into an to-be-answered atomic video question-answering task set based on the to-be-analyzed video, wherein the to-be-answered atomic video question-answering task set includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set;

[0148] A parsing answer determination unit, configured to determine, for the video to be parsed, the atomic video parsing answers corresponding to the candidate atomic video question-answering tasks in the set of atomic video question-answering tasks to be answered;

[0149] The parsing answer scheduling unit is used to input the determined atomic video parsing answer to the target video question-answering model, so that the target video question-answering model outputs the final video parsing answer corresponding to the question to be answered based on the obtained atomic video parsing answer.

[0150] Optionally, the parsing answer determination unit is specifically used to:

[0151] The atomic video parsing answer is determined by the target video question-answering model or other video question-answering models different from the target video question-answering model.

[0152] Optionally, the parsing answer determination unit is specifically used to:

[0153] Calling atomic task parsing libraries that match each candidate atomic video question-answering task in the set of atomic video question-answering tasks to be answered, wherein the atomic task parsing library is used to parse the provided video through the other video question-answering models to complete the corresponding candidate atomic video question-answering tasks;

[0154] Obtain the atomic video parsing answer returned by the atomic task parsing library for the video to be parsed.

[0155] Optionally, the parsing answer determination unit is specifically used to:

[0156] Input the to-be-analyzed video into the target video question-answering model or the other video question-answering model, and guide the target video question-answering model or the other video question-answering model to parse the to-be-analyzed video through video parsing prompt words to complete the corresponding candidate atomic video question-answering task, and output the corresponding atomic video parsing answer;

[0157] Among them, when the output of the target video question and answer model or the other video question and answer model for the candidate atomic video question and answer task is non-natural language, the video parsing prompt word is also used to guide the target video question and answer model or the other video question and answer model to convert the output into natural language as the corresponding atomic video parsing answer.

[0158] Optionally, also include:

[0159] A path planning guidance unit guides the target video question answering model to perform path planning for each received atomic video parsing answer;

[0160] Among them, the final video analysis answer is obtained by the target video question-answering model through comprehensive reasoning of the received atomic video analysis answers based on the planned path.

[0161] Optionally, the path planned by the target video question-answering model includes:

[0162] at least one concatenated subpath for concatenated comprehensive reasoning about the corresponding atomic video parsing answer; and / or,

[0163] At least one parallel subpath for parallel comprehensive reasoning on a corresponding atomic video parsing answer.

[0164] Optionally, the candidate atomic video question-answering tasks in the candidate atomic video question-answering task set include at least one of the following types:

[0165] Video object recognition tasks, used to identify specific objects in videos;

[0166] Video object tracking tasks, used to track objects in video sequences;

[0167] Video action recognition task, used to identify the action of an object in a video;

[0168] Video face detection and recognition tasks, which are used to detect and recognize the faces of objects in videos and perform sentiment analysis;

[0169] Video classification task, used to classify videos into preset categories;

[0170] Video segmentation tasks, which are used to divide videos into different parts based on scenes or objects;

[0171] Video subtitle generation task, which is used to generate text descriptions for videos to explain the video content;

[0172] The video summary generation task is used to extract video key frames or key segments and combine them into a video summary.

[0173] Optionally, the target video question answering model includes: a large multimodal model.

[0174] Please refer to Fig.11 , the atomic task parsing device for video question answering can be applied to Fig. 9 In the device shown, the technical solution of this specification is implemented. The atomic task parsing device for video question answering is applied to an atomic task parsing library corresponding to any candidate atomic video question answering task in the candidate atomic video question answering task set; wherein each candidate atomic video question answering task in the candidate atomic video question answering task set has a corresponding atomic task parsing library; the device includes:

[0175] A call request receiving unit, configured to receive a parsing library call request, wherein the parsing library call request is used to instruct the corresponding atomic task parsing library to parse the video to be parsed to complete any of the candidate atomic video question-answering tasks; wherein the parsing library call request is initiated when the target video question-answering model decomposes the question to be answered into a set of atomic video question-answering tasks to be answered based on the video to be parsed, and the set of atomic video question-answering tasks to be answered includes any of the candidate atomic video question-answering tasks;

[0176] A task processing guiding unit, configured to input the video to be analyzed into a predefined video question answering model, and guide the predefined video question answering model to provide an output corresponding to any candidate atomic video question answering task based on the video to be analyzed;

[0177] The parsing answer returning unit is used to return the atomic video parsing answer corresponding to any candidate atomic video question-answering task according to the output of the predefined video question-answering model, so that the target video question-answering model can output the final video parsing answer corresponding to the question to be answered according to the obtained atomic video parsing answer.

[0178] Please refer to Fig.12 The training device of the video question answering model can be applied to Fig. 9 The device shown in the figure is used to implement the technical solution of this specification. Among them, the training device of the video question answering model may include:

[0179] A training sample acquisition unit, used to acquire a training set, each training sample in the training set includes a sample video, a sample question and a sample atomic video question-answering task set to be answered, wherein the sample atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set;

[0180] a decomposition result acquisition unit, for inputting the training set and the candidate atomic video question-answering task set into the target video question-answering model, and obtaining the actual atomic video question-answering task set to be answered outputted for each training sample, wherein the actual atomic video question-answering task set to be answered is obtained by decomposing the sample questions in the corresponding training samples based on the sample videos in the corresponding training samples by the target video question-answering model, and the actual atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set;

[0181] The model iteration updating unit is used to iteratively update the target video question answering model according to the difference between the actual set of atomic video question answering tasks to be answered and the sample set of atomic video question answering tasks to be answered.

[0182] Please refer to Fig.13 , the training device of the video question answering model can be applied to Fig. 9 The device shown in the figure is used to implement the technical solution of this specification. Among them, the training device of the video question answering model may include:

[0183] A training sample acquisition unit, used to acquire a training set, wherein each training sample in the training set includes a sample video, a sample question, and a sample final video parsing answer;

[0184] a decomposition result acquisition unit, for inputting the training set and the candidate atomic video question-answering task set into the target video question-answering model, and obtaining the actual atomic video question-answering task set to be answered outputted for each training sample, wherein the actual atomic video question-answering task set to be answered is obtained by decomposing the sample questions in the corresponding training samples based on the sample videos in the corresponding training samples by the target video question-answering model, and the actual atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set;

[0185] A parsing answer acquisition unit is used to determine, for each sample video in each training sample, the atomic video parsing answers corresponding to each candidate atomic video question-answering task in the corresponding actual set of atomic video question-answering tasks to be answered, and input the determined atomic video parsing answers to the target video question-answering model, so that the target video question-answering model outputs the actual final video parsing answers corresponding to the sample questions in the corresponding training sample;

[0186] A model iterative update unit is used to iteratively update the target video question-answering model according to a predefined loss function; wherein the loss function includes an answer difference term, and the answer difference term is the difference between the actual final video analysis answer and the sample final video analysis answer.

[0187] Optionally, the decomposition result obtaining unit is specifically used for:

[0188] Calling atomic task parsing libraries that match each candidate atomic video question-answering task in the actual set of atomic video question-answering tasks to be answered, wherein the atomic task parsing library is used to parse the provided video through a predefined video question-answering model to complete the corresponding candidate atomic video question-answering task;

[0189] Obtain the atomic video parsing answer returned by the atomic task parsing library for the corresponding sample video.

[0190] Optionally, also include:

[0191] A path planning guidance unit, used to guide the target video question-answering model to perform path planning for each received atomic video parsing answer;

[0192] Among them, the actual final video analysis answer is obtained by the target video question-answering model through comprehensive reasoning of the received atomic video analysis answers based on the planned path.

[0193] Optional,

[0194] Each training sample further includes: a sample atomic video question-answering task set to be answered, wherein the sample atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task;

[0195] The loss function also includes: a task difference term, where the task difference term is the difference between the actual set of atomic video question-answering tasks to be answered and the sample set of atomic video question-answering tasks to be answered.

[0196] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0197] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0198] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

Claims

1. A video question-answering method, comprising: Inputting a video to be parsed, a question to be answered, and a set of candidate atomic video question-answering tasks into a target video question-answering model, and guiding the target video question-answering model to decompose the question to be answered into a set of atomic video question-answering tasks to be answered based on the video to be parsed, wherein the set of atomic video question-answering tasks to be answered includes at least one candidate atomic video question-answering task from the set of candidate atomic video question-answering tasks; For the video to be parsed, respectively determine the atomic video parsing answer corresponding to each candidate atomic video question-answering task in the set of atomic video question-answering tasks to be answered; The determined atomic video parsing answer is input into the target video question-answering model, so that the target video question-answering model outputs a final video parsing answer corresponding to the question to be answered based on the obtained atomic video parsing answer.

2. According to the method of claim 1, the step of respectively determining the atomic video parsing answer corresponding to each candidate atomic video question-answering task in the set of atomic video question-answering tasks to be answered comprises: The atomic video parsing answer is determined by the target video question-answering model or other video question-answering models different from the target video question-answering model.

3. The method according to claim 2, wherein determining the atomic video parsing answer by using another video question answering model different from the target video question answering model comprises: Calling atomic task parsing libraries that match each candidate atomic video question-answering task in the set of atomic video question-answering tasks to be answered, wherein the atomic task parsing library is used to parse the provided video through the other video question-answering models to complete the corresponding candidate atomic video question-answering tasks; Obtain the atomic video parsing answer returned by the atomic task parsing library for the video to be parsed.

4. The method according to claim 2, wherein determining the atomic video parsing answer by using the target video question-answering model or another video question-answering model different from the target video question-answering model comprises: Input the to-be-analyzed video into the target video question-answering model or the other video question-answering model, and guide the target video question-answering model or the other video question-answering model to parse the to-be-analyzed video through video parsing prompt words to complete the corresponding candidate atomic video question-answering task, and output the corresponding atomic video parsing answer; Among them, when the output of the target video question and answer model or the other video question and answer model for the candidate atomic video question and answer task is non-natural language, the video parsing prompt word is also used to guide the target video question and answer model or the other video question and answer model to convert the output into natural language as the corresponding atomic video parsing answer.

5. The method according to claim 1, further comprising: Guiding the target video question-answering model to perform path planning for each received atomic video parsing answer; Among them, the final video analysis answer is obtained by the target video question-answering model through comprehensive reasoning of the received atomic video analysis answers based on the planned path.

6. According to the method of claim 5, the path planned by the target video question answering model includes: at least one concatenated subpath for concatenated comprehensive reasoning about the corresponding atomic video parsing answer; and / or, At least one parallel subpath for parallel comprehensive reasoning on a corresponding atomic video parsing answer.

7. According to the method of claim 1, the candidate atomic video question answering tasks in the candidate atomic video question answering task set include at least one of the following types: Video object recognition tasks, used to identify specific objects in videos; Video object tracking tasks, used to track objects in video sequences; Video action recognition task, used to identify the action of an object in a video; Video face detection and recognition tasks, which are used to detect and recognize the faces of objects in videos and perform sentiment analysis; Video classification task, used to classify videos into preset categories; Video segmentation tasks, which are used to divide videos into different parts based on scenes or objects; Video subtitle generation task, which is used to generate text descriptions for videos to explain the video content; The video summary generation task is used to extract video key frames or key segments and combine them into a video summary.

8. The method according to claim 1, wherein the target video question answering model comprises: Large multimodal models.

9. An atomic task parsing method for video question answering, applied to an atomic task parsing library corresponding to any candidate atomic video question answering task in a candidate atomic video question answering task set; wherein: Each candidate atomic video question-answering task in the candidate atomic video question-answering task set has a corresponding atomic task parsing library; the method comprises: A parsing library call request is received, wherein the parsing library call request is used to instruct the corresponding atomic task parsing library to parse the video to be parsed to complete any of the candidate atomic video question-answering tasks; wherein the parsing library call request is initiated when the target video question-answering model decomposes the question to be answered into a set of atomic video question-answering tasks to be answered based on the video to be parsed, and the set of atomic video question-answering tasks to be answered includes any of the candidate atomic video question-answering tasks; Inputting the video to be parsed into a predefined video question answering model, and guiding the predefined video question answering model to provide an output corresponding to any candidate atomic video question answering task based on the video to be parsed; According to the output of the predefined video question-answering model, the atomic video parsing answer corresponding to any of the candidate atomic video question-answering tasks is returned, so that the target video question-answering model outputs a final video parsing answer corresponding to the question to be answered based on the obtained atomic video parsing answer.

10. A video question-answering collaborative system, comprising: An atomic task parsing library corresponding to each candidate atomic video question-answering task in the candidate atomic video question-answering task set, for parsing the provided video through a predefined video question-answering model to complete the corresponding atomic video question-answering task; Model proxy service for: Inputting a to-be-analyzed video, a to-be-answered question, and the candidate atomic video question-answering task set into a target video question-answering model, and guiding the target video question-answering model to decompose the to-be-answered question into an atomic video question-answering task set based on the to-be-analyzed video, wherein the atomic video question-answering task set includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set; Based on the set of atomic video question-answering tasks to be answered, a matching atomic task parsing library is called to obtain corresponding atomic video parsing answers; and, The determined atomic video parsing answer is input into the target video question-answering model, so that the target video question-answering model outputs a final video parsing answer corresponding to the question to be answered based on the obtained atomic video parsing answer.

11. The system according to claim 10, wherein the model proxy service is further configured to: Guiding the target video question-answering model to perform path planning for each received atomic video parsing answer; in, The final video analysis answer is obtained by the target video question-answering model through comprehensive reasoning of the received atomic video analysis answers based on the planned path.

12. A video question-answering system, comprising: Target video question answering model; A video question-answering collaborative system as described in claim 10 or 11.

13. A method for training a video question answering model, comprising: Obtain a training set, wherein each training sample in the training set includes a sample video, a sample question, and a sample atomic video question-answering task set to be answered, wherein the sample atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task from a candidate atomic video question-answering task set; Input the training set and the candidate atomic video question-answering task set into the target video question-answering model, and obtain the actual atomic video question-answering task set to be answered outputted by the target video question-answering model for each training sample, wherein the actual atomic video question-answering task set to be answered is obtained by decomposing the sample questions in the corresponding training samples based on the sample videos in the corresponding training samples by the target video question-answering model, and the actual atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set; The target video question answering model is iteratively updated according to the difference between the actual set of atomic video question answering tasks to be answered and the sample set of atomic video question answering tasks to be answered.

14. A method for training a video question answering model, comprising: Obtaining a training set, wherein each training sample in the training set includes a sample video, a sample question, and a sample final video parsing answer; Input the training set and the candidate atomic video question-answering task set into the target video question-answering model, and obtain the actual atomic video question-answering task set to be answered outputted by the target video question-answering model for each training sample, wherein the actual atomic video question-answering task set to be answered is obtained by decomposing the sample questions in the corresponding training samples based on the sample videos in the corresponding training samples by the target video question-answering model, and the actual atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task from the candidate atomic video question-answering task set; For each sample video in the training sample, the atomic video parsing answers corresponding to each candidate atomic video question-answering task in the corresponding actual set of atomic video question-answering tasks to be answered are respectively determined, and the determined atomic video parsing answers are input into the target video question-answering model, so that the target video question-answering model outputs the actual final video parsing answers corresponding to the sample questions in the corresponding training sample; The target video question-answering model is iteratively updated according to a predefined loss function; wherein the loss function includes an answer difference term, and the answer difference term is the difference between the actual final video parsing answer and the sample final video parsing answer.

15. The method according to claim 14, wherein for each sample video in the training sample, determining the atomic video parsing answer corresponding to each candidate atomic video question-answering task in the corresponding actual set of atomic video question-answering tasks to be answered comprises: Calling atomic task parsing libraries that match each candidate atomic video question-answering task in the actual set of atomic video question-answering tasks to be answered, wherein the atomic task parsing library is used to parse the provided video through a predefined video question-answering model to complete the corresponding candidate atomic video question-answering task; Obtain the atomic video parsing answer returned by the atomic task parsing library for the corresponding sample video.

16. The method according to claim 14, further comprising: Guiding the target video question-answering model to perform path planning for each received atomic video parsing answer; Among them, the actual final video analysis answer is obtained by the target video question-answering model through comprehensive reasoning of the received atomic video analysis answers based on the planned path.

17. The method according to claim 14, Each training sample also includes: A sample atomic video question-answering task set to be answered, wherein the sample atomic video question-answering task set to be answered includes at least one candidate atomic video question-answering task; The loss function also includes: a task difference term, where the task difference term is the difference between the actual set of atomic video question-answering tasks to be answered and the sample set of atomic video question-answering tasks to be answered.

18. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method according to any one of claims 1-9 and 13-17 by executing the executable instructions.

19. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 9 and 13 to 17.

20. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9, 13 to 17.

Citation Information

Patent Citations

  • Question answering method and device, electronic equipment and computer readable storage medium

    CN116244418A

  • Video question and answer processing method and device, electronic equipment and storage medium

    CN117112751A

  • Question and answer method and device, electronic equipment and medium

    CN117521625A

  • Zero-sample video question-answering method for guiding hybrid experts based on time sequence information

    CN117612049A

  • Video-text multi-mode question answering method and system based on module neural network

    CN117668744A