Method and apparatus for constructing intent reasoning datasets for video question answering

Abstract

This invention provides a method and apparatus for constructing an intent reasoning dataset for video question answering. The method includes: obtaining an initial intent reasoning dataset by filtering a pre-constructed dataset; annotating question-answer pairs in the initial intent reasoning dataset with actions to obtain an annotated initial dataset containing action annotation results, and obtaining an annotated initial intent reasoning dataset based on multiple sets of annotated initial datasets; and obtaining a target intent reasoning dataset by filtering the annotated initial intent reasoning dataset, wherein each set of target data in the target intent reasoning dataset satisfies the following conditions: the actions annotated in each set of target data correspond to the factual actions performed by the target person in the video of each set of target data, and the multiple factual actions in the video corresponding to multiple actions with the same action annotation results are the same factual actions. This achieves the construction of an intent reasoning dataset containing annotations of intent-related behaviors or actions.

Description

Technical Field

[0001] This invention relates to the field of dataset construction technology, and in particular to a method and apparatus for constructing an intent reasoning dataset for video question answering. Background Technology

[0002] According to relevant technologies, video question answering refers to answering natural language questions based on a given video.

[0003] Currently, video question answering is often performed using pre-trained video question answering models. However, the datasets used to train these models often focus on factual descriptions or the behavior itself, neglecting the intent behind the facts or actions. This prevents the models from being trained effectively to handle more complex contextual understanding and reasoning tasks, thus enabling them to provide more relevant answers.

[0004] Therefore, constructing an intent reasoning dataset that includes annotations of intent-related behaviors or actions has become a research hotspot. Summary of the Invention

[0005] This invention provides a method and apparatus for constructing an intent reasoning dataset for video question answering, which realizes the construction of an intent reasoning dataset containing annotations of intent-related behaviors or actions, thereby facilitating further understanding and prediction of the intent of human behavior.

[0006] This invention provides a method for constructing an intent reasoning dataset for video question answering. The method includes: filtering an initial intent reasoning dataset based on a pre-constructed dataset, wherein each set of initial data in the initial intent reasoning dataset is related to intent, and the question-answer pair includes a question and the answer corresponding to the question; performing action annotation on the question-answer pairs to obtain annotated initial data containing action annotation results, and obtaining an annotated initial intent reasoning dataset based on multiple sets of the annotated initial data; filtering a target intent reasoning dataset based on the annotated initial intent reasoning dataset, wherein each set of target data in the target intent reasoning dataset satisfies the following conditions: the actions annotated in each set of target data correspond to factual actions performed by a target person in the video of each set of target data, and multiple factual actions in the video corresponding to multiple actions with the same action annotation results are the same factual actions.

[0007] According to a method for constructing an intent reasoning dataset for video question answering provided by the present invention, the initial intent reasoning dataset is obtained by filtering in the following manner: for each set of data in the pre-constructed dataset, if the data satisfies any one of the following conditions, the data is used as the initial data, and the initial intent reasoning dataset is obtained based on the initial data: the question in the data contains first information, wherein the first information is used to characterize why a preset behavior should be performed; or the question in the data contains second information, wherein the second information is used to characterize how to achieve the preset intent; or the data contains third information, wherein the third information is used to characterize that the content described by the answer in the data occurs before or after the content described by the question in the data.

[0008] According to a method for constructing an intent reasoning dataset for video question answering provided by the present invention, the step of performing action annotation on the question-answer pairs specifically includes: obtaining a grammatical dependency tree of the question-answer pairs based on deep dependency parsing, wherein the grammatical dependency tree includes the part-of-speech tags of each word in the question-answer pairs; and performing action annotation on the question-answer pairs based on the part-of-speech tags of each word.

[0009] According to the present invention, a method for constructing an intent reasoning dataset for video question answering is provided. The step of obtaining the grammatical dependency tree of the question-answer pair based on deep dependency parsing specifically includes: determining the source of the action to be labeled in the question-answer pair based on deep dependency parsing; if the source is determined to be the question in the question-answer pair, obtaining the grammatical dependency tree of the question-answer pair based on the question; if the source is determined to be the answer in the question-answer pair, obtaining the grammatical dependency tree of the question-answer pair based on the answer, wherein, if the question in the question-answer pair is detected to include first information, or if the question-answer pair includes third information, the source is determined to be the question in the question-answer pair; if the question in the question-answer pair is detected to include second information, the source is determined to be the answer in the question-answer pair.

[0010] According to a method for constructing an intent reasoning dataset for video question answering provided by the present invention, after the target intent reasoning dataset is obtained through filtering, the method further includes: for any set of target data containing action annotation results in the target intent reasoning dataset, using the target question-answer pairs in the target data as anchor samples, determining the positive and negative samples of the anchor samples in the target question-answer pairs of the remaining target data, wherein the remaining target data are other target data in the target intent reasoning dataset besides the target data corresponding to the anchor samples; taking the target data including the anchor samples, the target data including the positive samples, and the target data including the negative samples as the final target data; and obtaining the final target intent reasoning dataset based on multiple sets of the final target data.

[0011] According to the present invention, a method for constructing an intent reasoning dataset for video question answering is provided. The positive and negative samples of the anchor sample are determined by the following method: candidate data with the same action annotation results as the anchor sample are determined from the remaining target data; the positive and negative samples of the anchor sample are determined based on the answer similarity between the question-answer pairs of the candidate data and the anchor sample.

[0012] According to a method for constructing an intent reasoning dataset for video question answering provided by the present invention, the step of determining positive and negative samples of the anchor sample based on the answer similarity between the question-answer pairs of the candidate data and the anchor sample specifically includes: when the answer similarity is greater than or equal to a similarity threshold, taking the question-answer pairs of the candidate data as positive samples of the anchor sample; and when the answer similarity is less than the similarity threshold, taking the question-answer pairs of the candidate data as negative samples of the anchor sample.

[0013] According to the present invention, a method for constructing an intent reasoning dataset for video question answering is provided. The action annotation results are the same, including any one of the following: action annotation results at the same action description level, action annotation results at the same word level of the action description, or action annotation results at the same semantic level of the action description, wherein the action description is a specific descriptive word about the action in the action annotation results.

[0014] According to the present invention, a method for constructing an intent reasoning dataset for video question answering, after the target intent reasoning dataset is obtained by filtering, the method further includes: dividing the target intent reasoning dataset into a training dataset, a validation dataset, and a test dataset according to a preset ratio, wherein the videos contained in the training dataset, the validation dataset, and the test dataset are different videos, and the training dataset, the validation dataset, and the test dataset contain target data with the same action annotation results.

[0015] The present invention also provides an apparatus for constructing an intent reasoning dataset for video question answering. The apparatus includes: an initial screening module for screening an initial intent reasoning dataset based on a pre-constructed dataset, wherein each set of initial data in the initial intent reasoning dataset is related to intent, and the question-answer pair includes a question and the answer corresponding to the question; an annotation module for annotating the question-answer pairs with actions to obtain an annotated initial data containing action annotation results, and obtaining an annotated initial intent reasoning dataset based on multiple sets of the annotated initial data; and a re-screening module for screening a target intent reasoning dataset based on the annotated initial intent reasoning dataset, wherein each set of target data in the target intent reasoning dataset satisfies the following conditions: the actions annotated in each set of target data correspond to the factual actions performed by the target person in the video of each set of target data, and the multiple factual actions in the video corresponding to multiple actions with the same action annotation results are the same factual actions.

[0016] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the intent reasoning dataset construction method for video question answering as described above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intent reasoning dataset construction method for video question answering as described above.

[0018] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the intent reasoning dataset construction method for video question answering as described above.

[0019] The present invention provides a method and apparatus for constructing an intent reasoning dataset for video question answering. Based on a pre-constructed dataset, an initial intent reasoning dataset related to intent is obtained by filtering. Then, action annotation is performed on question-answer pairs to obtain annotated initial data containing action annotation results. Based on multiple sets of annotated initial data, an annotated initial intent reasoning dataset is obtained. Finally, based on the annotated initial intent reasoning dataset, a target intent reasoning dataset is obtained by filtering, such that the actions annotated in the target data of the target intent reasoning dataset correspond to the factual actions performed by the target person in the video of each set of target data, and that multiple factual actions in the video corresponding to multiple actions with the same action annotation results are considered the same factual actions. The action annotation results reference the intent corresponding to the question-answer pairs, thus realizing the construction of an intent reasoning dataset containing intent-related behavioral or action annotations, which facilitates further understanding and prediction of human behavioral intent. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is one of the flowcharts illustrating the method for constructing an intent reasoning dataset for video question answering provided by the present invention;

[0022] Figure 2 This is a flowchart illustrating the action annotation process for question-answer pairs provided by the present invention;

[0023] Figure 3 This is the second flowchart illustrating the method for constructing an intent reasoning dataset for video question answering provided by this invention.

[0024] Figure 4 This is the third flowchart illustrating the method for constructing an intent reasoning dataset for video question answering provided by this invention;

[0025] Figure 5 This is a schematic diagram of the structure of the intent reasoning dataset construction device for video question answering provided by the present invention;

[0026] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] Figure 1 This is one of the flowcharts illustrating the method for constructing an intent reasoning dataset for video question answering provided by this invention.

[0029] To further introduce the method for constructing an intent reasoning dataset for video question answering provided by this invention, the following will combine... Figure 1 Please provide an explanation.

[0030] In an exemplary embodiment of the present invention, combined with Figure 1 As can be seen, the method for constructing an intent reasoning dataset for video question answering may include steps 110 to 130, which will be described in detail below.

[0031] In step 110, an initial intent reasoning dataset is obtained by filtering based on a pre-built dataset.

[0032] In the initial intent reasoning dataset, each set of initial data contains question-answer pairs related to intent, where each pair includes a question and its corresponding answer. In another example, the answers in a question-answer pair (also known as a QA) can include both correct and incorrect answers. For instance, for a QA, there might be one question and five answers, where one answer is correct and the others are constructed based on the video. It should be noted that the pre-built dataset is a video question-answering dataset.

[0033] In one embodiment, the pre-built dataset can be an existing NextQA dataset; however, no specific limitation is made to the pre-built dataset in this embodiment.

[0034] In another embodiment, a pre-built dataset can be filtered to obtain an initial intent reasoning dataset. In this dataset, the question-answer pairs in each set of initial data are related to the intent. The question-answer pairs can be understood as the QA (Question and Answer) in each set of data. It is understood that each set of initial data may include a video, a question, and a corresponding set of answers, where the set of answers includes multiple answers.

[0035] In yet another exemplary embodiment of the present invention, the initial intent reasoning dataset can be obtained by filtering in the following manner:

[0036] For each set of data in the pre-built dataset, the data is used as the initial data if any of the following conditions are met, and an initial intent inference dataset is obtained based on the initial data:

[0037] The problem in the data contains primary information, which characterizes why a pre-defined behavior should be performed, or

[0038] The problem in the data contains second information, which characterizes how to achieve the predetermined intent, or

[0039] The data contains third information, which is used to characterize whether the content described by the answer in the data occurs before or after the content described by the question in the data.

[0040] The problem in the data, which includes the first piece of information, can be illustrated through the following examples:

[0041] Data containing primary information can be Causal Why type QA, typically expressed as: [Why does a person perform a certain action? For a certain intention]. In this type of QA, the key action appears in the question, while the intention is in the answer. For these types of questions, we need to infer the underlying intention of the person in the video based on the key action.

[0042] The problem in the data containing second information can be illustrated through the following examples:

[0043] Data containing secondary information can be causal-how type QA, typically expressed as: "[How does a person achieve their intention? By performing a certain action]?" The key action in this type of question appears in the answer, while the intention is stated in the question. For these questions, we need to infer the key action for achieving that intention based on the known intention.

[0044] The inclusion of third-party information in the data can be illustrated through the following examples:

[0045] Data containing third-party information can be Temporal type QA, which typically manifests as follows: in the video, the events described in the answer always occur before or after the events described in the question; there are no instances where both occur simultaneously. Furthermore, key actions always appear after the words "after" or "before." In this type of question, the intent is usually not directly stated. For TN (after) questions, the basic form is "After what did B do?", where B's reaction reveals A's intent to perform the key action. For TP (before) questions, the basic form is "What did A do that caused B to react with the key action (before B performed the key action)?" The key action is a response to A's intent. In TN questions, the intent occurs after the key action, while in TP questions, the action occurs before the key intent. Answering these two types of questions does not require directly manipulating the intent, but understanding the intent is necessary to construct the relationship between the two events.

[0046] During application, if each set of data in the pre-built dataset satisfies any of the aforementioned conditions, the data can be used as the initial data, and the initial intent reasoning dataset can be obtained based on the initial data.

[0047] In step 120, action annotation is performed on the question-answer pairs to obtain initial annotation data containing action annotation results, and based on multiple sets of initial annotation data, an initial annotation intent inference dataset is obtained.

[0048] In one embodiment, action annotations can be applied to the question-answer pairs in the initial data to obtain annotated initial data containing action annotation results. In other words, the difference between annotated initial data and the initial data is that the annotated initial data includes action annotation results. Furthermore, based on multiple sets of annotated initial data containing action annotation results, an annotated initial intent reasoning dataset can be obtained. Since the action annotation results reference the intent corresponding to the question-answer pairs, an intent reasoning dataset containing intent-related behavior or action annotations is constructed, thereby facilitating further understanding and prediction of human behavioral intent.

[0049] In step 130, based on the labeled initial intent reasoning dataset, a target intent reasoning dataset is obtained by filtering, wherein each set of target data in the target intent reasoning dataset satisfies the following conditions:

[0050] The actions labeled in each set of target data correspond to the actual actions performed by the target person in the video of each set of target data, and the multiple actual actions in the video that correspond to multiple actions with the same action labeling results are the same actual actions.

[0051] In another embodiment, the initial intention reasoning dataset can be filtered to obtain a target intention reasoning dataset. In this dataset, the action to be controlled (corresponding to the action labeled) in each set of target data can be a physical action performed by a specific person and observable in the video within the target data, rather than a psychological behavior. This ensures the inclusion of social intent in the retained QA (corresponding to question-answer pairs in the target data).

[0052] Furthermore, it's necessary to ensure that multiple factual actions corresponding to multiple actions with the same action annotation results in the target data video are the same factual actions. In other words, for the same factual action in the video, the action annotation result should be the same even in different target data. This ensures that when comparing two samples, the comparison focuses on the intent of two identical annotated actions, and the difference in intent is due to the different contexts. For example, in one sample (corresponding to a set of target data), the physical behavior of a person in the video is raising their hand, while in another sample, the physical behavior of a person in the video is raising their leg. Although the labeled action is "raise," the intents are different; one intent is raising the hand, and the other is raising the leg.

[0053] This invention provides a method for constructing an intent reasoning dataset for video question answering. Based on a pre-built dataset, an initial intent reasoning dataset related to intent is obtained through filtering. Then, action annotation is performed on question-answer pairs to obtain annotated initial data containing action annotation results. Based on multiple sets of annotated initial data, an annotated initial intent reasoning dataset is obtained. Finally, based on the annotated initial intent reasoning dataset, a target intent reasoning dataset is obtained through filtering. This ensures that the actions annotated in the target data of the target intent reasoning dataset correspond to the factual actions performed by the target person in the video of each set of target data, and that multiple factual actions in the video corresponding to multiple actions with the same action annotation results are considered the same factual actions. The action annotation results reference the intent corresponding to the question-answer pairs, thus constructing an intent reasoning dataset containing intent-related behavioral or action annotations, which facilitates further understanding and prediction of human behavioral intent.

[0054] Figure 2 This is a flowchart illustrating the process of annotating action pairs for question answers provided by the present invention.

[0055] The following will combine Figure 2 Explain the process of annotating the answers to the questions with actions.

[0056] In an exemplary embodiment of the present invention, combined with Figure 2As can be seen, annotating the question and answer pairs with actions can include steps 210 and 220, which will be described in detail below.

[0057] In step 210, based on deep dependency parsing, a grammatical dependency tree of the question-answer pair is obtained, wherein the grammatical dependency tree includes the part-of-speech tag of each word in the question-answer pair;

[0058] In step 220, action tags are applied to the question-answer pairs based on the part of speech of each word.

[0059] In one embodiment, a grammatical dependency tree of the question-answer pair can be obtained through deep dependency parsing. This tree includes the part-of-speech tags (POS) of each word in the question-answer pair. POS can include verb POS, noun POS, and adjective POS, among others. Furthermore, action annotation is performed on the question-answer pair based on the POS of each word. During this action annotation process, the POS of the words can be used to determine the verb POS, thereby extracting key actions from the question-answer pair for subsequent action annotation.

[0060] In yet another exemplary embodiment of the present invention, the syntax dependency tree of question-answer pairs based on deep dependency parsing can be obtained in the following manner:

[0061] Based on deep dependency parsing, the source of the action to be labeled in the question answer pair is determined;

[0062] If the source is determined to be a question in the question-answer pair, a grammatical dependency tree of the question-answer pair is obtained based on the question;

[0063] If the source is determined to be the answer in the question-answer pair, a grammatical dependency tree of the question-answer pair is obtained based on the answer, wherein...

[0064] If the question in the question-answer pair is found to include the first information, or if the question-answer pair includes the third information, the source is determined to be the question in the question-answer pair.

[0065] If the question in the question-answer pair is found to include second information, the source is determined to be the answer in the question-answer pair.

[0066] In another embodiment, to more efficiently annotate question-answer pairs with actions, the following method can also be used:

[0067] Determine the source of the action to be annotated (also known as the key action) in the question-answer pair, where the source can include the question or the answer;

[0068] Specifically, when the question-answer pair is detected to include the first information or the third information, the source of the action to be labeled (also known as the key action) can be determined to be the question; further, based on the question in the question-answer pair, the grammatical dependency tree of the question in the question-answer pair is obtained; and based on the part of speech of each word in the grammatical dependency tree, the question-answer pair is labeled with actions.

[0069] In another embodiment, when it is detected that the question in the question-answer pair includes the second information, the source of the action to be labeled (also known as the key action) can be determined as the answer; further, based on the answer in the question-answer pair, the grammatical dependency tree of the answer in the question-answer pair is obtained; and based on the part of speech of each word in the grammatical dependency tree, the question-answer pair is labeled with actions.

[0070] In one embodiment, based on the characteristics of question-answer pairs (e.g., the key actions for the "Causal why" question appear in the question, while the key actions for the "Causal how" question are hidden in the answer), key actions can be extracted from the Q&A using grammatical dependency trees and part-of-speech tagging methods, and then the actions can be annotated.

[0071] In another embodiment, based on the characteristics of question-answer pairs (e.g., the key actions in temporal questions are hidden after target words such as "after / before"), key actions can be extracted from the Q&A using dependency trees and part-of-speech tagging methods, and then annotated. Through the aforementioned embodiments, action annotation can be performed quickly and accurately.

[0072] Figure 3 This is the second flowchart illustrating the method for constructing an intent reasoning dataset for video question answering provided by this invention.

[0073] To further introduce the method for constructing an intent reasoning dataset for video question answering provided by this invention, the following will combine... Figure 3 Please provide an explanation.

[0074] In an exemplary embodiment of the present invention, combined with Figure 3 As can be seen, the method for constructing an intent reasoning dataset for video question answering may include steps 310 to 360, wherein steps 310 to 330 are the same as or similar to steps 110 to 130. For specific implementation methods and beneficial effects, please refer to the previous description. In this embodiment, they will not be repeated. Steps 340 to 360 will be described below.

[0075] In step 340, for any set of target data containing action annotation results in the target intent reasoning dataset, the target question answer pair in the target data is used as the anchor sample, and the positive and negative samples of the anchor sample are determined in the target question answer pairs of the remaining target data.

[0076] The remaining target data consists of the target data in the target intent reasoning dataset other than the target data corresponding to the anchor sample.

[0077] In step 350, the target data including anchor samples, the target data including positive samples, and the target data including negative samples are used as the final target data.

[0078] In step 360, a final target intent inference dataset is obtained based on multiple sets of final target data.

[0079] In one embodiment, for any set of target data containing action annotation results in the target intent reasoning dataset, the target question-answer pairs in the target data can be used as anchor samples, and positive and negative samples of the anchor samples can be determined from the target question-answer pairs in the remaining target data. Further, the target data including anchor samples, target data including positive samples, and target data including negative samples are used as final target data, and a final target intent reasoning dataset is obtained based on multiple sets of final target data.

[0080] In yet another exemplary embodiment of the present invention, the determination of the positive and negative samples of the anchor sample can be achieved in the following manner:

[0081] Identify candidate data from the remaining target data that have the same action annotation results as the anchor samples;

[0082] Based on the similarity between the question-answer pairs in the candidate data and the anchor samples, positive and negative samples of the anchor samples are determined.

[0083] In yet another exemplary embodiment of the present invention, determining the positive and negative samples of the anchor sample based on the answer similarity between the candidate data question-answer pairs and the anchor sample can be achieved in the following manner:

[0084] If the answer similarity is greater than or equal to the similarity threshold, the question-answer pairs in the candidate data will be used as positive samples of the anchor samples.

[0085] If the answer similarity is less than the similarity threshold, the question-answer pairs in the candidate data are used as negative samples of the anchor samples.

[0086] During application, positive and negative examples can be selected based on the similarity between the action (corresponding to the action annotation result) and the answer. The answer similarity can be controlled based on the wups metric (corresponding to the similarity) of the true answers between the two Q&As. In one example, the criteria for selecting positive and negative samples are that the actions are consistent. Positive samples require a wups of 1 between the Q&As, while negative samples require a wups of less than 0.5 between the Q&As.

[0087] In another exemplary embodiment of the present invention, the same action annotation result may include any one of the following: action annotation results at the same action description level, action annotation results at the same word level of action description, or action annotation results at the same semantic level of action description, wherein the action description is a specific descriptive word about the action in the action annotation result.

[0088] In one embodiment, an action description can refer to specific descriptive words that describe an action, such as words that characterize an action like "lift," "push," and "take." It is understood that action annotation results at the same action description level refer to action annotation results where all specific descriptive words about the action are identical; action annotation results at the same lexical level refer to the next higher level description of the specific descriptive words about the action, i.e., action annotation results where the meanings are identical; and action annotation results at the same semantic level refer to the next higher level description of the lexical units of the action description, i.e., action annotation results where the meanings are identical. It is understood that, regarding the control of action similarity mentioned above, based on the three criteria of action consistency (corresponding to action annotation results at the same action description level), lemma consistency (corresponding to action annotation results at the same lexical level of the action description), and action id consistency (corresponding to action annotation results at the same semantic level of the action description), the similarity is naturally divided into three levels from strict to lenient.

[0089] Figure 4 This is the third flowchart illustrating the method for constructing an intent reasoning dataset for video question answering provided by this invention.

[0090] The following will combine Figure 4 The process of constructing an intent reasoning dataset for video question answering is described.

[0091] In an exemplary embodiment of the present invention, combined with Figure 4 As can be seen, the method for constructing an intent reasoning dataset for video question answering may include steps 410 to 440, wherein steps 410 to 430 are the same as or similar to steps 110 to 130. For specific implementation methods and beneficial effects, please refer to the previous description. In this embodiment, they will not be repeated. Step 440 will be introduced below.

[0092] In step 440, the target intent reasoning dataset is divided into a training dataset, a validation dataset, and a test dataset according to a preset ratio. The training dataset, the validation dataset, and the test dataset contain different videos, and the training dataset, the validation dataset, and the test dataset contain target data with the same action annotation results.

[0093] In one embodiment, the target intent reasoning dataset can be divided into a training dataset, a validation dataset, and a test dataset according to a preset ratio (e.g., 6:1:1). The preset ratio can be adjusted according to actual circumstances and is not specifically limited in this embodiment.

[0094] Furthermore, during application, it is ensured that each action term appearing in the validation / test set appears at least twice in the training set. For action terms with sufficient video samples, a 6:1:1 ratio is maintained across the three sets as much as possible. To avoid overfitting, it is ensured that the same video appears in only one set.

[0095] The present invention provides a method for constructing an intent reasoning dataset for video question answering, which focuses on various intents in daily social activities and creates a new IntentQA dataset (corresponding to the target intent reasoning dataset), which is of great significance for understanding and predicting the intent of human behavior.

[0096] In the application process, the existing NExT-QA dataset can be used to select reasoning-related QA questions (including causal and temporal questions) as the basis for data construction. Furthermore, dependency trees are used for dependency parsing to extract key actions in the QA and further annotation is performed.

[0097] To ensure annotation quality, cross-validation was employed. Each sample was included in the final IntentQA dataset only if all annotators agreed on its inclusion. This application constructs the dataset comparatively by designing that the same action leads to different underlying intentions in different contexts (corresponding to constructing target data including both positive and negative samples), thereby enhancing dataset quality and its contribution to model training. Furthermore, during dataset partitioning, it ensured that the same video appeared in only one dataset and maintained the proportion of action terms across training, validation, and test sets as much as possible to avoid overfitting and evaluation errors.

[0098] In addition, detailed statistics are provided, including the number of videos, the number of question-answer pairs, the number of actions, the number of lexical units, and the number of action IDs (corresponding word meanings), providing rich reference information for subsequent research.

[0099] Based on the same concept, the present invention also provides an apparatus for constructing an intent reasoning dataset for video question answering.

[0100] The following describes the apparatus for constructing an intent reasoning dataset for video question answering provided by the present invention. The apparatus for constructing an intent reasoning dataset for video question answering described below can be referred to in correspondence with the method for constructing an intent reasoning dataset for video question answering described above.

[0101] Figure 5 This is a schematic diagram of the device for constructing an intent reasoning dataset for video question answering provided by the present invention.

[0102] In an exemplary embodiment of the present invention, combined with Figure 5 As can be seen, the device for constructing an intent reasoning dataset for video question answering may include an initial screening module 510, an annotation module 520, and a re-screening module 530. Each module will be described below.

[0103] The initial screening module 510 can be configured to filter and obtain an initial intent reasoning dataset based on a pre-built dataset. In the initial intent reasoning dataset, the question-answer pairs in each set of initial data are related to the intent. The question-answer pairs include a question and the corresponding answer.

[0104] The annotation module 520 can be configured to perform action annotation on question-answer pairs to obtain initial annotation data containing action annotation results, and to obtain an initial annotation intent inference dataset based on multiple sets of initial annotation data;

[0105] The re-filtering module 530 can be configured to filter the initial labeled intent reasoning dataset to obtain a target intent reasoning dataset, wherein each set of target data in the target intent reasoning dataset satisfies the following conditions:

[0106] The actions labeled in each set of target data correspond to the actual actions performed by the target person in the video of each set of target data, and the multiple actual actions in the video that correspond to multiple actions with the same action labeling results are the same actual actions.

[0107] In an exemplary embodiment of this application, the initial screening module 510 can obtain the initial intent reasoning dataset by screening in the following manner:

[0108] For each set of data in the pre-built dataset, the data is used as the initial data if any of the following conditions are met, and an initial intent inference dataset is obtained based on the initial data:

[0109] The problem in the data contains primary information, which characterizes why a pre-defined behavior should be performed, or

[0110] The problem in the data contains second information, which characterizes how to achieve the predetermined intent, or

[0111] The data contains third information, which is used to characterize whether the content described by the answer in the data occurs before or after the content described by the question in the data.

[0112] In an exemplary embodiment of the present invention, the annotation module 520 may perform action annotation on the question-answer pairs in the following manner:

[0113] Based on deep dependency parsing, a grammatical dependency tree of the question-answer pair is obtained, wherein the grammatical dependency tree includes the part of speech of each word in the question-answer pair;

[0114] Based on the part of speech of each word, action tags are added to the question and answer pairs.

[0115] In an exemplary embodiment of the present invention, the annotation module 520 can implement the syntax dependency tree of question-answer pairs based on deep dependency parsing in the following manner:

[0116] Based on deep dependency parsing, the source of the action to be labeled in the question-answer pair is determined;

[0117] Given that the source is determined to be a question in a question-answer pair, a syntactic dependency tree of the question-answer pair is obtained based on the question.

[0118] If the source is determined to be the answer in a question-answer pair, a grammatical dependency tree of the question-answer pair is obtained based on the answer, where,

[0119] If the question in the question-answer pair is found to include the first piece of information, or if the question-answer pair includes the third piece of information, the source is determined to be the question in the question-answer pair.

[0120] If the question in the question-answer pair includes the second piece of information, the source is determined to be the answer in the question-answer pair.

[0121] In an exemplary embodiment of the present invention, the re-screening module 530 may further be configured to:

[0122] For any set of target data containing action annotation results in the target intent reasoning dataset, the target question answer pair in the target data is used as the anchor sample. Positive and negative samples of the anchor sample are determined in the target question answer pairs of the remaining target data. The remaining target data are other target data in the target intent reasoning dataset besides the target data corresponding to the anchor sample.

[0123] The target data, which includes anchor samples, positive samples, and negative samples, will be used as the final target data.

[0124] Based on multiple sets of final target data, a final target intent inference dataset is obtained.

[0125] In an exemplary embodiment of the present invention, the re-screening module 530 may also determine the positive and negative samples of the anchor sample in the following manner:

[0126] Identify candidate data from the remaining target data that have the same action annotation results as the anchor samples;

[0127] Based on the similarity between the question-answer pairs in the candidate data and the anchor samples, positive and negative samples of the anchor samples are determined.

[0128] In an exemplary embodiment of the present invention, the re-screening module 530 may further determine the positive and negative samples of the anchor sample by using the answer similarity between the candidate data and the anchor sample based on the question-answer pair.

[0129] If the answer similarity is greater than or equal to the similarity threshold, the question-answer pairs in the candidate data will be used as positive samples of the anchor samples.

[0130] If the answer similarity is less than the similarity threshold, the question-answer pairs in the candidate data are used as negative samples of the anchor samples.

[0131] In an exemplary embodiment of the present invention, the same action annotation result includes any one of the following: action annotation results at the same action description level, action annotation results at the same word level of the action description, or action annotation results at the same semantic level of the action description, wherein the action description is a specific descriptive word about the action in the action annotation result.

[0132] In an exemplary embodiment of the present invention, the re-screening module 530 may further be configured to:

[0133] The target intent reasoning dataset is divided into training dataset, validation dataset and test dataset according to a preset ratio. The videos contained in the training dataset, validation dataset and test dataset are different videos, and the training dataset, validation dataset and test dataset contain target data with the same action annotation results.

[0134] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a method for constructing an intent reasoning dataset for video question answering. The method includes: filtering an initial intent reasoning dataset based on a pre-constructed dataset, wherein each set of initial data in the initial intent reasoning dataset is related to intent, and the question-answer pair includes a question and the answer corresponding to the question; performing action annotation on the question-answer pairs to obtain annotated initial data containing action annotation results, and obtaining an annotated initial intent reasoning dataset based on multiple sets of the annotated initial data; filtering a target intent reasoning dataset based on the annotated initial intent reasoning dataset, wherein each set of target data in the target intent reasoning dataset satisfies the following conditions: the actions annotated in each set of target data correspond to factual actions performed by a target person in the video of each set of target data, and multiple factual actions in the video corresponding to multiple actions with the same action annotation results are the same factual actions.

[0135] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the intent reasoning dataset construction method for video question answering provided by the above methods. The method includes: filtering an initial intent reasoning dataset based on a pre-constructed dataset, wherein the question-answer pairs in each set of initial data in the initial intent reasoning dataset are related to intent, wherein the question-answer pairs include a question and the answer corresponding to the question; performing action annotation on the question-answer pairs to obtain annotated initial data containing action annotation results, and obtaining an annotated initial intent reasoning dataset based on multiple sets of annotated initial data; filtering a target intent reasoning dataset based on the annotated initial intent reasoning dataset, wherein each set of target data in the target intent reasoning dataset satisfies the following conditions: the action annotated in each set of target data corresponds to the factual action performed by the target person in the video of each set of target data, and the multiple factual actions in the video corresponding to multiple actions with the same action annotation results are the same factual actions.

[0137] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a method for constructing an intent reasoning dataset for video question answering provided by the methods described above. This method includes: filtering an initial intent reasoning dataset based on a pre-constructed dataset, wherein each set of initial data in the initial intent reasoning dataset is related to intent, and wherein the question-answer pair includes a question and the answer corresponding to the question; performing action annotation on the question-answer pairs to obtain annotated initial data containing action annotation results, and obtaining an annotated initial intent reasoning dataset based on multiple sets of the annotated initial data; filtering a target intent reasoning dataset based on the annotated initial intent reasoning dataset, wherein each set of target data in the target intent reasoning dataset satisfies the following conditions: the action annotated in each set of target data corresponds to a factual action performed by a target person in the video of each set of target data, and multiple factual actions in the video corresponding to multiple actions with the same action annotation results are the same factual actions.

[0138] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0139] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0140] It is further understood that although the operations are described in a specific order in the accompanying drawings in the embodiments of the present invention, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all the operations shown to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing an intent reasoning dataset for video question answering, characterized in that, The method includes: Based on a pre-built dataset, an initial intent reasoning dataset is obtained by filtering. In the initial intent reasoning dataset, the question-answer pairs in each set of initial data are related to the intent. The question-answer pairs include a question and the answer corresponding to the question. Action annotation is performed on the question-answer pairs to obtain initial annotation data containing action annotation results, and an initial annotation intent inference dataset is obtained based on multiple sets of the initial annotation data; Based on the labeled initial intent reasoning dataset, a target intent reasoning dataset is obtained by filtering, wherein each group of target data in the target intent reasoning dataset satisfies the following conditions: The actions labeled in each set of target data correspond to the factual actions performed by the target person in the video of each set of target data, and the multiple factual actions in the video that correspond to multiple actions with the same action labeling results are considered to be the same factual actions. The initial intent reasoning dataset is obtained by filtering in the following way: For each set of data in the pre-built dataset, the data is used as the initial data if the data satisfies any one of the following conditions, and the initial intent reasoning dataset is obtained based on the initial data: The problem in the data includes first information, wherein the first information is used to characterize why a preset behavior should be performed, or The problem in the data includes second information, wherein the second information is used to characterize how to achieve the preset intention, or The data includes third information, wherein the third information is used to characterize whether the content described by the answer in the data occurs before or after the content described by the question in the data. After obtaining the target intent reasoning dataset through the filtering process, the method further includes: For any set of target data containing action annotation results in the target intent reasoning dataset, the target question answer pair in the target data is used as the anchor sample. Positive and negative samples of the anchor sample are determined in the target question answer pairs of the remaining target data. The remaining target data are other target data in the target intent reasoning dataset other than the target data corresponding to the anchor sample. The target data including the anchor sample, the target data including the positive sample, and the target data including the negative sample are used as the final target data. Based on multiple sets of the final target data, a final target intent inference dataset is obtained.

2. The method for constructing an intent reasoning dataset for video question answering according to claim 1, characterized in that, The action annotation of the question-answer pairs specifically includes: Based on deep dependency parsing, a grammatical dependency tree of the question-answer pair is obtained, wherein the grammatical dependency tree includes the part-of-speech tag of each word in the question-answer pair; Based on the part of speech of each word, action tags are applied to the question-answer pairs.

3. The method for constructing an intent reasoning dataset for video question answering according to claim 2, characterized in that, The process of obtaining the syntactic dependency tree of the question-answer pair based on deep dependency parsing specifically includes: Based on deep dependency parsing, the source of the action to be labeled in the question answer pair is determined; If the source is determined to be a question in the question-answer pair, a grammatical dependency tree of the question-answer pair is obtained based on the question; If the source is determined to be the answer in the question-answer pair, a grammatical dependency tree of the question-answer pair is obtained based on the answer, wherein... If the question in the question-answer pair is found to include the first information, or if the question-answer pair includes the third information, the source is determined to be the question in the question-answer pair. If the question in the question-answer pair is found to include second information, the source is determined to be the answer in the question-answer pair.

4. The method for constructing an intent reasoning dataset for video question answering according to claim 1, characterized in that, The positive and negative samples of the anchor sample are determined in the following way: From the remaining target data, identify candidate data that have the same action annotation results as the anchor sample; Based on the similarity between the question-answer pairs in the candidate data and the answers in the anchor samples, the positive and negative samples of the anchor samples are determined.

5. The method for constructing an intent reasoning dataset for video question answering according to claim 4, characterized in that, The determination of positive and negative samples of the anchor sample based on the answer similarity between the candidate data and the anchor sample specifically includes: If the answer similarity is greater than or equal to the similarity threshold, the question-answer pairs of the candidate data are used as positive samples of the anchor samples; If the answer similarity is less than the similarity threshold, the question-answer pair of the candidate data is used as the negative sample of the anchor sample.

6. The method for constructing an intent reasoning dataset for video question answering according to claim 4, characterized in that, The same action annotation results include any one of the following: action annotation results at the same action description level, action annotation results at the same word level of the action description, or action annotation results at the same semantic level of the action description, wherein the action description is a specific descriptive word about the action in the action annotation results.

7. The method for constructing an intent reasoning dataset for video question answering according to claim 1, characterized in that, After obtaining the target intent reasoning dataset through the filtering process, the method further includes: The target intent reasoning dataset is divided into a training dataset, a validation dataset, and a test dataset according to a preset ratio. The videos contained in the training dataset, the validation dataset, and the test dataset are different videos, and the training dataset, the validation dataset, and the test dataset contain target data with the same action annotation results.

8. An apparatus for constructing an intent reasoning dataset for video question answering, characterized in that, The apparatus is used to implement the intent reasoning dataset construction method for video question answering according to any one of claims 1 to 7, the apparatus comprising: The initial screening module is used to filter and obtain an initial intent reasoning dataset based on a pre-built dataset. In the initial intent reasoning dataset, the question-answer pairs in each set of initial data are related to the intent. The question-answer pairs include a question and the answer corresponding to the question. The annotation module is used to annotate the question-answer pairs with actions to obtain an initial annotation data containing the action annotation results, and to obtain an initial annotation intent inference dataset based on multiple sets of the initial annotation data; The re-filtering module is used to filter the initial labeled intent reasoning dataset to obtain a target intent reasoning dataset, wherein each group of target data in the target intent reasoning dataset satisfies the following conditions: The actions labeled in each set of target data correspond to the actual actions performed by the target person in the video of each set of target data, and the multiple actual actions in the video that correspond to multiple actions with the same action labeling results are the same actual actions.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for constructing an intent reasoning dataset for video question answering as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the intent reasoning dataset construction method for video question answering as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the intent reasoning dataset construction method for video question answering as described in any one of claims 1 to 7.

Images