Video question and answer data set social reasoning evaluation method and system

Abstract

The invention provides a video question and answer data set social reasoning evaluation method and system. The method comprises the following steps: collecting social interaction videos from a video interaction big data platform, removing identified and labeled traces of a video data set, performing visual question answering and analysis labeling, and labeling questions and answers QA according with rules and describing an inference chain; according to the inference chain conforming to the rule questions and answers QA and descriptions, the event and psychological state causal relationship is constructed, an event and psychological state causal relationship model is established, and social causal diversification types are marked; according to the social causal diversification type, verifying the data through a first expert empirical model, and performing reasoning chain labeling through a second expert empirical model; performing question and answer QA and inference chain comprehensive evaluation through a multi-modal large model with multiple evaluation indexes; accurate data is provided for social behavior evaluation of an intelligent system; and verifying the generated question and answer QA to obtain a question and answer QA social reasoning result close to real social reasoning.

Description

Technical Field

[0001] The present invention relates to the technical field of social question-answering evaluation for intelligent system reasoning, and more specifically, to a method and system for social reasoning evaluation of a video question-answering dataset. Background Art

[0002] Theory of mind and social reasoning. Theory of mind is a language dataset that tests the social interaction and social reasoning ability of language models. It has been studied a lot (socialiqa, cleverhans, fantom, opentom, llmfailtrivialtom). High-quality videos are difficult to generate. In comparison, language datasets are easier to construct, but the information expressed in text is too direct, making it difficult to test the model's ability to establish causal relationships from a large number of small clues. Video question answering. With the emergence and development of multimodal large models, video question answering has become the main way to test the model's video understanding ability. Researchers have proposed some video question answering benchmarks for testing large models; MMBench-Video, Video-MME, Egoschema, MVBench, NextQA, ActivityNet-QA, etc. are commonly used test benchmarks. They focus more on the factual understanding of video content and lack the examination of people's psychological state. IntentQA and MELD only focus on one psychological state and lack a comprehensive examination of the model's ability to estimate the psychological state of others. VAR examines the model's ability to capture causal relationships and traceability, but it is not a video question-answering task and is not suitable for existing large multimodal models. It only focuses on the causal relationship between events. CausalChaos involves causal reasoning of psychological states, but its videos come from online video information, not the real world, and it does not combine language information to comprehensively examine the model's multimodal capabilities. In addition, its examination of causal reasoning ability is relatively simple, and it only gives explanations corresponding to QA. MMToM-QA is a video question-answering dataset for theory of mind, but the data is collected in a virtual environment, which has a gap with the real environment, and only examines the recognition of beleif and goal. SocialIQ is a dataset involving multiple aspects of social intelligence, but it lacks the examination of visual clue capture and reasoning ability, and other issues remain to be resolved. Therefore, it is necessary to propose a social reasoning evaluation method and system for video question-answering datasets to at least partially solve the problems existing in the prior art. Summary of the invention

[0003] A series of simplified concepts are introduced in the content of the invention, which will be further described in detail in the specific implementation method section; the content of the invention of the present invention does not mean to attempt to limit the key features and essential technical features of the technical solution claimed for protection, nor does it mean to attempt to determine the scope of protection of the technical solution claimed for protection.

[0004] To at least partially solve the above problems, the present invention provides a method for evaluating social reasoning of a video question-answering dataset, comprising: S100 collects social interaction videos from the video interaction big data platform, removes the traces of previous identification and annotation in the video dataset, performs visual question answering and analysis annotation, and annotates the rule-compliant question answering (QA) and description reasoning chains; S200, based on the reasoning chain of rule-compliant QA and description, construct the causal relationship between events and psychological states, establish a causal relationship model between events and psychological states, and annotate the diverse types of social causality; S300, based on the social causal diversity type, the data is verified by the first expert experience model, and the reasoning chain is annotated by the second expert experience model; S400 conducts comprehensive evaluation of QA and reasoning chains through a multi-modal large model with multiple evaluation indicators; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated QA and obtains QA social reasoning results that are close to real social reasoning.

[0005] Preferably, S100 includes: S101, collect social interaction videos from the video interaction big data platform, remove low-quality social interaction videos with few interactions, few behaviors, and few events; screen out high-quality social interaction videos with many interactions, many behaviors, or many events, and aggregate them to form a video data set; S102, selecting videos with incorrect answers and incorrect questions and answers (QA) from the large model, and removing the identification and annotation traces from the video dataset; S103, perform visual question answering and analysis annotation, perform data standard recognition and annotation platform rule training on the basic subject model, and obtain the basic subject training model; the basic subject training model annotates the rule-compliant question answering QA and description reasoning chain from the video interactive big data platform according to the challenge form; The reasoning chain includes: video event reasoning, video character psychological state reasoning, and causal relationship reasoning between events and psychological states; A set of related questions and answers (QA) is generated based on each reasoning chain through the big model. The generated rules are as follows: for each node in the reasoning chain, a factual or mental state identification question is generated; For each sub-chain of reasoning in the chain of reasoning, a single-step causal reasoning problem of causal cause and causal effect is generated.

[0006] Preferably, S200 includes: S201, establishing a causal relationship between events and mental states based on the reasoning chain of the rule-compliant question and answer QA and the description; S202, based on the causal relationship between events and psychological states, social causal reasoning is performed to understand the events occurring in the video and estimate the psychological states of the characters, and a causal relationship model between events and psychological states is established; S203, labeling social causal diversity types based on the causal relationship model between events and psychological states; According to the causal relationship between events and psychological states, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological states of the characters. The causal relationship model between events and psychological states is established, including: selecting sample reasoning chains; understanding the events that occurred in the video; estimating the psychological states of people; establishing causal relationships; dividing problems into diversified social causal types; diversified social causal types include: event understanding type, psychological state estimation type, causal cause type and causal result type; by evaluating the performance of the causal relationship model between events and psychological states on various types of problems, the lack of model segmentation capabilities is analyzed.

[0007] Preferably, S300 includes: S301, verifying the data through the first expert experience model according to the social causal diversity type, and obtaining the verified data; S302, verify that the data is annotated by the second expert experience model through reasoning chain.

[0008] Preferably, S400 includes: S401, select multiple evaluation indicators, and conduct comprehensive evaluation of QA and reasoning chain through multiple evaluation indicators and multi-modal large models; provide accurate data for social behavior evaluation of intelligent systems; S402, verify the generated question and answer QA through the third expert experience model; verify the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtain the question and answer QA social reasoning results that are close to the real social reasoning.

[0009] The present invention provides a social reasoning evaluation system for a video question-answering dataset, comprising: The video collection reasoning chain rule subsystem collects social interaction videos from the video interaction big data platform, removes the traces of previous identification and annotation in the video data set, performs visual question answering and analysis annotation, and annotates the rule-compliant question answering QA and description reasoning chains; The event and psychological state causal relationship model subsystem constructs the event and psychological state causal relationship based on the reasoning chain of rule-compliant QA and description, establishes the event and psychological state causal relationship model, and labels the diverse types of social causality; The reasoning chain expert verification subsystem verifies the data through the first expert experience model according to the social causal diversity type, and the reasoning chain is annotated by the second expert experience model; The question-answering reasoning chain evaluation system conducts comprehensive evaluation of question-answering QA and reasoning chains through a multi-modal large model with multiple evaluation indicators; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated question-answering QA and obtains question-answering QA social reasoning results that are close to real social reasoning.

[0010] Preferably, the video collection reasoning chain rule subsystem includes: The social interaction video collection and screening subsystem collects social interaction videos from the video interaction big data platform, removes low-quality social interaction videos with few interactions, few behaviors, and few events; screens out high-quality social interaction videos with many interactions, many behaviors, or many events, and aggregates them into a video data set; The video data extraction subsystem selects videos with incorrect answers and incorrect QA answers from the large model, and removes the traces of previous identification and annotation in the video data set; The visual question-answering, parsing and annotation subsystem performs visual question-answering and parsing annotation, performs data standard recognition and annotation platform rule training on the basic subject model, and obtains the basic subject training model; the basic subject training model annotates the rule-compliant QA and description reasoning chain from the video interactive big data platform according to the challenge form; The reasoning chain includes: video event reasoning, video character psychological state reasoning, and causal relationship reasoning between events and psychological states; A set of related questions and answers (QA) is generated based on each reasoning chain through the big model. The generated rules are as follows: for each node in the reasoning chain, a factual or mental state identification question is generated; For each sub-chain of reasoning in the chain of reasoning, a single-step causal reasoning problem of causal cause and causal effect is generated.

[0011] Preferably, the event and psychological state causal relationship model subsystem includes: The causal relationship architecture subsystem establishes the causal relationship between events and mental states based on the reasoning chain that conforms to the rule-based question and answer QA and description; The causal reasoning event and psychological state subsystem performs social causal reasoning based on the causal relationship between events and psychological states, understands the events that occur in the video, estimates the psychological state of the characters, and establishes a causal relationship model between events and psychological states; The social causal diversity subsystem labels the social causal diversity types according to the causal relationship model between events and psychological states; According to the causal relationship between events and psychological states, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological states of the characters. The causal relationship model between events and psychological states is established, including: selecting sample reasoning chains; understanding the events that occurred in the video; estimating the psychological states of people; establishing causal relationships; dividing problems into diversified social causal types; diversified social causal types include: event understanding type, psychological state estimation type, causal cause type and causal result type; by evaluating the performance of the causal relationship model between events and psychological states on various types of problems, the lack of model segmentation capabilities is analyzed.

[0012] Preferably, the reasoning chain expert verification subsystem includes: The expert model verification subsystem verifies the data through the first expert experience model according to the social causal diversity type and obtains the verified data; The data reasoning chain annotation subsystem verifies that the data is annotated by the second expert experience model through the reasoning chain.

[0013] Preferably, the question-answering reasoning chain scoring system includes: The question-answering comprehensive evaluation subsystem selects multiple evaluation indicators and uses a multi-modal large model with multiple evaluation indicators to conduct comprehensive evaluation of question-answering QA and reasoning chains; providing accurate data for the social behavior evaluation of intelligent systems; The question-and-answer (QA) social reasoning verification subsystem verifies the generated QA through the third expert experience model; verifies the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtains the QA social reasoning results that are close to the real social reasoning.

[0014] Compared with the prior art, the present invention has at least the following beneficial effects: The present invention discloses a method and system for evaluating social reasoning of a video question-and-answer data set. The method collects social interaction videos from a video interaction big data platform, removes the previously identified annotation traces of the video data set, performs visual question-and-answer and analytical annotation, and annotates the rule-compliant question-and-answer QA and description reasoning chains; constructs a causal relationship between events and psychological states based on the rule-compliant question-and-answer QA and description reasoning chains, establishes a causal relationship model between events and psychological states, and annotates the diversified types of social causality; verifies data based on the diversified types of social causality through a first expert experience model, and annotates the reasoning chains through a second expert experience model; comprehensively evaluates question-and-answer QA and reasoning chains through a multi-modal large model with multiple evaluation indicators; provides accurate data for social behavior evaluation of intelligent systems; verifies generated question-and-answer QA to obtain question-and-answer QA social reasoning results that are close to real social reasoning; QA is an intelligent question answering system. It is the abbreviation of Artificial Social Intelligence System; it can realize important social reasoning of artificial social intelligence, perceive and infer psychological states and clarify their causes and consequences during social interactions; it has very important technical significance to understand social reasoning in videos based on real-world records of videos; it can provide offline data support for social behavior evaluation and significantly improve the accuracy of reasoning.

[0015] The present invention describes a method and system for evaluating social reasoning in a video question-answering dataset. Other advantages, objectives, and features of the present invention will be partially reflected in the following description, and will also be partially understood by technicians in this field through research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a diagram of an embodiment of a social reasoning evaluation system for a video question-answering dataset described in the present invention.

[0017] Figure 2 This is another embodiment diagram of a social reasoning evaluation system for a video question-answering dataset described in the present invention.

[0018] Figure 3 This is another embodiment diagram of a social reasoning evaluation system and method for a video question-answering dataset described in the present invention. DETAILED DESCRIPTION

[0019] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments, so that those skilled in the art can implement it according to the instructions; as shown in the figure, the present invention provides a social reasoning evaluation method for a video question-answering dataset, including: S100 collects social interaction videos from the video interaction big data platform, removes the traces of previous identification and annotation in the video dataset, performs visual question answering and analysis annotation, and annotates the rule-compliant question answering (QA) and description reasoning chains; S200, based on the reasoning chain of rule-compliant QA and description, construct the causal relationship between events and psychological states, establish a causal relationship model between events and psychological states, and annotate the diverse types of social causality; S300, based on the social causal diversity type, the data is verified by the first expert experience model, and the reasoning chain is annotated by the second expert experience model; S400 conducts comprehensive evaluation of QA and reasoning chains through a multi-modal large model with multiple evaluation indicators; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated QA and obtains QA social reasoning results that are close to real social reasoning.

[0020] The principle and effect of the above technical solution are as follows: the present invention provides a social reasoning evaluation method for a video question-and-answer data set, including: collecting social interaction videos from a video interaction big data platform, removing the traces of previously identified annotations in the video data set, performing visual question-and-answer and analytical annotation, and annotating the rule-compliant question-and-answer QA and description reasoning chains; constructing a causal relationship between events and psychological states based on the reasoning chains that conform to the rule-compliant question-and-answer QA and descriptions, establishing a causal relationship model between events and psychological states, and annotating the diversified types of social causality; verifying the data based on the diversified types of social causality through a first expert experience model, and annotating the reasoning chains through a second expert experience model; conducting a comprehensive evaluation of question-and-answer QA and reasoning chains through a multi-modal large model with multiple evaluation indicators; providing accurate data for the social behavior evaluation of intelligent systems; verifying the generated question-and-answer QA to obtain question-and-answer QA social reasoning results that are close to real social reasoning; QA is an intelligent question answering system. It is the abbreviation of Artificial Social Intelligence System; it can realize important social reasoning of artificial social intelligence, perceive and infer psychological states and clarify their causes and consequences during social interactions; it has very important technical significance to understand social reasoning in videos based on real-world records of videos; it can provide offline data support for social behavior evaluation and significantly improve the accuracy of reasoning.

[0021] In one embodiment, S100 includes: S101, collect social interaction videos from the video interaction big data platform, remove low-quality social interaction videos with few interactions, few behaviors, and few events; screen out high-quality social interaction videos with many interactions, many behaviors, or many events, and aggregate them to form a video data set; S102, selecting videos with incorrect answers and incorrect questions and answers (QA) from the large model, and removing the identification and annotation traces from the video dataset; S103, perform visual question answering and analysis annotation, perform data standard recognition and annotation platform rule training on the basic subject model, and obtain the basic subject training model; the basic subject training model annotates the rule-compliant question answering QA and description reasoning chain from the video interactive big data platform according to the challenge form; The reasoning chain includes: video event reasoning, video character psychological state reasoning, and causal relationship reasoning between events and psychological states; A set of related questions and answers (QA) is generated based on each reasoning chain through the big model. The generated rules are as follows: for each node in the reasoning chain, a factual or mental state identification question is generated; For each sub-chain of reasoning in the chain of reasoning, a single-step causal reasoning problem of causal cause and causal effect is generated.

[0022] The principle and effect of the above technical solution are as follows: collect social interaction videos from the video interaction big data platform, remove low-quality social interaction videos with few interactions, few behaviors and few events; screen out high-quality social interaction videos with many interactions, many behaviors or many events, and aggregate them to form a video data set; select the large model to answer the wrong video and answer the wrong question and answer QA, and remove the recognition and annotation traces of the video data set; perform visual question and answer and analysis annotation, perform data standard identification and annotation platform rule training on the basic subject model, and obtain the basic subject training model; the basic subject training model annotates the rule-compliant QA and description reasoning chain from the video interaction big data platform according to the challenge form; the reasoning chain includes: video event reasoning, video character psychological state reasoning, and causal relationship reasoning between events and psychological states; generate a set of related questions and answers QA based on each reasoning chain through the large model; the generated rules are as follows: for each node in the reasoning chain, generate a factual or psychological state identification problem; for each sub-reasoning chain in the reasoning chain, generate a single-step causal reasoning problem of causal cause and causal result; based on the real-world record of the video, understanding the social reasoning in the video has very important technical significance.

[0023] In one embodiment, S200 includes: S201, establishing a causal relationship between events and mental states based on the reasoning chain of the rule-compliant question and answer QA and the description; S202, based on the causal relationship between events and psychological states, social causal reasoning is performed to understand the events occurring in the video and estimate the psychological states of the characters, and a causal relationship model between events and psychological states is established; S203, labeling social causal diversity types based on the causal relationship model between events and psychological states; According to the causal relationship between events and psychological states, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological states of the characters. The causal relationship model between events and psychological states is established, including: selecting sample reasoning chains; understanding the events that occurred in the video; estimating the psychological states of people; establishing causal relationships; dividing problems into diversified social causal types; diversified social causal types include: event understanding type, psychological state estimation type, causal cause type and causal result type; by evaluating the performance of the causal relationship model between events and psychological states on various types of problems, the lack of model segmentation capabilities is analyzed.

[0024] The principle and effect of the above technical solution are as follows: according to the reasoning chain of the question and answer QA and the description that conforms to the rules, the causal relationship between the event and the psychological state is established; according to the causal relationship between the event and the psychological state, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological state of the characters, and a causal relationship model between the event and the psychological state is established; according to the causal relationship model between the event and the psychological state, the diversified types of social causality are marked; according to the causal relationship between the event and the psychological state, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological state of the characters, and the causal relationship model between the event and the psychological state is established, including: selecting a sample reasoning chain; such as Figure 1-Figure 3 As shown; sample reasoning chains include: Robot2 reachesout his hand; to shake Robot1's hand, but Robot1 ignores him; -> Robot2 feels embarrassed -> Robot2 touches his head with his hand; understand the events in the video: Robot2 reachesout his hand, but Robot1 ignores him instead of shaking hands; Estimating a person's psychological state: Robot2 smiled after Robot1 ignored him, but this does not mean that he felt happy, but that he used the smile to cover up his embarrassment; Establish causality: Robot2 feels embarrassed because Robot2 reaches out his hand to shake Robot1's hand, but Robot1 ignores him, not because Robot1 shakes hands with an athlete in white clothes. Divide the problems into social causal diversity types: Social causal diversity types include: event understanding type, psychological state estimation type, causal cause type and causal result type; By evaluating the performance of the event and psychological state causal relationship model on various types of problems, the lack of model segmentation ability is analyzed; Event understanding type: Generate a factual question for each event node in the reasoning chain; To ensure the diversity of questions, two ways are designed to locate the event (Event) of the question: a. By the time period of the event, for example, what happened between 0s and 5s?; b. By the part of the event, for example, what did Robot1 do when Robot2 stretched out its hand; Establish the necessary condition of time and space perception in event understanding; Mental state estimation type: Establish social intelligence to evaluate mental state or infer mental state from behavior; divide mental state into affective mind and cognitive mind, namely emotion, belief, intent and desire; test the mental state estimation ability of the causal relationship model between events and mental states; generate questions based on mental state nodes in the reasoning chain, and locate them by time points or time periods, for example, what is the emotion of Robot2 at 0:03S; Causal reason type: corresponds to the cause type question; the cause type question forms include: Question: Why [effect]? Answer: Because [reasons] (Question: Why [effect]? Answer: Because [reasons]); Cause-effect type: corresponds to result-type questions; the forms of result-type questions include: Question: What do [reasons] lead to? Answer: [effect] (Question: [reasons] lead to what? Answer: [effect]); Causal cause type problems and causal result type problems. In the reasoning chain, the causes pointing to the same result node are in an AND relationship, while the result nodes pointed to by the same causes are in an OR relationship. The cause nodes pointing to the same result node in the reasoning chain are sufficient and necessary conditions, and the result cannot be deduced without any cause node. This enriches the examination form of the model's causal reasoning ability and avoids the situation where the model performs inconsistently on the two types of problems but cannot be tested.

[0025] In one embodiment, S300 includes: S301, verifying the data through the first expert experience model according to the social causal diversity type, and obtaining the verified data; S302, verify that the data is annotated by the second expert experience model through reasoning chain.

[0026] The principle and effect of the above technical solution are: according to the diversified types of social causality, the data is verified through the first expert experience model to obtain the verified data; the verified data is annotated with the reasoning chain by the second expert experience model; the expert experience model is evaluated and verified by the nominal multivariate expert framework based on a large amount of expert system knowledge database, and the multivariate expert experience is intelligently learned until the set verification accuracy is reached.

[0027] In one embodiment, S400 includes: S401, select multiple evaluation indicators, and conduct comprehensive evaluation of QA and reasoning chain through multiple evaluation indicators and multi-modal large models; provide accurate data for social behavior evaluation of intelligent systems; S402, verify the generated question and answer QA through the third expert experience model; verify the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtain the question and answer QA social reasoning results that are close to the real social reasoning.

[0028] The principle and effect of the above technical solution are: select a variety of evaluation indicators, and conduct a comprehensive evaluation of question-answering QA and reasoning chain through a multimodal large model with multiple evaluation indicators; provide accurate data for the social behavior evaluation of intelligent systems; verify the generated question-answering QA through a third expert experience model; verify the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtain the question-answering QA social reasoning results close to the real social reasoning; select a variety of evaluation indicators, and conduct a comprehensive evaluation of question-answering QA and reasoning chain of the multimodal large model through a variety of evaluation indicators, including: through the accuracy of psychological state recognition question-answering QA, the accuracy of event recognition question-answering QA, the accuracy of cause question-answering QA, and the accuracy of causal question-answering QA, the multimodal large model is comprehensively evaluated for question-answering QA and reasoning chain; the third expert experience model verifies the generated question-answering QA; the expert experience model is formed by a general expert model through QA data training to form QA data question-answering expert experience; the question of QA data training Question: Why does Robot 1 want to sleep in Robot 2's room? Training Q&A Evaluation Options: a. Because Robot 1 wants to discuss a scientific theory with Robot 2; b. Because Robot 1 is concerned about Robot 2's health and wants to monitor her overnight; c. Because Robot 1 needs a quiet place to work on his research and Robot 2's room is the quietest; d. Because Robot 1 is feeling afraid and seeks comfort by being in Robot 2's room; e.Because Robot 1 wants to surprise Robot 3 by being in Robot 2's room when he returns; ExpertsExplanation: Although Robot 1 said he was doing it for Robot 2, he clearlydidn't calmdown from his nightmare; His cautious tone indicated that he only wanted to sleep in Robot 2'shouse because he was afraid (Question: Why does Robot 1 want to sleep in Robot 2's room? Training Questions and Answers Assessment: Options: a. Because Robot 1 wants to discuss a scientific theory with Robot 2; b. Because Robot 1 is concerned about Robot 2's health and wants to monitor her all night; c. Because Robot 1 needs a quiet place to conduct research, and Robot 2's room is the quietest; d. Because Robot 1 is afraid and seeks comfort by staying in Robot 2's room; e. Because Robot 1 wanted to give Robot 3 a surprise when he came back, because he was in Robot 2's room; Expert experience model and explanation verification: Although Robot 1 said he was for Robot 2, he obviously did not calm down from the nightmare; his cautious tone showed that he just wanted to sleep in Robot 2's house because he was scared); Verify the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtain the QA social reasoning results close to the real social reasoning; significantly improve the accuracy of QA close to the real social reasoning. .

[0029] The present invention provides a social reasoning evaluation system for a video question-answering dataset, comprising: The video collection reasoning chain rule subsystem collects social interaction videos from the video interaction big data platform, removes the traces of previous identification and annotation in the video data set, performs visual question answering and analysis annotation, and annotates the rule-compliant question answering QA and description reasoning chains; The event and psychological state causal relationship model subsystem constructs the event and psychological state causal relationship based on the reasoning chain of rule-compliant QA and description, establishes the event and psychological state causal relationship model, and labels the diverse types of social causality; The reasoning chain expert verification subsystem verifies the data through the first expert experience model according to the social causal diversity type, and the reasoning chain is annotated by the second expert experience model; The question-answering reasoning chain evaluation system conducts comprehensive evaluation of question-answering QA and reasoning chains through a multi-modal large model with multiple evaluation indicators; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated question-answering QA and obtains question-answering QA social reasoning results that are close to real social reasoning.

[0030] The principle and effect of the above technical solution are as follows: the present invention provides a social reasoning evaluation system for video question-answering data sets, including: a video collection reasoning chain rule subsystem, which collects social interaction videos from a video interaction big data platform, removes the traces of identification and annotation of the video data set, performs visual question-answering and analysis annotation, and annotates the question-answering QA and description reasoning chains that meet the rules; an event and psychological state causal relationship model subsystem, which constructs the event and psychological state causal relationship, establishes the event and psychological state causal relationship model, and annotates the diversified types of social causality according to the reasoning chain that meets the rule question-answering QA and description; an inference chain expert verification subsystem, which verifies the data through the first expert experience model according to the diversified types of social causality, and annotates the reasoning chain by the second expert experience model; a question-answering reasoning chain evaluation subsystem, which conducts comprehensive evaluation of question-answering QA and reasoning chains through a multi-modal large model with multiple evaluation indicators; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated question-answering QA, and obtains the question-answering QA social reasoning results that are close to real social reasoning; social reasoning is an important basis for realizing artificial social intelligence, which can perceive and infer the psychological state of others and clarify its causes and consequences in the social process. Video is a record of the real world. Understanding social reasoning in videos is a very important topic and can also provide offline data for social behavior evaluation. QA is the abbreviation of Question Answering System. It can realize important social reasoning of artificial social intelligence and can perceive and infer psychological states and clarify their causes and consequences during social interactions. Understanding social reasoning in videos based on real-world records of videos is of great technical significance. It can provide offline data support for social behavior evaluation and significantly improve the accuracy of reasoning.

[0031] In one embodiment, the video collection reasoning chain rule subsystem includes: The social interaction video collection and screening subsystem collects social interaction videos from the video interaction big data platform, removes low-quality social interaction videos with few interactions, few behaviors, and few events; screens out high-quality social interaction videos with many interactions, many behaviors, or many events, and aggregates them into a video data set; The video data extraction subsystem selects videos with incorrect answers and incorrect QA answers from the large model, and removes the traces of previous identification and annotation in the video data set; The visual question-answering, parsing and annotation subsystem performs visual question-answering and parsing annotation, performs data standard recognition and annotation platform rule training on the basic subject model, and obtains the basic subject training model; the basic subject training model annotates the rule-compliant QA and description reasoning chain from the video interactive big data platform according to the challenge form; The reasoning chain includes: video event reasoning, video character psychological state reasoning, and causal relationship reasoning between events and psychological states; A set of related questions and answers (QA) is generated based on each reasoning chain through the big model. The generated rules are as follows: for each node in the reasoning chain, a factual or mental state identification question is generated; For each sub-chain of reasoning in the chain of reasoning, a single-step causal reasoning problem of causal cause and causal effect is generated.

[0032] The principles and effects of the above technical solution are as follows: the video collection reasoning chain rule subsystem includes: a social interaction video collection and screening subsystem, which collects social interaction videos from the video interaction big data platform, removes low-quality social interaction videos with few interactions, few behaviors and few events; screens out high-quality social interaction videos with many interactions, many behaviors or many events, and aggregates them into a video data set; a video data extraction subsystem, which selects large models to answer incorrect videos and answer incorrect questions and answers QA, and removes the traces of video data set recognition and annotation; a visual question and answer analysis and annotation subsystem, which performs visual question and answer and analysis annotation, performs data standard identification and annotation platform rule training on the basic subject model, and obtains the basic subject model. Trial training model; the basic subject training model marks the rule-compliant questions and answers QA and describes the reasoning chain from the video interaction big data platform according to the challenge form; the reasoning chain includes: video event reasoning, video character psychological state reasoning, and causal relationship reasoning between events and psychological states; a set of related questions and answers QA are generated based on each reasoning chain through the big model; the generated rules are as follows: for each node in the reasoning chain, a factual or psychological state identification question is generated; for each sub-reasoning chain in the reasoning chain, a single-step causal reasoning problem of causal cause and causal result is generated; based on the real-world records of the video, understanding the social reasoning in the video has very important technical significance.

[0033] In one embodiment, the event and mental state causal relationship model subsystem includes: The causal relationship architecture subsystem establishes the causal relationship between events and mental states based on the reasoning chain that conforms to the rule-based question and answer QA and description; The causal reasoning event and psychological state subsystem performs social causal reasoning based on the causal relationship between events and psychological states, understands the events that occur in the video, estimates the psychological state of the characters, and establishes a causal relationship model between events and psychological states; The social causal diversity subsystem labels the social causal diversity types according to the causal relationship model between events and psychological states; According to the causal relationship between events and psychological states, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological states of the characters. The causal relationship model between events and psychological states is established, including: selecting sample reasoning chains; understanding the events that occurred in the video; estimating the psychological states of people; establishing causal relationships; dividing problems into diversified social causal types; diversified social causal types include: event understanding type, psychological state estimation type, causal cause type and causal result type; by evaluating the performance of the causal relationship model between events and psychological states on various types of problems, the lack of model segmentation capabilities is analyzed.

[0034] The principle and effect of the above technical solution are as follows: the event and psychological state causal relationship model subsystem includes: the causal relationship architecture subsystem, which establishes the event and psychological state causal relationship between events and psychological states according to the reasoning chain that conforms to the rule question and answer QA and the description; the causal reasoning event and psychological state subsystem, which performs social causal reasoning based on the event and psychological state causal relationship, understands the events that occur in the video and estimates the psychological state of the characters, and establishes the event and psychological state causal relationship model; the social causal diversity subsystem, which labels the social causal diversity types according to the event and psychological state causal relationship model; performs social causal reasoning based on the event and psychological state causal relationship, understands the events that occur in the video and estimates the psychological state of the characters, and establishes the event and psychological state causal relationship model including: selecting sample reasoning chains; such as Figure 1-Figure 3 As shown; sample reasoning chains include: Robot2 reaches out his hand; to shake Robot1's hand, but Robot1 ignores him; -> Robot2 feels embarrassed -> Robot2 touches his head with his hand; understand the events in the video: Robot2 reaches out his hand;, but Robot1 ignores him instead of shaking hands; Estimating a person's psychological state: Robot2 smiled after Robot1 ignored him, but this does not mean that he felt happy, but that he used the smile to cover up his embarrassment; Establish causality: Robot2 feels embarrassed because Robot2 reaches out his hand to shake Robot1's hand, but Robot1 ignores him, not because Robot1 shakes hands with an athlete in white clothes. Divide the problems into social causal diversity types: Social causal diversity types include: event understanding type, psychological state estimation type, causal cause type and causal result type; By evaluating the performance of the event and psychological state causal relationship model on various types of problems, the lack of model segmentation ability is analyzed; Event understanding type: Generate a factual question for each event node in the reasoning chain; to ensure the diversity of questions, design two ways to locate the event of the question: a. By the time period when the event occurred, for example For example, what happened between 0s and 5s? b. Through the event part, for example, when Robot2 stretched out its hand, what did Robot1 do? Establish the necessary conditions of time and space perception in event understanding; Mental state estimation type: Establish social intelligence to evaluate mental state or infer mental state from behavior; divide mental state into affective mind and cognitive mind, namely emotion, belief, intent and desire; test the mental state estimation ability of the causal relationship model between events and mental states; generate questions based on mental state nodes in the reasoning chain, and locate them by time points or time periods, such as Robot2 in What is the emotion of 0:03S? Causal reason type: corresponds to the cause type question; the cause type question forms include: Question: Why [effect]? Answer: Because [reasons] (Question: Why [effect]? Answer: Because [reasons]); Cause-effect type: corresponds to result-type questions; the forms of result-type questions include: Question: What do [reasons] lead to? Answer: [effect] (Question: [reasons] lead to what? Answer: [effect]); Causal cause type problems and causal result type problems. In the reasoning chain, the causes pointing to the same result node are in an AND relationship, while the result nodes pointed to by the same causes are in an OR relationship. The cause nodes pointing to the same result node in the reasoning chain are sufficient and necessary conditions, and the result cannot be deduced without any cause node. This enriches the examination form of the model's causal reasoning ability and avoids the situation where the model performs inconsistently on the two types of problems but cannot be tested.

[0035] In one embodiment, the reasoning chain expert verification subsystem includes: The expert model verification subsystem verifies the data through the first expert experience model according to the social causal diversity type and obtains the verified data; The data reasoning chain annotation subsystem verifies that the data is annotated by the second expert experience model through the reasoning chain.

[0036] The principles and effects of the above technical solution are: the reasoning chain expert verification subsystem, including: the expert model verification subsystem, which verifies the data through the first expert experience model according to the social causal diversity type to obtain the verified data; the data reasoning chain labeling subsystem, the verified data is labeled by the second expert experience model for the reasoning chain; the expert experience model is based on a large number of expert system knowledge databases, through the nominal multivariate expert framework evaluation and verification, and multivariate expert experience intelligent learning until the set verification accuracy is reached.

[0037] In one embodiment, the question-answering reasoning chain scoring system includes: The question-answering comprehensive evaluation subsystem selects multiple evaluation indicators and uses a multi-modal large model with multiple evaluation indicators to conduct comprehensive evaluation of question-answering QA and reasoning chains; providing accurate data for the social behavior evaluation of intelligent systems; The question-and-answer (QA) social reasoning verification subsystem verifies the generated QA through the third expert experience model; verifies the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtains the QA social reasoning results that are close to the real social reasoning.

[0038] The principle and effect of the above technical solution are as follows: the question-answering reasoning chain evaluation system includes: a comprehensive question-answering evaluation subsystem, which selects a variety of evaluation indicators and conducts comprehensive evaluation of question-answering QA and reasoning chain through a multi-modal large model with multiple evaluation indicators; provides accurate data for the social behavior evaluation of intelligent systems; a question-answering QA social reasoning verification subsystem, which verifies the generated question-answering QA through a third expert experience model; verifies the clarity of questions, the rationality of answers, the spelling error rate, the diversity of expressions and the unique meaning, and obtains question-answering QA social reasoning results that are close to real social reasoning; selects multiple A variety of evaluation indicators are used to comprehensively evaluate the QA and reasoning chain of the multimodal large model, including: the accuracy of psychological state recognition QA, the accuracy of event recognition QA, the accuracy of cause QA, and the accuracy of causal QA, to comprehensively evaluate the QA and reasoning chain of the multimodal large model; the third expert experience model verifies the generated QA; the expert experience model is formed by the general expert model through QA data training to form QA data QA expert experience; QA data training questions Question: Why does Robot 1 want to sleep in Robot 2's room? Training Q&A Evaluation Options: a. Because Robot 1 wants to discuss a scientific theory with Robot 2; b. Because Robot 1 is concerned about Robot 2's health and wants to monitor her overnight; c. Because Robot 1 needs a quiet place to work on his research and Robot 2's room is the quietest; d. Because Robot 1 is feeling afraid and seeks comfort by being in Robot 2's room; e.Because Robot 1 wants to surprise Robot 3 by being in Robot 2's room when he returns; ExpertsExplanation: Although Robot 1 said he was doing it for Robot 2, he clearlydidn't calmdown from his nightmare; His cautious tone indicated that he only wanted to sleep in Robot 2'shouse because he was afraid (Question: Why does Robot 1 want to sleep in Robot 2's room? Training Questions and Answers Assessment: Options: a. Because Robot 1 wants to discuss a scientific theory with Robot 2; b. Because Robot 1 is concerned about Robot 2's health and wants to monitor her all night; c. Because Robot 1 needs a quiet place to conduct research, and Robot 2's room is the quietest; d. Because Robot 1 is afraid and seeks comfort by staying in Robot 2's room; e. Because Robot 1 wanted to give Robot 3 a surprise when he came back, because he was in Robot 2's room; Expert experience model and explanation verification: Although Robot 1 said he was for Robot 2, he obviously did not calm down from the nightmare; his cautious tone showed that he just wanted to sleep in Robot 2's house because he was scared); Verify the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtain the QA social reasoning results close to the real social reasoning; significantly improve the accuracy of QA close to the real social reasoning. .

[0039] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.

Claims

1. A social reasoning evaluation method for a video question-answering dataset, characterized in that: include: S100 collects social interaction videos from the video interaction big data platform, removes the traces of previous identification and annotation in the video dataset, performs visual question answering and analysis annotation, and annotates the rule-compliant question answering (QA) and description reasoning chains; S200, based on the reasoning chain of rule-compliant QA and description, construct the causal relationship between events and psychological states, establish a causal relationship model between events and psychological states, and annotate the diverse types of social causality; S300, based on the social causal diversity type, the data is verified by the first expert experience model, and the reasoning chain is annotated by the second expert experience model; S400 conducts comprehensive evaluation of QA and reasoning chains through a multi-modal large model with multiple evaluation indicators; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated QA and obtains QA social reasoning results that are close to real social reasoning.

2. According to claim 1, a method for evaluating social reasoning of a video question-answering dataset is characterized in that: S100 includes: S101, collect social interaction videos from the video interaction big data platform, remove low-quality social interaction videos with few interactions, few behaviors, and few events; screen out high-quality social interaction videos with many interactions, many behaviors, or many events, and aggregate them to form a video data set; S102, selecting videos with incorrect answers and incorrect questions and answers (QA) from the large model, and removing the identification and annotation traces from the video dataset; S103, perform visual question answering and analysis annotation, perform data standard recognition and annotation platform rule training on the basic subject model, and obtain the basic subject training model; the basic subject training model annotates the rule-compliant question answering QA and description reasoning chain from the video interactive big data platform according to the challenge form; The reasoning chain includes: video event reasoning, video character psychological state reasoning, and causal relationship reasoning between events and psychological states; A set of related questions and answers (QA) is generated based on each reasoning chain through the big model. The generated rules are as follows: for each node in the reasoning chain, a factual or mental state identification question is generated; For each sub-chain of reasoning in the chain of reasoning, a single-step causal reasoning problem of causal cause and causal effect is generated.

3. The social reasoning evaluation method for a video question-answering dataset according to claim 1, characterized in that: S200 includes: S201, establishing a causal relationship between events and mental states based on the reasoning chain of the rule-compliant question and answer QA and the description; S202, based on the causal relationship between events and psychological states, social causal reasoning is performed to understand the events occurring in the video and estimate the psychological states of the characters, and a causal relationship model between events and psychological states is established; S203, labeling social causal diversity types based on the causal relationship model between events and psychological states; According to the causal relationship between events and psychological states, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological states of the characters. The causal relationship model between events and psychological states is established, including: selecting sample reasoning chains; understanding the events that occurred in the video; estimating the psychological states of people; establishing causal relationships; dividing problems into diversified social causal types; diversified social causal types include: event understanding type, psychological state estimation type, causal cause type and causal result type; by evaluating the performance of the causal relationship model between events and psychological states on various types of problems, the lack of model segmentation capabilities is analyzed.

4. According to claim 1, a method for evaluating social reasoning of a video question-answering dataset is characterized in that: S300 includes: S301, verifying the data through the first expert experience model according to the social causal diversity type, and obtaining the verified data; S302, verify that the data is annotated by the second expert experience model through reasoning chain.

5. The social reasoning evaluation method for a video question-answering dataset according to claim 1, characterized in that: S400 includes: S401, select multiple evaluation indicators, and conduct comprehensive evaluation of QA and reasoning chain through multiple evaluation indicators and multi-modal large models; provide accurate data for the social behavior evaluation of intelligent systems; S402, verify the generated question and answer QA through the third expert experience model; verify the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtain the question and answer QA social reasoning results that are close to the real social reasoning.

6. A social reasoning evaluation system for video question-answering datasets, characterized in that: include: The video collection reasoning chain rule subsystem collects social interaction videos from the video interaction big data platform, removes the traces of previous identification and annotation in the video data set, performs visual question answering and analysis annotation, and annotates the rule-compliant question answering QA and description reasoning chains; The event and psychological state causal relationship model subsystem constructs the event and psychological state causal relationship based on the reasoning chain of rule-compliant QA and description, establishes the event and psychological state causal relationship model, and labels the diverse types of social causality; The reasoning chain expert verification subsystem verifies the data through the first expert experience model according to the social causal diversity type, and the reasoning chain is annotated by the second expert experience model; The question-answering reasoning chain evaluation system conducts comprehensive evaluation of question-answering QA and reasoning chains through a multi-modal large model with multiple evaluation indicators; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated question-answering QA and obtains question-answering QA social reasoning results that are close to real social reasoning.

7. The social reasoning evaluation system for video question-answering dataset according to claim 6, characterized in that: Video collection reasoning chain rule subsystem, including: The social interaction video collection and screening subsystem collects social interaction videos from the video interaction big data platform, removes low-quality social interaction videos with few interactions, few behaviors, and few events; screens out high-quality social interaction videos with many interactions, many behaviors, or many events, and aggregates them into a video data set; The video data extraction subsystem selects videos with incorrect answers and incorrect QA answers from the large model, and removes the traces of previous identification and annotation in the video data set; The visual question-answering, parsing and annotation subsystem performs visual question-answering and parsing annotation, performs data standard recognition and annotation platform rule training on the basic subject model, and obtains the basic subject training model; the basic subject training model annotates the rule-compliant QA and description reasoning chain from the video interactive big data platform according to the challenge form; The reasoning chain includes: video event reasoning, video character psychological state reasoning, and causal relationship reasoning between events and psychological states; A set of relevant questions and answers (QA) is generated based on each chain of reasoning through the big model. The generated rules are as follows: for each node in the chain of reasoning, a factual or mental state identification question is generated; For each sub-chain of reasoning in the chain of reasoning, a single-step causal reasoning problem of causal cause and causal effect is generated.

8. The social reasoning evaluation system for video question-answering dataset according to claim 6, characterized in that: The event and mental state causal relationship model subsystem includes: The causal relationship architecture subsystem establishes the causal relationship between events and mental states based on the reasoning chain that conforms to the rule-based question and answer QA and description; The causal reasoning event and psychological state subsystem performs social causal reasoning based on the causal relationship between events and psychological states, understands the events that occur in the video, estimates the psychological state of the characters, and establishes a causal relationship model between events and psychological states; The social causal diversity subsystem labels the social causal diversity types according to the causal relationship model between events and psychological states; According to the causal relationship between events and psychological states, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological states of the characters. The causal relationship model between events and psychological states is established, including: selecting sample reasoning chains; understanding the events that occurred in the video; estimating the psychological states of people; establishing causal relationships; dividing problems into diversified social causal types; diversified social causal types include: event understanding type, psychological state estimation type, causal cause type and causal result type; by evaluating the performance of the causal relationship model between events and psychological states on various types of problems, the lack of model segmentation capabilities is analyzed.

9. The social reasoning evaluation system for video question-answering dataset according to claim 6, characterized in that: Reasoning chain expert verification subsystem, including: The expert model verification subsystem verifies the data through the first expert experience model according to the social causal diversity type and obtains the verified data; The data inference chain annotation subsystem verifies that the data is annotated by the second expert experience model through the inference chain.

10. The social reasoning evaluation system for video question-answering dataset according to claim 6, characterized in that: The question-answering reasoning chain test scoring system includes: The question-answering comprehensive evaluation subsystem selects multiple evaluation indicators and uses a multi-modal large model with multiple evaluation indicators to conduct comprehensive evaluation of question-answering QA and reasoning chains; providing accurate data for the social behavior evaluation of intelligent systems; The question-and-answer (QA) social reasoning verification subsystem verifies the generated QA through the third expert experience model; verifies the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtains the QA social reasoning results that are close to the real social reasoning.

Images