A Social Reasoning Evaluation Method and System for Video Question-Answering Datasets

Abstract

The present invention provides a method and system for social reasoning evaluation of a video question-answer data set; by collecting social interaction videos from a video interaction big data platform, removing the identification and annotation traces of the video data set, performing visual question-answering and parsing annotation, and annotating the question-answer QAs and description inference chains that conform to the rules; according to the question-answer QAs and description inference chains that conform to the rules, constructing the causal relationship between events and mental states, establishing a causal relationship model between events and mental states, and annotating the types of social causality diversification; according to the types of social causality diversification, verifying the data through a first expert experience model, and performing inference chain annotation by a second expert experience model; comprehensively evaluating the question-answer QAs and inference chains through a multi-modal large model with multiple evaluation indicators; providing accurate data for the social behavior evaluation of intelligent systems; verifying the generated question-answer QAs, and obtaining the question-answer QA social reasoning results close to real social reasoning.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent system inference social question and answer evaluation, and more specifically, the present invention relates to a method and system for social inference evaluation of a video question and answer dataset. Background Art

[0002] Theory of Mind and social reasoning. The Theory of Mind is a language dataset for testing the social interaction and social reasoning abilities of language models that has been studied extensively (SocialIQa, CleverHans, Fantom, OpenToM, LLMFailTrivialToM). High-quality videos are difficult to generate. In contrast, language datasets are easier to construct, but the information expressed in text is too direct to examine the model's ability to establish causal relationships from a large number of small clues. Video question and answer. With the emergence and development of multimodal large models, video question and answer has become the main way to test the video understanding ability of models. Researchers have proposed some video question and answer benchmarks for testing large models; MMBench-Video, Video-MME, Egoschema, MVBench, NextQA, ActivityNet-QA, etc. are commonly used test benchmarks. They focus more on the factual understanding of video content and lack the examination of people's mental states. IntentQA and MELD only focus on one mental state and lack a comprehensive examination of the model's ability to estimate others' mental states. VAR examines the model's ability to capture causal relationships and abductive reasoning, but it is not a video question and answer task and is not applicable to existing multimodal large models. Moreover, it only focuses on the causal relationship between events; CausalChaos involves causal reasoning of mental states, but its videos are sourced from online video information, not the real world, and do not comprehensively examine the model's multimodal capabilities in combination with language information. Additionally, its method of examining causal reasoning ability is relatively simple, only providing explanations corresponding to the Q&A. MMToM-QA is a video question and answer dataset for the Theory of Mind, but the data is collected in a virtual environment, there is a gap with the real environment, and it only examines the recognition of belief and goal. SocialIQ is a dataset covering multiple aspects of social intelligence, but there are still problems to be solved such as the lack of examination of the ability to capture and reason about visual clues; therefore, it is necessary to propose a method and system for social inference evaluation of a video question and answer dataset to at least partially solve the problems existing in the prior art. Summary of the Invention

[0003] A series of simplified concepts are introduced in the Summary of the Invention section, which will be further elaborated in detail in the Detailed Implementation section; the Summary of the Invention section of the present invention does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the protection scope of the claimed technical solution.

[0004] To at least partially solve the above problems, the present invention provides a method for social reasoning evaluation of a video question-answering data set, including:

[0005] S100, collect social interaction videos from a video interaction big data platform, remove the identification and annotation traces of the video data set, perform visual question answering and parsing annotation, and annotate the question-and-answer QAs and descriptive reasoning chains that conform to the rules therein;

[0006] S200, construct the causal relationship between events and mental states according to the question-and-answer QAs and descriptive reasoning chains that conform to the rules, establish a causal relationship model between events and mental states, and annotate the types of social causality diversification;

[0007] S300, verify the data according to the types of social causality diversification through a first expert experience model, and perform reasoning chain annotation by a second expert experience model;

[0008] S400, comprehensively evaluate the question-and-answer QAs and reasoning chains through a multi-modal large model with multiple evaluation indicators; provide accurate data for the social behavior evaluation of intelligent systems; verify the generated question-and-answer QAs, and obtain the social reasoning results of the question-and-answer QAs that are close to real social reasoning.

[0009] Preferably, S100 includes:

[0010] S101, collect social interaction videos from a video interaction big data platform, remove the low-quality social interaction videos with few interactions, few behaviors, and few events; screen out the high-quality social interaction videos with many interactions, many behaviors, or many events, and form a video data set by aggregation;

[0011] S102, select the videos with incorrect answers by the large model and the incorrect question-and-answer QAs, and remove the identification and annotation traces of the video data set;

[0012] S103, perform visual question answering and parsing annotation, perform data standard recognition and annotation platform rule training on the basic subject model, and obtain a basic subject training model; the basic subject training model annotates the question-and-answer QAs and descriptive reasoning chains that conform to the rules therein from the video interaction big data platform according to the challenge form;

[0013] The reasoning chain includes: video event reasoning, video character mental state reasoning, and causal relationship reasoning between events and mental states;

[0014] Generate a group of related question-and-answer QAs for each reasoning chain through the large model; the generation rules are as follows: for each node in the reasoning chain, generate a factual or mental state discrimination question;

[0015] For each sub - inference chain in the inference chain, generate a single - step causal inference question about the causal cause and causal effect.

[0016] Preferably, S200 includes:

[0017] S201, establish the causal relationship between events and mental states according to the rule - compliant question - answering QA and the described inference chain;

[0018] S202, perform social causal reasoning based on the causal relationship between events and mental states, understand the events occurring in the video and estimate the mental states of the characters, and establish a causal relationship model between events and mental states;

[0019] S203, label the social causal diversification types according to the causal relationship model between events and mental states;

[0020] Performing social causal reasoning based on the causal relationship between events and mental states, understanding the events occurring in the video and estimating the mental states of the characters, and establishing a causal relationship model between events and mental states includes: selecting sample inference chains; understanding the events occurring in the video; estimating the mental states of people; establishing causal relationships; dividing the questions into social causal diversification types; the social causal diversification types include: event understanding type, mental state estimation type, causal cause type, and causal effect type; by evaluating the performance of the causal relationship model between events and mental states on various types of questions, analyzing the lack of ability in the model's detailed classification.

[0021] Preferably, S300 includes:

[0022] S301, verify the data according to the social causal diversification types through the first expert experience model to obtain the verified - passed data;

[0023] S302, the verified - passed data is used for inference chain annotation by the second expert experience model.

[0024] Preferably, S400 includes:

[0025] S401, select multiple evaluation metrics, and conduct a comprehensive evaluation of question - answering QA and inference chains through the multi - modal large - model with multiple evaluation metrics; provide accurate data for the social behavior evaluation of intelligent systems;

[0026] S402, verify the generated question - answering QA through the third expert experience model; verify the question clarity, answer rationality, spelling error rate, diverse expressions, and unique meanings, and obtain the social reasoning results of the question - answering QA close to real social reasoning.

[0027] The present invention provides a social reasoning evaluation system for a video question - answering data set, including:

[0028] The video collection inference chain rule subsystem collects social interaction videos from the video interaction big data platform, removes the identification and annotation traces in the video dataset, conducts visual question answering and parsing annotation, and annotates the question-and-answer QAs and description inference chains that conform to the rules among them;

[0029] The event and mental state causal relationship model subsystem constructs the causal relationship between events and mental states according to the question-and-answer QAs and the described inference chains that conform to the rules, establishes the event and mental state causal relationship model, and annotates the types of social causality diversification;

[0030] The inference chain expert verification subsystem verifies the data according to the types of social causality diversification through the first expert experience model, and conducts inference chain annotation by the second expert experience model;

[0031] The question-and-answer inference chain evaluation system conducts a comprehensive evaluation of the question-and-answer QAs and inference chains through a variety of evaluation indicators and multi-modal large models; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated question-and-answer QAs, and obtains the question-and-answer QA social inference results that are close to real social inferences.

[0032] Preferably, the video collection inference chain rule subsystem includes:

[0033] The social interaction video collection and screening subsystem collects social interaction videos from the video interaction big data platform, and removes the low-quality social interaction videos with less interaction, less behavior, and fewer events; screens out the high-quality social interaction videos with more interaction, more behavior, or more events, and forms a video dataset in a centralized manner;

[0034] The video data extraction subsystem selects the videos with incorrect answers from the large model and the question-and-answer QAs with incorrect answers, and removes the identification and annotation traces in the video dataset;

[0035] The visual question answering parsing annotation subsystem conducts visual question answering and parsing annotation, conducts data standard recognition and annotation platform rule training on the basic subject model, and obtains the basic subject training model; the basic subject training model annotates the question-and-answer QAs and description inference chains that conform to the rules among them from the video interaction big data platform according to the challenge form;

[0036] The inference chain includes: video event inference, video character mental state inference, and causal relationship inference between events and mental states;

[0037] Based on each inference chain, a large model generates a group of related question-and-answer QAs; the generation rules are as follows: for each node in the inference chain, generate a factual or mental state discrimination question;

[0038] For each sub-inference chain in the inference chain, generate a single-step causal inference question about the causal reason and causal result.

[0039] Preferably, the event and psychological state causal relationship model subsystem includes:

[0040] The causal relationship architecture subsystem establishes the causal relationship between events and mental states based on the reasoning chain that conforms to the rule-based question and answer QA and description;

[0041] The causal reasoning event and psychological state subsystem performs social causal reasoning based on the causal relationship between events and psychological states, understands the events that occur in the video, estimates the psychological state of the characters, and establishes a causal relationship model between events and psychological states;

[0042] The social causal diversity subsystem labels the social causal diversity types according to the causal relationship model between events and psychological states;

[0043] According to the causal relationship between events and psychological states, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological states of the characters. The causal relationship model between events and psychological states is established, including: selecting sample reasoning chains; understanding the events that occurred in the video; estimating the psychological states of people; establishing causal relationships; dividing problems into diversified social causal types; diversified social causal types include: event understanding type, psychological state estimation type, causal cause type and causal result type; by evaluating the performance of the causal relationship model between events and psychological states on various types of problems, the lack of model segmentation capabilities is analyzed.

[0044] Preferably, the reasoning chain expert verification subsystem includes:

[0045] The expert model verification subsystem verifies the data through the first expert experience model according to the social causal diversity type and obtains the verified data;

[0046] The data reasoning chain annotation subsystem verifies that the data is annotated by the second expert experience model through the reasoning chain.

[0047] Preferably, the question-answering reasoning chain scoring system includes:

[0048] The question-answering comprehensive evaluation subsystem selects multiple evaluation indicators and uses a multi-modal large model with multiple evaluation indicators to conduct comprehensive evaluation of question-answering QA and reasoning chains; providing accurate data for the social behavior evaluation of intelligent systems;

[0049] The question-and-answer (QA) social reasoning verification subsystem verifies the generated QA through the third expert experience model; verifies the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtains the QA social reasoning results that are close to the real social reasoning.

[0050] Compared with the prior art, the present invention has at least the following beneficial effects:

[0051] A social reasoning evaluation method and system for a video question-answering dataset of the present invention collects social interaction videos from a video interaction big data platform, removes the identification and annotation traces of the video dataset, performs visual question-answering and parsing annotation, and annotates the question-answer Q&A that conforms to the rules and the description of the reasoning chain; according to the question-answer Q&A that conforms to the rules and the reasoning chain of the description, constructs the causal relationship between events and mental states, establishes a causal relationship model between events and mental states, and annotates the diverse types of social causality; according to the diverse types of social causality, verifies the data through the first expert experience model, and performs reasoning chain annotation by the second expert experience model; comprehensively evaluates the question-answer Q&A and the reasoning chain through multiple evaluation indicators and a multi-modal large model; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated question-answer Q&A to obtain the question-answer Q&A social reasoning result close to real social reasoning; Q&A is the abbreviation of Question Answering System; can realize important social reasoning of artificial social intelligence, can perceive and infer mental states and clarify their causes and results during social interaction; understands the social reasoning in the video according to the real-world record of the video, which has very important technical significance; can provide offline data support for social behavior evaluation and significantly improve the reasoning accuracy.

[0052] A social reasoning evaluation method and system for a video question-answering dataset of the present invention, other advantages, objectives and features of the present invention will be partially reflected by the following description, and partially will be understood by those skilled in the art through the research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0054] Figure 1 It is a diagram of an embodiment of a social reasoning evaluation system for a video question-answering dataset of the present invention.

[0055] Figure 2 It is a diagram of another embodiment of a social reasoning evaluation system for a video question-answering dataset of the present invention.

[0056] Figure 3 It is a diagram of another embodiment of a social reasoning evaluation system and method for a video question-answering dataset of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] The following further describes the present invention in detail with reference to the drawings and embodiments, so that those skilled in the art can implement it according to the specification; as shown in the figure, the present invention provides a social reasoning evaluation method for a video question-answering dataset, including:

[0058] S100: Collect social interaction videos from the video interaction big data platform, remove the identification and annotation traces in the video dataset, perform visual question answering and parsing annotation, and annotate the question-and-answer Q&A and descriptive reasoning chains that conform to the rules therein.

[0059] S200: Based on the question-and-answer Q&A and descriptive reasoning chains that conform to the rules, construct the causal relationship between events and mental states, establish a causal relationship model between events and mental states, and annotate the diverse types of social causality.

[0060] S300: According to the diverse types of social causality, verify the data through the first expert experience model, and perform reasoning chain annotation by the second expert experience model.

[0061] S400: Conduct a comprehensive evaluation of the question-and-answer Q&A and reasoning chains through a multi-modal large model with multiple evaluation metrics; provide accurate data for the social behavior evaluation of intelligent systems; verify the generated question-and-answer Q&A to obtain the Q&A social reasoning results that are close to real social reasoning.

[0062] The principle and effect of the above technical solution are as follows: The present invention provides a method for social reasoning evaluation of a video question-and-answer dataset, including: collecting social interaction videos from the video interaction big data platform, removing the identification and annotation traces in the video dataset, performing visual question answering and parsing annotation, and annotating the question-and-answer Q&A and descriptive reasoning chains that conform to the rules therein; constructing the causal relationship between events and mental states based on the question-and-answer Q&A and descriptive reasoning chains that conform to the rules, establishing a causal relationship model between events and mental states, and annotating the diverse types of social causality; according to the diverse types of social causality, verifying the data through the first expert experience model, and performing reasoning chain annotation by the second expert experience model; conducting a comprehensive evaluation of the question-and-answer Q&A and reasoning chains through a multi-modal large model with multiple evaluation metrics; providing accurate data for the social behavior evaluation of intelligent systems; verifying the generated question-and-answer Q&A to obtain the Q&A social reasoning results that are close to real social reasoning; Q&A is the abbreviation of the intelligent question answering Question Answering System; it can realize important social reasoning of artificial social intelligence, can perceive and infer mental states and clarify their causes and results during the social interaction process; understand the social reasoning in the video according to the real-world records of the video, which has very important technical significance; it can provide offline data support for social behavior evaluation and significantly improve the reasoning accuracy.

[0063] In one embodiment, S100 includes:

[0064] S101: Collect social interaction videos from the video interaction big data platform, remove the low-quality social interaction videos with less interaction, fewer behaviors, and fewer events; screen out the high-quality social interaction videos with more interaction, more behaviors, or more events, and form a video dataset by aggregation.

[0065] S102, Select videos and Q&A (QA) with incorrect answers from the large model, and remove the identification and annotation traces previously recognized in the video dataset;

[0066] S103, Conduct visual question answering and parsing annotation, train the basic subject model on data standard recognition and annotation platform rules, and obtain the basic subject training model; The basic subject training model marks the Q&A (QA) that conforms to the rules and describes the reasoning chain from the video interaction big data platform according to the challenge form;

[0067] The reasoning chain includes: video event reasoning, video character mental state reasoning, and causal relationship reasoning between events and mental states;

[0068] Generate a set of related Q&A (QA) based on each reasoning chain through the large model; The generation rules are as follows: For each node in the reasoning chain, generate a factual or mental state discrimination question;

[0069] For each sub-reasoning chain in the reasoning chain, generate a single-step causal reasoning question about the causal cause and causal result.

[0070] The principle and effect of the above technical solution are: Collect social interaction videos from the video interaction big data platform, and remove low-quality social interaction videos with less interaction, fewer behaviors, and fewer events; Screen out high-quality social interaction videos with more interaction, more behaviors, or more events, and form a video dataset by aggregation; Select videos and Q&A (QA) with incorrect answers from the large model, and remove the identification and annotation traces previously recognized in the video dataset; Conduct visual question answering and parsing annotation, train the basic subject model on data standard recognition and annotation platform rules, and obtain the basic subject training model; The basic subject training model marks the Q&A (QA) that conforms to the rules and describes the reasoning chain from the video interaction big data platform according to the challenge form; The reasoning chain includes: video event reasoning, video character mental state reasoning, and causal relationship reasoning between events and mental states; Generate a set of related Q&A (QA) based on each reasoning chain through the large model; The generation rules are as follows: For each node in the reasoning chain, generate a factual or mental state discrimination question; For each sub-reasoning chain in the reasoning chain, generate a single-step causal reasoning question about the causal cause and causal result; Understanding the social reasoning in the video based on the real-world record of the video has very important technical significance.

[0071] In one embodiment, S200 includes:

[0072] S201, Establish the causal relationship between events and mental states based on the Q&A (QA) that conforms to the rules and the described reasoning chain;

[0073] S202. Perform social causal reasoning based on the causal relationship between events and mental states, understand the events occurring in the video, estimate the mental states of the characters, and establish a causal relationship model between events and mental states.

[0074] S203. Label diverse types of social causality according to the causal relationship model between events and mental states.

[0075] Performing social causal reasoning based on the causal relationship between events and mental states, understanding the events occurring in the video, estimating the mental states of the characters, and establishing a causal relationship model between events and mental states includes: selecting a sample reasoning chain; understanding the events occurring in the video; estimating the mental states of people; establishing a causal relationship; dividing the problem into diverse types of social causality; diverse types of social causality include: event understanding type, mental state estimation type, causal reason type, and causal result type; by evaluating the performance of the causal relationship model between events and mental states on various types of problems, it is analyzed that the model lacks in terms of its detailed capabilities.

[0076] The principle and effect of the above technical solution are as follows: Based on the reasoning chain that conforms to the rules of question - answering QA and description, establish a causal relationship between events and mental states; perform social causal reasoning based on the causal relationship between events and mental states, understand the events occurring in the video, estimate the mental states of the characters, and establish a causal relationship model between events and mental states; label diverse types of social causality according to the causal relationship model between events and mental states; performing social causal reasoning based on the causal relationship between events and mental states, understanding the events occurring in the video, estimating the mental states of the characters, and establishing a causal relationship model between events and mental states includes: selecting a sample reasoning chain; as Figures 1 - 3 shown; the sample reasoning chain includes: Robot2 reaches out his hand (Robot 2 extends his hand); to shake Robot1’s hand, but Robot1 ignores him (to shake hands with Robot 1, but Robot 1 ignores him); -> Robot2 feels embarrassed -> Robot2 touches his head with his hand (Robot 2 reaches out to shake hands with Robot 1, but Robot 1 ignores him -> Robot 2 feels embarrassed -> Robot 2 touches his head with his hand); understanding the events occurring in the video: Robot2 reaches out his hand (Robot 2 extends his hand), but Robot1 ignores him instead of shaking hands with him.

[0077] Estimating the mental state of people: Robot2 smiles after Robot1 ignores him, but this doesn't mean he is happy. Instead, he uses the smile to cover up his embarrassment.

[0078] Establishing causal relationships: The reason why Robot2 feels embarrassed is that Robot2 reaches out his hand; to shake Robot1’s hand, but Robot1 ignores him;, rather than Robot1 shaking hands with an athlete in white clothes;

[0079] Dividing the problem into socially causally diverse types: Socially causally diverse types include: event understanding type, mental state estimation type, causal reason type, and causal result type; By evaluating the performance of the event and mental state causal relationship model on various types of problems, it is analyzed that the model lacks the ability of fine-grained discrimination;

[0080] Event understanding type: Generate a factual question for each event node in the inference chain; To ensure the diversity of questions, design two ways to locate the event being questioned: a. By the time period when the event occurs, for example, what happened within 0s - 5s?; b. By part of the event, for example, what did Robot1 do after Robot2 reached out his hand; Establish the necessary condition spatio-temporal perception in event understanding;

[0081] Mental state estimation type: Establish a social intelligence to evaluate mental states or infer mental states from behaviors; Divide mental states into affective mind and cognitive mind, namely emotion, belief, intent, and desire; Test the mental state estimation ability of the event and mental state causal relationship model; Generate questions based on the mental state nodes in the inference chain, located by time points or time periods, for example, what is Robot2's emotion at

[0082] 0:03S?

[0083] Causal reason type: Corresponding to reason-based questions; The form of reason-based questions includes: Question: Why [effect]? Answer: Because [reasons];

[0084] Causal result type: Corresponding to result-based questions; The form of result-based questions includes: Question: What do [reasons] lead to? Answer: [effect];

[0085] Causal cause type problems and causal result type problems. In the inference chain, multiple causes pointing to the same result node are in an AND relationship, while the result nodes pointed to by the same multiple causes are in an OR relationship; the cause nodes pointing to the same result node in the inference chain are sufficient and necessary conditions, and the result cannot be deduced without any one of the cause nodes; it enriches the examination form of the model's causal reasoning ability and prevents the model from showing inconsistent performance on the two types of problems but being unable to be tested.

[0086] In one embodiment, S300 includes:

[0087] S301, according to the social causal diversification type, through the first expert experience model, verify the data and obtain the verified data;

[0088] S302, the verified data is annotated with an inference chain by the second expert experience model.

[0089] The principle and effect of the above technical solution are: according to the social causal diversification type, through the first expert experience model, verify the data and obtain the verified data; the verified data is annotated with an inference chain by the second expert experience model; the expert experience model is evaluated and verified through the nominal multiple expert framework based on a large number of expert system knowledge databases, and multiple expert experience intelligent learning is carried out until the set verification accuracy is reached.

[0090] In one embodiment, S400 includes:

[0091] S401, select multiple evaluation indicators, and conduct comprehensive evaluation of question answering QA and inference chain through the multi-modal large model with multiple evaluation indicators; provide accurate data for the social behavior evaluation of intelligent systems;

[0092] S402, verify the generated question answering QA through the third expert experience model; verify the question clarity, answer reasonableness, spelling error rate, diverse expressions and unique meanings, and obtain the question answering QA social reasoning results close to real social reasoning.

[0093] The principle and effect of the above technical solution are as follows: Select a variety of evaluation indicators, and conduct a comprehensive evaluation of question answering (QA) and reasoning chains for the multi-modal large model through a variety of evaluation indicators; provide accurate data for the social behavior evaluation of intelligent systems; verify the generated QA through the third expert experience model; verify the clarity of questions, the reasonableness of answers, the spelling error rate, diverse expressions and unique meanings, and obtain the QA social reasoning results close to real social reasoning; select a variety of evaluation indicators, and conduct a comprehensive evaluation of QA and reasoning chains for the multi-modal large model through a variety of evaluation indicators, including: comprehensively evaluating the QA and reasoning chains of the multi-modal large model through the accuracy rate of mental state recognition QA, the accuracy rate of event recognition QA, the accuracy rate of reason QA, and the accuracy rate of causal QA; the third expert experience model verifies the generated QA; the expert experience model is formed by training the general expert model with QA data to obtain the QA data question answering expert experience; the question for QA data training is Question: Why does Robot 1 want to sleep in Robot 2's room? The training QA evaluation options are: a. Because Robot 1 wants to discuss a scientific theory with Robot 2; b. Because Robot 1 is concerned about Robot 2's health and wants to monitor her overnight; c. Because Robot 1 needs a quiet place to work on his research and Robot 2's room is the quietest; d. Because Robot 1 is feeling afraid and seeks comfort by being in Robot 2's room; e.Because Robot 1 wants to surprise Robot 3 by being in Robot 2's room when he returns; Expert Experience Explanation: Although Robot 1 said he was doing it for Robot 2, he clearly didn't calm down from his nightmare; His cautious tone indicated that he only wanted to sleep in Robot 2's house because he was afraid (Question: Why does Robot 1 want to sleep in Robot 2's room? Training Q&A Assessment: Options: a. Because Robot 1 wants to discuss a scientific theory with Robot 2; b. Because Robot 1 cares about Robot 2's health and wants to monitor her all night; c. Because Robot 1 needs a quiet place to conduct research, and Robot 2's room is the quietest; d. Because Robot 1 is afraid and seeks comfort by staying in Robot 2's room; e. Because Robot 1 wants to surprise Robot 3 when he returns by being in Robot 2's room; Expert Experience Model and Explanation Verification: Although Robot 1 said he was doing it for Robot 2, he clearly didn't calm down from his nightmare; His cautious tone indicated that he only wanted to sleep in Robot 2's house because he was afraid); Verify the question clarity, answer reasonableness, spelling error rate, diverse expressions and unique meanings, and obtain Q&A QA social reasoning results close to real social reasoning; Significantly improve the accuracy of Q&A QA close to real social reasoning.

[0094] The present invention provides a social reasoning evaluation system for a video Q&A data set, including:

[0095] Video collection inference chain rule subsystem, which collects social interaction videos from the video interaction big data platform, removes the identification and annotation traces of the video data set, conducts visual Q&A and parsing annotation, and annotates the Q&A QA and description inference chains that conform to the rules therein;

[0096] Event and mental state causal relationship model subsystem, which constructs the causal relationship between events and mental states according to the Q&A QA and the described inference chains that conform to the rules, establishes an event and mental state causal relationship model, and annotates diverse types of social causality;

[0097] Inference chain expert verification subsystem, which verifies the data according to the diverse types of social causality through the first expert experience model, and conducts inference chain annotation by the second expert experience model;

[0098] Question Answering Inference Chain Testing and Scoring System, which comprehensively evaluates question - answering QA and inference chains of multi - modal large models through various evaluation metrics; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated question - answering QA to obtain question - answering QA social inference results close to real - world social inferences.

[0099] The principle and effect of the above - mentioned technical solution are as follows: The present invention provides a social inference evaluation system for video question - answering datasets, including: a video collection inference chain rule subsystem, which collects social interaction videos from a video interaction big - data platform, removes the identification and annotation traces in the video dataset, performs visual question - answering and parsing annotation, and annotates the question - answering QA and description inference chains that conform to the rules; an event - and - psychological - state causal - relationship model subsystem, which constructs an event - and - psychological - state causal relationship based on the question - answering QA and the described inference chains that conform to the rules, establishes an event - and - psychological - state causal - relationship model, and annotates various types of social causality; an inference - chain expert verification subsystem, which verifies the data through a first expert experience model according to various types of social causality and performs inference - chain annotation through a second expert experience model; a question - answering inference chain testing and scoring system, which comprehensively evaluates question - answering QA and inference chains of multi - modal large models through various evaluation metrics; provides accurate data for the social behavior evaluation of intelligent systems; verifies the generated question - answering QA to obtain question - answering QA social inference results close to real - world social inferences; Social inference is an important foundation for realizing artificial social intelligence. During social interactions, one can perceive and infer the psychological states of others and clarify their causes and results. Video is a record of the real world. Understanding social inferences in videos is a very important topic and can also provide offline data for social behavior evaluation; QA is short for Question Answering System; it can achieve important social inferences for artificial social intelligence, perceive and infer psychological states during social interactions and clarify their causes and results; understanding social inferences in videos based on the real - world records of videos has very important technical significance; it can provide offline data support for social behavior evaluation and significantly improve the accuracy of inferences.

[0100] In one embodiment, the video collection inference chain rule subsystem includes:

[0101] A social interaction video collection and screening subsystem, which collects social interaction videos from a video interaction big - data platform, removes low - quality social interaction videos with few interactions, few behaviors, and few events; screens out high - quality social interaction videos with many interactions, many behaviors, or many events, and forms a video dataset in a centralized manner;

[0102] A video data extraction subsystem, which selects videos with incorrect answers from the large model and incorrect question - answering QA, and removes the identification and annotation traces in the video dataset;

[0103] The Visual Question Answering Parsing and Annotation Subsystem conducts visual question answering, parsing, and annotation, trains the basic subject model on data standard recognition and annotation platform rules, and obtains the basic subject training model; the basic subject training model labels the compliant question-and-answer QAs and describes the inference chain from the video interaction big data platform according to the challenge form;

[0104] The inference chain includes: video event inference, video character mental state inference, and causal relationship inference between events and mental states;

[0105] Based on each inference chain, a large model generates a set of relevant question-and-answer QAs; the generation rules are as follows: for each node in the inference chain, generate a factual or mental state discrimination question;

[0106] For each sub-inference chain in the inference chain, generate a single-step causal inference question about the causal cause and causal result.

[0107] The principle and effect of the above technical solution are as follows: The video collection inference chain rule subsystem includes: the social interaction video collection and screening subsystem, which collects social interaction videos from the video interaction big data platform, removes low-quality social interaction videos with few interactions, few behaviors, and few events; screens out high-quality social interaction videos with many interactions, many behaviors, or many events, and forms a video dataset by aggregation; the video data extraction subsystem selects videos with incorrect answers from the large model and incorrect question-and-answer QAs, and removes the identification and annotation traces of the video dataset; the visual question answering parsing and annotation subsystem conducts visual question answering, parsing, and annotation, trains the basic subject model on data standard recognition and annotation platform rules, and obtains the basic subject training model; the basic subject training model labels the compliant question-and-answer QAs and describes the inference chain from the video interaction big data platform according to the challenge form; the inference chain includes: video event inference, video character mental state inference, and causal relationship inference between events and mental states; based on each inference chain, a large model generates a set of relevant question-and-answer QAs; the generation rules are as follows: for each node in the inference chain, generate a factual or mental state discrimination question; for each sub-inference chain in the inference chain, generate a single-step causal inference question about the causal cause and causal result; understanding the social inference in the video according to the real-world record of the video has very important technical significance.

[0108] In one embodiment, the event and mental state causal relationship model subsystem includes:

[0109] The causal relationship architecture subsystem establishes the causal relationship between events and mental states according to the compliant question-and-answer QAs and the described inference chain;

[0110] Causal reasoning events and mental state subsystem, according to the causal relationship between events and mental states, conduct social causal reasoning, understand the events occurring in the video and estimate the mental states of characters, and establish a causal relationship model between events and mental states;

[0111] Social causal diversification subsystem, according to the causal relationship model between events and mental states, label the types of social causal diversification;

[0112] Conduct social causal reasoning according to the causal relationship between events and mental states, understand the events occurring in the video and estimate the mental states of characters, and establish a causal relationship model between events and mental states, including: selecting sample reasoning chains; understanding the events occurring in the video; estimating the mental states of people; establishing causal relationships; dividing the problems into types of social causal diversification; the types of social causal diversification include: event understanding type, mental state estimation type, causal reason type, and causal result type; by evaluating the performance of the causal relationship model between events and mental states on various types of problems, analyze the lack of fine-grained capabilities of the model.

[0113] The principle and effect of the above technical solution are: the causal relationship model subsystem between events and mental states includes: a causal relationship architecture subsystem, which establishes the causal relationship between events and mental states according to the reasoning chains that conform to the rules of question answering QA and descriptions; causal reasoning events and mental state subsystem, according to the causal relationship between events and mental states, conduct social causal reasoning, understand the events occurring in the video and estimate the mental states of characters, and establish a causal relationship model between events and mental states; social causal diversification subsystem, according to the causal relationship model between events and mental states, label the types of social causal diversification; conduct social causal reasoning according to the causal relationship between events and mental states, understand the events occurring in the video and estimate the mental states of characters, and establish a causal relationship model between events and mental states, including: selecting sample reasoning chains; as Figures 1 - 3 shown; the sample reasoning chain includes: Robot2 reaches out his hand; to shake Robot1’s hand, but Robot1 ignores him; -> Robot2 feels embarrassed -> Robot2 touches his head with his hand; understand the events occurring in the video: Robot2 reaches out his hand;, but Robot1 ignores him instead of shaking hands with him;

[0114] Estimating a person's mental state: Robot2 laughed after Robot1 ignored him. However, this doesn't mean he was happy. Instead, he used the smile to cover up his embarrassment;

[0115] Establishing a causal relationship: The reason Robot2 felt embarrassed was that Robot2 reaches out his hand (Robot 2 extends his hand); to shake Robot1’s hand, but Robot1 ignores him (to shake hands with Robot 1, but Robot 1 ignores him); rather than Robot1 shaking hands with an athlete in white clothes;

[0116] Dividing the problem into socially causal diverse types: Socially causal diverse types include: event understanding type, mental state estimation type, causal reason type, and causal result type; By evaluating the performance of the event and mental state causal relationship model on various types of problems, it is analyzed that the model lacks the ability of detailed subdivision;

[0117] Event understanding type: Generate a factual question for each event node in the reasoning chain; To ensure the diversity of questions, design two ways to locate the event for questioning: a. By the time period when the event occurs, e.g.

[0118] What happened within 0s - 5s?; b. By part of the event, e.g. What did Robot1 do after Robot2 extended his hand; Establish the necessary condition spatio-temporal perception in event understanding;

[0119] Mental state estimation type: Establish social intelligence to evaluate mental states or infer mental states from behaviors; Divide mental states into affective mind and cognitive mind, namely emotion, belief, intent, and desire (emotional thinking and cognitive thinking, psychology, belief, intention, and desire); Test the mental state estimation ability of the event and mental state causal relationship model; Generate questions based on the mental state nodes in the reasoning chain, located by time points or time periods, e.g. What was Robot2's emotion at

[0120] 0:03S?

[0121] Causal reason type: Corresponding to reason-based questions; The form of reason-based questions includes: Question: Why [effect]? Answer: Because [reasons] (Question: Why [effect]? Answer: Because [reasons]);

[0122] Causal result type: Corresponding to result - type questions; The forms of result - type questions include: Question: What do [reasons] lead to? Answer: [effect] (Question: What does [cause] result in? Answer: [effect]);

[0123] For causal cause - type questions and causal result - type questions, in the inference chain, multiple causes pointing to the same result node are in an AND relationship, while the result nodes pointed to by the same multiple causes are in an OR relationship; The cause nodes pointing to the same result node in the inference chain are sufficient and necessary conditions, and the result cannot be deduced without any one of the cause nodes; This enriches the examination form of the model's causal reasoning ability and prevents the model from showing inconsistent performance on the two types of questions but not being detected.

[0124] In one embodiment, the inference - chain expert verification subsystem includes:

[0125] The expert - model verification subsystem verifies the data according to the social causal diversification type through the first expert - experience model and obtains the verified - passed data;

[0126] The data inference - chain annotation subsystem annotates the inference chain of the verified - passed data by the second expert - experience model.

[0127] The principle and effect of the above - mentioned technical solution are as follows: The inference - chain expert verification subsystem includes: The expert - model verification subsystem verifies the data according to the social causal diversification type through the first expert - experience model and obtains the verified - passed data; The data inference - chain annotation subsystem annotates the inference chain of the verified - passed data by the second expert - experience model; The expert - experience model is evaluated and verified through the nominal multiple - expert framework based on a large number of expert - system knowledge databases, and performs multiple - expert - experience intelligent learning until the set verification accuracy is reached.

[0128] In one embodiment, the question - answering inference - chain evaluation subsystem includes:

[0129] The question - answering comprehensive evaluation subsystem selects multiple evaluation indicators and conducts a comprehensive evaluation of question - answering QA and the inference chain for the multi - modal large - model through the multiple evaluation indicators; It provides accurate data for the social - behavior evaluation of intelligent systems;

[0130] The question - answering QA social - reasoning verification subsystem verifies the generated question - answering QA through the third expert - experience model; It verifies the question clarity, answer reasonableness, spelling error rate, diverse expressions, and unique meanings, and obtains the question - answering QA social - reasoning results close to real social reasoning.

[0131] The principles and effects of the above technical solutions are as follows: The question-and-answer reasoning chain measurement and evaluation system includes: a comprehensive question-and-answer evaluation subsystem that selects multiple evaluation indicators and conducts a comprehensive evaluation of question-and-answer QA and reasoning chains for multi-modal large models through multiple evaluation indicators; provides accurate data for the social behavior evaluation of intelligent systems; a question-and-answer QA social reasoning verification subsystem that verifies the generated question-and-answer QA through a third expert experience model; verifies the clarity of questions, the reasonableness of answers, the spelling error rate, diverse expressions, and unique meanings, and obtains question-and-answer QA social reasoning results close to real social reasoning; selecting multiple evaluation indicators and conducting a comprehensive evaluation of question-and-answer QA and reasoning chains for multi-modal large models through multiple evaluation indicators includes: comprehensively evaluating question-and-answer QA and reasoning chains for multi-modal large models through indicators such as the accuracy rate of question-and-answer QA for psychological state recognition, the accuracy rate of question-and-answer QA for event recognition, the accuracy rate of question-and-answer QA for reasons, and the accuracy rate of question-and-answer QA for causality; the third expert experience model verifies the generated question-and-answer QA; the expert experience model is formed by a general expert model through training with QA data to form question-and-answer expert experience for QA data; the question for QA data training is Question: Why does Robot 1 want to sleep in Robot 2's room? The training question-and-answer evaluation options are: a. Because Robot 1 wants to discuss a scientific theory with Robot 2; b. Because Robot 1 is concerned about Robot 2's health and wants to monitor her overnight; c. Because Robot 1 needs a quiet place to work on his research and Robot 2's room is the quietest; d. Because Robot 1 is feeling afraid and seeks comfort by being in Robot 2's room; e.Because Robot 1 wants to surprise Robot 3 by being in Robot 2's room when he returns; Expert Experience: Experts Explanation: Although Robot 1 said he was doing it for Robot 2, he clearly didn't calm down from his nightmare; His cautious tone indicated that he only wanted to sleep in Robot 2's house because he was afraid (Question: Why does Robot 1 want to sleep in Robot 2's room? Training Q&A Assessment: Options: a. Because Robot 1 wants to discuss a scientific theory with Robot 2; b. Because Robot 1 cares about Robot 2's health and wants to monitor her all night; c. Because Robot 1 needs a quiet place to conduct research, and Robot 2's room is the quietest; d. Because Robot 1 is afraid and seeks comfort by staying in Robot 2's room; e. Because Robot 1 wants to surprise Robot 3 when he returns by being in Robot 2's room; Expert Experience Model and Explanation Verification: Although Robot 1 said he was doing it for Robot 2, he clearly didn't calm down from his nightmare; His cautious tone indicated that he only wanted to sleep in Robot 2's house because he was afraid); Verify the clarity of the question, the reasonableness of the answer, the spelling error rate, the diversity of expressions and the uniqueness of meaning, and obtain the Q&A QA social reasoning result close to real social reasoning; Significantly improve the accuracy of the Q&A QA close to real social reasoning.

[0132] Although the embodiments of the present invention have been disclosed above, it is not limited to the applications listed in the specification and the embodiments. It can be fully applied to various fields suitable for the present invention. For those familiar with the field, additional modifications can be easily made. Therefore, without departing from the general concept defined by the claims and the equivalent scope, the present invention is not limited to the specific details and the illustrated and described examples here.

Claims

1. A social reasoning evaluation method for a video question-answering dataset, characterized in that: include: S100 collects social interaction videos from the video interaction big data platform, removes the traces of previous identification and annotation in the video dataset, performs visual question answering and analysis annotation, and annotates the rule-compliant question answering (QA) and description reasoning chains; S200, based on the reasoning chain of rule-compliant QA and description, construct the causal relationship between events and psychological states, establish a causal relationship model between events and psychological states, and annotate the diverse types of social causality; S300, based on the social causal diversity type, the data is verified by the first expert experience model, and the reasoning chain is annotated by the second expert experience model; S400, through a multi-modal large model with multiple evaluation indicators, conducts comprehensive evaluation of QA and reasoning chain; providing accurate data for the social behavior evaluation of intelligent systems; Verify the generated QA to obtain QA social reasoning results that are close to real social reasoning; S200 includes: S201, establishing a causal relationship between events and mental states based on the reasoning chain of the rule-compliant question and answer QA and the description; S202, based on the causal relationship between events and psychological states, social causal reasoning is performed to understand the events occurring in the video and estimate the psychological states of the characters, and a causal relationship model between events and psychological states is established; S203, labeling social causal diversity types based on the causal relationship model between events and psychological states; According to the causal relationship between events and psychological states, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological state of the characters. The establishment of a causal relationship model between events and psychological states includes: selecting sample reasoning chains; understanding the events that occurred in the video; estimating the psychological state of the characters; establishing causal relationships; dividing problems into diversified social causal types; diversified social causal types include: event understanding type, psychological state estimation type, causal cause type and causal result type; by evaluating the performance of the causal relationship model between events and psychological states on various types of problems, the lack of model segmentation capabilities is analyzed; Event understanding type: generate a factual question for each event node in the reasoning chain; establish spatiotemporal perception as a necessary condition in event understanding; psychological state estimation type: establish social intelligence to evaluate psychological state or infer psychological state from behavior; test the psychological state estimation ability of the causal relationship model between events and psychological states; generate questions based on psychological state nodes in the reasoning chain, and locate them by time points or time periods; causal cause type: corresponds to cause-type questions; causal result type: corresponds to result-type questions; causal cause type questions and causal result type questions, in the reasoning chain, multiple causes pointing to the same result node are in an and relationship, while the same multiple causes pointing to the result nodes are in an or relationship; the cause nodes pointing to the same result node in the reasoning chain are sufficient and necessary conditions, and the result cannot be deduced without any cause node.

2. According to claim 1, a method for evaluating social reasoning of a video question-answering dataset is characterized in that: S100 includes: S101, collect social interaction videos from the video interaction big data platform, remove low-quality social interaction videos with few interactions, few behaviors, and few events; screen out high-quality social interaction videos with many interactions, many behaviors, or many events, and aggregate them to form a video data set; S102, selecting videos with incorrect answers and incorrect questions and answers (QA) from the large model, and removing the identification and annotation traces from the video dataset; S103, perform visual question answering and analysis annotation, perform data standard recognition and annotation platform rule training on the basic subject model, and obtain the basic subject training model; the basic subject training model annotates the rule-compliant question answering QA and description reasoning chain from the video interactive big data platform according to the challenge form; The reasoning chain includes: video event reasoning, video character psychological state reasoning, and causal relationship reasoning between events and psychological states; A set of related questions and answers (QA) is generated based on each reasoning chain through the big model. The generated rules are as follows: for each node in the reasoning chain, a factual or mental state identification question is generated; For each sub-chain of reasoning in the chain of reasoning, a single-step causal reasoning problem of causal cause and causal effect is generated.

3. The social reasoning evaluation method for a video question-answering dataset according to claim 1, characterized in that: S300 includes: S301, verifying the data through the first expert experience model according to the social causal diversity type, and obtaining the verified data; S302, verify that the data is annotated by the second expert experience model through reasoning chain.

4. According to claim 1, a method for evaluating social reasoning of a video question-answering dataset is characterized in that: S400 includes: S401, select multiple evaluation indicators, and conduct comprehensive evaluation of QA and reasoning chain through multiple evaluation indicators and multi-modal large models; provide accurate data for social behavior evaluation of intelligent systems; S402, verify the generated question and answer QA through the third expert experience model; verify the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtain the question and answer QA social reasoning results that are close to the real social reasoning.

5. A social reasoning evaluation system for video question-answering datasets, characterized in that: include: The video collection reasoning chain rule subsystem collects social interaction videos from the video interaction big data platform, removes the traces of previous identification and annotation in the video data set, performs visual question answering and analysis annotation, and annotates the question answering QA and description reasoning chains that meet the rules; The event and psychological state causal relationship model subsystem constructs the event and psychological state causal relationship based on the reasoning chain of rule-compliant QA and description, establishes the event and psychological state causal relationship model, and labels the diverse types of social causality; The reasoning chain expert verification subsystem verifies the data through the first expert experience model according to the social causal diversity type, and the reasoning chain is annotated by the second expert experience model; The question-answering and reasoning chain evaluation system uses a multi-modal large model with multiple evaluation indicators to conduct comprehensive evaluation of question-answering QA and reasoning chains; providing accurate data for the social behavior evaluation of intelligent systems; Verify the generated QA to obtain QA social reasoning results that are close to real social reasoning; The event and mental state causal relationship model subsystem includes: The causal relationship architecture subsystem establishes the causal relationship between events and mental states based on the reasoning chain that conforms to the rule-based question and answer QA and description; The causal reasoning event and psychological state subsystem performs social causal reasoning based on the causal relationship between events and psychological states, understands the events that occur in the video, estimates the psychological state of the characters, and establishes a causal relationship model between events and psychological states; The social causal diversity subsystem labels the social causal diversity types according to the causal relationship model between events and psychological states; According to the causal relationship between events and psychological states, social causal reasoning is performed to understand the events that occurred in the video and estimate the psychological state of the characters. The establishment of a causal relationship model between events and psychological states includes: selecting sample reasoning chains; understanding the events that occurred in the video; estimating the psychological state of the characters; establishing causal relationships; dividing problems into diversified social causal types; diversified social causal types include: event understanding type, psychological state estimation type, causal cause type and causal result type; by evaluating the performance of the causal relationship model between events and psychological states on various types of problems, the lack of model segmentation capabilities is analyzed; Event understanding type: generate a factual question for each event node in the reasoning chain; establish spatiotemporal perception as a necessary condition in event understanding; psychological state estimation type: establish social intelligence to evaluate psychological state or infer psychological state from behavior; test the psychological state estimation ability of the causal relationship model between events and psychological states; generate questions based on psychological state nodes in the reasoning chain, and locate them by time points or time periods; causal cause type: corresponds to cause-type questions; causal result type: corresponds to result-type questions; causal cause type questions and causal result type questions, in the reasoning chain, multiple causes pointing to the same result node are in an and relationship, while the same multiple causes pointing to the result nodes are in an or relationship; the cause nodes pointing to the same result node in the reasoning chain are sufficient and necessary conditions, and the result cannot be deduced without any cause node.

6. A social reasoning evaluation system for video question-answering datasets according to claim 5, characterized in that: Video collection reasoning chain rule subsystem, including: The social interaction video collection and screening subsystem collects social interaction videos from the video interaction big data platform, removes low-quality social interaction videos with few interactions, few behaviors, and few events; screens out high-quality social interaction videos with many interactions, many behaviors, or many events, and aggregates them into a video data set; The video data extraction subsystem selects videos with incorrect answers and incorrect QA answers from the large model, and removes the traces of previous identification and annotation in the video data set; The visual question-answering, parsing and annotation subsystem performs visual question-answering and parsing annotation, performs data standard recognition and annotation platform rule training on the basic subject model, and obtains the basic subject training model; the basic subject training model annotates the rule-compliant QA and description reasoning chain from the video interactive big data platform according to the challenge form; The reasoning chain includes: video event reasoning, video character psychological state reasoning, and causal relationship reasoning between events and psychological states; A set of related questions and answers (QA) is generated based on each reasoning chain through the big model. The generated rules are as follows: for each node in the reasoning chain, a factual or mental state identification question is generated; For each sub-chain of reasoning in the chain of reasoning, a single-step causal reasoning problem of causal cause and causal effect is generated.

7. The social reasoning evaluation system for video question-answering dataset according to claim 5, characterized in that: Reasoning chain expert verification subsystem, including: The expert model verification subsystem verifies the data through the first expert experience model according to the social causal diversity type and obtains the verified data; The data reasoning chain annotation subsystem verifies that the data is annotated by the second expert experience model through the reasoning chain.

8. The social reasoning evaluation system for video question-answering dataset according to claim 5, characterized in that: The question-answering reasoning chain test scoring system includes: The question-answering comprehensive evaluation subsystem selects multiple evaluation indicators and uses a multi-modal large model with multiple evaluation indicators to conduct comprehensive evaluation of question-answering QA and reasoning chains; providing accurate data for the social behavior evaluation of intelligent systems; The question-and-answer (QA) social reasoning verification subsystem verifies the generated QA through the third expert experience model; verifies the clarity of the question, the rationality of the answer, the spelling error rate, the diversity of expressions and the unique meaning, and obtains the QA social reasoning results that are close to the real social reasoning.

Images