Dynamic hypothesis verification method and device for complex event understanding
Through the logical assumption verification and reasoning process of the dynamic multi-perspective thinking agent framework, evidence search is searched in combination with title, content and time correlation, and the problems of low search efficiency and high computing resource consumption in complex event understanding are solved, and efficient and comprehensive Answer generation.
Patent Information
- Application Number
- CN202411150973.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing technology has problems such as low search efficiency, low utilization of retrieval evidence and high computing resource consumption in the understanding of complex events, which is difficult to effectively support human efficient research and analysis processes.
A dynamic multi-perspective thinking agency framework is proposed, which dynamically adjusts hypotheses through logical assumption verification and inference process, combines title, content and time correlation to search and reason for evidence to generate comprehensive answers.
It improves the response efficiency of complex event understanding and the quality of final answers, significantly improves the efficiency of collecting relevant evidence and the comprehensiveness of answers, and reduces the consumption of computing resources.
Smart Images

Figure CN119088903A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a dynamic hypothesis verification method and device for complex event understanding. Background Art
[0002] In the era of information explosion, people are overwhelmed by a large amount of information about real-world events. Among them, complex emerging events often involve a long period of time, complex causes and major impacts, resulting in a large amount of complex and lengthy news on the Internet. The richness of this information makes it difficult for users to digest and understand it efficiently. Therefore, how to quickly understand the core ideas of these complex events has become a key issue and has attracted much attention.
[0003] To facilitate understanding of the large amount of information associated with complex events, early efforts have focused on tracking events over time. This includes creating summaries that provide a progression of events, as well as visualizing news corpora in a more structured manner. These approaches help maintain a chronological record of events, making it easier for users to follow the latest developments. However, simply understanding the progression of events is not enough. We hope that AI can provide deeper, more proactive, and more flexible assistance. This includes analyzing the causes of events and predicting future outcomes, greatly enhancing the human decision-making process.
[0004] Solving these problems requires complex information requirements and sophisticated reasoning capabilities. Recently, the rapid advancement of chat-based large language models (LLMs) has achieved remarkable achievements in complex task solving. By combining LLMs with external tools such as retrieval systems and turning LLMs into autonomous agents, they can also actively retrieve the latest knowledge to facilitate the reasoning process, which shows great potential in effective complex event understanding. Nevertheless, due to the complexity of the task, preliminary analysis shows that some challenges arise when directly adapting autonomous agents to complex event understanding tasks, including low retrieval efficiency, low utilization of retrieved evidence, and high consumption of computational resources. Therefore, it is crucial to study how to align these methods with the efficient human research and analysis process. Summary of the invention
[0005] The present invention aims to address the deficiencies of the prior art and proposes a dynamic hypothesis verification method and device for complex event understanding. By proposing a novel dynamic multi-perspective thinking agent framework, the language model can actively propose and dynamically adjust hypotheses, verify the hypotheses through retrieval and reasoning processes, and finally synthesize a comprehensive solution.
[0006] To achieve the above object, the present invention provides the following solution: a dynamic hypothesis verification method for complex event understanding, comprising the following steps:
[0007] S1, collect and annotate datasets;
[0008] S2. Construct a dynamic multi-perspective thinking agent framework, use the data set to train the dynamic multi-perspective thinking agent framework, and generate comprehensive answers to users' queries about complex events based on the trained dynamic multi-perspective thinking agent framework.
[0009] Further preferably, the method for obtaining the data set includes:
[0010] Complex event instances are collected and manually annotated; each annotated instance contains a triple (q, K, D), where q represents the user's query; K represents the list of key points contained in the answer; and D represents the set of news reports related to the complex event in the query.
[0011] Further preferably, the method for the dynamic multi-perspective thinking agent framework to obtain the comprehensive answer includes:
[0012] Step 1: Propose a set of logical hypotheses based on the user's query on complex events;
[0013] Step 2: Select a hypothesis from the set of logical hypotheses, retrieve evidence and perform reasoning based on the hypothesis to obtain a reasoning result; and delete the hypothesis from the set of logical hypotheses;
[0014] Step 3: Based on the inference result, modify the current logical assumption set, the modification includes deleting the logical assumption and adding a new logical assumption; repeat step 2 for the modified logical assumption set until the logical assumption set is empty; form the final argument set with all the assumptions whose inference results are true and the retrieved evidence, and proceed to step 4;
[0015] Step 4: Obtain the comprehensive answer based on the language model and the recorded final argument set.
[0016] Further preferably, for user queries on complex events, the logical hypothesis is proposed; a query generator is used to analyze the logical hypothesis to generate a number of independent queries; for each query, a retriever is used to search an external news database to provide the evidence; a reasoner is used to perform synthetic reasoning on the evidence to obtain the reasoning result.
[0017] Further preferably, the retriever obtains the evidence based on title relevance, content relevance and time relevance during the retrieval process;
[0018] The method for calculating the title relevance includes:
[0019] The language model is used to generate an embedding vector for each title text, and the cosine similarity between the title embedding vector and the query embedding vector is calculated as the title relevance;
[0020] The time correlation is an exponential decay function of the time difference between the news timestamp and the time sentinel;
[0021] The three correlation scores are normalized using the minimum-maximum scaling method, and the final retrieval score of the evidence is:
[0022] score=α1·title+α2·content+α3·\temporal,
[0023] Where α1, α2, and α3 represent weight coefficients respectively; title, content, and temporal represent the relevance scores between the input query and the corresponding article in terms of title, content, and time respectively.
[0024] The present invention also provides a dynamic hypothesis verification device for complex event understanding, comprising: a data set construction module and a dynamic multi-perspective thinking agent framework module;
[0025] The data set construction module is used to collect and annotate data sets;
[0026] The dynamic multi-perspective thinking agent framework module is used to construct a dynamic multi-perspective thinking agent framework, use the data set to train the dynamic multi-perspective thinking agent framework, and generate comprehensive answers to users' queries about complex events based on the trained dynamic multi-perspective thinking agent framework.
[0027] Further preferably, the method for the data set construction module to obtain the data set includes:
[0028] Complex event instances are collected and manually annotated; each annotated instance contains a triple (q, K, D), where q represents the user's query; K represents the list of key points contained in the answer; and D represents the set of news reports related to the complex event in the query.
[0029] Further preferably, the dynamic multi-perspective thinking agent framework module includes: a query input unit, a logic proof search and reasoning unit, a hypothesis modification unit and a solution generation unit;
[0030] The query input unit is used to propose a set of logical hypotheses based on the user's query on the complex event;
[0031] The query input unit is used to propose a set of logical hypotheses based on the user's query on the complex event;
[0032] The logic proof search and reasoning unit is used to analyze each hypothesis, search for evidence based on the hypothesis, and reason about the hypothesis based on the evidence to obtain a reasoning result; and delete the hypothesis from the logic hypothesis set;
[0033] The hypothesis modification unit is used to modify the logical hypothesis set based on the reasoning result, and the modification includes deleting the logical hypothesis and adding a new logical hypothesis; and inputting the new logical hypothesis set into the logical proof search and reasoning unit until the logical hypothesis set is empty; all the hypotheses with true reasoning results and the retrieved evidence form a final argument set, and input the final argument set into the solution generation unit;
[0034] The solution generation unit is used to obtain the comprehensive answer based on the language model and the final argument set.
[0035] Further preferably, the logic proof search and reasoning unit comprises: a query generator, a retriever and a reasoner;
[0036] The query generator is used to analyze the logical assumption and generate a plurality of independent queries;
[0037] The retriever is used to retrieve an external news database to provide the evidence based on each query;
[0038] The reasoner is used to perform synthetic reasoning on the evidence to obtain the reasoning result.
[0039] Further preferably, the retriever obtains the evidence based on title relevance, content relevance and time relevance during the retrieval process;
[0040] The method for calculating the title relevance includes:
[0041] The language model is used to generate an embedding vector for each title text, and the cosine similarity between the title embedding vector and the query embedding vector is calculated as the title relevance;
[0042] The time correlation is an exponential decay function of the time difference between the news timestamp and the time sentinel;
[0043] The three correlation scores are normalized using the minimum-maximum scaling method, and the final retrieval score of the evidence is:
[0044] score=α1·title+α2·content+α3·\temporal,
[0045] Where α1, α2, and α3 represent weight coefficients respectively; title, content, and temporal represent the relevance scores between the input query and the corresponding article in terms of title, content, and time respectively.
[0046] Compared with the prior art, the present invention has the following beneficial effects:
[0047] This paper proposes a new agent framework - dynamic multi-perspective thinking agent for complex event understanding. In the dynamic multi-perspective thinking agent framework, multi-perspective thinking is included, which enables LLM to make assumptions about possible event contexts. Based on these assumptions, more accurate retrieval and verification are performed through logical proof search and reasoning modules. At the same time, the proposed hypothesis schemes can be dynamically adjusted. This allows a favorable balance between response efficiency and comprehensiveness.
[0048] There is currently no public data specifically for understanding complex events, so this paper curates a complex event understanding dataset containing 100 pairs of question-keypoint lists. The dataset contains carefully crafted economic news questions from 2021 to 2023 and is manually annotated. Each question is annotated with key points that should be covered in the answer for automatic model evaluation to assess the key point coverage of the answer during generation and retrieval. Experimental results show that compared with the previously proposed RAG framework, our method has made significant progress in both the efficiency of collecting relevant evidence and the quality of the final response. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0050] Figure 1 This is a schematic diagram of the dynamic multi-perspective thinking agent framework process of an embodiment of the present invention;
[0051] Figure 2 Schematic diagram of a curve showing a change in the recall rate of key points according to an embodiment of the present invention;
[0052] Figure 3 The figure is a schematic diagram of a recall rate variation curve under different token consumptions in an embodiment of the present invention. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0055] Embodiment 1:
[0056] This embodiment provides a dynamic hypothesis verification method for complex event understanding, comprising the following steps:
[0057] S1. Collect and annotate datasets.
[0058] Complex event instances are collected and manually annotated; each annotated instance contains a triple (q, K, D), where q represents the user's query; K represents the list of key points contained in the answer; and D represents the set of news reports related to the complex event in the query.
[0059] S2. Construct a dynamic multi-perspective thinking agent framework, use the data set to train the dynamic multi-perspective thinking agent framework, and generate comprehensive answers to users' queries about complex events based on the trained dynamic multi-perspective thinking agent framework.
[0060] The methods of the dynamic multi-perspective thinking agent framework to obtain comprehensive answers include:
[0061] Step 1: Based on the user's query about complex events, a set of logical hypotheses is proposed.
[0062] Step 2: Select a hypothesis from the set of logical hypotheses, retrieve evidence and make inferences based on the hypothesis to obtain the inference result; and delete the hypothesis from the set of logical hypotheses.
[0063] Step 3: Based on the inference results, modify the current set of logical assumptions, including deleting logical assumptions and adding new logical assumptions; repeat step 2 on the modified set of logical assumptions until the set of logical assumptions is empty; form the final argument set with all the assumptions whose inference results are true and the retrieved evidence, and proceed to step 4.
[0064] Step 4: Get a comprehensive answer based on the language model and the final set of recorded arguments.
[0065] Specifically, for users' queries on complex events, logical hypotheses are proposed; a query generator is used to analyze the logical hypotheses and generate several independent queries; for each query, a retriever is used to search external news databases to provide evidence; a reasoner is used to perform synthetic reasoning on the evidence to obtain the reasoning result.
[0066] During the retrieval process, the retriever obtains evidence based on title relevance, content relevance, and time relevance; the calculation method of title relevance includes: using the language model to generate the embedding vector of each title text, and calculating the cosine similarity between the title embedding vector and the query embedding vector as the title relevance; content relevance gives higher scores to paragraphs related to the current query content, and its specific calculation method is the same as title relevance, which is also achieved through the cosine similarity between the content embedding vector and the query embedding vector. Time relevance is an exponential decay function of the time difference between the news timestamp and the time sentinel; the minimum-maximum scaling method is used to normalize the three relevance scores, and the final evidence retrieval score is:
[0067] score=α1·title+α2·content+α3·\temporal,
[0068] Where α1, α2, and α3 represent weight coefficients respectively; title, content, and temporal represent the relevance scores between the input query and the corresponding article in terms of title, content, and time respectively.
[0069] Embodiment 2:
[0070] This embodiment will explain the method proposed by the present invention in detail with reference to specific examples.
[0071] S1. Collect and annotate datasets.
[0072] In order to evaluate the proposed framework, complex event instances are collected and manually annotated. In this embodiment, a set of 100 instances are manually annotated for the complex event understanding task. Each annotated instance contains a triple (q, K, D), where q represents the user's query; K represents the list of key points contained in the answer; and D represents the set of news reports related to the complex event in the query. The annotated answer contains a list of key points, rather than requiring manual annotators to write long answers. In the process of generating complex event queries, the following principles are mainly considered: (1) Ethical considerations: Avoid ethical issues in the selection of complex events. Different groups hold different positions and attitudes towards controversial events. Therefore, the constructed dataset avoids involving such controversial events. (2) Timeliness: The dataset collects events between May 2022 and May 2023 to ensure that these events are not processed in the training process of large-scale language models. (3) Diversity of question types: The diversity of question types can reflect the extensiveness of queries in real applications. (4) Comprehensiveness of key points: Annotators should write the key points of the answers as clearly, comprehensively and systematically as possible.
[0073] To achieve this, this embodiment considers four types of complex event understanding queries: the evolution of events, the reactions of all parties, the causes of events, and predictions of future developments. After obtaining the query, news data related to complex events are crawled and queried from a series of well-known news portals to ensure the accuracy and completeness of relevant reports. Among the above principles, principle four is particularly challenging. Analyzing causes and predicting the potential consequences of complex events requires individuals to have a rich professional knowledge background. In order to ensure the accuracy and comprehensiveness of the answers, the annotator does not need to conduct his own analysis, but selects valuable events and looks for analysis from authoritative people, such as research papers, company technical reports, and videos of well-known influencers on platforms such as YouTube and Bilibili. After summarizing and organizing information from various sources, the annotator creates relevant key points to answer the questions. For each question, several NLP researchers with graduate degrees are used to participate in the annotation process. One of them plays the role of an arbitrator, integrating or revising the results of other annotators to ensure that the annotations are as comprehensive and accurate as possible.
[0074] S2. Construct a dynamic multi-perspective thinking agent framework, use the data set to train the dynamic multi-perspective thinking agent framework, and generate comprehensive answers to users' queries about complex events based on the trained dynamic multi-perspective thinking agent framework.
[0075] Specifically, Figure 1 As shown, logical hypotheses are proposed for user queries on complex events; wherein, considering the importance of multiple perspectives in solving complex problems, after receiving the user's query, the query event is summarized and a new question specifically focusing on the summarized event is generated; then, multiple detailed logical hypotheses are brainstormed about what may be included in the answer; these hypotheses are used to initialize the perspective set A.
[0076] For example, if a user is interested in understanding the reasons behind the event "Silicon Valley Bank collapsed", based on the user query input, the event "Silicon Valley Bank collapsed" is first summarized, and a new question specifically focusing on summarizing the event is generated, namely "the reasons for the bank's collapse". Subsequently, N perspectives are proposed and elaborated based on this question. One of the potential perspectives is the "financial perspective". From this perspective, it is necessary to explain that the bank's collapse may be related to credit risk, market volatility and liquidity issues, emphasizing factors related to financial operations and balance sheets.
[0077] The reason for introducing event summarization is that the language model LLM relies on general event evolution knowledge rather than forming comprehensive hypotheses for specific events. This prevents LLM from refusing to answer due to its own lack of specific background knowledge about the query event. Proactively formulating hypotheses about the query helps to solve the problem efficiently and helps to deal with one aspect at a time, thus achieving organized and effective evidence collection.
[0078] After that, each time a hypothesis a is extracted from the perspective set A, logical proofs are searched and reasoning is performed to verify a. Specifically, a query generator is used to analyze the logical hypothesis and generate several independent queries; for each query, a retriever is used to retrieve external news databases to provide evidence; a reasoner is used to perform synthetic reasoning on the evidence to generate additional factual events mentioned in the retrieved news, and at the same time obtain verification results for the initial hypothesis. Factual events can be used as evidence to support or reject their hypotheses, and can be used for subsequent hypothesis modification and refinement of the hypothesis set.
[0079] In the retrieval process, the retriever obtains evidence based on title relevance, content relevance, and time relevance; specifically, it takes the query as input and returns a subset of segmented paragraphs to pass to the language model. Depending on the importance of the collected evidence, a variety of retrieval functions can be implemented.
[0080] Title relevance assigns higher scores to paragraphs from news headlines that are relevant to the current query. The specific calculation method includes: using a language model to generate an embedding vector for each title text, and calculating the cosine similarity between the title embedding vector and the query embedding vector as title relevance. Content relevance assigns higher scores to paragraphs that are relevant to the current query content. Its specific calculation method is the same as title relevance, and is also achieved through the cosine similarity between the content embedding vector and the query embedding vector. Time relevance assigns higher scores to paragraphs whose timestamps are closer to the query. This is solved by setting a time sentinel. In the absence of a time context in the query, the sentinel defaults to the current moment; otherwise, it is aligned with the time context explicitly stated in the query. Time relevance is regarded as an exponentially decaying function of the time difference between the news timestamp and the time sentinel; the decay factor is set to 0.995.
[0081] In order to calculate the final retrieval score, the three relevance scores are normalized to the range of [0,1] using the minimum-maximum scaling method. The final evidence retrieval score is a weighted combination of the three elements:
[0082] score=α1·title+α2·content+α3·\temporal,
[0083] Where α1, α2, and α3 represent weight coefficients respectively; title, content, and temporal represent the relevance scores between the input query and the corresponding article in terms of title, content, and time respectively.
[0084] After the above process generates the factual events mentioned in the retrieved news, these factual events are reviewed and the initially obtained perspective set A is refined, including deleting or adding new hypotheses to the perspective set A. Dynamic modification of the perspective set A as the retrieval process proceeds is very important for reducing redundant calls of the LLM. This is because additional factual information to verify the current sampling hypothesis can often be used to verify or exclude other hypotheses in the set. For example, consider the query "the current progress of Microsoft's acquisition of Blizzard". The perspective proposed by the language model (LLM) may include various stages of this event, including the launch of the acquisition, legal and regulatory approvals in different countries, Microsoft's response, and results. If the retrieval results show that the EU has not approved the acquisition, there is no need to further verify the final acquisition results. Afterwards, a comprehensive solution is generated by integrating the verified hypotheses. This well-organized and screened content has a higher information density, making it more effective to generate comprehensive answers.
[0085] This embodiment uses the key evidence recall rate (KER) to evaluate the comprehensiveness of the generated answer. In order to calculate whether the generated answer contains a given key point k, the generated answer is divided into multiple paragraphs (p1, p2, ... p N ), for every pair (p i ,k), prompting the language model to recognize p i Whether to include k, if any p i If k is included, then k is retrieved. In addition to the key evidence recall rate (KER), the token consumption (TC) of the prompt LLM, the retrieval efficiency rate (RE) and the retrieval evidence utilization rate (EU) are also used to evaluate the efficiency of the entire retrieval process.
[0086] Embodiment three:
[0087] This embodiment selects several representative algorithms as baseline algorithms for comparative experiments. Table 1 shows the overall performance of the dynamic multi-perspective thinking agent framework (DAMPT) proposed in this application and other baseline algorithms.
[0088]
[0089] As shown in Table 1, DAMPT beats all baseline methods in the retrieval KER (key evidence recall rate) and answer KER indicators, which indicates that the DAMPT framework can obtain more comprehensive perspectives on the problem and generate better answers during the retrieval process. This means that the hypothesis verification reasoning process helps to improve the deep understanding of complex events. Compared with the baseline methods, the proposed method achieves higher results in the retrieval evidence utilization rate (EUR). At the same time, it is noted that not all valid evidence retrieved during the search process can be effectively utilized in generating the final answer. Therefore, the effectiveness of the final answer depends on two factors: first, the ability of the retrieval system to retrieve relevant evidence; second, the efficiency of synthesizing relevant evidence to generate the final answer, that is, the utilization efficiency of the retrieved evidence. The improvement in EUR can be attributed to the effective organization of relevant evidence through hypotheses, where only evidence supporting these hypotheses is used as input for context learning, thereby effectively reducing the impact of noise on the model output results. In terms of computational resource consumption, ReAct's resource usage is significantly lower than that of other methods, although its results are the worst. Its lower computational resource consumption is attributed to its limited involvement in multi-step retrieval and reasoning, which makes its processing of external information more efficient, but limits the quality of generated answers. Compared with the Plan-then-Execute method, which can also perform multi-step retrieval and reasoning, the DAMPT method consumes fewer tokens while performing more efficient retrieval. This is because the Plan-then-Execute method needs to integrate the results of all previous steps into its reasoning context after completing the subtask, resulting in additional token consumption. The DAMPT method focuses on a diverse problem-solving perspective and is able to complete different subtasks independently, thereby significantly reducing the length of the reasoning context.
[0090] At the same time, compared with the baseline method that does not actively propose hypotheses, the recall rate of key evidence in each iteration is compared, that is, how many golden key evidences can be covered by the retrieved documents after each retrieval step. Figure 2As shown, it can be seen that ReAct tends to stop retrieving after a few steps of retrieval. As the number of retrieval steps increases, the method of the present invention can quickly retrieve more relevant evidence present in the answer. The ReACT method performs poorly in generating a comprehensive retrieval plan; it often starts to generate the final answer after a few steps of retrieval. This limitation stems from the integration of the results of all previous steps into the reasoning context, which weakens the ability of the large model to infer subsequent retrieval behavior due to the complexity of the relevant information. The Plan-then-Execute method designs a complete plan after understanding the problem and solves it step by step. However, we observe that this method fails to effectively utilize the large language model's understanding of the occurrence of macroscopic events. It focuses more on organizing the problem-solving process and lacks the specificity of the decomposition of sub-processes. The method of the present invention adopts an active hypothesis and verification mode, which effectively stimulates the large model's understanding of the background knowledge of the event. Based on this understanding, it generates more precise retrieval targets, thereby improving retrieval efficiency. Figure 3 The relationship between the number of tokens consumed by the static framework and the recall of the final predicted answer is shown. DAMPT achieves the best performance among all baseline methods.
[0091] Embodiment 4:
[0092] The present embodiment provides a dynamic hypothesis verification device for complex event understanding, including: a data set construction module and a dynamic multi-perspective thinking agent framework module; the data set construction module is used to collect and annotate data sets; the dynamic multi-perspective thinking agent framework module is used to construct a dynamic multi-perspective thinking agent framework, use the data set to train the dynamic multi-perspective thinking agent framework, and generate comprehensive answers to users' queries about complex events based on the trained dynamic multi-perspective thinking agent framework.
[0093] A further implementation is that the method for obtaining the data set by the data set construction module includes: collecting complex event instances and manually annotating the complex event instances; each annotated instance contains a triple (q, K, D), where q represents the user's query; K represents a list of key points contained in the answer; and D represents a set of news reports related to the complex event in the query.
[0094] Further implementation is that the dynamic multi-perspective thinking agent framework module includes: a query input unit, a logic proof search and reasoning unit, a hypothesis modification unit and a solution generation unit; the query input unit is used to propose a set of logical assumptions based on the user's query on complex events; the logic proof search and reasoning unit is used to analyze each assumption, search for evidence based on the assumption, and reason about the assumption based on the evidence to obtain a reasoning result; and delete the assumption from the logical assumption set; the hypothesis modification unit is used to modify the logical assumption set based on the reasoning result, and the modification includes deleting the logical assumption and adding a new logical assumption; and input the new logical assumption set into the logic proof search and reasoning unit until the logical assumption set is empty; all assumptions with true reasoning results and the retrieved evidence form a final argument set, and the final argument set is input into the solution generation unit; the solution generation unit is used to obtain a comprehensive answer based on the language model and the final argument set.
[0095] Among them, the logical proof search and reasoning unit includes: a query generator, a retriever and a reasoner; the query generator is used to analyze logical assumptions and generate several independent queries; the retriever is used to retrieve external news databases to provide evidence based on each query; the reasoner is used to perform synthetic reasoning on the evidence to obtain the reasoning result.
[0096] During the retrieval process, the retriever obtains evidence based on title relevance, content relevance, and time relevance. The calculation method of title relevance includes: using the language model to generate the embedding vector of each title text, and calculating the cosine similarity between the title embedding vector and the query embedding vector as the title relevance; the time relevance is an exponential decay function of the time difference between the news timestamp and the time sentinel; using the minimum-maximum scaling method, the three relevance scores are normalized, and the final evidence retrieval score is:
[0097] score=α1·title+α2·content+α3·\temporal,
[0098] Where α1, α2, and α3 represent weight coefficients respectively; title, content, and temporal represent the relevance scores between the input query and the corresponding article in terms of title, content, and time respectively.
[0099] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.
Claims
1. A dynamic hypothesis verification method for complex event understanding, characterized in that: The following steps are involved: S1, collect and annotate datasets; S2. Construct a dynamic multi-perspective thinking agent framework, use the data set to train the dynamic multi-perspective thinking agent framework, and generate comprehensive answers to users' queries about complex events based on the trained dynamic multi-perspective thinking agent framework.
2. A dynamic hypothesis verification method for complex event understanding according to claim 1, characterized in that: The method of obtaining the data set includes: Complex event instances are collected and manually annotated; each annotated instance contains a triple (q, K, D), where q represents the user's query; K represents the list of key points contained in the answer; and D represents the set of news reports related to the complex event in the query.
3. A dynamic hypothesis verification method for complex event understanding according to claim 1, characterized in that: The method for the dynamic multi-perspective thinking agent framework to obtain the comprehensive answer includes: Step 1: Propose a set of logical hypotheses based on the user's query on complex events; Step 2: Select a hypothesis from the set of logical hypotheses, retrieve evidence and perform reasoning based on the hypothesis to obtain a reasoning result; and delete the hypothesis from the set of logical hypotheses; Step 3: Based on the inference result, modify the current logical assumption set, the modification includes deleting the logical assumption and adding a new logical assumption; repeat step 2 for the modified logical assumption set until the logical assumption set is empty; form the final argument set with all the assumptions whose inference results are true and the retrieved evidence, and proceed to step 4; Step 4: Obtain the comprehensive answer based on the language model and the recorded final argument set.
4. A dynamic hypothesis verification method for complex event understanding according to claim 3, characterized in that: For user queries on complex events, the logical hypothesis is proposed; a query generator is used to analyze the logical hypothesis to generate several independent queries; for each query, a retriever is used to search an external news database to provide the evidence; a reasoner is used to perform synthetic reasoning on the evidence to obtain the reasoning result.
5. A dynamic hypothesis verification method for complex event understanding according to claim 4, characterized in that: During the retrieval process, the retriever obtains the evidence based on title relevance, content relevance and time relevance; The method for calculating the title relevance includes: The language model is used to generate an embedding vector for each title text, and the cosine similarity between the title embedding vector and the query embedding vector is calculated as the title relevance; The time correlation is an exponential decay function of the time difference between the news timestamp and the time sentinel; The three correlation scores are normalized using the minimum-maximum scaling method, and the final retrieval score of the evidence is: score=α1·title+α2·content+α3·\temporal, Where α1, α2, and α3 represent weight coefficients respectively; title, content, and temporal represent the relevance scores between the input query and the corresponding article in terms of title, content, and time respectively.
6. A dynamic hypothesis verification device for complex event understanding, the device being used to implement the method according to any one of claims 1 to 5, characterized in that: include: Dataset construction module and dynamic multi-perspective thinking agent framework module; The data set construction module is used to collect and annotate data sets; The dynamic multi-perspective thinking agent framework module is used to construct a dynamic multi-perspective thinking agent framework, use the data set to train the dynamic multi-perspective thinking agent framework, and generate comprehensive answers to users' queries about complex events based on the trained dynamic multi-perspective thinking agent framework.
7. A dynamic hypothesis verification device for complex event understanding according to claim 6, characterized in that: The method for the data set construction module to obtain the data set includes: Complex event instances are collected and manually annotated; each annotated instance contains a triple (q, K, D), where q represents the user's query; K represents the list of key points contained in the answer; and D represents the set of news reports related to the complex event in the query.
8. A dynamic hypothesis verification device for complex event understanding according to claim 6, characterized in that: The dynamic multi-perspective thinking agent framework module includes: a query input unit, a logic proof search and reasoning unit, a hypothesis modification unit and a solution generation unit; The query input unit is used to propose a set of logical hypotheses based on the user's query on the complex event; The logic proof search and reasoning unit is used to analyze each hypothesis, search for evidence based on the hypothesis, and reason about the hypothesis based on the evidence to obtain a reasoning result; and delete the hypothesis from the logic hypothesis set; The hypothesis modification unit is used to modify the logical hypothesis set based on the reasoning result, and the modification includes deleting the logical hypothesis and adding a new logical hypothesis; and inputting the new logical hypothesis set into the logical proof search and reasoning unit until the logical hypothesis set is empty; all the hypotheses with true reasoning results and the retrieved evidence form a final argument set, and input the final argument set into the solution generation unit; The solution generation unit is used to obtain the comprehensive answer based on the language model and the final argument set.
9. A dynamic hypothesis verification device for complex event understanding according to claim 8, characterized in that: The logic proof search and reasoning unit includes: a query generator, a retriever and a reasoner; The query generator is used to analyze the logical assumption and generate a plurality of independent queries; The retriever is used to retrieve an external news database to provide the evidence based on each query; The reasoner is used to perform synthetic reasoning on the evidence to obtain the reasoning result.
10. A dynamic hypothesis verification device for complex event understanding according to claim 9, characterized in that: During the retrieval process, the retriever obtains the evidence based on title relevance, content relevance and time relevance; The method for calculating the title relevance includes: The language model is used to generate an embedding vector for each title text, and the cosine similarity between the title embedding vector and the query embedding vector is calculated as the title relevance; The time correlation is an exponential decay function of the time difference between the news timestamp and the time sentinel; The three correlation scores are normalized using the minimum-maximum scaling method, and the final retrieval score of the evidence is: score=α1·title+α2·content+α3·\temporal, Where α1, α2, and α3 represent weight coefficients respectively; title, content, and temporal represent the relevance scores between the input query and the corresponding article in terms of title, content, and time respectively.
Citation Information
Patent Citations
Multi-agent collaborative knowledge reasoning framework and system based on large language model
CN117744795A
