An agent and large language model-based financial document retrieval method, device, equipment and storage medium

Abstract

The application discloses a financial document retrieval method and device based on an agent and a large language model, equipment and a storage medium, relates to the technical field of artificial intelligence, and comprises the following steps: acquiring an initial evidence set from an external knowledge base through a retriever, and performing structured extraction on a document to generate supporting evidence notes; then, the notes are compared with each other to identify factual contradictions, and conflict evidence is formed; then, the conflict evidence is evaluated by using a plurality of independent evidence evaluation agents, and the result is corrected and debated by using a debate coordination agent; finally, the evidence is screened and integrated based on the debate result, an answer is generated by a large language model, the source and the reliability are marked, and finally, the target financial document retrieval result is output, so that the efficiency of financial document retrieval is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a financial document retrieval method, apparatus, device, and storage medium based on intelligent agents and large language models. Background Technology

[0002] Currently, Retrieval-augmented Generation (RAG) technology effectively compensates for the shortcomings of Large Language Models (LLMs) in terms of up-to-date and domain-specific knowledge by combining external knowledge base retrieval with Large Language Model (LLM) generation, thereby improving the factual consistency of generated content. However, existing RAG technologies still have the following key shortcomings in practical applications:

[0003] 1. Low signal-to-noise ratio problem: Retrieved documents often contain a large amount of redundant information irrelevant to the query. This noise can interfere with the generation process of large language models, causing the generated content to deviate from the core requirements and even introducing erroneous information. Existing technologies mostly rely on simple text truncation or keyword matching for noise filtering, which cannot achieve accurate extraction and structured organization of core evidence.

[0004] 2. Lack of Evidence Conflict Resolution: In multi-source information retrieval scenarios, different documents may contain factual contradictions (such as inconsistent descriptions of the same event from different sources, conflicts between different academic viewpoints, etc.). Existing RAG technology lacks an effective conflict identification and resolution mechanism, often directly inputting conflicting evidence into the model, leading to logical contradictions and factual distortions in the generated content, making it difficult to meet the needs of high-credibility scenarios.

[0005] 3. Error accumulation and unreliability: In complex multi-hop reasoning tasks, if the existing RAG technology generates intermediate conclusions based on incomplete or erroneous fragmented evidence, it will trigger a chain of errors in subsequent reasoning. Moreover, the generated content lacks traceability of the source of evidence and credibility labeling, making it impossible to verify its reliability.

[0006] To address the aforementioned issues, existing technologies have yielded two main improvement approaches: one focuses on improving the signal-to-noise ratio through structured evidence extraction, such as the "supportive evidence notes" approach. However, this approach only focuses on refining single pieces of evidence and fails to resolve conflicts arising from multiple sources. The other approach involves multi-agent interaction to process conflicting evidence, such as the "multi-agent debate aggregation" approach. However, this approach does not perform pre-processing of retrieved evidence, resulting in the need to process a large amount of noisy information during the debate process, leading to low efficiency and limited accuracy in conflict resolution. Therefore, improving the efficiency of financial document retrieval based on agents and large language models is a pressing issue that needs to be addressed. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a financial document retrieval method, apparatus, device, and storage medium based on intelligent agents and large language models, which can improve the efficiency of financial document retrieval in the process of financial document retrieval based on intelligent agents and large language models. The specific solution is as follows:

[0008] Firstly, this application provides a financial document retrieval method based on intelligent agents and large language models, including:

[0009] Using a preset retrieval tool and based on financial document query requirements, an initial evidence set, including each document to be processed, is determined from an external knowledge base that corresponds to the financial document query requirements.

[0010] The system utilizes preset supporting evidence note generation rules to structurally extract supporting evidence notes from each document in the initial evidence set, resulting in supporting evidence notes that include core facts, uncertainty annotations, and missing information. A target evidence set is then constructed based on these supporting evidence notes. The preset supporting evidence note generation rules include content focusing rules, uncertainty annotation rules, and structured format rules. The content focusing rules are used to extract initial core facts corresponding to the financial document query requirements based on a large language model, and then compress the text length of these initial core facts using a preset compression ratio to obtain target core facts. The uncertainty annotation rules are used to extract ambiguous statements and missing evidence portions from each document in the initial evidence set based on the large language model, and then annotate these ambiguous statements and missing evidence portions using preset standardized tags to obtain uncertainty annotations and missing information. The structured format rules are used to annotate the source credibility of each document in the initial evidence set according to a preset fixed template based on the large language model to obtain source annotation results. The supporting evidence notes are determined based on the target core facts, the uncertainty annotations, the missing information, and the source annotation results.

[0011] Each of the supporting evidence notes in the target evidence set is compared pairwise to obtain evidence pairs that have factual contradictions. Each evidence pair is then marked with a conflict type and a core conflict point to obtain conflict evidence that includes both conflict type and core conflict point.

[0012] Several independent evidence evaluation agents are used to independently evaluate the conflicting evidence based on several evaluation dimensions to obtain a target evaluation opinion. Then, a debate coordination agent is used to modify and debate the evaluation results of each evidence evaluation agent based on the target evaluation opinion until a preset condition is met to obtain a debate result.

[0013] The target evidence set is filtered and integrated using a preset aggregator based on the debate results to obtain a target evidence subset. Then, a large language model is used to generate answers for the target evidence subset. Finally, the answer generation results are labeled with evidence source and credibility level to obtain the target financial document retrieval results.

[0014] Optionally, the step of comparing each of the supporting evidence notes in the target evidence set pairwise to obtain evidence pairs with factual contradictions, and marking each evidence pair with conflict type and conflict core point, to obtain conflict evidence including conflict type and conflict core point, including:

[0015] Preset prompt words for conflict identification are input into a large language model, which is then used to identify the logical consistency of the core facts in each of the supporting evidence notes, thereby obtaining evidence pairs that contain factual contradictions.

[0016] The large language model is used to judge the conflict type corresponding to each piece of evidence, and the judgment result is obtained. The core conflict point that leads to the conflict is extracted, and then the judgment result and the core conflict point are labeled to obtain conflict evidence including conflict type and core conflict point.

[0017] Optionally, the step of evaluating the agent using several independent pieces of evidence and independently evaluating the conflicting evidence based on several evaluation dimensions to obtain the target evaluation opinion includes:

[0018] Several evaluation dimensions are defined; these evaluation dimensions include the evidence credibility dimension, the query relevance dimension, and the contextual fact consistency dimension.

[0019] The intelligent agent is evaluated using several independent pieces of evidence, and the source authority and information freshness of the documents to be processed corresponding to the conflicting evidence are evaluated based on the evidence credibility dimension to obtain the corresponding first evaluation opinion.

[0020] The intelligent agent is evaluated using several independent pieces of evidence, and the degree of direct matching between the core facts in the conflicting evidence and the financial document query requirement is assessed based on the query relevance dimension to obtain the corresponding second evaluation opinion.

[0021] The agent is evaluated using several independent pieces of evidence, and the compatibility of the core facts in the conflicting evidence with the facts recorded in other non-conflicting supporting evidence notes in the target evidence set is evaluated based on the contextual fact consistency dimension, to obtain the corresponding third evaluation opinion;

[0022] The target evaluation opinion is determined based on the first evaluation opinion, the second evaluation opinion, and the third evaluation opinion.

[0023] Optionally, the step of utilizing the debate coordination agent and revising and debating the evaluation results corresponding to each of the evidence evaluation agents based on the target evaluation opinion until preset conditions are met to obtain the debate result includes:

[0024] The debate coordination agent collects the evaluation opinions of each of the evidence evaluation agents, integrates the evaluation opinions, and then distributes the integrated results to each evidence evaluation agent so that each evidence evaluation agent can review, revise and debate the integrated results in turn to obtain the initial debate results.

[0025] Determine whether the variance of the score change corresponding to the initial debate result is less than a preset threshold. If the variance of the score change corresponding to the initial debate result is less than the preset threshold, then the initial debate result is determined to be the target debate result.

[0026] Optionally, the step of using a preset aggregator and filtering and integrating the target evidence set based on the debate results to obtain a target evidence subset includes:

[0027] Determine the credibility score and consensus state corresponding to each supporting evidence note in the debate results, and perform a consensus interpretation of the target evidence set based on the consensus state for specific terms or contexts, and perform disambiguation processing on the consensus interpretation results to obtain the disambiguated supporting evidence notes to be processed.

[0028] Based on the credibility score, the supporting evidence notes to be processed are sorted in descending order of the score, and then the top preset number of supporting evidence notes to be processed in the sorting result are determined.

[0029] Conflict resolution operations are performed on each of the aforementioned supporting evidence notes to obtain the target evidence subset.

[0030] Optionally, the step of generating answers using a large language model for the target evidence subset, and then labeling the evidence source and credibility level of the answer generation results to obtain the target financial document retrieval results, includes:

[0031] The target answer is decomposed into several independent factual statement units, and each factual statement unit is semantically matched and factually verified with each supporting evidence note in the target evidence subset to obtain the notes to be processed;

[0032] The credibility level of the source credibility field in each of the supporting evidence notes is determined based on a preset structured format to obtain the credibility level determination result. Then, the evidence source field in each of the supporting evidence notes is determined to determine the evidence source level to obtain the evidence source level determination result. The evidence source level determination result includes authoritative source level, reliable source level, and ordinary source level.

[0033] By associating and annotating each factual statement unit with its corresponding supporting evidence note's unique identifier, source information, credibility level determination result, and evidence source level determination result, the target financial document retrieval results are obtained.

[0034] Secondly, this application provides a financial document retrieval device based on intelligent agents and large language models, comprising:

[0035] The initial evidence set determination module is used to determine an initial evidence set, including each document to be processed, from an external knowledge base based on a preset retrieval device and the financial document query requirements.

[0036] The target evidence set determination module is used to structurally extract the structure of each document to be processed in the initial evidence set using preset supporting evidence note generation rules, obtaining supporting evidence notes including core facts, uncertainty annotations, and missing information, so as to construct a target evidence set based on each supporting evidence note; the preset supporting evidence note generation rules include content focusing rules, uncertainty annotation rules, and structured format rules; the content focusing rules are used to extract the initial core facts corresponding to the financial document query requirements based on a large language model, and then compress the text length of the initial core facts using a preset compression ratio to obtain the target core facts; the uncertainty annotation rules are used to extract the ambiguous expressions and missing evidence parts in each document to be processed based on the large language model, and then annotate the ambiguous expressions and missing evidence parts using preset standardized tags to obtain uncertainty annotations and missing information; the structured format rules are used to annotate the source credibility of each document to be processed according to a preset fixed template based on the large language model to obtain source annotation results; the supporting evidence notes are determined based on the target core facts, the uncertainty annotations, the missing information, and the source annotation results;

[0037] The conflict evidence identification module is used to compare each of the supporting evidence notes in the target evidence set in pairs to obtain evidence pairs with factual contradictions, and to mark each evidence pair with conflict type and conflict core point to obtain conflict evidence including conflict type and conflict core point.

[0038] The debate result generation module is used to use several independent evidence evaluation agents and independently evaluate the conflicting evidence based on several evaluation dimensions to obtain the target evaluation opinion. Then, the debate coordination agent is used to modify and debate the evaluation results of each evidence evaluation agent based on the target evaluation opinion until the preset conditions are met to obtain the debate result.

[0039] The retrieval result generation module is used to filter and integrate the target evidence set based on the debate results using a preset aggregator to obtain a target evidence subset, and to generate answers for the target evidence subset using a large language model. Then, the answer generation results are labeled with evidence source and credibility level to obtain the target financial document retrieval results.

[0040] Thirdly, this application provides an electronic device, comprising:

[0041] Memory, used to store computer programs;

[0042] A processor is used to execute the computer program to implement the aforementioned financial document retrieval method based on intelligent agents and large language models.

[0043] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned financial document retrieval method based on intelligent agents and large language models.

[0044] As can be seen from the above, before performing financial document retrieval based on intelligent agents and large language models, this application needs to use a pre-set retrieval tool and, based on the financial document query requirements, determine an initial evidence set from an external knowledge base, including each document to be processed, corresponding to the financial document query requirements. Then, using pre-set supporting evidence note generation rules, the documents to be processed in the initial evidence set are structurally extracted to obtain supporting evidence notes including core facts, uncertainty annotations, and missing information, which are used to construct an evidence set. Finally, each supporting evidence note in the evidence set is compared pairwise to obtain evidence pairs with factual contradictions, and the conflict type and conflict core of each evidence pair are determined. The process involves: identifying conflict points and obtaining conflict evidence, including conflict type and core points; evaluating conflict evidence independently using several independent evidence assessment agents based on several assessment dimensions to obtain target assessment opinions; then using a debate coordination agent to revise and debate the assessment results of each evidence assessment agent based on the target assessment opinions until preset conditions are met, resulting in a debate result; using a preset aggregator to filter and integrate the evidence set based on the debate result to obtain a target evidence subset; using a large language model to generate answers for the target evidence subset; and then labeling the answer generation results with evidence source and credibility level to obtain the target financial document retrieval results.

[0045] Therefore, this application first needs to utilize a pre-defined retrieval tool and, based on the financial document query requirements, determine an initial evidence set from an external knowledge base, including all documents to be processed, corresponding to the financial document query requirements. Then, using pre-defined supporting evidence note generation rules, the documents to be processed in the initial evidence set are structurally extracted to obtain supporting evidence notes including core facts, uncertainty annotations, and missing information, thereby constructing an evidence set based on these supporting evidence notes. Next, each supporting evidence note in the evidence set is compared pairwise to obtain evidence pairs with factual contradictions, and each evidence pair is marked with conflict type and conflict core points, resulting in an evidence set including conflict type and conflict core points. The process begins with identifying conflicting evidence at the core of the conflict. Then, several independent evidence evaluation agents independently assess the conflicting evidence based on several evaluation dimensions, yielding a target evaluation opinion. A debate coordination agent then refines and debates the evaluation results of each evidence evaluation agent based on the target evaluation opinion, continuing until preset conditions are met, resulting in a debate outcome. Finally, a preset aggregator filters and integrates the target evidence set based on the debate outcome, obtaining a target evidence subset. A large language model is then used to generate answers from this target evidence subset, and the generated answers are labeled with evidence sources and credibility levels to obtain the target financial document retrieval results. This approach improves the efficiency of financial document retrieval based on agents and a large language model, thereby enhancing the user experience. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0047] Figure 1 This is a flowchart of a financial document retrieval method based on intelligent agents and large language models disclosed in this application;

[0048] Figure 2 This is a schematic diagram of the structure of a financial document retrieval device based on an intelligent agent and a large language model disclosed in this application;

[0049] Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Currently, retrieval augmentation (RAG) technology effectively compensates for the shortcomings of large language models in terms of up-to-date and domain-specific knowledge by combining external knowledge base retrieval with large language model generation, thereby improving the factual consistency of generated content. However, existing RAG technology still suffers from the following key drawbacks in practical applications: low signal-to-noise ratio, lack of evidence conflict resolution, error accumulation, and unreliability. Therefore, this application provides a financial document retrieval method based on intelligent agents and large language models, which can improve the efficiency of financial document retrieval in the process of retrieving financial documents using intelligent agents and large language models.

[0052] See Figure 1 As shown in the figure, this invention discloses a financial document retrieval method based on intelligent agents and large language models, including:

[0053] Step S11: Using a preset retrieval tool and based on the financial document query requirements, determine an initial evidence set from an external knowledge base that corresponds to the financial document query requirements, including each document to be processed.

[0054] In this embodiment, during the financial document retrieval process based on intelligent agents and a large language model, the embodiment first needs to perform query reception and retrieval: receiving user queries, retrieving Top-K documents related to the user query from an external knowledge base through a retrieval tool to form an initial evidence set; secondly, refining the evidence: based on preset supporting evidence note generation rules as prompts for the large language model, each document in the initial evidence set is structurally refined to generate supporting evidence notes containing core facts, uncertainty annotations, and missing information, filtering redundant noise information to form a refined evidence set; then, conflict identification: comparing each supporting evidence note in the refined evidence set pairwise through the prompts set of the large language model, identifying evidence pairs with factual contradictions, and marking the conflict type and core conflict points; Furthermore, multi-agent debate resolution is performed: multiple independent evidence evaluation agents are activated to independently evaluate conflicting evidence based on different evaluation dimensions (credibility, relevance, and factual consistency) and output evaluation opinions; a debate coordination agent organizes multiple rounds of debate, guiding each evidence evaluation agent to revise its own evaluation results based on the evaluation opinions of other agents until a consensus is reached or a preset number of debate rounds are reached; then, evidence aggregation is performed: based on the debate results, an aggregator filters and integrates the purified evidence set, eliminating erroneous evidence and retaining highly credible and consistent evidence to form a final reliable evidence subset; finally, credible generation is performed: the large language model generates the target answer based on the final reliable evidence subset, and labels the evidence source (supporting evidence note ID, original document source) and credibility level of each factual statement in the answer, outputting the final result.

[0055] Step S12: The document to be processed in the initial evidence set is structurally extracted using preset supporting evidence note generation rules to obtain supporting evidence notes including core facts, uncertainty annotations, and missing information, thereby constructing a target evidence set based on these supporting evidence notes. The preset supporting evidence note generation rules include content focusing rules, uncertainty annotation rules, and structured format rules. The content focusing rules are used to extract initial core facts corresponding to the financial document query requirements based on a large language model, and then compress the text length of the initial core facts using a preset compression ratio to obtain target core facts. The uncertainty annotation rules are used to extract ambiguous expressions and missing evidence parts in each document to be processed based on the large language model, and then annotate the ambiguous expressions and missing evidence parts using preset standardized tags to obtain uncertainty annotations and missing information. The structured format rules are used to annotate the source credibility of each document to be processed according to a preset fixed template based on the large language model to obtain source annotation results. The supporting evidence notes are determined based on the target core facts, the uncertainty annotations, the missing information, and the source annotation results.

[0056] In this embodiment, the rules for generating supporting evidence notes include: content focus: retaining only core factual information directly related to the user's query, eliminating redundant descriptions, and compressing the text to 1 / 5-1 / 3 of the original document length; uncertainty labeling: for ambiguous information (such as "may" or "not explicitly mentioned") or parts lacking evidence, standardized labels (such as...) are used. <uncertain>,<No Evidence> The content is then annotated; subsequently, it is structured using a fixed template: "Core Facts: XXX; Uncertainties: XXX; Missing Information: XXX; Source Credibility: XXX", where source credibility is based on the authority of the original document's publishing institution and the validity of its publication time. Notably, the generation of supporting evidence notes is achieved through a finely tuned large language model, with training data consisting of triples of "user queries + original documents + manual annotations," and the training sample size is expanded through bootstrapping learning.

[0057] Step S13: Compare each of the supporting evidence notes in the target evidence set in pairs to obtain evidence pairs with factual contradictions, and mark the conflict type and core conflict point of each evidence pair to obtain conflict evidence including conflict type and core conflict point.

[0058] In this embodiment, each supporting evidence note in the target evidence set is compared pairwise to obtain evidence pairs with factual contradictions. Each evidence pair is then marked with a conflict type and a core conflict point to obtain conflict evidence including conflict type and core conflict point. This can include: inputting preset prompt words for conflict identification into a large language model to identify the logical consistency of core facts in each supporting evidence note using the large language model to obtain evidence pairs with factual contradictions; using the large language model to judge the conflict type corresponding to each evidence pair to obtain the judgment result, extracting the core conflict point that leads to the conflict, and then marking the judgment result and the core conflict point to obtain conflict evidence including conflict type and core conflict point.

[0059] Step S14: Utilize several independent evidence evaluation agents and independently evaluate the conflicting evidence based on several evaluation dimensions to obtain target evaluation opinions. Then, utilize a debate coordination agent and, based on the target evaluation opinions, revise and debate the evaluation results corresponding to each of the evidence evaluation agents until preset conditions are met to obtain the debate results.

[0060] In this embodiment, the specific process of multi-agent debate includes: Initialization: Each evidence evaluation agent independently analyzes conflicting evidence based on its own preset evaluation dimensions and outputs an evaluation report containing evidence scores (0-10 points), conflict point judgments, and supporting reasons. Specifically, using several independent evidence evaluation agents and based on several evaluation dimensions to independently evaluate conflicting evidence and obtain target evaluation opinions may include: determining several evaluation dimensions; evaluation dimensions include evidence credibility dimension, query relevance dimension, and contextual fact consistency dimension; using several independent evidence evaluation agents and based on the evidence credibility dimension to evaluate the source authority and information freshness of the document to be processed corresponding to the conflicting evidence, and obtaining the corresponding first evaluation opinion; using several independent evidence evaluation agents and based on the query relevance dimension to evaluate the direct matching degree between the core facts in the conflicting evidence and the financial document query requirements, and obtaining the corresponding second evaluation opinion; using several independent evidence evaluation agents and based on the contextual fact consistency dimension to evaluate the compatibility between the core facts in the conflicting evidence and the facts recorded in other non-conflicting supporting evidence notes in the target evidence set, and obtaining the corresponding third evaluation opinion; and determining the target evaluation opinion based on the first evaluation opinion, the second evaluation opinion, and the third evaluation opinion.

[0061] In this embodiment, multiple rounds of interaction are required. The debate coordination agent is invoked to summarize the evaluation reports and synchronize them to all evidence evaluation agents. Each evidence evaluation agent re-examines the conflicting evidence based on the evaluation reasons of other agents and corrects its own evaluation score and judgment. Subsequently, if the variance of the scores of each evidence evaluation agent is less than a preset threshold (e.g., 0.1) for two consecutive rounds, it is determined that a consensus has been reached and the debate is terminated. If no consensus is reached and the preset maximum number of debate rounds (default 3 rounds) is reached, the debate coordination agent determines the debate result based on the majority voting principle.

[0062] Specifically, the debate coordination agent is used to revise and debate the evaluation results of each evidence evaluation agent based on the target evaluation opinions until preset conditions are met to obtain the debate result. This can include: using the debate coordination agent to collect the evaluation opinions of each evidence evaluation agent, integrating the evaluation opinions, and then distributing the integrated results to each evidence evaluation agent so that each evidence evaluation agent can review, revise, and debate the integrated results in turn to obtain the initial debate result; determining whether the variance of the score change corresponding to the initial debate result is less than a preset threshold. If the variance of the score change corresponding to the initial debate result is less than the preset threshold, the initial debate result is determined to be the target debate result.

[0063] Step S15: Using a preset aggregator and based on the debate results, the target evidence set is filtered and integrated to obtain a target evidence subset. Then, a large language model is used to generate answers for the target evidence subset. Finally, the answer generation results are labeled with evidence source and credibility level to obtain the target financial document retrieval results.

[0064] In this embodiment, the aggregator's integration rules include: disambiguation processing: for ambiguous entities or concepts, the unique meaning most relevant to the user's query is associated with the debate results; priority ranking: evidence is ranked from high to low according to its score, and the top 60% of the evidence is retained first; conflict removal: conflicting evidence that has not reached a consensus is directly removed if its score is lower than a preset threshold (e.g., 4 points); if the score is higher than the threshold, the conflict points and the viewpoints of different evidence are clearly marked in the final answer.

[0065] Specifically, the target evidence set is filtered and integrated using a pre-defined aggregator based on the debate results to obtain a subset of target evidence. This may include: determining the credibility score and consensus state corresponding to each supporting evidence note in the debate results; performing a consensus interpretation of specific terms or situations on the target evidence set based on the consensus state; disambiguating the consensus interpretation results to obtain disambiguated supporting evidence notes to be processed; sorting each supporting evidence note to be processed based on the credibility score and in descending order of score; determining the top pre-defined number of supporting evidence notes to be processed in the sorted results; and performing conflict resolution operations on each supporting evidence note to be processed to obtain the subset of target evidence.

[0066] Furthermore, the credibility level includes three levels: "Authoritative Source," "Reliable Source," and "Ordinary Source," which correspond to the original document source being an official authoritative institution, a professional platform, and an ordinary online source, respectively.

[0067] Specifically, a large language model is used to generate answers for a subset of target evidence. The generated answers are then labeled with evidence sources and credibility levels to obtain the target financial document retrieval results. This process may include: decomposing the target answer into several independent factual statement units, performing semantic matching and fact-checking on each factual statement unit with each supporting evidence note in the target evidence subset to obtain notes to be processed; determining the credibility level of the source credibility field in each supporting evidence note based on a preset structured format to obtain credibility level determination results; then determining the evidence source level of the evidence source field in each supporting evidence note to obtain evidence source level determination results; the evidence source level determination results include authoritative source level, reliable source level, and ordinary source level; and associating and labeling each factual statement unit with the unique identifier, source information, credibility level determination results, and evidence source level determination results of the corresponding supporting evidence notes to obtain the target financial document retrieval results.

[0068] In one specific implementation, this application embodiment first performs query reception and retrieval: receiving user queries for "application conditions and amount limits for personal housing provident fund loans," and retrieving the Top-10 relevant documents from a financial knowledge base to form an initial evidence set. Subsequently, evidence refinement is performed: using a finely tuned LLaMA-2-13B model and based on preset supporting evidence note generation rules, the 10 documents in the initial evidence set are structurally refined. The generation rules can be set as follows: core facts are retained only for financial policy conclusions related to "application conditions for personal housing provident fund loans" and "amount limits"; uncertainty labeling uses... <uncertain>Ambiguous information such as "differentiated implementation standards in different cities" is marked; the structured template is "core facts: XXX; uncertainties: XXX; missing information: XXX; source credibility: XXX", with source credibility based on document type labeling, and finally 10 supporting evidence notes are generated to form a refined evidence set.

[0069] Then, conflict identification was performed: a pre-trained DeBERTa-large natural language inference model was used to compare the 10 supporting evidence notes pairwise, and two sets of conflicting evidence pairs were identified: one set was a conflict about the "down payment ratio for first-time home provident fund loans" (Evidence A believes that the down payment ratio should not be less than 20%, while Evidence B believes that it should not be less than 30%), and the other set was a conflict about the "required length of provident fund account contributions" (Evidence C believes that 6 consecutive months of contributions are sufficient to apply, while Evidence D believes that 12 consecutive months of contributions are required). The core points of the conflict were marked as "difference in down payment ratio" and "difference in contribution length threshold".

[0070] Furthermore, a multi-agent debate was conducted to resolve the issue: three evidence evaluation agents were activated to evaluate the evidence from three dimensions: "policy authority," "regional applicability," and "implementation timeliness." GPT-4 was used as the debate coordinating agent to organize three rounds of debate. Subsequently, initialization was performed: Agent 1 (policy authority dimension) believed that evidence A (unified policy of the Housing Provident Fund Management Center) was more authoritative than evidence B (supplementary guidelines from a local bank), scoring 9 points; Agent 2 (regional applicability dimension) believed that the 30% down payment ratio mentioned in evidence B was applicable to cities with purchase restrictions and was more in line with users' potential regional needs, scoring 8 points; Agent 3 (implementation timeliness dimension) believed that evidence B was released in 2025, later than evidence A in 2023, scoring 8.5 points.

[0071] Furthermore, this application embodiment requires multiple rounds of interaction: After the first round of debate, Agent 1, combining the evaluation opinions of Agents 2 and 3, corrects the score of Evidence B to 8.2 points; Agent 2, combining the authoritative evaluation of Agent 1, corrects the score of Evidence A to 8.8 points; After the second round of debate, the variance of the scores of each agent for the two sets of conflicting evidence is less than 0.1, reaching a consensus; Then, a termination judgment is made: Due to the consensus reached, the debate is terminated, and the debate results are output: Evidence A (down payment ratio not less than 20%) is a national standard, Evidence B (30%) is a special standard for cities with purchase restrictions; Evidence C (continuous payment for 6 months) is the latest adjusted policy, and Evidence D (12 months) is an old policy that has expired. Then, evidence aggregation is performed: based on the debate results, the aggregator removes erroneous evidence D and retains the remaining 9 supporting evidence notes; conflicting evidence pairs are sorted according to the principle of "general standards take precedence, special scenarios are supplemented", and the core facts of evidence A and C are adopted first; for the ambiguous "city scope of purchase restriction", it is uniformly marked as "real estate purchase restriction areas announced by each province and city" based on the debate results, and finally a final reliable evidence subset containing 7 highly credible pieces of evidence is formed.

[0072] In this embodiment, the application requires the use of the GPT-4 model and the generation of the answer based on the final reliable evidence subset. The content is as follows: "The application conditions and amount limits for personal housing provident fund loans are as follows: 1. Application conditions: Continuous contribution to the provident fund for 6 months or more, normal account status; stable source of income and good credit; valid purchase contract. 2. Down payment ratio: The minimum down payment ratio is no less than 20% in general, and no less than 30% in cities with purchase restrictions. 3. Amount limit: The maximum loan amount for a single person is no more than 600,000 yuan, and the maximum loan amount for a couple applying together is no more than 1,200,000 yuan." The source of evidence for each conclusion (e.g., "Continuous contribution for 6 months: Evidence C, source: 'Notice on Adjustment of Housing Provident Fund Loan Policy (2025)'") and its credibility level are also marked, and the final result is output.

[0073] As can be seen from the above, the embodiments of this application first need to use a preset retrieval tool and, based on the financial document query requirements, determine an initial evidence set from an external knowledge base, including each document to be processed, corresponding to the financial document query requirements. Then, using preset supporting evidence note generation rules, the documents to be processed in the initial evidence set are structurally extracted to obtain supporting evidence notes including core facts, uncertainty annotations, and missing information, so as to construct an evidence set based on each supporting evidence note. Next, each supporting evidence note in the evidence set is compared pairwise to obtain evidence pairs with factual contradictions, and each evidence pair is marked with conflict type and conflict core points to obtain a set including conflict categories. The process involves several steps: First, conflicting evidence is identified at the core of the conflict. Then, several independent evidence evaluation agents are used to independently evaluate the conflicting evidence based on several evaluation dimensions, yielding a target evaluation opinion. Next, a debate coordination agent is used to revise and debate the evaluation results of each evidence evaluation agent based on the target evaluation opinion, until preset conditions are met, resulting in a debate outcome. Finally, a preset aggregator is used to filter and integrate the target evidence set based on the debate outcome, resulting in a target evidence subset. A large language model is then used to generate answers for this target evidence subset, and the generated answers are labeled with evidence sources and credibility levels to obtain the target financial document retrieval results. This approach improves the efficiency of financial document retrieval based on agents and a large language model, thereby enhancing the user experience.

[0074] Accordingly, see Figure 2 As shown, this application also provides a financial document retrieval device based on intelligent agents and large language models, including:

[0075] The initial evidence set determination module 11 is used to determine an initial evidence set, including each document to be processed, from an external knowledge base based on a preset retrieval device and financial document query requirements.

[0076] The target evidence set determination module 12 is used to perform structured extraction on each of the documents to be processed in the initial evidence set using preset supporting evidence note generation rules, to obtain supporting evidence notes including core facts, uncertainty annotations, and missing information, so as to construct a target evidence set based on each of the supporting evidence notes; the preset supporting evidence note generation rules include content focusing rules, uncertainty annotation rules, and structured format rules; the content focusing rules are used to extract initial core facts corresponding to the financial document query requirements based on a large language model, and then compress the text length of the initial core facts using a preset compression ratio to obtain target core facts; the uncertainty annotation rules are used to extract ambiguous expressions and missing evidence parts in each of the documents to be processed based on the large language model, and then annotate the ambiguous expressions and missing evidence parts using preset standardized tags to obtain uncertainty annotations and missing information; the structured format rules are used to annotate the source credibility of each of the documents to be processed according to a preset fixed template based on the large language model to obtain source annotation results; the supporting evidence notes are determined based on the target core facts, the uncertainty annotations, the missing information, and the source annotation results;

[0077] The conflict evidence determination module 13 is used to compare each of the supporting evidence notes in the target evidence set in pairs to obtain evidence pairs with factual contradictions, and to mark each evidence pair with conflict type and conflict core point to obtain conflict evidence including conflict type and conflict core point.

[0078] The debate result generation module 14 is used to use several independent evidence evaluation agents and independently evaluate the conflicting evidence based on several evaluation dimensions to obtain the target evaluation opinion. Then, it uses a debate coordination agent and, based on the target evaluation opinion, corrects and debates the evaluation results corresponding to each of the evidence evaluation agents until the preset conditions are met to obtain the debate result.

[0079] The retrieval result generation module 15 is used to filter and integrate the target evidence set based on the debate results using a preset aggregator to obtain a target evidence subset, and to generate answers for the target evidence subset using a large language model. Then, the answer generation results are labeled with evidence source and credibility level to obtain the target financial document retrieval results.

[0080] In some specific embodiments, the conflict evidence determination module 13 may specifically include:

[0081] The evidence pair generation unit is used to input preset prompt words for conflict identification into a large language model, so as to use the large language model to identify the logical consistency relationship of the core facts of each of the supporting evidence notes, and obtain evidence pairs with factual contradictions.

[0082] The conflict core point extraction unit is used to use the large language model to judge the conflict type corresponding to each piece of evidence, obtain the judgment result, extract the conflict core point that leads to the conflict, and then label the judgment result and the conflict core point to obtain conflict evidence including conflict type and conflict core point.

[0083] In some specific embodiments, the debate result generation module 14 may specifically include:

[0084] An evaluation dimension determination unit is used to determine several evaluation dimensions; the evaluation dimensions include evidence credibility dimension, query relevance dimension, and contextual fact consistency dimension.

[0085] The first evaluation opinion determination unit is used to evaluate the intelligent agent using several independent pieces of evidence and to evaluate the source authority and information freshness of the document to be processed corresponding to the conflicting evidence based on the evidence credibility dimension, so as to obtain the corresponding first evaluation opinion.

[0086] The second evaluation opinion determination unit is used to evaluate the intelligent agent using several independent pieces of evidence and to evaluate the degree of direct matching between the core facts in the conflicting evidence and the financial document query requirements based on the query relevance dimension, so as to obtain the corresponding second evaluation opinion.

[0087] The third evaluation opinion determination unit is used to evaluate the agent using several independent pieces of evidence and, based on the contextual fact consistency dimension, evaluate the compatibility between the core facts in the conflicting evidence and the facts recorded in other non-conflicting supporting evidence notes in the target evidence set, so as to obtain the corresponding third evaluation opinion.

[0088] The target evaluation opinion generation unit is used to determine the target evaluation opinion based on the first evaluation opinion, the second evaluation opinion and the third evaluation opinion.

[0089] In some specific embodiments, the debate result generation module 14 may specifically include:

[0090] The integration result generation unit is used to collect the evaluation opinions of each of the evidence evaluation agents by the debate coordination agent, integrate the evaluation opinions, and then distribute the integration result to each evidence evaluation agent so that each evidence evaluation agent can review, correct and debate the integration result in turn to obtain the initial debate result.

[0091] The rating change variance judgment unit is used to determine whether the rating change variance corresponding to the initial debate result is less than a preset threshold. If the rating change variance corresponding to the initial debate result is less than the preset threshold, the initial debate result is determined to be the target debate result.

[0092] In some specific embodiments, the retrieval result generation module 15 may specifically include:

[0093] A credibility score generation unit is used to determine the credibility score and consensus state corresponding to each supporting evidence note in the debate results, so as to perform a consensus interpretation of specific terms or situations on the target evidence set based on the consensus state, and to disambiguate the consensus interpretation results to obtain the disambiguated supporting evidence notes to be processed.

[0094] The evidence note sorting unit is used to sort each of the supporting evidence notes to be processed based on the credibility score and in descending order of the score, and then determine the first preset number of supporting evidence notes to be processed in the sorting result.

[0095] The target evidence subset generation unit is used to perform conflict resolution operations on each of the supporting evidence notes to be processed to obtain the target evidence subset.

[0096] In some specific embodiments, the retrieval result generation module 15 may specifically include:

[0097] The note-to-be-processed unit is used to decompose the target answer into several independent factual statement units, and to perform semantic matching and fact verification between each factual statement unit and each supporting evidence note in the target evidence subset to obtain the note-to-be-processed.

[0098] The credibility level determination result unit is used to determine the credibility level of the source credibility field in each of the supporting evidence notes based on a preset structured format, and obtain the credibility level determination result. Then, it determines the evidence source level of the evidence source field in each of the supporting evidence notes, and obtains the evidence source level determination result. The evidence source level determination result includes authoritative source level, reliable source level, and ordinary source level.

[0099] The retrieval result generation sub-unit is used to associate and annotate each of the factual statement units with the unique identifier, source information, credibility level determination result, and evidence source level determination result of the corresponding supporting evidence notes to obtain the target financial document retrieval results.

[0100] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the financial document retrieval method based on intelligent agents and large language models disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0101] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0102] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0103] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the financial document retrieval method based on intelligent agents and large language models disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0104] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed financial document retrieval method based on intelligent agents and a large language model. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0105] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0106] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0107] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0108] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0109] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / uncertain> < / uncertain>

Claims

1. A method for financial document retrieval based on an agent and a large language model, characterized in that, include: Using a preset retrieval tool and based on financial document query requirements, an initial evidence set, including each document to be processed, is determined from an external knowledge base that corresponds to the financial document query requirements. The documents to be processed in the initial evidence set are structurally extracted using preset supporting evidence note generation rules to obtain supporting evidence notes including core facts, uncertainty annotations, and missing information, so as to construct a target evidence set based on the supporting evidence notes. The preset supporting evidence note generation rules include content focusing rules, uncertainty labeling rules, and structured format rules. The content focusing rules are used to extract initial core facts corresponding to the financial document query requirements based on a large language model, and then compress the text length of the initial core facts using a preset compression ratio to obtain the target core facts. The uncertainty labeling rules are used to extract ambiguous statements and missing evidence parts in each of the documents to be processed based on the large language model, and then label the ambiguous statements and missing evidence parts using preset standardized tags to obtain uncertainty labels and missing information. The structured format rules are used to label the source credibility of each of the documents to be processed according to a preset fixed template based on the large language model to obtain source labeling results. The supporting evidence notes are determined based on the target core facts, the uncertainty labels, the missing information, and the source labeling results. Each of the supporting evidence notes in the target evidence set is compared pairwise to obtain evidence pairs that have factual contradictions. Each evidence pair is then marked with a conflict type and a core conflict point to obtain conflict evidence that includes both conflict type and core conflict point. Several independent evidence evaluation agents are used to independently evaluate the conflicting evidence based on several evaluation dimensions to obtain a target evaluation opinion. Then, a debate coordination agent is used to modify and debate the evaluation results of each evidence evaluation agent based on the target evaluation opinion until a preset condition is met to obtain a debate result. The target evidence set is filtered and integrated using a preset aggregator based on the debate results to obtain a target evidence subset. Then, a large language model is used to generate answers for the target evidence subset. Finally, the answer generation results are labeled with evidence source and credibility level to obtain the target financial document retrieval results. The step of using a debate coordination agent to revise and debate the evaluation results of each evidence evaluation agent based on the target evaluation opinion until a preset condition is met to obtain a debate result includes: using the debate coordination agent to collect the evaluation opinions of each evidence evaluation agent, integrating the evaluation opinions, and then distributing the integrated results to each evidence evaluation agent so that each evidence evaluation agent can sequentially review, revise, and debate the integrated results to obtain an initial debate result; determining whether the variance of the score change corresponding to the initial debate result is less than a preset threshold; if the variance of the score change corresponding to the initial debate result is less than the preset threshold, then the initial debate result is determined to be the target debate result.

2. The agent and large language model based financial document retrieval method according to claim 1, characterized in that, The supporting evidence notes in the target evidence set are compared pairwise to obtain evidence pairs with factual contradictions. Each evidence pair is then marked with a conflict type and a core conflict point, resulting in conflict evidence including the conflict type and core conflict point, including: Preset prompt words for conflict identification are input into a large language model, which is then used to identify the logical consistency of the core facts in each of the supporting evidence notes, thereby obtaining evidence pairs that contain factual contradictions. The large language model is used to judge the conflict type corresponding to each piece of evidence, and the judgment result is obtained. The core conflict point that leads to the conflict is extracted, and then the judgment result and the core conflict point are labeled to obtain conflict evidence including conflict type and core conflict point.

3. The agent and large language model based financial document retrieval method of claim 1, wherein, The process of evaluating the agent using several independent pieces of evidence and independently evaluating the conflicting evidence based on several evaluation dimensions to obtain a target evaluation opinion includes: Several evaluation dimensions are defined; these evaluation dimensions include the evidence credibility dimension, the query relevance dimension, and the contextual fact consistency dimension. The intelligent agent is evaluated using several independent pieces of evidence, and the source authority and information freshness of the documents to be processed corresponding to the conflicting evidence are evaluated based on the evidence credibility dimension to obtain the corresponding first evaluation opinion. The intelligent agent is evaluated using several independent pieces of evidence, and the degree of direct matching between the core facts in the conflicting evidence and the financial document query requirement is assessed based on the query relevance dimension to obtain the corresponding second evaluation opinion. The agent is evaluated using several independent pieces of evidence, and the compatibility of the core facts in the conflicting evidence with the facts recorded in other non-conflicting supporting evidence notes in the target evidence set is evaluated based on the contextual fact consistency dimension, to obtain the corresponding third evaluation opinion; The target evaluation opinion is determined based on the first evaluation opinion, the second evaluation opinion, and the third evaluation opinion.

4. The agent and large language model based financial document retrieval method of claim 1, wherein, The step of using a preset aggregator and filtering and integrating the target evidence set based on the debate results to obtain a target evidence subset includes: Determine the credibility score and consensus state corresponding to each supporting evidence note in the debate results, and perform a consensus interpretation of the target evidence set based on the consensus state for specific terms or contexts, and perform disambiguation processing on the consensus interpretation results to obtain the disambiguated supporting evidence notes to be processed. Based on the credibility score, the supporting evidence notes to be processed are sorted in descending order of the score, and then the top preset number of supporting evidence notes to be processed in the sorting result are determined. Conflict resolution operations are performed on each of the aforementioned supporting evidence notes to obtain the target evidence subset.

5. The financial document retrieval method based on intelligent agents and large language models according to any one of claims 1 to 4, characterized in that, The process involves using a large language model to generate answers for the target evidence subset, then labeling the evidence sources and credibility levels of the generated answers to obtain the target financial document retrieval results, including: The answer generation result is decomposed into several independent factual statement units, and each factual statement unit is semantically matched and factually verified with each supporting evidence note in the target evidence subset to obtain the note to be processed; The credibility level of the source credibility field in each of the supporting evidence notes is determined based on a preset structured format to obtain the credibility level determination result. Then, the evidence source field in each of the supporting evidence notes is determined to determine the evidence source level to obtain the evidence source level determination result. The evidence source level determination result includes authoritative source level, reliable source level, and ordinary source level. By associating and annotating each factual statement unit with its corresponding supporting evidence note's unique identifier, source information, credibility level determination result, and evidence source level determination result, the target financial document retrieval results are obtained.

6. A financial document retrieval device based on intelligent agents and large language models, characterized in that, include: The initial evidence set determination module is used to determine an initial evidence set, including each document to be processed, from an external knowledge base based on a preset retrieval device and the financial document query requirements. The target evidence set determination module is used to perform structured extraction of each document to be processed in the initial evidence set using preset supporting evidence note generation rules, to obtain supporting evidence notes including core facts, uncertainty annotations, and missing information, so as to construct a target evidence set based on each of the supporting evidence notes; The preset supporting evidence note generation rules include content focusing rules, uncertainty labeling rules, and structured format rules. The content focusing rules are used to extract initial core facts corresponding to the financial document query requirements based on a large language model, and then compress the text length of the initial core facts using a preset compression ratio to obtain the target core facts. The uncertainty labeling rules are used to extract ambiguous statements and missing evidence parts in each of the documents to be processed based on the large language model, and then label the ambiguous statements and missing evidence parts using preset standardized tags to obtain uncertainty labels and missing information. The structured format rules are used to label the source credibility of each of the documents to be processed according to a preset fixed template based on the large language model to obtain source labeling results. The supporting evidence notes are determined based on the target core facts, the uncertainty labels, the missing information, and the source labeling results. The conflict evidence identification module is used to compare each of the supporting evidence notes in the target evidence set in pairs to obtain evidence pairs with factual contradictions, and to mark each evidence pair with conflict type and conflict core point to obtain conflict evidence including conflict type and conflict core point. The debate result generation module is used to use several independent evidence evaluation agents and independently evaluate the conflicting evidence based on several evaluation dimensions to obtain the target evaluation opinion. Then, the debate coordination agent is used to modify and debate the evaluation results of each evidence evaluation agent based on the target evaluation opinion until the preset conditions are met to obtain the debate result. The search result generation module is used to filter and integrate the target evidence set based on the debate results using a preset aggregator to obtain a target evidence subset, and to generate answers for the target evidence subset using a large language model. Then, the answer generation results are labeled with evidence source and credibility level to obtain the target financial document search results. Specifically, the debate result generation module is used for: The debate coordination agent collects the evaluation opinions of each evidence evaluation agent, integrates the evaluation opinions, and then distributes the integrated results to each evidence evaluation agent. Each evidence evaluation agent then reviews, corrects, and debates the integrated results in sequence to obtain an initial debate result. It is then determined whether the variance of the score change corresponding to the initial debate result is less than a preset threshold. If the variance of the score change corresponding to the initial debate result is less than the preset threshold, the initial debate result is determined to be the target debate result.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the financial document retrieval method based on intelligent agents and large language models as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the financial document retrieval method based on intelligent agents and large language models as described in any one of claims 1 to 5.

Images