A progressive retrieval enhanced dynamic knowledge question answering method

Abstract

The application discloses a dynamic knowledge question and answer method with progressive retrieval enhancement, and relates to the technical field of artificial intelligence.The application decomposes a multi-hop question into a series of logically continuous and simpler data query subtasks.In the process of analyzing each subtask, a standardized data retrieval request is iteratively generated to obtain the most needed information in the current step from an external knowledge base.Through the data retrieval mode of the whole into parts and step by step, the application ensures that each link in the complex analysis chain has accurate data supply, thereby fundamentally solving the problem that the traditional single retrieval mechanism can only obtain the first or most surface knowledge in the reasoning chain and cannot provide all the information required for the subsequent steps, thereby leading to the interruption of reasoning or the drawing of wrong conclusions.The technical scheme of the application can accurately respond to complex queries in real time.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a progressive retrieval-enhanced dynamic knowledge question answering method. Background Technology

[0002] Retrieval-Enhanced Generation (RAG)-based contextual knowledge injection takes a different approach. Instead of directly modifying model parameters, it explicitly stores all dynamically updated knowledge in an external knowledge base (such as a vector database). When processing user queries, the retrieval tool first finds the most relevant knowledge fragments from the external knowledge base. These fragments are then used as contextual information and fed into a large language model along with the original query, guiding the model to generate an answer based on the latest knowledge. This method avoids the complexity and risks associated with parameter modification and offers advantages such as efficiency and scalability. However, for complex problems requiring multiple steps and information sources (academically known as "multi-hop questions"), traditional single-retrieval mechanisms often only obtain the first or most superficial knowledge in the reasoning chain, failing to provide all the information needed for subsequent steps, leading to reasoning interruptions or incorrect conclusions. Summary of the Invention

[0003] The purpose of this invention is to address the problem that traditional single-retrieval mechanisms often only obtain the first or most superficial knowledge in the reasoning chain, failing to provide all the information required for subsequent steps, thus leading to reasoning interruptions or incorrect conclusions. This invention provides a progressively enhanced dynamic knowledge question-answering method.

[0004] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0005] A progressive retrieval-enhanced dynamic knowledge question answering method includes the following steps:

[0006] The system receives complex queries in natural language input from the user and inputs them into a trained LLM model to obtain the output answer. The trained LLM model specifically performs the following steps:

[0007] Step 1: Break down the complex problem into multiple sub-problems;

[0008] Step 2: For the first sub-problem, generate a standardized data retrieval request;

[0009] Step 3: When a retrieval trigger marker is detected in the data retrieval request, extract the sub-question from the data retrieval request;

[0010] Step 4: Based on the sub-problem, query and retrieve relevant data from an external knowledge base, format it as a text string and return it. The text string is the intermediate answer, thus obtaining the reasoning chain of "sub-problem - retrieval - intermediate answer";

[0011] Step 5: Concatenate the returned text string to the data retrieval request and use it as the starting point of the next inference chain;

[0012] Step 6: Repeat steps 2 to 5 to obtain the reasoning chains corresponding to all sub-problems;

[0013] Step 7: Based on all the reasoning chains, obtain the final output answer.

[0014] Furthermore, step 1 specifically includes:

[0015] For complex problems, autoregressive reasoning is used to identify information gaps at the current logical nodes in the text sequence, thereby determining a sub-problem.

[0016] Furthermore, the specific steps for querying and obtaining relevant data from an external knowledge base in step 4 are as follows:

[0017] Step 41: Determine whether the subproblem is structured data. If yes, proceed to step 42; otherwise, proceed to step 43.

[0018] Step 42: Start a Text-to-SQL conversion module, which dynamically compiles the sub-problems in natural language form into SQL query statements that can be directly executed in a relational database and obtain accurate query results;

[0019] Step 43: Use a text embedding model to transform the text of the sub-problem into a high-dimensional vector. Then, in a vector index built by an efficient approximate nearest neighbor search library, perform a maximum inner product or cosine similarity search to recall the most relevant text fragments as query results in milliseconds.

[0020] Furthermore, the structured data includes customer profiles and transaction records, while the unstructured data includes policy documents and research reports.

[0021] Furthermore, the trained LLM model is fine-tuned using a low-rank adaptation method, and the update matrix of the low-rank adaptation is applied to the query and value projection layers in the model's attention mechanism.

[0022] Furthermore, the text embedding model is BGE or M3E, and the efficient approximate nearest neighbor search library is FAISS or ScaNN.

[0023] Furthermore, step 4 also includes determining whether the intermediate answer has been updated. If it has been updated, the intermediate answer is updated; otherwise, it is not updated. The determination of whether the intermediate answer has been updated is performed using a trained LLM model. The training process is as follows:

[0024] Construct training samples and train an LLM model using the training samples. The training samples include direct update samples and indirect update samples. The direct update samples are target problem samples, and the indirect update samples are in-range problem samples and out-of-range problem samples.

[0025] The target question sample is a question that directly tests the data point, paired with an answer that must rely on new data to be generated.

[0026] The problem sample within the specified range is a problem that indirectly depends on the data point, which is paired with an answer that is calculated or reasoned based on new data.

[0027] The out-of-scope question sample is a question unrelated to the data fact, paired with an answer that indicates it cannot be answered or relies on internal model knowledge.

[0028] Furthermore, the loss function of the LLM model is expressed as:

[0029]

[0030] in, The actual token of the sequence at position t. After seeing the first t-1 tokens, how does the model predict the t-th token? The probability, Let N be the loss mask and N be the total number of tokens.

[0031] Furthermore, the LLM model is of the Llama series, Mistral series, or GPT series.

[0032] Furthermore, the LLM model is the Llama-2-7B model.

[0033] The beneficial effects of this invention are:

[0034] This application decomposes a multi-hop problem into a series of logically continuous and simpler data query subtasks. Furthermore, during the analysis of each subtask, standardized data retrieval requests are iteratively generated to obtain the most needed information for the current step from an external knowledge base. Through this step-by-step, modular data retrieval approach, this application ensures accurate data supply at every stage of the complex analysis chain, fundamentally solving the problem that traditional single-retrieval mechanisms often only obtain the first or most superficial knowledge in the reasoning chain, failing to provide all the information needed for subsequent steps, thus leading to reasoning interruptions or incorrect conclusions. The technical solution of this application can respond to complex queries in real time and accurately. Attached Figure Description

[0035] Figure 1 This is a diagram illustrating the overall structure of this application;

[0036] Figure 2 Example diagram showing the construction of data samples for training the "progressive retrieval" capability of this application;

[0037] Figure 3 Example diagram of data sample construction for training the "dynamic knowledge utilization" capability of this application;

[0038] Figure 4 A diagram illustrating the differences between this application and parameterized knowledge injection methods in terms of training and inference paradigms;

[0039] Figure 5 A performance comparison chart of this application and the parameterized knowledge injection method under sequential editing. Detailed Implementation

[0040] It should be noted that, where there is no conflict, the various embodiments disclosed in this application can be combined with each other.

[0041] Specific Implementation Method 1: The progressive retrieval-enhanced dynamic knowledge question answering method described in this implementation method includes the following steps:

[0042] The system receives complex queries in natural language input from the user and inputs them into a trained LLM model to obtain the output answer. The trained LLM model specifically performs the following steps:

[0043] Step 1: Break down the complex problem into multiple sub-problems;

[0044] Step 2: For the first sub-problem, generate a standardized data retrieval request;

[0045] Step 3: When a retrieval trigger marker is detected in the data retrieval request, extract the sub-question from the data retrieval request;

[0046] Step 4: Based on the sub-problem, query and retrieve relevant data from an external knowledge base, format it as a text string and return it. The text string is the intermediate answer, thus obtaining the reasoning chain of "sub-problem - retrieval - intermediate answer";

[0047] Step 5: Concatenate the returned text string to the data retrieval request and use it as the starting point of the next inference chain;

[0048] Step 6: Repeat steps 2 to 5 to obtain the reasoning chains corresponding to all sub-problems;

[0049] Step 7: Based on all the reasoning chains, obtain the final output answer.

[0050] This application innovatively combines model fine-tuning with retrieval enhancement. Its core idea is not to have the model "memorize" new data, but rather to "teach" a large language model a new "ability" to dynamically interact and reason with external data sources by fine-tuning it on a carefully designed domain-specific dataset. This ability transforms the model from a passive language processor into a processor capable of proactively planning data acquisition and dynamically analyzing data. This ability is specifically manifested in two interrelated core mechanisms, illustrated below using "financial credit review" as an example:

[0051] Progressive Retrieval Capability: This capability aims to address the insufficient data supply problem in complex multi-hop query scenarios. After training with this method, the model can autonomously and intelligently decompose a complex credit review problem that requires multi-step data analysis into a series of logically continuous and simpler data query subtasks. Furthermore, during the analysis of each subtask, the model can proactively and iteratively generate standardized data retrieval requests, retrieving the most needed information for the current step (such as credit reports, bank statements, and legal litigation information) from external knowledge bases. This step-by-step, modular data retrieval approach ensures accurate data supply at every stage of the complex analysis chain, fundamentally overcoming the limitations of traditional single-retrieval methods in multi-hop queries.

[0052] Dynamic Knowledge Utilization (DKU): This capability aims to resolve data source conflicts and ensure the accuracy of data updates. After training with this method, the model can flexibly and effectively utilize retrieved external data. Specifically, the model first determines the relevance of the retrieved information to the current (sub)question. When the information is highly relevant (i.e., the question data exists in the external database), the model prioritizes generating answers based on the newly retrieved data, effectively adopting the latest external data. When the information is irrelevant or has low relevance (i.e., the question data does not exist in the external database), the model intelligently suppresses interference from the retrieved information and instead relies on its inherent, unedited static data representation to generate answers. This mechanism ensures the "locality" of data updates, affecting only relevant data and not disrupting irrelevant data, thus effectively resolving data conflicts and maintaining the overall performance stability of the model.

[0053] Accordingly, this application also provides an intelligent system for implementing the above method, which includes at least: a large language model fine-tuned by the method of this application as an analysis and reasoning engine; an external knowledge base that can be dynamically read and written for storing real-time and background data in the domain; an efficient knowledge retrieval unit responsible for executing data retrieval instructions issued by the model; and a central control module responsible for coordinating the above components to fully execute the model fine-tuning training process and subsequent adaptive reasoning process.

[0054] Finally, this application further provides a specific implementation of applying the above-mentioned method and system to the field of financial credit review, namely, an intelligent approval and auxiliary decision-making system that can interact with credit reviewers in natural language. This system can respond to complex queries from credit reviewers in real time and accurately, and provide reliable risk assessment, approval suggestions and decision support.

[0055] System Overall Architecture

[0056] Reference Figure 1 The intelligent system proposed in this application mainly includes four core components that logically or physically cooperate with each other:

[0057] Large-scale language model: serving as the core of the system's cognition and reasoning. Theoretically, any large-scale language model with sound fundamental capabilities can serve as the base model for this application, such as the Llama and Mistral series from the open-source community, or commercial closed-source models like the GPT series. In this embodiment, the Llama-2-7B model can be selected as the base. After fine-tuning during the "capability acquisition" phase of this application, this model will be transformed from a general-purpose language model into a specialized model with dynamic domain knowledge processing capabilities.

[0058] External Editing Knowledge Base: This is a unified vector knowledge base independent of the LLM. Its core function is to transform all dynamically changing domain knowledge, regardless of its original format, into a standardized vector form that allows for efficient semantic retrieval by the model for storage and management. In a specific implementation of the financial credit review application, the data in this knowledge base originates from a comprehensive data platform that integrates multiple data sources, but ultimately all data is injected into the knowledge base through a standardized "natural language-vectorization" process, as follows:

[0059] Data sources and transformation process:

[0060] Processing of unstructured documents: For a large amount of text materials, such as banks' "Credit Approval Policy", "Anti-Money Laundering Operation Manual", and various financial product manuals, the system will first divide them into blocks (e.g., by paragraph or chapter), and then use an efficient text embedding model to convert each text block into a high-dimensional vector and store it in the vector knowledge base.

[0061] Processing of structured and semi-structured data: For customer files, credit reports, loan history, court default records, etc., from the core business system, the system first converts them into "natural language". For example, a credit record (Applicant ID: 3301...; Number of overdue payments: 3; Number of inquiries (last 3 months): 8) is converted into descriptive text: "Applicant 3301... has 3 historical overdue payments and 8 credit inquiries in the last 3 months." Then, this descriptive text is converted into a vector for storage.

[0062] Processing of high-frequency real-time data: For data streams from the transaction system, such as large transfers and abnormal credit card transactions, the system uses the same "natural language-vectorization" process for near real-time processing. For example, a transaction record (Account: 6222...: Time: 14:30:05: Type: Overseas cardless transaction: Amount: 5000 USD) will be converted into the text: "At 14:30:05, account 6222... made an overseas cardless transaction of 5000 USD." This text is then quickly vectorized and updated in the knowledge base.

[0063] Technical advantages:

[0064] The core advantage of this unified vectorization approach lies in:

[0065] Unified retrieval interface: Regardless of the original form of knowledge, LLM can be queried through a unified semantic retrieval interface, which greatly simplifies the design complexity of the system.

[0066] Deep semantic understanding: It enables models to transcend the structural boundaries of data and make deep semantic connections. For example, a model can match a fault phenomenon in natural language with a device state vector converted from real-time data to discover potential relationships.

[0067] Lightweight and Timely: Crucially, this vectorization process is executed by an efficient embedding model with minimal computational overhead, enabling near real-time processing of new and changed data without becoming a system bottleneck. This ensures that the external knowledge base keeps pace with real-world changes, providing the upper-level LLM with the latest and most accurate knowledge support.

[0068] Knowledge Retrieval Unit: This component acts as a bridge between the LLM and external knowledge bases. It receives sub-questions (natural language queries) generated by the LLM during the reasoning process and is responsible for translating them into actual query instructions for the external knowledge base. It then retrieves the most relevant knowledge fragments from the knowledge base and returns them to the LLM. The implementation of the retrieval unit can vary; for example, a "Text-to-SQL" module can be used for structured data, while a dense vector-based retrieval model, such as Contriever or BGE, can be used for unstructured documents.

[0069] The control module, acting as the system's "commander," is responsible for orchestrating and managing the entire workflow. Its functions cover the two core phases of this application: the capability acquisition phase and the adaptive inference phase. In the capability acquisition phase, the control module is responsible for loading the fine-tuning dataset, configuring training parameters, and performing the fine-tuning process of the LLM. In the adaptive inference phase, the control module is responsible for receiving user input, driving the LLM to perform inference, and coordinating the interaction between the LLM and the knowledge retrieval unit (i.e., capturing retrieval triggers, sending queries, receiving and concatenating the returned knowledge).

[0070] Detailed Explanation of Core Methods

[0071] The core of this application lies in endowing LLM with the two key capabilities mentioned above through a unique training paradigm. The implementation process is described in detail below.

[0072] Phase 1: Capability Acquisition (through data processing and model fine-tuning)

[0073] This stage is a one-time, offline training process. Its goal is not simply to make the LLM memorize knowledge itself, but to enable it to master a set of general capabilities for dynamic interaction with external knowledge bases. Specifically, through fine-tuning, the model learns precisely when to invoke retrieval (generating retrieval triggers), how to invoke retrieval (autonomously generating precise sub-questions), and how to dynamically utilize retrieval results to generate answers based on relevance.

[0074] Construction of a high-quality fine-tuning dataset:

[0075] The quality of the dataset directly determines the upper limit of the model's capabilities. The construction of the dataset should closely revolve around the two main goals of "incremental retrieval" and "dynamic knowledge utilization".

[0076] Progressive retrieval data construction:

[0077] The goal of this data section is to train the model to decompose complex problems into subtasks and proactively retrieve the necessary information for each subtask. The data construction process follows a "case decomposition - format conversion" workflow. (Reference) Figure 2 The specific construction process is as follows.

[0078] First, collect typical multi-hop complex problem cases from actual business in the field of financial credit review, such as cases from historical difficult credit approval reports, post-loan risk investigation logs, or credit reviewer training manuals.

[0079] Subsequently, domain experts broke down each complex problem, simulating its solution process as a logically clear chain of reasoning. This chain consists of multiple interconnected units of "sub-problem - retrieval - intermediate answer".

[0080] The problem to be solved is broken down into multiple sub-problems. For each sub-problem, firstly, typical multi-hop complex problem cases are collected from actual business operations in the financial credit review field, such as cases from historical difficult credit approval reports, post-loan risk screening logs, or credit reviewer training manuals. Then, using a teacher-student model-based paradigm, a large-scale language model with excellent reasoning capabilities is employed, combined with mind chain hinting engineering technology, to automatically generate reasoning paths for each complex problem. This process simulates complex approval decisions as a formatted reasoning chain consisting of multiple "sub-problem-retrieval-intermediate answer" units.

[0081] Specifically, each unit of the chain operates as follows:

[0082] Sub-problem generation: When performing autoregressive inference, the model identifies the information gaps at the current logical node based on the generated text sequence and actively generates a sub-problem that is encapsulated in a specific format and requires external knowledge to solve.

[0083] "Retrieval" Execution: The system employs different technical paths to perform the retrieval based on the intent of the sub-question and the type of external knowledge base.

[0084] For unstructured data (such as policy documents and research reports): Leading text embedding models (such as BGE, M3E, etc.) are used to transform the "sub-problem" text into high-dimensional vectors. Subsequently, in a vector index built by an efficient approximate nearest neighbor search library (such as FAISS or ScaNN), maximum inner product or cosine similarity search is performed to recall the most relevant text fragments in milliseconds.

[0085] For structured data (such as customer profiles and transaction records in a database): Launch a Text-to-SQL conversion module that dynamically compiles the "sub-questions" in natural language form into SQL query statements that can be executed directly in a relational database and obtain accurate query results.

[0086] The "intermediate answer" is obtained by dynamically concatenating the text fragments or formatted data query results returned by the above retrieval steps into the model's current input sequence through context injection. Based on this enhanced context, the model continues reasoning, generating a logically coherent analytical text that integrates new and old information. This text serves as the "intermediate answer" to solving the current sub-problem and as the starting point for the next round of reasoning.

[0087] Finally, this complete reasoning chain is converted into training samples in a specific format. Taking a specific financial scenario as an example, for the original question: "Assess the risk of a 500,000 yuan business loan applied for by individual business owner Li Si, and provide approval recommendations," experts analyze that his reasoning process might include: 1. Checking personal credit history; 2. Checking business entity information; 3. Analyzing bank statements; 4. Comparing with industry policies. Based on this, the constructed training samples are as follows:

[0088] "Assess the risks of the 500,000 yuan business loan applied for by individual business owner Li Si." <sep>[Retrieve (Check Li Si's personal credit report) → The credit report shows no overdue payments, but there are 5 loan approval inquiry records in the past six months.] Applicant Li Si has a good personal credit record, but his recent credit demand is active. [Retrieve (Check the business registration and litigation information of "Li Ji Business" under Li Si's name) → The business is operating normally and has no litigation records.] The business entity's qualifications are normal. [Retrieve (Check Li Ji Business's corporate bank statements and Li Si's personal bank statements for the past year) → The corporate bank statement shows an average monthly bank statement of about 200,000 yuan, but the personal bank statement shows a fixed large amount of funds transferred to a third-party account every month.] The corporate bank statement meets the requirements, but the applicant's personal bank statement shows suspicious expenditures, which may conceal other liabilities. [Retrieve (Check the "High-Risk Behavior" definition clause of our bank's micro and small enterprise business loan) → Policy 3.1.4 stipulates that frequent credit inquiries and unexplained large expenditures are considered high risk.] Based on the overall assessment, the applicant may have undeclared hidden liabilities, which poses a high risk. The final answer is: it is recommended to reject the loan application, or require the applicant to provide additional collateral and a reasonable explanation for the whereabouts of large sums of money.

[0089] In this format, <sep>[ is a separator used to distinguish between the original question and the reasoning chain; [Retrieve(...)] is a wrapper for the retrieval instructions that the model needs to learn to generate; → is a special marker used to separate the retrieval query from the retrieval results returned from the external knowledge base. By training on a large number of such formatted samples, the model can learn this dynamic data retrieval capability of "breaking down the whole into parts and proceeding step by step".

[0090] Dynamic knowledge is built using data:

[0091] The goal of this data section is to train the model to master the ability to "identify information relevance" and "resolve data conflicts." Its construction process revolves around a "data fact" that needs updating, and by constructing different types of question-answer pairs, it teaches the model how to appropriately utilize retrieved external data in different scenarios.

[0092] Taking a revision of a financial credit policy as an example, suppose the data that needs to be updated is: "(Old data) The maximum loan-to-value ratio for our bank's 'Housing Loan' product was 70% → (New data) According to the latest policy, the maximum loan-to-value ratio for the 'Housing Loan' product has been reduced to 65%." Based on this data, and referring to... Figure 3 Construct the following three types of training samples:

[0093] For the unresolved issue of financial credit verification, the reasoning chain is obtained through the large language model trained above.

[0094] For the intermediate answer in this reasoning chain, determine whether the answer has been updated. If it has been updated, update the intermediate answer; otherwise, do not update it.

[0095] Target sample: Used to train the model by directly applying new data. This sample pairs a question that directly tests the data point with an answer that must rely on new data to generate. For example:

[0096] Q: "What is the maximum loan amount available for housing loans now?"

[0097] A: "[Retrieve (check the 'Home Loan' product policy) → The latest policy document shows that the maximum loan-to-value ratio is 65%] According to the latest policy, the maximum loan-to-value ratio for the 'Home Loan' product is 65%."

[0098] In-scope sample questions: These are used to train the model to apply new data to related reasoning tasks, testing its knowledge transfer capabilities. This sample pairs a question that indirectly depends on the data point with an answer calculated or reasoned based on the new data. For example:

[0099] Q: "If a customer wants to use a 'housing loan' to buy a house worth 1 million, what is the maximum amount they can borrow?"

[0100] A: "[Retrieve (check the 'Housing Loan' product policy) → The latest policy document shows that the maximum loan-to-value ratio is 65%.] According to the latest 'Housing Loan' policy, a house worth 1 million can be loaned a maximum of 650,000 yuan."

[0101] Out-of-scope questions: These are used to train a model's ability to maintain its knowledge stability when external data is not applicable. This sample pairs a question unrelated to the facts of the data with an answer that indicates the model cannot answer or relies on its internal knowledge. For example:

[0102] Q: "Please predict whether the number of applicants for the 'Housing Loan' product will increase or decrease next year?"

[0103] A: "[Retrieve (predict the number of applicants for 'Housing Loan' next year) → Search failed, the knowledge base does not contain prediction function or related data] I'm sorry, I cannot predict the market application volume. My knowledge is mainly focused on querying existing product policies and customer data."

[0104] By training the model on the three types of samples mentioned above, the model can learn to dynamically decide whether to adopt new external data or rely on existing internal knowledge based on the relevance of the retrieved data to the problem, thereby achieving accurate and flexible utilization of dynamic data.

[0105] Innovative model fine-tuning paradigm:

[0106] Reference Figure 4 The training paradigm of this application differs fundamentally from traditional parameterized injection methods. Parameterized methods ( Figure 4 The first part calculates a parameter increment for each new piece of knowledge to update the model, essentially "giving a man a fish." The method in this application (…) Figure 4 The second half involves fine-tuning the model in one go using a carefully designed general dataset D, teaching it the general ability to interact with external knowledge bases, which is "teaching a man to fish".

[0107] Fine-tuning process:

[0108] Using the hybrid dataset constructed above, the LLM is trained through an efficient parameter fine-tuning technique (PEFT), such as LoRA (Low-Rank Adaptation). The LoRA rank (r) can be set to 8, alpha to 16, and dropout to 0.05, and it is applied only to the model's q_proj and v_proj matrices.

[0109] The most critical innovation during training lies in the way the loss function is calculated. For each training sample, when calculating the cross-entropy loss, it is necessary to mask the knowledge text portion provided by the retrieval machine in the input sequence (i.e., all tokens between the → symbol and the ] symbol).

[0110] The specific definition of the loss function:

[0111] The loss function used in this application is the cross-entropy loss function of causal language models. Its core idea is that at each position t in the sequence, the model outputs a probability distribution covering the entire vocabulary. This distribution represents the probability that the model predicts the next word to be any word in the vocabulary. The loss function measures the difference between this predicted probability distribution and the actual next word. The gap between them.

[0112] Specifically, for a training sequence containing N tokens The total loss L is calculated as follows:

[0113]

[0114] in: It is the actual token of the sequence at position t. The model predicts the t-th token after seeing the previous t-1 tokens. The probability of. This is a loss mask with a value of either 1 or 0. If it is 1, the loss at position t is included in the total loss; if it is 0, the loss at that position is ignored. This mask... This is the core of achieving the "blocking" operation.

[0115] This avoids the model learning to generate text that should be retrieved through retrieval, preventing interference with model training. This design forces the model to truly "understand" and "digest" the retrieved knowledge and learn how to use this knowledge to generate logically coherent and accurate subsequent text (i.e., intermediate or final answers). This mechanism ensures that the model's training objectives are highly consistent with its actual reasoning usage, effectively preventing the model from learning opportunistic "shortcut" behaviors and guaranteeing the generalization and robustness of the acquired capabilities.

[0116] Phase Two: Adaptive Reasoning (Workflow in Practical Applications)

[0117] Once the system's analysis and inference engine (i.e., the finely tuned large language model) has acquired its capabilities, the system can be deployed in real-world applications to execute online, dynamic, and adaptive data processing and inference workflows. This workflow is centrally coordinated by a control module, and the specific steps are as follows:

[0118] Query Reception and Initialization: The central control module receives complex queries in natural language form from users (e.g., financial credit auditors) through the interactive interface. For example: "Review Wang Wu's credit card limit increase application. He made several large overseas purchases last month. Please assess the risk." The control module uses this query as initial input and passes it to the analysis and inference engine to initiate the data processing sequence.

[0119] Dynamic Data Request Generation: Based on its trained "progressive data retrieval" capability, the analysis and inference engine analyzes the current input, generates preliminary analysis text, and autonomously identifies the first step of key data required to solve the problem. Subsequently, the engine generates a standardized data retrieval request (→), which encapsulates specific sub-problems. For example, the engine's output might be: "Received, analyzing Wang Wu's credit card limit increase application. First, we need to verify his current credit status and repayment ability. [Retrieve(Query Wang Wu's personal credit report and asset information in our bank) →" (Standardized data retrieval request).

[0120] Data Retrieval and Supply: The central control module monitors the engine's output stream in real time. When a predefined retrieval trigger marker (→) is detected, the module immediately pauses the engine's generation process. It precisely extracts the sub-question from the data retrieval request (i.e., "Query Wang Wu's personal credit report and asset information at our bank") from the engine's output and forwards it to the knowledge retrieval unit. Based on this sub-question, the knowledge retrieval unit queries and retrieves relevant data from external knowledge bases (such as credit databases and the bank's customer relationship management system) and formats it into a text string for return. For example, the returned data might be: "Retrieval results: No overdue payments on credit report, debt ratio 35%. 500,000 yuan in fixed deposits at our bank."

[0121] Data Integration and Iterative Analysis: The central control module appends the data string returned by the knowledge retrieval machine to the current engine's generated sequence (→) and resumes engine operation. At this point, the input to the analysis and reasoning engine is updated to include the complete context of the newly acquired data. Based on its "dynamic data utilization" capability acquired through training, the engine integrates and analyzes the new data, generates intermediate conclusions for the previous sub-problem, and continues with the next round of reasoning. If more data is needed to solve the entire problem, the engine will generate a new data retrieval request. For example, the engine might continue to generate: "The applicant has a good asset situation and sufficient repayment ability. Next, it is necessary to analyze whether there is a risk of money laundering or fraud in their overseas consumption behavior. [Retrieve (query Wang Wu's consumption details for last month and our bank's list of high-risk anti-money laundering merchants) →]".

[0122] Process Loop and Termination: The system iteratively executes steps 3 and 4 in a pattern of "generating data requirements → retrieving data → integrating and analyzing." This iterative process continues until the analysis and inference engine determines that it has obtained all the data needed to solve the original complex query and no longer generates new data retrieval requests. At this point, the engine generates a logically rigorous and clearly structured final conclusion based on a complete, multi-source data evidence chain. Inference Ends.

[0123] Results Presentation: The central control module formats the final conclusions output by the analysis and inference engine and presents them to the user. For example, the final report might read: "Comprehensive Analysis Conclusion: Applicant Wang Wu has strong assets and a good credit record. His overseas spending occurred at well-known chain hotels and department stores, which are not considered high-risk merchants. The risk is controllable. Recommended steps: 1. Approve his credit limit increase application to XX yuan. 2. Mark his recent spending patterns as normal, without triggering a high-risk alert." This complete analysis report, based on multi-source real-time data, provides strong technical support for the credit reviewer's final decision.

[0124]

[0125] Table 1. Detailed performance comparison data of the proposed method and several baseline methods on three single-hop knowledge editing datasets.

[0126]

[0127] Table 2. Performance comparison of the proposed method with several baseline methods on two multi-hop knowledge editing datasets.

[0128] Significantly Improved Accuracy and Reliability of Knowledge Editing: This application directly addresses the core challenges of knowledge editing and achieves outstanding results. Referring to Table 1, in single-hop tasks that measure the fundamental performance of knowledge editing, the method in this application far surpasses existing techniques in overall performance on three mainstream datasets (ZsRE, Recent, Counterfact). For example, in the two key metrics of portability (measuring whether the model can generalize new knowledge to related problems) and locality (measuring whether the model can edit without affecting irrelevant knowledge), this application achieves average scores of 81.5% and 81.3%, respectively, while traditional parameterized methods such as ROME only reach 40-50%. More importantly, in multi-hop tasks that directly affect the feasibility of complex applications, referring to Table 2, the advantages of this application are even more significant. On datasets such as MQUAKE, when the editing batch increases, the performance of traditional methods drops sharply to single digits, while the method in this application still maintains a high level of over 50%, with an average accuracy improvement of up to 25.3%. In the financial field, where risk control is extremely demanding, the reliability gains resulting from improved accuracy are the cornerstone for ensuring sound bank operations and reducing the risk of bad debts.

[0129] In practical applications, knowledge updates are not always one-off events. To comprehensively evaluate the capabilities of a knowledge editing method, it is necessary to examine its performance in different editing scenarios. We mainly distinguish the following three editing modes:

[0130] Single-point editing: refers to updating only one piece of knowledge in the model at a time.

[0131] Sequential editing: refers to editing single points continuously, one after another. This tests the stability of the model in an environment that continuously receives new information.

[0132] Batch editing: refers to injecting multiple pieces of new knowledge into the model at the same time.

[0133] A fatal weakness of digital editing methods lies precisely in their extremely limited sequential and batch editing capabilities; their performance deteriorates sharply with the increase in the number of edits or the amount of editing. (See reference...) Figure 3 The figure clearly illustrates the precipitous drop in the success rate, portability, and locality of methods such as FT-L, ROME, and MEMIT during sequential editing (from 1 to 1000 times). Each parameter modification is a perturbation to the model weights; when these perturbations accumulate, severe interference arises between different editing operations, ultimately destroying the model's original knowledge structure. This demonstrates their inability to handle large-scale, high-frequency knowledge updates.

[0134] The performance curve of this application (PRAE) is... Figure 5 The model remains almost perfectly horizontal, exhibiting exceptional stability. This is because this application does not modify the core parameters of the model; the addition of knowledge merely expands the external knowledge base without disturbing the model itself. Whether it's single-point editing, sequential editing, or batch editing, the essence of this application's method remains the same: during inference, retrieval is performed from a continuously expanding external knowledge base. Therefore, temporary adjustments to regulatory policies, iterations of risk models, and the launch of new financial products can all be injected into the system in real time at near-zero cost. This superior scalability allows it to perfectly handle the dynamic environment of rapidly changing financial market knowledge, ensuring that every decision is based on sound reasoning.

[0135] Powerful Deep Reasoning Capabilities for Complex Problems: Through an innovative "progressive retrieval" mechanism, this application endows large-scale language models with unprecedented deep reasoning capabilities. It no longer simply performs one-time information retrieval based on surface keywords, but rather, like a human domain expert, logically decomposes complex problems, forms reasoning paths, and proactively and precisely collects evidence for each step. This capability enables it to demonstrate an intelligence level far exceeding traditional RAG methods when handling complex scenarios such as credit risk assessment, fraud detection, and post-loan monitoring. The answers provided are no longer fragmented information piles, but logically rigorous and clearly structured analytical reports.

[0136]

[0137] Table 3 Comparison of average time consumption at each stage when editing a single knowledge point using the method of this application and various existing methods.

[0138] Extremely high editing efficiency and application economy: Referring to Table 3, this application has an overwhelming advantage in efficiency. For the "editing" of a new knowledge point, parameterization methods such as ROME and MEMIT require several seconds or even longer "editing time" to calculate and apply parameter updates. In contrast, the "editing time" of the method in this application is zero because the knowledge update only occurs in an external knowledge base, which is imperceptible to the model. Its online inference overhead mainly lies in retrieval time, which is typically in the millisecond range. This near-instantaneous knowledge update capability, combined with a mode that allows for long-term use after a one-time fine-tuning, greatly reduces maintenance costs and computational overhead throughout the model's lifecycle, making large-scale deployment in actual financial business possible and demonstrating significant economic advantages.

[0139] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solution of the present invention and should not be used to limit the scope of protection. Any modifications made in accordance with the claims and specification of the present invention that are only partial should still fall within the protection scope of the present invention.< / sep> < / sep>

Claims

1. A method of dynamic knowledge Q&A with progressive retrieval augmentation, characterized by Includes the following steps: The system receives complex queries in natural language input from the user and inputs them into a trained LLM model to obtain the output answer. The trained LLM model specifically performs the following steps: Step 1: Break down the complex problem into multiple sub-problems; Step 2: For the first sub-problem, generate a standardized data retrieval request; Step 3: When a retrieval trigger marker is detected in the data retrieval request, extract the sub-question from the data retrieval request; Step 4: Based on the sub-problem, query and retrieve relevant data from an external knowledge base, format it as a text string and return it. The text string is the intermediate answer, thus obtaining the reasoning chain of "sub-problem - retrieval - intermediate answer"; Step 5: Concatenate the returned text string to the data retrieval request and use it as the starting point of the next inference chain; Step 6: Repeat steps 2 to 5 to obtain the reasoning chains corresponding to all sub-problems; Step 7: Based on all the reasoning chains, obtain the final output answer; Step 4 also includes a step to determine whether the intermediate answer has been updated, specifically: The trained LLM model determines whether the intermediate answer has been updated. If it has, the intermediate answer is updated; otherwise, it is not updated. The model training process is as follows: Construct training samples and train an LLM model using the training samples. The training samples include direct update samples and indirect update samples. The direct update samples are target problem samples, and the indirect update samples are in-range problem samples and out-of-range problem samples. The target question sample is a question with a direct test data point, paired with an answer that must rely on new data to be generated. The problem sample within the specified range is a problem that indirectly depends on the data point, which is paired with an answer that is calculated or reasoned based on new data. The out-of-range question sample is a question unrelated to the data point, paired with an answer that indicates it cannot be answered or relies on internal model knowledge. The loss function of the LLM model is expressed as: where, is the true token at position t for the sequence, is the probability that the model predicts the tth token to be after seeing the first t-1 tokens, is the loss mask, and N is the total number of tokens.

2. The method of claim 1, wherein the progressive retrieval enhanced dynamic knowledge Q&A method is characterized by Step 1 specifically involves: For complex problems, autoregressive reasoning is used to identify information gaps at the current logical nodes in the text sequence, thereby determining a sub-problem.

3. The method of claim 1, wherein the progressive retrieval of the enhanced dynamic knowledge Q&A is based on a user's query. The specific steps for querying and obtaining relevant data from an external knowledge base in step 4 are as follows: Step 41: Determine whether the subproblem is structured data. If yes, proceed to step 42; otherwise, proceed to step 43. Step 42: Start a Text-to-SQL conversion module, which dynamically compiles the sub-problems in natural language form into SQL query statements that can be directly executed in a relational database and obtain accurate query results; Step 43: Use a text embedding model to transform the text of the sub-problem into a high-dimensional vector. Then, in a vector index built by an efficient approximate nearest neighbor search library, perform a maximum inner product or cosine similarity search to recall the most relevant text fragments as query results in milliseconds.

4. The method of claim 3, wherein the progressive retrieval enhancement is based on a user's search history. The structured data includes customer profiles and transaction records, while the unstructured data includes policy documents and research reports.

5. The method of claim 1, wherein the progressive retrieval enhanced dynamic question answering method is characterized by, The trained LLM model is fine-tuned using a low-rank adaptation method, and the update matrix of the low-rank adaptation is applied to the query and value projection layers in the model's attention mechanism.

6. The method of claim 3, wherein the progressive retrieval of the enhanced dynamic knowledge Q&A is based on a user's query. The text embedding model is BGE or M3E, and the efficient approximate nearest neighbor search library is FAISS or ScaNN.

7. The method of claim 1, wherein the progressive retrieval of enhanced dynamic knowledge Q&A is based on a user's query. The LLM model is either the Llama series, the Mistral series, or the GPT series.

8. The progressive retrieval-enhanced dynamic knowledge question answering method according to claim 1, characterized in that... The LLM model is the Llama-2-7B model.

Images