Large language model reasoning enhancement method and system based on logic chain verification
By constructing a logical chain and distinguishing between key and secondary nodes, and calculating evaluation scores based on historical evaluation data, the problem of misjudgment in complex reasoning tasks of large language models is solved. This enables in-depth verification and self-correction of the logical chain, thereby improving the model's reasoning accuracy and reliability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- KAIWU DIGITAL INTELLIGENCE (SHANGHAI) ARTIFICIAL INTELLIGENCE TECHNOLOGY CO LTD
- Filing Date
- 2026-01-26
- Publication Date
- 2026-04-24
AI Technical Summary
The reasoning ability of existing large language models relies on static judgment criteria, which cannot adapt to the complexity of different tasks and changes in context. This leads to erroneous reasoning when dealing with novel or boundary problems. Furthermore, the reasoning process is separated from the verification process, making it impossible to delve into the logical chain construction process inside the model, resulting in misjudgments and the inability to self-correct.
By collecting historical question-and-answer pairs, progressive question-and-answer pairs are constructed and logical chains are built. Key nodes and secondary nodes are distinguished. Evaluation scores are calculated based on the historical evaluation data of the nodes, and the scores are compared and verified before output, thus establishing an internal logical chain verification mechanism.
It enables in-depth verification of logical chains, quantitatively evaluates the credibility of logical chains, reduces the risk of erroneous reasoning, and enhances the accuracy and reliability of large language models in complex reasoning tasks.
Smart Images
Figure CN121920536A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the technical field of large language model reasoning, specifically relating to a method and system for enhancing large language model reasoning based on logical chain verification. Background Technology
[0002] As a driving force in the field of natural language processing, large language models have been widely used in many scenarios such as information retrieval and human-computer interaction. However, the existing knowledge question answering capabilities of large language models cannot meet the needs of practical applications. Improving the logical reasoning ability when facing complex problems is the core issue that needs to be addressed.
[0003] Existing large language models rely on static, pre-defined judgment criteria for reasoning ability, which cannot adapt to the complexity and context of different tasks. This leads to situations where, when dealing with novel or boundary problems, correct but low-confidence reasoning paths are incorrectly rejected, or incorrect but high-confidence reasoning paths are incorrectly accepted.
[0004] Furthermore, the reasoning and verification processes in existing technologies are separated. The verification module acts as a post-processing filter, performing a superficial review of the final results generated by the model. This separate architecture prevents the verification mechanism from delving into the logical chain construction process within the model, often leading to misjudgments. Moreover, it cannot provide effective guidance for correction after errors are discovered, resulting in the model's inability to form a closed-loop self-correction and optimization capability.
[0005] In view of this, the present invention proposes a method and system for enhancing reasoning in large language models based on logical chain verification. Summary of the Invention
[0006] The purpose of this invention is to provide a method and system for enhancing large language model reasoning based on logical chain verification, which can judge the results of logical chain reasoning and correct inaccurate output content in a timely manner.
[0007] The technical solution adopted by this invention is as follows: A large language model reasoning enhancement method based on logical chain verification, which executes: collecting a set of historical question-answer pairs, wherein each historical question-answer pair includes the question content and the response information;
[0008] Based on whether the question content of each historical question-and-answer pair is based on the response information of the previous historical question-and-answer pair, historical question-and-answer pairs are classified into progressive question-and-answer pairs.
[0009] Multiple logical chains are constructed based on progressive question-and-answer pairs; based on the distribution of each node in the multiple logical chains, the nodes are divided into key nodes and secondary nodes, and a weight value is assigned to each node; based on the historical evaluation data associated with the nodes, the evaluation scores of key nodes and secondary nodes are calculated.
[0010] When a query to be verified is received, the following steps are executed: Construct a logic chain to be verified in response to the query.
[0011] Generate a reasoning answer based on the logic chain to be verified; calculate the verification score of the logic chain to be verified; compare the verification score with a preset verification threshold;
[0012] Based on the comparison results, when the verification score is greater than or equal to the preset verification threshold, the reasoning answer is output; when the verification score is less than the preset verification threshold, the query content to be verified is marked as pending review.
[0013] Preferably, constructing multiple logical chains based on progressive question-answer pairs includes: selecting a starting question-answer pair from the progressive question-answer pairs as the starting node of the logical chain;
[0014] Using the response information of the starting node as the retrieval content, iteratively search for subsequent question-answer pairs in the progressive question-answer pairs, where the query content of the subsequent question-answer pair matches the response information of the previous node; and arrange the multiple nodes obtained through iterative search in the search order to construct a logical chain.
[0015] Preferably, classifying nodes into critical nodes and secondary nodes includes: if a node exists in every logical chain of multiple logical chains, then the node is identified as a critical node;
[0016] If a node exists only in a portion of multiple logical chains, then the node is identified as a minor node.
[0017] Preferably, calculating the evaluation scores of key nodes and secondary nodes includes: for each node, calculating an initial score based on the positive feedback count and negative feedback count in its historical evaluation data;
[0018] The initial scores of all key nodes are aggregated to obtain the key node evaluation score; and the initial scores of all secondary nodes are aggregated to obtain the secondary node evaluation score.
[0019] Based on the statistical differences between the initial score sets of all key nodes and the initial score sets of all minor nodes, a first preset weight coefficient and a second preset weight coefficient are determined for calculating the verification score.
[0020] Preferably, generating the reasoning answer based on the logic chain to be verified includes: generating the reasoning answer based on each node in the logic chain to be verified and the weight value associated with each node.
[0021] Preferably, calculating the verification score of the logic chain to be verified includes: identifying key nodes and secondary nodes in the logic chain to be verified; and obtaining the evaluation scores of key nodes and secondary nodes.
[0022] The evaluation scores of key nodes are multiplied by the first preset weighting coefficient, and the evaluation scores of secondary nodes are multiplied by the second preset weighting coefficient. The results are then weighted and summed to obtain the verification score.
[0023] Preferably, the collection of historical question-and-answer pairs includes: communicating with the user through at least one interactive entity deployed on a smart terminal, and capturing the user's posted questions and the response information generated in response to the questions, to form historical question-and-answer pairs.
[0024] A large language model reasoning enhancement system based on logical chain verification includes:
[0025] The logical model building module is used to collect a set of historical question-and-answer pairs, classify the historical question-and-answer pairs into progressive question-and-answer pairs, build multiple logical chains containing key nodes and secondary nodes based on the progressive question-and-answer pairs, and determine the evaluation scores and weight values associated with key nodes and secondary nodes.
[0026] The reasoning and verification module is used to construct a logic chain to be verified in response to the query content to be verified when it receives the query content to be verified. Based on the nodes in the logic chain to be verified and the evaluation scores and weight values determined by the logic model construction module, it generates a reasoning answer and calculates the verification score.
[0027] The result processing module compares the verification score calculated by the reasoning verification module with the preset verification threshold. When the verification score is greater than or equal to the preset verification threshold, the reasoning answer is output. When the verification score is less than the preset verification threshold, the query content to be verified is marked as pending review.
[0028] Preferably, the logic model building module is specifically used to: iteratively find progressive question-answer pairs that match the query content with the response information of the previous node to build multiple logic chains; and classify the nodes into key nodes and secondary nodes according to the distribution of the nodes in the multiple logic chains.
[0029] The initial score is calculated based on the positive and negative feedback counts in the historical evaluation data of each node, and the initial scores of all key nodes and all secondary nodes are aggregated to obtain the evaluation scores of key nodes and secondary nodes.
[0030] Preferably, the reasoning verification module is specifically used to: generate a reasoning answer based on each node in the logic chain to be verified and the weight values determined by the logic model construction module;
[0031] The verification score is calculated by weighted summing of the evaluation scores of the critical nodes and the evaluation scores of the secondary nodes in the logic chain to be verified.
[0032] Beneficial effects
[0033] 1. This invention classifies historical question-and-answer pairs into progressive and parallel question-and-answer pairs based on whether the query content of the historical question-and-answer pairs is based on the response information of previous historical question-and-answer pairs, and stores them accordingly. By selecting nodes from the progressive database, matching the response information of the previous node with the query content of the next node, a logical chain composed of multiple progressive question-and-answer pairs arranged in sequence is iteratively constructed. The logical chain is constructed using progressive question-and-answer pairs with inherent successive relationships. The reasoning process of constructing a logical chain can simulate and restore a rigorous step and coherent context, overcoming the problem in the prior art that the logical break or weak correlation between nodes is caused by the disordered data organization.
[0034] 2. This invention constructs multiple logical chains and, based on the distribution of each node across these chains, classifies nodes into critical nodes and secondary nodes. Specifically, nodes present in every logical chain are identified as critical nodes, while nodes present in some logical chains are identified as secondary nodes. Weight values are then assigned to each node to quantify the structural importance of different nodes within the logical chains. Critical nodes represent indispensable core parts of the reasoning process, while secondary nodes reflect optional or auxiliary reasoning branches. This serves as the basis for subsequent evaluation of the logical chains, avoiding evaluation bias caused by treating all nodes equally.
[0035] 3. Based on the historical evaluation data of nodes, this invention calculates the evaluation scores of key nodes and secondary nodes, and performs a weighted summation using a first preset weight coefficient and a second preset weight coefficient to obtain the final verification score. Before outputting the reasoning answer, the verification score is compared with a preset verification threshold. When the verification score is not less than the preset verification threshold, the reasoning answer is determined as a valid answer and output. Through this verification mechanism, a dual verification system that takes into account both the importance of node structure and the reliability of historical performance is established. This system quantifies the overall credibility of the logical chain, intercepts and filters reasoning answers with low credibility, and reduces the risk of inaccurate or fictitious content generated by the large language model. Attached Figure Description
[0036] Figure 1 This is a flowchart of the method of the present invention;
[0037] Figure 2 This is a system framework diagram of the present invention. Detailed Implementation
[0038] The technical solution of this patent will be further described in detail below with reference to specific embodiments. The following embodiments are used to illustrate the present invention, but should not be used to limit the scope of protection of the present invention. The conditions in the embodiments can be further adjusted according to specific conditions. Simple improvements to the method of the present invention under the premise of the concept of the present invention are all within the scope of protection claimed by the present invention.
[0039] Example 1
[0040] Please see Figure 1 As shown, this embodiment provides a reasoning enhancement method for large-scale natural language processing systems based on logical chain verification. By structurally organizing and logically analyzing the set of historical question-answer pairs, a mechanism is established to verify the content generated by the system.
[0041] The method specifically includes the following steps:
[0042] Collect and categorize historical question-and-answer pairs: Data is collected through interactive devices deployed on smart terminals, such as smartphones, in-vehicle infotainment systems, or smart home assistants. These interactive devices are responsible for capturing the user's questions and the responses generated in response to those questions. Together, they constitute historical question-and-answer pairs. The historical question-and-answer pairs collected in this way reflect the user's questioning habits and information needs in specific scenarios.
[0043] After data collection, a contextual dependency analysis is performed on each historical question-and-answer pair to determine its category. The specific process is as follows: the query content of the current question-and-answer pair is examined to determine whether it contains semantic components that refer to, follow up on, or supplement the response information of the previous historical question-and-answer pair. If the subsequent query content deepens or extends the response information of the previous historical question-and-answer pair, then the historical question-and-answer pair is classified as a progressive question-and-answer pair; otherwise, if the query content has no direct thematic inheritance relationship with the preceding dialogue, it is classified as a parallel question-and-answer pair. To improve the accuracy of the classification, this classification process can be combined with… The classification process utilizes pre-constructed standard question templates by domain experts to assist in the judgment. Specifically, the standard question templates serve as a reference during the classification process. By calculating the structured similarity between the query content of the question-answer pair to be classified and the standard question template, the category of the question-answer pair is determined. For example, by comparing the matching degree of the syntactic structure of the question, keyword sequence, or intent tag, the category can be determined when the similarity is higher than a preset classification threshold. After the classification is completed, all progressive question-answer pairs are stored in the progressive database, and all parallel question-answer pairs are stored in the parallel database, laying the data foundation for subsequent processing.
[0044] Structured similarity is calculated using a computational model that quantifies the degree of matching between the query content of the question-answer pair to be classified and the standard question template. The mathematical formula is defined as follows:
[0045] Input: The syntactic structure vector of the query content to be categorized and keyword set ; Syntactic structure vector of standard problem template and keyword set .
[0046] Output: Structured similarity score .
[0047]
[0048] In the formula, The structured similarity score represents the meaning of the query content. With template The similarity between them has a range of [0,1]; The syntactic structure vector represents a high-dimensional real vector that converts the syntactic parse tree or other structured representation of a sentence into a syntactic vector. This refers to a set of keywords, which is a set of words extracted from the query content and template that can represent its core semantics. The syntactic weight coefficient is a hyperparameter used to balance the importance of syntactic structure similarity and keyword similarity, and its value range is [0,1].
[0049] Furthermore, a logical chain is constructed: based on data in the progressive database, a logical chain representing a continuous reasoning path is constructed. A logical chain is an ordered sequence of nodes composed of semantically connected progressive question-and-answer pairs, used to represent a complete and continuous reasoning path, where each progressive question-and-answer pair serves as a node. The construction process is as follows: a progressive question-and-answer pair is selected from the progressive database as the starting node of the logical chain; then, using the response information of the starting node as the search content, a matching search is performed in the progressive database to determine subsequent nodes. The matching search can be exact matching or fuzzy matching, i.e., the text content is completely consistent or the semantic similarity or keyword overlap reaches a preset matching threshold; if a subsequent question-and-answer pair that meets the matching conditions is found, it is used as the subsequent node of the logical chain; and the above search process is repeated using the response information of the new node as the content until no subsequent node is found in the progressive database; by traversing the progressive database, multiple logical chains can be constructed. After construction, a path topology graph is generated. This path topology graph is a structured data graph used to intuitively display the intersection, convergence, and branching relationships between different logical chains, thereby revealing the association between different reasoning paths.
[0050] Furthermore, key nodes and secondary nodes are distinguished and weighted: After constructing multiple logical chains, the structural importance of each node in the entire dialogue logic framework is evaluated; the total number of times each node is used in all constructed logical chains is counted and recorded as the node's frequency of occurrence; then, based on the distribution of each node in multiple logical chains, nodes are distinguished as key nodes and secondary nodes; the specific distinction rules are: if a node exists in every logical chain of multiple logical chains, then the node is identified as a key node, representing the core and common link in the reasoning path; if a node exists only in some logical chains of multiple logical chains, then the node is identified as a secondary node, representing a non-essential or variable link in the reasoning path; all key nodes are categorized into a key content set, and all secondary nodes are categorized into an auxiliary content set, and each node is assigned a unique number for easy management; at the same time, weighted values are assigned to all nodes according to their category and frequency of occurrence; these weighted values are used to characterize the structural importance of the node in the logical chain structure.
[0051] One specific weight allocation method is to set a fixed high weight value, such as 1.0, for all key nodes, while the weight value of secondary nodes is obtained by normalizing their occurrence frequency to ensure that their value is within the range of (0, 1.0), thereby achieving a positive correlation between their weight value and occurrence frequency.
[0052] Further, the initial scores of nodes are calculated and weighting coefficients are determined: the historical interaction performance of each node is quantified, and weighting coefficients are determined for the final verification score; historical evaluation data associated with all nodes is obtained, which originates from user feedback on historical question-and-answer interactions; specifically, this includes the number of times users give positive feedback and the number of times they give negative feedback, i.e., positive feedback count and negative feedback count. For all nodes, their initial scores are calculated using their historical evaluation data; the specific calculation process is as follows: the probability of occurrence of positive feedback count and the probability of occurrence of negative feedback count are calculated, i.e., positive feedback count divided by the total feedback count and negative feedback count divided by the total feedback count; then, the difference between the probability of occurrence of positive feedback count and the probability of occurrence of negative feedback count is determined as the initial score of that node; finally, the initial scores of all key nodes are arithmetically averaged to obtain the key node evaluation score, which is used for macro-evaluation. The evaluation process involves assessing the overall historical performance of key nodes. Then, the initial scores of all secondary nodes are arithmetically averaged to obtain a secondary node evaluation score, used to macroscopically evaluate the overall historical performance of all secondary nodes. During final verification, the influence of key and secondary nodes is scientifically combined to determine a first and second preset weighting coefficient. These coefficients are dynamically determined based on the statistical differences between the initial score sets of all key nodes and all secondary nodes, adjusting the contribution of each type of node in the final score. Specifically, a set of judgment rules is established: when the mean of the initial score set of key nodes is greater than the mean of the initial score set of secondary nodes, and its variance is less than the variance of the initial score set of secondary nodes, the historical performance of the key node set is considered more stable and of higher quality. In this case, the first preset weighting coefficient is set to a value greater than the second preset weighting coefficient, increasing the influence weight of key nodes in the final score.
[0053] The initial score is calculated using a computational model that quantifies the credibility of a single node based on historical user feedback. The mathematical formula is defined as follows:
[0054] Input: For a specific node Historical positive feedback counting and historical negative feedback count .
[0055] Output: Nodes initial score .
[0056]
[0057] In the formula, This represents the initial score, which means that the node... The historical performance score is defined in the range of [-1, 1], with a higher score indicating better historical performance. This represents the positive feedback count, which means the number of times a user interacts with a node. The total number of positive responses to the replies, such as likes and acceptance; This represents the negative feedback count, which means the user's response to the node. The total number of times negative feedback was given in response to the reply information, such as downvoting or reporting; The smoothing factor is represented by a very small positive number, for example... This is used to prevent the denominator from being zero when the total feedback count is zero.
[0058] Furthermore, the inference verification is performed and the results are output: when the query content to be verified is received, the inference and verification process is started; based on the query content to be verified, a logic chain to be verified is constructed by performing a search and connection process similar to the "constructing a logic chain" step; then the key nodes and secondary nodes contained in the logic chain to be verified are identified, and one or more candidate response contents are generated based on the position order and weight value of each node in the chain.
[0059] One specific generation method is as follows: The response information of each node in the logic chain to be verified is concatenated according to the node order, and the level of detail of the corresponding response information in the final generated content is adjusted according to the weight value of each node; then the verification score of the logic chain to be verified is calculated, the initial scores of all key nodes in the logic chain to be verified are summed to obtain the score of the key nodes of the logic chain to be verified; the initial scores of all secondary nodes in the logic chain to be verified are summed to obtain the score of the secondary nodes of the logic chain to be verified.
[0060] The scores of key nodes in the verification chain are multiplied by a first preset weighting coefficient, and the scores of secondary nodes are multiplied by a second preset weighting coefficient. These two scores are then added together to obtain the verification score. The verification score is then converted to a percentage scale through a linear transformation and compared with a set of preset confidence intervals. This maps the score to one of several preset confidence levels, with each level associated with a specific answer output strategy. The confidence levels and their corresponding strategies are as follows: High confidence level indicates that the logical path of the candidate response is reliable, and it is determined to be a valid answer and output directly; Higher confidence level indicates that the candidate response has high confidence, and the reasoning answer is output along with a system-generated, highly accurate prompt; General confidence level indicates that the candidate response... The credibility of the response is generally low. The system outputs the reasoning answer with a verification suggestion that the content may contain errors and should be verified. The system monitors the availability zone level, which is the lowest credibility level. When the verification score falls into this range, the system will pause the direct output of the answer and initiate a re-verification or manual review process, indicating that the verification score is less than the preset verification threshold and the logical path of the candidate response content has a high risk. Then, the query content to be verified is marked as pending review, and an alternative logical chain is constructed by replacing or removing the node with the lowest initial score in the logical chain to be verified. The verification process of this step is then re-executed on the alternative logical chain. If the verification score is still at this level after a preset number of adjustments and re-verifications, the task is transferred to the manual review queue for manual intervention.
[0061] The verification score is calculated using a computational model that comprehensively evaluates the overall reliability of the logic chain under verification. The mathematical formula is defined as follows:
[0062] Input: The logic chain to be verified The set of all key nodes included and the set of all secondary nodes Initial score for each node First preset weighting coefficient Second preset weight coefficient .
[0063] Output: The verification score of the logic chain to be verified. .
[0064]
[0065] In the formula, The score represents the overall evaluation of the logic chain to be verified. A higher score indicates a more reliable logic path. This represents the preset weighting coefficients, which are used to adjust the proportions of the total score of key nodes and the total score of secondary nodes in the final verification score, respectively. This represents a set of nodes, meaning that each node is a logical chain to be verified. The set of key and secondary nodes contained therein; This represents the initial score, which means the node. or node The score is calculated based on historical user feedback.
[0066] Example 2
[0067] Please see Figure 2 As shown, this embodiment provides a reasoning enhancement system for large-scale natural language processing systems based on logic chain verification. By performing reliability verification on the logic chain generated internally before outputting the reasoning answer of the large-scale natural language processing system, it filters out answers that are logically flawed or contain factual errors, thereby improving the accuracy and reliability of the large-scale natural language processing system when handling complex, multi-step reasoning tasks.
[0068] In its implementation, this system can be deployed on servers, cloud computing platforms, or edge computing devices with corresponding computing power. Physically, it can consist of one or more processors, memory, and communication interfaces. The processor executes computer program instructions stored in memory to implement the following logical functional modules; the system specifically includes the following modules:
[0069] The logic model building module's core task is to build a basic model for evaluating the reliability of the logic chain by learning from and modeling historical data before the inference enhancement method is executed.
[0070] This module is configured to perform the following operations: communicate with users through interactive individuals deployed on smart terminals such as smartphones and personal computers, continuously collect user-posted queries and system-generated responses to those queries, and organize these pairs of data into historical question-and-answer pairs to form a historical question-and-answer pair set;
[0071] Then, the set of historical question-and-answer pairs is processed to identify dialogue sequences with contextual dependencies. Specifically, the historical question-and-answer pairs are traversed to determine whether the question content of each historical question-and-answer pair is based on the response information of the previous historical question-and-answer pair. If such a succession relationship exists, the historical question-and-answer pair is classified as a progressive question-and-answer pair.
[0072] Multiple logical chains are constructed based on progressive question-answer pairs. In the construction process, a starting question-answer pair is selected from the progressive question-answer pairs as the starting node of the logical chain. Then, the reply information of the starting node is used as the retrieval content to iteratively search for subsequent question-answer pairs in the progressive question-answer pair set. The query content of the subsequent question-answer pair matches or is highly related to the reply information of the previous node. By arranging the multiple nodes obtained by iterative search, i.e., question-answer pairs, in the search order, a complete logical chain is constructed. This process is repeated to construct multiple logical chains.
[0073] Then, all the constructed logical chains are analyzed to distinguish nodes of different importance. If a node exists in every logical chain, the module identifies this node as a key node because it represents a common and necessary link in the reasoning path of this type of problem. Conversely, if a node exists only in some logical chains, it is identified as a minor node. Then, a weight value is assigned to all nodes, which can be used in the subsequent reasoning answer generation process.
[0074] Then, the initial score for each node is calculated. For all nodes, historical evaluation data associated with them is obtained, such as the positive and negative feedback counts of user responses to the node. Based on these counts, the initial score for all nodes is calculated. The initial scores of all key nodes are aggregated, for example, by summing or averaging, to obtain the key node evaluation score. Then, the initial scores of all secondary nodes are aggregated to obtain the secondary node evaluation score.
[0075] Finally, based on the differences in statistical characteristics between the initial score sets of all critical nodes and the initial score sets of all secondary nodes, such as differences in mean, variance, and distribution pattern, the first and second preset weight coefficients for subsequent verification score calculations are determined; these two coefficients reflect the different importance of critical nodes and secondary nodes in the logical chain reliability assessment.
[0076] The reasoning and verification module is the core processing unit activated when the system receives a real-time request from a user. Its main task is to generate candidate response content and verify the underlying logical chain.
[0077] This module is configured to perform the following operations: when a query to be verified is received, a logic chain to be verified is constructed in response to the query. This process can be the thought chain generation process within a large-scale natural language processing system, that is, a series of intermediate reasoning steps generated by the model in order to answer the query. These steps constitute the nodes of the logic chain to be verified.
[0078] Then, based on each node in the logic chain to be verified and the weight values associated with each node determined by the logic model construction module, candidate response content is generated, indicating that the process of generating the answer is guided by the logic chain structure and constrained by the node weights.
[0079] While generating the answer, the verification score of the logic chain to be verified is calculated to identify which nodes in the logic chain are key nodes and which are secondary nodes. Then, the pre-calculated evaluation scores of key nodes and secondary nodes are obtained from the logic model construction module. The evaluation scores of key nodes are multiplied by the first preset weight coefficient, and the evaluation scores of secondary nodes are multiplied by the second preset weight coefficient. The two are then weighted and summed to obtain the verification score of the logic chain to be verified.
[0080] The result processing module is the final decision-making and output unit of the system. It determines how to process the generated candidate response content based on the calculation results of the reasoning and verification module.
[0081] This module is configured to perform the following operations: This module receives the verification score calculated by the inference verification module and compares it with a preset verification threshold, which represents the minimum acceptable standard for the reliability of the logic chain in the system;
[0082] Based on the comparison results, different operations are performed. When the verification score is greater than or equal to the preset verification threshold, it indicates that the system considers the logical chain on which the candidate response content depends to be reliable and trustworthy. At this time, the reasoned answer is output and presented to the user or passed to the downstream application.
[0083] Conversely, when the verification score is less than the preset verification threshold, it indicates that the system believes the logical basis of the candidate response is flawed or uncertain, and its reliability is low. In this case, in order to avoid providing incorrect or misleading information, the candidate response will not be output, but the query content to be verified will be marked as pending review. This status can be used to trigger the manual review queue or prompt the system to adopt other more conservative strategies to regenerate the answer.
[0084] Through the collaborative work of the above modules, the system in this embodiment can construct a closed-loop mechanism for self-examination and verification of the reasoning process of a large-scale natural language processing system. By quantitatively evaluating the reliability of the logical chain, a "quality gate" is implemented before the answer is output, thereby enhancing the model's performance in complex reasoning tasks.
Claims
1. A method for enhancing reasoning in large language models based on logical chain verification, characterized in that, Execution: Collect a set of historical question-and-answer pairs, where each historical question-and-answer pair includes the question content and the response information; Based on whether the question content of each historical question-and-answer pair is based on the response information of the previous historical question-and-answer pair, historical question-and-answer pairs are classified into progressive question-and-answer pairs. Multiple logical chains are constructed based on progressive question-and-answer pairs; Based on the distribution of each node in multiple logical chains, the nodes are divided into key nodes and secondary nodes, and a weight value is assigned to each node. Based on the historical evaluation data associated with the nodes, calculate the evaluation scores of key nodes and secondary nodes. When a query to be verified is received, the following steps are executed: Construct a logic chain to be verified in response to the query. Generate reasoning answers based on the logic chain to be verified; Calculate the verification score of the logic chain to be verified; Compare the verification score with the preset verification threshold; Based on the comparison results, when the verification score is greater than or equal to the preset verification threshold, the reasoning answer is output; when the verification score is less than the preset verification threshold, the query content to be verified is marked as pending review.
2. The method for enhancing reasoning in a large language model based on logical chain verification according to claim 1, characterized in that, Constructing multiple logical chains based on progressive question-answer pairs includes: selecting the starting question-answer pair from the progressive question-answer pairs as the starting node of the logical chain; Using the response information of the starting node as the retrieval content, iteratively search for subsequent question-answer pairs in the progressive question-answer pairs, where the query content of the subsequent question-answer pair matches the response information of the previous node; and arrange the multiple nodes obtained through iterative search in the search order to construct a logical chain.
3. The method for enhancing reasoning in a large language model based on logical chain verification according to claim 1, characterized in that, Distinguishing nodes into critical nodes and secondary nodes includes: if a node exists in every logical chain of multiple logical chains, then the node is identified as a critical node; If a node exists only in a portion of multiple logical chains, then the node is identified as a minor node.
4. The method for enhancing reasoning in a large language model based on logical chain verification according to claim 1, characterized in that, The calculation of the evaluation scores for key nodes and secondary nodes includes: for each node, calculating the initial score based on the positive feedback count and negative feedback count in its historical evaluation data; The initial scores of all key nodes are aggregated to obtain the key node evaluation score; and the initial scores of all secondary nodes are aggregated to obtain the secondary node evaluation score. Based on the statistical differences between the initial score sets of all key nodes and the initial score sets of all minor nodes, a first preset weight coefficient and a second preset weight coefficient are determined for calculating the verification score.
5. The method for enhancing reasoning in a large language model based on logical chain verification according to claim 1, characterized in that, The reasoning answers generated based on the logic chain to be verified include: Based on each node in the logic chain to be verified and the weight values associated with each node, a reasoning answer is generated.
6. The method for enhancing reasoning in a large language model based on logical chain verification according to claim 1, characterized in that, The calculation of the verification score of the logic chain to be verified includes: Identify the key and secondary nodes in the logic chain to be verified; Obtain the evaluation scores for key nodes and secondary nodes; The evaluation scores of key nodes are multiplied by the first preset weighting coefficient, and the evaluation scores of secondary nodes are multiplied by the second preset weighting coefficient. The results are then weighted and summed to obtain the verification score.
7. The method for enhancing reasoning in a large language model based on logical chain verification according to claim 1, characterized in that, The collection of historical question-and-answer pairs includes: By communicating with the user through at least one interactive entity deployed on a smart terminal, and capturing the user's posted questions and the response information generated in response to the questions, a historical question-and-answer pair is formed.
8. A large language model reasoning enhancement system based on logical chain verification, characterized in that, include: The logical model building module is used to collect a set of historical question-and-answer pairs, classify the historical question-and-answer pairs into progressive question-and-answer pairs, build multiple logical chains containing key nodes and secondary nodes based on the progressive question-and-answer pairs, and determine the evaluation scores and weight values associated with key nodes and secondary nodes. The reasoning and verification module is used to construct a logic chain to be verified in response to the query content to be verified when it receives the query content to be verified. Based on the nodes in the logic chain to be verified and the evaluation scores and weight values determined by the logic model construction module, it generates a reasoning answer and calculates the verification score. The result processing module compares the verification score calculated by the reasoning verification module with the preset verification threshold. When the verification score is greater than or equal to the preset verification threshold, the reasoning answer is output. When the verification score is less than the preset verification threshold, the query content to be verified is marked as pending review.
9. The large language model reasoning enhancement system based on logical chain verification according to claim 8, characterized in that, The logic model building module is specifically used to iteratively find progressive question-answer pairs that match the query content with the response information of the previous node in order to build multiple logical chains; Based on the distribution of nodes in multiple logical chains, nodes are divided into critical nodes and secondary nodes. The initial score is calculated based on the positive and negative feedback counts in the historical evaluation data of each node. The initial scores of all key nodes and all secondary nodes are then aggregated to obtain the evaluation scores of key nodes and secondary nodes.
10. The large language model reasoning enhancement system based on logical chain verification according to claim 8, characterized in that, The reasoning and verification module is specifically used to generate reasoning answers based on each node in the logic chain to be verified and the weight values determined by the logic model construction module. The verification score is calculated by weighted summing of the evaluation scores of the critical nodes and the evaluation scores of the secondary nodes in the logic chain to be verified.