A training method, inference method, apparatus, and equipment for a large language model.

By performing multi-level factual verification and hierarchical weight optimization on the large language model, the problem of local factual errors being difficult to suppress in complex structured reasoning tasks is solved, achieving accurate fact alignment between logical levels and improving the credibility of generated content.

CN123117338APending Publication Date: 2026-11-17TIANJIN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611322635.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-28
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing large language models struggle to accurately suppress local factual errors in complex structured reasoning tasks, leading to problems such as local illusions and insufficient factual consistency during the reasoning process.

Method used

By performing multi-level factual verification on candidate response sequences based on an external knowledge base, a word-level hierarchical mask matrix is ​​generated. Hierarchical weights are introduced in the preference optimization training, and the weighted log probability with mask is calculated to differentiate the generation probability of different logical levels, thus establishing a linkage mask mechanism between structured conclusions and thought chain reasoning.

Benefits of technology

It significantly improves the model's factual consistency and accuracy in complex structured reasoning tasks, reduces the local illusion rate, and enhances the logical rigor and credibility of the generated content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117338A_ABST
    Figure CN123117338A_ABST
Patent Text Reader

Abstract

This application relates to the fields of artificial intelligence and natural language processing technology, and provides a training method, inference method, apparatus, and device for a large language model. The method includes: acquiring candidate response sequences generated by the large language model for structured reasoning tasks; performing multi-level factual verification on each reasoning unit in the candidate response sequences based on an external knowledge base, and generating a lexical-level hierarchical mask matrix based on the verification results; and calculating a masked weighted log-probability during preference optimization training based on the lexical-level hierarchical mask matrix and preset hierarchical weights to differentially adjust the generation probability of different logical levels in the large language model. This application achieves fine-grained factual alignment, significantly reducing the local illusion rate in complex reasoning tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, specifically to a training method, inference method, apparatus, and device for a large language model. Background Technology

[0002] Currently, in the post-training optimization stage of large language models, preference alignment techniques are widely used to improve the consistency between model output and human preferences or domain-specific knowledge. However, existing preference optimization methods typically evaluate the model's generated responses as a whole. This coarse-grained optimization approach at the response level has significant drawbacks when dealing with complex structured reasoning tasks involving multiple logical levels. Specifically, when only a few logical levels in the long sequence of reasoning generated by the model contain factual errors, the global optimization signal struggles to accurately locate and penalize these locally erroneous units. Instead, it may incorrectly suppress or reinforce the correct reasoning parts within the same response, preventing the model from effectively learning the correct factual connections between different logical levels. Consequently, the model remains prone to local illusions in complex reasoning scenarios, resulting in insufficient factual consistency and interpretability in the reasoning process. Summary of the Invention

[0003] To address the problem that coarse-grained preference optimization in existing technologies makes it difficult to accurately suppress local factual errors in complex structured reasoning tasks, this application provides a training method, reasoning method, apparatus, and device for a large language model. This enables fine-grained factual alignment of different logical levels in the model-generated sequence, thereby significantly improving the factual consistency and accuracy of the model's reasoning.

[0004] To achieve the above objectives, this application adopts the following technical solution: A training method for a large language model includes: acquiring candidate response sequences generated by the large language model for a structured reasoning task, wherein the candidate response sequences consist of multiple lexical units and contain reasoning units corresponding to multiple logical levels; performing multi-level factual verification on each reasoning unit in the candidate response sequences based on an external knowledge base, and generating a lexical-level hierarchical mask matrix based on the verification results, wherein the lexical-level hierarchical mask matrix characterizes the factual correctness of each lexical unit in the candidate response sequences at different logical levels; and calculating a masked weighted log probability in preference optimization training based on the lexical-level hierarchical mask matrix and preset hierarchical weights to differentially adjust the generation probability of different logical levels in the large language model.

[0005] The above scheme maps the external knowledge verification results into a word-level hierarchical mask matrix and introduces hierarchical weights in preference optimization, thereby enabling the optimization signal to be refined from the response level to the word level and the hierarchical level. This allows the model to specifically correct factual errors at a particular logical level, rather than giving a general reward or punishment to the entire response.

[0006] As one implementation method, the step of performing multi-level factual verification on each reasoning unit in the candidate response sequence based on an external knowledge base and generating a word-level hierarchical mask matrix based on the verification results includes: extracting the structured conclusion part and the thought chain reasoning part from the candidate response sequence; performing factual verification on each reasoning unit in the structured conclusion part to obtain the conclusion verification result; marking the reasoning words in the thought chain reasoning part corresponding to the target logical level as factual errors based on the verification errors of the reasoning units at the target logical level in the structured conclusion part; and generating the word-level hierarchical mask matrix based on the conclusion verification result and the reasoning words marked as factual errors.

[0007] This implementation establishes a linkage masking mechanism between structured conclusions and reasoning chains, ensuring that when the final conclusion contains factual errors, the upstream reasoning process is also synchronously marked as invalid. This avoids the model learning erroneous association patterns of "erroneous reasoning leading to erroneous conclusions" or "correct reasoning leading to erroneous conclusions," thus guaranteeing the overall logical consistency of the reasoning chain.

[0008] As one implementation method, the factual verification of each reasoning unit in the structured conclusion section includes at least one of the following verification dimensions: consistency verification between input features and the first-level conclusion; consistency verification between the first-level conclusion and the second-level processing strategy; consistency verification between the second-level processing strategy and the specific execution parameters; and compliance verification between the specific execution parameters and the preset security constraint rules.

[0009] As one implementation, before calculating the masked weighted log-probability and loss function in preference optimization training based on the lexical-level hierarchical mask matrix and preset hierarchical weights, the method further includes: for the same input instruction, obtaining multiple candidate response sequences generated by the large language model; calculating the comprehensive factual score of each candidate response sequence based on the lexical-level hierarchical mask matrix and the hierarchical weights; selecting the candidate response sequence with the highest comprehensive factual score as the preferred response, and selecting the candidate response sequence with the lowest comprehensive factual score as the non-preferred response, thus constructing preference data pairs.

[0010] As one implementation method, the segment-level average log probability is calculated based on the word-level hierarchical mask matrix:

[0011] in, For a complete sequence response, For sequence length, For the corresponding level, For the current strategy model, For the first Each word element, For the first The lexical sequence preceding each lexical unit, Indicates the first The next level A mask for each word. , represents the number of tokens verified at this level, where . It normalizes the length of content at different length levels.

[0012] Calculate the masked log-odds during preference optimization training:

[0013] For the preferred and non-preferred responses in the preference data pairs and the preset hierarchical weights, the masked weighted log-probability of each logical level is calculated to differentiate the generation rate of different logical levels in the large language model. The calculation formula is as follows:

[0014] in, and These are respectively the preferred response and the non-preferred response. and These are the preferred and non-preferred responses, respectively. Hierarchical mask vectors, For the same input command, Let j=0 represent the total number of logical levels, j=0 represent the complete answer level dimension, retain the complete response sequence, and j=1……K represent the K levels in the labeled responses. Based on the weighted log odds of each logical level and the corresponding level weights, a multi-granularity contrastive loss function is constructed to update the parameters of the large language model.

[0015] This implementation ensures differentiated control of gradient updates across different logical levels based on factual correctness through masking normalization and hierarchical weighting terms in the formula. Specifically, the masking mechanism ensures that correct inferences in preference data are positively reinforced, while erroneous inferences in non-preference data are targeted and suppressed. Furthermore, the optimization intensity at different logical levels can be flexibly configured according to task requirements, preventing long entities or non-critical levels from dominating the gradient update direction and improving the stability and accuracy of preference optimization.

[0016] As one implementation, before obtaining the candidate response sequence generated by the large language model for the structured reasoning task, the method further includes: constructing a multi-level instruction fine-tuning dataset containing a thought chain reasoning part and a structured conclusion part; using the multi-level instruction fine-tuning dataset to perform supervised fine-tuning on the pre-trained large language model to obtain an initial policy model; the large language model is the initial policy model.

[0017] As one implementation method, the construction of the external knowledge base includes: extracting entities and relationships based on multi-source domain data to construct a knowledge graph containing hierarchical mapping relationships; constructing a rule base containing hard constraint rules based on domain expert rules; the external knowledge base includes the knowledge graph and the rule base.

[0018] Furthermore, this application also provides a reasoning method for a large language model, comprising: acquiring input data of a target object; inputting the input data into a large language model to generate a thought chain reasoning part and a structured conclusion part containing multiple logical levels, wherein the large language model is trained by the training method described above; and outputting the structured conclusion part and a comprehensive fact score of factual confidence scores and response sequences corresponding to each logical level.

[0019] The aforementioned reasoning method utilizes a model trained with fine-grained factual alignment. During the reasoning phase, it not only outputs structured conclusions but also provides factual confidence scores at each level, enhancing the interpretability and transparency of the model output and facilitating users' evaluation of the reliability of the reasoning results.

[0020] Furthermore, this application also provides a training device for a large language model, comprising: a sequence acquisition module for acquiring candidate response sequences generated by the large language model for a structured reasoning task, wherein the candidate response sequences consist of multiple lexical units and contain reasoning units corresponding to multiple logical levels; a fact verification and mask generation module for performing multi-level factual verification on each reasoning unit in the candidate response sequences based on an external knowledge base, and generating a lexical-level hierarchical mask matrix based on the verification results, wherein the lexical-level hierarchical mask matrix characterizes the factual correctness of each lexical unit in the candidate response sequences at different logical levels; and a preference optimization module for calculating a masked weighted log probability in preference optimization training based on the lexical-level hierarchical mask matrix and preset hierarchical weights, so as to differentially adjust the generation probability of different logical levels in the large language model.

[0021] In addition, this application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the method described above.

[0022] Beneficial effects: The technical solution provided in this application performs multi-level factual verification on candidate response sequences based on an external knowledge base and generates a word-level hierarchical mask matrix. This accurately maps the abstract factual verification results to the word space generated by the model, enabling the gradient update in the preference optimization process to accurately locate the specific word with factual errors and its logical level. This fundamentally solves the technical problem that traditional response-level preference optimization cannot distinguish between local factual errors and significantly reduces the local illusion rate of the model in complex structured reasoning tasks.

[0023] Furthermore, by introducing hierarchical weights and a masked weighted log-probability calculation mechanism into the preference optimization loss function, differentiated adjustment of the generation probability for different logical levels is achieved. This mechanism allows for dynamic allocation of optimization intensity based on the importance of each logical level in the inference task, ensuring that errors at key decision levels are more fully corrected. Simultaneously, mask normalization eliminates the interference of differences in sequence length and key information density on gradient updates, improving the convergence stability and fact alignment efficiency of model training.

[0024] Furthermore, by establishing a linkage masking mechanism between structured conclusions and the reasoning part of the thought chain, when a factual error is detected in the downstream conclusion level, the corresponding reasoning term in the upstream is automatically marked as an error. This causal linkage masking strategy forces the model to learn the factual consistency between the reasoning process and the conclusion, effectively avoiding the model learning erroneous reasoning paths and improving the overall logical rigor and credibility of the generated content. Attached Figure Description

[0025] Figure 1 This is a flowchart of the large language model training method according to an embodiment of this application; Figure 2 This is a schematic diagram of the structured reasoning chain hierarchy in an embodiment of this application; Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0028] Example 1 like Figure 1As shown, this embodiment provides a training method for a large language model. This method is mainly applied to the post-training optimization stage of a large language model, aiming to solve the problem that local factual errors are difficult to suppress accurately in complex structured reasoning tasks. The method of this embodiment includes the following steps: The steps involve obtaining candidate response sequences generated by a large language model for structured reasoning tasks. These candidate response sequences consist of multiple lexical units and contain reasoning units corresponding to multiple logical levels.

[0029] Specifically, structured reasoning tasks refer to those tasks that not only require the model to generate fluent text, but also need to follow a specific domain's logical paradigm and possess a rigorous causal chain. For example, such tasks could be diagnostic reasoning in the medical field, case analysis in the legal field, troubleshooting or code generation tasks in the industrial field, etc., and this application does not specifically limit them. Unlike ordinary open-ended dialogues, the output of structured reasoning tasks typically contains multiple logical levels with clear semantic boundaries, each level corresponding to a key link in the reasoning chain. The reasoning unit refers to the smallest semantic functional block that constitutes these logical levels; for example, in diagnostic tasks, it might correspond to symptom summarization, pathogenesis analysis, and treatment determination, while in legal tasks, it might correspond to fact-finding, legal provision matching, and sentencing recommendations. The candidate response sequence is usually one of multiple potential responses generated from the current strategy model through a pre-defined sampling strategy (such as Top-k sampling or kernel sampling), and it is composed of serialized tokens from the model's vocabulary. By obtaining candidate sequences containing complete logical structures, the necessary structured foundation is provided for subsequent fine-grained fact verification, enabling the optimization process to delve into the internal links of the reasoning chain, rather than merely staying on the surface of the final answer.

[0030] Step S102: Perform multi-level factual verification on each reasoning unit in the candidate response sequence based on an external knowledge base, and generate a word-level hierarchical mask matrix based on the verification results. The word-level hierarchical mask matrix represents the factual correctness of each word in the candidate response sequence at different logical levels.

[0031] Specifically, an external knowledge base is an authoritative knowledge source independent of model parameters, which may include domain knowledge graphs, expert rule bases, standard specification documents, or a combination thereof, used to provide objective fact-checking benchmarks. Multi-level fact-checking refers to judging the correctness of each inference unit identified in the candidate response sequence at its respective logical level. For example, verifying whether an intermediate conclusion is reasonably derived from upstream input features, or verifying whether an execution parameter conforms to safety constraints. Since the training and optimization of large language models are essentially performed in the lexical space, while fact-checking is performed in the semantic level space, there is a misalignment in granularity. Therefore, this embodiment introduces a lexical-level hierarchical mask matrix as a bridge connecting semantic verification and model optimization. This matrix is ​​a two-dimensional mapping structure, with one dimension corresponding to the lexical index in the candidate response sequence and the other dimension corresponding to predefined fact-checking units. Each element value in the matrix represents the factual state of the corresponding lexical at the corresponding logical level; for example, a value of 1 indicates that the lexical is factually correct at that level, and a value of 0 indicates factually incorrect. It should be understood that although this embodiment uses a binary mask as an example, in other embodiments, the mask value can also be a continuous probability value or a confidence score, as long as it can characterize the degree of factual correctness. Through this mapping mechanism, the abstract semantic verification result is accurately projected onto the model-aware lexical space, enabling subsequent gradient updates to precisely locate specific lexical positions.

[0032] Step S103: Based on the word-level hierarchical mask matrix and the preset hierarchical weights, calculate the masked weighted log probability and optimization loss in the preference optimization training to differentiate the generation probability of different logical levels in the large language model.

[0033] Specifically, preference optimization training aims to adjust model parameters by comparing preferred and non-preferred responses, making them more aligned with human preferences or factual standards. Traditional preference optimization methods typically calculate a single log-odds across the entire response sequence, resulting in a uniform distribution of the optimization signal across the entire sequence. This fails to distinguish between critical decision-making stages and ordinary descriptive stages, and also fails to eliminate the interference of erroneous terms in preferred responses and correct descriptive terms in non-preferred responses on the gradient. This embodiment introduces a word-level hierarchical masking matrix to strengthen the contribution of masked and valid terms when calculating log-odds, thereby enhancing the suppression of erroneous content generation. Simultaneously, combined with preset hierarchical weights, the optimization intensity of different logical levels is weighted and adjusted. For example, higher weights can be assigned to critical decision-making levels that determine the direction of reasoning (such as core conclusions and key parameters), while lower weights can be assigned to auxiliary descriptive levels. This differentiated adjustment mechanism allows the model to concentrate limited optimization resources on the most critical factual error correction during training, avoiding optimization bias caused by long-tailed descriptions or non-critical information dominating gradient updates. From a microscopic perspective, the masked weighted log odds calculation formula and the hierarchical preference optimization loss actually construct a multi-granularity contrastive loss function. This function can accurately enhance the generation probability of the correct reasoning path at the word level, while suppressing local illusions at specific logical levels. Thus, while ensuring the overall coherence of reasoning, it significantly improves the factual accuracy and logical consistency of the model output.

[0034] Through the steps described above up to S103, this embodiment establishes a complete closed loop from structured semantic verification to lexical-level gradient control. Compared to the coarse-grained optimization at the response level in existing technologies, this embodiment achieves fine-grained allocation of optimization signals in both spatial and importance dimensions by mapping the verification results to a lexical-level hierarchical mask matrix and combining it with hierarchical weights. This mechanism can not only accurately identify and penalize local factual errors in the reasoning chain, but also effectively protect the correct reasoning parts in the same response from being mistakenly penalized, fundamentally solving the technical problems of difficult illusion suppression and low fact alignment efficiency in complex structured reasoning tasks.

[0035] Example 2 Building upon Example 1, this example further specifies the concrete implementation method for performing multi-level factual verification on each reasoning unit in the candidate response sequence based on an external knowledge base, and generating a word-level hierarchical mask matrix according to the verification results. As one implementation method, this process includes: extracting the structured conclusion part and the thought chain reasoning part from the candidate response sequence; performing factual verification on each reasoning unit in the structured conclusion part to obtain the conclusion verification result; verifying errors in the reasoning units corresponding to the target logical level in the structured conclusion part, marking the reasoning words in the thought chain reasoning part corresponding to the target logical level as factual errors; and generating a word-level hierarchical mask matrix based on the conclusion verification result and the reasoning words marked as factual errors.

[0036] Specifically, referring to the diagram, candidate response sequences typically include an explicit structured conclusion section and an implicit or explicit thought chain reasoning section. In this application, the thought chain reasoning section can be located within specific labels (e.g., tags), recording the complete derivation process of the model from input to output; while the structured conclusion section usually presents the final decision result in the form of tables, lists, or standardized text blocks. In this embodiment, fact verification is not performed in isolation for these two parts, but rather a cross-regional linkage masking mechanism is established. When the system detects that a reasoning unit at a specific logical level in the structured conclusion section has not passed external knowledge base verification, not only will the lexical unit corresponding to that conclusion be marked as incorrect, but the system will also trace back to the thought chain reasoning section, locate the derivation fragment on which the conclusion depends, and synchronously mark these upstream reasoning lexical units as factual errors. The initial intention of this linkage mechanism is to solve the common "reasoning-conclusion inconsistency" problem in the training of large language models. Without using linkage masks, and only penalizing erroneous conclusions, the model might learn the erroneous pattern of "correct reasoning leading to incorrect conclusions" during training, or retain "erroneous reasoning processes" simply because the conclusion happens to be corrected and not sufficiently suppressed. By using linkage markers, the model is forced to recognize that an error in a downstream conclusion necessarily means the failure of the upstream reasoning path. This allows for the simultaneous suppression of erroneous reasoning paths and erroneous conclusion outputs during gradient updates, ensuring the overall logical rigor and factual consistency of the reasoning chain.

[0037] Furthermore, as one implementation method, factual verification is performed on each reasoning unit in the structured conclusion section, including at least one of the following verification dimensions: consistency verification between input features and first-level conclusion; consistency verification between first-level conclusion and second-level processing strategy; consistency verification between second-level processing strategy and specific execution parameters; and compliance verification between specific execution parameters and preset security constraint rules.

[0038] Specifically, the aforementioned verification dimensions constitute a progressive logical verification system, designed to ensure the close connection and compliance of each stage in structured reasoning tasks. To more clearly illustrate the technical solution of this application, two different application scenarios are used as examples below, but the scope of protection of this application is not limited thereto.

[0039] In the first exemplary structured reasoning scenario, the input features can be raw observation data or problem descriptions provided by the user. The first-level conclusion is a preliminary induction or qualitative judgment of these features. The second-level processing strategy is a solution framework determined based on this qualitative judgment, while the specific execution parameters are the precise values ​​or configuration items required to implement the solution. For example, in an intelligent diagnosis scenario, the input features correspond to the patient's chief complaint and vital signs, the first-level conclusion corresponds to the disease type or state determination, the second-level processing strategy corresponds to the treatment principle or intervention plan, and the specific execution parameters correspond to drug combinations, dosages, or surgical parameters. In this case, "consistency verification between input features and first-level conclusions" is used to check whether the inferred state is fully supported by the raw data; "consistency verification between first-level conclusions and second-level processing strategies" is used to check whether the selected solution is suitable for the current state; "consistency verification between second-level processing strategies and specific execution parameters" is used to check whether the specific parameters correctly reflect the established solution; and "safety constraint compliance verification" serves as a safety net, checking whether the execution parameters violate preset contraindications or safety thresholds (such as drug incompatibilities, equipment operating limits, etc.).

[0040] In the second exemplary structured reasoning scenario, such as a legal document analysis task, the input features can be case factual evidence, the first-level conclusion is the determination of the legal relationship, the second-level processing strategy is the applicable legal provisions or litigation claims, and the specific execution parameters are the calculation of compensation amount or the recommended sentence. In this case, the above four verification dimensions correspond to the matching degree between evidence and factual findings, the correspondence between factual findings and the application of legal provisions, the accuracy of the application of legal provisions and the calculation of claims, and whether the claim outcome complies with the statutory upper limit or procedural provisions.

[0041] Through this universal multi-dimensional verification architecture, this embodiment transforms domain-specific professional knowledge constraints into a standardized logical verification process. This enables the technical solution to be widely applied to various structured reasoning tasks requiring rigorous logical deduction, avoiding limitations in protection scope caused by differences in domain terminology. Simultaneously, the multi-dimensional verification results provide rich, fine-grained information for the word-level hierarchical mask matrix, allowing subsequent preference optimization to apply differentiated correction signals for errors of different natures (such as premise errors, logical breaks, parameter out-of-bounds errors, etc.).

[0042] Example 3 Based on Example 1 or Example 2, this example further defines the construction method of preference data pairs and the specific calculation process of the loss function in preference optimization training. As one implementation method, before calculating the masked weighted log-probability and preference optimization loss in preference optimization training based on the lexical-level hierarchical mask matrix and preset hierarchical weights, the method further includes: for the same input instruction, obtaining multiple candidate response sequences generated by a large language model; calculating the comprehensive factual score of each candidate response sequence based on the lexical-level hierarchical mask matrix and hierarchical weights; selecting the candidate response sequence with the highest comprehensive factual score as the preferred response, and selecting the candidate response sequence with the lowest comprehensive factual score as the non-preferred response, thus constructing preference data pairs.

[0043] Specifically, this step constitutes the data preparation stage for preference optimization, the core purpose of which is to select sample pairs with significant comparative value from the current generation distribution of the model. For the same input instruction, strategies such as Top-k sampling or kernel sampling are typically used to generate K (e.g., 8 to 16) candidate response sequences to ensure sample diversity. For each candidate response sequence, a weighted sum is performed using the word-level hierarchical mask matrix generated in the previous embodiment and the preset hierarchical weights to obtain a scalarized comprehensive factual score. This score reflects the factual correctness of the response in the overall logical chain. It is important to emphasize that the comprehensive factual score here is only used for sample selection and ranking, that is, to determine which response is a positive sample (preferred response) and which response is a negative sample (non-preferred response), and it does not directly participate in the subsequent gradient backpropagation calculation. This is different from the probability adjustment using the mask matrix inside the loss function described in step S103 of embodiment 1, which are two independent but related action boundaries. The former solves the problem of "what data to use for training", while the latter solves the problem of "how to use this data to update parameters". This separation design ensures both the efficiency of preference pair construction and the flexibility of fine-grained gradient control during the training phase.

[0044] Furthermore, as one implementation method, the segment-level average log probability is calculated based on the word-level hierarchical mask matrix:

[0045] in, For a complete sequence response, For sequence length, For the corresponding level, For the current strategy model, For the first Each word element, For the first The lexical sequence preceding each lexical unit, Indicates the first The next level A mask for each word. , represents the number of tokens verified at this level, where . It normalizes the length of content at different length levels.

[0046] Calculate the masked log-odds during preference optimization training:

[0047] For the preferred and non-preferred responses in the preference data pairs and the preset hierarchical weights, the masked weighted log-probability of each logical level is calculated to differentiate the generation rate of different logical levels in the large language model. The calculation formula is as follows:

[0048] in, and These are respectively the preferred response and the non-preferred response. and These are the preferred and non-preferred responses, respectively. Hierarchical mask vectors, For the same input command, Let j=0 represent the total number of logical levels, j=0 represent the complete answer level dimension, retain the complete response sequence, and j=1……K represent the K levels in the labeled responses. Based on the weighted log odds of each logical level and the corresponding level weights, a multi-granularity contrastive loss function is constructed to update the parameters of the large language model.

[0049] Specifically, the above formula is the mathematical core for achieving fine-grained factual alignment. The logarithmic term on the right-hand side of the first formula... Characterizes the generated lexical units of the current strategy model The mask value, relative to the log odds of not generating the term, reflects the model's local confidence in that term. It serves a dual purpose here: in the numerator summation, it acts as a selector, accumulating only the terms belonging to the first numerator. The log-odds ratio of each logical level of tokens (representing correctly judged fact units in preferred samples and incorrectly judged fact units in unpreferred samples) masks interference from other levels or irrelevant tokens; in the denominator, Represents the first The total number of effective lexical units contained in each logical level plays a crucial role in length normalization. Without this normalization, the cumulative logarithmic probability of long text levels containing more lexical units (such as detailed thought chain reasoning sections) will naturally be much greater than that of short text levels (such as brief conclusion labels). This causes gradient updates to be dominated by sequence length rather than factual importance, leading the model to overemphasize the fluency of generating long paragraphs while neglecting the accuracy of key short conclusions. By dividing by the number of effective lexical units, it is ensured that logical levels of different lengths have comparable numerical magnitudes in the optimization objective, allowing differentiated adjustment to truly be implemented at the semantic importance level. Based on this, the hierarchical weights... The introduction of this approach further optimizes the strategic allocation of resources. In practical applications, the importance of different logical levels to task success is uneven. For example, in structured reasoning tasks, the correctness of key decision levels such as intermediate conclusion determination and execution parameter configuration directly determines the effectiveness of subsequent reasoning; therefore, higher level weights can be set, for example, a value range of 1.0 to 1.5. Conversely, for basic description levels such as background description and input restatement, the error tolerance is relatively high, and lower level weights can be set, for example, a value range of 0.5 to 0.8. To verify the necessity of differentiated weights, this application conducted a comparative experiment: when the weights of all logical levels within the responses are set to 0, i.e., degenerating to only performing preference optimization on the complete sequence, the model's efficiency in correcting key decision errors during training significantly decreases, manifested as a slower convergence speed of the loss function, and the factual accuracy of key levels on the test set is approximately 5% to 8% lower than when using differentiated weight configuration. This result demonstrates that uniform weights cannot effectively guide the model to focus on core errors, while the differentiated weight mechanism proposed in this embodiment can significantly improve the model's sensitivity to and ability to correct key factual errors, thus proving that this technical feature has substantial technical effects and creativity.

[0050] It should be understood that the above-mentioned weight range is merely illustrative and not intended to limit the scope of protection of this application. In specific implementations, the hierarchical weights can be fixed hyperparameters, dynamically adjusted variables based on the confidence level of the verification results, or even automatically learned through meta-learning. As long as the hierarchical weights are used to non-uniformly adjust the optimization intensity of different logical levels, they should be covered within the scope of protection of this application. The final constructed multi-granularity contrastive loss function is usually obtained by weighted summing the weighted log-probability differences (preferred response minus non-preferred response) of each logical level after processing with an activation function (such as Log-Sigmoid), thereby driving the model parameters to be updated in the direction of enhancing the probability of generating correct levels in preferred responses and suppressing the probability of generating incorrect levels in non-preferred responses.

[0051] Example 4 Building upon Examples 1 to 3, this example further defines the model preparation stage before obtaining candidate response sequences and the specific construction method of the external knowledge base. As one implementation method, before obtaining the candidate response sequences generated by the large language model for the structured reasoning task, the method further includes: constructing a multi-level instruction fine-tuning dataset containing a thought chain reasoning part and a structured conclusion part; using the multi-level instruction fine-tuning dataset to perform supervised fine-tuning of the pre-trained large language model to obtain an initial policy model; the large language model serves as the initial policy model.

[0052] Specifically, this step aims to endow the base model with the fundamental ability to generate responses conforming to a specific logical paradigm, which is a necessary prerequisite for subsequent fine-grained preference optimization. Unlike ordinary dialogue fine-tuning, this embodiment emphasizes that the multi-level instruction fine-tuning dataset must have dual structural features, that is, each training sample must simultaneously contain an explicit thought chain reasoning part and a structured conclusion part. The thought chain reasoning part records the complete derivation process from input to output, usually presented in the form of natural language paragraphs or specific tags; the structured conclusion part displays the final decision result in the form of tables, lists, or standardized fields. This dual structure design is not simply to enrich the output format, but directly serves the linkage masking mechanism and multi-level verification logic mentioned in the previous embodiment. If the training data only contains the final question-answer pair and lacks the intermediate reasoning chain, or if the reasoning process and conclusion are not physically distinguished, then in the subsequent preference optimization stage, it will be impossible to extract independent reasoning units for fact verification, and it will be impossible to establish a causal mapping relationship between upstream reasoning lexical units and downstream conclusion lexical units, resulting in the inability to generate a lexical-level hierarchical masking matrix. Therefore, by constructing a fine-tuning dataset with a dual structure and performing supervised fine-tuning, the resulting initial policy model can internalize this structured generation pattern, thereby ensuring that it naturally produces objects that can be finely optimized by the method of this application when sampling and generating candidate response sequences. It should be understood that the specific hyperparameters of supervised fine-tuning (such as learning rate, batch size, number of rounds, etc.) can be flexibly adjusted according to the scale of the base model and the amount of data, and this application does not impose any restrictions on this.

[0053] Furthermore, as one implementation method, the construction of the external knowledge base includes: extracting entities and relationships based on multi-source domain data to construct a knowledge graph containing hierarchical mapping relationships; constructing a rule base containing hard constraint rules based on domain expert rules; the external knowledge base includes the knowledge graph and the rule base.

[0054] Specifically, the external knowledge base is the cornerstone supporting the multi-level factual verification in Example 1, and its quality directly determines the accuracy of the mask matrix and the effectiveness of preference optimization. This example divides the external knowledge base into two complementary forms to address different types of verification needs in structured reasoning tasks. The first form is a knowledge graph constructed based on multi-source domain data extraction, mainly used to provide a soft semantic verification benchmark. Here, "multi-source domain data" can cover authoritative textbooks, industry standard documents, historical case records, or professional literature, etc. The key is that this knowledge graph is not a flat collection of entities, but must contain clear hierarchical mapping relationships. For example, the edge relationships in the graph should reflect logical links such as "input features point to the first-level conclusion," "the first-level conclusion points to the second-level processing strategy," and "the processing strategy points to specific execution parameters." It is this hierarchical topology that enables the verification module to check consistency step by step along the reasoning path, rather than simply performing isolated fact point matching. If the knowledge base lacks hierarchical associations, it cannot support the implementation of cross-level verification dimensions such as "consistency between input features and conclusions" or "consistency between strategy and parameters" in the aforementioned examples.

[0055] The second form is a rule base built based on domain expert rules, primarily used to provide hard security constraints and compliance safeguards. Compared to the probabilistic or semantic matching of knowledge graphs, rule bases typically store deterministic logical judgment conditions or threshold ranges. For example, in tasks involving parameter configuration, the rule base can define safe upper and lower limits for values; in tasks involving combinatorial decision-making, the rule base can define mutually exclusive terms or a list of mandatory options. These rules are usually derived from mandatory industry standards, safe operating procedures, or the experience summaries of senior experts, possessing inviolable rigid characteristics. During multi-level verification, when the semantic verification result of the knowledge graph is in an ambiguous range, the hard constraints of the rule base can serve as the final decision-making basis, ensuring that the reasoning units generated by the model do not exceed the security threshold. By combining the flexible semantic associations of the knowledge graph with the rigid constraints of the rule base, this embodiment constructs an external knowledge system with both breadth and depth, providing not only multi-dimensional verification signals for the generation of word-level hierarchical mask matrices but also providing dual guarantees for the factual accuracy and security of the model in complex reasoning scenarios. It should be understood that the specific storage format of knowledge graphs and rule bases (such as graph databases, relational databases, vector indexes, etc.) and their update and maintenance mechanisms can be selected according to actual engineering needs. As long as they can provide hierarchical query interfaces and rule matching services, they should be covered within the scope of protection of this application.

[0056] Example 5 This embodiment provides a reasoning method for a large language model. This method is applied to a large language model optimized using the training methods described in Embodiments 1 to 4, aiming to solve the problems of credibility assessment of reasoning results and human-computer collaborative decision-making in practical application scenarios. As one implementation method, the reasoning method includes: acquiring input data of a target object; inputting the input data into a large language model to generate a thought chain reasoning part containing multiple logical levels and a structured conclusion part, wherein the large language model is trained using the training methods described in any one of Embodiments 1 to 4; and outputting the structured conclusion part, as well as a comprehensive factual score of the factual confidence score and response sequence corresponding to each logical level.

[0057] Specifically, this embodiment uses a traditional Chinese medicine (TCM) diagnostic and treatment scenario as an example for detailed explanation. However, it should be understood that this reasoning method is also applicable to other structured reasoning fields such as legal consultation and industrial fault diagnosis. In the TCM diagnostic and treatment scenario, the input data of the target object typically includes the patient's chief complaint, present medical history, past medical history, and physical signs collected through the four diagnostic methods of observation, auscultation, inquiry, and palpation, such as tongue appearance, pulse, complexion, and voice. This unstructured or semi-structured multimodal information is integrated into the model's input prompts. After receiving the input data, the large language model does not directly provide a final diagnostic conclusion, but first generates a thought chain reasoning section containing a complete derivation process. This section is usually located within specific tags and records in detail the model's internal thinking path from symptom summarization, pathogenesis analysis, syndrome judgment, treatment establishment to prescription selection, simulating the clinical thinking process of TCM experts who "examine the symptoms to seek the cause, formulate treatment based on the cause, and select prescriptions according to law." Subsequently, based on the analysis results of the thought chain reasoning part, the model generates a structured conclusion part, which is usually presented in a standardized table, key-value pair list or Markdown format, clearly listing the specific content of the five logical levels of "symptom-syndrome-treatment-prescription-medicine", such as the syndrome being "liver stagnation and spleen deficiency syndrome", the treatment being "soothing the liver and strengthening the spleen", and the prescription being "Xiaoyao San with modifications", etc.

[0058] Furthermore, the structured conclusion section is output, along with the comprehensive factual score of the factual confidence score and response sequence corresponding to each logical level.

[0059] In practical applications, factual confidence scores directly support the human-machine collaborative review and intervention mechanism. When the confidence score of a certain logical level falls below a preset safety threshold (e.g., 0.80), the system can automatically trigger an early warning, suggesting that the user manually review the content at that level. For licensed physicians, a low confidence score does not mean negating the model's output, but rather provides precise attention guidance, allowing them to focus on key nodes marked as "uncertain" by the model without having to review the entire reasoning process word by word. It should be understood that the above-mentioned TCM syndrome differentiation and treatment scenario is only a specific application example of the technical solution of this application. In other embodiments, the input data can be case files, equipment sensor logs, etc., and the logical level can be legal relationship determination, fault root cause location, etc. As long as the model optimized by the training method of this application is used and hierarchical factual confidence scores are output, they should be covered within the protection scope of this application.

[0060] Example 6 This embodiment provides a training device for a large language model. This device is a concrete manifestation of the training methods in Embodiments 1 to 5 at the virtual device level, used to achieve fine-grained factual alignment optimization for structured reasoning tasks. Figure 1 As shown, the training device includes a sequence acquisition module, a fact verification and mask generation module, and a preference optimization module.

[0061] The sequence acquisition module is used to acquire candidate response sequences generated by the large language model for structured reasoning tasks. These candidate response sequences consist of multiple lexical units and contain reasoning units corresponding to multiple logical levels. Specifically, the sequence acquisition module functionally corresponds to the steps in Example 1, serving as the data entry point for the entire training process and responsible for interacting with the current policy model. In actual deployment, this module can acquire candidate response sequences in batches by calling the model's generation interface and utilizing a preset sampling strategy. It should be understood that the sequences acquired by this module must retain complete logical hierarchical structure information so that subsequent modules can accurately identify the semantic boundaries of each reasoning unit. For example, this module can parse specific tags or format markers in the model output, transforming unstructured text streams into structured objects containing hierarchical metadata, thereby providing standardized input data for downstream fact verification.

[0062] The fact verification and mask generation module performs multi-level factual verification on each reasoning unit in the candidate response sequence based on an external knowledge base, and generates a word-level hierarchical mask matrix based on the verification results. This matrix represents the factual correctness of each word in the candidate response sequence at different logical levels. Specifically, this module functionally corresponds to step S102 in Example 1 and the linkage masking mechanism in Example 2. Internally, it integrates a knowledge base retrieval engine and a logic verifier, enabling precise mapping of semantic-level verification results to the word space. During data flow, this module receives the candidate response sequence from the sequence acquisition module, performs multi-level verification, and outputs a two-dimensional word-level hierarchical mask matrix. This matrix not only contains the factual state markers of each word but also implicitly includes causal relationship information between logical levels (such as the upstream and downstream linkage relationship described in Example 2). This design transforms abstract knowledge constraints into numerical signals that the model can process, solving the alignment problem between external knowledge and the model's internal representation.

[0063] The preference optimization module is used to calculate the masked weighted log odds and preference optimization loss during preference optimization training based on the word-level hierarchical mask matrix and preset hierarchical weights, so as to differentiate the generation probability of different logical levels in the large language model. Specifically, this module corresponds functionally to step S103 in Embodiment 1 and the loss function calculation process in Embodiment 3. It receives the mask matrix output from the fact verification and mask generation module, and performs masked weighted log odds calculation and gradient backpropagation in combination with the preset hierarchical weight configuration. The core function of this module is to transform fine-grained factual feedback into the update direction of model parameters, ensuring that optimization resources can be accurately focused on error correction of key logical levels. It should be understood that although this embodiment divides each function into independent modules, in actual engineering implementation, these modules can be integrated into a unified training framework component, or deployed on different computing nodes in a distributed service manner. As long as the overall data flow and processing logic is executed, it should be covered within the protection scope of this application.

[0064] Furthermore, this embodiment also provides an electronic device, including a memory and a processor. The memory stores a computer program; the processor executes the computer program to implement the large language model training method as described in any of Embodiments 1 to 5. Specifically, this electronic device constitutes the physical hardware implementation carrier of the technical solution of this application. The memory can be volatile memory (such as RAM) or non-volatile memory (such as ROM, flash memory, hard disk, etc.) for persistently storing the instruction code and related data for implementing the above-mentioned training method. The processor can be a general-purpose processor (such as a CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a graphics processing unit (GPU), etc., which instantiates the aforementioned sequence acquisition, fact verification and mask generation, preference optimization, and other functional logic by reading and executing the computer program in the memory. In practical applications, the electronic device may also include conventional components such as communication interfaces and input / output devices to support functions such as data loading, model interaction, and result display. Through this combination of hardware and software, the training method provided by this application can be implemented and run on specific computing devices, thereby effectively improving the factual consistency of large language models in complex structured reasoning tasks.

[0065] The above description is merely a specific implementation of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of variations or substitutions within the technical scope disclosed in this application. For example, the traditional Chinese medicine diagnosis and treatment scenario in the foregoing embodiments can be replaced with other structured reasoning tasks such as legal case analysis or industrial fault diagnosis; the binary mask matrix can be adjusted to a continuous confidence mask; or the fixed hierarchical weight configuration can be adjusted to a dynamic weight adjustment mechanism based on verification feedback. As long as these modifications do not deviate from the core technical concept of word-level hierarchical masking and differentiated preference optimization in this application, they should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a large language model, characterized in that, include: Obtain candidate response sequences generated by a large language model for a structured reasoning task. The candidate response sequences consist of multiple lexical units and contain reasoning units corresponding to multiple logical levels. Multi-level factual verification is performed on each reasoning unit in the candidate response sequence based on an external knowledge base, and a word-level hierarchical mask matrix is ​​generated based on the verification results. The word-level hierarchical mask matrix represents the factual correctness of each word in the candidate response sequence at different logical levels. Based on the lexical-level hierarchical mask matrix and the preset hierarchical weights, the masked weighted log probability is calculated in the preference optimization training to differentiate the generation probability of different logical levels in the large language model.

2. The method according to claim 1, characterized in that, The step of performing multi-level factual verification on each reasoning unit in the candidate response sequence based on an external knowledge base, and generating a word-level hierarchical mask matrix based on the verification results, includes: Extract the structured conclusion portion and the thought chain reasoning portion from the candidate response sequence; Each reasoning unit in the structured conclusion section is factually verified to obtain the conclusion verification results; Based on the error verification of the reasoning unit at the target logical level in the structured conclusion section, the reasoning terminology corresponding to the target logical level in the thought chain reasoning section is marked as a fact error; Based on the verification results of the conclusion and the inference lexical units marked as factual errors, the lexical-level hierarchical mask matrix is ​​generated.

3. The method according to claim 2, characterized in that, The factual verification of each reasoning unit in the structured conclusion section includes at least one of the following verification dimensions: Consistency verification between input features and first-level conclusions; Consistency verification between the first-level conclusion and the second-level processing strategy; Consistency verification between the second-level processing strategy and the specific execution parameters; The specific execution parameters are verified to conform to the preset security constraint rules.

4. The method according to claim 1, characterized in that, Before calculating the masked weighted log odds during preference optimization training based on the lexical-level hierarchical mask matrix and preset hierarchical weights, the method further includes: For the same input command, obtain multiple candidate response sequences generated by the large language model; Based on the word-level hierarchical mask matrix and the hierarchical weights, calculate the comprehensive factual score of each candidate response sequence; The candidate response sequence with the highest comprehensive factual score is selected as the preferred response, and the candidate response sequence with the lowest comprehensive factual score is selected as the non-preferred response, thus constructing a preference data pair.

5. The method according to claim 4, characterized in that, The segment-level average log probability is calculated based on the word-level hierarchical mask matrix: in, For a complete sequence response, For sequence length, For the corresponding level, For the current strategy model, For the first Each word element, For the first The lexical sequence preceding each lexical unit, Indicates the first The next level A mask for each word. , represents the number of lexical units verified at this level, where . It normalizes the length of content at different length levels; Calculate the masked log-odds during preference optimization training: For the preferred and non-preferred responses in the preference data pairs and the preset hierarchical weights, the masked weighted log-probability of each logical level is calculated to differentiate the generation rate of different logical levels in the large language model. The calculation formula is as follows: in, and These are respectively the preferred response and the non-preferred response. and These are the preferred and non-preferred responses, respectively. Hierarchical mask vectors, For the same input command, Let j=0 represent the total number of logical levels, j=0 represent the complete answer level dimension, retain the complete response sequence, and j=1……K represent the K levels in the labeled responses. Based on the weighted log odds of each logical level and the corresponding level weights, a multi-granularity contrastive loss function is constructed to update the parameters of the large language model.

6. The method according to claim 1, characterized in that, Before obtaining the candidate response sequence generated by the large language model for the structured reasoning task, the method further includes: Construct a multi-level instruction fine-tuning dataset that includes a thought chain reasoning part and a structured conclusion part; The pre-trained large language model is supervised and fine-tuned using the multi-level instruction fine-tuning dataset to obtain the initial policy model. ; The large language model is the initial strategy model.

7. The method according to claim 6, characterized in that, The external knowledge base is constructed in the following ways: Based on multi-source domain data, entities and relationships are extracted to construct a knowledge graph containing hierarchical mapping relationships; A rule base containing hard constraint rules is built based on domain expert rules; The external knowledge base includes the knowledge graph and the rule base.

8. A reasoning method for a large language model, characterized in that, include: Obtain the input data of the target object; The input data is input into a large language model to generate a thought chain reasoning part and a structured conclusion part containing multiple logical levels. The large language model is trained by the training method as described in any one of claims 1 to 7. Output the structured conclusion section, along with the comprehensive factual score of the factual confidence score and response sequence corresponding to each logical level.

9. A training device for a large language model, characterized in that, include: The sequence acquisition module is used to acquire candidate response sequences generated by the large language model for structured reasoning tasks. The candidate response sequences consist of multiple word units and contain reasoning units corresponding to multiple logical levels. The fact verification and mask generation module is used to perform multi-level fact verification on each reasoning unit in the candidate response sequence based on an external knowledge base, and generate a word-level hierarchical mask matrix based on the verification results. The word-level hierarchical mask matrix represents the factual correctness of each word in the candidate response sequence at different logical levels. The preference optimization module is used to calculate the masked weighted log probability during preference optimization training based on the word-level hierarchical mask matrix and preset hierarchical weights, so as to differentiate the generation probability of different logical levels in the large language model.

10. An electronic device, comprising: Memory, used to store computer programs; A processor for implementing the method as described in any one of claims 1 to 7 when executing the computer program.