A large language model credibility alignment method for financial risk control

Abstract

The application relates to the technical field of computer systems based on specific calculation models, and discloses a large language model credibility alignment method for financial risk control, which comprises the following steps: bidirectional implication verification is performed on candidate fact assertions generated by a large language model and the information sources thereof, the candidate fact assertions are converted into structured data after the verification, and the structured data is stored in a temporary fact account book established for a current task; then, when subsequent text is generated, the consistency of candidate content with the account book is checked, and the generation process of candidate content that constitutes a contradiction is intervened in real time; the internal calibration loop from source verification to generation process constraint is constructed; the inherent logic self-consistency of a calculation model is directly and technically associated with external objective facts; a novel conversation-level logic introspection and fact grounding capability of the model is improved; and the inherent generation logic forgetting defect of the model is solved.

Description

TECHNICAL FIELD

[0001] The application relates to a large language model credibility alignment method for financial risk control, and belongs to the technical field of computer systems based on specific computing models. BACKGROUND

[0002] Currently, when such computing models are applied to financial risk control and other tasks that have strict requirements for logical coherence, there is an inherent mismatch between the core working mechanism of the model and the task objective; specifically, when the model generates each subsequent token, its optimization direction is to make the generated sequence statistically closest to the distribution pattern of the training corpus, rather than strictly following the statement that has already been generated by itself in the current task. This running mechanism will cause the content of the text output by the model to have contradictory statements before and after when performing tasks that require long and coherent analysis. To address this phenomenon, external verification or post-processing methods are usually used, such as comparing the independent assertions output by the model with an external knowledge base, or manually or automatically reviewing the entire text after the model has completed all text generation.

[0003] Even if the prior art focuses on optimizing the classification or understanding ability of the model itself, it often ignores the necessity of real-time constraints on the behavior of the model in specific generation tasks, resulting in the persistence of logical contradictions. For example, the Chinese invention patent with the authorization publication number CN120429444B discloses a text knowledge base classification management method based on a large language model, which attempts to use the degree of interference between categories and the attention of the model to distinctive information as a basis for judgment, and then adjusts the learning rate during the training process to improve the classification accuracy of the model for specific domain text. However, the essence of this method still remains in the optimization of the model during the training phase, aiming to improve its macro accuracy in discrete classification tasks. It does not address the inherent limitations of large language models when generating long texts continuously, as their core mechanism focuses more on the statistical fluency of local text rather than the logical consistency of global information, resulting in possible contradictions in the content of the text before and after. Specifically, although this training-level optimization can help the model better distinguish static features of different categories, it cannot establish a logical benchmark that the model must follow during the dynamic process of a single generation task. Therefore, a model optimized by such a method may still produce contradictory statements in the content of the text before and after when generating a financial risk control report, and cannot guarantee the logical coherence and factual reliability of the generated content at the session level.

[0004] Analysis of the aforementioned existing technologies reveals the following shortcomings: 1. Comparison with external knowledge bases is primarily used to verify the accuracy of individual factual assertions, but it cannot assess the coherence between logical nodes along the internal reasoning path of the model within a complete generation task; 2. Post-review methods allow the model to expend computational resources to generate text that will ultimately be discarded due to internal logical contradictions, resulting in inefficient allocation of computational resources. These shortcomings indicate that existing technologies lack a technical solution capable of establishing and executing dynamic, task-specific internal logical consistency constraints within the text generation process of the computational model. Therefore, the technical problem this invention aims to solve is how to design a method that enables the large language model to identify its own output key factual assertions in real time during a single text generation task, using these assertions as dynamic logical benchmarks for the current task, and performing consistency checks based on these benchmarks when generating subsequent text, thereby preventing the model from outputting content with contradictory internal logic. Summary of the Invention

[0005] This invention provides a credibility alignment method for large language models for financial risk control. Its main purpose is to solve the problem that existing technologies lack a mechanism to ensure real-time internal consistency of computational models by referring to their own output content during the generation process.

[0006] To achieve the above objectives, this invention provides a method for aligning the credibility of large language models for financial risk control, comprising the following steps:

[0007] When using a large language model to perform a text generation task based on the input information source, an arbitration step is first performed on the candidate fact assertions generated by the large language model. The arbitration step includes: extracting the original text content corresponding to the candidate fact assertion from the input information source, and verifying whether there is a two-way logical implication relationship between the original text content and the candidate fact assertion.

[0008] Under the condition that the bidirectional logical implication relationship holds, the candidate fact assertion is transformed into a machine-readable structured triple containing a subject, attribute, and value, and stored in a temporary fact ledger created only for this text generation task;

[0009] During the process of generating subsequent text using a large language model, a natural language inference model is invoked for one or more candidate generated content to determine whether the candidate generated content constitutes a logical contradiction with any factual assertions already stored in the temporary fact ledger.

[0010] In one or more steps of a decoding algorithm, if the result of the judgment is that a logical contradiction exists, the logarithmic probability value of the candidate generated content is reduced to below a preset threshold that is sufficient to prevent it from being selected by the decoding algorithm.

[0011] Preferably, further comprising: when storing the structured triple into the temporary fact ledger, also storing a source data identifier pointing to a specific data point in the input source that gave birth to the structured triple, in association with the structured triple; and monitoring in parallel whether the data point pointed to by the source data identifier has changed; if changed, marking the structured triple directly associated with the source data identifier, and all downstream fact assertions derived from the structured triple, as invalid in the temporary fact ledger, and preventing the structured triple and downstream fact assertions marked as invalid from participating in subsequent logical contradiction judgment.

[0012] Preferably, further comprising: before the arbitration step, also determining a confidence score between 0 and 1 for the candidate fact assertion according to its expression; and reducing the log-likelihood value of the candidate generated content below a preset threshold, the strength of the reduction being proportional to the confidence score of the fact assertion in the temporary fact ledger that is refuted by the candidate generated content, the relationship being determined by the following formula: wherein, is the reduced final log-likelihood value, is the original log-likelihood value, is a preset base penalty coefficient, and is the confidence score of the refuted fact assertion in the temporary fact ledger.

[0013] Preferably, further comprising a contradiction probing step, which is executed before the step of determining whether the candidate generated content constitutes a logical contradiction, and includes: pausing the generation when the large language model generates text word by word to a semantic critical point of a syntactic structure, the semantic critical point being the moment when the subject and predicate have been generated but the key object or adverbial has not been generated; using a prediction model to generate one or more hypothetical concept completions of the subsequent core semantics of the text fragment based on the generated text fragment; pre-verifying the hypothetical concept completions with the temporary fact ledger to determine whether the current generation path has the risk of leading to a logical contradiction; and the step of reducing the log-likelihood value of the candidate generated content is executed in advance based on the determination result of the risk.

[0014] Preferably, the log-likelihood value of the candidate generated content is reduced below a preset threshold, specifically, in a beam search decoding algorithm, the log-likelihood value of the candidate generated content that constitutes a logical contradiction is directly set to a negative infinity value.

[0015] Preferably, the verification of whether the original content and the candidate fact assertion constitute a bidirectional logical entailment relationship specifically includes: performing a first verification, in which a natural language inference model is called to determine whether the original content and the candidate fact assertion constitute an entailment relationship when the original content is taken as a premise and the candidate fact assertion is taken as a hypothesis; and performing a second verification, in which the natural language inference model is called to determine whether the original content and the candidate fact assertion constitute an entailment relationship when the candidate fact assertion is taken as a premise and the core information of the original content is taken as a hypothesis.

[0016] Preferably, the subject of the structured triple is an entity, the attribute is a financial feature of the entity, and the value is a specific statement of the financial feature.

[0017] Preferably, the determination of the confidence score for the candidate fact assertion is performed by a confidence classifier, which generates the confidence score based on an analysis of modal verbs and adverbs contained in the candidate fact assertion.

[0018] Preferably, the step of generating the candidate fact assertion is implemented by a lightweight text classification model combined with keyword matching and regular expressions, and the step is configured to capture statements containing explicit financial indicators and numerical values, strong causal association words, or qualitative judgments in the text.

[0019] Preferably, after marking the downstream fact assertion as invalid, the method further includes guiding the large language model to regenerate a completely new text sequence based on the changed data point.

[0020] Compared with the prior art, the present application has the following beneficial effects:

[0021] 1. When processing text by using a large language model, first, a bidirectional logical entailment relationship between the original source and the assertion initially generated by the model is verified, and then the verified assertion is stored in a temporary fact ledger established for the task. In the subsequent text generation process, any candidate content must be consistency checked with the assertion that has passed the source verification in the ledger. This process directly links the pre-arbitration link of fact identification and the consistency constraint link of content generation, and constitutes a logically closed and traceable internal calibration loop from the source to the final output. As a result, the text generation behavior of the model in a single task changes from a sequence construction along a probability path without constraints to a self-correction process continuously referring to verified facts, so that the evolution of the internal state of the computing model when performing a task is directly technically associated with external objective information sources.

[0022] 2. When storing factual assertions in the fact ledger, not only is their content recorded, but a confidence level is also assigned to them, and a source data identifier pointing to their original source is associated with them. When the model generates subsequent content and performs consistency checks, the intensity of the intervention is directly related to the confidence level of the refuted fact. At the same time, a parallel monitoring mechanism monitors the data status pointed to by the source data identifier. Once the source data changes, all downstream factual assertions in the ledger derived from it will immediately become invalid. This method of synchronously managing the confidence and timeliness attributes of assertions enables the computational model to overcome complex inputs that mix deterministic and non-deterministic static and dynamic information. Its internal logical constraints are no longer singular and rigid, but rather it gains a multi-dimensional adaptability that matches the complexity of the external information environment.

[0023] 3. When the text generation reaches the preset semantic threshold in the sentence, the generation will be paused, and a prediction model will be used to generate several hypothetical concepts to complete the subsequent content. These hypothetical concepts will be quickly pre-verified against the fact ledger to determine whether there is a risk of logical contradiction in the current generation path. Based on the risk judgment result, the generation process of the path will be intervened in advance. This move decomposes the consistency verification action that was originally located at the end of the generation into a risk detection stage in the middle of the generation and a final verification stage. Therefore, the consistency verification mechanism not only ensures the logical accuracy of the output content, but also acts as a control signal for optimizing the allocation of computing resources. It acts on the generation process itself to guide the computing model to actively avoid those generation paths that will invest computing resources but are likely to be discarded due to logical contradictions. Attached Figure Description

[0024] Fig. 1 This is a schematic diagram of the overall process and internal calibration loop of the method of the present invention;

[0025] Fig. 2 This is a graph showing the performance impact and calibration curves of the basic penalty coefficient in this invention.

[0026] Fig. 3 This is a sequence diagram of the component interactions for the fact assertion arbitration step of this invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the scope of protection of the present invention.

[0028] The application provides a large language model credibility alignment method for financial risk control, which is a software method running in a computer system based on a specific calculation model, mainly including a fact assertion arbitration step, a dynamic fact storage step, a consistency verification step and a generation process intervention step, which cooperates to build a real-time calibration loop from source verification to generation process constraint in the calculation model, and the purpose is to solve the problem that the output content may have logical contradictions due to the inherent sequence generation mechanism of the calculation model when performing financial risk analysis and other long text generation tasks that require high logical coherence; in a specific application scenario, the method is used to generate an analysis report on the short-term solvency of a specified enterprise based on the quarterly financial report of the enterprise as the input source; in order to deal with the risk that the speculative expression in the input source may be misidentified as a fact in the generation process, or the details of the original data are distorted in the summary process, the method first performs the fact assertion arbitration step, when the large language model generates a candidate fact assertion after processing the input source, for example, the asset-liability ratio of the company in the third quarter is 60%, the assertion is not directly adopted, but placed in a temporary pending area, and an identifier pointing to its specific location in the original financial report document is attached; then, a source-assertion bidirectional implication verification mechanism as the arbitration core is called, which performs double verification through a natural language inference model fine-tuned on financial field corpus, the first verification is to verify whether the former logically implies the latter, that is, the asset-liability ratio of the company in the third quarter is 60%, based on the premise that the total liabilities of the company are 600 million yuan and the owner's equity is 400 million yuan at the end of the reporting period, which is shown in section 5.2 of the original content of the financial report, and the second verification is to verify whether the former logically implies the latter, that is, the asset-liability ratio of the company in the third quarter is 60%, based on the premise that the core information of the original text is not omitted or distorted, only when the candidate fact assertion passes both verification, it is proved that it constitutes an information equivalent restatement with the source, and the arbitration step determines that it is a qualified fact assertion.

[0029] After a candidate fact assertion passes arbitration, the dynamic fact storage step is executed. The system normalizes this qualified assertion, converting it into a machine-readable structured triple consisting of a subject, attribute, and value, specifically {subject: the company, property: debt-to-equity ratio, value: 60%}. This triple is then stored in a temporary fact ledger dynamically created in memory solely for this analysis report generation task. To address the potential for dynamic changes in information within financial scenarios, when storing this structured triple in the ledger, the system also assigns a source data identifier pointing to its originating data point, such as a financial identifier. Report Q3_Item_5.2, associated with this triplet for storage. A parallel source data monitor will track whether the data point pointed to by this identifier has changed. If the value of this item changes due to subsequent input of updated financial data, the monitor will trigger a logical failure propagation mechanism. This not only marks the triplet directly derived from this data in the ledger as invalid, but also traverses the ledger to mark all downstream fact assertions derived from this invalid fact as invalid as well, to ensure that the consistency basis of the model can remain synchronized with changes in external facts. As the analysis report generation process continues, when the model needs to generate subsequent text, consistency... The consistency verification step and the intervention step in the generation process begin. In an implementation using a beam search decoding algorithm, when the model generates a candidate content, such as "Therefore, the company has low financial leverage and short-term debt repayment risk is controllable," this candidate content is not immediately output. Instead, it first passes through a consistency verifier. The verifier calls a natural language inference model to compare the candidate content with every fact assertion already stored in the temporary fact ledger to determine if there is a logical contradiction. For example, if the ledger already contains the record {subject: the company, property: short-term loans, value: significant increase}... The natural language inference model will determine whether the conclusion that the short-term debt repayment risk is controllable in the candidate content constitutes a logical contradiction with the fact. Once the judgment result is that a logical contradiction exists, the generation process intervention step is triggered. The system will intervene in the generation process of the candidate content in real time. Specifically, it will impose a penalty on the logarithmic probability value in the beam search algorithm, for example, by setting it directly to a negative infinity value, or reducing it to a preset threshold that is sufficient to prevent it from being selected by the decoding algorithm. This operation prunes the candidate content with logical contradiction on the generation path, thereby preventing the content that constitutes a logical contradiction from being finally output.

[0030] Further, to handle the ambiguity and uncertainty expressions commonly seen in financial texts, the method can also integrate a consistency check method with confidence weighting. Before a fact assertion is stored in the ledger, a confidence classifier analyzes its expression. For the assertion that the company's cash flow seems to be weakening, the classifier determines a confidence score between 0 and 1, such as 0.4, based on the modal adverb seems. Accordingly, when consistency checking is performed and a contradiction penalty is applied, the strength of the penalty will be associated with the confidence score of the refuted fact assertion. The relationship can be determined by the following formula: wherein, is the final log-odds value after the penalty, is the original log-odds value, is a preset base penalty coefficient, and is the confidence score of the refuted fact assertion in the temporary fact ledger. In this way, refuting a certain fact with a confidence score of 1.0 will be punished with greater strength, while refuting a speculative assertion with a confidence score of 0.4 will be punished with less strength, allowing the model to reason more flexibly with uncertain information without violating core facts. In addition, to improve computational efficiency, the method can also introduce a contradiction exploration mechanism that executes before complete consistency checking. It will pause generation at a semantic critical point of the text generated by the model to a syntactic structure, such as after generating the fragment based on its tight cash flow, the company's future credit rating will be. A lightweight prediction model quickly generates several hypothetical conceptual completions for the subsequent core semantics, such as {improve, remain unchanged, and down}. The system then performs a quick pre-check with these hypothetical concepts and the fact ledger. If all high-probability subsequent concepts are found to potentially contradict the fact of cash flow tightness in the ledger, it is determined that the current generation path has a risk of directional logical contradiction. Based on this risk judgment result, a punitive adjustment is made to the score of the generation path, thereby guiding the decoding algorithm to avoid this generation path that is likely to lead to logical contradiction before investing more computational resources.

[0031] Example 1: In an automated due diligence scenario for a potential M&A transaction, a computational model is configured to generate a comprehensive risk assessment report based on a dynamic data room containing hundreds of documents, which is unique in that it contains early financial projections and business outlooks provided by the target company's management, and later legal documents containing a large number of risk warnings and pending litigation disclosures from third-party auditing agencies. This situation poses a challenge to the computational model's ability to maintain logical consistency in the final report when dealing with the time sequence and content conflicts of information flow; in the initial stage of the task, the model first processes the early financial projection documents, and when generating the summary text, the method's fact assertion arbitration step is triggered. A candidate fact assertion, i.e., the target company's projected annual compound growth rate for the next three years is 15%, is generated and sent to the pending review area. Through the source-assertion bidirectional entailment verification mechanism, it is confirmed that it has data support in the original source document and does not distort the original meaning. Subsequently, the assertion is transformed into a structured triple {subject: target company, property: projected annual compound growth rate, value: 15%} by the dynamic fact storage step and stored in the temporary fact ledger created for this report generation task. Based on this, the model generates a positive description of the target company's growth in the opening section of the report. All these early-generated text contents successfully pass the consistency check step because they do not logically contradict this fact assertion in the ledger.

[0032] As the processing proceeds, the model begins to analyze a late-joined legal due diligence report, in the process, the fact assertion arbitration step identifies a key information again, and after bidirectional entailment verification, a new fact assertion that the target company faces a potential environmental protection lawsuit whose maximum possible compensation amount accounts for 80% of the company's net assets is normalized as {subject: target company, property: potential environmental lawsuit liability, value: accounts for 80% of net assets} and stored in the temporary fact ledger; At this time, a common technical problem occurs, that is, the calculation model tends to continue the previous optimistic tone when pursuing local text fluency, and a model without the mechanism of the present application may generate a candidate content such as in conclusion, the financial fundamentals of the target company are stable, and the overall risk of this merger and acquisition is controllable in the conclusion section of the report; However, under the operation of the present method, the candidate content is immediately processed by the consistency verification step after being generated, the natural language inference model compares the candidate content with all records in the temporary fact ledger, and determines that the candidate content and the fact assertion about the potential environmental lawsuit liability in the ledger constitute a logical contradiction; Then, the generation process intervention step is activated, and the system sets the logarithmic ratio value of the candidate generated content which constitutes a logical contradiction to a negative infinity value in a beam search decoding algorithm, resulting in the generation path being immediately abandoned; This intervention makes the calculation model have to abandon the path of continuing the previous tone and explore other generation possibilities that are logically consistent with all facts in the temporary fact ledger, and finally, the model generates an alternative logically consistent concluding text, for example, although the target company shows business growth potential, the disclosed potential environmental lawsuit constitutes a major financial risk of the transaction, which may have a substantial negative impact on the value of the company after the merger; In this way, through the synergistic effect of the two technical features of pre-arbitration of fact assertions and real-time consistency constraints in the generation process, the final generated analysis report not only avoids logical contradictions caused by the order of information input, but also converts an open-domain text continuation task into a reasoning process within a closed logical boundary with constraints, and the output text sequence therefore has the required internal logical coherence in the field of financial risk control; On the basis of the scenario of the present embodiment, at T1 when the automated due diligence system completes the initial report generation, an audit file in the data room as a core source is updated, specifically, one contingent liability of the target company is confirmed as an actual liability, resulting in a change of its asset-liability ratio from 60% to 75%; The source data monitor built into the system detects this change immediately by comparing the checksums of the source files, and immediately triggers the logical failure transmission mechanism, marking the fact assertion {subject: target company, property: asset-liability ratio, value: 60%} in the temporary fact ledger derived from the old version of the file and all downstream fact assertions derived from this assertion as invalid;Subsequently, the large language model is guided to automatically regenerate a brand-new text sequence consistent with the current fact base based on the changed data points, replacing all analyses and conclusions in the report based on outdated data, thereby ensuring the timeliness and accuracy of the final report content.

[0033] Example 2: To objectively verify the effect of the method of the present application in inhibiting the generation of self-contradictory content by the computing model, a comparative test was designed and performed, which aimed to quantitatively evaluate the improvement of the logical consistency of the output text of the computing model when performing long text generation tasks containing inherent information conflicts after the introduction of the method; a standardized evaluation environment was built for this test, and the test platform was based on a pre-trained large language model with a parameter size of 7 billion, and two different running modes were configured as test groups, among which the control group used the original configuration of the model and combined with a standard beam search decoding algorithm with a beam width of 5 for text generation; the test group integrated the complete method of the present application including the fact assertion arbitration step, the dynamic fact storage step, the consistency checking step and the generation process intervention step on the basis of the control group; the data set used in the test was a set of financial analysis corpus containing 100 test cases constructed internally, each case contained an input source of about 1000 words, and the structure was set to contain 5 explicit positive fact assertions in the first half of the document, and 5 negative fact assertions which constituted potential logical contradictions with the former in the second half; the test task required the model to generate a summary analysis report with a length of no less than 500 words for each case.

[0034] The evaluation index of the experiment is the logical contradiction rate, which is defined as the percentage of the number of reports containing at least one self-contradictory statement in the final generated summary analysis report among all test cases. An independent natural language-based inference model automated judgment program is used for preliminary screening, and cross-validation is performed by three annotators. During the experiment, for a typical test case, the input source clearly states in the first half that the cash flow of Company A is sufficient, and in the second half, it discloses that the short-term debt risk of Company A is high. The control group model generates a statement consistent with the first half of the information in the opening of the report. According to the source, the cash flow of Company A is healthy, but in the conclusion part of the report, a sentence is generated that contradicts the previous statement, and the company's short-term debt pressure is small. In contrast, the experimental group model has stored two arbitrated fact assertions in the temporary fact book when processing the same case: {subject: Company A, property: cash flow, value: sufficient} and {subject: Company A, property: short-term debt risk, value: high}. Therefore, when the decoding algorithm generates the candidate content that the company's short-term debt pressure is small in the conclusion part, the consistency verification step determines that this content constitutes a logical contradiction with the fact assertion of high short-term debt risk in the book, and immediately triggers the generation process intervention step, which punishes the logarithmic odds ratio value of the candidate content, causing it to be abandoned by the decoding algorithm. The model finally generates alternative text that is not contradictory to all facts in the book. See Table 1, which lists the processing of 10 representative test cases.

[0035] Table 1: Comparison of behaviors of different test groups in processing representative test cases.

[0036]

[0037] After processing all 100 test cases, it is found that, among the 100 reports generated by the control group, 87 reports are determined to have at least one logical contradiction, and the logical contradiction rate is 87%; while in the test group using the method of the present application, among the 100 reports generated, only 4 reports have logical contradictions due to the misjudgment of the natural language inference model, and the logical contradiction rate is 4%; the test data show that, compared with the control group using the standard generation method, the logical consistency of the output text of the test group using the method of the present application is obviously improved, proving that the method can effectively suppress the generation of self-contradictory text content by establishing dynamic internal logical consistency constraints in the process of text generation by the computing model; to further verify the effect of the method of the present application on the utilization of computing resources, a supplementary test is added based on the test in Embodiment 2, which divides the test group into two subgroups, wherein the test subgroup A enables the contradiction detection step, and the test subgroup B disables the step, and the rest of the configuration is exactly the same as that in the test group in Embodiment 2; the test index is the total computing resource consumption required for processing all 100 test cases, and the floating point operation number (FLOPs) is used as the unit of measurement; the test results show that the total consumption of the test subgroup B is FLOPs, and the total consumption of the test subgroup A is FLOPs, and the computing resource consumption is reduced by about 33.3%; the data show that the contradiction detection step reduces the computing resources invested by the computing model in generating invalid text sequences that will be discarded in the early stage of the generation path by pre-identifying and pruning candidate contents that are likely to lead to logical contradictions.

[0038] Embodiment 3: This embodiment combines Figs. 1 to 3 a method for aligning the credibility of a large language model for financial risk control, as shown in Fig. 1 , the process starts with receiving an input source, such as a corporate financial report, and then a large language model (LLM) is used to preliminarily refine the fact statements to generate candidate fact assertions, which are not directly adopted but subjected to a source-assertion bidirectional entailment verification process. If the verification fails, it returns to retry, and if the verification passes, the assertion is converted into a structured triple and stored in a temporary fact ledger, which serves as the dynamic logical benchmark for this task. In parallel, a source data monitoring and invalidation mechanism monitors source changes and invalidates associated facts in the ledger. In subsequent text generation, the system can perform contradiction detection before the LLM generates subsequent candidate content to predict and avoid potential contradictions in the generation path. The generated candidate content must pass the consistency verification with the temporary fact ledger, and if there is no contradiction, the content is accepted and the final analysis report is output. If there is a contradiction, the generation process intervention step is triggered to adjust the intervention intensity based on the results of the confidence-weighted verification to reduce the generation probability of contradictory content and guide the LLM to regenerate text that is logically consistent with the ledger.

[0039] like Fig. 2 As shown, this provides a calibration basis for a key parameter in the intervention stage of the generation process. The horizontal axis in the figure represents the base penalty coefficient increasing from 1.0 to 20.0. The left side of the main vertical axis represents the conflict avoidance rate (%), and the right side of the secondary vertical axis represents the normalized text fluency. The solid curve in the graph shows that the conflict avoidance rate increases with the base penalty coefficient. The coefficient increases monotonically with increasing base penalty coefficient, while the dashed curve shows that text fluency increases with increasing base penalty coefficient. The curves increase and then decrease monotonically. The trends of the two curves clearly reveal the technical trade-off between improving logical consistency and maintaining the natural fluency of the text, providing a quantitative reference for selecting an engineering balance between the two.

[0040] like Fig. 3 As shown in the figure, this diagram illustrates the collaborative workflow among multiple system components, including a large language model, input source, original text extractor, natural language inference model, confidence classifier, and review area. The process begins with the large language model submitting a candidate fact assertion to the review area and requesting the corresponding original text content from the original text extractor. After obtaining the original text, the large language model sends the original text and assertion together to the natural language inference model to perform the first and second layers of verification. If the bidirectional entailment verification passes, the assertion is sent to the confidence classifier to calculate the confidence score, and the pass result with the confidence score is sent to the review area. If the verification fails, the natural language inference model directly returns a failure result to the review area. Finally, based on the received results, the review area returns the final status of whether the assertion is accepted or rejected to the large language model.

[0041] Example 4: In the preliminary stage of deploying the method of the present invention in an automated credit risk monitoring system that processes real-time financial news streams, its core natural language inference model and confidence classifier need to possess lightweight and domain-specific attributes. Furthermore, the basic penalty coefficient used in the method to adjust the intensity of contradiction penalties... The specific value of the coefficient affects the balance between maintaining logical consistency and ensuring naturalness of the generated text, and needs a reproducible procedure to determine; to meet this requirement, the embodiment adopts a systematic offline calibration and model construction procedure. First, for the construction of the natural language inference model, the procedure adopts the technical path of knowledge distillation, selects a large-scale general pre-training language model as the teacher model, and constructs a special dataset containing 100,000 pairs of sentences in the financial field, which are all labeled as implied, neutral or contradictory relationship; then, a lightweight MobileBERT is used as the basic architecture of the student model, and the teacher model's ability to judge the logical relationship of financial text is distilled into the student model through the special dataset, so as to obtain a natural language inference model with fast reasoning speed, small resource occupation and optimization for the financial context; similarly, the construction of the confidence classifier is also completed by using a dataset of 50,000 financial news sentences with labeled confidence scores to supervise the fine-tuning of the same lightweight model architecture.

[0042] Second, for the calibration of the basic penalty coefficient , the procedure performs a gradient optimization experiment, which selects 1000 pairs of sentences labeled as contradictory in the special dataset of the natural language inference model as the test set. The experiment process is as follows: the first sentence in each pair is taken as the input of the model, and the model is instructed to generate the subsequent text. At the same time, the value of the basic penalty coefficient is started from 1.0, with a step of 1.0, and increased to 20.0. At each value of , all 1000 test cases are run completely, and two performance indicators are recorded. The first is the contradiction avoidance rate, which is the percentage of cases where the model successfully prevents the generation of the second contradictory sentence. The second is the text fluency, which is the average perplexity score calculated for all successfully generated non-contradictory text outputs. Experimental data shows that when the value of increases from 1.0 to 15.0, the contradiction avoidance rate monotonically increases from 65.3% to 99.5%, while the average perplexity score of text fluency only deteriorates slightly. However, when the value of continues to increase from 15.0 to 20.0, the contradiction avoidance rate improves by less than 0.2%, while the average perplexity score deteriorates. Based on this data, to achieve a technical balance between contradiction avoidance rate and text fluency, the embodiment sets the basic penalty coefficient The engineering value of the is determined to be 15.0; by performing the above procedures, this embodiment not only obtains a lightweight and reliable core model component suitable for the target application scenario, but also converts a key parameter affecting system behavior from an empirically set value to a specific value determined through a standardized experimental process and quantitative data analysis, thereby providing technical details for the reproducibility and engineering deployment of the entire method.

[0043] In the application of the method of the present application to the financial intelligence analysis scenario involving a large amount of unstructured text such as analyst comments and market rumors, to ensure that the fact assertions entering the temporary fact ledger have high credibility, this embodiment further includes a procedure for setting the acceptance threshold of bidirectional entailment verification in the fact assertion arbitration step; this procedure is performed on an offline verification data set containing ten thousand pairs of financial domain text sentences, each pair consisting of a source text and a candidate fact assertion, and having been annotated by domain analysts with a fact fidelity score between 0 and 100.

[0044] During the execution of this calibration procedure, the system inputs each sentence pair in the verification data set into the constructed natural language inference model to perform bidirectional entailment verification and records the confidence scores output by the model in both verification directions; then, the system takes the geometric mean of the two confidence scores as the final comprehensive entailment score, and plots the performance curve between the comprehensive entailment score and the manually annotated fact fidelity score; by analyzing the performance curve, the point at which the precision and recall reach a pre-set balance point is selected, and the comprehensive entailment score corresponding to this point, i.e. 0.95, is set as the final acceptance threshold for the fact assertion arbitration step; in subsequent online operation, only when a candidate fact assertion has a comprehensive entailment score with its source text greater than or equal to the calibration threshold of 0.95, is it allowed to be stored in the temporary fact ledger, and this procedure provides a data-driven determination basis for the judgment criterion of the access control for fact identification.

[0045] In the application of the method of the present application to the financial intelligence analysis scenario involving a large amount of unstructured text such as analyst comments and market rumors, to ensure that the fact assertions entering the temporary fact ledger have high credibility, this embodiment further includes a procedure for setting the acceptance threshold of bidirectional entailment verification in the fact assertion arbitration step; this procedure is performed on an offline verification data set containing ten thousand pairs of financial domain text sentences, each pair consisting of a source text and a candidate fact assertion, and having been annotated by domain analysts with a fact fidelity score between 0 and 100.

[0046] During the execution of the calibration procedure, the system will verify each sentence pair in the dataset, input into the constructed natural language inference model to perform bidirectional entailment verification, and record the confidence score of the model output as an entailment relationship in both verification directions; then, the system takes the geometric mean of the two confidence scores as the final comprehensive entailment score, and draws the performance curve between the comprehensive entailment score and the artificially annotated fact fidelity score; by analyzing the performance curve, the position that reaches the preset balance point between precision and recall is selected, and the comprehensive entailment score corresponding to the point, that is, 0.95, is set as the final acceptance threshold of the fact assertion arbitration step; in the subsequent online operation, only when the comprehensive entailment score of a candidate fact assertion and its source is greater than or equal to the calibration threshold 0.95, the candidate fact assertion is allowed to be stored in the temporary fact ledger, and this procedure provides a quantifiable, data-driven determination basis for the access control of fact identification.

[0047] In order to further verify the necessity and technical effect of the internal calibration loop for ensuring the logical consistency of the generated content as the core technical feature in the method of the application, the following comparative examples are provided.

[0048] Comparative Example 1: This comparative example aims to verify the technical performance of a large language model in the prior art in handling financial analysis tasks containing inherent information conflicts, in the absence of the internal calibration loop of the application from source verification to generation process constraints; the experimental conditions adopted in this comparative example are consistent with the test group in Example 2 except for the specific differences pointed out below, specifically, this comparative example adopts the same pre-trained large language model as in Example 2 with a parameter size of 7 billion, and configures a standard beam search decoding algorithm with a beam width of 5; the essential difference lies in that the large language model in this comparative example does not integrate any of the claimed financial risk control-oriented large language model credibility alignment methods of the application, including fact assertion arbitration, dynamic fact storage, consistency verification, and generation process intervention, but adopts the conventional unconstrained text generation method in the art; the task of this comparative example is to require the model to generate an investment risk summary of no less than 500 words based on a comprehensive due diligence report on target company C (input source) constructed for this test, which contains potential logical conflicts in different chapters of technical facts, as follows, source fact 1 (report from market and sales department): clearly states that the target company C's main product X series chip has successfully signed a five-year exclusive supply agreement with the downstream leading automobile manufacturer Giant Auto, and is expected to account for more than 80% of the procurement share of Giant Auto's similar components, providing stable revenue for the company in the next five years; source fact 2 (report from supply chain risk review department): clearly states that the only supplier of core raw material K material for X series chip, upstream company S, has recently been ordered to suspend production for rectification due to environmental problems, and the resumption date is still uncertain, and the target company C's K material inventory can only maintain normal production for 30 days, and has not found a replacement supplier with equivalent quality and capacity.

[0049] During the execution of the experiment, the large language model of the comparative example adopts the conventional technical path of the comparative example, receives the above-mentioned input source and starts to generate the investment risk summary. In the opening part of the report, the model first generates the following positive statement based on the source fact 1: the core competitiveness of the target company C lies in the stable market position of its main product X series chip, and the company has locked a five-year exclusive supply agreement with the industry giant Giant Car, which greatly guarantees the certainty and growth potential of the company's future revenue; then, when the processing process of the model advances to the latter half of the input source, it contacts the source fact 2 about the supply chain interruption; when generating the conclusion part of the report, the model generates the following candidate content: in summary, the target company C, relying on the long-term exclusive supply agreement of its key product, shows strong market control and financial stability, and its investment risk level is evaluated as low; due to the lack of the real-time internal consistency constraint mechanism of the present application, the above-mentioned candidate content, although it maintains coherence in the local context with the positive statement in the opening part of the report, constitutes a serious logical contradiction with the source fact 2 about the core raw material supply interruption that the model has already processed. The candidate content is selected by the beam search decoding algorithm and finally output. After the complete output report is submitted to a downstream automated portfolio management system, the system fails to identify the deep logical contradiction in the text and mistakenly maintains the risk exposure of the target company C at a low level, until it is found by manual review 15 working days later, triggering an emergency position adjustment, see Table 2.

[0050] Table 2: Comparison table of experimental results.

[0051]

[0052] The experimental results of the comparative example show that, in the absence of the internal calibration loop from source verification to generation process constraint proposed by the present application, even if the basic model is the same, the conventional text generation method cannot process the input containing conflicting information in a single generation task in its technical principle, and may produce output content containing internal logical contradictions, thereby failing to meet the requirements of high logical coherence in serious application scenarios such as financial risk control.

[0053] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application.

[0054] Finally, it should be noted that the above examples are only used to illustrate the technical solutions of the present application and are not limiting. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application.

Claims

1. A method for aligning the credibility of a large language model for financial risk control, characterized in that, The method comprises the following steps: In the process of using a large language model to perform a text generation task according to an input source, first, a mediation step is performed on a candidate fact assertion generated by the large language model, the mediation step comprising: extracting original content corresponding to the candidate fact assertion from the input source, and verifying whether a bidirectional logical entailment relationship exists between the original content and the candidate fact assertion; Under the condition that the bidirectional logical entailment relationship exists, the candidate fact assertion is converted into a machine-readable structured triple containing a subject, a property and a value, and is stored in a temporary fact ledger created only for the current text generation task; In the process of the large language model generating subsequent text, for one or more candidate generated contents, a natural language inference model is called to determine whether the candidate generated content constitutes a logical contradiction with any candidate fact assertion stored in the temporary fact ledger; In one or more steps of a decoding algorithm, if the result of the determination is that a logical contradiction exists, the logarithmic probability value of the candidate generated content is reduced to below a preset threshold value that prevents it from being selected by the decoding algorithm; Further comprising: when the structured triple is stored in the temporary fact ledger, a source data identifier used to point to a specific data point in the input source that gave birth to the structured triple is also stored in association with the structured triple; and the data point pointed to by the source data identifier is monitored in parallel to determine whether it has changed; if it has changed, the structured triple directly associated with the source data identifier in the temporary fact ledger, as well as all downstream fact assertions derived from the structured triple, are marked as invalid, and the structured triple and downstream fact assertions marked as invalid are prevented from participating in subsequent logical contradiction determination; Verifying whether a bidirectional logical entailment relationship exists between the original content and the candidate fact assertion specifically comprises: performing a first verification, in which the natural language inference model is called to determine whether the relationship between the original content and the candidate fact assertion is an entailment when the original content is the premise and the candidate fact assertion is the hypothesis; and performing a second verification, in which the natural language inference model is called to determine whether the relationship between the original content and the candidate fact assertion is an entailment when the candidate fact assertion is the premise and the core information of the original content is the hypothesis.

2. The financial risk control-oriented large language model credibility alignment method according to claim 1, characterized in that, Further comprising: prior to the arbitration step, also determine a confidence score for the candidate fact assertion between 0 and 1 based on its representation, and reduce the log-odds value of the candidate generated content below a preset threshold by an amount proportional to the confidence score of the fact assertion in the provisional fact ledger that is refuted by the candidate generated content, the relationship being determined by the following formula: wherein, is the reduced final log-odds value, is the original log-odds value, is a preset base penalty coefficient, and is the confidence score of the refuted fact assertion in the provisional fact ledger.

3. The method of claim 1, wherein, Further comprising a contradiction exploration step, which is performed before the step of determining whether the candidate generated content constitutes a logical contradiction, and comprises: pausing generation when the large language model generates text word by word to a semantic critical point of a syntactic structure, the semantic critical point being a moment when the subject and the predicate have been generated but the key object or adverbial has not been generated; using a prediction model to generate one or more hypothetical concept completions of the subsequent core semantics of the text fragment based on the generated text fragment; pre-verifying the hypothetical concept completions against the temporary fact ledger to determine whether the current generation path has a risk of leading to a logical contradiction; and the step of reducing the logarithmic probability value of the candidate generated content is performed in advance based on the result of the determination. Further comprising:

4. The financial risk control-oriented large language model credibility alignment method according to claim 1, characterized in that, The log-likelihood ratio of the candidate generated content is reduced below a preset threshold, specifically, in a beam search decoding algorithm, the log-likelihood ratio of the candidate generated content constituting a logical contradiction is directly set to a negative infinity value.

5. The financial risk control-oriented large language model credibility alignment method according to claim 1, characterized in that, The subject of the structured triple is an entity, the property is a financial feature of the entity, and the value is a specific statement of the financial feature.

6. The financial risk control-oriented large language model credibility alignment method according to claim 2, characterized in that, A confidence score is determined for the candidate factual assertion by a confidence classifier that generates the confidence score based on analyzing modal verbs and adverbs included in the candidate factual assertion.

7. The method of claim 1, wherein the method is a method of financial risk control-oriented large language model credibility alignment. The step of generating the candidate factual assertion is implemented by a lightweight text classification model in combination with keyword matching and regular expressions, and the step is configured to capture statements in the text that include explicit financial indicators and numerical values, strong causal connection words, or qualitative judgments.

Images