Large language model knowledge verification and improvement method, equipment and medium

Abstract

The invention discloses a large language model knowledge verification and improvement method and device and a medium. The method comprises the steps that task description, specific problems and example data input by a user are received; in combination with a preset heuristic cue word and structured processing, generating a logic expression, and removing redundant contradiction rules to form a logic knowledge base; inputting specific questions into the large language model to generate preliminary answers, and performing symbolization conversion on the preliminary answers to extract entities, relationships and attributes so as to obtain symbolized propositions; matching the symbolized proposition with the logic knowledge base, if a conflict is matched, generating a candidate correction scheme through deductive reasoning, determining an optimal scheme according to a predefined correction cost function, feeding back the optimal scheme to the large language model, and regenerating an answer; and iterating the related steps until the answer passes verification, and outputting a final answer, a confidence score and an interpretability report. Through the deductive constraint and bidirectional feedback mechanism, the LLM output reliability is improved, the knowledge base cost is reduced, and dynamic updating and interpretability of knowledge are realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, device and medium for knowledge verification and improvement of large language models. Background Technology

[0002] With the rapid development of artificial intelligence and natural language processing technologies, large language models (LLMs), with their powerful semantic understanding and generation capabilities, have been widely applied in fields such as mathematical reasoning, legal document generation, and medical diagnostic assistance.

[0003] However, existing large language models have insufficient reliability in output. Probabilistic generation mechanisms are prone to "illusions," leading to factual errors or logical contradictions, making it difficult to meet the needs of high-precision scenarios. In addition, the lack of dynamic logical constraints means that the generated content often violates domain rules (such as contraindications for medical medication and legal regulations) and cannot be corrected autonomously. Furthermore, knowledge verification and management costs are high. Traditional manual review is inefficient. Although back-learning can correct errors by relying on a logical knowledge base, the knowledge base needs to be manually constructed by domain experts, which is not only time-consuming and labor-intensive but also difficult to cover complex scenarios. Moreover, static knowledge bases cannot be expanded as tasks change. Summary of the Invention

[0004] This application provides a method, device, and medium for knowledge verification and improvement of a large language model. The method includes: accepting a task description, a specific question, and example data input by a user; the example data being related to the task description and the specific question; inputting the task description and preset heuristic prompts into the large language model to generate logical rules related to the task description, and performing structured processing on the logical rules to obtain logical expressions; calculating the coverage of the logical expressions on the example data to eliminate redundant and contradictory rules, thereby obtaining a logical knowledge base; inputting the specific question into the large language model to generate a preliminary answer, and performing symbolic transformation on the preliminary answer to extract the entities, relations, and attributes of the preliminary answer. Obtain symbolic propositions; match the symbolic propositions with the logical knowledge base to detect whether there is a logical conflict between the symbolic propositions and the logical knowledge base; if a logical conflict is detected, generate multiple candidate correction schemes through reverse reasoning, and determine the optimal correction scheme according to a predefined correction cost function; the correction cost function is related to the proposition change, rule violation degree, and semantic deviation degree; feed the optimal correction scheme back to the large language model to regenerate the answer; iteratively execute steps 4 to 7 until the answer is verified, and obtain the final answer, confidence score, and interpretability report; the interpretability report includes the verification results, rule matching details, correction measures, and correction basis.

[0005] In one example, the task description and preset heuristic prompts are input into a large language model to generate logical rules related to the task description. These logical rules are then structured to obtain logical expressions. Specifically, this includes: guiding the large language model to analyze the input / output types, core objectives, and potential constraints of the task description based on the preset heuristic prompts, generating primary logical rules; using an iterative questioning strategy to further investigate whether the primary logical rules have any unconsidered constraints, thereby generating logical rules; and using a sequence-to-sequence model based on the Transformer architecture to structure the logical rules and generate first-order logical expressions.

[0006] In one example, the coverage of the logical expression on example data is calculated to eliminate redundant and contradictory rules, resulting in a logical knowledge base. Specifically, this includes: calculating the coverage of the logical expression on example data using a preset coverage formula to obtain a coverage result; identifying rules that are functionally redundant in the logical expression based on the coverage result through logical equivalence judgment and rule inclusion relationship analysis, and deleting them; identifying rule pairs with contradictory conclusions in the logical expression based on the coverage result by constructing counterexamples, and deleting them; and constructing a logical knowledge base based on the coverage result after eliminating redundant and contradictory rules.

[0007] In one example, the preliminary answer is symbolically transformed to extract the entities, relations, and attributes of the preliminary answer, resulting in a symbolic proposition. Specifically, this includes: accurately parsing the structured data in the preliminary answer using a syntactic analyzer based on a bottom-up operator priority analysis method to construct an abstract syntax tree; identifying entities in the free text of the preliminary answer using named entity recognition technology; identifying the syntactic relations between the entities using dependency parsing technology, and inferring the semantic relations between the entities based on the syntactic relations; and combining the entities, semantic relations, and attributes into a propositional form to form a symbolic proposition.

[0008] In one example, the symbolic propositions are matched with the logical knowledge base to detect whether there is a logical conflict between the symbolic propositions and the logical knowledge base. Specifically, this includes: binding the variables of the rules in the logical knowledge base with the specific entities in the symbolic propositions using pattern matching technology, and checking whether the preconditions of the rules are all satisfied in the set of symbolic propositions under the variable binding; if the preconditions of the rules are all satisfied, but the conclusion part is not found in the set of symbolic propositions under the variable binding, it is determined as a logical conflict; each detected logical conflict is evaluated to determine the severity of the conflict; the severity of the conflict is jointly determined by the importance weight of the violated rule itself, and whether the violation is a deterministic violation or a probabilistic violation.

[0009] In one example, if a logical conflict is detected, multiple candidate correction schemes are generated through reverse reasoning. Specifically, this includes: analyzing the logical conflict to determine whether it is a numerical error in the proposition itself or an error in the logical relationship between propositions; for the numerical error, modifying the parameters in the erroneous proposition through a reverse reasoning mechanism; for the logical relationship error, adding missing intermediate propositions to complete the reasoning chain through a reverse reasoning mechanism; generating multiple candidate correction schemes for each correction direction through a heuristic search strategy; verifying the feasibility of the candidate correction schemes, and storing the verified candidate correction schemes uniformly.

[0010] In one example, determining the optimal correction scheme based on a predefined correction cost function specifically includes: determining the component weights of the correction cost function according to the application domain of each candidate correction scheme; the correction cost function is a weighted sum of three components: proposition change, rule violation, and semantic deviation; and evaluating multiple candidate correction schemes based on the component weights and the predefined correction cost function to determine the candidate correction scheme with the minimum overall correction cost, which is then identified as the optimal correction scheme.

[0011] In one example, the method also includes: monitoring failed cases during the iteration process, calculating the cumulative failure score according to the preset scoring rules, and when the score exceeds the preset threshold, generating new rules by analyzing the failed cases and integrating them into the knowledge base.

[0012] On the other hand, embodiments of this application provide a large language model knowledge verification and improvement device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above-mentioned large language model knowledge verification and improvement methods.

[0013] On the other hand, embodiments of this application provide a non-volatile computer storage medium for verifying and improving large language model knowledge, which stores computer-executable instructions capable of executing any of the above-mentioned methods for verifying and improving large language model knowledge.

[0014] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: This application employs bidirectional matching verification through reverse reasoning and a logical knowledge base, combined with a modified cost function to select the optimal solution. It accurately detects factual errors and logical conflicts in the output of a large language model while maximizing semantic coherence, significantly improving the reliability and accuracy of the output content, effectively solving the "illusion" problem, and meeting the needs of high-precision scenarios. Heuristic prompts guide the large language model to automatically generate logical rules, which are then structured, hierarchically categorized, and redundant contradictions removed to form a knowledge base. This eliminates the need for manual rule writing, significantly reducing knowledge acquisition and management costs. A bidirectional feedback mechanism feeds error information and correction suggestions back to the large language model, guiding it to iteratively optimize its output and continuously evolving the knowledge base. This enhances the transparency of system decision-making and creates a collaborative, closed-loop process between the large language model and the knowledge base, improving the system's performance and stability in handling complex tasks over the long term. Attached Figure Description

[0015] To more clearly illustrate the technical solution of this application, some embodiments of this application will be described in detail below with reference to the accompanying drawings, in which: Figure 1 A flowchart illustrating a method for verifying and improving knowledge of a large language model provided in an embodiment of this application; Figure 2 A system architecture diagram of a large language model knowledge verification and improvement method provided in the embodiments of this application; Figure 3 A flowchart illustrating the reverse verification module processing of a large language model knowledge verification and improvement method provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of a large language model knowledge verification and improvement device provided in an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] Some embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart illustrating a method for verifying and improving knowledge from a large language model, as provided in an embodiment of this application. This method can be applied to different business domains. Certain input parameters or intermediate results in this process can be manually adjusted to help improve accuracy.

[0019] The analysis method involved in the embodiments of this application can be implemented by a terminal device or a server, and this application does not impose any special limitations on it. For ease of understanding and description, the following embodiments are all described in detail using a server as an example.

[0020] Based on this Figure 1 The process may include the following steps: S101: Accept user input of a task description, a specific problem, and sample data; the sample data is related to the task description and the specific problem.

[0021] In some embodiments of this application, the system input interface receives task descriptions, specific questions, and optional sample data provided by the user, and performs normalization processing and quality assessment on the input content.

[0022] First, the user input is structured and parsed to extract key semantic elements from the task description. For example, in mathematical reasoning tasks, task types such as "equation solving" and "logical proof" are identified, and in legal documents, application scenarios such as "lawsuit generation" and "contract review" are identified.

[0023] For example data, a quality detection mechanism based on semantic consistency and task relevance is adopted. By calculating the cosine similarity between the semantic embedding vectors of the example question and the target task description, as well as the degree of matching between the example answer and the task target, low-quality examples with noise or distribution bias are automatically identified and filtered.

[0024] It should also be noted that if the user does not provide an example, it has zero-shot start capability, relying entirely on the prior knowledge of the large language model for subsequent processing. Experimental data shows that the rule extraction accuracy can reach 68% in the case of zero examples, improves to 75% with one example, and tends to stabilize at around 90% with five examples, demonstrating strong small-shot adaptability.

[0025] S102: Input the task description and preset heuristic prompts into the large language model to generate logical rules related to the task description, and perform structured processing on the logical rules to obtain logical expressions.

[0026] In some embodiments of this application, after receiving user input data, the task description is combined with carefully designed heuristic prompts and input into a large language model to guide the model to generate task-related logical rules, and then converted into formal logical expressions through multi-stage processing.

[0027] Specifically, the first step is to identify the input and output characteristics of the task, such as the input of solving mathematical equations being the equation expression and the output being the solution set and verification process; the second step is to analyze the core objectives and constraints of the task, such as the need to follow mathematical identity transformation rules when solving equations; and finally, to summarize the logical principles that need to be followed.

[0028] Based on the above analysis, an iterative questioning strategy can be used to stimulate the model to generate a more complete set of rules. For example, after generating the initial rules, follow up by asking "Are there any unconsidered constraints or special cases?"

[0029] Furthermore, a Transformer-based sequence-to-sequence model is used to convert natural language rules into first-level logical expressions. This transformation model is trained on tens of thousands of rule-paired corpora and can accurately identify elements such as quantifiers and logical connectors.

[0030] It should also be noted that for newly emerging domain-specific concepts, a dynamic embedding strategy is adopted, which utilizes the contextual understanding capabilities of the large language model to generate symbolic representations for them, ensuring that the formal expression of the rules is both accurate and complete.

[0031] S103: Calculate the coverage of the logical expression on the example data to eliminate redundant and contradictory rules and obtain a logical knowledge base.

[0032] In some embodiments of this application, after obtaining the logical expression, a layered strategy is first implemented based on the rule complexity: the rules are divided into a basic layer (simple constraints), an intermediate layer (reasoning rules), and a complex layer (multi-step reasoning) according to indicators such as the number of logical operators, the depth of quantifier nesting, and the number of concepts involved.

[0033] Furthermore, a pre-defined coverage formula is used to evaluate the applicability of each rule on the example data, calculating its effective application ratio in the example set. The formula is:

[0034] in, Representing the The coverage of a rule, its value range is within Between these values, the larger the value, the wider the scope of application of the rule. Indicates the first in the knowledge base A set of logical rules, expressed in the form of Horn clauses in first-order logic. The total number of samples in the example set is a positive integer. Represents the first in the example set Each sample contains a pair of input questions and expected outputs. It is an indicator function, when the rule Applicable to samples The value is 1 if the summation condition is met, and 0 otherwise. (Summarization symbol) This indicates that the calculation is performed cumulatively for all samples.

[0035] To improve the robustness of this formula in small sample scenarios, when the number of examples... At that time, the system uses Laplace smoothing technology to correct the formula to ,in The smoothing parameter is typically taken as the square root of the total number of rules. This smoothing strategy can avoid coverage estimation bias caused by insufficient samples.

[0036] Based on coverage assessment, redundancy and contradiction detection are performed: rules with repeated functions are identified by logical equivalence judgment, such as "both sides of the equation can be added to the same number" and "both sides of the equation can be added to the same quantity" are considered redundant; opposing rules are detected by constructing counterexamples, such as the conflict between "all equations have solutions" and "some equations have no solutions".

[0037] The final knowledge base combines hierarchy and simplicity. During verification, simple rules can be applied first for quick screening, followed by in-depth checks using complex rules, significantly improving verification efficiency. The knowledge base also supports dynamic weight adjustment, with the importance of rules continuously optimized based on usage.

[0038] S104: Input the specific problem into the large language model to generate a preliminary answer, and perform symbolic transformation on the preliminary answer to extract the entities, relations and attributes of the preliminary answer to obtain a symbolic proposition.

[0039] In some embodiments of this application, after obtaining the logical knowledge base, the specific question is input into the large language model to generate a preliminary answer, and then converted into a symbolic propositional representation through a hybrid parsing strategy.

[0040] Specifically, different processing methods are adopted for different types of content: for structured content such as mathematical expressions and logical statements, a bottom-up operator priority analyzer is used for precise parsing to construct an abstract syntax tree to capture the complete logical structure.

[0041] For free text content, the system combines named entity recognition and dependency parsing techniques to extract syntactic and semantic relationships between entities. For example, for the answer "the solution to the equation is x=2", the system not only identifies the entities "equation", "x", and "2", but also extracts the "solution" relationship and constructs a predicate logic proposition of solution (equation, x, 2).

[0042] When processing multilingual content, a unified vector space of a multilingual pre-trained model is used to ensure semantic consistency, enabling the same concepts expressed in different languages ​​to obtain comparable symbolic representations. The entire symbolization process ultimately outputs a structured set of propositions, laying the foundation for subsequent logical verification.

[0043] S105: Match the symbolic proposition with the logical knowledge base to detect whether there is a logical conflict between the symbolic proposition and the logical knowledge base.

[0044] In some embodiments of this application, after obtaining the symbolic proposition, the symbolic proposition is deeply matched with the logical knowledge base to detect potential logical conflicts and assess their severity.

[0045] Specifically, a strategy combining forward and backward reasoning is employed. Forward reasoning starts from known propositions and applies rules to deduce new propositions, expanding the boundaries of knowledge. Backward reasoning works backward from the proposition to be verified to find supporting evidence and test its rationality. After binding rule variables with proposition entities using pattern matching technology, it is checked whether the preconditions of each rule are met and whether the conclusion is valid.

[0046] When a premise is met but the conclusion is not, it is marked as a logical conflict, and the severity is assessed according to the rule type and nature of the violation: violations of hard constraints (such as mathematical rules) directly lead to the rejection of the answer, while violations of soft constraints (such as format specifications) affect the confidence score.

[0047] It should also be noted that this application has established a conflict classification mechanism, which comprehensively considers the weight coefficient of the violated rule and the accuracy of the violation, to ensure that important rule violations receive higher attention and provide priority guidance for subsequent amendments.

[0048] S106: If a logical conflict is detected, multiple candidate correction schemes are generated through reverse reasoning, and the optimal correction scheme is determined according to a predefined correction cost function; the correction cost function is related to the proposition change, rule violation degree, and semantic deviation degree.

[0049] In some embodiments of this application, after a logical conflict is detected, candidate correction schemes are generated through reverse reasoning, and the optimal solution is determined based on multi-objective optimization.

[0050] Specifically, the system first analyzes the root causes of conflicts, distinguishing between numerical errors and logical relationship errors. For numerical errors, a parameter correction strategy is employed, such as recalculating the equation solution; for logical relationship errors, the reasoning chain is improved by adding intermediate propositions. Based on the mechanism of adjusting premises from abnormal results in reverse reasoning, the system generates multiple candidate solutions for each correction direction and uses heuristic search to efficiently explore the solution space. All candidate solutions must pass a feasibility check, excluding those that introduce new conflicts or violate core constraints.

[0051] Furthermore, a predefined correction cost function is used for comprehensive evaluation. This function quantifies three key indicators: propositional change measures the magnitude of modification, rule violation assesses remaining conflicts, and semantic deviation measures the degree of semantic preservation. Through domain-adaptive weight configuration (e.g., mathematical reasoning emphasizes rule violation, legal documents emphasize semantic deviation), the solution with the minimum comprehensive cost is selected as the optimal correction, ensuring that the correction result is both logically correct and semantically coherent.

[0052] The corrected cost function formula is as follows:

[0053] in, Indicates the first The smaller the total cost of each candidate modification scheme, the better the scheme is. Indicates the first Each candidate modification scheme contains a modified set of propositions and supporting rules. This represents the change in propositions, specifically calculated as the size of the symmetric difference between the sets of propositions before and after the modification. , in It is the set of original propositions. This represents the rule violation rate, calculated as the number of rules that are still violated in the corrected answer. Ideally, it should be 0. To represent semantic deviation, a sentence embedding model is used to calculate the vector representations of the answers before and after correction. Then, the semantic difference is measured using cosine distance. , in and These are the vector representations of the original answer and the revised solution, respectively. , , There are three weighting coefficients, all of which are non-negative real numbers and satisfy the following conditions: This is used to adjust the relative importance of the three indicators.

[0054] Choose to The smallest candidate solution is chosen as the final modified solution. This selection process can be expressed as follows: .

[0055] It should be noted that, in order to adapt the modified cost function to different application domains, an adaptive weight update mechanism based on reinforcement learning is adopted. The state space is defined as the feature vector of the current task. The action space is a weight vector. The reward function is a weighted sum of the validation pass rate of the corrected answer and user satisfaction. The weight update formula uses the policy gradient method:

[0056] in To accumulate rewards, This represents the trajectory of the correction process. For parameterized weight selection strategy, The total reward for the trajectory, This is the learning rate. This represents the gradient of the objective function with respect to the weights, which is approximated by Monte Carlo sampling.

[0057] To prevent oscillations caused by weight drift, a regularization constraint is applied to the weight updates. Specifically, a Kullback-Leibler divergence term is added to the objective function to ensure that the updated weight distribution does not deviate too far from the initial empirical values.

[0058] Experiments show that in the field of mathematical reasoning It converges to approximately 0.25, in the field of legal documents. It converges to approximately 0.60, which is significant in the field of medical diagnostics. The convergence to approximately 0.35 reflects the different preferences of various fields for logical rigor and semantic fidelity.

[0059] S107: Feed the optimal correction scheme back to the large language model to regenerate the answer.

[0060] In some embodiments of this application, after obtaining the optimal correction scheme, the optimal correction scheme is transformed into structured feedback information to guide the large language model to regenerate the answer.

[0061] Specifically, the constructed feedback text contains three key elements: error location clearly indicates the location of the problem, such as the failure to verify the solution x=3 in the third line of the answer; error explanation describes the specific rule violated and the verification process, such as substituting into the original equation to get 9≠0; and correction suggestions provide directions for improvement without directly giving the answer, such as suggesting checking the discriminant calculation.

[0062] This feedback, combined with the original question, forms a new prompt input to the large language model, prompting the model to self-correct based on error information. This feedback mechanism not only improves the quality of the answers but also promotes the continuous learning of the large language model on specific tasks.

[0063] S108: Iteratively execute steps 4 to 7 until the answer is verified, and obtain the final answer, confidence score, and interpretability report; the interpretability report includes verification results, rule matching details, corrective measures, and basis for correction.

[0064] In some embodiments of this application, after obtaining the optimal correction scheme for the first time, the process of steps 4 to 7 is iteratively executed until the answer is verified, and finally the answer containing multiple guarantees and the complete decision basis are output.

[0065] Once the answer passes all logical verifications, a comprehensive confidence score is calculated. This score takes into account factors such as the verification pass rate, rule coverage, and correction costs, and generates a quantitative indicator in the range of 0-1 using a weighted formula.

[0066] Simultaneously, it generates an in-depth interpretability report, including a summary of the validation results (e.g., all rules passed the check), rule matching details (listing all applicable rules and their validation status), correction records (if any corrections were made, their location and basis are explained), and confidence analysis (explaining the composition of the score and the reliability level).

[0067] The entire output package ensures that users not only obtain the final answer, but also fully understand the system's verification process and decision-making logic, greatly enhancing the credibility of the results and the transparency of the system.

[0068] In addition, failure cases in the verification process are continuously monitored, and the knowledge base is automatically updated when the accumulated failure score exceeds the dynamic threshold, forming a complete self-evolution closed loop.

[0069] Cumulative failure score The calculation formula is:

[0070] The meanings of each symbol are as follows. The cumulative failure score is a non-negative real number that reflects the degree of incompleteness of the knowledge base. This represents the total number of tasks processed since the system started running, and is a positive integer. It's a summation index, traversing from 1 to... , representing each verification process. Indicates the first The answer generated by the large language model in this sub-task is text in the form of natural language. This represents the current knowledge base, which contains all logical rules and their weights. It's a conflict degree function; it calculates the answer. Violation of knowledge base The severity of a rule is calculated by dividing the sum of the weights of rule violations by the total weight of the rules; the larger the value, the more severe the conflict. Indicates the first The error pattern type to which the verification failed will be determined, and similar errors will be automatically classified into the same pattern. Indicates error mode The frequency of occurrence in historical records is calculated as the number of times the pattern appears divided by the total number of verifications.

[0071] when At that time, the system performs a knowledge base update operation, in which... This is a preset threshold parameter. To adapt to different risk levels in different fields, a dynamic threshold strategy is adopted. ,in As the initial threshold, The attenuation coefficient is... The accumulated regret value represents the performance loss caused by an improperly set threshold. In high-risk fields such as medical diagnostics, Set it to a lower value to ensure timely updates, while in low-risk areas such as mathematical reasoning, a higher value can be set to maintain stability.

[0072] It should also be noted that, considering that simple cosine similarity may not accurately reflect the degree of semantic preservation in some cases, this application introduces a nonlinear correction based on an attention mechanism. The corrected semantic deviation... Defined as:

[0073] in The sigmoid function is used to map similarity to... Interval. This indicates the total number of sentences or semantic units in the answer. and These represent the first and second solutions, respectively. The hidden state vector of each semantic unit. The similarity metric function can be cosine similarity or bilinear transformation. For attention weights, through Calculate, where For the first The importance score of each semantic unit is learned by an independent attention network.

[0074] This nonlinear correction enables the system to distinguish the importance of different parts of the answer. For example, in mathematical proofs, the semantic preservation of key reasoning steps is more important than transitional descriptions, and the attention mechanism can automatically learn this difference and take it into account during correction.

[0075] It should be noted that, although the embodiments in this application are based on... Figure 1 Steps S101 to S108 will be described sequentially, but this does not mean that steps S101 and S108 must be performed in a strict order. The reason this embodiment follows this order is... Figure 1The order in which steps S101 to S108 are described is provided to facilitate understanding of the technical solutions of the embodiments of this application by those skilled in the art. In other words, in the embodiments of this application, the order of steps S101 to S108 can be appropriately adjusted according to actual needs.

[0076] pass Figure 1 This application employs a method that combines reverse reasoning with bidirectional matching verification using a logical knowledge base. By adjusting the cost function to select the optimal solution, it accurately detects factual errors and logical conflicts in the output of a large language model, while maintaining semantic coherence to the greatest extent possible. This significantly improves the reliability and accuracy of the output content, effectively solving the "illusion" problem and meeting the needs of high-precision scenarios. Heuristic prompts guide the large language model to automatically generate logical rules, which are then structured, hierarchically categorized, and redundant contradictions are removed to form a knowledge base. This eliminates the need for manual rule writing, greatly reducing knowledge acquisition and management costs. A bidirectional feedback mechanism feeds error information and correction suggestions back to the large language model, guiding it to iteratively optimize its output and continuously evolving the knowledge base. This enhances the transparency of system decision-making and creates a collaborative progress loop between the large language model and the knowledge base, improving the system's performance and stability in handling complex tasks over the long term.

[0077] Figure 2 This is a system architecture diagram of a method for verifying and improving knowledge of a large language model provided in an embodiment of this application.

[0078] Figure 3 The flowchart shows the inverse verification module of a large language model knowledge verification and improvement method provided in this application embodiment.

[0079] Figure 4 A schematic diagram of a large language model knowledge verification and improvement device provided in this application embodiment includes: At least one processor; and, A memory that is communicatively connected to at least one processor; wherein, A method for verifying and improving knowledge of a large language model, wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable at least one processor to perform any of the above.

[0080] Some embodiments of this application provide a non-volatile computer storage medium for verifying and improving large language model knowledge, which stores computer-executable instructions capable of executing any of the above-mentioned methods for verifying and improving large language model knowledge.

[0081] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0082] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0083] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0087] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0088] Memory may include non-persistent storage in computer-readable media, random access memory (RAM), and non-volatile memory such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0089] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0090] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0091] The above are merely embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the technical principles of this application should fall within the protection scope of this application.

Claims

1. A method for knowledge verification and improvement of a large language model, characterized in that, The method includes: The system accepts user input of a task description, a specific problem, and sample data; the sample data is related to the task description and the specific problem. The task description and preset heuristic prompts are input into a large language model to generate logical rules related to the task description. The logical rules are then structured to obtain logical expressions. Calculate the coverage of the logical expression on the example data to eliminate redundant and contradictory rules and obtain a logical knowledge base; The specific problem is input into a large language model to generate a preliminary answer. The preliminary answer is then symbolically transformed to extract the entities, relations, and attributes of the preliminary answer, resulting in a symbolic proposition. The symbolic propositions are matched with the logical knowledge base to detect whether there is a logical conflict between the symbolic propositions and the logical knowledge base; If a logical conflict is detected, multiple candidate correction schemes are generated through reverse reasoning, and the optimal correction scheme is determined according to a predefined correction cost function; the correction cost function is related to the proposition change, rule violation degree, and semantic deviation degree. The optimal correction scheme is fed back to the large language model to regenerate the answer; The process of steps 4 to 7 is executed iteratively until the answer is verified, resulting in the final answer, confidence score, and interpretability report. The interpretability report includes the verification results, rule matching details, corrective measures, and the basis for the corrections.

2. The method according to claim 1, characterized in that, The process of inputting the task description and preset heuristic prompts into a large language model to generate logical rules related to the task description, and then performing structured processing on the logical rules to obtain logical expressions, specifically includes: Based on preset heuristic prompts, the large language model is guided to analyze the input and output types, core objectives and potential constraints of the task description, and generate primary logical rules. By employing an iterative questioning strategy, the primary logical rules are questioned to determine if there are any unconsidered constraints, thereby generating logical rules. The logical rules are structured using a sequence-to-sequence model based on the Transformer architecture to generate first-order logical expressions.

3. The method according to claim 1, characterized in that, The calculation of the coverage of the logical expression on the example data, in order to eliminate redundant and contradictory rules and obtain a logical knowledge base, specifically includes: The coverage of the logical expression on the example data is calculated using a preset coverage formula to obtain the coverage result. Based on the coverage results, logical equivalence judgment and rule inclusion relationship analysis are used to identify rules that are functionally redundant in the logical expressions for deletion. Based on the coverage results, counterexamples are constructed to identify rule pairs with opposing conclusions in logical expressions for deletion. A logical knowledge base is constructed based on the coverage results of redundant and contradictory rules.

4. The method according to claim 1, characterized in that, The symbolic transformation of the preliminary answer to extract its entities, relations, and attributes, resulting in a symbolic proposition, specifically includes: The structured data in the preliminary answer is precisely parsed using a syntactic parser based on a bottom-up operator priority analysis method to construct an abstract syntax tree; For the free text in the initial answer, entities in the text are identified using named entity recognition technology; The syntactic relations between the entities are identified using dependency parsing techniques, and the semantic relations between the entities are inferred based on the syntactic relations. The entities, semantic relationships, and attributes are combined into propositional forms to form symbolic propositions.

5. The method according to claim 1, characterized in that, The step of matching the symbolic proposition with the logical knowledge base to detect whether there is a logical conflict between the symbolic proposition and the logical knowledge base specifically includes: Pattern matching technology is used to bind the variables of rules in the logical knowledge base to the specific entities in the symbolic propositions, and to check whether the preconditions of the rules are all satisfied in the set of symbolic propositions under the variable binding. If all the preconditions of the rule are met, but the conclusion is not found in the set of symbolized propositions under the variable binding, it is judged as a logical conflict. Each detected logical conflict is evaluated to determine its severity; the severity is determined by the importance weight of the violated rule itself, and whether the violation is deterministic or probabilistic.

6. The method according to claim 1, characterized in that, If a logical conflict is detected, multiple candidate correction schemes are generated through reverse reasoning, specifically including: Analyze logical conflicts to determine whether the error lies in the numerical value of the proposition itself or in the logical relationship between the propositions. For the numerical errors, the parameters in the erroneous proposition are modified through a reverse reasoning mechanism; For errors in the logical relationship, the missing intermediate propositions are added to complete the reasoning chain through a reverse reasoning mechanism; For each correction direction, multiple candidate correction schemes are generated using a heuristic search strategy; The feasibility of the candidate correction schemes is verified, and the verified candidate correction schemes are stored uniformly.

7. The method according to claim 1, characterized in that, The optimal correction scheme is determined based on a predefined correction cost function, specifically including: Based on the application domain of each candidate correction scheme, the component weights of the correction cost function are determined; the correction cost function is a weighted sum of three components: proposition change, rule violation, and semantic deviation. Based on the component weights, multiple candidate correction schemes are evaluated using a predefined correction cost function to determine the candidate correction scheme with the minimum overall correction cost, which is then identified as the optimal correction scheme.

8. The method according to claim 1, characterized in that, The method further includes: The system monitors failed cases during the iteration process, calculates the cumulative failure score according to the preset scoring rules, and generates new rules by analyzing the failed cases and integrating them into the knowledge base when the score exceeds the preset threshold.

9. A large language model knowledge verification and improvement device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the large language model knowledge verification and improvement method according to any one of claims 1-8.

10. A large language model knowledge verification and improvement storage medium, storing computer-executable instructions, characterized in that, The computer-executable instructions are capable of executing the large language model knowledge verification and improvement method described in any one of claims 1-8.

Images