A method and system for state reconstruction and semantic verification of a symbolic reasoning process

Abstract

The application discloses a kind of state reconstruction and semantic verification method and system of symbol reasoning process, it is related to computer data processing and artificial intelligence evaluation technical field, including: obtaining SAT problem instance and large language model reasoning track text and parsing into structured action instruction sequence;Log integrity and structure alignment detection are carried out;CDCL state machine model is constructed in memory, and state object sequence is generated according to the dynamic update state object of action instruction;For each state conversion, corresponding verification subprogram is called to carry out semantic legality automatic verification;Record error and introduce multi-dimensional problem scale parameter to generate fidelity attenuation curve;Multi-dimensional evaluation vector is calculated and visual evaluation report is generated.The application can realize automatic evaluation without calling external solver, can accurately locate reasoning error step and reveal the reliability boundary of model under different problem scale.

Description

Technical Field

[0001] This invention relates to the fields of computer data processing and artificial intelligence evaluation technology, and in particular to a method and system for state reconstruction and semantic verification of symbolic reasoning processes. Background Technology

[0002] In recent years, large language models (LLMs) have achieved good results in tasks such as mathematical reasoning, code generation, and logical question answering. Increasing research is focusing on their application in symbolic reasoning tasks such as the Boolean satisfiability problem (SAT). Existing evaluation methods mainly assess the accuracy of the model by comparing its final output (such as SAT or UNSAT classification) with the standard answer.

[0003] However, existing technologies have the following obvious defects and shortcomings: First, the evaluation process is computationally resource-intensive and inefficient. To verify the correctness of the intermediate inference process of a large language model, existing methods typically require calling an external standard SAT solver to rerun the same instance, or relying on manual verification of inference logs. This approach lacks an automated intermediate state verification mechanism, resulting in long evaluation cycles and high computational consumption, which cannot meet the needs of large-scale model evaluation.

[0004] Second, it is impossible to verify the consistency and semantic validity of the underlying data state. For modern SAT solving frameworks such as Conflict-Driven Clause Learning (CDCL), the process involves discrete and deterministic symbolic state updates, including unit propagation, decision assignment, implication graph construction, conflict detection, learning clause derivation, and non-temporally sequential backtracking. Although the inference text generated by large language models may conform to expectations in natural language format, its underlying variable assignment data and clause state data often contain logical conflicts (i.e., "illusions"). Existing methods cannot directly map and verify the consistency of the underlying data structure of the model-generated text.

[0005] Third, there is a lack of multi-dimensional quantitative evaluation data output and generalization analysis. Existing methods can only output a binary index of "right / wrong" for the final result. When the model makes an error, it is difficult to pinpoint which specific inference subroutine has a semantic error. At the same time, it is impossible to analyze the model's generalization ability and reliability boundaries under different problem scales, making it difficult to guide further fine-tuning of large language models or optimization of inference chains.

[0006] Therefore, there is an urgent need for a method and system for state reconstruction and semantic verification of symbolic reasoning processes to overcome the defects and shortcomings of existing technologies. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for state reconstruction and semantic verification of symbolic reasoning process. By performing structured parsing of the reasoning trajectory output by a large language model, reconstructing the memory state of the CDCL solver, and performing subroutine-level automatic verification based on CDCL semantic constraints, the invention achieves automated, fine-grained, and multi-dimensional fidelity evaluation of the symbolic reasoning process of a large language model.

[0008] To achieve the above objectives, this invention provides a method for state reconstruction and semantic verification of symbolic reasoning processes, comprising the following steps: Step S1: Obtain the SAT question instance data to be evaluated and the inference trajectory text output by the large language model for the SAT question instance; Step S2: Parse the inference trajectory text obtained in step S1, and convert the unstructured text into a sequence of action instructions through the natural language parsing module; each action instruction in the sequence of action instructions maps to the symbol state update operation in the CDCL solution process. Step S3: Perform log integrity and structure alignment checks on the action instruction sequence of step S2. For valid trajectories that pass the integrity check, perform index alignment with the standard action sequence generated by the standard SAT solver for the same SAT instance, and filter out invalid trajectories whose search branches deviate from the standard solver tree. Step S4: Based on the action instruction sequence obtained after alignment in step S3, construct the state machine model of the CDCL solver in computer memory, starting from the initial empty state object. Initially, the action instructions in the instruction queue are read sequentially, and the state update function is called to dynamically update the state objects in memory, generating a sequence of state objects. Each state object It must contain at least an assignment map. Hierarchical counter Clause set array and directed implication graph Based on the current state object Execute action instructions Through the state update function Generate the next state object ; Step S5: For each state transition in the state object sequence generated in step S4. The system invokes a verification subroutine corresponding to the action instruction type to perform automatic semantic legality verification based on data feature matching. This automatic semantic legality verification includes at least decision data verification, unit propagation data verification, conflict data verification, learning clause data verification, and non-temporally sequential backtracking data verification. This indicates the specific action instruction executed during the i-th state transition; Step S6: When any verification subroutine in step S5 returns a verification failure signal, record the error information in the log database and evaluate the cascading impact; introduce the number of variables. Number of clauses and clause variable ratio The system uses multidimensional problem size parameters to generate random 3-SAT instances in batches under different size combinations, and automatically executes steps S1 to S5 to collect process-level verification data. Based on the verification data, the system calculates the verification pass rate of each verification subroutine under different problem sizes and generates multidimensional fidelity decay curves to define the reliability boundary. Step S7: Based on the verification results of step S6, calculate the multidimensional evaluation vector representing the inference fidelity of the large language model in each CDCL subroutine, obtain the basic score data through weighted calculation, and render and generate a multidimensional visual evaluation report containing a subroutine capability radar chart, an error type and hierarchy heatmap, and a process credibility diagnostic summary.

[0009] Preferably, in step S2, the natural language parsing module combines a preset Prompt template, regular expression, or sequence labeling model to convert the unstructured reasoning trajectory text into a structured sequence of action instructions; the sequence of action instructions includes one or more of the following: decision assignment, unit propagation, implication graph construction, conflict detection, learned clausederivation, or non-chronological backjump.

[0010] Preferably, the execution log integrity and structure alignment detection in step S3 specifically includes: first, determining whether the action instruction sequence constitutes a complete CDCL solution loop, specifically starting with an initial empty state and ending with a SAT conclusion, UNSAT conclusion, or exhaustive search space; for trajectories with missing key subroutine instructions, damaged data structures, or unparseable data, marking them as invalid data and terminating subsequent evaluation; for valid trajectories that pass the integrity check, indexing and aligning them with the standard action sequence generated by the standard SAT solver.

[0011] Preferably, the initial empty state object in step S4 It has explicit initialization semantics, specifically: assignment mapping table Empty hash table; level counter Set to zero; array of clauses The original clauses read from the CNF file have been loaded; the learning clause portion is empty; Directed Implication Graph An empty graph; The state update function performs the following operations: when a decision instruction is read, it assigns values ​​to the mapping table. Insert assignment and increment the level counter In directed implication graphs Add a new decision node; when the unit propagation instruction is read, assign a value to the mapping table. Insert propagation assignment in the directed implication graph Add new nodes and connect predecessor edges from the false variables of the triggering clause; when a conflict instruction is read, traverse the clause set array. Confirm conflicting clauses; read the instruction to derive the learning clause and examine the directed implication graph. The graph is traversed to locate the 1-UIP (first unique implication point) and the learned clauses are generated and written to the clause set array. When a rollback instruction is read, the rollback assignment mapping table is rolled back according to the target level. Hierarchical counter With directed implication graph .

[0012] Preferably, the automatic semantic legality verification based on data feature matching in step S5 specifically includes: (1) Decision data verification: query the assignment mapping table Verify that the variable identifier specified by the decision instruction exists in the set of unassigned variables, and verify that its assignment is a valid Boolean value; (2) Unit propagation data verification: Extract the target variable from the propagation instruction and traverse the clause set array. Verify whether there exists at least one target clause such that all literals in that clause except the target variable are in the assignment map. The query results in the above are all false; (3) Conflict data verification: When a conflict command is received, the clause set array is traversed. Verify if all characters exist in the assignment mapping table. The query results in the clause are all false, which is a conflict clause. (4) Learning Clause Data Validation: Extract the generated learning clause data and validate it based on the directed implication graph. Execute the graph traversal algorithm to verify whether the data of the learning clause is generated by the conflicting nodes and their predecessor nodes through the data merging operation, and verify whether its node topology features satisfy the 1-UIP (first unique implied point) condition. (5) Non-time-order backtracking data verification: Extract the target level data in the backtracking instruction, parse the decision level identifier contained in the learning clause data, and verify whether the target level data is strictly equal to the second highest decision level identifier in the clause.

[0013] Preferably, step S6, which records error information and assesses cascading effects, specifically includes: recording the error type, the instruction index where the error occurred, and the corresponding memory state object snapshot in the log database; dynamically classifying the error into fatal and non-fatal errors based on whether the error causes subsequent state machine deadlock or search tree deviation, and recording the error's affiliation in the CDCL subroutine; The fidelity decay curve is generated in the following way: the number of variables is used as the horizontal axis and the verification pass rate of each verification subroutine is used as the vertical axis. The reliability boundary of the large language model on different symbolic reasoning subtasks is accurately quantified through curve fitting and threshold analysis.

[0014] Preferably, step S6 further includes: performing feature extraction and cluster analysis on the error records in the log database, extracting error feature vectors, using a clustering algorithm to unsupervised group the error patterns, identifying the systematic defects of the large language model in CDCL inference, and using this as prior knowledge for subsequent model fine-tuning or reinforcement learning reward function design.

[0015] Preferably, the multidimensional evaluation vector in step S7 Specifically, it includes: Log integrity metrics : The number of valid and structurally complete action instruction sequences divided by the total number of trajectories generated by the model; Decision fidelity index : The number of valid decision assignment steps divided by the total number of decision steps output by the model; Unit propagation fidelity index : The number of valid propagation steps that satisfy the unit clause triggering condition divided by the total number of propagation steps output by the model; Conflict detection fidelity index The harmonic mean of precision and recall, F1-score, is used. Learning Clause Fidelity Index The number of learning clauses that satisfy the 1-UIP topological features and have correct derivation logic, divided by the total number of learning clauses generated by the model; Backtracking Fidelity Index The number of correct backtracking levels is divided by the total number of backtracking operations performed by the model.

[0016] Preferably, the basic scoring data obtained through weighted calculation in step S7 specifically involves: calculating the multidimensional evaluation vector... With weight vector Perform dot product operation Weight vector It supports dynamic configuration based on the evaluation scenario or automatic adjustment based on the information entropy weight method; the final comprehensive score is based on the basic score data, and corresponding penalty items are deducted according to the number of fatal errors in the verification results.

[0017] This invention also provides a system for state reconstruction and semantic verification of symbolic reasoning processes, comprising: Data access and parsing module: used to read CNF files in DIMACS format and inference trajectory text generated by large language model LLM, and use regular expressions and natural language NLP parser to convert unstructured text into a structured JSON format action instruction queue. State Reconstruction Engine: Used to maintain a virtual CDCL state machine in memory, including a state initializer, state transition functions, and a state snapshot manager. It dynamically updates the state objects in memory according to the sequence of action instructions, generating a sequence of state objects. Semantic verification engine: It contains five independent verifiers corresponding to decision, propagation, conflict, learning and backtracking subroutines respectively. Each verifier reads the current state snapshot and action instructions and outputs a boolean verification result (Pass / Fail) and error diagnosis code. Evaluation, analysis and visualization module: used to aggregate all verification results, perform multi-dimensional index calculations and multi-scale decay analysis, and render and generate an evaluation report including radar charts and decay curves.

[0018] Therefore, the present invention employs the above-mentioned method and system for state reconstruction and semantic verification of symbolic reasoning process, which has the following beneficial effects: (1) By directly reconstructing the state object in the computer memory and performing data feature matching, there is no need to call the external standard SAT solver for secondary solving or to require manual intervention, which greatly reduces the computational overhead of CPU / GPU and realizes the automated and efficient evaluation of large-scale inference trajectories.

[0019] (2) The natural language output of the large language model is converted into a computer-executable memory data structure, which can accurately identify the "illusion" errors in the model in the underlying data operations such as variable assignment and clause deduction; it can automatically locate the specific error steps and error types in the reasoning trajectory, and reflect the model's true symbolic reasoning ability from the process level.

[0020] (3) It can generate a multidimensional evaluation vector containing the pass rates of multiple subroutines, and generate a fidelity decay curve in combination with the problem size parameter, revealing the generalization decay law and reliability boundary of the large language model under different problem sizes.

[0021] (4) By using error type and decision level heatmaps and error pattern clustering analysis, we can identify the systematic defects in the model in CDCL inference, providing accurate data support and optimization targets for subsequent reinforcement learning or process supervision fine-tuning of large language models.

[0022] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the architecture of CDCL trajectory generation, model fine-tuning, and symbolic reasoning evaluation in a state reconstruction and semantic verification method for a symbolic reasoning process according to an embodiment of the present invention. Figure 2 This is a flowchart of a CDCL-based large language model fine-tuning method for state reconstruction and semantic verification of a symbolic reasoning process according to an embodiment of the present invention. Figure 3 This is a heatmap showing the error types and decision levels in an embodiment of the present invention. Figure 4 This is a bar chart of CDCL subroutine-level verification and error statistics in an embodiment of the present invention, where (a) is the pass rate of each subroutine verification and (b) is the number of error steps in each subroutine. Figure 5 This is a radar chart showing the inference fidelity of the CDCL subroutine in an embodiment of the present invention. Figure 6 This is a graph showing the change in the verification pass rate of the CDCL subroutine in an embodiment of the present invention. Detailed Implementation

[0024] The following detailed description of embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0025] Example Please see Figures 1-5 Let's take an SAT question as an example to be evaluated. Its CNF (Conjunctive Normal Form) formula is: ; Among them: clauses clause Number of variables ; number of clauses Clause variable ratio .

[0026] In SAT questions, CNF refers to a logical formula representation, namely: A logical expression consisting of multiple clauses connected by "AND" (∧), where each clause consists of several literals connected by "OR" (∨).

[0027] This evaluation assesses the performance of the finely tuned gemma3-4b. The output consists of 6000 high-quality solver step trajectories. The input is a pre-designed prompt: "You are proficient in CDCL solver steps and can solve based on the input CNF file, generating complete CDCL execution decisions for each step." The prompt is "xxx[input] {cnf}", where {cnf} is a variable that represents each corresponding CNF problem during training. A CNF problem is randomly generated using a Python script. The specific process is as follows... Figure 2 As shown.

[0028] The object to be evaluated is an open-source large language model that has been fine-tuned with CDCL inference trajectories. Its output for the aforementioned SAT instance is as follows: "First, a decision is assigned, entering decision level 1; then, unit propagation is executed; a conflict is detected; a learning clause is generated based on the conflict result; a backtracking operation is performed; finally, it is determined that the current SAT instance is unsatisfiable (UNSAT)." The specific steps are as follows: Step S1, Data Acquisition: The DIMACS CNF file corresponding to the SAT instance to be evaluated (a DIMACS CNF file is a standardized file format used to describe the Boolean satisfiability problem (SAT), in which integer encoding is used to represent variable and clause information in CNF formulas) is read, and the CDCL inference trajectory text generated by the large language model to be evaluated is obtained. The CNF file serves as the original constraint set on which state reconstruction is based, and the inference trajectory output by the model serves as the input trajectory parsing module for the object to be verified.

[0029] Step S2, Trajectory Analysis and Structuring: The model output is parsed using a natural language parsing module. This module combines regular expression matching with Prompt parsing to convert the natural language inference process into a unified CDCL action representation.

[0030] In this embodiment, the following sequence of action instructions is obtained through parsing: Decision( ); UnitPropagation( ); ConflictDetect( ); LearnClause( ); :Backjump(0); Ultimately, UNSAT.

[0031] The instructions are then written to the instruction queue in the order they occur. Simultaneously, corresponding metadata is created for each action, including variable number, Boolean assignment, decision level, involved clause number, and action sequence number, providing input for subsequent state reconstruction.

[0032] Step S3: Track integrity verification and alignment with standard track: First, the parsed action sequence is checked for integrity. In this embodiment, the trajectory sequentially includes three actions: decision-making, unit propagation, and collision detection. The order of these actions conforms to the execution flow of the CDCL algorithm, therefore the trajectory structure is determined to be complete, and the integrity check is passed.

[0033] Subsequently, the model output trajectory is aligned with the standard solution trajectory generated by the standard CDCL solver for the same SAT instance. For this embodiment, the standard solution process is as follows: Decision ); UnitPropagation( The reason clause is ); All clauses are satisfied; Return to SAT; The comparison revealed that the two trajectories maintained the same decision-making action in the first step, but differed in the unit propagation action in the second step. The standard solver propagation result is... The model output is Therefore, the action The error is marked as a candidate semantic anomaly, but the entire trajectory is still retained for subsequent state reconstruction in order to analyze the impact of the error on the subsequent reasoning process.

[0034] Step S4, CDCL state reconstruction: Construct a CDCL state machine in memory. The state object includes: a current assignment mapping table. Hierarchical counter Clause set array Learning the set of clauses Directed implication graph .

[0035] First, initialize to obtain the initial state. .in: Empty; =0; Contains original clause and ; Empty; Empty.

[0036] The state is then updated sequentially according to the action sequence.

[0037] (1) Performing actions : Read Decision( In the assignment mapping table New variables The decision level is Level 1. Simultaneously, the current decision level is updated to 1, and a new decision node is created in the implication graph. Generate a state snapshot. .

[0038] (2) Performing actions : Read UnitPropagation( Based on the model output, in the assignment mapping table... New variables The decision-making level remains at Level 1. Subsequently, implication relations are established based on the propagation cause clause, and propagation nodes are created in the directed implication graph. And establish corresponding cause edges. Generate state snapshots. .

[0039] (3) Performing actions : Read ConflictDetect(). Iterate through the set of original clauses. Based on the current state of the model, record the state information corresponding to the conflict detection and generate a state snapshot. .

[0040] (4) Perform the action : Read the LearnClause() function. Generate a learning clause based on the conflict analysis results output by the model and add it to the learning clause set. Simultaneously, generate a corresponding state snapshot due to the addition of a new learning clause. .

[0041] (5) Perform the action : Read Backjump(). Trace back to the root of the assignment that caused the conflict and jump back to update the next level, generating a corresponding state snapshot. .

[0042] The final sequence of state objects is obtained: .

[0043] Step S5: Verify the legality of state semantics: The corresponding verification subroutines are called sequentially to perform semantic verification on each state transition process.

[0044] (1) Decision data verification: verify to State transition. Query the current assignment mapping table and find the variable. Since no value has been assigned before, the result of the assignment is a valid Boolean value, and the decision level is valid. Therefore, the decision action is deemed to have passed verification.

[0045] (2) Unit propagation data verification: verify to State transition. In the current state, Iterate through all clauses. For each clause... ,because Therefore, divide the variable Apart from the above, all other text is False, satisfying the unit propagation trigger condition. According to the CDCL algorithm rules, the only valid propagation result for this clause should be... However, the model output propagation result is... The propagation direction is inconsistent with the only legitimate direction, therefore the unit propagation verification is deemed to have failed. The error type is recorded as Propagation Hallucination.

[0046] (3) Conflict data verification: verify to State transition. Based on the current state of the model, , By traversing all the original clauses, we can find the clauses. Both characters in the text are False, so the current state of the model does indeed correspond to a conflict clause. However, this conflict is based on an illegal state formed by error propagation in the previous stage, and therefore belongs to a pseudo-conflict induced by error propagation, rather than a legitimate conflict generated during the actual execution of CDCL. Therefore, this stage records the completion of the conflict detection action, but its source of error is attributed to the error propagated in the previous stage, and it will be treated as a cascading effect of the propagation error in the final evaluation.

[0047] (4) Learning clause data validation: Verify the state transition.

[0048] The conflict analysis process output by the model is verified based on the conflict clauses obtained in the conflict detection phase and the current implication graph structure.

[0049] First, check whether the learning clauses generated by the model satisfy the conflict learning constraints in the CDCL algorithm. Specifically, this includes: ① Study whether each literal in the clause originates from the variables relevant to the current conflict; ②Whether the learning clause can prevent the current conflict state from recurring; ③ Check whether the learning clause satisfies the generation rules corresponding to the First Unique Implication Point (UIP) or the preset conflict analysis strategy.

[0050] In this embodiment, the model outputs the learning clause as follows: ; By traversing the current implication graph, it was found that the variables in the learning clauses generated by the model can all be found to have corresponding sources in the conflict implication path, and the clause is a negation constraint of the conflict clause in the current assignment state, thus satisfying the conditions for generating learning clauses.

[0051] (5) Verification of rebound motion data: Verify the state transition; Read the Backjump action output by the model to obtain the model's predicted backjump level; Based on the current conflict clauses and the implication graph structure, recalculate the decision level of the conflict-related variables; Subsequently, according to the CDCL nonlinear backtracking rule, the target level for the backtracking should satisfy: Less than the current conflict level; This is equivalent to the second highest decision level in the learning clause.

[0052] In this embodiment: The current conflict occurs at Level 1, therefore the theoretical bounce level should be 0; The model outputs a bounce level of 0; Comparison revealed that the model's output jump level was consistent with the standard CDCL state transition result, thus validating the Backjump action.

[0053] Step S6, Error Logging and Batch Evaluation: First, the verification result of the current instance is recorded in the error log database. The record includes: Error type: propagation semantic error; Subroutine: unit propagation; Instruction number: Status snapshot: Decision-making level: Level 1; Cascading effect: Yes; Error level: Fatal error.

[0054] Subsequently, random 3-SAT instances are automatically generated under different variable sizes and different clause variable ratios. For example, variable sizes are set separately. The clause variables are 20, 50, 100, and 200. The steps are 2, 3, 4, 4.2, 5, and 6. The system automatically repeats steps S1 to S5 for all test instances and compiles the semantic verification results for each subroutine.

[0055] Furthermore, based on the statistical results, the fidelity changes of each subroutine under different parameter conditions are calculated, and fidelity decay curves are generated to analyze the trend of model process reliability as a function of problem size and constraint density. Figure 6 As shown, the pass rate of each CDCL subroutine generally decreased as the number of SAT problem variables increased from 3 to 7. The decision module maintained high stability throughout, indicating that the large language model can perform variable assignment operations well. The unit propagation and clause learning modules showed a more significant decrease, suggesting that the model is more prone to errors in symbolic state maintenance and multi-step inference dependencies as the problem size increases. The conflict detection and bounce-back modules showed a relatively gradual decrease, but were still affected by the bias of preceding inference states. These results indicate that the large language model has high reliability in simple symbolic decision tasks, but exhibits fidelity decay in complex state evolution processes.

[0056] Finally, features such as subroutine category, decision level, error location, and inference context are extracted from all error records, and the DBSCAN clustering algorithm is used to identify potential systemic inference defects in the model.

[0057] Step S7: Fidelity index calculation and evaluation report generation: Based on the verification results obtained in step S5, the six-dimensional process fidelity index is calculated. In this embodiment: Log completeness It is 1.0; Decision fidelity It is 1.0; Unit propagation fidelity It is 0.0; Conflict detection fidelity The result, calculated according to the preset evaluation rules, is 0.5. Learning Clause Fidelity Recorded as 0.5; Backtracking fidelity It is recorded as 0.5.

[0058] The final solution is incorrect.

[0059] Therefore, the process fidelity vector obtained in this embodiment is: ; The weighted calculation was performed using the weight vector corresponding to the advanced solution strategy evaluation mode, resulting in a basic process fidelity score of 0.45. Because a semantic error propagated in this embodiment, which further led to subsequent conflict detection being based on an illegal state, a fatal error penalty of 0.25 was deducted according to the preset scoring rules, ultimately yielding a comprehensive score of 0.20.

[0060] Finally, a model evaluation report is generated. The evaluation report includes a radar chart of the inference fidelity of the CDCL subroutine (e.g., ...). Figure 5 (As shown), CDCL subroutine-level verification and error statistics bar chart (as shown) Figure 4 (as shown), error type and decision level heatmap (e.g.) Figure 3 (As shown), the fidelity decay curve and the model process reliability analysis results; by Figure 3 It can be seen that there is one propagation illusion error at level L1. This diagram intuitively realizes two-dimensional localization of error type and inference depth, providing a clear diagnostic basis for subsequent targeted optimization of the model. Figure 4 As can be seen from (a) in the figure, the decision subroutine has a 100% pass rate with no error steps; Figure 4 As shown in (b), the errors are concentrated in the unit propagation subroutine, resulting in a 1-step inference error; conflict detection, learning clauses, and backtracking subroutines are all normal. This diagram completes the fault decomposition from the modular perspective of the CDCL algorithm, locating the unit propagation module as the weak link in model inference, which corroborates the conclusion of the heatmap, proving that this evaluation framework can accurately distinguish the inference reliability of each CDCL subroutine. Figure 5 It can be observed that the log completeness and decision fidelity are high in the single example; however, the fidelity of conflict detection, learning clauses, and backtracking is only 50%; and the fidelity of unit propagation is 0 due to failure. The radar chart intuitively quantifies the boundary of the symbolic reasoning ability of the model at each stage of the CDCL process, vividly demonstrating the shortcomings of the model's unbalanced capabilities, and forming a multi-dimensional evidence chain with the first two statistical charts.

[0061] In this embodiment, the evaluation report points out that the main errors of the model are concentrated in the unit propagation stage, where semantic errors occur in the propagation direction judgment, which further affects the subsequent conflict detection process. Based on this, the key weak links in the model inference process can be accurately located, providing a basis for model optimization and training data improvement.

[0062] Therefore, this invention employs the aforementioned method and system for state reconstruction and semantic verification of symbolic reasoning processes. First, the unstructured reasoning trajectory output by the model is parsed into a structured CDCL action instruction sequence and stored in a memory queue to solve the problem of uncomputable trajectories. Then, integrity verification and standard trajectory alignment filtering are implemented to filter invalid trajectories and ensure the validity of subsequent state reconstruction data. Next, a state machine model containing an assignment mapping table, a hierarchy counter, a clause set, and a directed implication graph is constructed in memory. This model is then driven by five types of instructions—decision, propagation, conflict, learning, and backtracking—to update the various data structures in a coordinated manner, generating a complete state object. The sequence replaces the cumbersome process of repeatedly calling external solvers; using the current state snapshot as the operation object, it performs data feature matching verification on each subroutine one by one based on the standard semantic rules of alignment, accurately identifying which specific subroutine the illusion occurs in; if the verification fails, it triggers cascading error recording, and the above steps are encapsulated into batch execution units, which are run concurrently on multiple scale SAT instances to generate fidelity decay curves for each subroutine, revealing the reliability boundary of the model as the problem complexity increases, and finally outputting a multi-dimensional weighted score and a visualized diagnostic report, realizing full-link automated fine-grained evaluation of "analysis → screening → reconstruction → verification → analysis".

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for state reconstruction and semantic verification in a symbolic reasoning process, characterized in that, Includes the following steps: Step S1: Obtain the SAT question instance data to be evaluated and the inference trajectory text output by the large language model for the SAT question instance; Step S2: Parse the inference trajectory text obtained in step S1, and convert the unstructured text into a sequence of action instructions through the natural language parsing module; each action instruction in the sequence of action instructions maps to the symbol state update operation in the CDCL solution process. Step S3: Perform log integrity and structure alignment checks on the action instruction sequence of step S2. For valid trajectories that pass the integrity check, perform index alignment with the standard action sequence generated by the standard SAT solver for the same SAT instance, and filter out invalid trajectories whose search branches deviate from the standard solver tree. Step S4: Based on the action instruction sequence obtained after alignment in step S3, construct the state machine model of the CDCL solver in computer memory, starting from the initial empty state object. Initially, the action instructions in the instruction queue are read sequentially, and the state update function is called to dynamically update the state objects in memory, generating a sequence of state objects. Each state object It must contain at least an assignment map. Hierarchical counter Clause set array and directed implication graph Based on the current state object Execute action instructions Through the state update function Generate the next state object ; Step S5: For each state transition in the state object sequence generated in step S4. The system invokes a verification subroutine corresponding to the action instruction type to perform automatic semantic legality verification based on data feature matching. This automatic semantic legality verification includes at least decision data verification, unit propagation data verification, conflict data verification, learning clause data verification, and non-temporally sequential backtracking data verification. This indicates the specific action instruction executed during the i-th state transition; Step S6: When any verification subroutine in step S5 returns a verification failure signal, record the error information in the log database and evaluate the cascading impact; introduce the number of variables. Number of clauses and clause variable ratio The multidimensional problem size parameter generates random 3-SAT instances in batches under different size combinations, and automatically and concurrently executes steps S1 to S5 to collect process-level verification data. Based on the verification data, the pass rate of each verification subroutine under different problem scales is statistically analyzed to generate a multi-dimensional fidelity decay curve to define the reliability boundary. Step S7: Based on the verification results of step S6, calculate the multidimensional evaluation vector representing the inference fidelity of the large language model in each CDCL subroutine, obtain the basic score data through weighted calculation, and render and generate a multidimensional visual evaluation report containing a subroutine capability radar chart, an error type and hierarchy heatmap, and a process credibility diagnostic summary.

2. The method for state reconstruction and semantic verification of a symbolic reasoning process according to claim 1, characterized in that: In step S2, the natural language parsing module combines a preset Prompt template, regular expressions, or sequence labeling model to convert unstructured reasoning trajectory text into a structured sequence of action instructions. The action instruction sequence includes one or more of the following: decision assignment instruction, unit propagation instruction, implication graph construction instruction, conflict detection instruction, learning clause derivation instruction, or non-temporal backtracking instruction.

3. The method for state reconstruction and semantic verification of a symbolic reasoning process according to claim 2, characterized in that, Step S3, which involves executing log integrity and structure alignment checks, specifically includes: first, determining whether the action instruction sequence constitutes a complete CDCL solution loop, specifically starting with an initial empty state and ending with a SAT conclusion, an UNSAT conclusion, or an exhaustive search space; for trajectories with missing key subroutine instructions, corrupted data structures, or that cannot be parsed, marking them as invalid data and terminating subsequent evaluation; and for valid trajectories that pass the integrity check, indexing and aligning them with the standard action sequence generated by the standard SAT solver.

4. The method for state reconstruction and semantic verification of a symbolic reasoning process according to claim 3, characterized in that: Initial empty state object in step S4 It has explicit initialization semantics, specifically: assignment mapping table Empty hash table; level counter Set to zero; array of clauses The original clauses read from the CNF file have been loaded; the learning clause portion is empty; Directed Implication Graph An empty graph; The state update function performs the following operations: when a decision instruction is read, it assigns values ​​to the mapping table. Insert assignment and increment the level counter In directed implication graphs Add a new decision node; when the unit propagation instruction is read, assign a value to the mapping table. Insert propagation assignment in the directed implication graph Add new nodes and connect predecessor edges from the false variables of the triggering clause; when a conflict instruction is read, traverse the clause set array. Confirm conflict clause; When reading the instruction to derive the learning clause, the directed implication graph is read. The execution graph is traversed to locate 1-UIP and a learning clause is generated and written to the clause set array. ; When a rollback instruction is read, the rollback assignment mapping table is rolled back according to the target level. Hierarchical counter With directed implication graph .

5. The method for state reconstruction and semantic verification of a symbolic reasoning process according to claim 4, characterized in that, Step S5, the automatic semantic legality verification based on data feature matching, specifically includes: (1) Decision data verification: query the assignment mapping table Verify that the variable identifier specified by the decision instruction exists in the set of unassigned variables, and verify that its assignment is a valid Boolean value; (2) Unit propagation data verification: Extract the target variable from the propagation instruction and traverse the clause set array. Verify whether there exists at least one target clause such that all literals in that clause except the target variable are in the assignment map. The query results in the above are all false; (3) Conflict data verification: When a conflict command is received, the clause set array is traversed. Verify if all characters exist in the assignment mapping table. The query results in the clause are all false, which is a conflict clause. (4) Learning Clause Data Validation: Extract the generated learning clause data and validate it based on the directed implication graph. Execute the graph traversal algorithm to verify whether the data of the learning clause is generated by the conflicting nodes and their predecessor nodes through the data merging operation, and verify whether its node topology features satisfy the 1-UIP condition. (5) Non-time-order backtracking data verification: Extract the target level data in the backtracking instruction, parse the decision level identifier contained in the learning clause data, and verify whether the target level data is strictly equal to the second highest decision level identifier in the clause.

6. The method for state reconstruction and semantic verification of a symbolic reasoning process according to claim 5, characterized in that, Step S6, which records error information and assesses cascading effects, specifically includes: recording the error type, the instruction index where the error occurred, and the corresponding memory state object snapshot in the log database; dynamically classifying errors into fatal and non-fatal errors based on whether the error causes subsequent state machine deadlock or search tree deviation, and recording the error's affiliation in the CDCL subroutine.

7. The method for state reconstruction and semantic verification of a symbolic reasoning process according to claim 6, characterized in that, Step S6 also includes: performing feature extraction and cluster analysis on error records in the log database, extracting error feature vectors, using clustering algorithms to unsupervised group error patterns, and identifying systemic defects in large language models in CDCL inference.

8. The method for state reconstruction and semantic verification of a symbolic reasoning process according to claim 7, characterized in that, Multidimensional evaluation vector in step S7 Specifically, it includes: Log integrity metrics : The number of valid and structurally complete action instruction sequences divided by the total number of trajectories generated by the model; Decision fidelity index : The number of valid decision assignment steps divided by the total number of decision steps output by the model; Unit propagation fidelity index : The number of valid propagation steps that satisfy the unit clause triggering condition divided by the total number of propagation steps output by the model; Conflict detection fidelity index The harmonic mean of precision and recall, F1-score, is used. Learning Clause Fidelity Index The number of learning clauses that satisfy the 1-UIP topological features and have correct derivation logic, divided by the total number of learning clauses generated by the model; Backtracking Fidelity Index The number of correct backtracking levels is divided by the total number of backtracking operations performed by the model.

9. The method for state reconstruction and semantic verification of a symbolic reasoning process according to claim 8, characterized in that, In step S7, the basic scoring data is obtained through weighted calculation, specifically by converting the multidimensional evaluation vector... With weight vector Perform dot product operation Weight vector It supports dynamic configuration based on the evaluation scenario or automatic adjustment based on the information entropy weight method; the final comprehensive score is based on the basic score data, and corresponding penalty items are deducted according to the number of fatal errors in the verification results.

10. A system for state reconstruction and semantic verification of a symbolic reasoning process, applied to the method for state reconstruction and semantic verification of a symbolic reasoning process as described in any one of claims 1-9, characterized in that, include: Data access and parsing module: used to read CNF files in DIMACS format and inference trajectory text generated by large language models, and to convert unstructured text into a structured action instruction queue using regular expressions and a natural language parser; State Reconstruction Engine: Used to maintain a virtual CDCL state machine in memory, including a state initializer, state transition functions, and a state snapshot manager. It dynamically updates the state objects in memory according to the sequence of action instructions, generating a sequence of state objects. Semantic verification engine: It contains five independent verifiers corresponding to decision, propagation, conflict, learning and backtracking subroutines respectively. Each verifier reads the current state snapshot and action instructions and outputs a boolean verification result and error diagnosis code. Evaluation, analysis and visualization module: used to aggregate all verification results, perform multi-dimensional index calculations and multi-scale decay analysis, and render and generate an evaluation report including radar charts and decay curves.

Images