Question analysis text difference diagnosis method and system based on dependency graph and sandbox

Abstract

The application discloses a question analysis text difference diagnosis method and system based on a dependency graph and a sandbox, and belongs to the fields of artificial intelligence education technology and automatic reasoning. First, a to-be-diagnosed question analysis text and a reference answer are acquired, and an analysis verification dependency graph is constructed. Static safety checking is performed on scripts in the analysis verification dependency graph, if all the scripts pass, execution and safety state objects are obtained in a sandbox, if any script fails, a safety violation difference item is generated. Based on the safety violation difference item or the execution and safety state objects, root cause failure nodes, safety type failure nodes or secondary failure nodes are divided. According to the division result, a minimum difference diagnosis subgraph is extracted in the analysis verification dependency graph, a difference diagnosis report and a structured feedback package containing an automatic pass gate conclusion are generated. The application is helpful to improve the consistency of question analysis error positioning, failure propagation identification and automatic pass gating.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence education technology and automatic reasoning, specifically involving a method and system for diagnosing text differences in test question parsing based on dependency graphs and sandboxes. Background Technology

[0002] With the increasing application of large language models in question bank construction, online tutoring, and educational content quality inspection, automatically generated test question analysis texts need to be quickly verified. For quantitative calculation questions, the correctness of the analysis text depends not only on whether the final answer is consistent with the reference answer, but also on whether a stable derivation relationship is formed between variable definitions, formula calls, unit handling, intermediate conclusions, and the final conclusion.

[0003] Existing technologies can be broadly categorized into three types.

[0004] The first type of approach is the automatic grading of word problems or math problems. This type of solution typically acquires the problem stem, student answers, and sequences of operations or expressions, and judges the correctness of the answer and some steps based on the standard answer or standard sequence. For example, CN113987296A and CN114722245A both involve word problem solution detection or grading. This type of solution can handle structured answers or expression sequences, but it is usually not designed for full-text parsing of natural language generated by large language models, nor does it establish a unified graph structure between the original parsing steps, script execution nodes, and security states.

[0005] The second type of approach is the natural language to code generation or code interpreter approach. This type of solution focuses on how to generate code based on user intent, run the code in a test environment or sandbox, and update it based on runtime feedback. For example, CN118519660A involves large model code generation, code sandboxing, runtime feedback, and code updates; CN118778939A involves code generation and inconsistency correction. The core object of this type of solution is the code task or code description, rather than the step-by-step reasoning chain in the test question parsing text. Its feedback is usually geared towards code updates and does not directly pinpoint root cause errors in the natural language parsing steps.

[0006] The third type of approach is assertion generation or code testing. This type of solution can generate test assertions or evaluate the correctness of assertions. For example, CN117951011A involves unit test assertion generation based on a large language model and retrieval enhancement. However, this type of solution primarily serves software testing and does not distinguish between variable bindings, relational expressions, intermediate conclusions, final conclusions, script nodes, and security states in test question parsing, nor does it address the issue of differentiating between root cause failures and secondary failures among multiple failing assertions.

[0007] The aforementioned technical approaches share common shortcomings: First, if the final answer or a certain expression is judged as incorrect, the system has difficulty in determining whether the error is caused by variable binding, relational expressions, unit handling, script construction, security violations, or subsequent propagation. Second, if the script is blocked due to non-whitelist calls, dynamic imports, file access, or network access, existing solutions easily confuse this with parsing errors. Third, when multiple intermediate assertions fail simultaneously, existing solutions typically list all failure nodes, making it difficult to pinpoint the smallest root cause range, resulting in an excessively large regeneration range. Fourth, when key relational expressions are not covered by assertions or the number of valid assertions is insufficient, the system may still output an automatic pass conclusion.

[0008] The core technical problem arising from this is that existing solutions lack a traceable diagnostic closed loop for full-text parsing of natural language test questions, and lack data objects that uniformly constrain parsing dependencies, script security status, failure propagation paths, and automatic pass conditions.

[0009] Therefore, a differential diagnostic technology for the parsing text of quantitative calculation test questions is needed. This technology should be able to organize the natural language parsing steps, variable bindings, relations, intermediate conclusions, assertions, script execution nodes, and security states into a unified parsing verification dependency graph. When execution results, security violations, or assertion failures occur, it should distinguish between root cause failures, secondary failures, and security failures, and output a minimum differential diagnostic subgraph and an automatically passed gating conclusion. Summary of the Invention

[0010] Existing automated test question parsing verification schemes struggle to pinpoint the root cause among multiple failure points, separate security violations from incorrect parsing conclusions, and prevent erroneous conclusions from being automatically passed when assertion coverage is insufficient. To address these technical problems, this invention proposes a test question parsing text difference diagnosis method and system based on dependency graphs and sandboxing. Taking the parsing diagnosis input package as input, the system sequentially executes logic chain extraction, parsing verification dependency graph construction, verification script generation, pre-execution static security checks, restricted sandbox execution, difference diagnosis subgraph localization, and feedback gating, forming a processing chain from test question parsing text to difference diagnosis report.

[0011] In a first aspect, this invention proposes a method for diagnosing textual discrepancies in test question parsing based on dependency graphs and sandboxes, including:

[0012] S1, obtain the reference answers for quantitative calculation questions, the analysis text of the questions to be diagnosed, and the error threshold configuration, and extract the structured logic chain from the analysis text of the questions;

[0013] S2, based on the relational expressions, intermediate conclusions, and final conclusions in the structured logic chain, generate assertions and corresponding execution scripts, and further obtain a parsing and verification dependency graph; the parsing and verification dependency graph at least includes nodes reflecting the original parsing steps, variable bindings, relational expressions, conclusions, assertions, scripts, and security states, as well as edges reflecting the dependencies, derivations, bindings, mappings, and checks between nodes;

[0014] S3, perform static security checks on the execution scripts. If all execution scripts pass the check, they are executed in a restricted sandbox and an execution and security status object is obtained. If any execution script fails the check, a security violation difference item is generated.

[0015] S4, based on the difference comparison results or security violation difference items, divide the nodes in the parsing verification dependency graph into root cause failure nodes, secondary failure nodes, or security failure nodes; the difference comparison results are obtained by comparing the execution and security status objects with the reference answer and the test question parsing text in sequence according to the error threshold configuration;

[0016] S5. Based on the partitioning results, the minimum difference diagnostic subgraph is extracted from the parsing and verification dependency graph, and a difference diagnostic report and a structured feedback package containing automatically passed gating conclusions are further generated.

[0017] Furthermore, in S1, the structured logic chain includes variables, relational expressions, intermediate conclusions, final conclusions, original parsing step identifiers, and binding relationships between variables.

[0018] Furthermore, in S2, the nodes of the parsing and verification dependency graph include original parsing step nodes, variable binding nodes, relation nodes, intermediate conclusion nodes, final conclusion nodes, assertion nodes, script nodes, and security state nodes; the edges include variable reference edges between variable binding nodes and relation nodes, relation derivation edges between relation nodes and intermediate or final conclusion nodes, assertion binding edges between conclusion nodes and assertion nodes, script mapping edges between assertion nodes and script nodes, and security state edges between script nodes and security state nodes.

[0019] Furthermore, in S3, the process of performing a static security check on the execution script is as follows: the execution script is compared with a preset whitelist. If the execution script contains script elements or execution behaviors that are not in the whitelist, the execution script is considered to have failed the static security check; otherwise, it is considered to have passed.

[0020] Furthermore, the whitelist is used to constrain one or more of the following: import statements, function call statements, external access statements, dynamic import behavior, and operator sets.

[0021] Furthermore, in S3, the execution and security status object includes the execution result of the execution script, intermediate assertion results, execution log, and all exception types and exception information that occur during the execution process.

[0022] Furthermore, in step S4, the process of dividing nodes based on the difference comparison results or safety violation difference items is specifically as follows:

[0023] If there are security violation discrepancies, the security status node corresponding to the executed script will be classified as a security failure node in the parsing and verification dependency graph, and a security violation evidence chain will be generated.

[0024] If the difference comparison results show that the script execution is abnormal, then all the predecessor nodes of the script node are found by following the reverse dependency edge of the parsing and verification dependency graph. If there is no source node that causes the execution abnormality among the predecessor nodes, then the script node is classified as the root cause failure node; otherwise, the source node is classified as the root cause failure node.

[0025] If the difference comparison results show an assertion node that fails, then the node is classified as a secondary failure node. Starting from this node, all predecessor nodes of the node are found along the reverse dependency edge of the parsing verification dependency graph. The source node that caused the error among the predecessor nodes is marked as the root cause failure node.

[0026] Furthermore, in step S5, the process of extracting the diagnostic sub-map with the minimum difference is specifically as follows:

[0027] If a safe failure node exists, the minimum difference diagnosis subgraph consists of the safe failure node, the script node with a safe state edge to the safe failure node, the node corresponding to the original parsing step, and the directly related nodes in the safety violation evidence chain. If no safe failure node exists, the minimum connected node set consisting of the root cause failure node and the corresponding failure evidence chain is found in the parsing verification dependency graph, which is the minimum difference diagnosis subgraph.

[0028] Furthermore, in S5, when the relational assertion coverage rate in the parsing and verification dependency graph is greater than or equal to the preset coverage threshold, the number of valid assertions is not zero, there are no safe failure nodes, there are no nodes to be reviewed, and the minimum difference diagnostic subgraph is empty, the automatic pass gating conclusion in the structured feedback package outputs an automatic pass conclusion, indicating that the test question parsing text has passed the diagnosis. If any gating condition is not met, the automatic pass gating conclusion in the structured feedback package outputs an automatic pass conclusion or a node to be reviewed conclusion that is not allowed, and records the unmet gating condition.

[0029] The nodes to be reviewed include the analytical steps in the test question analysis text that cannot be converted into symbolic expressions, lack executable relational expressions, or lack variable binding relationships but contain conclusive text, as well as their corresponding intermediate or final conclusions; in the analysis verification dependency graph, the nodes to be reviewed do not have associated assertion nodes;

[0030] The relational assertion coverage rate is the ratio of the number of relational nodes in the parsing verification dependency graph that have corresponding assertions and whose execution and difference comparisons are completed to the total number of relational nodes.

[0031] The number of valid assertions refers to the number of assertion nodes that have been generated and executed and compared in the dependency graph of the parsing verification.

[0032] Secondly, this invention proposes a test question parsing text difference diagnosis system based on dependency graphs and sandboxes, which is used to implement the aforementioned test question parsing text difference diagnosis method based on dependency graphs and sandboxes.

[0033] The beneficial effects of this invention are:

[0034] (1) This invention organizes the original parsing steps, variable bindings, relations, assertions, script nodes and security states into the same diagnostic object by parsing and verifying the dependency graph, which helps to trace back the source of failure based on the graph structure.

[0035] (2) The present invention distinguishes between root cause failure and secondary failure by using the minimum difference diagnostic subgraph, which can avoid feeding back all failure steps to the upstream generation system, thereby narrowing the regeneration range.

[0036] (3) This invention separates script security violations from parsing conclusion errors through security failure nodes, avoiding misjudging non-whitelist calls, dynamic imports or external accesses as parsing content errors.

[0037] (4) By combining automatic gating with relational assertion coverage, effective assertion quantity and security status, this invention can prevent erroneous automatic passing conclusions when assertion coverage is insufficient or there are nodes to be verified.

[0038] (5) The present invention binds failure types to processing actions through structured feedback packages, which helps to improve the consistency of traffic distribution between local regeneration, script reconstruction, threshold review and manual review. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the overall verification and differential diagnosis process of the present invention.

[0040] Figure 2 This is a flowchart for constructing the structured logic chain and the dependency graph for parsing and verification in this invention.

[0041] Figure 3This is a flowchart illustrating the restricted sandbox execution and security state node violation state generation process of the present invention.

[0042] Figure 4 This is a flowchart of the minimum difference diagnostic subgraph localization and feedback gating of the present invention. Detailed Implementation

[0043] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0044] For ease of explanation, the names appearing in the specific embodiments of the present invention will first be explained:

[0045] An assertion is an executable verification rule used in programming and software testing to verify whether a program meets expectations. It checks whether a certain conclusion or relation is true. When an assertion condition is false, the program will stop execution and display an error message.

[0046] The test question analysis text difference diagnosis method proposed in this invention is applicable to quantitative calculation test questions, that is, test questions containing executable mathematical relations (such as equations, inequalities, and function expressions), including test questions in mathematics, physics, chemistry, engineering mechanics, etc. that require numerical calculation or formula derivation.

[0047] like Figure 1 As shown, the method of this invention takes the parsed diagnostic input package as input and sequentially performs five steps: logic chain extraction, construction of a verification dependency graph, static security check, difference comparison, and feedback generation, forming a processing chain from test question parsing text to difference diagnostic report. The data object chains involved include D1 to D7, where the flowcharts of S1 to S2 correspond to... Figure 2 The diagram shows the pre-execution static security check and restricted sandbox execution process of S3. Figure 3 The diagrams for failure propagation analysis, minimum difference diagnostic subgraph localization, and feedback gating processes from S4 to S5 correspond to... Figure 4 .

[0048] The specific details of each step are as follows:

[0049] S1: The system extracts the logical chain of the parsing text of the test questions to be verified in D1.

[0050] First, the system obtains the text of the quantitative calculation question to be diagnosed, the question analysis text of the quantitative calculation question, the reference answer or the target conclusion, and combines them with the error threshold configuration to form the analysis and diagnosis input package D1.

[0051] The error threshold configuration in the analytical diagnostic input package D1 can be jointly determined by the question's accuracy requirements, the accuracy of the reference answer, the subject type, and the numerical unit. For algebra questions with exact integers or no error requirements, the equality assertion threshold can be 0; for real number calculation questions, the numerical error threshold can be between 1e-9 and 1e-6; for engineering quantity, mechanics of materials, or physical quantity calculation questions, the threshold can be between 0.1% and 1% of the nominal value of the target quantity, or half of the minimum resolution corresponding to the given significant figures in the question; for angle-related results, the angle threshold can be between 0.01 and 0.1 degrees. If the question text does not declare accuracy requirements, the system configures a default threshold according to the corresponding question type and records the threshold source in the feedback package generated in S5; if the threshold cannot be determined, the system will subsequently mark all nodes related to threshold determination in the verification dependency graph as nodes to be reviewed.

[0052] Secondly, the logical chain extraction is performed on the test question analysis text to be diagnosed: according to the original analysis steps (such as "step one", "step two"), expression boundaries (identified by expression start and end markers), conjunctions (such as "therefore it can be concluded") and conclusive statements (such as "the final result is 5"), the analysis text is decomposed into multiple candidate steps, and the variable binding relationships, relational expressions, intermediate conclusions and final conclusions in the analysis text are identified.

[0053] Specifically:

[0054] The test question analysis text is divided into multiple candidate analysis steps according to the step number, line breaks, punctuation, conjunctions, and concluding statements; and the algebraic expressions are symbolized to transform the analytical expressions in the analysis text into symbolic expressions to support the verification of algebraic identity transformations.

[0055] Based on the expression fragments containing equal signs, inequalities, function symbols, operators, numerical values, or units in each candidate parsing step, extract the relational expression and variable names, variable values, variable units, and variable sources. The variable binding relationship is composed of the variable name, variable value, unit, and variable source.

[0056] At the same time, based on the guiding words and concluding statements in each candidate step, the conclusion is extracted. If the conclusion appears in the last candidate parsing step, it is taken as the final conclusion.

[0057] For steps that lack an executable relation but contain conclusive text (i.e. steps that cannot be converted into symbolic expressions), the system will subsequently mark the corresponding parsing step node and conclusive node in the verification dependency graph as nodes to be reviewed. These nodes will not trigger automatic passage in subsequent gating.

[0058] The final structured logic chain D2 is obtained, which includes at least the original parsing step identifier, the original step text, variables, variable binding relationships, relational expressions, intermediate conclusions, and final conclusions.

[0059] S2: The system builds D3 based on D2.

[0060] Based on the structured logic chain, construct a parsing verification dependency graph D3 that includes at least a set of nodes, a set of edges, and a set of node attributes.

[0061] The node set includes original parsing step nodes, variable binding nodes, relational nodes, intermediate conclusion nodes, final conclusion nodes, assertion nodes, script nodes, and security status nodes.

[0062] The original parsing step node is obtained from the original parsing step identifier in D2 and is used to represent the natural language parsing step; the variable binding node is obtained from the variable set in D2 and is used to represent the variable name, variable value, and unit source; the relation node is obtained from the relation set in D2 and is used to represent equality, inequality, or functional relations; the intermediate conclusion node and the final conclusion node are obtained from the intermediate conclusion set and the final conclusion field in D2, respectively, and are used to represent the conclusions in the parsed text; the assertion node is used to represent the verification conditions that need to be executed or compared, and corresponds to the relation node, intermediate conclusion node, or final conclusion node; the script node is used to represent the executable unit converted from the assertion node; the security status node is used to represent the status of the script node in the security check before execution.

[0063] It should be noted that the system will not generate corresponding assertions and assertion nodes for the nodes to be reviewed in S1. The nodes to be reviewed are passed to S5 as negative conditions for automatically passing the gating conclusion. A conclusion that does not allow automatic passing is generated in the structured feedback package, and the processing action is set to manual review.

[0064] The edge set includes variable reference edges, relation deduction edges, assertion binding edges, script mapping edges, and security state edges. Variable reference edges indicate that a variable-bound node is referenced by a relation node, obtained from the variable binding relationships in D2; relation deduction edges indicate the derivation effect of relation nodes on intermediate or final conclusions; assertion binding edges indicate the correspondence between conclusion nodes and assertion nodes; script mapping edges indicate the execution correspondence between assertion nodes and script nodes; and security state edges indicate the check relationship between script nodes and security state nodes.

[0065] In the node attribute set, the node attributes of the original parsing step node, variable binding node, relation node, intermediate conclusion node, and final conclusion node are all obtained from the structured logic chain D2. Assertion nodes and script nodes are obtained from the intermediate conclusion node and final conclusion node; the node attributes of the safe state node are obtained after a safety check by S3.

[0066] S3: The system generates D4 based on D3 and performs a static security check on D4 before execution.

[0067] This step is used to verify the script nodes in the dependency graph D3.

[0068] Specifically:

[0069] First, based on the script node and its corresponding attributes, the set of script nodes to be verified and the execution script text to be verified are obtained, and an assertion mapping table is established, thus forming a verification script object D4. D4 contains script nodes, execution script text, assertion mapping table, exception recording rules, and whitelist configuration.

[0070] The assertion mapping table records the original parsing step identifier, assertion node identifier, script node identifier, assertion expression, expected state, and failure backtracking target for each assertion node.

[0071] Secondly, static security checks are performed on the verification script objects. The static security checks are based on whitelist configurations. Each executable script in the verification script object is compared against the whitelist for at least one type of import statement, function call statement, external access statement, dynamic import behavior, and operator set. For example, whether the import statement in the script is in the whitelist, whether the function call is allowed, whether there is external file or network access behavior, whether there is dynamic import, and whether the operator set used is legal.

[0072] When all execution scripts pass the static security check, the system executes all execution scripts sequentially in a restricted sandbox. Based on the exception logging rules, it captures the execution results, intermediate assertion results (i.e., the assertion results corresponding to the execution scripts), execution logs, and all exception types and information that occur during execution. These are then combined to obtain the execution and security state object D5. The system also updates the attribute of all security state nodes associated with script nodes to "Pass".

[0073] When any script node fails to meet the whitelist constraints, the entire verification script object for this round does not enter the restricted sandbox execution environment. Instead, in the parsing verification dependency graph D3, the node attributes of the security status node corresponding to the script node that fails to meet the whitelist constraints are modified to a violation status to mark the corresponding script node as having a security issue. A security violation difference item is then generated, which includes the violation node identifier, violation type, violation evidence, and the original parsing step identifier. The resulting execution and security status object D5 at this point only includes the security violation difference item.

[0074] S4: The system performs difference comparison and failure propagation analysis based on D5.

[0075] Based on the error threshold configuration, this step compares the execution and security status object D5 obtained in step S3 with the reference answer or target conclusion in D1 and the intermediate and final conclusions in the test question analysis text to be verified, in turn, to determine the execution differences.

[0076] When a numerical deviation, assertion failure, or script execution anomaly is found after comparison, the system first identifies the assertion node that failed or the script node that executed abnormally based on the parsing and verification dependency graph D3. Then, it searches backward along the dependency edges in D3 to find the preceding variable binding node, relation node, and original parsing step node.

[0077] When there is an execution exception (such as execution failure) in the script, the script node is recorded as the execution exception node, and its corresponding original parsing step node, relation node or conclusion node is recorded according to the assertion mapping table; when it is impossible to continue to locate a more preceding node, the script node is regarded as the root cause failure node, and a failure evidence chain is generated.

[0078] When an assertion fails, the failure is propagated from the preceding node. The assertion node corresponding to the failed assertion is marked as a secondary failure node. The source node of the error is analyzed and marked as the root cause failure node, and a chain of failure evidence is generated.

[0079] If no difference is found after comparison, there are no failed nodes, and there is no need to distinguish between root cause failure nodes, secondary failure nodes, and safe failure nodes. The minimum difference subgraph generated in S5 will be empty.

[0080] It should be noted that when the static security check fails in S3, all execution scripts will not enter the sandbox for execution. At this time, there are no failed assertion nodes or execution exception nodes. The only failure node is the security state node of the violation state. The system marks the corresponding security state node as a security failure node and uses the script node associated with it as the cause node of the security constraint failure. Furthermore, it combines the original parsing step node and the nodes directly associated with the script node to form a security violation evidence chain.

[0081] Based on the aforementioned root cause failure nodes, secondary failure nodes, failure evidence chains or safe failure nodes, and safe violation evidence chains, the system forms a minimum connected node set and evidence chain for extracting the minimum difference diagnostic subgraph, and then passes it to S5.

[0082] S5: The system generates D7 based on D6.

[0083] Based on the results obtained in S4, this step extracts the minimum difference diagnostic subgraph and generates a difference diagnostic report and a structured feedback package D7 based on the minimum difference diagnostic subgraph D6.

[0084] If a safe failure node exists, the minimum difference diagnostic subgraph consists of the safe failure node, the script node with a safe state edge to the safe failure node, the node corresponding to the original parsing step, and the directly related nodes in the safety violation evidence chain; if no safe failure node exists, the minimum difference diagnostic subgraph is formed based on the minimum connected node set and the failure evidence chain.

[0085] If a root cause failure node is found in the difference comparison and failure propagation analysis in S4, the difference diagnosis report generated in D7 shall include at least the root cause failure node (which variable binding, relation or conclusion is wrong), the failure propagation path (how the error is passed from one node to other nodes), the secondary failure node, the safe failure node, the corresponding original analysis step and failure evidence; the structured feedback package shall include at least the failure type, failure step, anomaly information, correction direction, processing action and automatic pass gating conclusion.

[0086] Failure types need to be categorized based on the nature of the failed node, and should include at least one or more of the following: numerical deviation, step assertion failure, script construction failure, script syntax exception, runtime exception, timeout exception, non-whitelisted call, dynamic import violation, file access violation, network access violation, secondary assertion failure, and insufficient assertion coverage.

[0087] The processing action is selected based on the failure type: partial parsing and regeneration, verification script reconstruction, error threshold review, disabling automatic pass, or manual review.

[0088] In the automatic pass gating conclusion, whether automatic pass is allowed is determined based on relational assertion coverage, the number of valid assertions, whether the minimum difference diagnostic subgraph is empty, the existence of safe failure nodes, and the existence of nodes awaiting review. When the relational assertion coverage is lower than a preset coverage threshold, the number of valid assertions is zero, a safe failure node exists, the minimum difference diagnostic subgraph is not empty, or a node awaiting review exists, the system outputs an automatic pass conclusion or a pending review conclusion, and records the unmet gating conditions. Only when none of the above conditions are met, and the minimum difference diagnostic subgraph D6 is empty (i.e., no failure nodes were found), does the system output an automatic pass conclusion.

[0089] Among them, the relational assertion coverage rate is the ratio of the number of relational nodes that have generated assertions and participated in execution and difference comparison to the total number of relational nodes in the resolution and verification dependency graph, and the effective assertion number is the number of assertions that have been generated and completed execution and difference comparison.

[0090] The preset coverage threshold is used to represent the minimum relational assertion coverage required to support automatic conclusions. It is set by the user and can be determined according to the question type, subject, number of relations, accuracy requirements, and whether unit conversion is required. The preferred value range is 0.80 to 1.00.

[0091] For example, for problems involving algebraic identity manipulation and equation solving that heavily rely on derivation, a threshold of 0.95 to 1.00 is preferred; for routine numerical calculation problems, a threshold of 0.90 to 1.00 is preferred; for problems involving engineering quantities, physical quantities, chemometrics, or mechanics of materials, if unit conversions, approximate values, or significant figure processing are involved, a threshold of 0.80 to 0.95 is preferred. For problems requiring complete derivation or where the final conclusion depends on a few key relationships, the assertion coverage threshold for these key relationships is preferably set to 1.00.

[0092] It is important to note that security-type failure nodes are not directly treated as errors in the parsing conclusion. In other words, the system will not determine that the mathematical logic of the question parsing text is flawed simply because a script security violation occurs; instead, it will treat it as a script security constraint failure. Therefore, when a failure node is only a security-type failure node, the processing action in the structured feedback package can only be selected from verifying script reconstruction, disabling automatic passing, or manual review.

[0093] The present invention further explains and illustrates the proposed method in conjunction with several specific embodiments:

[0094] (1) Example 1: Normal parsing path

[0095] First, the system receives the parsing diagnostic input packet D1. The question text in D1 is "Given x=2, y=3, find z=x+y", the parsing text to be verified is "From x=2, y=3, we can get z=x+y=5, therefore the final result is 5", the reference answer is 5, and the error threshold is 0.

[0096] In S1, the system extracts variable binding nodes V1 and V2 from the parsed text, where V1 represents x=2 and V2 represents y=3; it extracts relation node R1, where R1 represents z=x+y; it extracts intermediate conclusion node C1, where C1 represents z=5; and it extracts final conclusion node F1, where F1 represents the final result as 5. The system writes these nodes into D2 to obtain a structured logic chain.

[0097] Among them, the variable binding nodes V1 and V2 are determined by combining the variable definition statements such as "given x=2, y=3" in the question text and "by x=2, y=3" in the analysis text. The identified variable binding relationships are as follows: the value of variable x is 2, which comes from the question stem; the value of variable y is 3, which comes from the question stem; the value of variable z is 5, which is bound to x and y through the relation z=x+y, which comes from the calculation result of the relation.

[0098] The relation node R1 is obtained by further normalizing the relation part of the expression fragment “z=x+y=5”.

[0099] Intermediate conclusion node C1 and final conclusion node F1 are identified by the concluding statements “can be obtained” and “final result is”. Since F1 is located in the last parsing step, it is marked as the final conclusion node.

[0100] In S2, the system constructs and verifies the dependency graph D3. In D3, V1 and V2 are connected to R1 via variable reference edges, R1 is connected to C1 via relation deduction edges, C1 is connected to F1 via relation deduction edges, C1 and F1 are connected to assertion nodes A1 and A2 via assertion binding edges, and A1 and A2 are connected to script nodes SC1 and SC2 via script mapping edges. The system also generates security state nodes SS1 and SS2 for SC1 and SC2.

[0101] In S3, the system performs static security checks on SC1 and SC2. Since the script only contains basic addition operations and equality comparisons, both SC1 and SC2 pass the security check. The system then executes the script in the restricted sandbox, obtaining z=5, and both A1 and A2 meet the expected state.

[0102] In S4, the system did not identify any failed assertion nodes, execution exception nodes, or safe state nodes with violation status. The set of failed nodes and the chain of failed evidence were empty. Therefore, in S5, the extracted minimum difference diagnostic subgraph was empty. In the generated D7, the overall status was recorded as passed, the minimum difference diagnostic subgraph was empty, and automatic pass was allowed.

[0103] (2) Example 2: Separation of root cause failure and secondary failure

[0104] The system receives another parsed text: "From x=2, y=3, we can get z=xy=5, therefore the final result is 5." This parsed text states the final result as 5, but the relation uses xy.

[0105] In S1, the system extracts variable binding node V1 representing x=2, variable binding node V2 representing y=3, relation node R1 representing z=xy, intermediate conclusion node C1 representing z=5, and final conclusion node F1 representing the final result as 5. In S2, the system generates assertion nodes, script nodes, and security state nodes to construct D3, and connects R1 to C1 and F1.

[0106] In S3, the script performs a static security check and runs in a restricted sandbox, obtaining the actual computation result of R1, z=-1. However, the assertion node A1 corresponding to C1 requires z=5, and the comparison reveals an inconsistency, therefore the assertion of A1 fails; the assertion of the assertion node A2 corresponding to F1 also fails.

[0107] In S4, the system analyzes the sources of failure along the reverse dependency edges of D3. According to the reference answer, z=5, therefore the failure of A1 is directly caused by the inconsistency between the result of R1 and C1. The system then determines R1 as the source of the error, and thus R1 is marked as the root cause failure node. The failure of A2 is caused by the subsequent propagation between C1 and F1, and is marked as a secondary failure node. Finally, the minimum difference diagnostic subgraph D6 generated by the system contains R1, C1, A1 and their corresponding original analytical steps, and does not consider F1 as an independent root cause.

[0108] In S5, D7 records the root cause of the failure as relation node R1, and the failure propagation path is R1->C1->F1. The handling action is to regenerate the local parsing. Therefore, the system does not need to rewrite the entire parsing; it only needs to correct the original parsing steps that use the incorrect relation.

[0109] (3) Example 3: Separation of security violations and parsing errors

[0110] The system receives the test question analysis text to be diagnosed, and generates an execution script in S1-S2 to verify the conclusions and relations in the analysis text. After conversion, a script node SC3 is formed.

[0111] For ease of explanation, in this embodiment, the script fragment corresponding to SC3 contains one of the following non-whitelisted statements: `import os`, `open(' / tmp / a.txt')`, or `__import__('os')`. This statement may originate from an abnormal fragment in the parsed text or from an incorrect conversion during the script construction process. The whitelist configuration only allows basic arithmetic operations, comparison operations, and preset basic functions in `math` or `numpy`, and does not allow dynamic imports, file access, network access, or system command calls.

[0112] In S3, during the pre-execution static security check, the system identifies that SC3 contains dynamic import statements or file access statements that do not meet the whitelist constraints, and therefore does not proceed to the script execution phase. In D3, the system records the node attributes of the security state node SS3 associated with this script node as a violation state and generates a security violation difference item, which is written to the execution and security state object D5. This security violation difference item includes the script node identifier of SC3, the violation type, the evidence of the violation, and the identifier of the corresponding original parsing step.

[0113] In S4, the system marks the safe state node SS3 of the violation state as a safe failure node, instead of directly marking it as an incorrect parsing conclusion. At the same time, the script node SC3, which has a safe state edge with SS3, is regarded as the cause node of the safety constraint failure, and a safety violation evidence chain is formed.

[0114] In S5, the system extracts the minimum difference diagnostic subgraph based on the security violation evidence chain. This subgraph contains script node SC3, security state node SS3 of the violation state and its associated edges. The system generates a structured feedback package and selects to verify script reconstruction, prohibit automatic passage or manual review as the processing action.

[0115] This embodiment illustrates that the present invention can separate script security violations from parsing logic errors, avoiding the misjudgment of security violations caused by abnormal text fragments in script construction or parsing as incorrect test question analysis conclusions.

[0116] (4) Example 4: Assertion Insufficient Coverage Gating

[0117] The system receives a parsed text containing only the final answer: "Substituting the values, the answer is 5." This text lacks variable binding relationships and key relational expressions, containing only the final conclusion. The system can identify that F1 indicates the final result is 5 from the text, but it cannot identify enough variable binding nodes or relational expression nodes.

[0118] In S1, the system marks the steps involving missing variable bindings and relations as nodes to be reviewed. In S2, the system is unable to create assertion nodes for the relations, therefore the relation assertion coverage is lower than the preset coverage threshold. Even if the final answer matches the reference answer, the feedback gating module in S5 still outputs a conclusion that automatic approval is not allowed and sets the processing action to manual review or requiring supplementary parsing steps.

[0119] In this embodiment, the parsed text of this type only provides the final answer but lacks key relational expressions. The corresponding preset coverage threshold is preferably 0.90, or preferably 1.00 in scenarios requiring complete derivation. Since no key relational nodes and valid assertions are formed that can be used for assertion coverage calculation, the system treats it as insufficient coverage, and the relational assertion coverage rate does not meet the preset coverage threshold.

[0120] This embodiment illustrates that the present invention focuses not only on "whether an error is detected," but also on "whether the verification coverage is sufficient to support automatic pass." When a critical relation is not covered by assertions, the system will not output pass simply because the final answer is consistent.

[0121] (5) Example 5: Diagnosis of textual differences in the analysis of mechanics of materials test questions

[0122] In the materials mechanics analysis sample, the system receives stress components σx=50MPa, σy=0MPa, τxy=20MPa, and the analysis text to be verified. The analysis text requires calculation of principal stresses, principal plane azimuth, and maximum shear stress. The conclusions are as follows: mean normal stress is 25 MPa, half-difference term is 25 MPa, principal stress radius is sqrt(1025), approximately 32.0 MPa; maximum principal stress in the plane is approximately 57.0 MPa, minimum principal stress in the plane is approximately -7.0 MPa, maximum shear stress is approximately 32.0 MPa, and principal plane azimuth is +19.3 degrees.

[0123] In S1, the system identifies the average normal stress variable nodes, the semi-difference variable nodes, the principal stress radius variable nodes, the principal stress variable nodes, the principal plane angle variable nodes, the maximum shear stress variable nodes, and the corresponding relational nodes.

[0124] Given the inputs σx=50, σy=0, and τxy=20, the system calculates the average normal stress as 25 MPa, the half-difference term as 25 MPa, and the principal stress radius as sqrt(1025), approximately 32.0 MPa, based on the executed script. The maximum principal stress in the plane is approximately 57.0 MPa, the minimum principal stress in the plane is approximately -7.0 MPa, and the maximum shear stress is approximately 32.0 MPa. For the principal plane azimuth, under the notation convention adopted in this embodiment, tensile stress is taken as positive, τxy=20 MPa is substituted according to the input normal shear stress, and α0 represents the angle of rotation from the positive x-axis to the normal direction of the plane of maximum principal stress, with counterclockwise being positive and clockwise being negative. Therefore... It is approximately -19.3 degrees Celsius.

[0125] Execute S2-S4 sequentially. In S4, the system compares the script execution results with the conclusion node and the reference answer. It finds that the negative sign in the formula for calculating the principal plane azimuth is omitted in the parsed text. Therefore, the calculation of the maximum principal stress and the minimum principal stress in the plane is correct, but the calculation of the principal plane azimuth is incorrect. As a result, the assertion related to the magnitude of the principal stress can pass, but the assertion of the principal plane azimuth fails.

[0126] At this point, the dependent edge reverse identification based on D3 results in the principal plane angle relation node as the root cause failure node, the principal plane angle intermediate conclusion node and the principal plane angle final conclusion node as the secondary failure node, and the principal stress magnitude node does not belong to the difference diagnosis subgraph.

[0127] The minimum difference diagnostic subgraph output by S5 only covers the principal plane angle calculation chain and prompts for verification of shear stress sign conventions and tan(2α0) calculation formula in the generated structured feedback package.

[0128] As can be seen from this embodiment, the present invention can distinguish between locally failed and non-failed links in the same analytical text, avoiding the need to rewrite the entire material mechanics analysis due to an error in a single angle formula.

[0129] Based on the same inventive concept, this invention proposes a test question parsing text difference diagnosis system based on dependency graphs and sandboxes, including:

[0130] The logic chain extraction module is used to obtain the reference answers to quantitative calculation questions, the analysis text of the questions to be diagnosed, and the error threshold configuration, and to extract the structured logic chain from the analysis text of the questions.

[0131] The dependency graph parsing and verification module generates assertions and corresponding execution scripts based on the relationships, intermediate conclusions, and final conclusions in the structured logic chain, thereby obtaining the dependency graph parsing and verification.

[0132] The execution and security status generation module is used to perform static security checks on the execution scripts. If all execution scripts pass, they are executed in a restricted sandbox and an execution and security status object is obtained. If any execution script fails, a security violation difference item is generated.

[0133] The difference comparison and failure propagation analysis module is used to classify nodes in the parsing verification dependency graph into root cause failure nodes, secondary failure nodes, or safety-type failure nodes based on the difference comparison results or safety violation difference items.

[0134] The feedback gating module is used to extract the minimum difference diagnostic subgraph from the parsing and verification dependency graph based on the partitioning results; based on the minimum difference diagnostic subgraph, it generates a difference diagnostic report for the test question parsing text diagnostic results and a structured feedback package containing the automatically passed gating conclusion.

[0135] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0136] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution, thereby creating the corresponding logical device.

[0137] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A text discrepancy diagnosis method for test question analysis based on dependency graphs and sandboxes, characterized in that, include: S1, obtain the reference answers to quantitative calculation questions, the analysis text of the questions to be diagnosed, and the error threshold configuration, and extract the structured logic chain from the analysis text of the questions; S2, based on the relational expressions, intermediate conclusions, and final conclusions in the structured logic chain, generate assertions and corresponding execution scripts, and further obtain a parsing and verification dependency graph; the parsing and verification dependency graph at least includes nodes reflecting the original parsing steps, variable bindings, relational expressions, conclusions, assertions, scripts, and security states, as well as edges reflecting the dependencies, derivations, bindings, mappings, and checks between nodes; S3, perform static security checks on the execution scripts. If all execution scripts pass the check, they are executed in a restricted sandbox and an execution and security status object is obtained. If any execution script fails the check, a security violation difference item is generated. S4, based on the difference comparison results or security violation difference items, divide the nodes in the parsing verification dependency graph into root cause failure nodes, secondary failure nodes, or security failure nodes; the difference comparison results are obtained by comparing the execution and security status objects with the reference answer and the test question parsing text in sequence according to the error threshold configuration; S5. Based on the partitioning results, the minimum difference diagnostic subgraph is extracted from the parsing and verification dependency graph, and a difference diagnostic report and a structured feedback package containing automatically passed gating conclusions are further generated.

2. The text difference diagnosis method for test question parsing based on dependency graphs and sandboxes according to claim 1, characterized in that, In S1, the structured logic chain includes variables, relational expressions, intermediate conclusions, final conclusions, original parsing step identifiers, and the binding relationships between variables.

3. The text difference diagnosis method for test question parsing based on dependency graphs and sandboxes according to claim 1, characterized in that, In S2, the nodes of the parsing and verification dependency graph include original parsing step nodes, variable binding nodes, relation nodes, intermediate conclusion nodes, final conclusion nodes, assertion nodes, script nodes, and security state nodes; the edges include variable reference edges between variable binding nodes and relation nodes, relationship derivation edges between relation nodes and intermediate or final conclusion nodes, assertion binding edges between conclusion nodes and assertion nodes, script mapping edges between assertion nodes and script nodes, and security state edges between script nodes and security state nodes.

4. The text difference diagnosis method for test question parsing based on dependency graphs and sandboxes according to claim 1, characterized in that, In S3, the process of performing static security checks on the execution script is as follows: the execution script is compared with a preset whitelist. If the execution script contains script elements or execution behaviors that are not in the whitelist, the execution script is considered to have failed the static security check; otherwise, it is considered to have passed.

5. The text difference diagnosis method for test question parsing based on dependency graphs and sandboxes according to claim 4, characterized in that, The whitelist is used to restrict one or more of the following: import statements, function call statements, external access statements, dynamic import behavior, and operator sets.

6. The text difference diagnosis method for test question parsing based on dependency graphs and sandboxes according to claim 1, characterized in that, In S3, the execution and security status object includes the execution result of the execution script, intermediate assertion results, execution log, and all exception types and exception information that occur during the execution process.

7. The text difference diagnosis method for test question parsing based on dependency graphs and sandboxes according to claim 1, characterized in that, In step S4, the process of dividing nodes based on the difference comparison results or safety violation difference items is as follows: If there are security violation discrepancies, the security status node corresponding to the executed script will be classified as a security failure node in the parsing and verification dependency graph, and a security violation evidence chain will be generated. If the difference comparison results show that the script execution is abnormal, then all the predecessor nodes of the script node are found by following the reverse dependency edge of the parsing and verification dependency graph. If there is no source node that causes the execution abnormality among the predecessor nodes, then the script node is classified as the root cause failure node; otherwise, the source node is classified as the root cause failure node. If the difference comparison results show an assertion node that fails, then the node is classified as a secondary failure node. Starting from this node, all predecessor nodes of the node are found along the reverse dependency edge of the parsing verification dependency graph. The source node that caused the error among the predecessor nodes is marked as the root cause failure node.

8. The text difference diagnosis method for test question parsing based on dependency graphs and sandboxes according to claim 1, characterized in that, In step S5, the process of extracting the diagnostic sub-map with the minimum difference is as follows: If a safe failure node exists, the minimum difference diagnosis subgraph consists of the safe failure node, the script node with a safe state edge to the safe failure node, the node corresponding to the original parsing step, and the directly related nodes in the safety violation evidence chain. If no safe failure node exists, the minimum connected node set consisting of the root cause failure node and the corresponding failure evidence chain is found in the parsing verification dependency graph, which is the minimum difference diagnosis subgraph.

9. The text difference diagnosis method for test question parsing based on dependency graphs and sandboxes according to claim 1, characterized in that, In S5, when the relational assertion coverage rate in the parsing and verification dependency graph is greater than or equal to the preset coverage threshold, the number of valid assertions is not zero, there are no safe failure nodes, there are no nodes to be reviewed, and the minimum difference diagnosis subgraph is empty, the automatic pass gating conclusion in the structured feedback package outputs an automatic pass conclusion, indicating that the test question parsing text has passed the diagnosis. If any gating condition is not met, the automatic pass gating conclusion in the structured feedback package outputs an automatic pass conclusion or a node to be reviewed conclusion, and records the unmet gating condition. The nodes to be reviewed include the analytical steps in the test question analysis text that cannot be converted into symbolic expressions, lack executable relational expressions, or lack variable binding relationships but contain conclusive text, as well as their corresponding intermediate or final conclusions; in the analysis verification dependency graph, the nodes to be reviewed do not have associated assertion nodes; The relational assertion coverage rate is the ratio of the number of relational nodes in the parsing verification dependency graph that have corresponding assertions and whose assertions have been executed and whose differences have been compared to the total number of relational nodes. The number of valid assertions refers to the number of assertion nodes that have been generated and executed and compared in the dependency graph of the parsing verification.

10. A test question parsing text difference diagnosis system based on dependency graphs and sandboxes, used to implement the test question parsing text difference diagnosis method based on dependency graphs and sandboxes as described in claim 1, characterized in that, include: The logic chain extraction module is used to obtain the reference answers to quantitative calculation questions, the analysis text of the questions to be diagnosed, and the error threshold configuration, and to extract the structured logic chain from the analysis text of the questions. The dependency graph construction module generates assertions and corresponding execution scripts based on the relationships, intermediate conclusions, and final conclusions in the structured logic chain, thereby obtaining the dependency graph. The execution and security status generation module is used to perform static security checks on the execution scripts. If all execution scripts pass, they are executed in a restricted sandbox and an execution and security status object is obtained. If any execution script fails, a security violation difference item is generated. The difference comparison and failure propagation analysis module is used to classify nodes in the parsing verification dependency graph into root cause failure nodes, secondary failure nodes, or safety-type failure nodes based on the difference comparison results or security violation difference items. The feedback gating module is used to extract the minimum difference diagnostic subgraph from the parsing and verification dependency graph based on the partitioning results; based on the minimum difference diagnostic subgraph, it generates a difference diagnostic report for the test question parsing text diagnostic results and a structured feedback package containing the automatically passed gating conclusion.

Images