Code error feedback method combining static analysis and dynamic analysis

Abstract

The invention relates to a code error feedback method combining static analysis and dynamic analysis, and mainly solves the problems of insufficient error feedback and low efficiency in an Online Judge system: reconstructing codes of correct and error versions, including alignment and renaming of variables, and rearranging statements on the premise of not influencing semantics; static and dynamic multi-level alignment is carried out, in the static alignment, an editing script with the minimum cost is calculated through a heuristic editing distance algorithm, an alignment block and a difference list are obtained, in the dynamic alignment, compiling execution is carried out on source codes of two versions, test cases which do not pass operation are dynamically printed through gdb, and execution track information of the test cases is dynamically printed through gdb; error positioning is achieved, data flow backtracking analysis and scoring are carried out on error positions, and error feedback is generated. The method has the characteristics of high execution efficiency, high accuracy and easier understanding of generated feedback, automatic feedback of error codes can be realized, and the learning efficiency of students in programming tasks is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer science, particularly software technology. It proposes a code error feedback method that combines static and dynamic analysis, effectively solving the problem that current Online Judge (OJ) systems cannot accurately locate and report errors in student code, thereby improving teaching effectiveness and student learning efficiency. Background Technology

[0002] Online Judge (OJ) systems can automatically evaluate student-submitted code, playing a crucial role in teaching, competitions, and job applications. The main workflow of an OJ system involves compiling and executing the student's code, running several test cases. Each test case has a standard input and an expected output. The OJ system accepts the standard input, receives the actual output, and then compares the student's code's actual output with the expected output. If they match, the code passes; otherwise, it fails. Therefore, student code may receive an accepted answer or a wrong answer. Furthermore, during compilation and execution, errors such as compilation errors, runtime errors, time limit expiration, and memory limit expiration may occur.

[0003] In practical use, the existing OJ system has some shortcomings: (1) The prompts are simple, usually simple feedback such as WrongAnswer, which cannot provide more effective information. Beginners cannot independently and accurately locate the error code and correct it; (2) The efficiency of manual feedback is low. If students seek help from teachers or teaching assistants, manual search for code errors can provide accurate help, but it is time-consuming and laborious. Therefore, in teaching, only failed test cases are usually provided to students, which is very limited in helping students. Students need to have the ability to analyze and debug code; (3) Manual feedback is not fair enough. Students cannot take the initiative to seek help due to personality and other reasons, resulting in different feedback received by students with different personalities.

[0004] To address the shortcomings of existing online judge (OJ) systems, an effective solution is to provide an online code error feedback system. This would not only provide students with accurate error feedback at any time, but also ensure that everyone has an equal opportunity to receive help. The error feedback approach is as follows: In the history of correct commits, find the version closest to the erroneous code; analyze and align the two versions; compare the differences between the correct and erroneous code; infer the location of the error based on the differences; and provide specific feedback and suggestions for improvement.

[0005] However, due to the complexity of programming languages ​​and the significant differences in student code implementations, it is almost impossible for a correct version and an incorrect version to differ only in one place. Therefore, it is necessary to increase the fault tolerance of alignment through static refactoring (e.g., renaming variables, rearranging statement order), and to compensate for the semantic gaps in static alignment through dynamic execution path analysis (e.g., statements in two versions have different syntax but similar functionality). Then, multi-level alignment using static and dynamic methods is used to obtain the final alignment information. Finally, based on the alignment information, the possible locations of errors are inferred, and different error locations are scored and ranked to generate appropriate error feedback. Summary of the Invention

[0006] The problem this invention aims to solve is that traditional online judge (OJ) systems provide relatively simple prompts, making it difficult to accurately locate error codes. In addition, human feedback is inefficient and unfair.

[0007] The technical solution of the present invention includes: collecting correct and incorrect submissions of online judge problems, performing alignment analysis on input variables and variables with initial values, normalizing and reordering statements within basic blocks, and using a multi-level heuristic strategy to perform alignment analysis on lines of code, and finally performing error location analysis based on the code alignment results and generating an error feedback report.

[0008] This invention includes the following specific steps:

[0009] 1) Alignment analysis for input variables and variables with initial values: For the standard input of the OJ problem, even if the variable names are different in the two versions, they can be aligned. Similarly, the output can also be aligned. For some variables that do not require input and are directly assigned initial values, their execution trajectory is printed out through dynamic analysis. If the value changes of the two variables are consistent during the execution process, they are aligned one by one.

[0010] 2) Normalize and reorder the statements within the basic block: Based on 1), reorder the statement sequence, that is, reorder the statements in the basic block according to the lexicographical order of the first variable of the statement, while ensuring that the original data dependencies are not changed.

[0011] 3) Employ a multi-level heuristic strategy to perform alignment analysis on lines of code: Based on 2), use static alignment methods to obtain basic alignment blocks; by obtaining the path trajectory during execution, including variable values, stack information, current line number, etc., calculate the similarity of source code based on the similarity of execution trajectory to obtain dynamic alignment information.

[0012] 4) Error location analysis based on code alignment results: Based on static and dynamic alignment, check whether the execution trajectory values ​​corresponding to the statements in the two versions are consistent. If the values ​​are inconsistent, it is a possible error location; summarize the analysis results of all potential error locations and generate an error feedback report.

[0013] In step 1), an alignment analysis is performed on the input variables and the variables with initial values.

[0014] First, for variables with standard input, although the variable names may differ in different versions, a corresponding relationship can be established for alignment based on their input order; for variables with initial values, the value execution trajectory during the execution process is recorded through dynamic analysis. If the value changes of two variables are consistent during the execution process, then the two variables are considered to be aligned.

[0015] Secondly, a data dependency graph between variables is constructed and variable propagation alignment is performed. Constructing a dependency graph based on the data flow relationships between variables requires handling dependency graphs that do not contain loops. That is, using the aligned variables obtained in step 1), if the other variables that variables in two versions depend on can be matched one-to-one, then these two variables can also be aligned. This graph propagation method gradually expands the range of alignable variables.

[0016] Finally, we can obtain the variable correspondence between the two versions and replace the variable names in the correct version with the corresponding variable names in the incorrect version.

[0017] Step 2) Normalize and reorder the statements within the basic block.

[0018] Analyze the dependencies between statements by performing data flow analysis on the statements within the basic block to identify the dependencies between them. If statement 2 uses a variable defined in statement 1, then a dependency exists between statement 1 and statement 2, and the relative position of statement 1 before statement 2 must be maintained during reordering. For statements without dependencies, perform standardized sorting. That is, for statements within the basic block that do not have mutual dependencies, use a uniform sorting rule (such as alphabetical order of variable names) to reorder them, ensuring that different versions of the code have a consistent statement order after reordering, and maintaining the final order under dependency constraints.

[0019] Step 3) Employ a multi-level heuristic strategy to perform alignment analysis on lines of code, specifically including the following processing:

[0020] First, based on the variable alignment and statement sorting results completed in steps 1) and 2), completely identical statements in the two versions are identified as alignment anchors. Using these anchors as a benchmark, similar statements are searched for upward and downward.

[0021] Secondly, for statements that do not match completely, the similarity between statements is calculated, and when the similarity exceeds a preset threshold, it is marked as an aligned statement.

[0022] Finally, for statements that are still not aligned, a behavioral similarity evaluation method based on dynamic analysis is introduced. This method defines the program execution state as a triple (Li, C, S), where Li represents the i-th execution of line L, C represents the source code of that line, and S is a state dictionary recording the values ​​of each variable at the execution point. By collecting the execution paths of the program on failed test cases, the consistency of the variable state dictionaries of the misaligned statements is analyzed, thereby evaluating the semantic similarity between statements and ultimately determining the alignment relationship of these statements.

[0023] Step 4) Perform error localization analysis based on code alignment results. That is, based on the static and dynamic alignment relationships obtained in Steps 1) to 3), perform comparative analysis of the execution trajectories of the two versions of the program.

[0024] First, the complete execution trajectory of the program under test cases is collected. For each pair of aligned statements, the state and values ​​of all relevant variables at the same execution point are compared. When inconsistent variable values ​​are found during the execution of the aligned statements, the location is marked as a potential error location. By performing data flow backtracking analysis on these locations, the root cause of the difference in variable values ​​is determined.

[0025] Secondly, by combining the program's contextual information, including the location of variable definitions, data dependencies, and control flow information, the credibility of each potential error location is evaluated by comparing the consistency of behavior, the degree of abnormality of variable changes, and the complexity of the difference propagation chain between the two versions of the program in the execution trajectory.

[0026] Finally, the analysis results of all potential error locations are summarized, and the error locations with the highest scores are identified by comparing their confidence levels. An error feedback report is then output, which includes the error location, information on the differences in relevant variable values, and data dependency links.

[0027] The present invention, by adopting the above technical solution, has the following advantages:

[0028] 1. High execution efficiency: This invention can provide timely feedback, which is faster than manual error finding by teachers or teaching assistants, and it is more convenient for students to obtain feedback.

[0029] 2. High accuracy: The accuracy of this invention is higher than that of some current purely static or purely dynamic analysis tools, and because it uses reconstruction technology, it does not require strict consistency between the correct and incorrect versions.

[0030] 3. Improved fault tolerance: This invention combines static similarity and dynamic behavior analysis, which can more comprehensively establish the correspondence between two versions of code, requiring fewer correct version samples. Attached Figure Description

[0031] Figure 1 Flowchart of a code error feedback method combining static and dynamic analysis

[0032] Figure 2 A schematic diagram of a code error feedback method combining static and dynamic analysis. Detailed Implementation

[0033] To make the content, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will now be described in detail with reference to the embodiments and accompanying drawings. It should be noted that the described embodiments are only some specific implementations of this invention and are not intended to limit all implementations. Those skilled in the art should understand that any modifications, equivalent substitutions, or improvements made based on the content described in this invention without departing from the concept and essence of this invention are within the protection scope of this invention.

[0034] Furthermore, the various steps and components in the embodiments of the present invention can be arranged, designed, and implemented in multiple ways. Therefore, the following detailed description of the present invention is limited to specific embodiments and is intended to better understand the technical solution of the present invention, rather than limiting the scope of patent protection. Other modifications and extensions that can be obtained by those skilled in the art based on the technical solution of the present invention without inventive effort should be considered as covered by the scope of protection of the present invention.

[0035] This invention proposes a code error feedback method that combines static and dynamic analysis. This method performs variable name normalization, statement order standardization, and multi-level code alignment analysis on two versions of the program, combining static analysis and dynamic execution information to achieve precise location of program errors. The specific implementation process of this invention includes four main steps.

[0036] Step 1: Normalize the variables in the program.

[0037] First, Treesitter is used to parse the code AST, obtain the input and output statements, the location of variable definitions, analyze the usage of variables in the program, and build a dependency graph of variables in the program based on the definition and usage of variables.

[0038] Next, identify all the input statements for variables that appear in the code. Variables that can be aligned in both versions are read in the same order, meaning the first variable read in the correct version corresponds to the first variable read in the incorrect version, and so on.

[0039] Additionally, variable alignment is performed based on the execution trajectory. That is, if the variables in two versions change in the same way in the execution trajectory, then they can also be considered aligned.

[0040] Then, based on data dependencies, the alignment relationship between the two versions of the variable can be propagated: if the correct and incorrect variables depend on variables that can be matched one-to-one, and if their operations are the same, then they can also be aligned. The alignment relationship is propagated according to this rule.

[0041] Finally, rename all variable names from the correct versions to those from the incorrect versions.

[0042] Step 2: Reorder the statements within the basic block.

[0043] First, perform control flow analysis on the program and divide the code into basic blocks according to their scope.

[0044] Next, for each statement within a basic block, the dependencies between statements are established. Data dependencies are defined as follows: if statement 2 uses a variable defined or reassigned in statement 1, then statement 2 is considered dependent on statement 1; if statement 2 does not use a variable defined or reassigned in statement 1, then statement 2 is not dependent on statement 1; if two statements do not depend on each other, then they are independent.

[0045] Finally, the statements within the basic block are sorted according to the following principles: for statements with dependencies, the dependent statement comes first, followed by the dependent statement; for independent statements, the alphabetical order of the variables can be used for sorting. Variables in both versions can follow the same sorting rules, resulting in the longest possible sequence of similar subsequences.

[0046] Step 3: Use a multi-level strategy to perform alignment analysis on the two versions of the code.

[0047] First, use the static alignment tool to obtain basic static alignment analysis of the alignment statements.

[0048] The code reads two versions of the code file and uses the Python third-party library difflib to perform heuristic alignment between the two versions of the code. Specifically, it adds blank lines to complete the code of the two versions to the same length, and then uses the edit distance algorithm to calculate the minimum cost operation method, that is, to make the two versions of the code identical through insertion, deletion and replacement operations.

[0049] Differences are categorized according to operation type (insertion, deletion, replacement), generating a list of difference sequences. Each difference consists of three parts: the original code snippet, the modified code snippet, and the operation type. This yields a list of identical statements and a list of differences requiring specific operations to be identical. For replacement operations, the statement similarity is calculated separately; if the statement similarity exceeds a threshold, it is considered alignable.

[0050] Next, dynamic analysis is used to align the parts that were not aligned in the static analysis.

[0051] For statements where alignment cannot be determined through exact matching, especially substitution operations, which may be different ways of writing the same semantics, a dynamic analysis-based approach is introduced. A Python script is used to automatically execute the gdb tool to obtain the necessary information, which is then converted into a triple (Li, C, S), where Li represents the i-th execution of line L, C is the source code content, and S is a dictionary of variable states.

[0052] For example, for the statement "x = y + 1", its state dictionary might be {"x": 5, "y": 4}, indicating that when the statement is executed, the value of y is 4, and after execution, the value of x is 5.

[0053] By collecting the execution trajectory of the program on failed test cases, comparing the semantic consistency of the state of unaligned statements, evaluating their semantic similarity, and finally determining the alignment relationship between statements.

[0054] Step 4: Error location analysis based on code alignment results.

[0055] First, compare the variable states. For each pair of alignment statements, compare the states and values ​​of all relevant variables at their execution point. If there is a difference in the state or value of a variable, record the line number of the statement, the number of executions, the current value of the variable, and the test case information used. This statement is a potential error location.

[0056] Next, the data flow is analyzed back, which includes the following steps.

[0057] 1) Construction of the difference propagation link: Data flow backtracking is performed on the potentially erroneous locations marked, tracing from the current variable state to the variable definition location or the most recent assignment operation, and analyzing the root cause of the difference.

[0058] 2) Backtracking path recording: For each variable value difference, construct its data dependency path, trace back from the difference location to the root location, record all relevant variable and statement information, and form a complete difference propagation chain.

[0059] Next, assess the credibility of potential errors, which includes the following steps.

[0060] 1) Context analysis: By combining the program's context information, including the location of variable definitions, data dependencies, and control flow information, the credibility of each potential error location is evaluated.

[0061] 2) Credibility Score: By comparing the differences between the two versions of the program, a credibility score is assigned to each potential error location. The credibility score is composed of the following key factors: the degree of anomalousness of variable changes (S). e Execution trajectory consistency S t Frequency and position of variable S f .

[0062] ① The degree of abnormality of variable change S e It measures the degree to which the value of a variable changes across different versions of a program, denoted as:

[0063]

[0064] Where v1 and v2 are the values ​​of the variable in the two versions, and σ is the standard deviation of the variable values, used to measure the relative degree of change. e The larger the value, the more serious the error.

[0065] ② Execution trajectory consistency S t This measures the consistency of the execution path across different versions of the program, and is calculated in the following way:

[0066]

[0067] Here, P1 and P2 are the execution paths of two versions of the program. A result close to 1 indicates that the execution paths are highly consistent.

[0068] ③ Frequency and position of variable S f It measures the frequency of a variable's use in the process and its distance from error points.

[0069]

[0070] Where var is the variable used in this statement, and f var It represents the frequency of use of the variable, and max(fall) is the maximum frequency of use of the variable in the program. (1-dist(L)) var L error The variable's frequency is the distance between its most recently used location and the potential error location. The more frequently a variable appears and the more likely it is to be an error location if it's on the error propagation path, the more likely the statement is to be an error location. Otherwise, if a variable is only frequently used but not on the path, or infrequent but on the path, its impact will be minimal.

[0071] These indicators are comprehensively scored, with higher scores indicating that the location is more likely to be an actual error location. The scoring formula is as follows:

[0072] S = W e ·S e +w t ·S t +w f ·S f

[0073] Finally, generate an error feedback report. This step involves summarizing the analysis results of all potential error locations and generating an error localization report that includes the following: clearly indicating the specific location of the potential error in the code, including the file name, line number, and code snippet; describing in detail the variable value differences found at the potential error location, including the variable name, its value in the two versions and its changes, and the test case input that caused the difference; and showing the complete data dependency chain from the potential error location to the root cause location to help developers understand the propagation path of the difference.

Claims

1. A code error feedback method combining static and dynamic analysis, characterized in that: The source code is parsed into an Abstract Syntax Tree (AST) to obtain the alignment relationship between the correct and incorrect versions of variable names, and the variables in the source code are renamed accordingly. Then, based on the dependency relationship of the statements, the two versions of the source code are sorted according to the same rules. Next, a heuristic algorithm is used to perform static alignment of the source code. For the parts that cannot be aligned, the execution trajectory is obtained by compiling and running, and the information of the execution trajectory is dynamically analyzed to perform semantic alignment. Finally, for the parts that still have differences, it is evaluated whether they are error locations, and a feedback report is generated.

2. The code error feedback method combining static analysis and dynamic analysis according to claim 1, characterized in that... Includes the following steps: 1) Parse the source code into an Abstract Syntax Tree (AST), obtain the initial alignment of the input, propagate the alignment method through the dependencies between variables, obtain the alignment information, and rename the variables in the code; 2) Analyze the dependencies between statements and rearrange the statements in the program without changing the program semantics; 3) Static and dynamic multi-level alignment: Static alignment statements are obtained through heuristic algorithms. For results that cannot be aligned, variables in the execution trajectory are dynamically analyzed to obtain dynamic alignment statements. 4) Based on the alignment, mark the locations where variable values ​​are inconsistent as potential error locations and trace back the data flow to analyze the root causes of the differences. Combine the context information and data dependencies to conduct a credibility assessment of the potential error locations and generate an error location report that includes the error location, variable value differences, and data dependency links.

3. The code error feedback method combining static analysis and dynamic analysis according to claim 2, characterized in that, In step 1), the source code is first parsed into an abstract syntax tree (AST) to obtain the locations of variable declarations and usages, and a dependency graph between variables is constructed. Then, the variables that can be initially aligned are obtained by inputting the locations. Next, the scope of alignable variables is gradually propagated through the dependency graph. Finally, the variables are renamed according to the alignment relationship.

4. The code error feedback method combining static analysis and dynamic analysis according to claim 2, characterized in that, In step 2), the dependencies between statements are first analyzed, and data flow analysis is performed on the statements within the basic block to identify the dependencies between statements. Then, statements without dependencies are standardized and sorted using a unified sorting rule (such as alphabetical order of variable names) to ensure that different versions of the code have a consistent statement order after reordering, thus maintaining the final order under dependency constraints.

5. The code error feedback method combining static analysis and dynamic analysis according to claim 2, characterized in that, In step 3), the two versions of the code are first aligned using a heuristic method. The edit distance algorithm is used to calculate the editing method with the least cost, resulting in a static alignment block and a difference list. Then, the source code is compiled and executed on the failed test cases. The Python script is used to execute gdb, and the execution trajectory of the program on the failed test cases is collected. The statements corresponding to the replacement operations in the difference list are found, and the values ​​of variables in the execution trajectories of these statements are analyzed to evaluate their semantic similarity. Finally, the alignment relationship between the statements is determined.

6. The code error feedback method combining static analysis and dynamic analysis according to claim 2, characterized in that, In step 4), firstly, for each pair of alignment statements, the statements where the state or value of the variable differs are marked as potential error locations; then, data flow backtracking is performed on the potential error locations, tracing from the current variable state to the last correct use of the variable, analyzing the root cause of the difference, and recording the path; then, the credibility of each potential error location is evaluated in conjunction with the program's context information; finally, the error location analysis results are summarized, and an error feedback report containing detailed information such as the error location and propagation path is given.

Images