Code difference division technology based on data flow analysis

Abstract

The invention relates to a code difference division technology based on data flow analysis, which aims at Java language, designs a static analysis framework taking an abstract syntax tree as granularity, extracts data dependence among syntax tree nodes by using a data flow analysis algorithm, and selects a code difference block in Git as a basic unit to divide the code difference block into the Java language. The relation between syntax tree nodes is improved to be between code snippets, and finally, code difference snippets are divided through a design drawing algorithm to form different code change clusters. According to the method, developers can be assisted in understanding code differences, and the time cost consumed in the code review process is reduced.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer, mainly aiming at the technical field of software engineering. The present application proposes a code difference division technology based on data flow analysis, which converts the relevance between nodes to the code difference fragments by using data flow analysis on abstract syntax tree, and groups different types of difference fragments, thereby improving the efficiency of developers in understanding code differences. BACKGROUND

[0002] With the continuous expansion of software size and the improvement of complexity, the version control system Git has become one of the indispensable tools in the software development process. Git not only can maintain the history of code submission, but also supports parallel development, branch management and code merging, etc., which greatly improves the efficiency of collaborative development. Code review as an important part of parallel development is a major means to improve code quality and stability. By analyzing the submitted code changes, potential problems can be found in time to avoid the continuous pollution of error code to the downstream.

[0003] However, the existing code review method mostly depends on the manual operation of the developer, using Git to compare the difference fragments before and after the code submission, analyzing the rationality of the code changes, and judging whether the modification meets the expectation. With the gradual expansion of the size of the code base, a submission often involves the modification of multiple modules and files, and the cost of review will quickly rise. Therefore, how to help developers quickly understand the code difference has gradually become a research and practice hotspot in the field of software engineering.

[0004] Git diff is a command in Git to calculate the difference of text, which uses Histogram difference algorithm to construct a frequency histogram of text lines, and preferentially matches the lines with lower frequency, which can have more accurate code difference presentation for files containing a large number of repeated lines, code block movement or code rearrangement.

[0005] To solve the problem of coarse text line granularity, the existing code difference analysis is mostly based on abstract syntax tree expansion. Abstract syntax tree AST (Abstract Syntax Tree) is an intermediate tree structure used by compilers and interpreters when analyzing and processing code, which represents the syntax structure of the source code and organizes the various parts of the code (such as classes, methods and expression statements, etc.) into a tree form according to their syntax relationship. After converting the source code before and after Git submission into abstract syntax tree, the matching relationship between tree nodes can be calculated to obtain the change information of the syntax tree.

[0006] In general, matching algorithms solve for the longest common substructure between two data structures. For example, the well-known longest common subsequence (LCS) can represent the degree of matching between two text strings. However, solving for the longest common substructure in a tree or most graph-based data structures is very complex. For example, the longest common embedding subtree and the longest common subgraph are NP-hard problems, requiring constraints and heuristics to find approximate solutions in polynomial time complexity.

[0007] Currently, many code difference analysis algorithms (such as GumTree) use a combination of top-down and bottom-up matching methods to find approximate solutions for the maximum common substructure (GCS). These methods make assumptions based on the syntax rules of programming languages ​​and can quickly find the GCS of two code samples in a graphical representation of the code. Simply put, top-down matching calculates the overall framework of the syntax tree structure. Since the two code samples to be matched usually have some similarity, such as possibly having the same class names or function names, these elements are matched first. Bottom-up matching, on the other hand, after most classes and functions have been matched, tries to match as many remaining program elements as possible. The combination of these two matching strategies, along with some heuristic rules, allows existing algorithms to achieve O(n log n) performance. 2 Solving for the maximum matching with a time complexity of 1 / 2. Tsantalis et al., based on GumTree's tree difference algorithm, customized the change pattern of code refactoring at the syntax tree node level, and implemented the more accurate code refactoring detection framework RMiner.

[0008] When faced with large-scale code differences, Git or syntax tree-based diffing algorithms can only reveal relatively fragmented difference information. While refactoring detection can reconstruct refactoring operations in the code, it still relies on tree-based diffing algorithms for other types of code changes.

[0009] Therefore, the main objective of this invention is to select an appropriate granularity to combine code change fragments in a more intuitive way to help developers understand code differences. Summary of the Invention

[0010] This invention aims to solve several problems that exist in the process of human interpretation of code differences:

[0011] 1. High time cost: As the complexity of projects increases and the size of the codebase expands, a single commit often involves changes to multiple modules and files. Developers will spend a lot of time manually comparing and analyzing changes.

[0012] 2. Difficulty in discovering correlations: When a commit involves multiple modules or files, the changes between the files often have varying degrees of correlation. For example, different types of code refactoring will involve different ranges of code changes, and it may be difficult to fully discover them by relying solely on manual analysis.

[0013] 3. Unstable review quality: The diversity of code writing and the complex relationships between files may cause developers to overlook hidden errors in code submissions, making it difficult to maintain stable code review quality.

[0014] The technical solution of this invention is as follows: For the Java language, a static analysis framework with an abstract syntax tree as the granularity is designed, and a data flow analysis algorithm is used to extract the data dependencies between syntax tree nodes; at the same time, code difference blocks in Git are selected as the basic unit to elevate the relationship between syntax tree nodes to the relationship between code fragments; finally, a graph algorithm is designed to divide the code difference fragments, thereby helping developers understand code differences.

[0015] This invention includes the following specific steps:

[0016] 1. Parse the code repository to obtain the code metadata information before and after Git commits;

[0017] 2. Parse the text content into an abstract syntax tree, analyze the syntax tree nodes that intersect with the difference segments, and design corresponding hierarchical parsing and data flow analysis algorithms based on structural and data dependency relationships to clarify the dependency relationships between syntax tree nodes;

[0018] 3. Select the difference fragments in Git as the basic unit of partitioning, convert the dependencies in the previous step into a graph of relationships between the difference fragments, and design a graph partitioning algorithm to process the difference fragments with strong correlations, so as to obtain the final partitioning result set.

[0019] In step 1, the Git repository is parsed to extract metadata information before and after code commits.

[0020] Since subsequent analysis steps rely on the existence of corresponding physical files locally, the file tree information of the old and new commits needs to be cloned to the local machine. The parsing process involves converting binary data in the Git repository and calculating file mapping relationships to determine changes such as file additions, deletions, modifications, and renamings. After parsing, metadata for code difference fragments is generated to record the textual differences between the two sets of code, including file paths and line position information of the fragments. All parsed data needs to be globally cached and managed to avoid redundant calculations.

[0021] In step 2, the text content is parsed into an abstract syntax tree, position information is filtered, and corresponding hierarchical parsing and data flow analysis algorithms are designed based on structural and data dependencies to clarify the dependencies between syntax tree nodes.

[0022] The filtering rule for location information is as follows: access the syntax tree in a preorder traversal manner, determine whether the source code location corresponding to the currently accessed node intersects with the segment where the code difference occurs, and if there is no intersection, the complete subtree corresponding to this node can be excluded.

[0023] Structural dependency is essentially the hierarchical relationship between code elements in a program structure. This invention mainly focuses on four types of structural relationships:

[0024] 1. The referencing relationship between the Import statement and the imported information;

[0025] 2. Inheritance relationship between parent and child classes;

[0026] 3. The implementation relationship between interfaces and classes;

[0027] 4. Overriding relationship of methods in a subclass to a method in a parent class.

[0028] In this invention, data dependency is based on type analysis that focuses on the definition-use relationship between variables and methods. Specifically, after constructing a control flow graph at the syntax tree level, different types of expressions in the tree are analyzed to calculate the actual types that might correspond to as the control flow propagates. The control flow graph, based on statements and nodes in the tree, is constructed in reverse, and seven edge types are designed to represent the program execution path: sequential execution, if condition fulfillment, switch condition fulfillment, and exception handling and throwing.

[0029] Data flow analysis focuses on all Def-Use relations and the reference types that expressions may generate. It uses iterative forward analysis to propagate and update along the control flow graph until the algorithm converges.

[0030] In step 3, Git difference fragments are selected as the basic unit of partitioning. The dependencies between syntax tree nodes are converted into a relationship graph between difference fragments. A graph partitioning algorithm is designed to handle the associations in the directed graph and generate the final grouping results.

[0031] Before the partitioning algorithm begins, the original dependency graph needs to be expanded, the reachability and distance between nodes need to be recalculated, and the detection of refactoring, formatting and annotations needs to be introduced to transform the original graph information into code difference fragments.

[0032] In this invention, the partitioning algorithm focuses on the connectivity between points, namely strong connectivity and weak connectivity. The algorithm first uses the Targan algorithm to find all strongly connected components, each corresponding to a set of more closely related difference segments. Then, it uses a disjoint-set data structure to merge the remaining directed edges to form multiple sets of weakly connected components. Finally, all connected components together constitute the partitioning result of the code differences.

[0033] The present invention, by adopting the above technical solution, has the following four advantages:

[0034] 1. High execution efficiency: This invention only analyzes files that differ before and after code submission, and all designed algorithms have low time complexity.

[0035] 2. High accuracy: Based on the abstract syntax tree, this invention designs a static analysis algorithm to extract the structure and data dependencies in the code, and uses refactoring tools to supplement additional difference information, which can effectively handle code changes in different patterns.

[0036] 3. High interpretability: This invention belongs to the field of code analysis in software engineering, and the static analysis and graph partitioning algorithms used have complete theoretical support and optimization.

[0037] 4. High scalability: Although this invention is designed for the analysis of the Java language, it does not involve analysis steps specific to the Java language, making it easy to extend to other languages ​​such as C++ and C#. Attached Figure Description

[0038] Figure 1 A flowchart of a code difference partitioning technique based on data flow analysis

[0039] Figure 2 Paradigm diagram of data flow analysis algorithms Detailed Implementation Plan

[0040] This invention proposes a code difference partitioning technique based on data flow analysis. For the Java language, using the abstract syntax tree (AST) as the analysis granularity, a data flow analysis algorithm is designed to extract the structure and data dependencies between AST nodes. Simultaneously, using Git's Diff Hunk as the basic granularity for code partitioning, a graph algorithm is designed to partition code differences by elevating the dependencies between AST nodes to the levels between code segments, thereby helping developers understand code differences.

[0041] The overall process of this invention is as follows: Figure 1 As shown, the specific steps include:

[0042] Step 1: Parsing the Git repository.

[0043] Because this invention performs difference analysis on code information already stored in the Git repository, and parsing the syntax tree and calculating dependencies both require the corresponding physical files to exist on the local machine, it is necessary to clone the file tree information of the old and new commits in the code repository to the local machine. This invention supports two forms of difference analysis: one is to specify the code difference between the hash values ​​of the old and new commits; the other is the difference between the commit pointed to by the Git header pointer and the cached code in the staging area.

[0044] The entire analysis is mainly divided into three stages:

[0045] 1. Parse the commit file tree: Using JGit, for a specified analysis method, obtain the internal file tree information and filter out non-Java files.

[0046] 2. Data Writing: Based on the acquired file tree information, the file information from the first stage is cloned to a specified local directory, maintaining the file structure consistent with the original code data to facilitate subsequent program analysis. The cloned directory uses a combination of repository name, timestamp, and version number, in the form of / repo_name / timestamp / old(new)_stage, to distinguish file information from different source data.

[0047] 3. Calculate file mapping relationships: By calculating the mapping relationship between the file trees of the previous and current versions, the types of changes in different files can be determined, such as additions, deletions, modifications, and renamings. For all files that have changed, metadata DiffHunk representing the code differences needs to be generated. DiffHunk can be used to record the text differences between two sets of code. It consists of two basic blocks, Hunk, where Hunk is a triple consisting of the file path, the initial line, and the end line, representing a code segment under the specified file, in the form of [path](begin,end).

[0048] After the parsing process is complete, the Hunk data in each file and all the Diff Hunk data are globally cached and managed to avoid duplicate calculations.

[0049] Step 2: Establish dependencies between nodes.

[0050] After the first step, a set of files that have changed between versions is obtained. This set can be converted into a syntax tree forest using a syntax tree parsing tool. Within the changed file set, only the text of newly added and deleted files needs to be analyzed. For modified or renamed files, only program elements overlapping with code differences need to be considered. Therefore, the analysis scope can be further narrowed by filtering the syntax tree.

[0051] The specific approach is as follows: Perform a pre-order traversal of the syntax tree. If a node's position in the source code does not overlap with a code difference segment, then it and all its child nodes do not need to be analyzed. After preprocessing, all syntax tree nodes to be analyzed will be obtained.

[0052] Structural dependencies are constructed for all nodes to be analyzed. Structural dependencies refer to the hierarchical relationships of code elements in the program structure. This invention mainly focuses on the following four structural relationships:

[0053] 1. The referencing relationship between the Import statement and the imported information

[0054] In Java, the Import statement can be used to import visible classes, interfaces, and methods from other modules or files. There are two special forms of the Import statement: static import with the prefix Import static and full import with the suffix as an asterisk. The former indicates that static methods or variables specified inside a class are imported, while the latter indicates that all visible fields and methods in a package or class are imported.

[0055] Therefore, for each reference type to be analyzed in the code difference block, it is determined whether it is imported by the Import statement of the file in which it is located: if so, an edge is created from the Import statement to the original class definition of the reference type, and the type of the edge is IMPORT.

[0056] 2. Inheritance relationship between parent and child classes

[0057] In Java, classes can inherit from other classes, and interfaces can inherit from other interfaces. Therefore, when analyzing inheritance relationships, both can be analyzed simultaneously.

[0058] For classes or interfaces, the system first performs static parsing of the class definition based on the file tree structure obtained in the first step, checking if a corresponding parent class declaration exists in the source file. If it exists, an edge of type EXTEND is introduced; otherwise, no operation is performed. This is because this invention only focuses on the changed source code content and not on third-party dependencies introduced using Gradle or Maven build tools.

[0059] 3. Implementation relationship between interfaces and classes

[0060] The analysis of interface implementation relationships mainly focuses on the implementation relationship between classes and interfaces in Java. The type of the edge is defined as IMPLEMENT. Since the analysis method is similar to that of the previous inheritance relationship, the specific process will not be repeated.

[0061] 4. Overriding relationship of methods in a subclass to a method in a parent class

[0062] In Java, polymorphism can be achieved by subclasses overriding methods of their parent classes. The analysis of overriding relationships involves a two-phase check:

[0063] 1) Check the inheritance chain: Since the inheritance relationship of a class can be multi-level, search from bottom to top according to the class inheritance relationship. If the class being searched is not introduced from the outside, check the method declarations inside the class.

[0064] 2) Check the interfaces implemented by direct or indirect parent classes: If no corresponding method was found in the previous stage, it is necessary to search for it in the implemented interfaces. For each parent class implementing an interface in the class inheritance chain and the inheritance relationship between interfaces, search upwards to see if there is an interface method with the same method signature.

[0065] After the two-stage search described above, for each method in the subclass, if there is a parent class method in its source code, an edge will be constructed pointing from the subclass method to the parent class method, and the edge type will be OVERRIDE.

[0066] Since structural dependencies can only be applied to larger code structures, finer-grained data dependencies require the combined involvement of control flow and data flow. This invention uses type analysis as the primary analysis method. Type analysis targets Def-Use and method calls, utilizing normalization to define and propagate reference types in specific expressions. This method can be directly based on syntax tree expansion, ensuring the correspondence between nodes and source code while maintaining good analysis speed and acceptable precision loss.

[0067] This invention proposes a control flow construction algorithm based on an abstract syntax tree. The control flow graph consists of nodes and edges. A node typically corresponds to a basic block in the program, i.e., a sequence of instructions executed sequentially without branch exits. Edges represent the transition relationships in the program execution order, such as sequential control flow, branch jumps, and loop jumps, thereby expressing the control dependencies between instructions. The control flow graph contains unique entry and exit nodes, corresponding to the start and end positions of the program, respectively.

[0068] Unlike other types of intermediate representations (IR), syntax trees inherently possess tree-like structural characteristics, and the more complex the code structure, the larger the number of corresponding nodes. Therefore, unlike existing research, this invention uses statements and expressions together as the basic granularity of control flow nodes. A statement corresponds to a complete line of code and consists of multiple expressions, thus maintaining a balance between analysis granularity and precision. The design of control flow edges can be divided into the following seven types: sequential execution, if condition fulfillment, switch condition fulfillment, exception handling, and exception throwing.

[0069] Simultaneously, recursive construction algorithms need to be designed specifically for different types of syntax tree nodes, and the context information of the current node must be maintained at all times to ensure the correctness of predecessors and successors between nodes. The algorithm as a whole adopts a reverse construction approach. For If, For, While, Switch, and Try-Catch statements, after analyzing the sub-statements, the internal control flow graph is constructed according to its syntactic structure, and the entry node of the current statement is returned to the upper layer, thus obtaining the control flow graph based on the syntax tree.

[0070] In data flow analysis, since the entire process mainly focuses on the definition and usage relationships of variables (DefUse), and how the reference types corresponding to expressions should propagate along the program control flow graph, the algorithm is based on classic iterative forward analysis. Two sets are introduced to represent the data propagation process: IN and OUT, representing the sets of all possible reference types that data can point to after entering and exiting nodes, respectively. The two sets have the following conversion function during analysis:

[0071] 1.

[0072] 2.OUT[n]=(IN[n]\kill(n))∪ gen(n).

[0073] Because syntax tree structures are far more complex than IR (Input / Output Logic), data flow analysis requires analyzing subexpressions first before propagating them to their corresponding statements. A specific analysis paradigm is as follows: Figure 2 As shown. At the same time, all DefUse relationships are saved during data stream analysis.

[0074] Step 3: Divide the code into different segments.

[0075] The output in the second step is a dependency relationship based on an abstract syntax tree. However, during code reading, the granularity of the syntax tree varies, ranging from coarse-grained class and method declarations to fine-grained expression statements, making it impossible to highlight the code changes themselves with a uniform and intuitive granularity. Therefore, this invention uses Git's DiffHunk as the basic granularity for final division and presentation. This step can be divided into two stages: relationship granularity improvement and code difference segmentation.

[0076] 1. Improved relationship granularity

[0077] The goal of this phase is to elevate the structural and data dependencies between nodes to the level of code differences:

[0078] First, the relationships between nodes are directly calculated, in the form of directed edges like a→b→c. However, it's possible that b isn't located in any code-differenced segments, causing information loss between a and c. Therefore, it's necessary to recalculate the reachability relationships between point pairs and use the reciprocal of the distance between the two points to represent the strength of the relationship.

[0079] Secondly, if there is a directed edge between two syntax tree nodes, and both nodes are covered by different hunks, then connect the two hunks with edges in the same direction. After processing the old and new versions separately, we can obtain directed graphs G and G′ based on hunks. Since a hunk belongs to only one Diff hunk, integrating G and G′ yields the DiffHunk relationship graph.

[0080] Furthermore, in order to better express the code change patterns, this invention also detects code refactoring, commenting, and formatting, and supplements the obtained change information into the DiffHunk relationship diagram.

[0081] 2. Code Differences

[0082] The essence of code diff partitioning is to divide the improved granularity Diff Hunk relationship graph into different change clusters, and each cluster contains a set of interrelated diff fragments.

[0083] First, the partitioning algorithm identifies all strongly correlated change clusters, meaning all segments within a cluster are mutually reachable, thus transforming the problem into finding the number of strongly connected components in the relational graph. Second, after the search is complete, any remaining nodes that are connected are weakly connected and may contain isolated nodes with no incoming or outgoing edges. Third, by further processing the weakly connected components in the remaining graph using a disjoint-set data structure, all change clusters become the final partitioning result of the code difference segments.

[0084] Step 4: Verify the validity of the partitioning results.

[0085] To verify the validity of the partitioning results, the method needs to be evaluated in a real code repository. The specific method is as follows:

[0086] First, select influential and long-active code repositories from GitHub, manually integrate several unrelated consecutive commits in the commit history into a single code commit, and treat this commit and the original code as the old and new versions, respectively.

[0087] Secondly, this method is used to divide the code differences, determine whether the division results can match the original commit data, and calculate the accuracy of the results.

Claims

1. A code difference partitioning technique based on data flow analysis, characterized by: For the Java language, a static analysis framework with abstract syntax trees as the granularity is designed. Data flow analysis algorithms are used to extract data dependencies between syntax tree nodes, and code difference blocks in Git are selected as the basic unit to elevate the relationships between syntax trees to the relationships between code fragments. Finally, the code difference set is classified based on graph partitioning algorithms to help developers understand code differences.

2. The code difference partitioning technique based on data flow analysis described in claim 1, characterized in that... Includes the following steps: 1) Parse the code repository to obtain the code metadata information before and after Git commits; 2) Parse the text content into an abstract syntax tree, analyze the syntax tree nodes that intersect with the difference segments, and design corresponding hierarchical parsing and data flow analysis algorithms based on structural and data dependency relationships to clarify the dependency relationships between syntax tree nodes; 3) Select the difference fragments in Git as the basic unit of partitioning, convert the dependencies in step 2) into a graph of relationships between difference fragments, and design a graph partitioning algorithm to process the difference fragments with strong correlations, so as to obtain the final partitioning result set.

3. The code difference partitioning technique based on data flow analysis according to claim 2, characterized in that: In step 1), the Git repository is parsed to extract metadata information before and after code commits: Since subsequent analysis steps rely on the existence of corresponding physical files locally, the file tree information of the old and new commits needs to be cloned to the local machine. The parsing process involves the conversion of binary data in the Git repository and the calculation of file mapping relationships to determine the changes in file addition, deletion, modification, and renaming. After parsing, metadata of code difference fragments will be generated to record the text differences between the two code files, including file paths and line position information of the fragments. All data obtained after parsing needs to be globally cached and managed to avoid duplicate calculations.

4. The code difference partitioning technique based on data flow analysis according to claim 2, characterized in that: In step 2), the text content is parsed into an abstract syntax tree, positional information is filtered, and based on structural and data dependencies, corresponding hierarchical parsing and data flow analysis algorithms are designed to clarify the dependencies between syntax tree nodes. The filtering rule for location information is as follows: access the syntax tree in a preorder traversal manner, determine whether the source code location corresponding to the currently accessed node intersects with the segment where the code difference occurs, and if there is no intersection, the complete subtree corresponding to this node can be excluded. Structural relationships include: the reference relationship between the Import statement and the imported information, the inheritance relationship between the parent and child classes, the implementation relationship between the interface and the class, and the overriding relationship of the parent class method by the child class; The control flow graph is based on statements and nodes in a tree and is constructed in reverse. It is designed with 7 edge types to represent the program execution path, namely: sequential execution, if condition is true or false, switch condition is satisfied or not, and exception catching and throwing. Data flow analysis focuses on all DefUse relations and the reference types that expressions may generate. It uses iterative forward analysis to propagate and update along the control flow graph until the algorithm converges.

5. The code difference partitioning technique based on data flow analysis as described in claim 2, characterized in that: In step 3), Git difference fragments are selected as the basic unit for partitioning. The dependencies between syntax tree nodes are transformed into a relationship graph between difference fragments. A graph partitioning algorithm is designed to handle the associations in the directed graph, generating the final grouping results. Before the partitioning algorithm begins, the original dependency graph needs to be expanded, the reachability and distance between nodes need to be recalculated, and the detection of refactoring, formatting and annotations needs to be introduced to transform the original graph information into code difference fragments; The partitioning algorithm focuses on the connectivity between points, namely strong connectivity and weak connectivity. The algorithm first uses the Targan algorithm to find all the strongly connected components, which correspond to a set of more closely related difference segments. Then, it uses a disjoint-set data structure to merge the remaining directed edges to form multiple sets of weakly connected components. Finally, all the connected components together constitute the partitioning result of the code differences.

Images