Code duplication checking method and system based on identification coding

Abstract

This invention discloses a code deduplication method and system based on identifier encoding, comprising the following steps: Step 1: Obtain the code to be deduplicated and the code to be compared, and parse them to obtain an abstract syntax tree, symbol table, and scope tree; Step 2: Parse the binding relationships and generate an identifier binding set and an identifier encoding vector; Step 3: Perform binding constraint renaming intervention and generate a renaming intervention sample set; Step 4: Perform semantic preservation transformation, generate a transformation sample set, and construct a track convergence kernel; Step 5: Apply Hilbert-Schmidt information bottleneck constraints to train a representation learning model and generate code representation vectors; Step 6: Divide the code into basic blocks and update the segment representation vectors; Step 7: Calculate global similarity and segment-level similarity, and output the judgment result and segment location result. This invention reduces interference from renaming and equivalent rewriting, and improves the stability and accuracy of deduplication detection.

Description

Technical Field

[0001] This invention relates to the field of software engineering technology, and in particular to a code deduplication method and system based on identifier encoding. Background Technology

[0002] Code plagiarism detection falls under the category of program analysis and similarity detection techniques in software engineering. Existing solutions typically target applications such as source code clone detection, plagiarism detection, and reusable fragment retrieval. These solutions involve parsing the source code to obtain an abstract syntax tree (AST), further extracting token sequences, syntax subtree features, and control flow structure features, and then using fingerprint comparison, tree edit distance, feature hashing, and structural similarity metrics to determine similarity. As engineering scales increase, deep learning-based code representation methods are becoming more prevalent. A common approach is to input the AST or its traversal sequence into a neural network model to obtain a vector representation, then use cosine similarity or metric learning to determine plagiarism, and, when necessary, combine function-level or code block-level segmentation to locate similar fragments.

[0003] Existing technologies still have significant shortcomings in resisting source code rewriting. The first type of deficiency stems from the variability at the identifier level. Renaming variable names, function names, and member names does not change the program's semantics, but it significantly alters token distribution and the local morphology of the abstract syntax tree, causing similarity drift in models that rely primarily on lexical meaning or local structure. Some solutions only reduce the impact through simple normalization or lexical truncation, failing to characterize the binding relationships between scope, definition points, and reference points, easily confusing "same name, different meaning" with "different name, same meaning." The second type of deficiency arises from the widespread existence of semantically preserving transformations. Transformations such as comment and whitespace rewriting, equivalent expression substitution, equivalent condition and loop rewriting, and statement swapping that satisfies dependency constraints generate a large number of superficially different samples. Existing methods mostly rely on limited data augmentation or rule filtering, lacking a unified measurement mechanism for the stability of transformation trajectories, resulting in insufficient generalization ability across projects, styles, and obfuscation intensities.

[0004] Existing deep representation learning schemes suffer from a mismatch between training objectives and plagiarism detection requirements. Common training objectives tend to maximize discriminative power, and models easily exploit "shortcut features" such as identifier literals, constant patterns, and local templates to achieve lower training errors. This results in sensitivity to renaming interventions, instability in maintaining semantic integrity, and the occurrence of high similarity but semantically uncorrelated segments when locating similar code. Even with segmented comparisons to illustrate similar positions, segment-level vectors are often influenced by segment length and local patterns, lacking constraints on inter-segment dependencies. This makes it difficult to maintain consistency in segment representations while ensuring consistent control and data flow, leading to both false positives and false negatives at the segment level. Furthermore, existing schemes lack controllable mechanisms to suppress the "dependency between representation and identifier information," making it difficult to systematically reduce the impact of identifier renaming while maintaining semantic discriminative capabilities.

[0005] Therefore, how to provide a code deduplication method and system based on identifier encoding is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a code deduplication method and system based on identifier encoding. This invention adopts a combination of static program analysis and representation learning to construct an abstract syntax tree, symbol table, and scope tree for the source code, explicitly modeling identifier definition points, reference points, and binding relationships. Through renaming intervention, semantic preservation transformation, and orbit convergence kernel constraint training process, the invention introduces the Hilbert-Schmidt information bottleneck to suppress the dependence of code representation on identifier literals, thereby achieving a stable representation of the semantic structure of the code. This method has the advantages of strong semantic consistency, high anti-confusion ability, and accurate location of similar segments.

[0007] A code deduplication method based on identifier encoding according to an embodiment of the present invention includes the following steps: Step 1: Obtain the code to be checked for plagiarism and the code to be compared, and parse them to obtain the abstract syntax tree, symbol table, and scope tree; Step 2: Based on the symbol table and scope tree, resolve the identifier definition point, identifier reference point, and binding relationship to generate the identifier binding set, and generate the identifier encoding vector based on the identifier binding set; Step 3: Generate a binding constraint renaming permutation set based on the identifier binding set. Perform renaming intervention on the code to be checked and the code to be compared according to the binding constraint renaming permutation set to obtain the renaming intervention sample set; Step 4: Set up a semantically preserved transformation set, perform the semantically preserved transformation set on the code to be checked and the code to be compared, obtain the transformation sample set, and construct the track convergence kernel based on the transformation sample set; Step 5: Train the representation learning model using the orbit convergence kernel to obtain the code representation vector. During the training process, apply the Hilbert-Schmidt information bottleneck constraint to limit the dependency between the code representation vector and the identifier encoding vector of the renaming intervention sample set. Step 6: Divide the basic blocks based on the abstract syntax tree, calculate the segment representation vector of the basic blocks, and update the segment representation vector based on the dependency path budget constraint; Step 7: Calculate global similarity based on code representation vectors, calculate segment-level similarity based on segment representation vectors, and generate plagiarism detection results and similar segment location results.

[0008] Optionally, step one specifically includes: Obtain the code to be checked and the code to be compared, determine the programming language identifier, and perform character encoding unification, line break normalization, and indentation normalization on the source code text to obtain the standardized source code text; Based on programming language identifiers, syntax rules are selected, and lexical and syntactic analysis are performed on the normalized source code text to generate an abstract syntax tree. Identifier declaration nodes are identified based on the abstract syntax tree, and symbol table entries are generated. The symbol table entries contain the identifier text, declaration category, declaration location, and visibility boundary. The scope tree is generated from the block structure nodes based on the abstract syntax tree. The scope tree contains the parent-child scope relationship, the start and end positions of the scope, and the set of entries in the scope association symbol table.

[0009] Optionally, step two specifically involves: The symbol table entries are assigned to different categories based on the start and end positions of the scope tree, and the association between the scope tree nodes and the set of symbol table entries is established. Traverse the abstract syntax tree, locate the node where the identifier appears and extract the source code interval position. Determine whether the node where the identifier appears belongs to the identifier definition point or the identifier reference point according to the node type and parent node type of the abstract syntax tree. For an identifier reference point, the corresponding scope tree node is determined based on the source code range position. The association relationship is retrieved level by level along the parent-child scope relationship of the scope tree and the symbol table entry is matched. The identifier text that is consistent with the identifier reference point is determined and the declaration position is recorded. A binding relationship record is generated. The binding relationship record includes the source code range position of the identifier reference point, the symbol table entry declaration category, the symbol table entry declaration position, and the scope tree node identifier. The binding relationship records are aggregated to generate an identifier binding set, and the identifier text, declaration category, scope depth, and reference number are mapped to fixed-length numerical codes and concatenated to generate an identifier encoding vector.

[0010] Optionally, step three specifically includes: Extract the symbol table entry set based on the identifier binding set, and divide the symbol table entry set according to the scope tree node identifier to obtain the symbol table entry set corresponding to the scope tree node identifier; For each scope tree node, extract the identifier text set corresponding to the symbol table entry set, generate the renaming target dictionary, and the renaming target dictionary contains the replacement identifier text list and satisfies that the elements of the replacement identifier text list are all different; For each scope tree node identifier, construct a renaming mapping corresponding to the set of symbol table entries. The renaming mapping maps the symbol table entry declaration position to the replacement identifier text. Collect the renaming mappings corresponding to each scope tree node identifier to generate a set of binding constraint renaming permutations. Based on the binding constraint renaming replacement set, the identifier definition points and identifier reference points of the abstract syntax tree are replaced. The replacement uses the symbol table entry declaration position of the binding relationship record corresponding to the identifier reference point to retrieve the renaming mapping and replace the identifier text, generating a renaming intervention sample set.

[0011] Optionally, step four specifically includes: Define a semantically preserved transformation set, which includes comment insertion and deletion transformation, whitespace character insertion and deletion transformation, bracket insertion and deletion transformation, equivalent expression substitution transformation, conditional structure equivalent rewriting transformation, loop structure equivalent rewriting transformation, and statement exchange transformation. Statement exchange and transformation satisfy data dependency constraints and control dependency constraints. Data dependency constraints extract the statement read set and statement write set through the abstract syntax tree and determine post-write read dependency, read-after-write dependency, and post-write dependency. Control dependency constraints locate branch nodes through the abstract syntax tree and determine statement reachability relationships. The semantic-preserving transformation set is generated by performing a semantic-preserving transformation on the code to be checked and the code to be compared. After performing a semantic-preserving transformation, an abstract syntax tree is generated and an identifier binding set is generated. When the abstract syntax tree and the identifier binding set are successfully generated, the transformation result is written to the transformation sample set. The basic kernel value of the sample pair is calculated for the sample to be deduplicated and the comparison sample in the transformed sample set. The orbital average is then performed on the basic kernel value of the sample pair to obtain the orbital convergence kernel.

[0012] Optionally, step five specifically includes: Select the transformation samples to be checked and the comparison transformation samples from the transformation sample set, pair them according to the sample index to form a sample pair set, calculate the track convergence core value for the sample pair set and generate a core value table; Abstract syntax tree encoding is performed on the deduplication transformation samples and comparison transformation samples in the sample pair set. The abstract syntax tree encoding includes node type encoding, parent-child relationship encoding, and node depth encoding. The abstract syntax tree encoding is mapped by the representation learning model to generate a set of code representation vectors. The representation similarity of the code representation vector set is calculated for the sample pair set, and the alignment loss is generated based on the difference between the representation similarity and the orbit convergence kernel value in the kernel value table; Rename intervention samples are selected from the rename intervention sample set. The corresponding identifier encoding vectors of the rename intervention samples are extracted and a set of identifier encoding vectors is formed. A representation kernel matrix is ​​constructed based on the code representation vector set. An identifier kernel matrix is ​​constructed based on the identifier encoding vector set. The representation kernel matrix and the identifier kernel matrix are centered and the Hilbert-Schmidt dependency is calculated to generate information bottleneck constraint terms. Alignment loss and information bottleneck constraint terms are combined into an optimization objective by weights, and the parameters of the representation learning model are updated iteratively to generate the code representation vector of the code to be checked for duplication and the code representation vector of the code to be compared.

[0013] Optionally, step six specifically includes: Locate the function block node in the abstract syntax tree, scan the statement nodes of the function block node in statement order, set the basic block boundary when encountering branch statement node, loop statement node, and jump statement node, form a basic block segment by combining the statement nodes between adjacent basic block boundaries and assign segment identifiers to the basic block segment. Extract statement node subtrees from basic block segments, perform abstract syntax tree encoding to generate basic block encoding vectors, and use a representation learning model to map the basic block encoding vectors to generate segment representation vectors. The segment identifier where the identifier definition point is located is determined by the source code interval location of the identifier definition point, and the segment identifier where the identifier reference point is located is determined by the source code interval location of the identifier reference point. Dependency edges are generated for the binding relationship records in the identifier binding set. The dependency edges contain the definition segment identifier and the reference segment identifier. The segment identifier sequence from the definition segment identifier to the reference segment identifier is calculated in the statement order of the function body block node. The segment identifier sequence is used as a dependency path and is aggregated to form a dependency path set. Set a dependency path budget value, sort the dependency path set in ascending order of dependency path length, and extract the dependency path corresponding to the budget value as the budget dependency path. Extract the segment representation vector corresponding to the adjacent segment identifiers in the segment identifier sequence based on the budget dependency path and perform weighted aggregation to obtain the updated segment representation vector.

[0014] Optionally, step seven specifically includes: Normalization is performed on the code representation vectors of the code to be checked and the code representation vectors of the comparison code. The cosine similarity of the code representation vectors is calculated and used as the global similarity. Construct a set of segment identifier pairs for the segment identifier set of the code to be checked and the segment identifier set of the code to be compared. Calculate the cosine similarity of the segment representation vectors corresponding to the segment identifier pairs and form a segment similarity matrix. The maximum value of the segment similarity matrix is ​​taken as the segment-level similarity. The global similarity is compared with the global threshold to generate a global judgment flag, the segment-level similarity is compared with the segment threshold to generate a segment-level judgment flag, and the plagiarism detection result is generated based on the global judgment flag and the segment-level judgment flag. A threshold filtering is performed on the segment similarity matrix to obtain a set of similar segment identifier pairs. The similar segment location result is determined based on the set of similar segment identifier pairs. The similar segment location result includes the identifier of the segment to be checked for plagiarism, the identifier of the comparison segment, the source code interval position of the segment to be checked for plagiarism, and the source code interval position of the comparison segment.

[0015] A code deduplication system based on identifier encoding according to an embodiment of the present invention includes: The parsing module is used to obtain the code to be checked for plagiarism and the code to be compared, and to parse it to obtain the abstract syntax tree, symbol table, and scope tree; The binding parsing module is used to parse identifier definition points, identifier reference points, and binding relationships, generate identifier binding sets, and generate identifier encoding vectors. The renaming intervention module is used to generate a set of binding constraint renaming replacements, perform renaming interventions on the code to be checked and the code to be compared, and generate a renaming intervention sample set; The transformation kernel module is used to set the semantically preserved transformation set, perform the semantically preserved transformation set on the code to be checked and the code to be compared, generate the transformation sample set, and build the track convergence kernel; The bottleneck training module is used to train the representation learning model using the orbit convergence kernel and apply the Hilbert-Schmidt information bottleneck constraint to generate code representation vectors. The segmented budget module is used to divide basic blocks into segments based on the abstract syntax tree, generate segment representation vectors, and update segment representation vectors based on dependency path budget constraints. The determination and location module is used to calculate global similarity based on code representation vectors, calculate segment-level similarity based on segment representation vectors, and generate plagiarism determination results and similar segment location results.

[0016] The beneficial effects of this invention are: This invention introduces a collaborative modeling approach using abstract syntax trees, symbol tables, and scope trees during the code parsing phase. This approach explicitly characterizes the definition points, reference points, and binding relationships of identifiers. Based on this, it constructs identifier binding sets and identifier encoding vectors. This allows code representation to no longer depend on the surface naming form of identifiers, but rather to be built upon the scope structure and semantic binding relationships. By renaming binding constraints and semantically preserving transformations, multi-perspective samples are generated. Furthermore, a convergent kernel is used to uniformly measure similarity in the equivalent transformation space. This ensures that the code maintains a stable and consistent representation even after undergoing variable renaming, expression rewriting, and equivalent transformations of control structures. Consequently, this significantly improves the reliability and consistency of plagiarism detection results in complex rewriting scenarios.

[0017] Meanwhile, this invention introduces the Hilbert-Schmidt information bottleneck constraint during the representation learning process, systematically limiting the dependence of the code representation vector on the identifier encoding vector, avoiding the model from using identifier literals or local templates to form shortcut features, reducing the risk of false positives while maintaining semantic discrimination ability. Combined with basic block segment partitioning and dependency path budgeting mechanism, the structural constraint update of the segment-level representation is performed to keep the segment-level similarity consistent with the relationship between control flow and data flow, and finally realizes the collaborative output of global similarity judgment and similar segment location, improving the accuracy of deduplication judgment and the interpretability of location results. Attached Figure Description

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a code deduplication method based on identifier encoding proposed in this invention; Figure 2 This is a schematic diagram illustrating the generation of an identifier binding set in a code deduplication method based on identifier encoding proposed in this invention; Figure 3 This is a schematic diagram of the Hilbert-Schmidt information bottleneck constraint training for a code deduplication method based on identifier encoding proposed in this invention. Detailed Implementation

[0019] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0020] refer to Figure 1-3 A code deduplication method based on identifier encoding includes the following steps: Step 1: Obtain the code to be checked for plagiarism and the code to be compared, and parse them to obtain the abstract syntax tree, symbol table, and scope tree; Step 2: Based on the symbol table and scope tree, resolve the identifier definition point, identifier reference point, and binding relationship to generate the identifier binding set, and generate the identifier encoding vector based on the identifier binding set; Step 3: Generate a binding constraint renaming permutation set based on the identifier binding set. Perform renaming intervention on the code to be checked and the code to be compared according to the binding constraint renaming permutation set to obtain the renaming intervention sample set; Step 4: Set up a semantically preserved transformation set, perform the semantically preserved transformation set on the code to be checked and the code to be compared, obtain the transformation sample set, and construct the track convergence kernel based on the transformation sample set; Step 5: Train the representation learning model using the orbit convergence kernel to obtain the code representation vector. During the training process, apply the Hilbert-Schmidt information bottleneck constraint to limit the dependency between the code representation vector and the identifier encoding vector of the renaming intervention sample set. Step 6: Divide the basic blocks based on the abstract syntax tree, calculate the segment representation vector of the basic blocks, and update the segment representation vector based on the dependency path budget constraint; Step 7: Calculate global similarity based on code representation vectors, calculate segment-level similarity based on segment representation vectors, and generate plagiarism detection results and similar segment location results.

[0021] In this embodiment, step one specifically includes: Obtain the code to be checked and the code to be compared, determine the programming language identifier, and perform character encoding unification, line break normalization, and indentation normalization on the source code text. Character encoding unification includes converting the source code text into a unified character set and removing invisible control characters. Line break normalization includes converting carriage returns and line feeds into line breaks. Indentation normalization includes replacing tabs with a fixed number of spaces and cleaning up trailing whitespace characters. Generate normalized source code text and maintain the correspondence between character offsets and row and column positions for declaration position determination. Based on the programming language identifier, the syntax rules are selected, and lexical analysis and syntax analysis are performed on the normalized source code text. Lexical analysis includes dividing the normalized source code text into lexical units according to the syntax rules and recording the start and end character offsets and row and column positions of each lexical unit. Syntax analysis includes reducing the lexical units according to the syntax rules and constructing abstract syntax tree nodes. The abstract syntax tree nodes include node type identifiers, child node references, and source code interval positions. Identifier declaration nodes are identified based on the abstract syntax tree and symbol table entries are generated. Identifier declaration nodes include variable declaration nodes, function declaration nodes, type declaration nodes, formal parameter declaration nodes, and member declaration nodes. Symbol table entries consist of identifier text, declaration category, declaration position, and visibility boundary. The declaration position is represented by character offset and row / column position, and the visibility boundary is represented by scope tree node identifier and scope start and end positions. The scope tree is generated based on the block structure nodes of the abstract syntax tree. The block structure nodes include file-level block nodes, namespace block nodes, type block nodes, function body block nodes, and compound statement block nodes. The scope tree includes parent-child scope relationships, scope start and end positions, and a set of scope-associated symbol table entries. The scope-associated symbol table entries are assigned to a specific category by comparing the declaration position of the symbol table entries with the scope start and end positions, and an association record is established.

[0022] In this embodiment, step two specifically includes: The symbol table entries are assigned according to the start and end positions of the scope tree. The assignment includes determining the interval inclusion between the declaration position of the symbol table entry and the start and end positions of the scope. If the interval inclusion determination satisfies that the declaration position falls within the source code interval defined by the start and end positions of the scope, the association relationship between the scope tree nodes and the set of symbol table entries is generated and the scope depth is recorded. Traverse the abstract syntax tree, locate the nodes where identifiers appear and extract the source code interval positions. Locating the nodes where identifiers appear includes selecting nodes in the abstract syntax tree whose node type is identifier and recording the node's source code interval position. Determining the type of the nodes where identifiers appear includes distinguishing between the declaration context and the usage context based on the node type and the parent node type in the abstract syntax tree. Mark the identifier nodes in the declaration context as identifier definition points and the identifier nodes in the usage context as identifier reference points. For identifier reference points, the corresponding scope tree node is determined based on the source code interval position of the identifier reference point. Determining the corresponding scope tree node includes using the source code interval position and the start and end positions of the scope to determine the interval inclusion and selecting the scope tree node with the smallest interval length. The association relationship is retrieved level by level along the parent-child scope relationship of the scope tree and the symbol table entries are matched. This includes filtering symbol table entries with the same identifier text in the symbol table entry set of the corresponding scope tree node and iteratively filtering along the parent scope direction if no match is found. When a symbol table entry is successfully matched, the declaration category and declaration position are extracted and a binding relationship record is generated. The binding relationship record includes the source code interval position of the identifier reference point, the identifier text, the symbol table entry declaration category, the symbol table entry declaration position, and the scope tree node identifier. The binding relationship records are aggregated to generate an identifier binding set. The identifier binding set consists of binding relationship records and is deduplicated according to the identifier text and declaration position. The identifier encoding vector generation includes mapping the identifier text to lexical encoding, mapping the declaration category to category encoding, mapping the scope depth to depth encoding, mapping the order of occurrence of identifier reference points in the same declaration position to reference number encoding, and concatenating the lexical encoding, category encoding, depth encoding, and reference number encoding to generate a fixed-length identifier encoding vector.

[0023] In this embodiment, step three specifically includes: The symbol table entry set is extracted based on the identifier binding set. The symbol table entry set is obtained by deduplicating the symbol table entries referenced by the binding relationship records. The symbol table entry set is divided according to the scope tree node identifier, which includes determining the scope tree node identifier using the visibility boundary of the symbol table entry and assigning the symbol table entry to the symbol table entry set corresponding to the corresponding scope tree node identifier. Extract the identifier text set from the symbol table entry set corresponding to the scope tree node identifier. The identifier text set is obtained by deduplicating the identifier text contained in the symbol table entry. The renaming target dictionary generation includes constructing a replacement identifier text list and satisfying that the elements of the replacement identifier text list are all different and are also different from the elements of the identifier text set. A renaming mapping is constructed for the set of symbol table entries corresponding to the scope tree node identifier. The renaming mapping maps the symbol table entry declaration position to the replacement identifier text. The construction of the renaming mapping includes sorting the symbol table entries by declaration position and assigning replacement identifier text to the symbol table entries according to the sort position. The renaming mappings corresponding to each scope tree node identifier are collected to generate a set of binding constraint renaming permutations. Based on the binding constraint renaming replacement set, the identifier definition points and identifier reference points of the abstract syntax tree are replaced. The replacement includes using the declaration position of the identifier definition point as the key to retrieve the renaming map and replace the identifier text, and using the declaration position of the symbol table entry of the binding relationship record corresponding to the identifier reference point as the key to retrieve the renaming map and replace the identifier text. After the replacement is completed, a renaming intervention sample set is generated according to the replaced abstract syntax tree and the association between the renaming map and the renaming intervention sample is maintained.

[0024] In this embodiment, step four specifically includes: Define a semantically preserved transformation set, which includes comment insertion / deletion transformation, whitespace character insertion / deletion transformation, bracket insertion / deletion transformation, equivalent expression substitution transformation, conditional structure equivalent rewriting transformation, loop structure equivalent rewriting transformation, and statement exchange transformation. The comment insertion / deletion transformation performs insertion or deletion on lexical comment intervals while preserving the node type of the abstract syntax tree. The whitespace character insertion / deletion transformation performs insertion or deletion on spaces and newlines while preserving token boundaries. The bracket insertion / deletion transformation performs bracket insertion or deletion on expression subtrees while preserving the operation precedence. The equivalent expression substitution transformation performs algebraic equivalent substitution on expression subtrees while preserving the evaluation order of the abstract syntax tree. The conditional structure equivalent rewriting transformation replaces the conditional expression subtree with a logically equivalent form while preserving the branch body abstract syntax tree. The loop structure equivalent rewriting transformation replaces the loop control expression subtree with an equivalent form while preserving the loop body abstract syntax tree. Statement exchange transformation satisfies data dependency constraints and control dependency constraints. Data dependency constraints are achieved by locating statement nodes through an abstract syntax tree and extracting statement read sets and statement write sets. The statement read set is obtained by collecting the text of the identifier reference points in the statement subtree, and the statement write set is obtained by collecting the text of the identifiers corresponding to the assignment lvalues ​​in the statement subtree. The write-read dependency judgment satisfies that the preceding statement write set and the following statement read set have an intersection, and the read-write dependency judgment satisfies that the preceding statement read set and the following statement write set have an intersection. The write-write dependency judgment satisfies that the preceding statement write set and the following statement write set have an intersection. Control dependency constraints are achieved by locating branch nodes through an abstract syntax tree and extracting the statement intervals of the branch body. The statement reachability judgment satisfies that statement exchange does not change the statement interval inclusion relationship controlled by the branch node. The semantic preservation transformation set is generated by performing semantic preservation transformation on the code to be checked and the code to be compared. Performing a semantic preservation transformation includes replacing the node subtree at the abstract syntax tree level and generating the source code text in reverse from the replacement result. Parsing the source code text to generate the abstract syntax tree includes parsing the syntax rules again and generating the abstract syntax tree node and source code interval position. The identification binding set generation includes generating the identification binding set according to the binding relationship record generation process in claim 3. When the abstract syntax tree is successfully generated and the identification binding set is successfully generated, the transformation result is written into the transformation sample set and the source code interval position of the transformation type identifier and the transformation position is recorded. The basic kernel value of the sample pair is calculated for the sample to be deduplicated and the comparison sample in the transformed sample set. The calculation of the basic kernel value of the sample pair includes generating a structural feature vector from the abstract syntax tree of the sample and calculating the kernel function value of the structural feature vector. The structural feature vector is obtained by concatenating the node type count vector and the node depth count vector of the abstract syntax tree. The orbital average is performed on the basic kernel value of the sample pair to obtain the orbital convergence kernel. The orbital average includes taking the arithmetic mean of the basic kernel values ​​of a set of sample to be deduplicated and a set of comparison samples to generate the orbital convergence kernel.

[0025] In this embodiment, step five specifically includes: Select the transformation sample to be checked and the comparison transformation sample from the transformation sample set, and form a sample pair set according to the sample index. The sample index is determined by the sequence number of the transformation sample in the transformation sample set. The track convergence kernel value calculation includes calculating the track convergence kernel for each transformation sample to be checked and the comparison transformation sample, and writing the calculation result into the kernel value table. The kernel value table uses the sample index pair as the key and the track convergence kernel value as the value. Abstract syntax tree encoding is performed on the deduplication transformation samples and comparison transformation samples in the sample pair set. The abstract syntax tree encoding includes node type encoding, parent-child relationship encoding, and node depth encoding. The node type encoding is obtained by mapping the abstract syntax tree node type identifier to an integer encoding. The parent-child relationship encoding is obtained by concatenating the parent node type encoding and the child node type encoding according to the edge relationship. The node depth encoding is obtained by mapping the number of edges from the node to the root node of the abstract syntax tree to an integer encoding. The abstract syntax tree encoding is vectorized to obtain sample vectors and fed into the representation learning model. The representation learning model performs forward mapping on the sample vectors to generate a set of code representation vectors. The representation similarity of the code representation vector set is calculated for the sample pair set. The representation similarity calculation includes normalizing each pair of code representation vectors and calculating the vector dot product to obtain the similarity value. The alignment loss is generated based on the difference between the representation similarity and the orbit convergence kernel value in the kernel value table. The alignment loss is summed over the sample pair set to obtain the alignment loss value of the training batch. The process involves selecting renaming intervention samples from the renaming intervention sample set, extracting the corresponding identifier encoding vectors for each renaming intervention sample, forming an identifier encoding vector set, constructing the representation kernel matrix (using the code representation vector set as the sample set to calculate the basic kernel values ​​of any two code representation vectors and filling them to form the representation kernel matrix), constructing the identifier kernel matrix (using the identifier encoding vector set as the sample set to calculate the basic kernel values ​​of any two identifier encoding vectors and filling them to form the identifier kernel matrix), centering the representation kernel matrix and identifier kernel matrix by performing row mean elimination and column mean elimination on the representation kernel matrix and the identifier kernel matrix respectively, and compensating for the overall mean, calculating the Hilbert-Schmidt dependency degree by performing matrix multiplication on the centered representation kernel matrix and the centered identifier kernel matrix and calculating the trace value, and obtaining the information bottleneck constraint term by scaling the Hilbert-Schmidt dependency degree value according to the weights. Alignment loss and information bottleneck constraint terms are combined into an optimization objective by weights. The gradient of the optimization objective with respect to the parameters of the representation learning model is calculated and the parameters of the representation learning model are updated iteratively. At the end of the iteration update, the parameters of the representation learning model are fixed and abstract syntax tree encoding is performed on the abstract syntax tree of the code to be deduplicated and the abstract syntax tree of the comparison code to generate the code representation vector of the code to be deduplicated and the code representation vector of the comparison code.

[0026] In this embodiment, step six specifically includes: In the abstract syntax tree, the function block node is located. The function block node is determined by the node of type function in the abstract syntax tree. The statement nodes of the function block node are scanned in statement order. The statement order is determined by the occurrence index of the statement node in the function block node. When a branch statement node, loop statement node, or jump statement node is encountered, the basic block boundary is set. The basic block boundary is recorded as the boundary position of the statement node source code interval. The statement nodes between adjacent basic block boundaries are combined into basic block segments and a segment identifier is assigned to the basic block segment. The segment identifier is generated by concatenating the function block node identifier and the basic block segment index. Extract statement node subtrees from basic block segments. Each statement node subtree consists of a set of abstract syntax tree subtrees with statement nodes contained in the basic block segment as root nodes. Perform abstract syntax tree encoding to generate basic block encoding vectors. The abstract syntax tree encoding includes node type encoding, parent-child relationship encoding, and node depth encoding, and is vectorized in traversal order. Use a representation learning model to map the basic block encoding vectors to generate segment representation vectors. The segment representation vector has the same length as the code representation vector and serves as the representation of the basic block segment. The segment identifier of the identifier definition point is determined by using the source code interval location of the identifier definition point. Determining the segment identifier of the identifier definition point involves determining the interval inclusion between the source code interval location of the identifier definition point and the source code interval location of the statement node contained in the basic block segment, and selecting the segment identifier of the basic block segment containing the source code interval location of the identifier definition point. The segment identifier of the identifier reference point is determined by using the same interval inclusion determination method. Dependency edges are generated for the binding relationship records in the identifier binding set. The dependency edges are composed of the definition segment identifier and the reference segment identifier and are generated one by one according to the binding relationship records. Calculate the segment identifier sequence from the definition segment identifier to the reference segment identifier in the statement order of the function block node. The segment identifier sequence is composed of the definition segment identifier, the segment identifier located between the definition segment identifier and the reference segment identifier in the statement order, and the reference segment identifier in sequence. The segment identifier sequence is used as a dependency path and is collected to form a dependency path set. The dependency path set is obtained by deduplicating the segment identifier sequences corresponding to all dependency edges. Set a dependency path budget value, which is a positive integer. Sort the dependency path set in ascending order by dependency path length and extract the dependency paths corresponding to the budget value as budgeted dependency paths. The dependency path length is determined by the number of segment identifiers contained in the segment identifier sequence. Extract the segment representation vectors corresponding to adjacent segment identifiers in the segment identifier sequence based on the budgeted dependency paths and perform weighted aggregation. The weighted aggregation of segment representation vectors includes assigning weights to the segment representation vectors corresponding to adjacent segment identifiers according to their positional distance and calculating a weighted sum to obtain the updated segment representation vector and replace the original segment representation vector.

[0027] In this embodiment, step seven specifically includes: Normalization is performed on the code representation vector of the code to be checked and the code representation vector of the comparison code. The normalization process includes calculating the vector norm of the code representation vector and dividing the vector components by the vector norm one by one. The cosine similarity calculation includes calculating the vector dot product between the normalized code representation vector of the code to be checked and the normalized code representation vector of the comparison code and using the dot product value as the global similarity. A set of segment identifier pairs is constructed from the set of segment identifiers of the code to be checked and the set of segment identifiers of the code to be compared. The set of segment identifier pairs is obtained by performing a Cartesian product of the segment identifiers to be checked and the segment identifiers to be compared. The cosine similarity of the segment representation vectors corresponding to the segment identifier pairs is calculated by performing normalization processing on each pair of segment representation vectors in the segment identifier pairs and calculating the vector inner product. The segment similarity matrix is ​​formed by filling the corresponding cosine similarity of the segment identifier pairs according to the segment identifier index. The segment-level similarity is taken as the maximum value of the segment similarity matrix. The global similarity is compared with the global threshold to generate a global judgment flag. The global judgment flag is mapped to a binary identifier by the comparison result. The segment-level judgment flag is mapped to a binary identifier by the comparison result of the segment-level similarity and the segment threshold. The deduplication judgment result is synthesized by the global judgment flag and the segment-level judgment flag according to the preset logic rules and output as a binary judgment. A threshold filtering process is performed on the segment similarity matrix to obtain a set of similar segment identifier pairs. When the threshold filtering satisfies the condition that the element value of the segment similarity matrix is ​​greater than or equal to the segment threshold, the corresponding segment identifier pair is written into the set of similar segment identifier pairs. Based on the set of similar segment identifier pairs, the similar segment location result is determined by searching the basic block segment corresponding to the segment identifier pair for each pair in the set of similar segment identifier pairs, and extracting the source code interval position of the statement node contained in the basic block segment to form the source code interval position of the segment to be checked for plagiarism and the source code interval position of the comparison segment. The similar segment location result consists of the segment identifier to be checked for plagiarism, the comparison segment identifier, the source code interval position of the segment to be checked for plagiarism, and the source code interval position of the comparison segment.

[0028] A code deduplication detection system based on identifier encoding includes: The parsing module is used to obtain the code to be checked for plagiarism and the code to be compared, and to parse it to obtain the abstract syntax tree, symbol table, and scope tree; The binding parsing module is used to parse identifier definition points, identifier reference points, and binding relationships, generate identifier binding sets, and generate identifier encoding vectors. The renaming intervention module is used to generate a set of binding constraint renaming replacements, perform renaming interventions on the code to be checked and the code to be compared, and generate a renaming intervention sample set; The transformation kernel module is used to set the semantically preserved transformation set, perform the semantically preserved transformation set on the code to be checked and the code to be compared, generate the transformation sample set, and build the track convergence kernel; The bottleneck training module is used to train the representation learning model using the orbit convergence kernel and apply the Hilbert-Schmidt information bottleneck constraint to generate code representation vectors. The segmented budget module is used to divide basic blocks into segments based on the abstract syntax tree, generate segment representation vectors, and update segment representation vectors based on dependency path budget constraints. The determination and location module is used to calculate global similarity based on code representation vectors, calculate segment-level similarity based on segment representation vectors, and generate plagiarism determination results and similar segment location results.

[0029] Example 1: To verify the feasibility of this invention in practice, it was applied to source code similarity detection and code plagiarism review in a large-scale software development scenario. In this scenario, the code sources are complex, including code submitted by different developers within the same project, as well as code versions from different projects, different development standards, and different coding styles. The code scale covers function level, file level, and module level. In practical application, it was found that a large amount of code is highly similar in logical structure and semantic behavior, but there are significant differences in variable naming, function naming, comment content, whitespace format, and local control structure expression. Traditional plagiarism detection methods based on text features or simple syntactic features are difficult to accurately identify such similarities, and are prone to missed or false detections, which brings great difficulties to code auditing and quality assessment.

[0030] In this application scenario, the code set to be checked and the code set to be compared are first parsed to construct an abstract syntax tree, symbol table, and scope tree, clarifying the declaration position, visibility scope, and reference relationship of each identifier in the program. Then, by parsing the identifier definition points and reference points, an identifier binding set is generated, and the text information, declaration category, scope depth, and reference order of the identifiers are encoded into identifier encoding vectors. On this basis, binding constraint renaming intervention is performed on the code, and a renaming intervention sample set is systematically generated, so that the same semantic structure forms a set of equivalent code samples under different naming methods. At the same time, a semantic preservation transformation set is introduced to perform comment rewriting, equivalent expression replacement, equivalent rewriting of conditional and loop structures, and exchange of statements that satisfy dependency constraints on the code, forming a transformation sample set covering multiple rewriting methods. The similarity in the equivalent transformation space is uniformly measured by track convergence check.

[0031] In the representation learning phase, the model is trained by constraining sample pairs using orbit convergence kernels, enabling the model to learn stable code representation vectors. During training, Hilbert-Schmidt information bottleneck constraints are applied to explicitly limit the dependence of code representation vectors on identifier encoding vectors, preventing the model from overusing surface information such as variable names and function names. In the similarity calculation phase, the code is divided into basic blocks based on abstract syntax trees, and a dependency path set is constructed by combining identifier binding relationships. The segment representation vector is updated through dependency path budget constraints, thereby strengthening the constraint of consistency between control flow and data flow in segment-level similarity calculation. Finally, the global deduplication judgment result and similar segment location result are output simultaneously.

[0032] To verify the beneficial effects, comparative experiments were conducted on the same code dataset, comparing the method of this invention with traditional token-based methods, methods based on abstract syntax tree structure similarity, and deep representation methods without introducing information bottleneck constraints. The experimental results show that on a test set containing large-scale variable renaming and structural equivalence rewriting, the present invention demonstrates significant advantages in deduplication accuracy, recall, and localization accuracy, while maintaining stable overall computational efficiency. The statistical results of the comparative experiments are shown in Table 1. Table 1. Statistical Table of Key Indicators in Code Deduplication Comparison Experiment

[0033] Experimental data shows that this invention maintains high accuracy and recall rates even under complex rewriting conditions, with a significant improvement in similar segment localization accuracy. This indicates that through the synergistic effect of identifier encoding, renaming intervention, semantic preservation transformation, and Hilbert-Schmidt information bottleneck constraints, the code representation focuses more on the semantic and structural essence of the program, effectively reducing interference from naming differences and superficial rewriting. Meanwhile, despite introducing a more complex modeling and training process, the overall processing time remains within an acceptable range, making it suitable for batch deduplication and auditing of large-scale code in practical engineering environments.

[0034] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A code deduplication method based on identifier encoding, characterized in that, Includes the following steps: Step 1: Obtain the code to be checked for plagiarism and the code to be compared, and parse them to obtain the abstract syntax tree, symbol table, and scope tree; Step 2: Based on the symbol table and scope tree, resolve the identifier definition point, identifier reference point, and binding relationship to generate the identifier binding set, and generate the identifier encoding vector based on the identifier binding set; Step 3: Generate a binding constraint renaming permutation set based on the identifier binding set. Perform renaming intervention on the code to be checked and the code to be compared according to the binding constraint renaming permutation set to obtain the renaming intervention sample set; Step 4: Set up a semantically preserved transformation set, perform the semantically preserved transformation set on the code to be checked and the code to be compared, obtain the transformation sample set, and construct the track convergence kernel based on the transformation sample set; Step 5: Train the representation learning model using the orbit convergence kernel to obtain the code representation vector. During the training process, apply the Hilbert-Schmidt information bottleneck constraint to limit the dependency between the code representation vector and the identifier encoding vector of the renaming intervention sample set. Step 6: Divide the basic blocks based on the abstract syntax tree, calculate the segment representation vector of the basic blocks, and update the segment representation vector based on the dependency path budget constraint; Step 7: Calculate global similarity based on code representation vectors, calculate segment-level similarity based on segment representation vectors, and generate plagiarism detection results and similar segment location results.

2. The code deduplication method based on identifier encoding according to claim 1, characterized in that, Step one specifically involves: Obtain the code to be checked and the code to be compared, determine the programming language identifier, and perform character encoding unification, line break normalization, and indentation normalization on the source code text to obtain the standardized source code text; Based on programming language identifiers, syntax rules are selected, and lexical and syntactic analysis are performed on the normalized source code text to generate an abstract syntax tree. Identifier declaration nodes are identified based on the abstract syntax tree, and symbol table entries are generated. The symbol table entries contain the identifier text, declaration category, declaration location, and visibility boundary. The scope tree is generated from the block structure nodes based on the abstract syntax tree. The scope tree contains the parent-child scope relationship, the start and end positions of the scope, and the set of entries in the scope association symbol table.

3. The code deduplication method based on identifier encoding according to claim 1, characterized in that, Step two specifically involves: The symbol table entries are assigned to different categories based on the start and end positions of the scope tree, and the association between the scope tree nodes and the set of symbol table entries is established. Traverse the abstract syntax tree, locate the node where the identifier appears and extract the source code interval position. Determine whether the node where the identifier appears belongs to the identifier definition point or the identifier reference point according to the node type and parent node type of the abstract syntax tree. For an identifier reference point, the corresponding scope tree node is determined based on the source code range position. The association relationship is retrieved level by level along the parent-child scope relationship of the scope tree and the symbol table entry is matched. The identifier text that is consistent with the identifier reference point is determined and the declaration position is recorded. A binding relationship record is generated. The binding relationship record includes the source code range position of the identifier reference point, the symbol table entry declaration category, the symbol table entry declaration position, and the scope tree node identifier. The binding relationship records are aggregated to generate an identifier binding set, and the identifier text, declaration category, scope depth, and reference number are mapped to fixed-length numerical codes and concatenated to generate an identifier encoding vector.

4. The code deduplication method based on identifier encoding according to claim 1, characterized in that, Step three specifically involves: Extract the symbol table entry set based on the identifier binding set, and divide the symbol table entry set according to the scope tree node identifier to obtain the symbol table entry set corresponding to the scope tree node identifier; For each scope tree node, extract the identifier text set corresponding to the symbol table entry set, generate the renaming target dictionary, and the renaming target dictionary contains the replacement identifier text list and satisfies that the elements of the replacement identifier text list are all different; For each scope tree node identifier, construct a renaming mapping corresponding to the set of symbol table entries. The renaming mapping maps the symbol table entry declaration position to the replacement identifier text. Collect the renaming mappings corresponding to each scope tree node identifier to generate a set of binding constraint renaming permutations. Based on the binding constraint renaming replacement set, the identifier definition points and identifier reference points of the abstract syntax tree are replaced. The replacement uses the symbol table entry declaration position of the binding relationship record corresponding to the identifier reference point to retrieve the renaming mapping and replace the identifier text, generating a renaming intervention sample set.

5. The code deduplication method based on identifier encoding according to claim 1, characterized in that, Step four specifically involves: Define a semantically preserved transformation set, which includes comment insertion and deletion transformation, whitespace character insertion and deletion transformation, bracket insertion and deletion transformation, equivalent expression substitution transformation, conditional structure equivalent rewriting transformation, loop structure equivalent rewriting transformation, and statement exchange transformation. Statement exchange and transformation satisfy data dependency constraints and control dependency constraints. Data dependency constraints extract the statement read set and statement write set through the abstract syntax tree and determine post-write read dependency, read-after-write dependency, and post-write dependency. Control dependency constraints locate branch nodes through the abstract syntax tree and determine statement reachability relationships. The semantic-preserving transformation set is generated by performing a semantic-preserving transformation on the code to be checked and the code to be compared. After performing a semantic-preserving transformation, an abstract syntax tree is generated and an identifier binding set is generated. When the abstract syntax tree and the identifier binding set are successfully generated, the transformation result is written to the transformation sample set. The basic kernel value of the sample pair is calculated for the sample to be deduplicated and the comparison sample in the transformed sample set. The orbital average is then performed on the basic kernel value of the sample pair to obtain the orbital convergence kernel.

6. The code deduplication method based on identifier encoding according to claim 1, characterized in that, Step five specifically involves: Select the transformation samples to be checked and the comparison transformation samples from the transformation sample set, pair them according to the sample index to form a sample pair set, calculate the track convergence core value for the sample pair set and generate a core value table; Abstract syntax tree encoding is performed on the deduplication transformation samples and comparison transformation samples in the sample pair set. The abstract syntax tree encoding includes node type encoding, parent-child relationship encoding, and node depth encoding. The abstract syntax tree encoding is mapped by the representation learning model to generate a set of code representation vectors. The representation similarity of the code representation vector set is calculated for the sample pair set, and the alignment loss is generated based on the difference between the representation similarity and the orbit convergence kernel value in the kernel value table; Rename intervention samples are selected from the rename intervention sample set. The corresponding identifier encoding vectors of the rename intervention samples are extracted and a set of identifier encoding vectors is formed. A representation kernel matrix is ​​constructed based on the code representation vector set. An identifier kernel matrix is ​​constructed based on the identifier encoding vector set. The representation kernel matrix and the identifier kernel matrix are centered and the Hilbert-Schmidt dependency is calculated to generate information bottleneck constraint terms. Alignment loss and information bottleneck constraint terms are combined into an optimization objective by weights, and the parameters of the representation learning model are updated iteratively to generate the code representation vector of the code to be checked for duplication and the code representation vector of the code to be compared.

7. The code deduplication method based on identifier encoding according to claim 1, characterized in that, Step six specifically involves: Locate the function block node in the abstract syntax tree, scan the statement nodes of the function block node in statement order, set the basic block boundary when encountering branch statement node, loop statement node, and jump statement node, form a basic block segment by combining the statement nodes between adjacent basic block boundaries and assign segment identifiers to the basic block segment. Extract statement node subtrees from basic block segments, perform abstract syntax tree encoding to generate basic block encoding vectors, and use a representation learning model to map the basic block encoding vectors to generate segment representation vectors. The segment identifier where the identifier definition point is located is determined by the source code interval location of the identifier definition point, and the segment identifier where the identifier reference point is located is determined by the source code interval location of the identifier reference point. Dependency edges are generated for the binding relationship records in the identifier binding set. The dependency edges contain the definition segment identifier and the reference segment identifier. The segment identifier sequence from the definition segment identifier to the reference segment identifier is calculated in the statement order of the function body block node. The segment identifier sequence is used as a dependency path and is aggregated to form a dependency path set. Set a dependency path budget value, sort the dependency path set in ascending order of dependency path length, and extract the dependency path corresponding to the budget value as the budget dependency path. Extract the segment representation vector corresponding to the adjacent segment identifiers in the segment identifier sequence based on the budget dependency path and perform weighted aggregation to obtain the updated segment representation vector.

8. The code deduplication method based on identifier encoding according to claim 1, characterized in that, Step seven specifically involves: Normalization is performed on the code representation vectors of the code to be checked and the code representation vectors of the comparison code. The cosine similarity of the code representation vectors is calculated and used as the global similarity. Construct a set of segment identifier pairs for the segment identifier set of the code to be checked and the segment identifier set of the code to be compared. Calculate the cosine similarity of the segment representation vectors corresponding to the segment identifier pairs and form a segment similarity matrix. The maximum value of the segment similarity matrix is ​​taken as the segment-level similarity. The global similarity is compared with the global threshold to generate a global judgment flag, the segment-level similarity is compared with the segment threshold to generate a segment-level judgment flag, and the plagiarism detection result is generated based on the global judgment flag and the segment-level judgment flag. A threshold filtering is performed on the segment similarity matrix to obtain a set of similar segment identifier pairs. The similar segment location result is determined based on the set of similar segment identifier pairs. The similar segment location result includes the identifier of the segment to be checked for plagiarism, the identifier of the comparison segment, the source code interval position of the segment to be checked for plagiarism, and the source code interval position of the comparison segment.

9. A code deduplication system based on identifier encoding, comprising executing the code deduplication method based on identifier encoding as described in any one of claims 1 to 8, characterized in that, include: The parsing module is used to obtain the code to be checked for plagiarism and the code to be compared, and to parse it to obtain the abstract syntax tree, symbol table, and scope tree; The binding parsing module is used to parse identifier definition points, identifier reference points, and binding relationships, generate identifier binding sets, and generate identifier encoding vectors. The renaming intervention module is used to generate a set of binding constraint renaming replacements, perform renaming interventions on the code to be checked and the code to be compared, and generate a renaming intervention sample set; The transformation kernel module is used to set the semantically preserved transformation set, perform the semantically preserved transformation set on the code to be checked and the code to be compared, generate the transformation sample set, and build the track convergence kernel; The bottleneck training module is used to train the representation learning model using the orbit convergence kernel and apply the Hilbert-Schmidt information bottleneck constraint to generate code representation vectors. The segmented budget module is used to divide basic blocks into segments based on the abstract syntax tree, generate segment representation vectors, and update segment representation vectors based on dependency path budget constraints. The determination and location module is used to calculate global similarity based on code representation vectors, calculate segment-level similarity based on segment representation vectors, and generate plagiarism determination results and similar segment location results.

Images