Repeated code segment extraction method, system and storage medium

Abstract

The application provides a repeated code fragment extraction method, system and storage medium. The repeated code fragment extraction method comprises the following steps: obtaining a plurality of code files, and converting source codes of the plurality of code files into syntax trees respectively; extracting a first function call statement from a function definition node, and replacing a parameter value in the first function call statement with a parameter value corresponding type placeholder to obtain a second function call statement; grouping all second function call statements corresponding to at least one function definition node to obtain a plurality of first groups; combining at least two adjacent second function call statements in the first group to obtain a first sequence; splicing command signatures of all second function call statements contained in the first sequence to obtain a sequence signature of the first sequence; and grouping all first sequences based on the sequence signature to obtain a plurality of second groups. The application can improve the recognition accuracy of repeated codes with equivalent functions and similar structures.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method, system, and storage medium for extracting repetitive code fragments. Background Technology

[0002] In large-scale software testing projects, the number of automated test scripts often reaches thousands or even tens of thousands, and they are mostly written by different developers over a long period of time, resulting in a large amount of duplicate code with the same or similar functions but different writing styles.

[0003] This kind of repetitive code requires modification one by one when the interface changes, which can easily lead to omissions and runtime errors. Traditional text matching tools can only identify completely identical code, but cannot identify variants with "same function and similar parameter structure". They also lack semantic understanding capabilities and have difficulty identifying code snippets with different variable names and parameters but the same logic. Summary of the Invention

[0004] In view of this, this application proposes a method, system and storage medium for extracting duplicate code fragments to solve the problem that related technologies cannot identify duplicate code that is functionally equivalent and structurally similar.

[0005] The first aspect of this application proposes a method for extracting duplicate code snippets, which is applied to a system for extracting duplicate code snippets from automated test scripts. The method includes: Obtain multiple code files and convert the source code of the multiple code files into syntax trees respectively; each syntax tree includes at least one function definition node of the corresponding code file. For any function definition node, a first function call statement is extracted from the function definition node, and the parameter values ​​in the first function call statement are replaced with the type placeholders corresponding to the parameter values ​​to obtain a second function call statement; the first function call statement includes a function code snippet, a file path, and a function name. All second function call statements corresponding to the at least one function definition node are grouped to generate multiple first groups; the second function call statements in each first group belong to the same function in the same code file; For any first group, combine at least two adjacent second function call statements in the first group to obtain a first sequence; For any first sequence, the command signatures of all second function call statements contained in the first sequence are concatenated to generate the sequence signature of the first sequence; the command signature includes the command type and key parameter template of the second function call statement; Based on the sequence signature, all first sequences are grouped to generate multiple second groups. For any second group, all first sequences contained in the second group are determined to be duplicate code fragments.

[0006] The embodiments of this application have the following technical effects: 1. By abstracting the parameter values ​​in function call statements into type placeholders, the interference of differences in specific parameter values ​​on similarity judgment is eliminated, so that code snippets with the same function but different parameters (such as click operations on different buttons) can be accurately identified as duplicate code, avoiding the missed detection problem caused by differences in parameter values ​​in traditional text matching methods; 2. Function definition nodes are obtained by parsing the syntax tree, and statement combination and sequence signature generation are performed within the same function to ensure the cohesion of repeated code snippets in the code structure; at the same time, sequence signatures are constructed by command signatures (including command type and key parameter templates), which can accurately depict the function call pattern of the code and effectively distinguish code snippets with similar functions but different structures. 3. A hierarchical grouping strategy is adopted. First, the first grouping is performed according to the file and function dimensions to limit the context scope. Then, the second grouping is performed based on the sequence signature. This achieves progressive and fine-grained positioning from code file → function → statement sequence, avoiding interference from irrelevant code across functions or files and improving the accuracy of duplicate code extraction.

[0007] 4. For the scenario of "same operation mode, different test data" that exists in a large number of automated test scripts, parameter type abstraction and sequence signature matching can effectively identify duplicate test code with similar test steps but different test data, providing a reliable basis for the simplification and optimization of test scripts and the merging of test cases.

[0008] In this embodiment of the application, after obtaining the second function call statement, the method further includes: Determine the function name of the second function call statement; The command type of the second function call statement is determined based on the function name; Extract the key parameter template from the second function call statement; The command signature of the second function call statement is generated based on the command type and the key parameter template.

[0009] In this embodiment of the application, after grouping all first sequences based on sequence signatures to generate multiple second groups, the method further includes: Select several second subsequences with different sequence signatures from all first subsequences; For any second subsequence, calculate multiple similarities between multiple second function call statements of the second subsequence and multiple second function call statements of the target second subsequence; the target second subsequence is any subsequence other than the second subsequence among the plurality of second subsequences. The sequence similarity between the second sub-sequence and the target second sub-sequence is calculated based on the multiple similarities. If the sequence similarity is greater than or equal to a preset similarity threshold, the second subsequence and the target second subsequence are assigned to the same third group; all second subsequences within the same third group are identified as functionally equivalent and structurally similar duplicate code segments.

[0010] In this embodiment of the application, calculating multiple similarities between multiple second function call statements of the second sub-sequence and multiple second function call statements of the target second sub-sequence includes: For any second function call statement in the second subsequence, match the second function call statement with the corresponding second function call statement in the target second subsequence to generate multiple statement pairs; For any pair of statements, the similarity between the two second function call statements in the statement pair is calculated based on the functional category features, object chain features, parameter structure features, and text features, resulting in multiple similarity scores that correspond one-to-one with the multiple pairs of statements.

[0011] In this embodiment of the application, the similarity between the two second function call statements in the statement pair is calculated based on functional category features, object chain features, parameter structure features, and text features, including: For the two second function call statements in the statement pair, the function category corresponding to the function name of each second function call statement is determined according to a preset mapping table; the preset mapping table includes multiple function categories and the function name corresponding to each function category; Compare the functional categories of the two second function call statements, and determine the functional category feature score based on the functional category comparison results; Extract the object chain for each second function call statement; the object chain includes multiple objects during the function call process; Calculate the object chain similarity between the two second function call statements, and determine the object chain feature score based on the object chain similarity; Obtain multiple parameters for each second function call statement; these multiple parameters include the number of positional arguments, keyword arguments, and test framework-specific parameters. The parameters of the two second function call statements are compared one by one, and the parameter structure feature score is determined based on the comparison results of the multiple parameters. Calculate the text similarity between the two second function call statements, and determine the text feature score based on the text similarity. The similarity between the two second function call statements is calculated based on the function category feature score, the object chain feature score, the parameter structure feature score, and the text feature score.

[0012] In this embodiment of the application, after assigning the second sub-sequence and the target second sub-sequence to the same third group, the method further includes: Select several third subsequences from the plurality of second subsequences whose sequence similarity is less than the preset similarity threshold; For any third subsequence, each second function call statement in the third subsequence is converted into a corresponding structured text representation; The complete text description of the third subsequence is obtained by concatenating all the structured text representations in the third subsequence. The complete text description is converted into a corresponding semantic vector, and a hash value is obtained by hashing the complete text description. Based on the semantic vector and the hash value, all third subsequences are clustered to obtain multiple clusters; all third subsequences contained in each cluster are identified as functionally equivalent and structurally similar duplicate code fragments.

[0013] In this embodiment of the application, multiple clusters are obtained by clustering all third subsequences based on the semantic vector and the hash value, including: For any two third subsequences, if the hash values ​​of the two third subsequences are the same, then the two third subsequences are assigned to the same cluster. For several third subsequences with different hash values, multiple clusters are obtained by clustering the several third subsequences according to their semantic vectors.

[0014] In this embodiment of the application, the method further includes: If the total number of sequences in all third subsequences is greater than the preset number of sequences, then all third subsequences are divided into multiple batches. For any batch containing multiple third subsequences, the multiple third subsequences are clustered to obtain multiple clusters and a centroid vector for each cluster; the centroid vector is calculated based on the semantic vectors of all third subsequences within the cluster. Clustering the centroid vectors of all clusters in the multiple batches yields multiple classes; each class includes multiple clusters from different batches. If the number of classes is greater than the preset number of classes, the centroid vector of the class is used as a new third subsequence, and the step of dividing all third subsequences into multiple batches is repeated until the number of classes is no greater than the preset number of classes. If the number of classes is not greater than the preset number of classes, then stop the loop and take the currently obtained classes as the final clustering result; For any one of the aforementioned classes, all third subsequences of that class are identified as functionally equivalent and structurally similar duplicate code fragments.

[0015] An embodiment of the second aspect of this application provides a system for extracting duplicate code snippets, including: The syntax tree generation module is used to acquire multiple code files and convert the source code of the multiple code files into syntax trees respectively; each syntax tree includes at least one function definition node of the corresponding code file. The second function call statement generation module is used to extract a first function call statement from any function definition node, and replace the parameter values ​​in the first function call statement with the type placeholders corresponding to the parameter values ​​to obtain a second function call statement; the first function call statement includes a function code snippet, a file path, and a function name. The statement grouping module is used to group all second function call statements corresponding to the at least one function definition node to generate multiple first groups; the second function call statements in each first group belong to the same function in the same code file; The statement combination module is used to combine at least two adjacent second function call statements in any first group to obtain a first sequence; A sequence signature generation module is used to concatenate the command signatures of all second function call statements contained in any first sequence to generate a sequence signature for the first sequence; the command signature includes the command type and key parameter template of the second function call statement; The duplicate code fragment determination module is used to group all first sequences based on sequence signatures to generate multiple second groups, and for any second group, determine all first sequences contained in the second group as duplicate code fragments.

[0016] An embodiment of the third aspect of this application provides a computer device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the repetitive code fragment extraction method described in the first aspect above.

[0017] An embodiment of the fourth aspect of this application provides a computer-readable storage medium storing computer instructions for causing a computer to perform the repetitive code fragment extraction method described in the first aspect above.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a method for extracting duplicate code fragments according to an embodiment of this application is shown. Figure 2 This illustration shows a schematic diagram of the structure of a repetitive code fragment extraction system provided in an embodiment of this application; Figure 3 This illustration shows a schematic diagram of the structure of a computer device according to an embodiment of this application; Figure 4 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0020] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0021] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0022] The technical scenarios involved in the embodiments of this application are described below.

[0023] In large-scale software testing projects, the number of automated test scripts often reaches thousands or even tens of thousands. Because test scripts are typically written by multiple people over a long period, different developers may write code snippets with the same or similar functionality but slightly different syntax for similar test scenarios. This large amount of redundant and repetitive code leads to the following technical problems: (1) High code redundancy and poor maintainability: When the underlying interface or test requirements change, all scripts containing similar logic need to be modified one by one, which is a huge workload and is very easy to miss, resulting in some scripts having runtime errors due to not being updated in time.

[0024] (2) Difficulty in manually identifying duplicate code: Due to the large number of test scripts (e.g., thousands of files), and the fact that duplicate code is not exactly the same but "similar in structure but different in parameters", it is difficult to fully discover all reusable common patterns by relying on manual review. Traditional code duplication detection tools (such as text matching tools) can only identify completely identical or nearly identical code, and cannot identify code patterns that are similar in structure but different in parameterization.

[0025] (3) Lack of semantic-level similarity understanding: Existing code clone detection methods are mainly based on text comparison or simple syntax tree comparison, which cannot understand the functional semantics of the code. For example, two pieces of code use different variable names and different command parameters, but implement the same test logic. Existing methods have difficulty identifying them as the same pattern.

[0026] (4) Performance bottleneck of large-scale analysis: When it is necessary to perform full similarity analysis and clustering on hundreds of thousands of statements in thousands of files, traditional methods face serious computational performance and memory bottlenecks, making it difficult to complete the analysis within an acceptable time.

[0027] The embodiments of this application are used to automatically identify repetitive code segments with similar structures and equivalent functions from a large-scale test script collection, cluster them into reusable common patterns, and generate parameterized general module templates, thereby significantly improving code reusability and maintainability.

[0028] According to an embodiment of this application, a method for extracting repeated code fragments is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0029] This embodiment provides a method for extracting duplicate code snippets, applied to a system for extracting duplicate code snippets from automated test scripts. Figure 1 This is a flowchart of a method for extracting duplicate code fragments according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain multiple code files and convert the source code of the multiple code files into syntax trees respectively.

[0030] Specifically, the code files contained in the code file set are parsed separately, and the source code in each code file is converted into a syntax tree by an abstract syntax tree parser.

[0031] In some specific embodiments, a multi-encoding trial strategy is employed during the parsing process (trying UTF-8, UTF-8-sig, GBK, GB2312, Latin-1, etc. sequentially) to ensure broad compatibility with source code of different encoding formats. Simultaneously, the extracted source code undergoes cleaning: redundant blank lines are removed, the original indentation structure is preserved, and multiple consecutive line breaks are merged into a single line break to ensure the accuracy of subsequent comparisons.

[0032] Step S102: For any function definition node, extract the first function call statement from the function definition node, and replace the parameter value in the first function call statement with the type placeholder corresponding to the parameter value to obtain the second function call statement.

[0033] In some specific embodiments, extracting the first function call statement from the function definition node includes: after obtaining the syntax tree corresponding to the source code, traversing all function definition nodes (FunctionDef) in the syntax tree, extracting function call statements (e.g., Call expressions) from them, and determining the quadruple information of each function call statement, including the function code fragment, the syntax tree node object, the file path to which it belongs, and the name of the function to which it belongs.

[0034] In some specific implementations, a cache key can be generated for each code file based on the hash value of its content, and the parsing results can be stored in the cache. When the file content remains unchanged, it can be loaded directly from the cache, avoiding repeated parsing. The cache adopts a categorized storage strategy, managing six types of data separately: statements, signatures, features, sequences, patterns, and embeddings. It also supports gzip compression to reduce disk usage. The cache also introduces thread locks to ensure concurrency safety and a cache isolation mechanism based on configuration hashes to ensure that different analysis configurations use independent cache spaces.

[0035] In some specific embodiments, the second function call statement is obtained by replacing the parameter value in the first function call statement with a placeholder of the type corresponding to the parameter value, including: Each extracted first function call statement is standardized by replacing the specific parameter value with a placeholder of the type corresponding to its parameter value, so as to perform a structural comparison.

[0036] In this application embodiment, the standardization rules include: (a) Replace the string literal with the 'STR' placeholder; (b) Replace the variable references in the f-string with the {VAR} placeholders; (c) Replace numeric literals with NUM placeholders; (d) Replace the IP address pattern (e.g., 192.168.1.1) with IP placeholders; (e) Replace the number in the network interface name with the {INTF} placeholder; (f) Replace resource identifiers with number suffixes (such as Resource1, Resource2) with placeholders {RES}{NUM}.

[0037] In some specific embodiments, after obtaining the second function call statement, the method further includes steps a1-a4: Step a1: Determine the function name of the second function call statement.

[0038] Step a2: Determine the command type of the second function call statement based on the function name.

[0039] Specifically, the function name can be obtained through the attribute access chain (ast.Attribute) or the name node (ast.Name), thereby determining the command type of the second function call statement. For example, when the second function call statement involves the function name "run_check", it can be determined that the command type of the second function call statement is a verification command or a check command.

[0040] Step a3: Extract the key parameter template from the second function call statement.

[0041] Specifically, the key parameter templates can be obtained by classifying the functions according to their names. For example, for run_check, the content of the cmd keyword parameter can be extracted from the second function call statement as the key parameter template; for send, the content of its first positional parameter can be extracted from the second function call statement as the key parameter template.

[0042] Step a4: Generate the command signature of the second function call statement based on the command type and the key parameter template.

[0043] Specifically, the command signature is a structured summary of the second function call statement, consisting of the command type and key parameter templates.

[0044] This application embodiment can quickly determine whether two function call statements belong to the same pattern category by matching the command signature of the statements, laying the foundation for subsequent rapid grouping.

[0045] The following specific examples illustrate step S102 and subsequent steps a1-a4: Suppose that in the test script collection, file A and file B each contain the following source code: The test_step_1 function in file A (file_A.py): tester.run_check('Check entry A1', cmd=f'query status {ctx.addr1} {ctx.mask1}', expect=['active'], strict_mode=False) tester.run_check('Check entry A2', cmd=f'query detail info', expect=['type: normal']) The test_step_1 function in file B (file_B.py): tester.run_check('Verify entry B1', cmd=f'query status {ctx.addr2} {ctx.mask2}', expect=['active'], strict_mode=False) tester.run_check('Verify entry B2', cmd=f'query detail info', expect=['type: normal']) The processing steps are as follows: Parse files A and B respectively, and extract the `run_check` call statement from the function body of `test_step_1`. Extract two first function call statements, A.1 and A.2, from file A, including: First function call statement A.1: tester.run_check('Check entry A1',cmd=f'query status {ctx.addr1}{ctx.mask1}',expect=['active'],strict_mode=False) First function call statement A.2: tester.run_check('Check entry A2',cmd=f'query detail info',expect=['type:normal']) File A extracts two first function call statements, B.1 and B.2, including: First function call statement B.1: tester.run_check('Verify entry B1',cmd=f'query status {ctx.addr2}{ctx.mask2}',expect=['active'], strict_mode=False) First function call statement B.2: tester.run_check('Verify entry B2',cmd=f'query detail info',expect=['type:normal']) The second function call statement is obtained by standardizing the first function call statement A.1 in file A; the second function call statement is: tester.run_check('STR', cmd=f'query status {VAR} {VAR}',expect=['STR'], strict_mode=False); The command signature of the second function call statement is: ("run_check", "query status{VAR}") The command signatures obtained by similarly processing the first function call statement A.2 in file A and the first function call statements B.1 and B.2 in file B are: ("run_check", "query detail info"), ("run_check", "query status {VAR}"), ("run_check", "query detail info") Step S103: Group all second function call statements corresponding to the at least one function definition node to generate multiple first groups.

[0046] Specifically, all second function call statements can be divided into multiple first groups by using the file path and function name to which the second function call statement belongs, so that the second function call statements in each first group belong to the same function in the same code file.

[0047] Step S104: For any first group, combine at least two adjacent second function call statements in the first group to obtain a first sequence.

[0048] Specifically, within each first group, all possible first subsequences can be generated by traversing from the minimum number of functions to the maximum number of functions using a sliding window approach. For example, when the first group contains [statement A1, statement A2, statement A3, statement A4], it can be divided into first subsequences [statement A1, statement A2], [statement A2, statement A3], and [statement A3, statement A4] based on a function count of 2; and into first subsequences [statement A1, statement A2, statement A3] and [statement A2, statement A3, statement A4] based on a function count of 3. The minimum number of functions is 2, while the maximum number of functions is not specifically limited here.

[0049] Step S105: For any first sequence, concatenate the command signatures of all second function call statements contained in the first sequence to generate a sequence signature for the first sequence.

[0050] Specifically, each second function call statement has a corresponding command signature. The command signatures of all second function call statements in the first sequence can be concatenated (e.g., concatenated sequentially using the "||" separator) to generate the sequence signature of the first sequence, for example: In the example above, the first sequence X is formed by the second function call statements A.1 and A.2 from file A, and the sequence signature of the first sequence X is: "run_check::query status {VAR} || run_check::query detail info" The first sequence Y is formed by the second function call statements B.1 and B.2 from file B. The sequence signature of the first sequence Y is: "run_check::query status {VAR} || run_check::query detail info" Step S106: Based on the sequence signature, all first sequences are grouped to generate multiple second groups. For any second group, all first sequences contained in the second group are determined to be duplicate code fragments.

[0051] Specifically, each second group contains at least two first sequences, and the sequence signatures of the at least two first sequences are the same.

[0052] The fast grouping strategy based on sequence signatures provided in this application is a significant performance innovation of the present invention. Compared to the traditional method that requires pairwise similarity calculation (time complexity O(n²)), signature-based grouping only requires O(n) time complexity to complete the initial pattern recognition, making it possible to process hundreds of thousands of sentences.

[0053] In some specific embodiments, after step S106 above, steps b1-b4 are further included: Step b1: Select several second subsequences with different sequence signatures from all the first subsequences.

[0054] Specifically, after generating multiple second groups by grouping all first sequences based on sequence signatures, the first sequences contained in these multiple second groups are removed from all the first sequences mentioned above, and the remaining first sequences are used as second subsequences.

[0055] Step b2: For any second subsequence, calculate multiple similarities between multiple second function call statements of the second subsequence and multiple second function call statements of the target second subsequence.

[0056] Specifically, the target second subsequence is any subsequence other than the second subsequence among the plurality of second subsequences.

[0057] In some specific embodiments, step b2 above further includes steps b21-b22: Step b21: For any second function call statement in the second subsequence, match the second function call statement with the corresponding second function call statement in the target second subsequence to generate multiple statement pairs.

[0058] Step b22: For any pair of statements, calculate the similarity between the two second function call statements in the statement pair based on the function category features, object chain features, parameter structure features, and text features, to obtain multiple similarity scores that correspond one-to-one with the multiple pairs of statements.

[0059] In this embodiment of the application, when the second subsequence is [x1,x2,x3] and the target second subsequence is [y1,y2,y3], statement pair 1 [x1,y1], statement pair 2 [x2,y2], and statement pair 3 [x3,y3] are generated, and the similarity between x1 and y1, x2 and y2, and x3 and y3 are calculated.

[0060] In some specific embodiments, step b22 above includes steps c1-c8: Step c1: For the two second function call statements in the statement pair, determine the function category corresponding to the function name of each second function call statement according to the preset mapping table.

[0061] Specifically, the preset mapping table includes multiple functional categories and the function names corresponding to each functional category, such as: stream class (including CheckStreamStats, CreateIPStream), port class (including CheckPortStats), capture class (including StartCapture, StopCapture), snmp class (including set_snmp_data), control class (such as Run, Stop, Reset), and util class (such as sleep, log_util).

[0062] Step c2: Compare the functional categories of the two second function call statements, and determine the functional category feature score based on the functional category comparison results.

[0063] Specifically, the score for this dimension is 1.0 if the two second function call statements have the same function category, and 0.0 otherwise.

[0064] Step c3: Extract the object chain of each second function call statement; the object chain includes multiple objects in the function call process, for example: the object chain in "tester.run_check" is ['tester'], and the object chain in "proj.tester.run_check" is ['proj', 'tester'].

[0065] Step c4: Calculate the object chain similarity between the two second function call statements, and determine the object chain feature score based on the object chain similarity.

[0066] Specifically, the object chain similarity between two second function call statements can be calculated using the SequenceMatcher algorithm.

[0067] Step c5: Obtain multiple parameters for each second function call statement.

[0068] Specifically, the multiple parameters include the number of positional parameters, keyword parameters, and test framework-specific parameters.

[0069] Step c6 involves comparing the multiple parameters of the two second function call statements one by one, and determining the parameter structure feature score based on the comparison results of the multiple parameters.

[0070] Specifically, multiple parameters are compared one by one, including: whether the number of positional parameters is the same (0.3%), whether they are all keyword parameters (0.2%), the Jaccard intersection ratio of keyword parameter names (0.3%), and the degree of matching of specific parameters of the test framework (0.2%).

[0071] Step c7: Calculate the text similarity between the two second function call statements, and determine the text feature score based on the text similarity.

[0072] Step c8: Calculate the similarity between the two second function call statements based on the function category feature score, the object chain feature score, the parameter structure feature score, and the text feature score.

[0073] Specifically, the similarity between two second function call statements can be calculated using the following formula: Score = (w1×S_func + w2×S_obj + w3×S_param + w4×S_text) / (w1+w2+w3+w4) Here, Score represents the similarity between two second function call statements; S_func represents the function category feature score; w1 represents the weight corresponding to the function category feature score (e.g., 0.3); S_obj represents the object chain feature score; w2 represents the weight corresponding to the object chain feature score (e.g., 0.2); S_param represents the parameter structure feature score; w3 represents the weight corresponding to the parameter structure feature score (e.g., 0.25); S_text represents the text feature score; and w4 represents the weight corresponding to the text feature score (e.g., 0.25). When a dimension cannot be calculated due to missing data, that dimension is automatically excluded from the weighted calculation.

[0074] Step b3: Calculate the sequence similarity between the second subsequence and the target second subsequence based on the multiple similarities.

[0075] Specifically, the average of multiple similarities can be used as the sequence similarity between the second subsequence and the target second subsequence.

[0076] Step b4: If the sequence similarity is greater than or equal to a preset similarity threshold, then the second subsequence and the target second subsequence are assigned to the same third group.

[0077] Specifically, all second subsequences within the same third group were identified as functionally equivalent and structurally similar repeated code fragments.

[0078] In this application embodiment, since the sequence signature cannot be determined, the sequence similarity between two second sub-sequences is calculated by weighting multiple dimensions of features, and the sequence similarity is used to determine whether the two are functionally equivalent and structurally similar duplicate code segments, which can further improve the accuracy of duplicate code segment determination.

[0079] In some specific embodiments, after step b4 above, the method further includes steps d1-d5: Step d1: Select several third subsequences from the several second subsequences whose sequence similarity is less than the preset similarity threshold.

[0080] Specifically, the second subsequence that was not assigned to the third group is determined as the third subsequence, and then it is further determined whether there are duplicate code segments among the multiple third subsequences.

[0081] Step d2: For any third subsequence, convert each second function call statement in the third subsequence into a corresponding structured text representation.

[0082] Specifically, the structured text representation includes the hierarchical path of function calls in the abstract syntax tree and the contextual information of the function call statement (used to distinguish functions with the same name but different meanings).

[0083] Step d3 involves concatenating all structured text representations in the third subsequence to obtain the complete text description of the third subsequence.

[0084] Specifically, the complete text description of the third subsequence is obtained by concatenating all the structured text representations in the third subsequence according to the execution sequence, where the execution sequence refers to the order in which the code is executed.

[0085] Step d4: Convert the complete text description into a corresponding semantic vector, and perform a hash calculation on the complete text description to obtain a hash value.

[0086] Specifically, the complete text description can be sent to the vector embedding service to obtain the corresponding semantic vector, and the semantic vector is uniformly stored using float16 precision.

[0087] Step d5: Cluster all third subsequences according to the semantic vector and the hash value to obtain multiple clusters; all third subsequences contained in each cluster are identified as functionally equivalent and structurally similar duplicate code fragments.

[0088] In some specific embodiments, step d5 includes steps d51-d52: Step d51: For any two third subsequences, if the hash values ​​of the two third subsequences are the same, then the two third subsequences are assigned to the same cluster.

[0089] Step d52: For several third subsequences with different hash values, cluster the several third subsequences according to their semantic vectors to obtain multiple clusters.

[0090] Specifically, clustering can be density clustering or hierarchical clustering, without any specific limitations.

[0091] In some specific embodiments, the method further includes steps e1-e6: Step e1: If the total number of sequences in all third subsequences is greater than the preset number of sequences, then divide all third subsequences into multiple batches.

[0092] Specifically, the number of preset sequences can be predefined and is not specifically limited here. The number of third subsequences included in each batch can also be predefined, for example, 5000.

[0093] Step e2: For any batch containing multiple third subsequences, cluster the multiple third subsequences to obtain multiple clusters and the centroid vector of each cluster.

[0094] Specifically, hierarchical clustering or density clustering can be performed on multiple third subsequences.

[0095] More specifically, the centroid vector is the average of the semantic vectors of all third subsequences within a cluster.

[0096] Step e3: Cluster the centroid vectors of all clusters in the multiple batches to obtain multiple classes; each class includes multiple clusters from different batches.

[0097] Specifically, similar cross-batch clusters are merged by performing secondary clustering on the centroid vectors of clusters generated in all batches.

[0098] Step e4: If the number of classes is greater than the preset number of classes, then the centroid vector of the class is used as a new third subsequence, and the step of dividing all third subsequences into multiple batches is repeated until the number of classes is not greater than the preset number of classes.

[0099] Specifically, the number of preset classes can be predefined and is not specifically limited here.

[0100] Step e5: If the number of classes is not greater than the preset number of classes, stop the loop and take the currently obtained classes as the final clustering result.

[0101] Step e6: For any one of the several classes, determine all third subsequences of the class as functionally equivalent and structurally similar duplicate code fragments.

[0102] The embodiments of this application employ a two-stage merging strategy of "intra-batch clustering → centroid extraction → secondary centroid clustering," which can effectively address the identification of duplicate code segments in large-scale datasets.

[0103] In some specific embodiments, for the second group, the third group, and the repeated code fragments in each cluster, this invention adopts a two-stage summarization strategy that combines a "rule engine + large language model (LLM)": Phase 1 (Rule Engine Summary): Based on structured information such as function names, command types, and parameter structures within groups or clusters, basic functional descriptions are quickly generated using predefined rule templates. This phase requires no external dependencies, can be completed instantly, and serves as a baseline summary.

[0104] Specifically, for example, the rule engine summary of the above example is: "run_check sequence mode: first execute querystatus to check the entry status, then execute query detail info to view the entry details."

[0105] The second phase (LLM enhancement summary): The code examples, basic descriptions and context information in the groups or clusters are built into structured prompts, which are then submitted in batches to the Large Language Model API for in-depth analysis to generate more accurate and easier-to-understand function summaries and usage instructions.

[0106] Specifically, for example, the LLM enhancement summary of the above example: "This mode is used to verify the state consistency of a specified resource: first check whether the entry under the specified parameter is in an active state, and then check the entry details to confirm that the type is normal."

[0107] In this embodiment, a batch concurrent call mechanism is designed in the LLM call stage: a thread pool (ThreadPoolExecutor) is used to manage concurrent requests, supporting a configurable maximum concurrency and request interval. Timeout control and failure degradation mechanisms are also provided—when an API call times out or returns an error, it automatically falls back to using the rule engine summary from the first stage, ensuring system robustness. Patterns requiring LLM summarization are collected centrally and submitted in batches at once, avoiding the inefficiency of sequential calls.

[0108] In some specific embodiments, all extracted patterns (i.e., each second group, third group, and cluster) are sorted in descending order of frequency of occurrence and output in a specified format.

[0109] Specifically, the specified formats include JSON, JSONL, Python module, and Markdown document formats. The JSON format includes structured data such as version number, configuration information, a complete list of patterns, and statistical information (total number of patterns, total frequency, and average frequency). The JSONL format includes metadata on the first line, with each subsequent line containing an independent pattern JSON object, suitable for streaming reading and processing of large-scale data. The Python module format converts each pattern into a Python function definition with a parameter list and docstring, which can be directly imported into the project. The Markdown document format generates readable documentation containing pattern descriptions, sample code, and usage instructions.

[0110] Corresponding to the above implementation methods for extracting duplicate code snippets, this application also provides a system for extracting duplicate code snippets, used to execute the method for extracting duplicate code snippets described in any of the above embodiments. For example... Figure 2 As shown, the duplicate code snippet extraction system includes: The syntax tree generation module is used to acquire multiple code files and convert the source code of the multiple code files into syntax trees respectively; each syntax tree includes at least one function definition node of the corresponding code file. The second function call statement generation module is used to extract a first function call statement from any function definition node, and replace the parameter values ​​in the first function call statement with the type placeholders corresponding to the parameter values ​​to obtain a second function call statement; the first function call statement includes a function code snippet, a file path, and a function name. The statement grouping module is used to group all second function call statements corresponding to the at least one function definition node to generate multiple first groups; the second function call statements in each first group belong to the same function in the same code file; The statement combination module is used to combine at least two adjacent second function call statements in any first group to obtain a first sequence; A sequence signature generation module is used to concatenate the command signatures of all second function call statements contained in any first sequence to generate a sequence signature for the first sequence; the command signature includes the command type and key parameter template of the second function call statement; The duplicate code fragment determination module is used to group all first sequences based on sequence signatures to generate multiple second groups, and for any second group, determine all first sequences contained in the second group as duplicate code fragments.

[0111] Optionally, the system further includes: a command signature generation module, used to determine the function name of the second function call statement; determine the command type of the second function call statement based on the function name; extract a key parameter template from the second function call statement; and generate a command signature of the second function call statement based on the command type and the key parameter template.

[0112] Optionally, the system further includes: a third grouping module, used to filter out several second subsequences with different sequence signatures from all first subsequences; for any second subsequence, calculate multiple similarities between multiple second function call statements of the second subsequence and multiple second function call statements of the target second subsequence; the target second subsequence is any subsequence other than the second subsequence among the several second subsequences; calculate the sequence similarity between the second subsequence and the target second subsequence based on the multiple similarities; if the sequence similarity is greater than or equal to a preset similarity threshold, then assign the second subsequence and the target second subsequence to the same third group; all second subsequences within the same third group are identified as functionally equivalent and structurally similar duplicate code segments.

[0113] Optionally, the third grouping module is further configured to match any second function call statement in the second subsequence with the corresponding second function call statement in the target second subsequence to generate multiple statement pairs; for any statement pair, calculate the similarity between the two second function call statements in the statement pair based on functional category features, object chain features, parameter structure features, and text features to obtain multiple similarities that correspond one-to-one with the multiple statement pairs.

[0114] Optionally, the third grouping module is further configured to, for the two second function call statements in the statement pair, determine the function category corresponding to the function name of each second function call statement according to a preset mapping table; the preset mapping table includes multiple function categories and the function name corresponding to each function category; compare the function categories of the two second function call statements and determine the function category feature score based on the function category comparison result; extract the object chain of each second function call statement; the object chain includes multiple objects in the function call process; calculate the object chain similarity of the two second function call statements and determine the object chain feature score based on the object chain similarity; obtain multiple parameters of each second function call statement; the multiple parameters include the number of positional parameters, keyword parameters, and test framework-specific parameters; compare the multiple parameters of the two second function call statements one by one, and determine the parameter structure feature score based on the comparison result of the multiple parameters; calculate the statement text similarity of the two second function call statements and determine the text feature score based on the statement text similarity; calculate the similarity of the two second function call statements based on the function category feature score, the object chain feature score, the parameter structure feature score, and the text feature score.

[0115] Optionally, the system further includes: a third subsequence clustering module, used to filter out several third subsequences from the several second subsequences whose sequence similarity is less than the preset similarity threshold; for any third subsequence, converting each second function call statement in the third subsequence into a corresponding structured text representation; concatenating all structured text representations in the third subsequence to obtain a complete text description of the third subsequence; converting the complete text description into a corresponding semantic vector, and performing hash calculation on the complete text description to obtain a hash value; clustering all third subsequences according to the semantic vector and the hash value to obtain multiple clusters; all third subsequences contained in each cluster are identified as functionally equivalent and structurally similar duplicate code fragments.

[0116] Optionally, a third subsequence clustering module is used to assign any two third subsequences to the same cluster if the hash values ​​of the two third subsequences are the same; and to cluster several third subsequences with different hash values ​​according to the semantic vectors of the several third subsequences to obtain multiple clusters.

[0117] Optionally, the system further includes: a batch partitioning module, used to divide all third subsequences into multiple batches if the number of sequences in all third subsequences is greater than a preset number of sequences; for any batch containing multiple third subsequences, clustering the multiple third subsequences to obtain multiple clusters and a centroid vector for each cluster; the centroid vector is calculated based on the semantic vectors of all third subsequences within the cluster; clustering the centroid vectors of all clusters in the multiple batches to obtain multiple classes; each class includes multiple clusters from different batches; if the number of classes is greater than the preset number of classes, the centroid vector of the class is used as a new third subsequence, and the step of dividing all third subsequences into multiple batches is repeated until the number of classes is not greater than the preset number of classes; if the number of classes is not greater than the preset number of classes, the loop is stopped, and the currently obtained classes are used as the final clustering result; for any class among the multiple classes, all third subsequences of the class are determined as functionally equivalent and structurally similar duplicate code fragments.

[0118] The duplicate code segment extraction system provided in the above embodiments of this application and the duplicate code segment extraction method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods used, run or implemented by the applications they store.

[0119] This application also provides a computer device for performing the above-described method for extracting duplicate code fragments. Please refer to... Figure 3 This illustrates a schematic diagram of a computer device provided by some embodiments of this application. For example... Figure 3 As shown, the computer device 3 includes: a processor 300, a memory 301, a bus 302, and a communication interface 303. The processor 300, the communication interface 303, and the memory 301 are connected via the bus 302. The memory 301 stores a computer program that can run on the processor 300. When the processor 300 runs the computer program, it executes the method for extracting duplicate code segments provided in any of the foregoing embodiments of this application.

[0120] The memory 301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0121] Bus 302 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 301 is used to store programs. After receiving an execution instruction, the processor 300 executes the program. The method for extracting repetitive code segments disclosed in any of the foregoing embodiments can be applied to the processor 300, or implemented by the processor 300.

[0122] The processor 300 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the steps of the above method.

[0123] The computer device provided in this application embodiment and the method for extracting duplicate code segments provided in this application embodiment are based on the same inventive concept and have the same beneficial effects as the methods they adopt, run or implement.

[0124] This application also provides a computer-readable storage medium corresponding to the duplicate code fragment extraction method provided in the foregoing embodiments. Please refer to... Figure 4 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the method for extracting repeated code fragments provided in any of the foregoing embodiments.

[0125] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0126] The computer-readable storage medium provided in the above embodiments of this application and the method for extracting duplicate code fragments provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods used, run or implemented by the applications stored therein.

[0127] It should be noted that: Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0128] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0129] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0130] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for extracting repetitive code snippets, characterized in that, The method is applied to a system for extracting repetitive code snippets from automated test scripts. The method includes: Obtain multiple code files and convert the source code of the multiple code files into syntax trees respectively; each syntax tree includes at least one function definition node of the corresponding code file. For any function definition node, a first function call statement is extracted from the function definition node, and the parameter values ​​in the first function call statement are replaced with the type placeholders corresponding to the parameter values ​​to obtain a second function call statement; the first function call statement includes a function code snippet, a file path, and a function name. All second function call statements corresponding to the at least one function definition node are grouped to generate multiple first groups; the second function call statements in each first group belong to the same function in the same code file; For any first group, combine at least two adjacent second function call statements in the first group to obtain a first sequence; For any first sequence, the command signatures of all second function call statements contained in the first sequence are concatenated to generate the sequence signature of the first sequence; the command signature includes the command type and key parameter template of the second function call statement; Based on the sequence signature, all first sequences are grouped to generate multiple second groups. For any second group, all first sequences contained in the second group are determined to be duplicate code fragments. After obtaining the second function call statement, the method further includes: Determine the function name of the second function call statement; The command type of the second function call statement is determined based on the function name; Extract the key parameter template from the second function call statement; The command signature of the second function call statement is generated based on the command type and the key parameter template.

2. The method according to claim 1, characterized in that, After grouping all first sequences based on sequence signatures to generate multiple second groups, the method further includes: Select several second subsequences with different sequence signatures from all first sequences; For any second subsequence, calculate multiple similarities between multiple second function call statements of the second subsequence and multiple second function call statements of the target second subsequence; the target second subsequence is any subsequence other than the second subsequence among the plurality of second subsequences. The sequence similarity between the second sub-sequence and the target second sub-sequence is calculated based on the multiple similarities. If the sequence similarity is greater than or equal to a preset similarity threshold, the second subsequence and the target second subsequence are assigned to the same third group; all second subsequences within the same third group are identified as functionally equivalent and structurally similar duplicate code segments.

3. The method according to claim 2, characterized in that, Calculating multiple similarities between multiple second function call statements of the second subsequence and multiple second function call statements of the target second subsequence includes: For any second function call statement in the second subsequence, match the second function call statement with the corresponding second function call statement in the target second subsequence to generate multiple statement pairs; For any pair of statements, the similarity between the two second function call statements in the statement pair is calculated based on the functional category features, object chain features, parameter structure features, and text features, resulting in multiple similarity scores that correspond one-to-one with the multiple pairs of statements.

4. The method according to claim 3, characterized in that, The similarity between the two second function call statements in the statement pair is calculated based on functional category features, object chain features, parameter structure features, and text features, including: For the two second function call statements in the statement pair, the function category corresponding to the function name of each second function call statement is determined according to a preset mapping table; the preset mapping table includes multiple function categories and the function name corresponding to each function category; Compare the functional categories of the two second function call statements, and determine the functional category feature score based on the functional category comparison results; Extract the object chain for each second function call statement; the object chain includes multiple objects during the function call process; Calculate the object chain similarity between the two second function call statements, and determine the object chain feature score based on the object chain similarity; Obtain multiple parameters for each second function call statement; these multiple parameters include the number of positional arguments, keyword arguments, and test framework-specific parameters. The parameters of the two second function call statements are compared one by one, and the parameter structure feature score is determined based on the comparison results of the multiple parameters. Calculate the text similarity between the two second function call statements, and determine the text feature score based on the text similarity. The similarity between the two second function call statements is calculated based on the function category feature score, the object chain feature score, the parameter structure feature score, and the text feature score.

5. The method according to claim 2, characterized in that, After assigning the second subsequence and the target second subsequence to the same third group, the method further includes: Select several third subsequences from the plurality of second subsequences whose sequence similarity is less than the preset similarity threshold; For any third subsequence, each second function call statement in the third subsequence is converted into a corresponding structured text representation; The complete text description of the third subsequence is obtained by concatenating all the structured text representations in the third subsequence. The complete text description is converted into a corresponding semantic vector, and a hash value is obtained by hashing the complete text description. Based on the semantic vector and the hash value, all third subsequences are clustered to obtain multiple clusters; all third subsequences contained in each cluster are identified as functionally equivalent and structurally similar duplicate code fragments.

6. The method according to claim 5, characterized in that, Based on the semantic vector and the hash value, all third subsequences are clustered to obtain multiple clusters, including: For any two third subsequences, if the hash values ​​of the two third subsequences are the same, then the two third subsequences are assigned to the same cluster; For several third subsequences with different hash values, multiple clusters are obtained by clustering the several third subsequences according to their semantic vectors.

7. The method according to claim 5, characterized in that, The method further includes: If the total number of sequences in all third subsequences is greater than the preset number of sequences, then all third subsequences are divided into multiple batches. For any batch containing multiple third subsequences, the multiple third subsequences are clustered to obtain multiple clusters and a centroid vector for each cluster; the centroid vector is calculated based on the semantic vectors of all third subsequences within the cluster. Clustering the centroid vectors of all clusters in the multiple batches yields multiple classes; each class includes multiple clusters from different batches. If the number of classes is greater than the preset number of classes, the centroid vector of the class is used as a new third subsequence, and the step of dividing all third subsequences into multiple batches is repeated until the number of classes is no greater than the preset number of classes. If the number of classes is not greater than the preset number of classes, then stop the loop and take the currently obtained classes as the final clustering result; For any one of the aforementioned classes, all third subsequences of that class are identified as functionally equivalent and structurally similar duplicate code fragments.

8. A system for extracting repetitive code snippets, characterized in that, The system includes: The syntax tree generation module is used to acquire multiple code files and convert the source code of the multiple code files into syntax trees respectively; each syntax tree includes at least one function definition node of the corresponding code file. The second function call statement generation module is used to extract a first function call statement from any function definition node, and replace the parameter values ​​in the first function call statement with the type placeholders corresponding to the parameter values ​​to obtain a second function call statement; the first function call statement includes a function code snippet, a file path, and a function name. The statement grouping module is used to group all second function call statements corresponding to the at least one function definition node to generate multiple first groups; the second function call statements in each first group belong to the same function in the same code file; The statement combination module is used to combine at least two adjacent second function call statements in any first group to obtain a first sequence; A sequence signature generation module is used to concatenate the command signatures of all second function call statements contained in any first sequence to generate a sequence signature for the first sequence; the command signature includes the command type and key parameter template of the second function call statement; The duplicate code fragment determination module is used to group all first sequences based on sequence signatures to generate multiple second groups, and for any second group, determine all first sequences contained in the second group as duplicate code fragments; After receiving the second function call statement, it also includes: Determine the function name of the second function call statement; The command type of the second function call statement is determined based on the function name; Extract the key parameter template from the second function call statement; The command signature of the second function call statement is generated based on the command type and the key parameter template.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method for extracting repeated code fragments as described in any one of claims 1 to 7.

Images