Code clone detection method, system, device, medium and product

Abstract

This application discloses a code clone detection method, system, device, medium, and product, relating to the field of software engineering. The method comprises obtaining code segments to be detected from a code repository, inputting all of the code segments into a trained clone code detection model, and outputting first code representation vectors corresponding to the code segments to be detected. Two first code representation vectors are then randomly selected as code pairs, and code pairs whose semantic distance is less than a preset threshold are marked as cloned code. This application can improve the accuracy of code clone detection.

Description

Technical Field

[0001] The present application relates to the field of software engineering, and in particular to a code clone detection method, system, device, medium and product. Background Art

[0002] To speed up the development process, developers often rely on frameworks to provide infrastructure and common functions to avoid duplication of work. However, this practice leads to the phenomenon of code cloning, that is, the appearance of identical or similar code segments in the software system.

[0003] Research has shown that large numbers of code clones can negatively impact software systems. For example, reusing code segments containing undetected defects can cause defects to propagate, thereby reducing the stability of information systems. Furthermore, if code clones in a system are not properly managed, the codebase may bloat unnecessarily, leading to code redundancy and increased maintenance costs. Therefore, identifying code clones in software systems is an important research topic in software engineering.

[0004] Code clones can be divided into four types based on the degree of similarity between code segments:

[0005] Type-1: Code pairs that differ only in comments and whitespace characters;

[0006] Type-2: In addition to the differences in Type-1, there are also code pairs with different variable names and constant values;

[0007] Type-3: In addition to the differences between Type-1 and Type-2, it also includes code pairs that add or delete statements;

[0008] Type-4: Code pairs with different syntax but identical functionality.

[0009] Traditional code clone detection techniques based on text similarity perform well for Type-1 to Type-3 code clones, as these clones share high textual similarity. However, when it comes to Type-4 clones, due to their syntactical differences, traditional code clone detection methods struggle to effectively identify these code segments that essentially share the same functionality but differ in form. Summary of the Invention

[0010] The purpose of this application is to provide a code clone detection method, system, device, medium and product, which can improve the accuracy of code clone detection.

[0011] To achieve the above objectives, this application provides the following solutions:

[0012] In a first aspect, the present application provides a code clone detection method, comprising:

[0013] Obtain code segments to be detected from a code repository to be detected, input all the code segments to be detected into a trained clone code detection model, and output a first code representation vector corresponding to the code segments to be detected;

[0014] arbitrarily selecting two of the first code representation vectors as a code pair, and marking the code pair whose semantic distance is less than a preset threshold as a clone code;

[0015] The clone code detection model is trained in the following way:

[0016] Randomly sampling from a code clone detection dataset, obtaining an equal number of source code segments and first target code segments that have a unique correspondence relationship, wherein each pair of the source code segment and the first target code segment that have the unique correspondence relationship is a clone of each other, and the unique correspondence relationship refers to semantic equivalence or syntactically different but functionally equivalent;

[0017] All the source code segments and the first target code segment are input into an initial clone code detection model, and the initial clone code detection model is trained using a multi-task learning loss function to obtain a clone code detection model. The multi-task learning loss function includes a contrastive learning loss function and a translation enhancement learning loss function.

[0018] In a second aspect, the present application provides a code clone detection system, comprising:

[0019] an acquisition module, configured to acquire code segments to be detected from a code repository to be detected, input all the code segments to be detected into a trained clone code detection model, and output a first code representation vector corresponding to the code segments to be detected;

[0020] a marking module, configured to arbitrarily select two of the first code representation vectors as a code pair, and mark the code pair whose semantic distance is less than a preset threshold as a clone code;

[0021] The marking module is further used to train the clone code detection model in the following manner:

[0022] Randomly sampling from a code clone detection dataset, obtaining an equal number of source code segments and first target code segments that have a unique correspondence relationship, wherein each pair of the source code segment and the first target code segment that have the unique correspondence relationship is a clone of each other, and the unique correspondence relationship refers to semantic equivalence or syntactically different but functionally equivalent;

[0023] All the source code segments and the first target code segment are input into an initial clone code detection model, and the initial clone code detection model is trained using a multi-task learning loss function to obtain a clone code detection model. The multi-task learning loss function includes a contrastive learning loss function and a translation enhancement learning loss function.

[0024] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the above-described code clone detection methods.

[0025] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the above-described code clone detection methods.

[0026] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned code clone detection methods.

[0027] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0028] The present application provides a code clone detection method, system, device, medium, and product. The method obtains code segments to be detected from a code repository to be detected, inputs all the code segments to be detected into a trained clone code detection model, and outputs first code representation vectors corresponding to the code segments to be detected. Two first code representation vectors are arbitrarily selected as code pairs, and code pairs with a semantic distance less than a preset threshold are marked as clone codes. The clone code detection model is trained by randomly sampling from a code clone detection dataset to obtain an equal number of source code segments and first target code segments with unique corresponding relationships, wherein each pair of source code segments and first target code segments with unique corresponding relationships are clone codes of each other, and the unique corresponding relationship refers to semantic equivalence or syntactically different but functionally identical. All source code segments and the first target code segment are input into an initial clone code detection model, and the initial clone code detection model is trained using a multi-task learning loss function to obtain a clone code detection model. The multi-task learning loss function includes a contrastive learning loss function and a translation enhancement learning loss function. This application combines a contrastive learning loss function with a translation-enhanced learning loss function. This loss function learns the semantic features of source code segments, bringing cloned code closer together in the embedding space. The translation-enhanced learning loss function then learns to translate each source code segment into its corresponding cloned code, capturing the fine-grained semantic relationships between code segments. This combination improves the accuracy of the first code representation vector, resulting in more accurate code clone detection results. This ensures that the cloned code detection model can accurately translate the code segment being detected into a semantically equivalent first code representation vector. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 This is an application environment diagram of a code clone detection method in an embodiment of the present application;

[0031] Figure 2 A flowchart of a code clone detection method provided in one embodiment of the present application;

[0032] Figure 3 A flowchart of a code clone detection method provided in another embodiment of the present application;

[0033] Figure 4A schematic diagram of the functional modules of a code clone detection system provided in one embodiment of the present application;

[0034] Figure 5 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0036] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0037] The code clone detection method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the source code segment to be processed and the first target code segment to the server 104. After the server 104 receives the source code segment to be processed and the first target code segment, for the source code segment to be processed and the first target code segment, the server 104 inputs all the source code segments and the first target code segment into the initial clone code detection model, and trains the initial clone code detection model through the multi-task learning loss function to obtain a clone code detection model. The server 104 obtains the code segment to be detected from the code repository to be detected, and inputs all the code segments to be detected into the clone code detection model, and outputs the first code representation vector corresponding to the code segment to be detected; arbitrarily selects two first code representation vectors as code pairs, and marks the code pairs whose semantic distance is less than a preset threshold as clone code. The server 104 can feedback the obtained clone code to the terminal 102. In addition, in some embodiments, the code clone detection method can also be implemented independently by the server 104 or the terminal 102. For example, the terminal 102 can directly process the source code segment to be processed, or the server 104 can obtain the source code segment to be processed from the data storage system and process the source code segment to be processed.

[0038] Terminal 102 may include, but is not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices. Server 104 may be implemented as a standalone server or a server cluster consisting of multiple servers, or may be a cloud server.

[0039] In an exemplary embodiment, Figure 2 As shown, a code clone detection method is provided, which is executed by a computer device, specifically a computer device such as a terminal or a server, or a terminal and a server. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in FIG. 1 is taken as an example to illustrate the method, which includes the following steps S201 to S204. Steps S201 to S204 correspond to Figure 3 The model fine-tuning process and model inference process in . Among them:

[0040] In step S201 , code segments to be detected are obtained from a code repository to be detected, and all the code segments to be detected are input into a trained clone code detection model, which outputs a first code representation vector corresponding to the code segments to be detected.

[0041] In step S202, two of the first code representation vectors are arbitrarily selected as a code pair, and the code pairs whose semantic distance is less than a preset threshold are marked as clone codes.

[0042] The clone code detection model is trained in the following way:

[0043] In step S203, random sampling is performed from the code clone detection dataset to obtain the same number of source code segments and first target code segments that have a unique correspondence relationship. Each pair of source code segments and first target code segments that have a unique correspondence relationship is a clone code of each other.

[0044] Specifically, N pairs of cloned code are randomly found from a code clone detection dataset. In each pair, one code segment is used as the source code segment, and the other is used as the first target code segment. This disclosure uses the open-source code clone detection dataset Google Code Jam. The Google Code Jam dataset contains code in multiple programming languages (such as Python, Java, and C++), covering different programming styles and techniques. The code segments in the dataset are derived from real-world programming competitions, reflecting the code clone issues that programmers may encounter in real-world development. The code segments in the dataset are labeled as clones or non-clones, providing a reliable benchmark for subsequent model training and validation. In the Google Code Jam dataset, different answers to each question are considered Type-4 code clones. Type-4 code clones are code segments that are semantically similar but may have completely different syntactic structures. This type of clone detection is more difficult because it requires the model to have a deep understanding of the code semantics, not just syntactic matching.

[0045] To train the model, we randomly sample multiple sets of data from the Google Code Jam dataset. The goal of this sampling is to construct a dataset containing both cloned and non-cloned code segments for contrastive learning. Contrastive learning is a method that trains a model using positive samples (clone code pairs) and negative samples (non-clone code pairs). This effectively improves the model's ability to identify code clones.

[0046] The source code segments and target code segments have been annotated with clone labels or non-clone labels. These labels will be used for comparative learning of the model to help the model distinguish between clone code pairs and non-clone code pairs.

[0047] In step S204, all source code segments and the first target code segment are input into an initial clone code detection model, and the initial clone code detection model is trained using a multi-task learning loss function to obtain a clone code detection model. The multi-task learning loss function includes a contrastive learning loss function and a translation enhancement learning loss function.

[0048] In one embodiment, step S204 corresponds to Figure 3 The multi-task learning process in step S204 of "training the initial clone code detection model through multi-task learning to obtain a clone code detection model" includes the following sub-steps S2041-S2045:

[0049] S2041. Encode the source code segment using an encoder, and output a second code representation vector corresponding to the source code segment. Each source code segment and the corresponding second code representation vector are attached with the same label, and the label is used to represent the function of the source code segment.

[0050] Specifically, the initial clone code detection model consists of an encoder and a decoder. It is built based on the pre-trained CodeT5p model. CodeT5p is a large-scale, open-source language model that utilizes an encoder-decoder architecture and can operate in different modes, including encoder-only, decoder-only, and encoder-decoder. This architecture enables CodeT5p to support a wide range of code understanding and generation tasks. In the clone code detection task, the model built on CodeT5p can leverage its powerful semantic representation capabilities to extract semantic features from code segments, enabling more accurate code clone detection.

[0051] The labels for source code segments are predefined and represent the functionality or semantic category of the code segment. For example, if a source code segment implements a sorting algorithm, its label might be "sort"; if a code segment implements data search, its label might be "search." These labels are assigned based on the functional characteristics of the code. After being processed by the encoder, source code segments 1 through N are converted into second code representation vectors 1 through N. The second code representation vector is a numerical representation of the semantic and structural features of the source code segment, capturing the core information of the code. The encoder converts the textual form of the code into a vector form that the model can understand and process. Assigning the same label to a source code segment and its corresponding second code representation vector ensures semantic consistency between the two. The source code segment label reflects its functionality, while the second code representation vector is a numerical representation of the semantic features of the source code segment. By assigning the same label, the model can learn the direct correspondence between the second code representation vector and the source code segment's functionality. This consistency enables the model to better understand the semantics of the code.

[0052] S2042: Pair the source code segments in pairs to obtain multiple code pairs. Determine whether a code pair is a clone code pair based on the matching of the labels of the source code segments. If the labels match, mark the code pair as a clone code pair; otherwise, mark the code pair as a non-clone code pair.

[0053] Specifically, in a batch, for N source code segments, each source code segment Ci is processed by the encoder to obtain a second code representation vector Ei. This Ei is obtained by linearly transforming the [CLS] tag output of the encoder. Each source code segment also has a label. The extracted source code segments are paired to form multiple code pairs. For the case of n code segments, For each code pair, the label information of the two source code segments is compared. If the labels of the two source code segments completely match, the code pair is marked as a clone code pair, otherwise, it is marked as a non-clone code pair.

[0054] S2043. Calculate the contrastive learning loss of the clone code pair according to the contrastive learning loss function.

[0055] The contrastive learning loss function is expressed as follows:

[0056] ;

[0057] in, represents contrastive learning loss; N represents the number of source code segments in a training batch; and A second code representation vector representing the correspondence between any two source code segments; express The positive sample of source code segments that form clone code pairs; express and The cosine similarity between Used for calculation and its positive sample The similarity index of Used for calculation The sum of similarity indices with all other source code segments; represents a hyperparameter used to control the distribution, and its empirical value is set to 0.11; the goal of the contrastive learning loss function is to make each source code segment Its positive sample The similarity of has a higher relative importance among all code segments.

[0058] S2044. All second code representation vectors are decoded by a decoder, and a second target code segment corresponding to the second code representation vector is output. The translation enhancement loss of the second target code segment relative to the first target code segment is calculated using a translation enhancement learning loss function.

[0059] Specifically, an encoder-decoder architecture is used to handle code translation tasks. A source code segment Ci is input to the encoder, which then encodes it and outputs a hidden state, a second code representation vector, that contains both semantic and structural information about the input source code segment. A translation instruction is then spliced into the decoder input. This instruction could be something like "translate code to Java" or "translate code to Python," depending on the language of the second target code segment. This instruction instructs the decoder on how to decode the encoder's hidden state into the second target code segment.

[0060] The first target code segment is the original code segment, and the second target code segment is the translated code segment. The first target code segment is the original code that the model attempts to translate into the second target code segment through the translation task. The first target code segment is the correct answer or goal summarized by the model learning process. The difference between the second target code segment and the first target code segment is minimized using the translation reinforcement learning loss function. The translation reinforcement learning loss function is expressed as follows:

[0061] ;

[0062] Where N represents the number of first target code segments in a training batch; T represents the length of the first target code segment, that is, the total number of tokens in the first code segment; represents the token the model attempts to generate at time step t; Indicates that a tag sequence has been generated in the first target code segment before time step t, represents the second code representation vector generated by the encoder, Indicates that given a generated tag sequence and the second code represents the vector Under the condition of probability.

[0063] The translation enhancement loss function is based on the standard causal language model design. The translation enhancement loss function calculates the token that the model attempts to generate at time step t (the predicted token). The actual tth token in the first target code segment The difference between. is not the actual token in the first target code segment. By minimizing the translation enhancement loss function, the model learns how to better represent the vector based on the input second target code. and the generated sequence , to predict the next token During training, the model's parameters are adjusted using optimization algorithms, such as gradient descent, to minimize the loss function The model parameters are updated based on the gradient to reduce the difference between the predicted second target code segment and the actual first target code segment. Through multiple iterations, the model gradually learns how to better replace the second target code segment with a code segment that is semantically and structurally similar to the first target code segment.

[0064] Because the loss function quantifies the gap between the model's prediction and the actual result, the input data is passed through the model in a complete forward propagation to calculate the model's output. The loss function calculates the gap between the predicted and actual outputs to obtain the loss value. The gradient of the loss function with respect to the model parameters—that is, the partial derivative of the loss function with respect to each model parameter—is calculated. These gradients indicate the direction in which the loss function rises fastest in parameter space. Therefore, to reduce the loss, the parameters should be updated in the opposite direction of the gradient. Based on the calculated gradients, an optimization algorithm, such as gradient descent, is used to update the model parameters. The optimization algorithm determines the step size and direction of the parameter update based on hyperparameters such as the gradient and the learning rate. The aforementioned steps of forward propagation, loss calculation, backward propagation, and parameter update are repeated until the loss converges to a low level or the preset number of iterations is reached. During this process of adjusting model parameters using the loss value, the model parameters are continuously updated, the loss value gradually decreases, and the model performance gradually improves, ultimately resulting in a model that can accurately predict.

[0065] Through the translation enhancement task, the model is forced to learn how to convert code snippets from representation vectors to target code, which helps enhance the semantic and structural representation of the code snippets. By splicing translation instructions into the decoder input, the model can learn to translate the same code representation into different or the same language, thus supporting cross-language code clone detection.

[0066] It's important to note that the encoder converts the input source code segment into a fixed-length second code representation vector. This vector captures the semantics and structure of the code, regardless of the programming language. In other words, the encoder extracts the core meaning of the code regardless of the input language. The decoder uses this vector to generate a second target code segment, which can be in a different language or an equivalent code in the same language. For example, if the input code is Java, the target code can be Python, or still Java but in a different style.

[0067] Translation instructions (such as "translate code to Python") are spliced into the decoder input, guiding the decoder on how to decode the encoder's hidden state into the target code segment. This instruction tells the decoder which language to generate code but is not involved in the loss calculation or the encoder's encoding process. The loss function only calculates the difference between the second target code segment generated by the decoder and the first target code segment. The loss function focuses solely on whether the second target code segment is semantically equivalent to the first target code segment, regardless of the generated language. The vector representation generated by the encoder is programming language-agnostic and captures only the semantics and structure of the code. This means that the same code logic will produce the same vector representation in different languages. Based on the translation instructions and the same vector representation, the decoder generates code in different languages. These codes, despite being in different languages, are functionally equivalent. The model learns how to generate equivalent code based on the semantics and structure of the input code segment, rather than relying on a specific programming language. As a result, the model can translate the same code representation into different or the same language without affecting the resulting representation. This approach is suitable for addressing code cloning between different programming languages, as it allows the model to translate code between different languages while maintaining semantic consistency. For example, it can detect functionally equivalent code snippets in different languages, or translate code from one language to another while maintaining semantic consistency.

[0068] In one embodiment, the second target code segment is input into the encoder to generate a representation vector, which is then input into the decoder to generate a third target code segment, and the difference between the third target code segment and the second target code segment is calculated using a translation reinforcement learning loss function.

[0069] Specifically, for the source code segment Ci and the first target code segment Cj, they are functionally equivalent code clone pairs, which means they perform the same task but may use different programming languages or coding styles. Step S2024 performs the forward translation task: the training model translates the source code segment Ci into the second target code segment , which involves inputting the source code segment Ci into the encoder, generating a second code representation vector, and then inputting the second code representation vector into the decoder to generate the second target code segment , Cj is calculated by translation enhancement learning loss function and The reverse translation task is to convert the second target code segment Input the encoder to generate a representation vector, which is then input into the decoder to generate the third target code. , and the third target code is calculated by the translation reinforcement learning loss function and the second target code segment The difference between.

[0070] The translation reinforcement learning loss functions of the forward translation task and the back translation task are combined and the model is trained simultaneously. This ensures that the model learns the conversion between code segments from two directions, which helps to improve the generalization ability and robustness of the model.

[0071] S2045. A multi-task learning loss is obtained by using a contrastive learning loss and a translation enhancement learning loss. Model parameters of the initial clone code detection model are continuously iterated using the multi-task learning loss, so that the model continuously learns how to capture the semantics of the source code segment, thereby obtaining the clone code detection model.

[0072] The model parameters of the initial clone code detection model are continuously updated and iterated through multi-task learning loss to minimize the multi-task learning loss function. Each iterative learning includes the encoding process of the encoder, the decoding process of the decoder, the calculation of the contrastive learning loss, the calculation of the translation enhancement loss, and the optimization of this iteration. The contrastive learning loss function and the translation enhancement learning loss function are combined to obtain the multi-task learning loss function, which is expressed as follows:

[0073] ;

[0074] in, represents a hyperparameter that controls the balance between contrastive learning loss and translation enhancement loss. =1 means that the contrastive learning loss accounts for 0.1 of the final loss.

[0075] In the code clone detection task, each set of data (code pair) is trained using a combination of contrastive learning loss and translation enhancement loss to update the model's parameters. The core of contrastive learning loss is to distinguish the similarities between positive and negative pairs. It helps the model learn the semantic features of source code segments by shortening the distance between cloned code pairs in feature space while extending the distance between non-cloned code pairs. The translation loss focuses on optimizing the model's generative capabilities, ensuring that it can accurately translate source code segments into semantically equivalent target code segments. This combination not only improves the model's ability to identify code clones, but also strengthens its deep understanding of code semantics. Through this training approach, the model can more accurately capture code semantics in complex code environments, effectively identifying cloned code pairs. This paper combines a contrastive learning loss function with a translation-enhanced learning loss function. This loss function learns the semantic features of source code segments, bringing cloned code closer together in the embedding space. The translation-enhanced learning loss function then learns to translate each source code segment into its corresponding cloned code, capturing the fine-grained semantic relationships between code segments. This combination improves the accuracy of the first code representation vector, resulting in more accurate code clone detection results. This ensures that the cloned code detection model can accurately translate the code segment being detected into a semantically equivalent first code representation vector.

[0076] In one embodiment, the step S201 of "obtaining the code segment to be detected from the code repository to be detected" includes the following sub-steps S2011-S2013:

[0077] S2011. Read the code file to be tested from the code repository to be tested;

[0078] S2012: passing the read content of the code file to be detected to a pre-specified AST parsing tool, and converting the code file to be detected into a structured tree representation by the AST parsing tool;

[0079] S2013. Find the function definition nodes of the code file to be detected by traversing the tree representation, and extract the code segment corresponding to each function definition node as the function-level code segment to be detected.

[0080] Specifically, a code repository is where the code is stored. It can be a local file system, a version control system (such as Git), or a remote code repository (such as GitHub, Bitbucket, etc.). The code files to be tested are files that need to be analyzed to find cloned code. They can be source code files, header files, or any other files containing executable code. An AST parsing tool is a software tool that can parse code files and generate an abstract syntax tree. This tree structure represents the grammatical structure of the code, including all statements, expressions, and the relationships between them. Before starting the analysis, you need to select an AST parsing tool for an appropriate code language. For example, for Python code, you can use the ast module; for Java code, you can use JavaParser.

[0081] An AST parsing tool reads the code file, parses the code according to grammatical rules, and generates a tree structure. In this tree structure, each node represents a code construct (such as a function definition, loop, conditional statement, etc.). This structured representation makes code analysis easier because it allows the code to be broken down into its basic components and the relationships between them to be examined. Once the code AST is obtained, the tree structure can be traversed to find specific nodes. In code clone detection, function definition nodes represent function or method definitions in the code. Each function definition node contains the function name, parameters, return type, and function body.

[0082] For each function definition node found, the corresponding code segment is extracted. This code segment includes the entire function definition, starting from the function declaration and ending with the function body. The function-level code segment to be tested is the basic unit for clone detection. By comparing the code segments of different function definition nodes, potential cloned code can be identified. Through steps S2011-S2013, the code file to be tested is converted into a structured tree representation, and then the function-level code segment is extracted to prepare for subsequent clone detection. This structured representation makes code analysis more systematic and automated, helping to improve the accuracy and efficiency of code clone detection.

[0083] In step S202, two first code representation vectors are arbitrarily selected as code pairs, and code pairs whose semantic distance is less than a preset threshold are marked as clone codes.

[0084] In one embodiment, step S202 includes the following sub-steps S2021-S2023:

[0085] S2021. Calculate the Euclidean distance between any two first code representation vectors, and measure the similarity between any two first code representation vectors using the Euclidean distance calculation result.

[0086] Euclidean distance is used to measure the straight-line distance between two points in a plane or space. The calculation formula of Euclidean distance is as follows:

[0087] ;

[0088] in, and They represent the coordinate values of point 1 and point 2 in the i-th dimension respectively. In the code clone detection process, the representation vector of each code segment can be regarded as a point. Calculating the Euclidean distance between two points can measure the similarity between the two code segments.

[0089] S2022. Classify similar first code representation vectors into the same category based on the Euclidean distance calculation result.

[0090] Specifically, the calculated Euclidean distance can be used to measure the similarity between any two first code representation vectors. The smaller the distance, the more similar the two code segments are. This is because a smaller distance means that the two vectors are closer in multidimensional space, making their semantics and structures more likely to be similar. Based on the calculated Euclidean distance, similar code representation vectors can be classified into the same category. This usually involves setting a threshold. When the Euclidean distance between two code segments is less than this threshold, they are considered similar and classified into the same category.

[0091] S2023: Perform pairwise distance comparison on the first code representation vectors within each category, and mark the code pairs whose semantic distance between the first code representation vectors within each category is less than a preset threshold as the clone code pairs.

[0092] Specifically, within each category, the first code representation vectors are compared pairwise. For each code pair within a category, the semantic distance between them is calculated. If the semantic distance between a code pair is less than a preset threshold, the pair is marked as a clone. The semantic distance can be Euclidean distance or other distance types, such as cosine similarity or Jaccard similarity.

[0093] The first code representation vector is clustered by calculating the Euclidean distance, and then the distance comparison is performed between each category. This method takes into account both the structure of the code and the similarity at the text level, making the result more accurate.

[0094] After training the model in a multi-task framework, the decoder is usually discarded and only the encoder is retained. This is because the encoder has learned how to convert the input code segment into a fixed-length vector representation that captures the semantic and structural information of the code segment. For N samples, the encoder is used to calculate the vector representation of each sample. Then, the cosine distance between these vectors is calculated to determine whether the two code segments are cloned code segments. Cosine similarity is used to measure the angle between two vectors. The closer the cosine similarity value is to 1, the more similar the two vectors are. The formula for calculating cosine similarity is as follows:

[0095] ;

[0096] Among them, A and B are two vectors to be compared, and the value of cosine similarity ranges from -1 to 1. The closer to 1, the more similar the vectors are. The preset threshold is an empirical value used to determine whether the similarity between two code representation vectors is high enough to mark them as clone code pairs. If the cosine similarity of two code segments is greater than this preset threshold, then the two code segments are considered semantically similar and can therefore be marked as clone code pairs. This threshold is usually determined based on experiments and may need to be adjusted according to the specific application scenario and dataset. In actual operation, a cosine similarity greater than 0.5 or 0.6 is generally considered to be similar, but this value can be adjusted according to the specific situation.

[0097] Based on the same inventive concept, embodiments of the present application also provide a system for implementing the aforementioned code clone detection method. The solution provided by this system is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more code clone detection system embodiments provided below can be found in the above-described limitations on code clone detection systems and will not be further elaborated here.

[0098] In an exemplary embodiment, Figure 4 As shown, a code clone detection system is provided, including:

[0099] An acquisition module 410 is configured to acquire code segments to be detected from a code repository to be detected, input all of the code segments to be detected into a trained clone code detection model, and output a first code representation vector corresponding to the code segment to be detected;

[0100] a marking module 420 configured to arbitrarily select two of the first code representation vectors as a code pair, and mark the code pair having a semantic distance less than a preset threshold as a clone code;

[0101] The marking module is further used to train the clone code detection model in the following manner:

[0102] Randomly sampling from a code clone detection dataset, obtaining an equal number of source code segments and first target code segments that have a unique correspondence relationship, wherein each pair of the source code segment and the first target code segment that have the unique correspondence relationship is a clone of each other, and the unique correspondence relationship refers to semantic equivalence or syntactically different but functionally equivalent;

[0103] All the source code segments and the first target code segment are input into an initial clone code detection model, and the initial clone code detection model is trained using a multi-task learning loss function to obtain a clone code detection model. The multi-task learning loss function includes a contrastive learning loss function and a translation enhancement learning loss function.

[0104] As an optional implementation, the initial clone code detection model includes an encoder and a decoder.

[0105] As an optional implementation, the code clone detection system is further used to:

[0106] encoding the source code segment by an encoder, and outputting a second code representation vector corresponding to the source code segment, wherein each source code segment and the corresponding second code representation vector are attached with the same label, and the label is used to represent the function of the source code segment;

[0107] Pairing all the source code segments in pairs to obtain a plurality of code pairs, and determining whether the code pairs are clone code pairs based on matching of the labels of the source code segments; if the labels match, marking the code pairs as clone code pairs; otherwise, marking the code pairs as non-clone code pairs;

[0108] Calculating the contrastive learning loss of the clone code pair according to the contrastive learning loss function;

[0109] Decoding all the second code representation vectors through a decoder, outputting a second target code segment corresponding to the second code representation vector, and calculating a translation enhancement loss of the second target code segment relative to the first target code segment using a translation enhancement learning loss function;

[0110] A multi-task learning loss is obtained by using the contrastive learning loss and the translation enhancement learning loss. Model parameters of the initial clone code detection model are continuously iterated using the multi-task learning loss, so that the model continuously learns how to capture the semantics of the source code segment, thereby obtaining the clone code detection model.

[0111] As an optional implementation, in the aspect of obtaining the code segment to be detected from the code repository to be detected, the acquisition module 410 is specifically configured to:

[0112] Read the code file to be tested from the code repository to be tested;

[0113] Passing the read content of the code file to be detected to a pre-specified AST parsing tool, and converting the code file to be detected into a structured tree representation through the AST parsing tool;

[0114] By traversing the tree representation, the function definition nodes of the code file to be detected are found, and the code segment corresponding to each function definition node is extracted as the code segment to be detected at the function level.

[0115] As an optional implementation, the marking module 420 is specifically configured to:

[0116] Calculating the Euclidean distance between any two of the first code representation vectors, and measuring the similarity between any two of the first code representation vectors using the Euclidean distance calculation result;

[0117] Classifying similar first code representation vectors into the same category according to the Euclidean distance calculation result;

[0118] The first code representation vectors within each category are compared with each other in distance, and the code pairs whose semantic distance between the first code representation vectors within each category is less than a preset threshold are marked as clone code pairs.

[0119] As an optional implementation, the initial clone code detection model is constructed based on a pre-trained CodeT5p model.

[0120] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store source code segments and first target code segments. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a code clone detection method is implemented.

[0121] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0122] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0123] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0124] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0125] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0126] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0127] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0128] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A code clone detection method, characterized in that: The code clone detection method includes: Obtain code segments to be detected from a code repository to be detected, input all the code segments to be detected into a trained clone code detection model, and output a first code representation vector corresponding to the code segments to be detected; arbitrarily selecting two of the first code representation vectors as a code pair, and marking the code pair whose semantic distance is less than a preset threshold as a clone code; The clone code detection model is trained in the following way: Randomly sampling from a code clone detection dataset, obtaining an equal number of source code segments and first target code segments that have a unique correspondence relationship, wherein each pair of the source code segment and the first target code segment that have the unique correspondence relationship is a clone of each other, and the unique correspondence relationship refers to semantic equivalence or syntactically different but functionally equivalent; inputting all the source code segments and the first target code segment into an initial clone code detection model, and training the initial clone code detection model using a multi-task learning loss function to obtain a clone code detection model, wherein the multi-task learning loss function includes a contrastive learning loss function and a translation enhancement learning loss function; The initial clone code detection model includes an encoder and a decoder; The initial clone code detection model is trained using a multi-task learning loss function to obtain a clone code detection model, including: encoding the source code segment by an encoder, and outputting a second code representation vector corresponding to the source code segment, wherein each source code segment and the corresponding second code representation vector are attached with the same label, and the label is used to represent the function of the source code segment; Pairing all the source code segments in pairs to obtain a plurality of code pairs, and determining whether the code pairs are clone code pairs based on matching of the labels of the source code segments; if the labels match, marking the code pairs as clone code pairs; otherwise, marking the code pairs as non-clone code pairs; Calculating the contrastive learning loss of the clone code pair according to the contrastive learning loss function; Decoding all the second code representation vectors through a decoder, outputting a second target code segment corresponding to the second code representation vector, and calculating a translation enhancement loss of the second target code segment relative to the first target code segment using a translation enhancement learning loss function; A multi-task learning loss is obtained by using the contrastive learning loss and the translation enhancement learning loss. Model parameters of the initial clone code detection model are continuously iterated using the multi-task learning loss, so that the model continuously learns how to capture the semantics of the source code segment, thereby obtaining the clone code detection model.

2. The code clone detection method according to claim 1, characterized in that: The step of obtaining the code segment to be detected from the code repository to be detected includes: Reading the code file to be detected from the code repository to be detected; Passing the read content of the code file to be detected to a pre-specified AST parsing tool, and converting the code file to be detected into a structured tree representation through the AST parsing tool; By traversing the tree representation, the function definition nodes of the code file to be detected are found, and the code segment corresponding to each function definition node is extracted as the code segment to be detected at the function level.

3. The code clone detection method according to claim 1, characterized in that: The arbitrarily selecting two of the first code representation vectors as a code pair, and marking the code pair having a semantic distance less than a preset threshold as a clone code, includes: Calculating the Euclidean distance between any two of the first code representation vectors, and measuring the similarity between any two of the first code representation vectors using the Euclidean distance calculation result; Classifying similar first code representation vectors into the same category according to the Euclidean distance calculation result; Perform pairwise distance comparison on the first code representation vectors within each of the categories, and mark the code pairs whose semantic distance between the first code representation vectors within each of the categories is less than the preset threshold as the clone code pairs.

4. The code clone detection method according to claim 1, wherein: The initial clone code detection model is built based on the pre-trained CodeT5p model.

5. A code clone detection device, characterized in that: The code clone detection device comprises: an acquisition module, configured to acquire code segments to be detected from a code repository to be detected, input all the code segments to be detected into a trained clone code detection model, and output a first code representation vector corresponding to the code segments to be detected; a marking module, configured to arbitrarily select two of the first code representation vectors as a code pair, and mark the code pair whose semantic distance is less than a preset threshold as a clone code; The marking module is further used to train the clone code detection model in the following manner: Randomly sampling from a code clone detection dataset, obtaining an equal number of source code segments and first target code segments that have a unique correspondence relationship, wherein each pair of the source code segment and the first target code segment that have the unique correspondence relationship is a clone of each other, and the unique correspondence relationship refers to semantic equivalence or syntactically different but functionally equivalent; inputting all the source code segments and the first target code segment into an initial clone code detection model, and training the initial clone code detection model using a multi-task learning loss function to obtain a clone code detection model, wherein the multi-task learning loss function includes a contrastive learning loss function and a translation enhancement learning loss function; The initial clone code detection model includes an encoder and a decoder; In terms of training the initial clone code detection model using a multi-task learning loss function to obtain a clone code detection model, the labeling module is specifically used to: encoding the source code segment by an encoder, and outputting a second code representation vector corresponding to the source code segment, wherein each source code segment and the corresponding second code representation vector are attached with the same label, and the label is used to represent the function of the source code segment; Pairing all the source code segments in pairs to obtain a plurality of code pairs, and determining whether the code pairs are clone code pairs based on matching of the labels of the source code segments; if the labels match, marking the code pairs as clone code pairs; otherwise, marking the code pairs as non-clone code pairs; Calculating the contrastive learning loss of the clone code pair according to the contrastive learning loss function; Decoding all the second code representation vectors through a decoder, outputting a second target code segment corresponding to the second code representation vector, and calculating a translation enhancement loss of the second target code segment relative to the first target code segment using a translation enhancement learning loss function; A multi-task learning loss is obtained by using the contrastive learning loss and the translation enhancement learning loss. Model parameters of the initial clone code detection model are continuously iterated using the multi-task learning loss, so that the model continuously learns how to capture the semantics of the source code segment, thereby obtaining the clone code detection model.

6. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the code clone detection method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the code clone detection method according to any one of claims 1 to 4 are implemented.

8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the code clone detection method according to any one of claims 1 to 4 are implemented.

Images