Code duplicate checking method and device, computer equipment and storage medium

Abstract

The invention relates to a code duplicate checking method and device, computer equipment and a storage medium. The method belongs to the technical field of duplicate checking retrieval, and comprises the following steps: acquiring a first vector representation of a code file to be subjected to duplicate checking, and determining a correlation value between the first vector representation and each second vector representation in a target vector database; and obtaining the target code file corresponding to the second vector representation with the correlation value greater than the correlation threshold. And according to the target code file and the to-be-duplicated code file, determining a code duplicate checking result of the to-be-duplicated code file. When code duplicate checking is carried out, duplicate checking is carried out in a vector representation mode, the duplicate checking speed is higher, and calculation of correlation values can be completed within shorter time. And obtaining the target code file based on the correlation value. And then according to the target code file and the code file to be subjected to duplicate checking, the code duplicate checking result of the code file to be subjected to duplicate checking is determined, and the accuracy of the duplicate checking result is effectively improved based on a dual duplicate checking mechanism.

Description

Technical Field

[0001] The present application relates to the technical field of duplicate code detection and retrieval, and in particular to a code duplicate code detection method, device, computer equipment and storage medium. Background Art

[0002] As software projects continue to grow in size and complexity, code duplication and redundancy are becoming more and more serious, leading to rising maintenance costs. The use of open source code allows developers to reuse existing code, but this also brings potential code infringement and plagiarism issues, which has promoted the development of code duplication detection technology.

[0003] Code duplication detection, commonly known as code similarity detection or code clone detection, is a technique used to identify and manage duplicate code snippets in computer programs. Research and applications in this area are gaining more and more attention, especially in software engineering, maintenance, and code review processes.

[0004] The initial code duplication checking technology was mainly based on text comparison, such as string matching algorithms (such as the KMP algorithm). These methods are simple and easy to implement, but are not sensitive enough to structure and semantics, resulting in inaccurate results of existing code duplication checking methods. Summary of the invention

[0005] Based on this, it is necessary to provide a code duplication checking method, device, computer equipment and storage medium that can effectively improve the accuracy of code duplication checking in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for checking for duplicate code. The method comprises:

[0007] Obtaining a first vector representation of the code file to be checked for duplicates, and determining a correlation value between the first vector representation and each second vector representation in the target vector database;

[0008] Acquire a reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and use the acquired reference code file as a target code file;

[0009] According to the target code file and the code file to be checked for duplicate code, the code duplication checking result of the code file to be checked for duplicate code is determined.

[0010] In one embodiment, determining a correlation value between the first vector representation and each second vector representation in the target vector database includes:

[0011] determining a similarity between the first vector representation and each second vector representation in a target vector database;

[0012] Select the maximum similarity and the minimum similarity from each similarity;

[0013] A correlation value between the first vector representation and each second vector representation in the target vector database is determined according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database.

[0014] In one embodiment, determining a correlation value between the first vector representation and each second vector representation in the target vector database according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database includes:

[0015] The difference between the maximum similarity and the minimum similarity is taken as the first value;

[0016] For each second vector representation, taking the difference between the similarity between the first vector representation and the second vector representation and the minimum similarity as the second value;

[0017] The ratio of the first value to the second value is taken as the correlation value between the first vector representation and the second vector representation.

[0018] In one embodiment, obtaining a first vector representation of a code file to be checked for duplicate content includes:

[0019] Preprocess the code file to be checked for duplicate content to obtain a preprocessed code file;

[0020] The preprocessed code file is encoded based on the language analysis model to obtain a first vector representation of the code file to be checked for duplicates.

[0021] In one embodiment, the method further comprises:

[0022] Obtain the first code type of the code file to be checked for duplicates;

[0023] A target vector database is selected from the candidate vector database according to the first code type and the second code type of the reference code file corresponding to each second vector representation in the candidate vector database; wherein the second code type of the reference code file corresponding to each second vector representation in each candidate vector database is the same; and the second code type of the reference code file corresponding to each second vector representation in the target vector database is the same as the first code type.

[0024] In one embodiment, determining the code duplication check result of the code file to be checked for duplicate content according to the target code file and the code file to be checked for duplicate content includes:

[0025] Obtain a first AST representation of the code file to be checked for duplicate content and a second AST representation of the target code file;

[0026] Determine a code duplication check result of the code file to be checked for duplicate content based on the first AST representation and the second AST representation.

[0027] In a second aspect, the present application also provides a code duplication checking device. The device comprises:

[0028] A first acquisition module is used to acquire a first vector representation of the code file to be checked for duplicates, and determine a correlation value between the first vector representation and each second vector representation in the target vector database;

[0029] A second acquisition module, used to acquire a reference code file corresponding to a second vector representation whose correlation value is greater than a correlation threshold, and use the acquired reference code file as a target code file;

[0030] The determination module is used to determine the code duplication checking result of the code file to be checked for duplicates based on the target code file and the code file to be checked for duplicates.

[0031] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program.

[0032] Obtain a first vector representation of the code file to be checked for duplicates, and determine a correlation value between the first vector representation and each second vector representation in the target vector database;

[0033] Acquire a reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and use the acquired reference code file as a target code file;

[0034] According to the target code file and the code file to be checked for duplicate code, the code duplication checking result of the code file to be checked for duplicate code is determined.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0036] Obtain a first vector representation of the code file to be checked for duplicates, and determine a correlation value between the first vector representation and each second vector representation in the target vector database;

[0037] Acquire a reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and use the acquired reference code file as a target code file;

[0038] According to the target code file and the code file to be checked for duplicate code, the code duplication checking result of the code file to be checked for duplicate code is determined.

[0039] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0040] Obtain a first vector representation of the code file to be checked for duplicates, and determine a correlation value between the first vector representation and each second vector representation in the target vector database;

[0041] Acquire a reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and use the acquired reference code file as a target code file;

[0042] According to the target code file and the code file to be checked for duplicate code, the code duplication checking result of the code file to be checked for duplicate code is determined.

[0043] The above-mentioned code duplicate checking method, device, computer equipment and storage medium obtain the first vector representation of the code file to be checked for duplicates, and determine the correlation value between the first vector representation and each second vector representation in the target vector database. Obtain the reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and use the obtained reference code file as the target code file. According to the target code file and the code file to be checked for duplicates, determine the code duplicate checking result of the code file to be checked for duplicates. When performing code duplicate checking, the present application adopts the form of vector representation for duplicate checking. Compared with the linear scanning of the traditional method, the present application has a faster duplicate checking speed and can complete the calculation of the correlation value in a shorter time. And based on the correlation value, obtain the target code file. Then according to the target code file and the code file to be checked for duplicates, determine the code duplicate checking result of the code file to be checked for duplicates, based on the double duplicate checking mechanism, effectively improve the accuracy of the duplicate checking result. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 An application environment diagram of the code duplication checking method provided in this embodiment;

[0045] Figure 2 A schematic diagram of the flow chart of the first code duplication checking method provided in this embodiment;

[0046] Figure 3 A schematic diagram of a flow chart for determining a correlation value between a first vector representation and each second vector representation in a target vector database provided in this embodiment;

[0047] Figure 4 A schematic diagram of a process for determining a code duplication check result for a code file to be checked for duplicate content provided in this embodiment;

[0048] Figure 5 A schematic diagram of the flow chart of the second code duplication checking method provided in this embodiment;

[0049] Figure 6 A structural block diagram of a code duplication checking device provided in this embodiment;

[0050] Figure 7 This is a diagram of the internal structure of the computer device provided in this embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The vehicle status analysis method provided in the embodiment of the present application can be executed by a terminal device or by a server. That is, the method can be applied to Figure 1 In the application environment shown. Taking the application to the server as an example, the server 104 obtains the first vector representation of the code file to be checked for duplicates after receiving the code file to be checked for duplicates sent by the terminal device 102, and determines the correlation value between the first vector representation and each second vector representation in the target vector database. The reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold is obtained, and the obtained reference code file is used as the target code file; according to the target code file and the code file to be checked for duplicates, the code duplication checking result of the code file to be checked for duplicates is determined, and the code duplication checking result is sent to the terminal device 102.

[0053] The server 104 may be implemented as an independent server or a server cluster composed of multiple servers. The terminal device 102 may be a smart terminal such as a mobile phone or a computer.

[0054] In one embodiment, if Figure 2 As shown, a code duplication checking method is provided, which is applied to Figure 1 The server in the example is used as an example. Figure 2 As shown, the following steps are included:

[0055] S201, obtaining a first vector representation of a code file to be checked for duplicate content, and determining a correlation value between the first vector representation and each second vector representation in a target vector database.

[0056] The code file to be checked for duplicate content refers to a code file that needs to be checked for duplicate content. Checking for duplicate content refers to similarity detection. The first vector representation refers to the vector corresponding to the code file to be checked for duplicate content. The target vector database refers to a database for storing the second vector representation that needs to be analyzed for correlation with the first vector representation. The correlation value refers to the value obtained by performing the correlation analysis.

[0057] Optionally, in this embodiment, the code file to be checked for duplicates is preprocessed to obtain a preprocessed code file. The preprocessed code file is encoded based on the language analysis model to obtain a first vector representation of the code file to be checked for duplicates.

[0058] In this embodiment, an optional implementation method for preprocessing the code file to be checked for duplicates is to first delete blank lines in the code file to be checked for duplicates to obtain a first code file. According to the file type of the first code file, the grammatical rules of the first code file are determined, and the import statements for introducing external modules and lines consisting only of symbols in the first code file are deleted to obtain a preprocessed file, for example, a file in C tn '' indicates that. Based on the language model M, the preprocessed file is encoded to obtain the first vector representation v of the duplicate code file to be checked with a length of L tn .

[0059] Optionally, an optional implementation method of a code duplication checking method in this embodiment is to obtain a first code type of a code file to be checked for duplication. According to the first code type and the second code type of the reference code file corresponding to each second vector representation in the candidate vector database, a target vector database is selected from the candidate vector database; wherein the second code type of the reference code file corresponding to each second vector representation in each candidate vector database is the same; and the second code type of the reference code file corresponding to each second vector representation in the target vector database is the same as the first code type.

[0060] It should be noted that in the present application, multiple candidate vector databases are constructed according to the code type, and the code type corresponding to each candidate vector database is different.

[0061] Optionally, in this embodiment, an optional implementation method for constructing a candidate vector database is to first perform a search on the mth reference code file C of the tth file type. tm , delete the blank lines in the reference code file and get the code file C tm '; For the code file C obtained after preprocessing tm ', according to the code type corresponding to the code file, determine the grammatical rules of the code file, delete the import statements used to introduce external modules and the lines consisting only of symbols in the file, and obtain the preprocessed code file C tm ''; Preprocess all code files of T language types to obtain all preprocessed code files {C1(C 1a '',C 1b '',…),C2(C 2a '', C 2b '',…),…,C T (C Ta '', C Tb'',…)}. Among them, C T Represents the code file library of the code type in T. Ta The ath preprocessed code file representing the code type in the Tth code. Based on the pretrained language model M, the preprocessed file is encoded to obtain the corresponding second vector representation, and the code type corresponding to the second vector representation is stored to obtain a candidate vector database, which can be expressed as: W=[W1(v 1a , v 1b ,…),W2(v 2a , v 2b ,…),…,W T (v Ta , v Tb ,…)]; where W1 represents the candidate vector database of the first code type. 1b The b-th second vector representation in the candidate vector database representing the first code type.

[0062] In this embodiment, an optional implementation method for determining the correlation value between the first vector representation and each second vector representation in the target vector database is to perform a correlation analysis on the first vector representation and each second vector representation in the target vector database, and obtain the correlation value between the first vector representation and each second vector representation in the target vector database based on the correlation analysis result.

[0063] In this embodiment, different candidate vector databases are constructed for code files of different code types. When determining the correlation value, the correlation value between the first vector representation of the code file to be checked for duplicate content and each second vector representation in the target vector database is determined, rather than determining the correlation value with the second vector representation in the candidate vector database, which effectively improves the efficiency of code duplication checking. In addition, the first vector representation and the second vector representation are obtained based on language model encoding, that is, when checking for code duplication, grammatical and semantic analysis are also considered, further improving the accuracy of code duplication checking.

[0064] S202, obtaining a reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and using the obtained reference code file as a target code file.

[0065] The reference code file refers to a code file used as a reference, generally a code file that has been made public or disclosed internally. The target code file refers to a reference code file whose correlation value meets the requirements.

[0066] Optionally, in this embodiment, the second vector representations are sorted according to the correlation value, and the reference code files corresponding to the second vector representations whose correlation values ​​are greater than the correlation threshold are used as the target code files. For example, when the correlation threshold is 5, the reference code files corresponding to the second vector representations whose sorting results are in the top 5 are used as the target code files.

[0067] S203, determining a code duplication checking result of the code file to be checked for duplicate content according to the target code file and the code file to be checked for duplicate content.

[0068] The code duplication check result refers to the similarity judgment result obtained by comparing and analyzing the code file to be checked for duplication with the target code file.

[0069] As an optional implementation method of the present application, semantic analysis is performed on the target code file and the code file to be checked for duplicates, and the similarity value between the code file to be checked for duplicates and each target code file is further determined, and the target code file with the highest similarity value or higher than a preset threshold and the corresponding similarity value are used as the code duplication checking result.

[0070] Another optional implementation method of the embodiment of the present application is to input the target code file and the code file to be checked for duplicate code into a language analysis model, and the language analysis model outputs the code duplication checking result of the code file to be checked for duplicate code.

[0071] The above-mentioned code duplicate checking method obtains the first vector representation of the code file to be checked for duplicates, and determines the correlation value between the first vector representation and each second vector representation in the target vector database. Obtain the reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and use the obtained reference code file as the target code file. According to the target code file and the code file to be checked for duplicates, determine the code duplicate checking result of the code file to be checked for duplicates. When performing code duplicate checking, the present application adopts the form of vector representation to check for duplicates. Compared with the linear scanning of the traditional method, the present application has a faster duplicate checking speed and can complete the calculation of the correlation value in a shorter time. And based on the correlation value, obtain the target code file. According to the target code file and the code file to be checked for duplicates, determine the code duplicate checking result of the code file to be checked for duplicates, based on the double duplicate checking mechanism, effectively improve the accuracy of the duplicate checking result.

[0072] In one embodiment, in order to more accurately determine the correlation value, such as Figure 3 As shown, an optional implementation of determining the correlation value between the first vector representation and each second vector representation in the target vector database in S20 includes:

[0073] S301, determining the similarity between the first vector representation and each second vector representation in the target vector database.

[0074] Here, similarity refers to the degree of similarity.

[0075] As an optional implementation of the present application, based on a similarity algorithm, a similarity value between the first vector representation and each second vector representation in the target vector database is determined. The similarity value is used as the similarity between the first vector representation and each second vector representation in the target vector database. The similarity algorithm includes but is not limited to Euclidean distance, cosine similarity, inner product, Manhattan distance, Hamming distance, and Jaccard similarity coefficient.

[0076] As another optional implementation of the present application, the first vector representation and each second vector representation in the target vector database are input into a similarity analysis model, and the similarity analysis model outputs the similarity between the first vector representation and each second vector representation in the target vector database. The similarity analysis model includes but is not limited to a neural network model.

[0077] S302: Select a maximum similarity and a minimum similarity from each similarity.

[0078] Optionally, in this embodiment, the similarities are sorted, and according to the sorting result, the maximum similarity and the minimum similarity are selected from the similarities.

[0079] S303, determining a correlation value between the first vector representation and each second vector representation in the target vector database according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database.

[0080] Optionally, in this embodiment, the difference between the maximum similarity and the minimum similarity is used as the first value. For each second vector representation, the difference between the similarity between the first vector representation and the second vector representation and the minimum similarity is used as the second value. The ratio of the first value to the second value is used as the correlation value between the first vector representation and the second vector representation.

[0081] Optionally, in this embodiment, the correlation value between the first vector representation and each second vector representation in the target vector database may be determined by the following formula (1):

[0082] ξ nx '=(ξ nx – min(ξ na, ξ nb,… )) / ( max(ξ na, ξ nb,… ) – min(ξ na, ξ nb,… )) (1)

[0083] Among them, ξ in formula (1)nx represents the similarity between the first vector representation and the x-th second vector representation in the target vector database; ξ na represents the similarity between the first vector representation and the a-th second vector representation in the target vector database; ξ nb represents the similarity between the first vector representation and the a-th second vector representation in the target vector database; ξ nx ' represents the correlation value between the first vector representation and the x-th second vector representation in the target vector database.

[0084] In this embodiment, the similarity between the first vector representation and each second vector representation in the target vector database is determined; the maximum similarity and the minimum similarity are selected from each similarity; and the correlation value between the first vector representation and each second vector representation in the target vector database is determined based on the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database. Based on this embodiment, the obtained correlation value can be made more accurate, and the duplicate checking in the form of vector representation can effectively improve the efficiency of duplicate checking.

[0085] In one embodiment, in order to obtain more accurate code duplication checking results, the present application adopts a double duplication checking mechanism, such as Figure 4 As shown, an optional implementation of S204 includes:

[0086] S401, obtaining a first AST representation of a code file to be checked for duplicate content and a second AST representation of a target code file.

[0087] The first AST (Abstract Syntax Tree) representation refers to the way in which the code file to be checked for duplicates is converted into an abstract syntax tree to represent the structure and semantics of the code. The second AST representation refers to the way in which the target code file is converted into an abstract syntax tree to represent the structure and semantics of the code.

[0088] Optionally, in the present embodiment, according to the code type of the code file to be checked for duplicates, the language type of the code file to be checked for duplicates is determined. According to the language type, the target parsing library is determined. Based on the target parsing library, the code file to be checked for duplicates and the target code file are parsed to obtain the first AST representation of the code file to be checked for duplicates and the second AST representation of the target code file. Wherein, when the target parsing library parses the code file to be checked for duplicates and the target code file, it is necessary to standardize the variable name, and normalize constants, string literals and normalized formats to ensure the consistency of the structure in the AST representation.

[0089] S402: Determine a code duplication check result of the code file to be checked for duplicate content based on the first AST representation and the second AST representation.

[0090] Optionally, in this embodiment, the node type and node content of each node in the first AST representation and the second AST representation are determined respectively. For each node, a hash operation is performed according to the node content and the node type to obtain the hash value of the node. Finally, the node type and hash value of each node of the first AST representation are compared and analyzed with the node type and hash value of each node of each second AST representation to obtain the similarity between the first AST representation and the second AST representation. The similarity between the first AST representation and the second AST representation is used as the similarity between the code file to be checked for duplicate content and the target code file, and the code duplicate checking result is determined based on the similarity. For example, the degree of similarity with the highest similarity (for example, a similarity of 95%) and the corresponding target code file are used as the code duplicate checking result.

[0091] In this embodiment, the first abstract syntax tree AST representation of the code file to be checked for duplicates and the second AST representation of the target code file are obtained. According to the first AST representation and the second AST representation, the code duplicate checking result of the code file to be checked for duplicates is determined. The present application innovatively adopts a dual duplicate checking mechanism, that is, the first layer uses the form of vector representation for duplicate checking, which improves the accuracy of duplicate checking and improves the efficiency of duplicate checking. The second layer uses the form of AST representation for duplicate checking, and more finely analyzes the similarity between the code file to be checked for duplicates and the target code file, so that the obtained code duplicate checking result is more accurate.

[0092] In one embodiment, if Figure 5 As shown, an optional implementation of a code duplication checking method includes:

[0093] S501, preprocessing the code file to be checked for duplicate content to obtain a preprocessed code file.

[0094] S502, encoding the preprocessed code file based on the language analysis model to obtain a first vector representation of the code file to be checked for duplicates.

[0095] S503, obtaining the first code type of the code file to be checked for duplicate code.

[0096] S504, selecting a target vector database from the candidate vector database according to the first code type and the second code type of the reference code file corresponding to each second vector representation in the candidate vector database. The second code type of the reference code file corresponding to each second vector representation in each candidate vector database is the same; and the second code type of the reference code file corresponding to each second vector representation in the target vector database is the same as the first code type.

[0097] S505: Determine the similarity between the first vector representation and each second vector representation in the target vector database.

[0098] S506: Select the maximum similarity and the minimum similarity from the similarities.

[0099] S507: Taking the difference between the maximum similarity and the minimum similarity as the first value.

[0100] S508 : For each second vector representation, use the difference between the similarity between the first vector representation and the second vector representation and the minimum similarity as the second value.

[0101] S509: Use the ratio of the first value to the second value as a correlation value between the first vector representation and the second vector representation.

[0102] S510, obtaining a reference code file corresponding to a second vector representation having a correlation value greater than a correlation threshold, and using the obtained reference code file as a target code file.

[0103] S511, obtaining a first AST representation of the code file to be checked for duplicate content and a second AST representation of the target code file.

[0104] S512: Determine a code duplication check result of the code file to be checked for duplication according to the first AST representation and the second AST representation.

[0105] In this embodiment, a first vector representation of a code file to be checked for duplicate content is obtained, and a correlation value between the first vector representation and each second vector representation in a target vector database is determined. A reference code file corresponding to a second vector representation whose correlation value is greater than a correlation threshold is obtained, and the reference code file obtained is used as a target code file. According to the target code file and the code file to be checked for duplicate content, a code duplicate checking result of the code file to be checked for duplicate content is determined. When performing code duplicate checking, the present application adopts a vector representation form to perform duplicate checking. Compared with the linear scanning of the traditional method, the present application has a faster duplicate checking speed and can complete the calculation of the correlation value in a shorter time. And based on the correlation value, the target code file is obtained. According to the target code file and the code file to be checked for duplicate content, the code duplicate checking result of the code file to be checked for duplicate content is determined, and based on the double duplicate checking mechanism, the accuracy of the duplicate checking result is effectively improved.

[0106] Based on the same inventive concept, the embodiment of the present application also provides a code duplication checking device for implementing the code duplication checking method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiment of a code duplication checking device provided below can refer to the limitations of the code duplication checking method above, and will not be repeated here.

[0107] In one embodiment, if Figure 6 As shown, a code duplication checking device 1 is provided, comprising:

[0108] A first acquisition module 10 is used to acquire a first vector representation of the code file to be checked for duplicates, and determine a correlation value between the first vector representation and each second vector representation in the target vector database;

[0109] A second acquisition module 20, used to acquire a reference code file corresponding to a second vector representation whose correlation value is greater than a correlation threshold, and use the acquired reference code file as a target code file;

[0110] The determination module 30 is used to determine the code duplication checking result of the code file to be checked for duplicates based on the target code file and the code file to be checked for duplicates.

[0111] Based on the device in the present embodiment, the first vector representation of the code file to be checked for duplicate content is obtained, and the correlation value between the first vector representation and each second vector representation in the target vector database is determined. The reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold is obtained, and the reference code file obtained is used as the target code file. According to the target code file and the code file to be checked for duplicate content, the code duplicate checking result of the code file to be checked for duplicate content is determined. When performing code duplicate checking, the present application adopts the form of vector representation to check for duplicate content. Compared with the linear scanning of the traditional method, the present application has a faster duplicate checking speed and can complete the calculation of the correlation value in a shorter time. And based on the correlation value, the target code file is obtained. According to the target code file and the code file to be checked for duplicate content, the code duplicate checking result of the code file to be checked for duplicate content is determined, and based on the double duplicate checking mechanism, the accuracy of the duplicate checking result is effectively improved.

[0112] In one embodiment, the Figure 6 The first determining module 10 is further specifically used for:

[0113] determining a similarity between the first vector representation and each second vector representation in a target vector database;

[0114] Select the maximum similarity and the minimum similarity from each similarity;

[0115] A correlation value between the first vector representation and each second vector representation in the target vector database is determined according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database.

[0116] In one embodiment, the Figure 6 The first determining module 10 is further specifically used for:

[0117] The difference between the maximum similarity and the minimum similarity is taken as the first value;

[0118] For each second vector representation, taking the difference between the similarity between the first vector representation and the second vector representation and the minimum similarity as the second value;

[0119] The ratio of the first value to the second value is taken as the correlation value between the first vector representation and the second vector representation.

[0120] In one embodiment, the Figure 6 The first determining module 10 is further specifically used for:

[0121] Preprocess the code file to be checked for duplicate content to obtain a preprocessed code file;

[0122] The preprocessed code file is encoded based on the language analysis model to obtain a first vector representation of the code file to be checked for duplicates.

[0123] In one embodiment, the Figure 6 The code duplication checking device 1 further comprises:

[0124] A third acquisition module is used to acquire a first code type of the code file to be checked for duplicates;

[0125] A selection module is used to select a target vector database from a candidate vector database according to a first code type and a second code type of a reference code file corresponding to each second vector representation in the candidate vector database; wherein the second code type of the reference code file corresponding to each second vector representation in each candidate vector database is the same; and the second code type of the reference code file corresponding to each second vector representation in the target vector database is the same as the first code type.

[0126] In one embodiment, the Figure 6 The determination module 30 is further specifically used for:

[0127] Obtain a first AST representation of the code file to be checked for duplicate content and a second AST representation of the target code file;

[0128] Determine a code duplication check result of the code file to be checked for duplicate content based on the first AST representation and the second AST representation.

[0129] Each module in the above-mentioned code duplication checking device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0130] In one embodiment, a computer device is provided, whose internal structure diagram can be as follows: Figure 7As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a code duplication detection method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0131] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0133] Obtaining a first vector representation of the code file to be checked for duplicates, and determining a correlation value between the first vector representation and each second vector representation in the target vector database;

[0134] Acquire a reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and use the acquired reference code file as a target code file;

[0135] According to the target code file and the code file to be checked for duplicate code, the code duplication checking result of the code file to be checked for duplicate code is determined.

[0136] In one embodiment, when the processor executes the computer program, the following steps are further implemented: determining a correlation value between the first vector representation and each second vector representation in the target vector database, including:

[0137] determining a similarity between the first vector representation and each second vector representation in a target vector database;

[0138] Select the maximum similarity and the minimum similarity from each similarity;

[0139] A correlation value between the first vector representation and each second vector representation in the target vector database is determined according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database.

[0140] In one embodiment, when the processor executes the computer program, the following steps are further implemented: determining a correlation value between the first vector representation and each second vector representation in the target vector database according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database, including:

[0141] The difference between the maximum similarity and the minimum similarity is taken as the first value;

[0142] For each second vector representation, taking the difference between the similarity between the first vector representation and the second vector representation and the minimum similarity as the second value;

[0143] The ratio of the first value to the second value is taken as the correlation value between the first vector representation and the second vector representation.

[0144] In one embodiment, when the processor executes the computer program, the following steps are further implemented: obtaining a first vector representation of the code file to be checked for duplicate content, including:

[0145] Preprocess the code file to be checked for duplicate content to obtain a preprocessed code file;

[0146] The preprocessed code file is encoded based on the language analysis model to obtain a first vector representation of the code file to be checked for duplicates.

[0147] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0148] Obtain the first code type of the code file to be checked for duplicates;

[0149] A target vector database is selected from the candidate vector database according to the first code type and the second code type of the reference code file corresponding to each second vector representation in the candidate vector database; wherein the second code type of the reference code file corresponding to each second vector representation in each candidate vector database is the same; and the second code type of the reference code file corresponding to each second vector representation in the target vector database is the same as the first code type.

[0150] In one embodiment, when the processor executes the computer program, the following steps are further implemented: determining the code duplication check result of the code file to be checked for duplicates according to the target code file and the code file to be checked for duplicates, including:

[0151] Obtain a first AST representation of the code file to be checked for duplicate content and a second AST representation of the target code file;

[0152] Determine a code duplication check result of the code file to be checked for duplicate content based on the first AST representation and the second AST representation.

[0153] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0154] Obtaining a first vector representation of the code file to be checked for duplicates, and determining a correlation value between the first vector representation and each second vector representation in the target vector database;

[0155] Acquire a reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and use the acquired reference code file as a target code file;

[0156] According to the target code file and the code file to be checked for duplicate code, the code duplication checking result of the code file to be checked for duplicate code is determined.

[0157] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: determining a correlation value between the first vector representation and each second vector representation in the target vector database, comprising:

[0158] determining a similarity between the first vector representation and each second vector representation in a target vector database;

[0159] Select the maximum similarity and the minimum similarity from each similarity;

[0160] A correlation value between the first vector representation and each second vector representation in the target vector database is determined according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database.

[0161] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a correlation value between the first vector representation and each second vector representation in the target vector database according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database, including:

[0162] The difference between the maximum similarity and the minimum similarity is taken as the first value;

[0163] For each second vector representation, taking the difference between the similarity between the first vector representation and the second vector representation and the minimum similarity as the second value;

[0164] The ratio of the first value to the second value is taken as the correlation value between the first vector representation and the second vector representation.

[0165] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: obtaining a first vector representation of the code file to be checked for duplicate content, including:

[0166] Preprocess the code file to be checked for duplicate content to obtain a preprocessed code file;

[0167] The preprocessed code file is encoded based on the language analysis model to obtain a first vector representation of the code file to be checked for duplicates.

[0168] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0169] Obtain the first code type of the code file to be checked for duplicates;

[0170] A target vector database is selected from the candidate vector database according to the first code type and the second code type of the reference code file corresponding to each second vector representation in the candidate vector database; wherein the second code type of the reference code file corresponding to each second vector representation in each candidate vector database is the same; and the second code type of the reference code file corresponding to each second vector representation in the target vector database is the same as the first code type.

[0171] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining the code duplication check result of the code file to be checked for duplicates according to the target code file and the code file to be checked for duplicates, including:

[0172] Obtain a first AST representation of the code file to be checked for duplicate content and a second AST representation of the target code file;

[0173] Determine a code duplication check result of the code file to be checked for duplicate content based on the first AST representation and the second AST representation.

[0174] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0175] Obtain a first vector representation of the code file to be checked for duplicates, and determine a correlation value between the first vector representation and each second vector representation in the target vector database;

[0176] Acquire a reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and use the acquired reference code file as a target code file;

[0177] According to the target code file and the code file to be checked for duplicate code, the code duplication checking result of the code file to be checked for duplicate code is determined.

[0178] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: determining a correlation value between the first vector representation and each second vector representation in the target vector database, comprising:

[0179] determining a similarity between the first vector representation and each second vector representation in a target vector database;

[0180] Select the maximum similarity and the minimum similarity from each similarity;

[0181] A correlation value between the first vector representation and each second vector representation in the target vector database is determined according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database.

[0182] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining a correlation value between the first vector representation and each second vector representation in the target vector database according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each second vector representation in the target vector database, including:

[0183] The difference between the maximum similarity and the minimum similarity is taken as the first value;

[0184] For each second vector representation, taking the difference between the similarity between the first vector representation and the second vector representation and the minimum similarity as the second value;

[0185] The ratio of the first value to the second value is taken as the correlation value between the first vector representation and the second vector representation.

[0186] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: obtaining a first vector representation of the code file to be checked for duplicate content, including:

[0187] Preprocess the code file to be checked for duplicate content to obtain a preprocessed code file;

[0188] The preprocessed code file is encoded based on the language analysis model to obtain a first vector representation of the code file to be checked for duplicates.

[0189] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0190] Obtain the first code type of the code file to be checked for duplicates;

[0191] A target vector database is selected from the candidate vector database according to the first code type and the second code type of the reference code file corresponding to each second vector representation in the candidate vector database; wherein the second code type of the reference code file corresponding to each second vector representation in each candidate vector database is the same; and the second code type of the reference code file corresponding to each second vector representation in the target vector database is the same as the first code type.

[0192] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: determining the code duplication check result of the code file to be checked for duplicates according to the target code file and the code file to be checked for duplicates, including:

[0193] Obtain a first AST representation of the code file to be checked for duplicate content and a second AST representation of the target code file;

[0194] Determine a code duplication check result of the code file to be checked for duplicate content based on the first AST representation and the second AST representation.

[0195] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0196] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0197] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A code duplication checking method, characterized in that: The method comprises: Obtain a first vector representation of the code file to be checked for duplicates, and determine a correlation value between the first vector representation and each second vector representation in a target vector database; Acquire a reference code file corresponding to the second vector representation whose correlation value is greater than the correlation threshold, and use the acquired reference code file as a target code file; The code duplication checking result of the code file to be checked for duplicates is determined according to the target code file and the code file to be checked for duplicates.

2. The method according to claim 1, characterized in that: The determining of the correlation value between the first vector representation and each second vector representation in the target vector database comprises: determining a similarity between the first vector representation and each second vector representation in a target vector database; Select the maximum similarity and the minimum similarity from each similarity; Determine a correlation value between the first vector representation and each of the second vector representations in the target vector database based on the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each of the second vector representations in the target vector database.

3. The method according to claim 2, characterized in that Determining the correlation value between the first vector representation and each of the second vector representations in the target vector database according to the maximum similarity, the minimum similarity, and the similarity between the first vector representation and each of the second vector representations in the target vector database includes: Taking the difference between the maximum similarity and the minimum similarity as the first value; For each second vector representation, taking the difference between the similarity between the first vector representation and the second vector representation and the minimum similarity as a second value; The ratio of the first value to the second value is used as a correlation value between the first vector representation and the second vector representation.

4. The method according to claim 1, characterized in that The step of obtaining a first vector representation of the code file to be checked for duplicate content includes: Preprocess the code file to be checked for duplicate content to obtain a preprocessed code file; The preprocessed code file is encoded based on a language analysis model to obtain a first vector representation of the code file to be checked for duplicates.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining a first code type of the code file to be checked for duplicates; According to the first code type and the second code type of the reference code file corresponding to each second vector representation in the candidate vector database, a target vector database is selected from the candidate vector database; wherein the second code type of the reference code file corresponding to each second vector representation in each candidate vector database is the same; and the second code type of the reference code file corresponding to each second vector representation in the target vector database is the same as the first code type.

6. The method according to claim 1, characterized in that The step of determining the code duplication checking result of the code file to be checked for duplicate content according to the target code file and the code file to be checked for duplicate content includes: Obtain a first abstract syntax tree (AST) representation of the code file to be checked for duplicates and a second AST representation of the target code file; Determine a code duplication check result of the code file to be checked for duplication according to the first AST representation and the second AST representation.

7. A code duplication checking device, characterized in that: The device comprises: A first acquisition module, used for acquiring a first vector representation of a code file to be checked for duplicates, and determining a correlation value between the first vector representation and each second vector representation in a target vector database; A second acquisition module, used to acquire a reference code file corresponding to a second vector representation whose correlation value is greater than a correlation threshold, and use the acquired reference code file as a target code file; A determination module is used to determine the code duplication checking result of the code file to be checked for duplicate content based on the target code file and the code file to be checked for duplicate content.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the code duplication checking method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images