Homology detection method, device and equipment for large-scale codes

Abstract

The invention provides a homology detection method, device and equipment for large-scale codes, and relates to the technical field of code homology detection.The homology detection method comprises the steps that different code files contained in a large-scale code project to be detected and abstract syntax trees corresponding to the code files are obtained; for any code file, extracting a plurality of target code elements from the abstract syntax tree of the code file; a plurality of target code elements in the code file are converted into Token characters, and a converted first code is obtained; performing multi-level segmentation on the first code to obtain second codes of different levels; respectively extracting feature values of the code file and the second codes of different levels, and constructing code features of the to-be-detected large-scale code project; and comparing the code features with different code features in a configured code feature database to obtain a homology detection result of the to-be-detected large-scale code project. According to the invention, a homology detection result with relatively high accuracy can be obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of code homology detection, in particular to a large-scale code homology detection method, device and equipment. BACKGROUND

[0002] The prior art generally uses regular expressions to tokenize and replace source code when performing homology detection, and then extracts the features of the code file by taking the hash value of the full text. This method not only incorrectly identifies normal function code in the source code as strings or comments, but also has a low recognition accuracy and a large number of false positives, which affects the accuracy of the comparison results. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide a large-scale code homology detection method, device and equipment to solve the above problems existing in the prior art and obtain a homology detection result with high accuracy.

[0004] In a first aspect, a large-scale code homology detection method is provided, which can include: Obtaining different code files contained in a large-scale code project to be detected and an abstract syntax tree corresponding to each code file; For any code file, extracting a plurality of target code elements from the abstract syntax tree of the code file; Converting the plurality of target code elements in the code file into Token characters to obtain a first code after conversion; Performing multi-level segmentation on the first code to obtain second codes at different levels; Respectively extracting feature values of the second codes at different levels to obtain first features of the code file; Extracting feature values of the code file to obtain second features of the code file; Constructing code features of the large-scale code project to be detected according to the first features and the second features of the different code files; Comparing the code features with different code features in a configured code feature database to obtain a homology detection result of the large-scale code project to be detected.

[0005] In an optional implementation, obtaining different code files contained in a large-scale code project to be detected and an abstract syntax tree corresponding to each code file includes: Obtaining different initial code files contained in a large-scale code project to be detected and file types of the different initial code files; According to a configured filtering rule, filtering the different initial code files to obtain a plurality of code files; For any code file, the code file is parsed according to a file parser corresponding to a file type of the code file configured, to obtain an abstract syntax tree corresponding to the code file.

[0006] In an optional implementation, a plurality of target code elements are extracted from the abstract syntax tree of the code file, including: A plurality of initial code elements are extracted from the abstract syntax tree of the code file. For any initial code element, if the initial code element is not included in a code element library configured, the initial code element is taken as a target code element.

[0007] In an optional implementation, the target code elements include: empty lines, comments, variable names, parameter names, and function names.

[0008] In an optional implementation, a plurality of target code elements in the code file are converted into Token characters to obtain a first code after conversion, including: The plurality of target code elements are classified according to element types of different target code elements. For any category of target code elements, a target Token character corresponding to the category of target code elements is matched according to different Token characters corresponding to different element types configured. Different categories of target code elements in the code file are replaced with target Token characters corresponding to the target code elements of the corresponding categories to obtain the first code after conversion.

[0009] In an optional implementation, the first code is divided into a plurality of levels to obtain second codes of different levels, including: The first code is divided into a plurality of code lines and a plurality of function blocks to obtain second line codes and second function codes, respectively. Feature values of the second codes of different levels are extracted to obtain first features of the code file, including: A plurality of continuous code combinations are extracted from the second line codes using a dynamic window. Hash values of different continuous code combinations are calculated. For any function block in the second function codes, a hash value of the function block is extracted. Based on the hash values of the different continuous code combinations and the hash values of the different function blocks, the first features of the code file are obtained.

[0010] In an optional implementation, feature values of the code file are extracted to obtain second features of the code file, including: A hash value of the code file is extracted to obtain a first hash value. extracting a hash value of the first code to obtain a second hash value; obtaining a second feature of the code file based on the first hash value and the second hash value.

[0011] In a second aspect, a device for homology detection of large-scale codes is provided, which can include: a obtaining unit configured to obtain different code files contained in a large-scale code project to be detected and an abstract syntax tree corresponding to each code file; a first extracting unit configured to extract a plurality of target code elements from the abstract syntax tree of any code file; a converting unit configured to convert the plurality of target code elements in the code file into Token characters to obtain a converted first code; a cutting unit configured to cut the first code in multiple levels to obtain second codes at different levels; a second extracting unit configured to extract feature values of the second codes at different levels respectively to obtain a first feature of the code file and extract feature values of the code file to obtain a second feature of the code file; a constructing unit configured to construct code features of the large-scale code project to be detected according to the first features and the second features of different code files; a detecting unit configured to compare the code features with different code features in a configured code feature database to obtain a homology detection result of the large-scale code project to be detected.

[0012] In a third aspect, an electronic device is provided, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; the memory is configured to store a computer program; the processor is configured to execute the program stored on the memory to implement the method steps of any of the first aspect.

[0013] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the method steps of any of the first aspect.

[0014] The present application can significantly improve the recognition accuracy by analyzing and processing codes through abstract syntax trees, and can improve the detection accuracy, reduce the false positive rate and improve the reliability of the homology detection result by extracting HASH values in multiple levels and obtaining features of code files. BRIEF DESCRIPTION OF DRAWINGS In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0015] Figure 1 A schematic diagram of an architecture of a homology detection system for large-scale code provided by the embodiments of the present application; Figure 2 A schematic diagram of a flow of a homology detection method for large-scale code provided by the embodiments of the present application; Figure 3 A schematic diagram of a homology detection result for large-scale code provided by the embodiments of the present application; Figure 4 A schematic diagram of a structure of a homology detection device for large-scale code provided by the embodiments of the present application; Figure 5 A schematic diagram of a structure of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0016] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, and not all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application. Unless otherwise defined, the technical terms or scientific terms used in the present application should be understood as the general meaning understood by those skilled in the art. The words "first", "second", and similar words used in the present application do not represent any order, number, or importance, but are only used to distinguish different components. The words "include" or "contain" and similar words mean that the elements or objects before the words cover the elements or objects listed after the words and their equivalents, and do not exclude other elements or objects. The words "connect", "couple", or "connect" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right", and the like are only used to represent relative positional relationships, and when the absolute positions of the described objects change, the relative positional relationships may also change accordingly.

[0017] In order to facilitate understanding, the following explains the terms involved in the embodiments of the present application: Homogeneity detection, also known as code similarity analysis or plagiarism detection, refers to judging the similarity of two or more large-scale code projects in structure, logic and implementation by algorithms and technical means.

[0018] The homogeneity detection method for large-scale code provided by the embodiments of the present application can be applied to Figure 1 The system architecture shown in the system is run on a server with multi-thread processing; the system can include a file traversal layer, a file processing layer and a result aggregation layer; The file traversal layer includes a traversal thread pool and an atomic counter; a parallel directory traversal engine is used to scan the directory results of the large-scale code project in multiple threads, and different initial code files are filtered according to the configured filtering rules to obtain multiple code files; the file paths of each code file are generated; The file processing layer is configured to build a producer-consumer model according to a lock-free channel, and different processing threads obtain the file paths of different code files to read the file contents of the corresponding code files and execute the homogeneity detection method provided by the embodiments of the present application, so as to obtain the homogeneity detection results of each code file and write the homogeneity detection results of each code file into the configured result temporary queue; As shown in Figure 1 The file processing layer includes a fingerprint extraction module and a fingerprint comparison module; the fingerprint extraction module is used to extract the fingerprint features (i.e. code features) in the source code directory, specifically, by traversing the directory structure of the code repository, parsing the ignore file of the version control tool (such as Git, SVN) to exclude unnecessary files and directories, and ensuring that the extracted code information is valid; a special file parser is designed for different programming languages to eliminate syntax differences; blank lines, comments, variable names, parameter names, function names, etc. are uniformly replaced with Token characters to eliminate naming differences and facilitate subsequent feature extraction; the code files are cut according to file level, line level and function level, and dynamic window technology is used to extract feature Hash values to form project fingerprints; the fingerprint comparison module is used to compare the extracted project fingerprints or other source code features with the feature database to form a detailed result report, and the homogeneity results are intuitively displayed in a visual manner; Specifically, the fingerprint comparison module includes an interface module, a database module, a middleware module, a data structure definition module, and a routing module. The interface module is used to define the interface of the backend service and provides functions such as user management, task management, code library management, and project management. The database module is used to implement the CRUD operations of the databases of users, tasks, code libraries, projects, and the like. The database module also implements index optimization, data sharding, and other operations to improve the performance and scalability of the system. The middleware module is used to implement the JWT authentication middleware to ensure the security of the system and the verification of user identities. Through JWT authentication, unauthorized access and operations can be effectively prevented. The data structure definition module is used to define the data table structure and implement serialization and deserialization functions. The data structure definition module ensures the structuring and consistency of data, facilitating data storage and retrieval. The routing module is used to define interface routing and route different interface requests to corresponding processing modules.

[0019] The result aggregation layer is used to batch read the homoplasy detection results of each code file from the result temporary queue, and obtain the homoplasy detection results of the large-scale code project after deduplication and sorting.

[0020] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0021] Figure 2 A flowchart of a homoplasy detection method for large-scale code provided by an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the method can include the following steps. Figure 2 In step S210, different code files contained in the large-scale code project to be detected and the abstract syntax trees corresponding to each code file are obtained.

[0022] ​Specifically, different initial code files contained in the large-scale code project to be detected and file types of the different initial code files are acquired; the different initial code files include directory structures, source code files, configuration files, build script files, dependency management files, version control related files, document files, test code files, resource files, and environment-specific setting files; different initial code files are screened according to a configured screening rule, to obtain a plurality of code files; specifically, a directory structure of the large-scale code project is traversed, and the different initial code files are screened by parsing an ignore file of a version control tool (such as Git or SVN), to exclude unnecessary files and directories, and to obtain the plurality of code files, including source code files, configuration files, script files, and version control related files, to ensure the effectiveness of the extracted code; for any code file, the code file is parsed according to a file parser corresponding to a file type of the code file according to a configured code file, to obtain a first abstract syntax tree corresponding to the code file; a corresponding file parser is constructed in advance for different programming languages, to eliminate syntax differences; the file parser is used to perform lexical analysis (Token splitting) and syntax analysis (constructing a hierarchical structure) on the code file according to syntax rules of the corresponding programming language, to obtain a structured first abstract syntax tree; the first abstract syntax tree contains all syntax-level code elements in the large-scale code project, covering file, package or module, class or function declaration at the top level, variable, parameter, statement block at the middle level, and expression, identifier reference and other structured syntax units at the bottom level, and their hierarchical associations.

[0023] In some embodiments of the present application, the first abstract syntax tree of any code file directly references the original code memory without copying, and the AST nodes in the first abstract syntax tree only store position metadata without storing code content.

[0024] In some embodiments of the present application, after obtaining the first abstract syntax tree of the code file, a configured type tag is set for different types of AST nodes in the first abstract syntax tree of each code file according to a configured syntax tree query rule; different types of AST nodes are hierarchically marked according to a configured hierarchical marking strategy based on the type tag; a basic mark is generated for the AST nodes of common types such as comments, strings, and numbers; a semantic mark is generated for the AST nodes of semantic types such as types, fields, and parameters; an anchor point mark is generated for the AST nodes of types such as function definitions; based on all the marked AST nodes, an abstract syntax tree of the corresponding code file is obtained; specifically, based on all the marked AST nodes, a second abstract syntax tree of the corresponding code file is obtained; nodes satisfying a configured node exclusion rule are filtered from the second abstract syntax tree, to obtain an abstract syntax tree of the corresponding code file; the node exclusion rule is used to exclude language built-in identifiers, standard library functions, design pattern high-frequency words, and template code.

[0025] Step S220: For any code file, extract multiple target code elements from the abstract syntax tree of the code file.

[0026] Specifically, based on the different types of tags in the abstract syntax tree of the code file, multiple initial code elements are extracted from the abstract syntax tree of the code file. For any initial code element, if it is not contained in the configured code element library, it is used as the target code element. The target code element is the key information of the code file, including: blank lines, comments, variable names, parameter names, and function names. The code element library is constructed by extracting sample sets that are incorrectly marked as clone pairs by the homology detection tool from the non-cloned pairs in the standard test set, and counting the code elements (such as variable names, function structures, comments, general logic, etc.) that appear more than a preset threshold in the sample set.

[0027] Specifically, blank lines, comments, variable names, parameter names, and function names are extracted from the abstract syntax tree (AST) of the code file, including: Record the line number range of the code file corresponding to the AST node, compare all line numbers in the code file with the line numbers covered by the AST node, and identify blank lines that are not covered by any node and contain only whitespace characters; use a parsing tool that supports preserving comments to identify the type of comment node (single-line or multi-line) and its position in the code (line number), and extract the comment text; extract variable names through the identifiers of variable declaration or assignment nodes in the AST; extract function definition nodes from the parameter lists of function or method definition nodes in the AST, access parameter list attributes, traverse parameter nodes, and extract the arg or getNam attribute of each parameter, which is the parameter name; extract the function name through the name attribute of the function or method definition node.

[0028] Step S230: Convert multiple target code elements in the code file into token characters to obtain the first converted code.

[0029] Specifically, multiple target code elements are categorized according to their element types. For any given category, the target token character corresponding to the category's target code element is matched based on the configured token characters corresponding to different element types. The target code elements of different categories in the code file are then replaced with the corresponding target token characters to obtain the converted first code, thus eliminating naming differences and facilitating subsequent feature extraction. This application improves the efficiency of code feature extraction by using thread processing for operations such as removing blank lines, removing comments, normalizing variable names, and normalizing function names.

[0030] like Figure 3As shown, after conversion, code elements of different element types are displayed in different colors, indicating that their corresponding target token characters are different; in Figure 3 In the code example, if regular expressions are used for tokenization, the " / *" in line 6 to the "* / " in line 8 of the code within the box in the image will be incorrectly identified as comments, but they actually contain function and variable assignments.

[0031] Step S240: Perform multi-level segmentation on the first code to obtain second code at different levels.

[0032] Specifically, the first code is split at the line level and function level to obtain a second line of code containing multiple lines of code and a second function code containing multiple function blocks. Specifically, the first code is split into two parts based on newline characters to obtain a second line of code consisting of single lines of code. The first code is then extracted by function or method, extracting complete code blocks defining the function (including function name, parameters, or function body), resulting in a second function code containing multiple function blocks. When extracting complete code blocks defining function definitions, due to the complexity of the code format, the initially identified function intervals may be scattered, overlapping, or contain irrelevant content. Therefore, any function block contains multiple discontinuous line intervals, for example, function declaration line: [2,2], first part of function block: [3,5], second part of function block: [7,10], and function end line: [11,11]. This application uses an overlapping interval merging algorithm to dynamically intervene during the extraction process, gradually modifying... The positive interval specifically includes: for any function block, obtaining all row intervals of that function block; sorting all row intervals by the starting row to obtain an initial ordered interval set; merging overlapping or adjacent intervals in the initial ordered interval set to obtain an ordered interval set; for any function row in the ordered interval set, obtaining multiple AST nodes of that function row; sorting the multiple AST nodes of that function row by the starting column position; if any two AST nodes overlap, obtaining the priority of the configured corresponding AST nodes, retaining the AST node with higher priority, to obtain the merged node list of that function row; filtering target code nodes from the merged node list of that function row to obtain the valid code intervals within that function row; target code nodes include code nodes and string nodes; generating a reverse replacement sequence for that function row based on the valid code intervals within that function row; and obtaining the second function code of that function block based on the reverse replacement sequences of all function rows.

[0033] In other embodiments of this application, the first code is further segmented into multiple levels to obtain second code at different levels, which may include: The first code is divided into line-level, function-level, and semantic-interval segments respectively, resulting in a second line of code containing multiple lines of code, a second function code containing multiple function blocks, and a second semantic code containing multiple semantic intervals.

[0034] Step S250: Extract the feature values ​​of the second code at different levels to obtain the first feature of the code file; extract the feature values ​​of the code file to obtain the second feature of the code file.

[0035] Specifically, the hash value of the code file is extracted to obtain a first hash value, which is generated using a lightweight hash algorithm. The hash value of the first code is extracted to obtain a second hash value. Based on the first and second hash values, a second feature of the code file is obtained. Specifically, hash values ​​are extracted from the code file before and after the token character replacement to obtain features covering the entire code file. Multiple consecutive code combinations are extracted from the second line of code using a dynamic window. Depending on the configured dynamic window size, the window slides sequentially on the second line of code (moving one or more lines at a time) to obtain multiple consecutive code combinations. The hash values ​​of different consecutive code combinations are calculated. For each window, the consecutive code combinations are concatenated into a string, and the hash value of the string for each consecutive code combination is calculated to obtain the corresponding hash value. For any function block in the second function code, the hash value of the function block is extracted. Based on the hash values ​​of different consecutive code combinations and different function blocks, a first feature of the code file is obtained.

[0036] In other embodiments of this application, extracting the hash value of the first code to obtain the second hash value may further include: Based on the second semantic code containing multiple semantic intervals and the second function code containing multiple function blocks, determine the data flow path of the target element in the code file that affects the final output or key state; extract the hash value of the second semantic code; based on the hash value of the second semantic code and the data flow path of the target element; and based on the hash values ​​of different consecutive code combinations, the hash values ​​of different function blocks, the hash value of the second semantic code, and the data flow path of the target element, obtain the first feature of the code file.

[0037] Step S260: Construct code features for the large-scale code project to be detected based on the first and second features of different code files.

[0038] Specifically, the code features of the large-scale code project to be detected are persistently stored on the hard disk in a specific format for subsequent comparison and analysis.

[0039] Step S270: Compare the code features with different code features in the configured code feature database to obtain the homology detection results of the large-scale code project to be detected.

[0040] Specifically, the code features of the large-scale code project to be detected are compared with the code features of other large-scale code projects in the configured code feature database to obtain the homology detection results.

[0041] Corresponding to the above method, embodiments of this application also provide a homology detection device for large-scale code, such as... Figure 4 As shown, the device includes: The acquisition unit 410 is used to acquire the different code files contained in the large-scale code project to be detected and the abstract syntax tree corresponding to each code file; The first extraction unit 420 is used to extract multiple target code elements from the abstract syntax tree of any code file. The conversion unit 430 is used to convert multiple target code elements in the code file into token characters to obtain the first converted code; The segmentation unit 440 is used to perform multi-level segmentation on the first code to obtain second code at different levels; The second extraction unit 450 is used to extract feature values ​​of the second code at different levels to obtain the first feature of the code file; and to extract feature values ​​of the code file to obtain the second feature of the code file. Construction unit 460 is used to construct code features of the large-scale code project to be detected based on the first and second features of different code files; The detection unit 470 is used to compare code features with different code features in the configured code feature database to obtain the homology detection results of the large-scale code project to be detected.

[0042] The functions of each functional unit in the homology detection device for large-scale code provided in the above embodiments of this application can be implemented through the above method steps. Therefore, the specific working process and beneficial effects of each unit in the homology detection device for large-scale code provided in the embodiments of this application will not be repeated here.

[0043] This application also provides an electronic device, such as... Figure 5 As shown, it includes a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540.

[0044] Memory 530 is used to store computer programs; When the processor 510 executes the program stored in the memory 530, it performs the following steps: Obtain the different code files contained in the large-scale code project to be tested and the abstract syntax tree corresponding to each code file; For any given code file, extract multiple target code elements from the abstract syntax tree of the code file; Convert multiple target code elements in the code file into token characters to obtain the first converted code; The first code is split into multiple levels to obtain second code at different levels; The feature values ​​of the second code at different levels are extracted to obtain the first feature of the code file; Extract the feature values ​​from the code file to obtain the second feature of the code file; Based on the first and second features of different code files, construct the code features of the large-scale code project to be detected; By comparing the code features with different code features in the configured code feature database, the homology detection results of the large-scale code projects to be detected are obtained.

[0045] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0046] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0047] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0048] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0049] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 2 The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0050] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the above embodiments of the homology detection method for large-scale code.

[0051] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the above embodiments of the method for detecting the homology of large-scale code.

[0052] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0054] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The function specified in one or more boxes.

[0055] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0056] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0057] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of this application and its equivalents, then these modifications and variations are also intended to be included in the embodiments of this application.

Claims

1. A method for homology detection of large-scale code, characterized in that, The method includes: Obtain the different code files contained in the large-scale code project to be tested and the abstract syntax tree corresponding to each code file; For any given code file, extract multiple target code elements from the abstract syntax tree of the code file; Multiple target code elements in the code file are converted into token characters to obtain the first converted code; The first code is split into multiple levels to obtain second code at different levels; The feature values ​​of the second code at different levels are extracted to obtain the first feature of the code file; Extract the feature values ​​of the code file to obtain the second feature of the code file; Based on the first and second features of different code files, construct the code features of the large-scale code project to be detected; The code features are compared with different code features in the configured code feature database to obtain the homology detection results of the large-scale code project to be detected.

2. The method as described in claim 1, characterized in that, Obtain the different code files contained in the large-scale code project to be tested and the abstract syntax tree corresponding to each code file, including: Obtain the different initial code files and file types contained in the large-scale code project to be tested; Based on the configured filtering rules, different initial code files are filtered to obtain multiple code files; For any given code file, the code file is parsed according to the configured file parser corresponding to the file type of the code file to obtain the abstract syntax tree corresponding to the code file.

3. The method as described in claim 1, characterized in that, From the abstract syntax tree of the code file, extract multiple target code elements, including: Extract multiple initial code elements from the abstract syntax tree of the code file; For any initial code element, if the configured code element library does not contain the initial code element, then the initial code element is used as the target code element.

4. The method as described in claim 1, characterized in that, The target code elements include: blank lines, comments, variable names, parameter names, and function names.

5. The method as described in claim 1, characterized in that, Multiple target code elements in the code file are converted into token characters to obtain the first converted code, including: Multiple target code elements are classified according to their element types. For any type of target code element, match the target token character corresponding to the target code element of that type based on the different token characters configured for different element types; Replace the target code elements of different categories in the code file with the target token characters corresponding to the target code elements of the corresponding categories to obtain the first code after conversion.

6. The method as described in claim 1, characterized in that, The first code is split into multiple levels to obtain second code at different levels, including: The first code is split into line-level and function-level segments to obtain a second line of code containing multiple lines of code and a second function code containing multiple function blocks. The feature values ​​of the second code at different levels are extracted to obtain the first feature of the code file, including: A dynamic window is used to extract multiple consecutive code combinations from the second line of code; Calculate the hash value of different consecutive code combinations; For any function block in the second function code, extract the hash value of the function block; Based on the hash values ​​of the different consecutive code combinations and the hash values ​​of different function blocks, the first feature of the code file is obtained.

7. The method as described in claim 1, characterized in that, Extracting the feature values ​​of the code file to obtain the second feature of the code file includes: Extract the hash value from the code file to obtain the first hash value; Extract the hash value of the first code to obtain the second hash value; Based on the first hash value and the second hash value, the second feature of the code file is obtained.

8. A homology detection device for large-scale code, characterized in that, The device includes: The acquisition unit is used to acquire the different code files contained in the large-scale code project to be detected and the abstract syntax tree corresponding to each code file; The first extraction unit is used to extract multiple target code elements from the abstract syntax tree of any code file. A conversion unit is used to convert multiple target code elements in the code file into token characters to obtain the converted first code; The segmentation unit is used to perform multi-level segmentation of the first code to obtain second code at different levels; The second extraction unit is used to extract feature values ​​of the second code at different levels to obtain the first feature of the code file; and to extract the feature values ​​of the code file to obtain the second feature of the code file. The building unit is used to construct code features of a large-scale code project to be detected based on the first and second features of different code files. The detection unit is used to compare the code features with different code features in the configured code feature database to obtain the homology detection results of the large-scale code project to be detected.

9. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images