Code homologous detection method and device and storage medium

Abstract

The invention relates to the technical field of network security, in particular to a code homology detection method and device and a storage medium, and is used for solving the problem that code traceability detection methods in related technologies have many limitations. Each candidate code snippet matched with each to-be-tested code snippet is queried from the code set, any lexical item feature comprises features corresponding to each lexical item included in the code set in the corresponding to-be-tested code snippet, and each feature is based on code complexity, annotation density and code line number weight of the corresponding to-be-tested code snippet. The word frequency of the corresponding lexical item and the inverse document frequency are determined; then, on the basis of structural features of all nodes in the graph structure of each extracted code snippet to be tested, screening out a same source code snippet corresponding to the code to be tested from all candidate code snippets; the same source code fragment can be quickly and accurately positioned, and the real-time performance, the high efficiency and the applicability of tracing are improved.

Description

Technical Field

[0001] The present application relates to the field of network security technology, and in particular to a method, device and storage medium for detecting homology of codes. Background Art

[0002] With the continuous development of software development, code traceability technology has attracted more and more attention from the industry. Clarifying the code traceability relationship is a prerequisite for understanding the security bottom line, repairing input vulnerabilities through reference code, and determining the degree of autonomous control. However, most code traceability detection methods in related technologies cannot take into account both efficiency and accuracy, especially in dealing with large-scale code bases, responding to code changes, and cross-language code traceability. There are many performance limitations. Summary of the invention

[0003] The embodiments of the present application provide a code homology detection method, device and storage medium for quickly and accurately locating homology source fragments that match the code to be tested, thereby improving the real-time, efficiency and applicability of the code to be tested traceability.

[0004] The specific technical solutions provided by the embodiments of this application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for detecting homology of a code, comprising:

[0006] Based on the lexical features of each code fragment to be tested in the code to be tested, from a plurality of candidate code fragments included in the code set, querying each candidate code fragment that matches each code fragment to be tested respectively;

[0007] Extracting structural features of each node in the graph structure of each code snippet to be tested;

[0008] Based on the extracted structural features and the structural features of each node corresponding to each candidate code fragment, the same source code fragment corresponding to the code to be tested is selected from each candidate code fragment;

[0009] Among them, any term feature includes features corresponding to each term included in the code set in the corresponding code snippet to be tested, and each feature is determined based on the code complexity, comment density and code line weight of the corresponding code snippet to be tested, as well as the term frequency and inverse document frequency of the corresponding term.

[0010] In a possible implementation, the step of searching for candidate code snippets matching each code snippet to be tested from a plurality of candidate code snippets included in the code set based on the term feature of each code snippet to be tested in the code to be tested includes:

[0011] Perform the following operations for each code snippet to be tested:

[0012] Based on any code snippet to be tested and the terms, extracting term features of the code snippet to be tested;

[0013] Based on the term feature of any code snippet to be tested, candidate code snippets matching any code snippet to be tested are searched from a plurality of candidate code snippets included in the code set.

[0014] In a possible implementation, the extracting the term feature of any code snippet to be tested based on any code snippet to be tested and the terms includes:

[0015] Based on the code set, respectively counting the word frequency and inverse document frequency of each term in any code snippet to be tested, and based on the cyclomatic complexity of any code snippet to be tested, determining the code complexity of any code snippet to be tested;

[0016] Determine the ratio of the number of comment lines to the number of code lines of any code snippet to be tested as the comment density of any code snippet to be tested, and determine the code line weight of any code snippet to be tested based on the number of code lines of any code snippet to be tested;

[0017] The statistically obtained word frequency and inverse document frequency of each term, the code complexity and the comment density, and the product of the code line weight are respectively determined as the features of the corresponding term to obtain the term features of any code snippet to be tested.

[0018] In a possible implementation, searching for candidate code snippets matching any of the code snippets to be tested from a plurality of candidate code snippets included in the code set based on the term feature of any of the code snippets to be tested includes:

[0019] Respectively determining the similarity between the term feature of any of the code snippets to be tested and the term features corresponding to the plurality of candidate code snippets included in the code set;

[0020] The candidate code segments corresponding to a preset number of similarities among the determined similarities are determined as candidate code segments that match any of the code segments to be tested.

[0021] In a possible implementation, based on the extracted structural features and the structural features of each node corresponding to each candidate code fragment, selecting the same source code fragment corresponding to the code to be tested from each candidate code fragment includes:

[0022] Extracting structural features of each node in the graph structure corresponding to each candidate code snippet respectively;

[0023] For each structural feature of each candidate code snippet, perform the following operations:

[0024] For any structural feature, create an index entry for the structural feature, the index entry including a feature identifier of the structural feature, an identifier of the code snippet where the feature is located, and a position of the node in the graph structure;

[0025] Inserting the index entry into an inverted index structure;

[0026] Based on the extracted structural features of each node of each code snippet to be tested, searching for index entries matching the corresponding structural features from the inverted index structure respectively;

[0027] Based on the identifier of the code fragment included in each queried index entry, a same source code fragment corresponding to the code to be tested is obtained.

[0028] In a possible implementation manner, obtaining the same source code fragment corresponding to the code to be tested based on the identifier of the code fragment included in each queried index entry includes:

[0029] Determine candidate code segments corresponding to identifiers of code segments included in the queried index entries as reference code segments corresponding to the code to be tested;

[0030] Based on the graph structure of each code snippet to be tested and the graph structures corresponding to the reference code snippets, the same source code snippets corresponding to the code snippets to be tested are matched from the reference code snippets.

[0031] In a possible implementation, the structural features of each node in the graph structure of any code snippet are extracted in the following manner, wherein the any code snippet is any code snippet to be tested, or any candidate code snippet:

[0032] Based on the syntax elements included in any one of the code snippets and the mapping relationship between syntax elements of multiple programming languages ​​and common node types, a graph structure of any one of the code snippets consisting of nodes of the common node type is obtained;

[0033] The following operation is performed for each node in the graph structure of any code snippet: based on the node type feature, semantic feature and context feature of any node, a structural feature of any node of any code snippet is obtained.

[0034] In a possible implementation, obtaining the structural feature of any node of any code snippet based on the node type feature, semantic feature, and context feature of any node includes:

[0035] Based on the index information of the semantic information of any node in the vocabulary set, the semantic feature of any node is obtained, wherein the semantic information includes part or all of the node type, variable name and function name, and is obtained by parsing the graph structure where the any node is located;

[0036] The initial context features of any of the nodes are set, and based on the node type features of any of the nodes, the semantic features and the initial context features, and the structural features of each of the neighboring nodes of any of the nodes, the context information of any of the nodes in any of the code fragments is aggregated to obtain the structural features of any of the nodes.

[0037] In a second aspect, an embodiment of the present application provides a code homology detection device, comprising:

[0038] A query module, for respectively querying, based on the term features of each code fragment to be tested in the code to be tested, candidate code fragments matching each code fragment to be tested from a plurality of candidate code fragments included in the code set;

[0039] An extraction module, used to extract the structural features of each node in the graph structure of each code snippet to be tested;

[0040] A screening module, configured to screen out the same source code fragments corresponding to the code to be tested from the candidate code fragments based on the extracted structural features and the structural features of each node corresponding to the candidate code fragments;

[0041] Among them, any term feature includes features corresponding to each term included in the code set in the corresponding code snippet to be tested, and each feature is determined based on the code complexity, comment density and code line weight of the corresponding code snippet to be tested, as well as the term frequency and inverse document frequency of the corresponding term.

[0042] In a possible implementation, the query module is specifically used to:

[0043] Perform the following operations for each code snippet to be tested:

[0044] Based on any code snippet to be tested and the terms, extracting term features of the code snippet to be tested;

[0045] Based on the term feature of any code snippet to be tested, candidate code snippets matching any code snippet to be tested are searched from a plurality of candidate code snippets included in the code set.

[0046] In a possible implementation, the query module is specifically used to:

[0047] Based on the code set, respectively counting the word frequency and inverse document frequency of each term in any code snippet to be tested, and based on the cyclomatic complexity of any code snippet to be tested, determining the code complexity of any code snippet to be tested;

[0048] Determine the ratio of the number of comment lines to the number of code lines of any code snippet to be tested as the comment density of any code snippet to be tested, and determine the code line weight of any code snippet to be tested based on the number of code lines of any code snippet to be tested;

[0049] The statistically obtained word frequency and inverse document frequency of each term, the code complexity and the comment density, and the product of the code line weight are respectively determined as the features of the corresponding term to obtain the term features of any code snippet to be tested.

[0050] In a possible implementation, the query module is specifically used to:

[0051] Respectively determining the similarity between the term feature of any of the code snippets to be tested and the term features corresponding to the plurality of candidate code snippets included in the code set;

[0052] The candidate code segments corresponding to a preset number of similarities among the determined similarities are determined as candidate code segments that match any of the code segments to be tested.

[0053] In a possible implementation, the screening module is specifically used for:

[0054] Extracting structural features of each node in the graph structure corresponding to each candidate code snippet respectively;

[0055] For each structural feature of each candidate code snippet, perform the following operations:

[0056] For any structural feature, create an index entry for the structural feature, the index entry including a feature identifier of the structural feature, an identifier of the code snippet where the feature is located, and a position of the node in the graph structure;

[0057] Inserting the index entry into an inverted index structure;

[0058] Based on the extracted structural features of each node of each code snippet to be tested, searching for index entries matching the corresponding structural features from the inverted index structure respectively;

[0059] Based on the identifier of the code fragment included in each queried index entry, a same source code fragment corresponding to the code to be tested is obtained.

[0060] In a possible implementation, the screening module is specifically used for:

[0061] Determine candidate code segments corresponding to identifiers of code segments included in the queried index entries as reference code segments corresponding to the code to be tested;

[0062] Based on the graph structure of each code snippet to be tested and the graph structures corresponding to the reference code snippets, the same source code snippets corresponding to the code snippets to be tested are matched from the reference code snippets.

[0063] In a possible implementation, the extraction module is specifically used to extract the structural features of each node in the graph structure of any code snippet in the following manner, wherein the any code snippet is any code snippet to be tested, or any candidate code snippet:

[0064] Based on the syntax elements included in any one of the code snippets and the mapping relationship between syntax elements of multiple programming languages ​​and common node types, a graph structure of any one of the code snippets consisting of nodes of the common node type is obtained;

[0065] The following operation is performed for each node in the graph structure of any code snippet: based on the node type feature, semantic feature and context feature of any node, a structural feature of any node of any code snippet is obtained.

[0066] In a possible implementation, the extraction module is specifically used to:

[0067] Based on the index information of the semantic information of any node in the vocabulary set, the semantic feature of any node is obtained, wherein the semantic information includes part or all of the node type, variable name and function name, and is obtained by parsing the graph structure where the any node is located;

[0068] The initial context features of any of the nodes are set, and based on the node type features of any of the nodes, the semantic features and the initial context features, and the structural features of each of the neighboring nodes of any of the nodes, the context information of any of the nodes in any of the code fragments is aggregated to obtain the structural features of any of the nodes.

[0069] In a third aspect, an embodiment of the present application provides a computer device, including:

[0070] Memory, used to store computer programs or instructions;

[0071] A processor is used to execute the computer program or instructions in the memory so that any method described in the first aspect above is executed.

[0072] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any method described in the first aspect above.

[0073] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, the computer executes any method as described in the first aspect above.

[0074] In an embodiment of the present application, based on the term features of each code snippet to be tested in the code to be tested, candidate code snippets matching each code snippet to be tested are queried from multiple candidate code snippets included in the code set, wherein any term feature includes features corresponding to each term included in the code set in the corresponding code snippet to be tested, and each feature is determined based on the code complexity, comment density and code line weight of the corresponding code snippet to be tested, as well as the term frequency and inverse document frequency of the corresponding term; then, the structural features of each node in the graph structure of each code snippet to be tested are extracted; based on the extracted structural features and the structural features of each node corresponding to each candidate code snippet, the same source code snippet corresponding to the code to be tested is screened out from each candidate code snippet; in this way, the same source code snippet matching the code to be tested can be quickly and accurately located, thereby improving the real-time, high efficiency and applicability of the traceability of the code to be tested. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A schematic diagram of a flow chart of a method for detecting homology of a code in some embodiments of the present application;

[0076] Figure 2 A schematic diagram of a preliminary screening process in some embodiments of the present application;

[0077] Figure 3 A schematic diagram of a flow chart of determining a lexical feature of a code snippet to be tested in some embodiments of the present application;

[0078] Figure 4A This is a schematic diagram of a code snippet before reconstruction in some embodiments of the present application;

[0079] Figure 4B A schematic diagram of a reconstructed code snippet in some embodiments of the present application;

[0080] Figure 5 This is a schematic diagram of a specific process of preliminary screening in some embodiments of the present application;

[0081] Figure 6 A schematic diagram of a process of determining structural features of a node in some embodiments of the present application;

[0082] Figure 7 A schematic diagram of a specific determination process of a structural feature of a node in some embodiments of the present application;

[0083] Figure 8 A flowchart of a method for obtaining a code fragment with the same source code corresponding to the code to be tested in some embodiments of the present application;

[0084] Fig. 9 A schematic diagram of a specific process for determining the same source code fragment corresponding to the code to be tested in some embodiments of the present application;

[0085] Fig.10 A schematic diagram of the logical architecture of a code homology detection device in some embodiments of the present application;

[0086] Fig.11 This is a schematic diagram of the physical architecture of a terminal device in some embodiments of the present application. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0088] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein.

[0089] In addition, the term "and / or" in the embodiments of the present application describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0090] There are several code traceability detection methods in the related technology: First, traceability based on version control system, which mainly uses the submission history recorded by the version control system to track the evolution of the code; second, traceability based on static analysis, through static analysis of the source code, extracting code structure, control flow, data flow and other information, building a code dependency graph or abstract syntax tree (Abstract Syntax Tree, AST), and matching and tracing code snippets based on these structured information; third, traceability based on feature matching, which mainly extracts code features (such as keywords, variable names, function names, comments, etc.), and uses similarity calculation methods (such as TF-IDF, cosine similarity, etc.) to match code snippets; fourth, traceability based on graph model, which mainly represents the code as a graph structure, and matches and traces code snippets through graph matching algorithm.

[0091] However, most of the above methods match through a single level and cannot take into account both efficiency and accuracy. In particular, there are performance bottlenecks and many limitations in dealing with large-scale code bases, responding to code changes, and cross-language code tracing. For example, the huge amount of data is difficult to retrieve and analyze, and it is impossible to handle code refactoring and renaming. It is impossible to capture the deep semantics of the code, and it is impossible to handle cross-language and cross-platform situations. Excessive resource consumption leads to low accuracy and efficiency of code tracing.

[0092] In view of this, in order to solve the many limitations of the code tracing detection methods in the related art, an embodiment of the present application provides a code homology detection method, which is based on the term features of each code fragment to be tested in the code to be tested, and from the multiple candidate code fragments included in the code set, each candidate code fragment matching each code fragment to be tested is queried respectively, wherein any term feature includes features corresponding to each term included in the code set in the corresponding code fragment to be tested, and each feature is determined based on the code complexity, comment density and code line weight of the corresponding code fragment to be tested, as well as the term frequency and inverse document frequency of the corresponding term; then, the structural features of each node in the graph structure of each code fragment to be tested are extracted; based on the extracted structural features and the structural features of each node corresponding to each candidate code fragment, the homology code fragment corresponding to the code to be tested is screened out from each candidate code fragment.

[0093] In the present application, by obtaining the code complexity, comment density and code line weight of each code snippet to be tested in the code to be tested, as well as the term frequency and inverse document frequency of the corresponding term, the term feature of the corresponding code snippet to be tested is determined, and then based on the term feature, each candidate code snippet that matches the corresponding code snippet to be tested is queried from the multiple candidate code snippets included in the code set, and then based on the structural features of each node in the graph structure of each code snippet to be tested, and the structural features of each node corresponding to each candidate code snippet, the queried candidate code snippets are further screened, so that the candidate code snippets that may match, that is, the same source code snippet corresponding to the code to be tested, can be quickly located, which significantly improves the real-time, efficiency and applicability of code traceability and detection.

[0094] The preferred implementation modes of the present application are further described in detail below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments may be combined with each other if there is no conflict.

[0095] See also Figure 1 As shown, the embodiment of the present application provides a method for detecting homology of codes, and the specific process of the method includes but is not limited to the following:

[0096] Step 100: Based on the term features of each code snippet to be tested in the code to be tested, query each candidate code snippet that matches each code snippet to be tested from multiple candidate code snippets included in the code set; wherein any term feature includes features corresponding to each term included in the code set in the corresponding code snippet to be tested, and each feature is determined based on the code complexity, comment density and code line weight of the corresponding code snippet to be tested, as well as the term frequency and inverse document frequency of the corresponding term.

[0097] In some embodiments of this application, refer to Figure 2 As shown, when executing step 100, the following steps can be performed for each code segment to be tested:

[0098] Step 200: Based on any code snippet to be tested and each term, extract the term feature of the code snippet to be tested.

[0099] In some embodiments of the present application, when executing step 200, refer to Figure 3 As shown, this can be achieved by executing the following steps:

[0100] Step 2001: Based on the code set, the word frequency and inverse document frequency of each word in the code snippet to be tested are counted respectively;

[0101] Step 2002: Determine the code complexity of the code snippet to be tested based on the cyclomatic complexity of the code snippet to be tested;

[0102] Step 2003: determining the ratio of the number of comment lines to the number of code lines of the code snippet to be tested as the comment density of the code snippet to be tested;

[0103] Step 2004: determining a code line weight of the code segment to be tested based on the code line number of the code segment to be tested;

[0104] Step 2005: The statistically obtained word frequency and inverse document frequency of each word, code complexity and comment density, and the product of the code line weight are respectively determined as the features of the corresponding word to obtain the word feature of the code snippet to be tested.

[0105] Concerning step 2001, in some embodiments of the present application, it is possible to construct a term set based on the code set in advance by counting the terms included in the code set, wherein each term can be a keyword, a variable name, a function name, etc.; then, the term frequency and inverse document frequency of each term in the code snippet to be tested are counted separately.

[0106] In other embodiments, when executing step 2001, the terms included in the code set may be counted first to construct a term set, and then, based on the term set, the term frequency and inverse document frequency of each term in the code snippet to be tested may be counted respectively.

[0107] In some embodiments of the present application, the term frequency (TF or tf) and inverse document frequency (IDF or idf) of each term of the code snippet (d) to be tested in the code set (D) can be expressed by the following formula:

[0108] df(t)=log(N / |d∈D:t∈d|)

[0109] Among them, N is the total number of documents in the code set (that is, the total number of candidate code snippets), |{d∈D:t∈d}| is the number of documents containing term t (that is, the number of code snippets to be tested containing term t).

[0110] In step 2002, code complexity is a measure of the complexity of the code, which can usually be calculated by the structure, control flow and other indicators of the code. In some embodiments of the present application, cyclomatic complexity can be used to calculate the aforementioned code complexity, and the calculation formula of code complexity can be expressed as follows:

[0111] c(d)=1 / (1+f(cfg d ))

[0112] Among them, c(d) is the code complexity, f(cfg d ) is the cyclomatic complexity.

[0113] In this application, cyclomatic complexity is a measure of code complexity, usually used to evaluate the complexity of a program. It can be calculated based on the control flow graph of the program, where the nodes in the control flow graph represent basic blocks and the edges represent control flow paths. Specifically, the control flow graph can be constructed first, and the basic properties of the control flow graph can be calculated. Let Edge be the number of edges in the graph, Node be the number of nodes in the graph, and P be the number of connected components in the graph, which is usually 1, indicating a connected graph. In some embodiments of this application, the cyclomatic complexity f(cfg d ) can be expressed by the following formula:

[0114] f(cfg d )=Edge-Node+2P

[0115] In step 2003, the comment density measures the proportion of comments in the code. In some embodiments of the present application, the comment density r(d) can be expressed by the following formula:

[0116] r(d)=q(d) / z(d)

[0117] Where q(d) is the number of comment lines in the code snippet to be tested; z(d) is the number of code lines in the code snippet to be tested, that is, the total number of lines, including the number of code lines and the number of comment lines.

[0118] In step 2004, refactoring is a common task in the software development process. The refactored code may become more concise, but the functionality remains the same. For example, Figure 4A This is a schematic diagram of a code snippet before reconstruction in some embodiments of the present application. Figure 4B This is a schematic diagram of a reconstructed code snippet in some embodiments of the present application. Figure 4A and Figure 4B ,The code complexity and comment density before and after the refactoring are almost the same, but the number of code lines has changed significantly.,If we only rely on code complexity and comment density, we may not be able to effectively,distinguish the code snippets before and after the refactoring.

[0119] Therefore, in some embodiments of the present application, when determining the term features of the code snippet to be tested, a factor of Lines of Code (LOC) weight is introduced based on code complexity, comment density, and word frequency-inverse document frequency to better distinguish the code snippet from the original code snippet. Figure 4A and Figure 4B The code shown here improves the distinguishability of the code feature representation, so that different types of code fragments can be better and more accurately distinguished. The LOC weight (u) in some embodiments of the present application can be expressed by the following formula:

[0120] u=1+log(LOC(di ))

[0121] Among them, i is the sequence number, d i Characterizes the i-th code snippet to be tested; LOC(d i ) represents the code snippet to be tested i The above formula can be used to determine the LOC weight of a code snippet based on the number of lines of code in the code snippet, where the influence of the number of lines of code is adjusted by taking the logarithm to avoid excessive weight due to too large a number of lines.

[0122] Involving step 2005, after obtaining the word frequency and inverse document frequency, code complexity, comment density, and LOC weight of the code snippet to be tested through steps 2001 to 2004, step 2005 is executed to determine the word feature of the code snippet to be tested.

[0123] In some embodiments of the present application, the feature corresponding to any term in the term feature of the code snippet to be tested can be expressed by the following formula, such as tfidf * :

[0124] tfidf * =tf*idf*c(d)*r(d)*u

[0125] Among them, tf is the term frequency of any of the aforementioned terms in the code snippet to be tested; idf is the inverse document frequency of any of the aforementioned terms in the code snippet to be tested; c(d) is the code complexity of the code snippet to be tested; r(d) is the comment density of the code snippet to be tested; u is the LOC weight of the code snippet to be tested.

[0126] It should be noted that the above steps 2001 to 2004 may be executed in parallel or in any order. It is understandable that the execution order of steps 2001 to 2004 is not limited.

[0127] For example, suppose the term set consists of {t1,t2,...,t m}. For each code snippet d i and each term t j , calculate tfidf * (t j ,d i ). For each code snippet d i Based on the number of terms included in the term set, the dimension of the term feature of the code snippet is determined.

[0128] Assume that the constructed term features are represented by feature vector v i To represent it, its dimension is equal to the size of the term set, that is, the number of terms, denoted as m. Then, the vector v i The jth element of is tfidf* (t j ,d i ). If the code snippet d i does not contain the term t j , then tfidf * (t j ,d i )=0.

[0129] For another example, suppose there are two code snippets d1 and d2, and three terms {t1, t2, t3}.

[0130] Then, for d1, calculate tfidf * (t1,d1),tfidf * (t2,d1),tfidf * (t3,d1);

[0131] For d2, calculate tfidf * (t1,d2),tfidf * (t2,d2),tfidf * (t3,d2);

[0132] Then the feature vector representations of the term features of d1 and d2 can be respectively:

[0133] V1=(tfidf * (t1,d1),tfidf * (t2,d1),tfidf * (t3,d1))

[0134] V2=(tfidf * (t1,d2),tfidf * (t2,d2),tfidf * (t3,d2))

[0135] In the present application, the term features of each code snippet to be tested comprehensively consider multi-level code features such as code complexity, comment density, term frequency-inverse document frequency, and the number of lines of the code snippet. These features rely on the surface features of the code and are intended to enable subsequent operations to quickly filter out irrelevant code snippets from the code set based on the term features, thereby quickly narrowing the scope of candidate code snippets, thereby reducing the amount of computational effort for subsequent matching and significantly improving the real-time, efficiency, and applicability of code tracing and detection.

[0136] Step 210: Based on the term feature of the code snippet to be tested, query each candidate code snippet that matches the code snippet to be tested from a plurality of candidate code snippets included in the code set.

[0137] In some embodiments of the present application, when executing step 210, refer to Figure 5 This can be achieved by performing the following steps:

[0138] Step 2101: respectively determining the similarity between the term features of the code snippet to be tested and the term features corresponding to the plurality of candidate code snippets included in the code set;

[0139] Step 2102: Determine candidate code snippets corresponding to a preset number of similarities among the determined similarities as candidate code snippets that match the code snippet to be tested.

[0140] Concerning step 2101, in some embodiments of the present application, when executing step 2101, the aforementioned steps 2001 to 2005 can be used to determine the term features corresponding to the multiple candidate code fragments included in the code set; then, based on the similarity comparison method, the similarities between the term features of the code fragment to be tested and the term features corresponding to the multiple candidate code fragments included in the code set are respectively determined.

[0141] In other embodiments, the lexical features corresponding to the multiple candidate code snippets included in the above code set can be determined and stored in advance using the aforementioned steps 2001 to 2005. The specific method for determining the lexical features of each candidate code snippet is described in the aforementioned content and will not be repeated here. Then, when executing step 2101, the similarities between the lexical features of the code snippet to be tested and the lexical features corresponding to the multiple candidate code snippets included in the code set can be determined respectively according to the similarity comparison method.

[0142] Regarding step 2102, in some embodiments of the present application, the similarities can be sorted, such as sorting the similarities in order from large to small, and then the top K candidate code fragments are selected as the candidate code fragments that match the code fragment to be tested, thereby obtaining a preliminary screening result of the code fragment to be tested.

[0143] In some embodiments of the present application, in the preliminary screening stage, the distinctiveness of the code feature representation can be enhanced by calculating the code complexity and annotation density for each code snippet in the code to be tested and the code set; and when the code complexity and annotation density are not sufficient to fully reflect the code characteristics, the addition of LOC weights can more accurately distinguish different types of code snippets. In this way, the term frequency-inverse document frequency (TF-IDF) is weighted in combination with the cyclomatic complexity, annotation density, and LOC weight. Compared with the single-level matching in the related technologies, such as the traceability based on version control systems, the traceability based on feature matching, the traceability based on podium analysis, the traceability based on graph models, etc. in the related technologies, the distinctiveness of the term feature representation is enhanced, the accuracy and distinctiveness of the preliminary screening stage are significantly improved, and the rapid filtering of irrelevant code snippets is achieved.

[0144] Step 110: extracting structural features of each node in the graph structure of each code snippet to be tested;

[0145] In some embodiments of the present application, when executing step 110, each code segment to be tested is respectively Figure 6 and Figure 7 The processing flow shown (see subsequent content for details) is used to extract the structural features of each node in the graph structure of each code fragment to be tested.

[0146] Step 120: Based on the extracted structural features and the structural features of each node corresponding to each candidate code segment, the same source code segment corresponding to the code to be tested is selected from each candidate code segment.

[0147] In some embodiments of the present application, when executing step 120, refer to Figure 8 As shown, this can be achieved by executing the following steps:

[0148] Step 1201: extracting structural features of each node in the graph structure corresponding to each of the candidate code snippets.

[0149] In some embodiments of the present application, when executing step 1201, the Figure 6 and Figure 7 The processing flow shown (see subsequent content for details) extracts the structural features of each node in the graph structure corresponding to the above-mentioned candidate code fragments. It should be noted that the above-mentioned candidate code fragments are executed in step 100, and are candidate code fragments that match each code fragment to be tested queried from the code set.

[0150] Step 1202: Perform the following operations for each structural feature of each candidate code snippet:

[0151] Step 12021: for any structural feature, create an index entry for the structural feature, the index entry including the feature identifier of the structural feature, the identifier of the code snippet where the feature resides, and the position of the node in the graph structure;

[0152] Step 12022: Insert the index entry into the inverted index structure.

[0153] In some embodiments of the present application, each structural feature can be represented as one or more feature vectors. When executing step 1202, an index entry is created for each structural feature. An index entry includes but is not limited to the feature identifier of the structural feature, the identifier of the code snippet where it is located, and the position of the associated node in the graph structure; then, the index entry is inserted into the inverted index structure, wherein the inverted index is a data structure, often referred to as a directional index, an insertion file or a reverse file. Compared with the forward index in the related art, the inverted index stores content that is not a mapping of documents (code snippets) to words (features), but a mapping relationship between words (features) and documents (code snippets). In the actual query stage, the index is not organized according to documents or code snippets, but according to features or keywords, which improves query efficiency and enables efficient retrieval of documents or code snippets containing specific features or keywords.

[0154] It should be noted that in the process of creating index entries, if for the structural feature currently being processed, an index entry corresponding to the feature identifier of the structural feature already exists in the inverted index structure, then only the identifier of the code fragment where the structural feature currently being processed is located, and the position in the graph structure of the node associated with the structural feature currently being processed are integrated into the index entry corresponding to the existing feature identifier in the inverted index structure, without the need to rebuild a new index entry.

[0155] For example, suppose there is a structural feature called "variable name x", whose index entry can be expressed as:

[0156] Feature identifier: variable name x;

[0157] The identifier of the code fragment: code_fragment_1, code_fragment_2, ...

[0158] The position of the node in the graph structure: position_1, position_2, ...

[0159] These index entries are inserted into the inverted index structure for subsequent fast query and matching.

[0160] Step 1203: Based on the extracted structural features of each node of each code snippet to be tested, index entries matching the corresponding structural features are searched from the inverted index structure.

[0161] In some embodiments of the present application, when executing step 1203, the structural features of each node of the code snippet to be tested are queried in the above-mentioned inverted index structure to obtain index entries that match the structural features of each node.

[0162] Step 1204: Based on the identifier of the code snippet included in each queried index entry, obtain the same source code snippet corresponding to the code to be tested.

[0163] In some embodiments of the present application, when executing step 1204, refer to Fig. 9 As shown, the following steps can be performed:

[0164] Step 900: Determine candidate code segments corresponding to identifiers of code segments included in the queried index entries as reference code segments corresponding to the code to be tested.

[0165] In some embodiments of the present application, when executing step 900, the identifier of the code fragment in each index entry that matches the queried structural feature is obtained, and the candidate code fragments corresponding to the obtained identifiers of each code fragment are respectively determined as reference code fragments of the code to be tested, wherein, for the structural features of any node of each code fragment to be tested, at least one index entry may be queried, and each index entry may include the identifier of at least one code fragment, then, each obtained candidate code fragment is respectively determined as a reference code fragment of the code to be tested, thereby obtaining each reference code fragment of the code to be tested.

[0166] In some embodiments of the present application, after executing step 900 to obtain each reference code fragment, the reference code fragment can also be sorted according to the frequency of feature occurrence or other statistical information, and the sorted code fragments can be matched in a more fine-grained manner, that is, executing step 910, to improve the matching accuracy.

[0167] Step 910: Based on the graph structure of each code snippet to be tested and the graph structure corresponding to each reference code snippet, match the same source code snippet corresponding to the code to be tested from each reference code snippet.

[0168] In some embodiments of the present application, when executing step 910, a graph matching algorithm can be used to perform a more fine-grained match on each reference code fragment, that is, based on the graph structure of each code fragment to be tested, and the graph structure of each reference code fragment, similar or identical reference code fragments are found through graph matching, and the same source code fragment corresponding to the code to be tested is obtained.

[0169] In some embodiments of the present application, the above-mentioned graph matching algorithm can be at least one algorithm of subgraph isomorphism, graph edit distance, etc., but it should be noted that the present application does not make specific limitations.

[0170] In some other embodiments of the present application, an efficient graph isomorphism algorithm may be used to quickly check the graph structure for isomorphism, so as to improve the matching rate.

[0171] In other embodiments, considering the limitation of computing load, each reference code fragment corresponding to the identifier of the code fragment included in each obtained index entry may also be determined as the same source code fragment corresponding to the code to be tested.

[0172] The following is a detailed description of the processing flow of how to extract the structural features of each node in the graph structure of each code snippet to be tested and each candidate code snippet. For the sake of ease of description, only one code snippet is used as an example to describe in detail the processing flow of extracting the structural features of each node in the graph structure of the code snippet in this application. It should be noted that the code snippet can be any code snippet to be tested in the aforementioned code to be tested, or any candidate code snippet queried above.

[0173] In some embodiments of this application, refer to Figure 6 As shown, the structural features of each node in the graph structure of any code snippet can be extracted by executing the following steps:

[0174] Step 600: Based on the syntax elements included in the code snippet and the mapping relationship between syntax elements of multiple programming languages ​​and common node types, a graph structure of the code snippet consisting of nodes of the common node type is obtained.

[0175] In some embodiments of the present application, the above-mentioned graph structure can be an abstract syntax tree structure, or a control flow graph, a data flow graph, etc., or other forms of graph structures, which are not specifically limited in the present application.

[0176] Taking the abstract syntax tree as an example, an abstract syntax tree (AST) can be represented as an ordered tree structure, where each node represents a construct element in the source code, or called a syntax element. Let T = (V, E) represent an AST, where V is a node set, each node represents a construct element in the source code; E is an edge set, each edge represents the hierarchical relationship between construct elements.

[0177] In order to map the code structures of different programming languages ​​to a unified intermediate representation, this application sets a cross-language mapping rule, that is, the mapping relationship between the syntax elements of the aforementioned multiple programming languages ​​and the common node type. Taking the abstract syntax tree as an example, the specific method can be to define the mapping relationship between the syntax elements of different programming languages ​​and the common AST node type. Let L iRepresents the i-th programming language, whose grammatical element set is ε i , the set of universal AST node types is V'. The mapping relationship can be characterized by defining a mapping function to map the syntax elements to the node types of the universal AST.

[0178] For example, for the following if statements in Java and Python:

[0179] Java: if(condition){statements}

[0180] Python: if condition:statements

[0181] In some embodiments of the present application, a mapping relationship may be defined, that is, mapping the if statements of the two to a unified AST node type, such as IF_STATEMENT.

[0182] Through the above mapping relationship, the AST of the code snippet can be generated.

[0183] In some embodiments, when generating AST, unnecessary nodes may be removed, and only key structural nodes may be retained, for example, empty statements, simple expression nodes, etc. This may reduce the size of AST, thereby speeding up the matching process.

[0184] Through the above processing, each code snippet can be parsed into an AST, and each node represents a grammatical structure; and through the cross-language mapping rules (i.e., the aforementioned mapping relationship), the AST nodes of different programming languages ​​are mapped to a unified set of general AST node types, that is, the above-mentioned cross-language mapping function can standardize the grammatical elements of different languages ​​into a unified representation, so that in the subsequent matching operation (i.e., the aforementioned step 120), efficient matching can be achieved to improve the matching rate.

[0185] Step 610: Perform the following operation for each node in the graph structure of the code snippet: Based on the node type feature, semantic feature and context feature of any node, obtain the structural feature of the node of the code snippet.

[0186] In some embodiments of the present application, when executing step 610, refer to Figure 7 As shown, the corresponding functions can be realized by executing the following steps:

[0187] Step 6101: Based on the index information of the node's semantic information in the vocabulary, obtain the semantic features of the node, wherein the semantic information includes part or all of the node type, variable name, and function name, and is obtained by parsing the graph structure where the node is located.

[0188] In some embodiments of the present application, the semantic information of the node may include but is not limited to the node type, variable name, function name, etc. This information can be obtained by parsing the AST. The specific extraction time can be during the process of generating the AST, or it can be extracted after the AST is generated. This application does not make specific limitations.

[0189] In some embodiments of the present application, when executing step 6101, the extracted semantic information is mapped to the index in the above vocabulary set to obtain the index information, and then the word embedding vector corresponding to the index information is obtained to obtain the semantic feature of the node, so as to better capture the semantic information. For example, if the index information in the vocabulary set is i, then the semantic feature V s V can be the i-th row of the word embedding matrix corresponding to the vocabulary set, denoted by V s =E[i], where E[i] represents the vector of the i-th row in the word embedding matrix. For another example, if the corresponding index information of "variable_name" in the vocabulary is 5, it is mapped to 5. According to the mapped index information (5), the word embedding vector (V) of the 5th row is obtained from the word embedding matrix. s =E[5]).

[0190] It should be noted that a vocabulary is a set of all possible terms, which can be keywords, variable names, function names, etc. In code processing, a vocabulary is usually stored in the form of a vocabulary table, which usually contains all unique terms that appear in the source code library and the code to be tested. In some embodiments of the present application, the extracted semantic information is mapped to an index in the vocabulary, and this information can be converted into a unified digital representation, which can facilitate subsequent calculations and processing, reduce the dimension of the feature vector, and improve calculation efficiency.

[0191] Step 6102: Set the initial context features of the node, and based on the node type features of the node, the above semantic features and the above initial context features, as well as the structural features of each neighbor node of the node, aggregate the context information of the node in the code fragment to obtain the structural features of the node.

[0192] In some embodiments of the present application, the node type feature is a feature derived from the node type of the node, and may include more information or context of the node type. For example, the node type may include, but is not limited to, some or all of the following:

[0193] 1. Encoding representation of node types, such as one-hot encoding or embedding vectors;

[0194] 2. Metadata of the node type, such as whether it is a control flow node, a data flow node, etc.;

[0195] 3. The association between node types and other node types;

[0196] 4. Statistical characteristics of node types, such as frequency of occurrence.

[0197] Compared with node type, node type refers to the specific type represented by each node in the code graph structure. It is a specific classification label and a basic description. For example, in AST, nodes can represent different programming language constructs, such as function definitions, variable declarations, operators, etc.

[0198] Compared with the semantic information of semantic features, semantic information focuses on understanding code behavior and functions. For example, the semantic information of a function call node may involve the function's parameters, return values, and its role in the program;

[0199] Node type features focus on the construction and structure of the code. For example, the node type features of the same function call node may include the fact that it is a function call, as well as its position in the AST and its relationship with other nodes. In this way, describing the code from different angles can provide different perspectives, provide a more comprehensive understanding, and help to more accurately aggregate the contextual information of the node.

[0200] In this application, context features describe neighbor node information of a node. In some embodiments of this application, it can be obtained by aggregating features of neighbor nodes. The following is a detailed description of the context information of a specific aggregated node.

[0201] Assume that the initial context feature of each node can be represented in vector form, denoted by V c ; Then, the features are updated layer by layer through the graph network to obtain the final context features, denoted as V c '.

[0202] Then, based on the node type feature, semantic feature and initial context feature of the node, the initial feature of the node is formed. In some embodiments of the present application, the initial feature can be represented by the following feature vector:

[0203]

[0204] Among them, V n Indicates node type characteristics, such as V n =[0,0,...,1,...,0]; [;] represents the vector connection operation, that is, through the vector connection operation [;], the node type feature vector V n , semantic feature vector V s and the initial context feature vector V c Connect to form the initial features The vector representation of .

[0205] In some embodiments of the present application, the aggregation of the context information of the node in the code snippet can be achieved through a graph convolutional network (GCN). In GCN, multiple layers of GCN are used to capture deeper graph structure information. Among them, each layer of GCN can update the node features by aggregating neighbor node features. The aggregation of multiple layers of GCN is to propagate node features on the basis of multiple layers of GCN and continuously update node features at each layer. For the node update of each layer, the following operations are performed:

[0206] Operation 1: Initialize node features: First input the AST node initial feature vector The initial features of each node include but are not limited to node type, semantic information and initial context information, that is, the aforementioned node type features, semantic features and initial context features.

[0207] Operation 2: Aggregate neighbor node features: For each node v, calculate the feature aggregation of its neighbor node u:

[0208]

[0209] Among them, N(v) represents the set of neighbor nodes of node v. Represents the feature vector of neighbor node u at the k-1 layer. vu represents the normalization coefficient; in some embodiments of the present application, the normalization coefficient coff vu It is used to balance the influence of different nodes on feature aggregation. For each pair of nodes (v, u), the normalization coefficient coff can be calculated by the following formula: vu :

[0210] coff vu =|N(v)| 1 / 2 .

[0211] For example, suppose there is a node v whose neighbor node set is N(v) = {u1, u2, u3}. The feature vector of each neighbor node at the k-1th layer is

[0212] Then, calculate the normalization coefficient coff vu :

[0213] For each neighbor node u i , the normalized coefficient is:

[0214]

[0215] Again, calculate the aggregate eigenvector:

[0216] The aggregate feature vector of node v at layer k is:

[0217]

[0218] From the formula of operation 2 and the above example, it can be seen that the aggregation process is achieved by weighted summation of the feature vectors of neighboring nodes.

[0219] In some embodiments of the present application, after the aggregation and updating of the node features, the aggregated features are linearly transformed and biased. In some embodiments of the present application, the linear transformation and bias processing can be identified by the following formula:

[0220]

[0221] in, represents the eigenvector after linear transformation and bias; represents the weight matrix of the kth layer, bias (k) represents the bias vector of the kth layer. In some embodiments of the present application, bias (k) They can be learned through training respectively, and the weight matrix and bias vector of each layer are independent and can be optimized according to the data during the training process.

[0222] Then, the linearly transformed feature vector is used to obtain the structural characteristics of the node through the activation function

[0223]

[0224] Among them, σ represents the activation function, which is used to introduce nonlinear features.

[0225] In some embodiments of the present application, after multi-layer GCN aggregation, the feature vector (i.e., structural feature) of each node will gradually include information of more distant neighboring nodes, thereby obtaining the final structural feature of the node, i.e., the final structural feature that aggregates the contextual information of the node in the code snippet.

[0226] In some embodiments, after obtaining the feature vector representation of the structural features of each node in the code snippet, the feature vector representations of all nodes included in the code snippet can also be aggregated into an embedded representation of the entire graph structure to facilitate subsequent operations, such as graph matching, graph isomorphism checking, etc.

[0227] In this application, by constructing a graph structure and multi-layer GCN aggregation, deeper graph structure information can be captured, thereby improving the accuracy of subsequent matching code snippets, and further significantly improving the real-time, efficiency and applicability of code tracing.

[0228] In this application, on the basis of preliminary screening, the process of further refinement and precise matching through structural features specifically includes several aspects, namely, feature combination matching, context matching, semantic matching, structural matching and statistical information matching. Among them, for feature combination matching, for each preliminary matched code fragment, the combination consistency of multiple features is checked, such as not only considering the variable name x (node ​​type), but also considering its context and semantic information; for context matching, the consistency of context information in the code fragment is confirmed, such as checking whether the context of the function or class where the variable x is located is the same; for semantic matching, the functional and logical consistency of the code fragment is checked; for structural matching, structural matching is performed to ensure that the structure of the code fragment is similar or identical; for statistical information matching, it can be sorted and screened according to the frequency of occurrence of features or other statistical information to further improve the accuracy of matching. In this way, through the screening of multi-level code feature representation (such as the term feature, structural feature, etc. determined based on the surface features of the multi-level code mentioned above), the same source code fragment corresponding to the final code to be tested is obtained, which significantly improves the accuracy, real-time, efficiency and applicability of code tracing.

[0229] Based on the same inventive concept, refer to Fig.10 As shown, in an embodiment of the present application, a device for detecting homology of a code is provided, comprising:

[0230] A query module 1010 is used to query, based on the term features of each code segment to be tested in the code to be tested, from a plurality of candidate code segments included in the code set, candidate code segments that match each code segment to be tested;

[0231] An extraction module 1020 is used to extract structural features of each node in the graph structure of each code snippet to be tested;

[0232] A screening module 1030 is used to screen out the same source code fragments corresponding to the code to be tested from the candidate code fragments based on the extracted structural features and the structural features of each node corresponding to the candidate code fragments;

[0233] Among them, any term feature includes features corresponding to each term included in the code set in the corresponding code snippet to be tested, and each feature is determined based on the code complexity, comment density and code line weight of the corresponding code snippet to be tested, as well as the term frequency and inverse document frequency of the corresponding term.

[0234] In a possible implementation, the query module 1010 is specifically used to:

[0235] Perform the following operations for each code snippet to be tested:

[0236] Based on any code snippet to be tested and the terms, extracting term features of the code snippet to be tested;

[0237] Based on the term feature of any code snippet to be tested, candidate code snippets matching any code snippet to be tested are searched from a plurality of candidate code snippets included in the code set.

[0238] In a possible implementation, the query module 1010 is specifically used to:

[0239] Based on the code set, respectively counting the word frequency and inverse document frequency of each term in any code snippet to be tested, and based on the cyclomatic complexity of any code snippet to be tested, determining the code complexity of any code snippet to be tested;

[0240] Determine the ratio of the number of comment lines to the number of code lines of any code snippet to be tested as the comment density of any code snippet to be tested, and determine the code line weight of any code snippet to be tested based on the number of code lines of any code snippet to be tested;

[0241] The statistically obtained word frequency and inverse document frequency of each term, the code complexity and the comment density, and the product of the code line weight are respectively determined as the features of the corresponding term to obtain the term features of any code snippet to be tested.

[0242] In a possible implementation, the query module 1010 is specifically used to:

[0243] Respectively determining the similarity between the term features of any of the code snippets to be tested and the term features corresponding to the plurality of candidate code snippets included in the code set;

[0244] The candidate code segments corresponding to a preset number of similarities among the determined similarities are determined as candidate code segments that match any of the code segments to be tested.

[0245] In a possible implementation, the screening module 1030 is specifically used to:

[0246] Extracting structural features of each node in the graph structure corresponding to each candidate code snippet respectively;

[0247] For each structural feature of each candidate code snippet, perform the following operations:

[0248] For any structural feature, create an index entry for the structural feature, the index entry including a feature identifier of the structural feature, an identifier of the code snippet where the feature is located, and a position of the node in the graph structure;

[0249] Inserting the index entry into an inverted index structure;

[0250] Based on the extracted structural features of each node of each code snippet to be tested, searching for index entries matching the corresponding structural features from the inverted index structure respectively;

[0251] Based on the identifier of the code fragment included in each queried index entry, a same source code fragment corresponding to the code to be tested is obtained.

[0252] In a possible implementation, the screening module 1030 is specifically used to:

[0253] Determine candidate code segments corresponding to identifiers of code segments included in the queried index entries as reference code segments corresponding to the code to be tested;

[0254] Based on the graph structure of each code snippet to be tested and the graph structures corresponding to the reference code snippets, the same source code snippets corresponding to the code snippets to be tested are matched from the reference code snippets.

[0255] In a possible implementation, the extraction module 1020 is specifically used to extract the structural features of each node in the graph structure of any code snippet in the following manner, wherein the any code snippet is any code snippet to be tested, or any candidate code snippet:

[0256] Based on the syntax elements included in any one of the code snippets and the mapping relationship between syntax elements of multiple programming languages ​​and common node types, a graph structure of any one of the code snippets consisting of nodes of the common node type is obtained;

[0257] The following operation is performed for each node in the graph structure of any code snippet: based on the node type feature, semantic feature and context feature of any node, a structural feature of any node of any code snippet is obtained.

[0258] In a possible implementation, the extraction module 1020 is specifically used to:

[0259] Based on the index information of the semantic information of any node in the vocabulary set, the semantic feature of any node is obtained, wherein the semantic information includes part or all of the node type, variable name and function name, and is obtained by parsing the graph structure where the any node is located;

[0260] The initial context features of any of the nodes are set, and based on the node type features of any of the nodes, the semantic features and the initial context features, and the structural features of each of the neighboring nodes of any of the nodes, the context information of any of the nodes in any of the code fragments is aggregated to obtain the structural features of any of the nodes.

[0261] See also Fig.11 As shown, a computer device is provided in an embodiment of the present application, and the computer device can implement the functions of the aforementioned method, referring to Fig.11 The computer device may include: at least one processor 111 and a memory 112 connected to the at least one processor 111.

[0262] The specific connection medium between the processor 111 and the memory 112 is not limited in the embodiment of the present application. Fig.11 In the example, the processor 111 and the memory 112 are connected via the bus 110. The bus 110 is Fig.11 The connections between other components are shown in bold lines, and are not intended to be limiting. The bus 110 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Fig.11 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. Alternatively, the processor 111 can also be called a controller, and there is no limitation on the name.

[0263] In the embodiment of the present application, the memory 112 stores a computer program or instruction that can be executed by at least one processor 111. The at least one processor 111 can perform the method discussed above by executing the computer program or instruction stored in the memory 112. The processor 111 can implement the functions of the aforementioned device.

[0264] In a possible implementation, the processor 111 may include one or more processing units, and the processor 111 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 111. In some embodiments, the processor 111 and the memory 112 may be implemented on the same chip, and in other embodiments, they may also be implemented separately on separate chips.

[0265] The processor 111 may be a general-purpose processor, for example, a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0266] The memory 112 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 112 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 112 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 112 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0267] By designing and programming the processor 111, the code corresponding to the method described in the above embodiment can be fixed into the chip, so that the chip can execute the steps of the method of the above embodiment when running. How to design and program the processor 111 is a technology well known to those skilled in the art and will not be described in detail here.

[0268] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the processor is enabled to execute any one of the methods in the above embodiments.

[0269] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a device, the program code is used to enable the device to execute the steps of the method according to various exemplary implementations of the present application described above in this specification.

[0270] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0271] The present application is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0272] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0273] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0274] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for detecting homology of codes, characterized in that: include: Based on the lexical features of each code fragment to be tested in the code to be tested, from a plurality of candidate code fragments included in the code set, querying each candidate code fragment that matches each code fragment to be tested respectively; Extracting structural features of each node in the graph structure of each code snippet to be tested; Based on the extracted structural features and the structural features of each node corresponding to each candidate code fragment, the same source code fragment corresponding to the code to be tested is selected from each candidate code fragment; Among them, any term feature includes features corresponding to each term included in the code set in the corresponding code snippet to be tested, and each feature is determined based on the code complexity, comment density and code line weight of the corresponding code snippet to be tested, as well as the term frequency and inverse document frequency of the corresponding term.

2. The method according to claim 1, characterized in that The method of searching for candidate code fragments matching each code fragment to be tested from a plurality of candidate code fragments included in the code set based on the term feature of each code fragment to be tested in the code to be tested includes: Perform the following operations for each code snippet to be tested: Based on any code snippet to be tested and the terms, extracting term features of the code snippet to be tested; Based on the term feature of any code snippet to be tested, candidate code snippets matching any code snippet to be tested are searched from a plurality of candidate code snippets included in the code set.

3. The method according to claim 2, characterized in that The step of extracting the term feature of any code snippet to be tested based on any code snippet to be tested and each term includes: Based on the code set, respectively counting the word frequency and inverse document frequency of each term in any code snippet to be tested, and based on the cyclomatic complexity of any code snippet to be tested, determining the code complexity of any code snippet to be tested; Determine the ratio of the number of comment lines to the number of code lines of any code snippet to be tested as the comment density of any code snippet to be tested, and determine the code line weight of any code snippet to be tested based on the number of code lines of any code snippet to be tested; The statistically obtained word frequency and inverse document frequency of each term, the code complexity and the comment density, and the product of the code line weight are respectively determined as the features of the corresponding term to obtain the term features of any code snippet to be tested.

4. The method according to claim 2, characterized in that The step of searching for candidate code snippets matching any of the code snippets to be tested from a plurality of candidate code snippets included in the code set based on the term feature of any of the code snippets to be tested comprises: Respectively determining the similarity between the term features of any of the code snippets to be tested and the term features corresponding to the plurality of candidate code snippets included in the code set; The candidate code segments corresponding to a preset number of similarities among the determined similarities are determined as candidate code segments that match any of the code segments to be tested.

5. The method according to any one of claims 1 to 4, characterized in that: The method of selecting the same source code fragment corresponding to the code to be tested from the candidate code fragments based on the extracted structural features and the structural features of each node corresponding to each candidate code fragment comprises: Extracting structural features of each node in the graph structure corresponding to each candidate code snippet respectively; For each structural feature of each candidate code snippet, perform the following operations: For any structural feature, create an index entry for the structural feature, the index entry including a feature identifier of the structural feature, an identifier of the code snippet where the feature is located, and a position of the node in the graph structure; Inserting the index entry into an inverted index structure; Based on the extracted structural features of each node of each code snippet to be tested, searching for index entries matching the corresponding structural features from the inverted index structure respectively; Based on the identifier of the code fragment included in each queried index entry, a same source code fragment corresponding to the code to be tested is obtained.

6. The method according to claim 5, characterized in that The identification of the code snippet included in each queried index entry is used to obtain a code snippet of the same source code corresponding to the code to be tested, including: Determine candidate code segments corresponding to identifiers of code segments included in the queried index entries as reference code segments corresponding to the code to be tested; Based on the graph structure of each code snippet to be tested and the graph structures corresponding to the reference code snippets, the same source code snippets corresponding to the code snippets to be tested are matched from the reference code snippets.

7. The method according to claim 5, characterized in that The structural features of each node in the graph structure of any code snippet are extracted in the following manner, wherein the any code snippet is any code snippet to be tested, or any candidate code snippet: Based on the syntax elements included in any one of the code snippets and the mapping relationship between syntax elements of multiple programming languages ​​and common node types, a graph structure of any one of the code snippets consisting of nodes of the common node type is obtained; The following operation is performed for each node in the graph structure of any code snippet: based on the node type feature, semantic feature and context feature of any node, a structural feature of any node of any code snippet is obtained.

8. The method according to claim 7, characterized in that The obtaining the structural features of any node of any code snippet based on the node type feature, semantic feature and context feature of any node includes: Based on the index information of the semantic information of any node in the vocabulary set, the semantic feature of any node is obtained, wherein the semantic information includes part or all of the node type, variable name and function name, and is obtained by parsing the graph structure where the any node is located; The initial context features of any of the nodes are set, and based on the node type features of any of the nodes, the semantic features and the initial context features, and the structural features of each of the neighboring nodes of any of the nodes, the context information of any of the nodes in any of the code fragments is aggregated to obtain the structural features of any of the nodes.

9. A computer device, characterized in that: include: Memory, used to store computer programs or instructions; A processor is used to execute the computer program or instructions in the memory so that the method according to any one of claims 1 to 8 is performed.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 8.

Images