A large-scale source code similarity detection method, system and device

Abstract

The present application discloses a large-scale source code similarity detection method, system and device, which is applied to the field of large-scale source code similarity detection technology. Receive the project to be detected sent by the user terminal and process it to obtain the first token feature of each function block in the project to be detected; compare the first token feature with the second token feature of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarity. Collect open source projects through the code collection service, and then generate a token feature library through the inverted index algorithm. Therefore, in the token feature library, fast search can be achieved in massive data based on the inverted index algorithm, and then large-scale source code similarity detection between the first token feature and massive data can be achieved through the token feature library.

Description

Technical Field

[0001] The present invention relates to the technical field of large-scale source code similarity detection, and in particular to a large-scale source code similarity detection method, system and device. Background Art

[0002] Large-scale source code similarity detection is a basic research topic in the field of software engineering, and plays an important role in application fields such as software project maintenance, plagiarism detection, malware detection, and vulnerability search. In the process of software project development, developers will reuse open source software code to improve code quality in order to speed up the development process. Generally, they will reuse existing code by copying and pasting and modifying it, and automatically generate code snippets by IDE. Studies have shown that in general software systems, there is about 7-24% source code similarity, and in some software it even reaches 50%. Therefore, source code similarity detection is needed to reduce the many problems caused by excessive code cloning to software project maintenance. For example, reusing a code snippet containing unknown vulnerabilities will bring unknown security risks to the software. However, existing source code similarity detection methods can only cope with detection in small-scale data and cannot handle massive data. Summary of the invention

[0003] The purpose of this application is to provide a large-scale source code similarity detection method, system and device, which is applied to the field of large-scale source code similarity detection technology. Open source projects are collected through code collection services, and then processed by an inverted index algorithm to generate a token feature library. Therefore, in the token feature library, a fast search in a large amount of data can be achieved based on the inverted index algorithm, and then the large-scale source code similarity detection between the first token feature and the massive data is achieved through the token feature library.

[0004] To solve the above technical problems, the present application provides a large-scale source code similarity detection method, including:

[0005] Receiving the item to be detected sent by the user terminal and processing to obtain the first token feature of each function block in the item to be detected;

[0006] The first token feature is compared with the second token features of all function blocks in a token feature library to obtain all third token features similar to the first token feature and corresponding similarities. The token feature library is generated by processing the open source projects collected by the code collection service with an inverted index algorithm.

[0007] Preferably, receiving the item to be detected sent by the user terminal and processing to obtain the first token feature of each function block in the item to be detected includes:

[0008] Receive the items to be detected sent by the user terminal and perform function slicing, data preprocessing and function tokenization on the items to be detected in sequence to obtain the first token feature of each function block in the items to be detected.

[0009] Preferably, after comparing the first token feature with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities, the method further includes:

[0010] The first token feature, all the third token features and the corresponding similarities are sent to the user terminal, and all the third token features are sorted from large to small based on the corresponding similarities.

[0011] Preferably, comparing the first token feature with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities includes:

[0012] Based on big data parallel computing technology, the first token feature is compared with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities.

[0013] Preferably, the big data platform receives the open source projects collected by the code collection service and generates the token feature library after processing through the inverted index algorithm.

[0014] Preferably, comparing the first token feature with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities includes:

[0015] Obtain the words contained in the first token feature;

[0016] Based on the inverted index algorithm, all the third token features are obtained from the second token features of all function blocks in the token feature library, and the number of types of the words included in the third token features is greater than a preset number;

[0017] Obtain all key words from all the words by using the TF-IDF algorithm, wherein the key words are words whose TF-IDF values ​​in the set of the first token feature and all the third token features are greater than a preset TF-IDF value;

[0018] By the formula S = a / b*cos<Vt,Vi> Get the similarity corresponding to the i-th third token feature, where Vt is a first word frequency vector based on the frequency of all the key words in the first token feature, Vi is a second word frequency vector based on the frequency of all the key words in the i-th third token feature, cos<Vt,Vi> is the cosine similarity between the Vt and the Vi, a is the number of the key words contained in the i-th third token feature, and b is the total number of all the key words.

[0019] Preferably, the big data platform receives the open source projects collected by the code collection service and processes the open source projects through an inverted index algorithm to sequentially generate a function dictionary table, a word dictionary table and an inverted index table to construct the token feature library;

[0020] Based on the inverted index algorithm, all the third token features are obtained from the second token features of all function blocks in the token feature library, and the number of types of the words contained in the third token features is greater than a preset number, including:

[0021] Find the word ID of each word through the word dictionary table;

[0022] Finding a function inverted list corresponding to each word ID through the inverted index table, wherein the function inverted list is composed of a vector (feature ID, frequency), wherein the feature ID is the ID of the second token feature containing the word, and the frequency is the number of the words contained in the second token feature;

[0023] Based on the function inverted list corresponding to each word ID, obtain all the words and corresponding frequencies contained in each second token feature;

[0024] All the second token features whose number of categories containing the word is greater than a preset number are used as all the third token features.

[0025] Preferably, all key words are obtained from all the words through the TF-IDF algorithm, wherein the key words are words whose TF-IDF values ​​are greater than a preset TF-IDF value in the set of the first token feature and all the third token features, including:

[0026] Through the formula TF-IDF=TF W *IDF W Get the TF-IDF value of the Wth word, the TF W =cW / d, IDF W =log(e / (f+1)), the c W is the frequency of the Wth word in the first token feature, d is the total number of the words, e is the sum of the number of the first token feature and all the third token features, and f is the sum of the number of features containing the Wth word in the first token feature and all the third token features;

[0027] All words whose TF-IDF values ​​are greater than a preset TF-IDF value are taken as all the key words.

[0028] In order to solve the above technical problems, the present application also provides a large-scale source code similarity detection system, including:

[0029] A token feature acquisition unit, configured to receive an item to be detected sent by a user terminal and process the item to be detected to obtain a first token feature of each function block in the item to be detected;

[0030] A similarity acquisition unit is used to compare the first token feature with the second token features of all function blocks in a token feature library to obtain all third token features similar to the first token feature and corresponding similarities, wherein the token feature library is generated by processing the open source projects collected by the code collection service with an inverted index algorithm.

[0031] In order to solve the above technical problems, the present application also provides a large-scale source code similarity detection device, comprising:

[0032] Memory for storing computer programs;

[0033] A processor is used to execute the computer program to implement the steps of the large-scale source code similarity detection method.

[0034] The present application provides a large-scale source code similarity detection method, system and device, which is applied to the field of large-scale source code similarity detection technology. Receive the project to be detected sent by the user terminal and process it to obtain the first token feature of each function block in the project to be detected; compare the first token feature with the second token feature of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities. Collect open source projects through the code collection service, and then generate a token feature library through the inverted index algorithm. Therefore, in the token feature library, fast search can be achieved in massive data based on the inverted index algorithm, and then large-scale source code similarity detection between the first token feature and massive data can be achieved through the token feature library. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the prior art and the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 A flowchart of a large-scale source code similarity detection method provided by this application;

[0037] Figure 2 A flowchart of a large-scale source code similarity detection provided by this application;

[0038] Figure 3 A device deployment diagram for large-scale source code similarity detection provided by this application;

[0039] Figure 4 A flowchart for building a token feature library provided for this application;

[0040] Figure 5 A flowchart of an algorithm for obtaining large-scale source code similarity provided by this application;

[0041] Figure 6 A schematic diagram of the structure of a large-scale source code similarity detection system provided by this application;

[0042] Figure 7 A schematic diagram of the structure of a large-scale source code similarity detection device provided by the present application. DETAILED DESCRIPTION

[0043] The core of this application is to provide a large-scale source code similarity detection method, system and device, which is applied to the field of large-scale source code similarity detection technology. Open source projects are collected through code collection services, and then processed by an inverted index algorithm to generate a token feature library. Therefore, in the token feature library, a fast search in a large amount of data can be achieved based on the inverted index algorithm, and then the large-scale source code similarity detection between the first token feature and the massive data can be achieved through the token feature library.

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0045] Figure 1 A flowchart of a large-scale source code similarity detection method provided in this application includes:

[0046] S11: receiving the item to be detected sent by the user terminal and processing to obtain the first token feature of each function block in the item to be detected;

[0047] S12: Compare the first token feature with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities. The token feature library is generated by processing the open source projects collected by the code collection service with an inverted index algorithm.

[0048] Large-scale source code similarity detection is a basic research topic in the field of software engineering, and plays an important role in application fields such as software project maintenance, plagiarism detection, malware detection, and vulnerability search. In the process of software project development, developers will reuse open source software code to improve code quality in order to speed up the development process. Generally, they will reuse existing code by copying and pasting and modifying it, and automatically generate code snippets by IDE. Studies have shown that in general software systems, there is about 7-24% source code similarity, and in some software it even reaches 50%. Therefore, source code similarity detection is needed to reduce the many problems caused by excessive code cloning to software project maintenance. For example, reusing a code snippet containing unknown vulnerabilities will bring unknown security risks to the software.

[0049] With the rapid development of computer and Internet technologies, the number of software system applications and mobile applications has also exploded, which has put forward higher requirements for source code similarity detection. However, the existing source code similarity detection methods can only cope with detection in small-scale data and cannot handle massive data.

[0050] In order to solve the above technical problems, this application generates a token feature library based on an inverted index algorithm for the open source projects collected by the code collection service, and then the token feature library can be used to realize the similarity detection of the first token feature of each function block in the project to be detected.

[0051] Specifically, in S11, in order to compare with the second token features of all function blocks in the token feature library, it is first necessary to obtain the first token feature of each function block in the item to be detected. The process of obtaining the first token feature of each function block can be: the set of items to be detected uploaded to the processor by the user terminal is P = {p1, p2, ..., px}, and after the processor performs function slicing, a series of preprocessing and function tokenization, the set P of items to be detected can be expressed as P = {b1, b2, ..., bm}, where bi (1≤i≤m) is the first token feature of the function block.

[0052] In S12, the code collection service can collect a large number of open source projects, and then obtain the second token features of all function blocks in a large number of open source projects and generate a token feature library based on the inverted index algorithm. Since the token feature library is built based on the inverted index algorithm, when the first token feature is compared with the second token features of all function blocks in the token feature library, the inverted index algorithm can be called to quickly find the target data in the token feature library containing a large number of open source projects, and obtain all third token features similar to the first token feature and the corresponding similarity, so as to complete the large-scale source code similarity detection of massive data, with higher efficiency and practicality. Among them, the first token feature and the third token feature can be judged as similar if they contain the same words, and the similarity is divided into type1 (similarity at the text level), type2 (similarity at the lexical or token level), and type3 (similarity at the grammatical level) similarity codes.

[0053] In addition, open source projects can provide a large number of C, C++, Java, Python and other programming language source code projects, which greatly increases the source code base for large-scale source code similarity detection, that is, it greatly enriches the token feature library, thereby increasing the optional range of the second token feature in the token feature library, and laying a good foundation for further manual inspection and screening of all third token features similar to the first token feature.

[0054] In summary, the present application provides a large-scale source code similarity detection method, which is applied to the technical field of large-scale source code similarity detection. Receive the project to be detected sent by the user terminal and process it to obtain the first token feature of each function block in the project to be detected; compare the first token feature with the second token feature of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities. Collect open source projects through the code collection service, and then generate a token feature library through the inverted index algorithm. Therefore, in the token feature library, fast search can be achieved in massive data based on the inverted index algorithm, and then large-scale source code similarity detection between the first token feature and massive data can be achieved through the token feature library.

[0055] Based on the above embodiments:

[0056] As a preferred embodiment, receiving the item to be detected sent by the user terminal and processing to obtain the first token feature of each function block in the item to be detected includes:

[0057] Receive the items to be detected sent by the user terminal and perform function slicing, data preprocessing and function tokenization on the items to be detected in sequence to obtain the first token feature of each function block in the items to be detected.

[0058] In this embodiment, the set of items to be detected sent by the user terminal can be P = {p1, p2, ..., px}. After the processor performs function slicing, a series of preprocessing and function tokenization, the set of items to be detected P can be expressed as P = {b1, b2, ..., bm}, where bi (1≤i≤m) is the first token feature of the function block. Different slicing strategies are used for items to be detected in different languages, and then the function blocks generated by the slicing are preprocessed, which includes operations such as removing comments, removing blank lines, and removing keywords. Finally, function tokenization is performed to remove special symbols (such as "+" and "-") in the function block to generate the first token feature.

[0059] As a preferred embodiment, after comparing the first token feature with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities, the method further includes:

[0060] The first token feature, all third token features and the corresponding similarities are sent to the user terminal, and all third token features are sorted from large to small based on the corresponding similarities.

[0061] All third token features similar to the first token feature and the corresponding similarities obtained can be returned to the user terminal and sorted from high to low based on the similarity, so that the operator can optimize the items to be detected based on the vulnerabilities existing in the third token features with higher similarities at the user terminal to prevent the vulnerabilities from affecting the items to be detected.

[0062] As a preferred embodiment, the first token feature is compared with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities, including:

[0063] Based on big data parallel computing technology, the first token feature is compared with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities.

[0064] Since human society entered the 21st century, with the rapid development of the data age, the amount of data has expanded rapidly, and the data dimension has continued to increase. Big data technologies represented by Hadoop and Spark have also continued to develop and mature. Big data technologies have been widely used in various fields of social development and people's lives, constantly improving people's quality of life and promoting the development and progress of human society.

[0065] Considering the efficiency of large-scale source code similarity detection in the face of massive data, big data parallel computing technology can be used to compare the first token feature with the second token feature of all function blocks in the token feature library to improve efficiency. The big data parallel computing technology can be Hadoop or Spark, etc.

[0066] For details, please refer to Figure 2, the set of items to be detected P = {p1, p2, ..., px} is obtained after function slicing, preprocessing and function tokenization, and the first token feature bi (1≤i≤m) of the function block is obtained. The first token feature bi is used as the granularity (that is, the first token feature bi is the smallest indivisible function block), and the large-scale source code similarity detection of each first token feature bi and all the second token features cj (1≤j≤n) in the token feature library are processed in parallel through big data parallel computing technology. When there is a large amount of data in the token feature library, compared with each first token feature bi being sequentially performed on a large scale source code similarity detection, the efficiency and practicality are higher. At this time, with the help of the inverted index algorithm and big data parallel computing technology, the speed of data processing and search during large-scale source code similarity detection is greatly improved.

[0067] As a preferred embodiment, the big data platform receives open source projects collected by the code collection service and generates a token feature library after processing through an inverted index algorithm.

[0068] In this embodiment, the process of generating a token feature library based on a large number of open source projects can be assisted by a big data platform, such as an HDFS distributed file system.

[0069] Specifically, the first step is to collect open source projects. The basis for large-scale source code similarity detection is to obtain a large number of open source projects (i.e., open source code). There are many open source source code hosting platforms on the Internet, such as GitHub, Gitee, BitBucket, and SourceForge. These open source code platforms provide support for building a token feature library generated based on massive source code. On the basis of complying with various open source code protocols (such as BSD, GPL, LGPL, MIT, MPL, and EPL), open source projects in various programming languages ​​can be obtained from these platforms, and the open source projects collected from various platforms can be uploaded to the big data platform in real time through the code collection service; then the big data platform can use the inverted index algorithm to periodically batch process token serialization to generate a token feature library.

[0070] On this basis, the device deployment scheme for large-scale source code similarity detection can refer to Figure 3, which is mainly divided into the following three steps: 1. Collect massive open source projects from various open source platforms through code collection services and upload them to the big data platform; 2. The big data platform can batch process the token serialization of massive open source projects based on the inverted index algorithm and big data parallel computing technology, obtain the second token feature, and store it in the token feature library; 3. The user terminal can send the project to be detected to the processor for large-scale source code similarity detection through the access detection interface. At this time, the token feature library provides services for large-scale source code similarity detection and can return the detection results to the user terminal. The detection results are all third token features and corresponding similarities. In summary, the device deployment scheme for large-scale source code similarity detection is mainly composed of four parts: code collection service, big data platform, token feature library and large-scale source code similarity detection. It realizes rapid large-scale source code similarity detection from massive open source projects through the interaction between upstream and downstream.

[0071] As a preferred embodiment, the first token feature is compared with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities, including:

[0072] Get the words contained in the first token feature;

[0073] Based on the inverted index algorithm, all third token features are obtained from the second token features of all function blocks in the token feature library, and the number of types of words included in the third token features is greater than a preset number;

[0074] Obtain all key words from all words through the TF-IDF algorithm, where the key words are words whose TF-IDF values ​​in the set of the first token feature and all third token features are greater than a preset TF-IDF value;

[0075] By the formula S = a / b*cos<Vt,Vi> Get the similarity corresponding to the i-th third token feature, Vt is the first word frequency vector based on the frequency of all key words in the first token feature, Vi is the second word frequency vector based on the frequency of all key words in the i-th third token feature, cos<Vt,Vi> is the cosine similarity between Vt and Vi, a is the number of key words contained in the i-th third token feature, and b is the total number of all key words.

[0076] In this embodiment, all third token features similar to the first token feature and the corresponding similarities are obtained based on the inverted index algorithm and the TF-IDF algorithm, thereby improving the detection efficiency and accuracy of large-scale source code similarity detection.

[0077] Specifically, the first token feature is composed of a token sequence, and the token sequence is composed of multiple words. First, the first token feature can be segmented and deduplicated to obtain the words it contains; then the second token feature containing any of the above words can be quickly found from the token feature library through the inverted index algorithm, and the more types of the above words contained in a second token feature, the higher the similarity. Therefore, according to a certain configurable threshold, that is, a preset number, all the third token features in the token feature library are preliminarily screened out, thereby greatly reducing the workload of the next step of calculation; finally, on the basis of the preliminary screening, the TF-IDF algorithm is used Then perform a secondary screening to calculate the TF-IDF values ​​of all the words that constitute the first token feature in the set of the first token feature and all the third token features. The larger the TF-IDF value of a word, the more important it is to the first token feature and all the third token features. When calculating the similarity of the code, the more attention should be paid to the similarity brought by the word. Therefore, according to a certain configurable threshold, that is, the preset TF-IDF value, the key words are screened out for a secondary time. When the number of key words screened out for the secondary time is large, the key words can be screened again in terms of quantity according to their corresponding TF-IDF values ​​from large to small, thereby greatly reducing the workload of the next step of calculation.

[0078] After the initial screening and the secondary screening, the similarity corresponding to each third token feature is obtained based on the key words and each third token feature in the token feature library. For example, the process of obtaining the similarity corresponding to the i-th third token feature is as follows: first, the first word frequency vector Vt composed of the frequency of all key words in the first token feature and the second word frequency vector Vi composed of the frequency of all key words in the i-th third token feature are obtained, and Vt and Vi can be quickly obtained through the inverted index algorithm; then, through cos<Vt,Vi> Get the cosine similarity of Vt and Vi, where cos<Vt,Vi> =Vt*Vi / (|Vt|*|Vi|); Finally, obtain the ratio of the number of key words a contained in the i-th third token feature to the total number of all key words b, and use the formula S=a / b*cos<Vt,Vi> Get the similarity corresponding to the feature of the third token of the i-th order. The similarity obtained by this formula is more accurate.

[0079] In summary, in the face of a large-scale token feature library, efficient and accurate large-scale source code similarity detection can be performed through big data parallel computing technology, inverted index algorithm and TF-IDF algorithm, which has better performance than the existing technology.

[0080] As a preferred embodiment, the big data platform receives the open source projects collected by the code collection service and processes the open source projects through an inverted index algorithm to sequentially generate a function dictionary table, a word dictionary table and an inverted index table to construct a token feature library;

[0081] Based on the inverted index algorithm, all third token features are obtained from the second token features of all function blocks in the token feature library. The number of types of words contained in the third token features is greater than the preset number, including:

[0082] Find the word ID of each word through the word dictionary table;

[0083] Find the function inverted list corresponding to each word ID through the inverted index table. The function inverted list is composed of a vector (feature ID, frequency). The feature ID is the ID of the second token feature containing the word, and the frequency is the number of words contained in the second token feature.

[0084] Based on the function inverted list corresponding to each word ID, obtain all words and corresponding frequencies contained in each second token feature;

[0085] All second token features whose number of categories containing words is greater than a preset number are used as all third token features.

[0086] In this embodiment, all third token features are obtained based on the function dictionary table, word dictionary table and inverted index table generated in sequence in the process of building the token feature library.

[0087] Specifically, the process of building a token feature library is as follows: Figure 4, the big data platform performs function slicing on the open source projects q1, q2...qy stored in itself to obtain function blocks f1, f2...fn, and then performs data preprocessing and function tokenization on the function blocks to obtain the second token features c1, c2...cn. The big data parallel computing technology can be used to speed up the process of building the token feature library, and the relevant information of the second token features c1, c2...cn after function tokenization is saved in the function dictionary table, as shown in Table 1. Among them, the function dictionary table can contain six fields such as the second token feature ID, name, project, file, token sequence, and word frequency; w1, w2, and w3 in the token sequence of the second token feature c1 are the words contained in the second token feature c1.

[0088] Table 1

[0089]

[0090] All token sequences in Table 1 are segmented and deduplicated, and the word information is saved in the word dictionary table, as shown in Table 2. The word dictionary table contains two fields: word ID and word.

[0091] Table 2

[0092] Word ID 1 2 3 4 5 6 …… word w1 w2 w3 w4 w5 w6 ……

[0093] Finally, the information in the function dictionary table and the word dictionary table is extracted to construct an inverted index table, as shown in Table 3, which records in detail the second token feature of each word and the corresponding frequency. Among them, the inverted index table contains two fields, namely, word ID and function inverted list.

[0094] In addition, the second word frequency vector Vi composed of the frequencies of all key words in the i-th third token feature can be generated based on Table 3. For example, when the key words are w2 and w3 and the ID of the i-th third token feature is 1, the second word frequency vector Vi is (1, 2), 1 is the frequency of w2 in c1, and 2 is the frequency of w3 in c1.

[0095] Table 3

[0096]

[0097]

[0098] Based on the above, the specific process of obtaining all the third token features is:

[0099] First, the first token feature is divided into words, and then the word IDs are found in the word dictionary table in the token feature library, taking unit IDs 1, 2, 3, 4, 5 and 6 as examples; then, according to these word IDs, the function inverted list of each word is found in the inverted index table of the token feature library. Here, a search function L(x) can be assumed, where x is a word, and the search result of the function inverted list can be expressed as:

[0100]

[0101] By aggregating the search results of the inverted list of the above function, we can get the aggregation results shown in Table 4, which records each second token feature containing the word to be detected. The word to be detected is the word contained in the first token feature. Table 4 also records the frequency of occurrence of the word to be detected in each second token feature.

[0102] Table 4

[0103]

[0104] All second token features that contain more than a preset number of categories of words to be detected are taken as all third token features. When the preset number is 3, all third token features are c3 and c4.

[0105] In summary, the function dictionary table, word dictionary table and inverted index table are used to search for all third token features, which has higher search efficiency.

[0106] In addition, combining all the above embodiments, a specific algorithm flow chart for obtaining large-scale source code similarity can be obtained. Figure 5 .

[0107] As a preferred embodiment, all key words are obtained from all words through the TF-IDF algorithm, and the key words are words whose TF-IDF value is greater than a preset TF-IDF value in the set of the first token feature and all third token features, including:

[0108] Through the formula TF-IDF=TF W *IDF W Get the TF-IDF value of the Wth word, TF W =c W / d, IDF W =log(e / (f+1)), c Wis the frequency of the Wth word in the first token feature, d is the total number of words, e is the sum of the number of the first token feature and all third token features, and f is the sum of the number of features containing the Wth word in the first token feature and all third token features;

[0109] All words with TF-IDF values ​​greater than a preset TF-IDF value are regarded as all key words.

[0110] In this embodiment, the process of filtering out key words from the words contained in the first token feature according to the TF-IDF value obtained by the TF-IDF algorithm is defined. The larger the TF-IDF value of a word is, the more important the word is to the first token feature and all the third token features. When calculating the similarity of the code, the more attention should be paid to the similarity brought by the word. Then, according to a certain configurable threshold, that is, the preset TF-IDF value, the key words are filtered out, and the computational workload is reduced while ensuring the reliability of the large-scale source code similarity finally obtained.

[0111] Please refer to Figure 6 , Figure 6 A schematic diagram of the structure of a large-scale source code similarity detection system provided for this application includes:

[0112] A token feature acquisition unit 21 is used to receive the item to be detected sent by the user terminal and process it to obtain the first token feature of each function block in the item to be detected;

[0113] The similarity acquisition unit 22 is used to compare the first token feature with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities. The token feature library is generated by processing the open source projects collected by the code collection service with an inverted index algorithm.

[0114] For an introduction to a large-scale source code similarity detection system provided by the present application, please refer to the above-mentioned embodiment, and the present application will not go into details here.

[0115] As a preferred embodiment, the token feature acquisition unit 21 is specifically used to receive the items to be detected sent by the user terminal and perform function slicing, data preprocessing and function tokenization on the items to be detected in sequence to obtain the first token feature of each function block in the items to be detected.

[0116] As a preferred embodiment, it also includes:

[0117] The returning unit is used to send the first token feature, all the third token features and the corresponding similarities to the user terminal after the similarity obtaining unit 22, and all the third token features are sorted from large to small based on the corresponding similarities.

[0118] As a preferred embodiment, the similarity acquisition unit 22 is specifically used to compare the first token feature with the second token features of all function blocks in the token feature library based on big data parallel computing technology to obtain all third token features similar to the first token feature and the corresponding similarities.

[0119] As a preferred embodiment, the big data platform receives open source projects collected by the code collection service and generates a token feature library after processing through an inverted index algorithm.

[0120] As a preferred embodiment, the similarity acquisition unit 22 includes:

[0121] A word acquisition unit, used to acquire the word contained in the first token feature;

[0122] A third token feature acquisition unit, configured to acquire all third token features from the second token features of all function blocks in the token feature library based on an inverted index algorithm, wherein the number of types of words included in the third token features is greater than a preset number;

[0123] A keyword acquisition unit, used to acquire all keyword words from all words by using a TF-IDF algorithm, wherein the keyword words are words whose TF-IDF values ​​in the set of the first token feature and all third token features are greater than a preset TF-IDF value;

[0124] The similarity acquisition unit corresponding to the i-th third token feature is used to obtain the similarity through the formula S = a / b*cos<Vt,Vi> Get the similarity corresponding to the i-th third token feature, Vt is the first word frequency vector based on the frequency of all key words in the first token feature, Vi is the second word frequency vector based on the frequency of all key words in the i-th third token feature, cos<Vt,Vi> is the cosine similarity between Vt and Vi, a is the number of key words contained in the i-th third token feature, and b is the total number of all key words.

[0125] As a preferred embodiment, the big data platform receives the open source projects collected by the code collection service and processes the open source projects through an inverted index algorithm to sequentially generate a function dictionary table, a word dictionary table and an inverted index table to construct a token feature library;

[0126] The third token feature acquisition unit includes:

[0127] A word ID acquisition unit is used to find the word ID of each word through a word dictionary table;

[0128] A function inverted list acquisition unit is used to find the function inverted list corresponding to each word ID through the inverted index table. The function inverted list is composed of a vector (feature ID, frequency), where the feature ID is the ID of the second token feature containing the word, and the frequency is the number of words contained in the second token feature;

[0129] A unit for acquiring words and corresponding frequencies in the second token feature, used for acquiring all words and corresponding frequencies contained in each second token feature based on the function inverted list corresponding to each word ID;

[0130] The word-based third token feature acquisition unit is used to take all second token features whose number of categories containing words is greater than a preset number as all third token features.

[0131] As a preferred embodiment, the keyword acquisition unit includes:

[0132] TF-IDF value acquisition unit, used to obtain the value of TF-IDF by the formula TF-IDF=TF W *IDF W Get the TF-IDF value of the Wth word, TF W =c W / d, IDF W =log(e / (f+1)), c W is the frequency of the Wth word in the first token feature, d is the total number of words, e is the sum of the number of the first token feature and all third token features, and f is the sum of the number of features containing the Wth word in the first token feature and all third token features;

[0133] The key word acquisition unit based on the TF-IDF value is used to take all words with a TF-IDF value greater than a preset TF-IDF value as all key words.

[0134] Please refer to Figure 7 , Figure 7 A schematic diagram of the structure of a large-scale source code similarity detection device provided in this application includes:

[0135] A memory 31, used for storing computer programs;

[0136] The processor 32 is used to execute the computer program to implement the steps of the large-scale source code similarity detection method.

[0137] For an introduction to a large-scale source code similarity detection device provided by the present application, please refer to the above-mentioned embodiment, and the present application will not go into details here.

[0138] It should be noted that, in this specification, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0139] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large-scale source code similarity detection method, characterized in that: include: Receiving the item to be detected sent by the user terminal and processing to obtain the first token feature of each function block in the item to be detected; Comparing the first token feature with the second token features of all function blocks in a token feature library to obtain all third token features similar to the first token feature and corresponding similarities, wherein the token feature library is generated by processing the open source projects collected by the code collection service with an inverted index algorithm; The step of comparing the first token feature with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities includes: Obtain the words contained in the first token feature; Based on the inverted index algorithm, all the third token features are obtained from the second token features of all function blocks in the token feature library, and the number of types of the words included in the third token features is greater than a preset number; Obtain all key words from all the words by using the TF-IDF algorithm, wherein the key words are words whose TF-IDF values ​​in the set of the first token feature and all the third token features are greater than a preset TF-IDF value; By the formula S=a / b*cos<Vt,Vi> Get the similarity corresponding to the i-th third token feature, where Vt is a first word frequency vector based on the frequency of all the key words in the first token feature, Vi is a second word frequency vector based on the frequency of all the key words in the i-th third token feature, cos<Vt,Vi> is the cosine similarity between the Vt and the Vi, a is the number of the key words contained in the i-th third token feature, and b is the total number of all the key words.

2. The large-scale source code similarity detection method according to claim 1, characterized in that: Receiving the item to be detected sent by the user terminal and processing to obtain the first token feature of each function block in the item to be detected, including: Receive the items to be detected sent by the user terminal and perform function slicing, data preprocessing and function tokenization on the items to be detected in sequence to obtain the first token feature of each function block in the items to be detected.

3. The large-scale source code similarity detection method according to claim 1, characterized in that: After comparing the first token feature with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities, the method further includes: The first token feature, all the third token features and the corresponding similarities are sent to the user terminal, and all the third token features are sorted from large to small based on the corresponding similarities.

4. The large-scale source code similarity detection method according to claim 1, characterized in that: Comparing the first token feature with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and corresponding similarities, including: Based on big data parallel computing technology, the first token feature is compared with the second token features of all function blocks in the token feature library to obtain all third token features similar to the first token feature and the corresponding similarities.

5. The large-scale source code similarity detection method according to claim 1, characterized in that: The big data platform receives the open source projects collected by the code collection service and generates the token feature library after processing through the inverted index algorithm.

6. The large-scale source code similarity detection method according to claim 1, characterized in that: The big data platform receives the open source projects collected by the code collection service and processes the open source projects through an inverted index algorithm to sequentially generate a function dictionary table, a word dictionary table and an inverted index table to construct the token feature library; Based on the inverted index algorithm, all the third token features are obtained from the second token features of all function blocks in the token feature library, and the number of types of the words contained in the third token features is greater than a preset number, including: Find the word ID of each word through the word dictionary table; Find the function inverted list corresponding to each word ID through the inverted index table, wherein the function inverted list is composed of a vector (feature ID, frequency), wherein the feature ID is the ID of the second token feature containing the word, and the frequency is the number of the words contained in the second token feature; Based on the function inverted list corresponding to each word ID, obtain all the words and corresponding frequencies contained in each second token feature; All the second token features whose number of categories containing the word is greater than a preset number are used as all the third token features.

7. The large-scale source code similarity detection method according to any one of claims 1 to 6, characterized in that: By using the TF-IDF algorithm, all key words are obtained from all the words, where the key words are words whose TF-IDF values ​​in the set of the first token feature and all the third token features are greater than a preset TF-IDF value, including: By formula TF-IDF=TF W *IDF W Get the TF-IDF value of the Wth word, the TF W =c W / d, IDF W =log(e / (f+1)), c W is the frequency of the Wth word in the first token feature, d is the total number of the words, e is the sum of the number of the first token feature and all the third token features, and f is the sum of the number of features containing the Wth word in the first token feature and all the third token features; All words whose TF-IDF values ​​are greater than a preset TF-IDF value are taken as all the key words.

8. A large-scale source code similarity detection system, characterized in that: include: A token feature acquisition unit, configured to receive an item to be detected sent by a user terminal and process the item to be detected to obtain a first token feature of each function block in the item to be detected; A similarity acquisition unit is used to acquire words included in the first token feature; based on an inverted index algorithm, acquire all third token features from the second token features of all function blocks in a token feature library, wherein the number of types of the words included in the third token features is greater than a preset number; and acquire all key words from all the words through a TF-IDF algorithm, wherein the key words are words whose TF-IDF values ​​in the set of the first token feature and all the third token features are greater than a preset TF-IDF value; By the formula S=a / b*cos<Vt,Vi> Get the similarity corresponding to the i-th third token feature, where Vt is a first word frequency vector based on the frequency of all the key words in the first token feature, Vi is a second word frequency vector based on the frequency of all the key words in the i-th third token feature, cos<Vt,Vi> is the cosine similarity between the Vt and the Vi, a is the number of the key words contained in the i-th third token feature, b is the total number of all the key words, and the token feature library is generated by processing the open source projects collected by the code collection service with an inverted index algorithm.

9. A large-scale source code similarity detection device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program to implement the steps of the large-scale source code similarity detection method according to any one of claims 1 to 7.

Images