Vulnerability detection method and system based on code similarity analysis

Abstract

The invention discloses a vulnerability detection method and system based on code similarity analysis, and the method comprises the steps: obtaining patches in an open source project warehouse to generate vulnerability codes, and constructing a vulnerability code library; analyzing the vulnerability code library and the source code library of the target program, and respectively generating abstract syntax tree sets of the vulnerability code library and the source code library of the target program according to the generation rules of the abstract syntax trees; abstracting the abstract syntax tree set through a conversion rule to obtain a new abstract syntax tree set; key values and mark strings are extracted from the abstract syntax tree of each function to serve as code features, the mark strings are converted into hash values, a vulnerability dictionary and a target dictionary are obtained, similarity is calculated, whether a target program has reuse of vulnerability codes or not is judged, and a result is output. According to the method, similarity comparison of codes is converted into comparison of key values and hash values, and security vulnerabilities of large-scale software programs can be rapidly detected on the premise of ensuring high accuracy.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a vulnerability detection method and system based on code similarity analysis. Background Art

[0002] In recent years, network security incidents have occurred frequently, causing significant economic losses. The key factor in network security incidents is the existence of vulnerabilities in the program, so vulnerability detection technology is a research hotspot both in academia and industry. In the field of network security, vulnerabilities refer to defects in the specific implementation of hardware, software, protocols, or system security policies, which can enable attackers to access or damage the system without authorization.

[0003] Code reuse, as the name implies, refers to the process of directly copying or slightly modifying some code from some open source software projects and using them in other software during the software development process. Although code duplication can speed up the process of software development, it often seriously affects the security of software because code duplication can easily spread vulnerabilities. Moreover, with the rapid growth of the number of open source software, the reuse of vulnerable codes has also greatly increased, thus polluting many software systems. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a vulnerability detection method and system based on code similarity analysis to solve the problems of vulnerability propagation, difficulty in detecting unknown vulnerabilities, and low detection efficiency caused by code reuse.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a vulnerability detection method based on code similarity analysis, comprising:

[0009] Obtain patches from the open source project repository to generate vulnerability codes and build a vulnerability code library;

[0010] Parsing the vulnerability code library and the source code library of the target program, and generating abstract syntax tree sets of the vulnerability code library and the source code library of the target program respectively according to the generation rules of the abstract syntax tree;

[0011] Abstract the set of abstract syntax trees by using transformation rules to obtain a new set of abstract syntax trees;

[0012] Based on the new abstract syntax tree set, key values ​​and tag strings are extracted from the abstract syntax tree of each function as code features, and the tag strings are converted into hash values ​​to obtain vulnerability dictionaries and target dictionaries.

[0013] The similarity is calculated based on the obtained vulnerability dictionary and the target dictionary to determine whether the target program has reused vulnerable code and output the vulnerability detection results.

[0014] As a preferred solution of the vulnerability detection method based on code similarity analysis described in the present invention, wherein:

[0015] The process of obtaining patches in the open source project repository to generate vulnerability codes and constructing a vulnerability code library includes the following steps:

[0016] Select and download active and well-known open source project repositories;

[0017] Use the CVE number as one of the keywords to filter out security-related commits;

[0018] Extract commit logs from security-related commits to obtain descriptions of CVE-related vulnerabilities and security patch information in unified diff format;

[0019] Filter irrelevant commits and retrieve vulnerable functions from old, unpatched files;

[0020] All the vulnerable functions finally retrieved are packaged into a vulnerable code library.

[0021] As a preferred solution of the vulnerability detection method based on code similarity analysis described in the present invention, wherein:

[0022] The vulnerability code library is saved in the form of code files in units of functions.

[0023] As a preferred solution of the vulnerability detection method based on code similarity analysis described in the present invention, wherein:

[0024] The step of parsing the vulnerability code library and the target program source code library, and generating the abstract syntax tree sets of the vulnerability code library and the target program source code library respectively according to the generation rule of the abstract syntax tree, comprises the following steps:

[0025] Use regular expressions to lexically analyze program source code and decompose the source code into a series of tokens;

[0026] Perform grammatical analysis on the token sequence according to the context-free grammar to generate an abstract syntax tree;

[0027] Perform lexical analysis and grammatical analysis on the vulnerability code library and the source code library of the target program respectively to generate their respective abstract syntax tree sets.

[0028] As a preferred solution of the vulnerability detection method based on code similarity analysis described in the present invention, wherein:

[0029] The process of abstracting the set of abstract syntax trees by using the conversion rules to obtain a new set of abstract syntax trees includes the following steps:

[0030] Collect all formal parameters from the function declaration and replace each occurrence of the formal parameter variable in the function with f_param;

[0031] Identify all local variables that appear in the function body and replace them with l_var;

[0032] Identify all data types that appear in the function, including standard C language data types, qualifiers, and user-defined data types, and replace all data types with d_type;

[0033] Identify all called function names in the function body and replace each called function name with fun_call.

[0034] As a preferred solution of the vulnerability detection method based on code similarity analysis described in the present invention, wherein:

[0035] The process of extracting key values ​​and tag strings from the abstract syntax tree of each function as code features, converting the tag strings into hash values, and obtaining vulnerability dictionaries and target dictionaries includes the following steps:

[0036] Traverse the abstract syntax tree of each function, count and record the key values ​​in the abstract syntax tree;

[0037] Merge the information of each node of the abstract syntax tree together to form a string of tokens;

[0038] The tuple consisting of the key value and the tag string is used as the code feature of the function, and the tag string is hashed and converted into a corresponding hash value;

[0039] The key value counting, tag string generation and hashing feature steps are performed on all elements in the abstract syntax tree sets of the vulnerability code library and the target program source code library respectively, and two vulnerability dictionaries and target dictionaries consisting of two tuples are obtained, which correspond to the vulnerability code library and the target program source code library respectively.

[0040] As a preferred solution of the vulnerability detection method based on code similarity analysis described in the present invention, wherein:

[0041] The method of calculating similarity based on the obtained vulnerability dictionary and the target dictionary, determining whether the target program has reuse of vulnerable code, and outputting the vulnerability detection result includes the following steps:

[0042] Select a function f from the target dictionary, get the key value of function f, and search for a function with the same key value in the vulnerability dictionary;

[0043] If no function with the same key value is found in the vulnerability dictionary, the function f is judged to be dissimilar to the known vulnerability and the process jumps to the iterative detection step;

[0044] If a function with the same key value is found in the vulnerability dictionary, set the key value to key and execute the next step;

[0045] For the function set with the same key value, further search in the vulnerability dictionary whether there is a function with the same hash value as the hash value of the function to be tested. If not, it is considered that the current function to be tested is not similar to the vulnerable code segment. If so, it is determined to be a vulnerable function and recorded;

[0046] Determine whether all the functions to be tested in the target dictionary have been tested. If not, jump back to the key-value matching step; if completed, output the vulnerability detection results.

[0047] In a second aspect, the present invention provides a vulnerability detection system based on code similarity analysis, comprising:

[0048] The vulnerability code library construction module is used to obtain patches in the open source project repository to generate vulnerability codes and build a vulnerability code library;

[0049] An abstract syntax tree generation module is used to parse the vulnerability code library and the source code library of the target program, and generate abstract syntax tree sets of the vulnerability code library and the source code library of the target program respectively according to the generation rules of the abstract syntax tree;

[0050] An abstract processing module is used to perform abstract processing on the abstract syntax tree set through conversion rules to obtain a new abstract syntax tree set;

[0051] The feature extraction and hashing module is used to extract key values ​​and tag strings as code features from the abstract syntax tree of each function based on the new abstract syntax tree set, and convert the tag strings into hash values ​​to obtain vulnerability dictionaries and target dictionaries;

[0052] The similarity detection module is used to calculate the similarity based on the obtained vulnerability dictionary and the target dictionary, determine whether the target program has reused vulnerable code, and output the vulnerability detection result.

[0053] In a third aspect, the present invention provides a computing device, comprising:

[0054] Memory, used to store programs;

[0055] A processor is used to execute the computer executable instructions, which, when executed by the processor, implement the steps of the vulnerability detection method based on code similarity analysis.

[0056] In a fourth aspect, the present invention provides a computer-readable storage medium, comprising: when the program is executed by a processor, the steps of implementing the vulnerability detection method based on code similarity analysis.

[0057] The beneficial effects of the present invention are as follows: the present invention adopts a method of analyzing submission information and patch files in a public vulnerability library and an open source project repository to reconstruct a vulnerability function to construct a vulnerability library, thereby realizing full automation of the vulnerability library construction process; the present invention uses an optimized abstract method for detecting vulnerability code reuse, thereby being able to detect unknown vulnerability code reuse; the method for calculating similarity used in the present invention has a very low time complexity, and it is based on a new type of code feature representing a function, which greatly improves the detection efficiency while ensuring the detection precision and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0059] Figure 1 A basic flow chart of a vulnerability detection method based on code similarity analysis provided by an embodiment of the present invention;

[0060] Figure 2 A schematic diagram of a flow chart of a vulnerability detection method based on code similarity analysis provided by an embodiment of the present invention;

[0061] Figure 3 A flowchart of extracting code features and constructing a dictionary of a vulnerability detection method based on code similarity analysis provided by an embodiment of the present invention;

[0062] Figure 4 A flowchart of the steps of performing vulnerability code reuse detection in a vulnerability detection method based on code similarity analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0064] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0065] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0066] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0067] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0068] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0069] Example 1

[0070] Reference Figure 1-4, is an embodiment of the present invention, and provides a vulnerability detection method based on code similarity analysis, such as Figure 1 , as shown in 2, comprising the following steps:

[0071] S1: Obtain patches from the open source project repository to generate vulnerability codes and build a vulnerability code library;

[0072] In the embodiment of the present application, S1 includes the following steps:

[0073] S1.1: Download a large number of newer, popular Git repositories;

[0074] S1.2: Search for related commits based on some common keywords, such as vulnerability type, vulnerability name, CVE number, etc.

[0075] S1.3: Obtain descriptions of CVE-related vulnerabilities and security patch information in unified diff format based on the submission log;

[0076] S1.4: Filter irrelevant commits and retrieve vulnerable functions from old, unpatched files.

[0077] All the vulnerability functions finally retrieved are packaged into a vulnerability code library, and the vulnerability code library is saved in the form of code files in units of functions.

[0078] S2: Parse the vulnerability code library and the target program source code library, and generate abstract syntax tree sets of the vulnerability code library and the target program source code library respectively according to the generation rules of the abstract syntax tree;

[0079] In the embodiment of the present application, the process of generating an abstract syntax tree is mainly to first perform lexical analysis on the program source code according to regular expressions to obtain Tokens, and then perform grammatical analysis according to a context-free grammar to obtain an abstract syntax tree (AST).

[0080] S3: abstract the set of abstract syntax trees through transformation rules to obtain a new set of abstract syntax trees;

[0081] In the embodiment of the present application, S3 includes the following steps:

[0082] S3.1: Parameter abstraction: Collect the parameter variables from the function declaration and replace each occurrence of the parameter variable in the function with the symbol f_param;

[0083] S3.2: Local variable abstraction: Replace all local variables appearing in the function body with the symbol l_var;

[0084] S3.3: Data type abstraction: Replace all data types that appear in the function with the symbol d_type. The data types here include not only standard C language data types and qualifiers, but also user-defined data types;

[0085] S3.4: Function call abstraction: Replace the name of each called function with the symbol fun_call.

[0086] S4: Based on the new abstract syntax tree set, extract the key value and tag string from the abstract syntax tree of each function as the code feature, and convert the tag string into a hash value to obtain the vulnerability dictionary and target dictionary;

[0087] In an embodiment of the present application, the abstract syntax tree of each function is traversed and parsed to obtain the number of nodes in the abstract syntax tree and the tag string of the function, to obtain a code feature consisting of the number of function nodes and the corresponding tag string, to convert the tag string into a hash value, to obtain two dictionaries consisting of two-tuples such as (number of nodes, hash value), which map the number of nodes to the hash value. Since different hash values ​​may correspond to the same number of nodes, different functions are classified and stored according to the number of nodes.

[0088] In the embodiments of the present application, Figure 3 As shown, S4 includes the following steps:

[0089] S4.1: Traverse the abstract syntax tree and record the number of nodes;

[0090] S4.2: Merge the information of each node of the abstract syntax tree together to form a tag string of the function, where the tag string is represented in the form of a character string;

[0091] S4.3: The tuple consisting of the number of nodes and the tag string is the code feature of the function, where the first element of the tuple (i.e. the number of nodes) is also called the key value. The MD5 hash algorithm is used to convert the tag string into the corresponding hash value, thereby converting the code feature into the representation form of (key value, hash value).

[0092] S4.4: Perform the above three steps on all elements in the abstract syntax tree set of the vulnerability code library and the target program source code library respectively, and obtain two code feature dictionaries G consisting of two tuples. V and G T , corresponding to the vulnerability code library and the target program source code library respectively. A dictionary is an associative container that maps key values ​​to hash values. Since different functions may share the same key value, the functions are classified and stored according to the key value for easy search.

[0093] S5: Calculate similarity based on the obtained vulnerability dictionary and target dictionary, determine whether the target program has reused vulnerable code, and output vulnerability detection results.

[0094] In the embodiments of the present application, Figure 4 As shown, S5 includes the following steps:

[0095] S5.1: Key-value lookup: Read target program dictionary G T The key value of a function f in (i.e. the number of nodes in the corresponding AST) is used to search the vulnerability dictionary G. V Is there a function with the same key value in the search? If the search fails, it is determined that the function to be tested is not similar to the vulnerable code segment and jumps to step 5.3, otherwise the key value is set to key and the next step is executed;

[0096] S5.2: Hash value search: Search for the function under the category with the key value of key in the vulnerability dictionary, and compare whether there is a function with the same hash value as the hash value of the function to be tested. If not, it is considered that the current function to be tested is not similar to the vulnerable code segment. Otherwise, it is determined to be a vulnerable function and recorded;

[0097] S5.3: Determine G T Check whether all the functions to be tested in have been tested. If not, jump to step 5.1; otherwise, output the vulnerability detection results and the process ends.

[0098] This embodiment also provides a vulnerability detection system based on code similarity analysis, including:

[0099] The vulnerability code library construction module is used to obtain patches in the open source project repository to generate vulnerability codes and build a vulnerability code library;

[0100] An abstract syntax tree generation module is used to parse the vulnerability code library and the source code library of the target program, and generate abstract syntax tree sets of the vulnerability code library and the source code library of the target program respectively according to the generation rules of the abstract syntax tree;

[0101] An abstract processing module is used to perform abstract processing on the abstract syntax tree set through conversion rules to obtain a new abstract syntax tree set;

[0102] The feature extraction and hashing module is used to extract key values ​​and tag strings as code features from the abstract syntax tree of each function based on the new abstract syntax tree set, and convert the tag strings into hash values ​​to obtain vulnerability dictionaries and target dictionaries;

[0103] The similarity detection module is used to calculate the similarity based on the obtained vulnerability dictionary and the target dictionary, determine whether the target program has reused vulnerable code, and output the vulnerability detection result.

[0104] Furthermore, it also includes:

[0105] Memory, used to store programs;

[0106] A processor is used to load the program to execute the vulnerability detection method based on code similarity analysis.

[0107] This embodiment also provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the vulnerability detection method based on code similarity analysis.

[0108] The storage medium proposed in this embodiment and the vulnerability detection method based on code similarity analysis proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0109] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (RandomAccess Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.

[0110] Example 2

[0111] This is an embodiment of the present invention, which provides a vulnerability detection system based on code similarity analysis, including a vulnerability code library construction module, an abstract syntax tree generation module, an abstract processing module, a feature extraction and hashing module and a similarity detection module;

[0112] In the embodiment of the present application, the vulnerability code library construction module includes obtaining patches in the open source project repository to generate vulnerability codes and construct a vulnerability code library;

[0113] In the embodiment of the present application, building a vulnerability code library includes analyzing patch files in the open source project repository to identify which security issues have been fixed. These patches usually contain modifications to the code snippets with vulnerabilities. The original (i.e., unfixed) code snippets are extracted from these patches to build a vulnerability code library. This step provides basic data for subsequent vulnerability detection.

[0114] In an embodiment of the present application, the abstract syntax tree generation module includes parsing the vulnerability code library and the source code library of the target program, and generating abstract syntax tree sets of the vulnerability code library and the source code library of the target program respectively according to the generation rules of the abstract syntax tree;

[0115] In the embodiment of the present application, generating an abstract syntax tree set includes once there is a vulnerability code library and a source code library of the target program, the next step is to parse the codes in the two libraries and convert them into AST form according to the generation rules of the abstract syntax tree (AST). AST provides a structured representation method for subsequent processing and comparison. The AST set generated in this step is the basis for the subsequent abstraction and feature extraction.

[0116] In the embodiment of the present application, the abstraction processing module includes performing abstraction processing on the abstract syntax tree set through a conversion rule to obtain a new abstract syntax tree set;

[0117] In an embodiment of the present application, abstracting a new abstract syntax tree set includes further abstracting the AST, such as unifying variable names, ignoring specific numerical values, etc., to reduce the impact of code differences. The abstracted AST set makes different but functionally similar code segments more consistent in structure, which helps to improve the accuracy of similarity calculations.

[0118] In the embodiment of the present application, the feature extraction and hashing module includes extracting key values ​​and tag strings as code features from the abstract syntax tree of each function based on the new abstract syntax tree set, and converting the tag strings into hash values ​​to obtain vulnerability dictionaries and target dictionaries;

[0119] In an embodiment of the present application, obtaining the vulnerability dictionary and the target dictionary includes calculating the number of nodes and generating a tag string for each function's AST, and then converting the tag string into a hash value, organizing this information into a tuple of (number of nodes, hash value), and storing them in two dictionaries respectively: one is the vulnerability dictionary and the other is the target dictionary. These dictionaries provide direct data support for the final similarity calculation.

[0120] In an embodiment of the present application, the similarity detection module includes calculating similarity based on the obtained vulnerability dictionary and the target dictionary, determining whether there is reuse of vulnerable code in the target program, and outputting vulnerability detection results.

[0121] In the embodiment of the present application, calculating similarity and outputting the result includes using the generated vulnerability dictionary and target dictionary to evaluate whether there is a part in the target program that is similar to the code in the vulnerability code library by comparing the tuples. The similarity calculation can adopt a variety of methods, such as Jaccard similarity, cosine similarity, etc. If it is found that the similarity exceeds the preset threshold, it is considered that there may be vulnerable code reuse. According to the result of the similarity calculation, a vulnerability detection report is output to point out the possible security risks.

[0122] Through the close connection between the above modules, a complete vulnerability detection process is formed. Each step depends on the result of the previous step and provides necessary input for the next step, thereby achieving effective identification of potential security vulnerabilities in the target program.

[0123] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A vulnerability detection method based on code similarity analysis, characterized in that: include: Obtain patches from the open source project repository to generate vulnerability codes and build a vulnerability code library; Parsing the vulnerability code library and the source code library of the target program, and generating abstract syntax tree sets of the vulnerability code library and the source code library of the target program respectively according to the generation rules of the abstract syntax tree; Abstract the abstract syntax tree set by using the conversion rules to obtain a new abstract syntax tree set; Based on the new abstract syntax tree set, key values ​​and tag strings are extracted from the abstract syntax tree of each function as code features, and the tag strings are converted into hash values ​​to obtain vulnerability dictionaries and target dictionaries. The similarity is calculated based on the obtained vulnerability dictionary and the target dictionary to determine whether the target program has reused vulnerable code and output the vulnerability detection results.

2. The vulnerability detection method based on code similarity analysis according to claim 1, characterized in that: The process of obtaining patches in the open source project repository to generate vulnerability codes and constructing a vulnerability code library includes the following steps: Select and download active and well-known open source project repositories; Use the CVE number as one of the keywords to filter out security-related commits; Extract commit logs from security-related commits to obtain descriptions of CVE-related vulnerabilities and security patch information in unified diff format; Filter irrelevant commits and retrieve vulnerable functions from old, unpatched files; All the vulnerable functions finally retrieved are packaged into a vulnerable code library.

3. The vulnerability detection method based on code similarity analysis according to claim 1 or 2, characterized in that: The vulnerability code library is saved in the form of code files in units of functions.

4. The vulnerability detection method based on code similarity analysis as claimed in claim 3, characterized in that: The step of parsing the vulnerability code library and the target program source code library, and generating the abstract syntax tree sets of the vulnerability code library and the target program source code library respectively according to the generation rule of the abstract syntax tree, comprises the following steps: Use regular expressions to lexically analyze program source code and decompose the source code into a series of tokens; Perform grammatical analysis on the token sequence according to the context-free grammar to generate an abstract syntax tree; Perform lexical analysis and grammatical analysis on the vulnerability code library and the source code library of the target program respectively to generate their respective abstract syntax tree sets.

5. The vulnerability detection method based on code similarity analysis according to claim 4, characterized in that: The process of abstracting the set of abstract syntax trees by using the conversion rules to obtain a new set of abstract syntax trees includes the following steps: Collect all formal parameters from the function declaration and replace each occurrence of the formal parameter variable in the function with f_param; Identify all local variables that appear in the function body and replace them with l_var; Identify all data types that appear in the function, including standard C language data types, qualifiers, and user-defined data types, and replace all data types with d_type; Identify all called function names in the function body and replace each called function name with fun_call.

6. The vulnerability detection method based on code similarity analysis according to claim 5, characterized in that: The process of extracting key values ​​and tag strings from the abstract syntax tree of each function as code features, converting the tag strings into hash values, and obtaining vulnerability dictionaries and target dictionaries includes the following steps: Traverse the abstract syntax tree of each function, count and record the key values ​​in the abstract syntax tree; Merge the information of each node of the abstract syntax tree together to form a string of tokens; The tuple consisting of the key value and the tag string is used as the code feature of the function, and the tag string is hashed and converted into a corresponding hash value; The key value counting, tag string generation and hashing feature steps are performed on all elements in the abstract syntax tree sets of the vulnerability code library and the target program source code library respectively, and two vulnerability dictionaries and target dictionaries consisting of two tuples are obtained, which correspond to the vulnerability code library and the target program source code library respectively.

7. The vulnerability detection method based on code similarity analysis according to claim 6, characterized in that: The method of calculating similarity based on the obtained vulnerability dictionary and the target dictionary, determining whether the target program has reuse of vulnerable code, and outputting the vulnerability detection result includes the following steps: Select a function f from the target dictionary, get the key value of function f, and search for a function with the same key value in the vulnerability dictionary; If no function with the same key value is found in the vulnerability dictionary, the function f is judged to be dissimilar to the known vulnerability and the process jumps to the iterative detection step; If a function with the same key value is found in the vulnerability dictionary, set the key value to key and execute the next step; For the function set with the same key value, further search in the vulnerability dictionary whether there is a function with the same hash value as the hash value of the function to be tested. If not, it is considered that the current function to be tested is not similar to the vulnerable code segment. If so, it is determined to be a vulnerable function and recorded; Determine whether all the functions to be tested in the target dictionary have been tested. If not, jump back to the key-value matching step; if completed, output the vulnerability detection results.

8. A system based on the vulnerability detection method based on code similarity analysis according to claim 1, characterized in that: The vulnerability code library construction module is used to obtain patches in the open source project repository to generate vulnerability codes and build a vulnerability code library; An abstract syntax tree generation module is used to parse the vulnerability code library and the source code library of the target program, and generate abstract syntax tree sets of the vulnerability code library and the source code library of the target program respectively according to the generation rules of the abstract syntax tree; An abstract processing module is used to perform abstract processing on the abstract syntax tree set through conversion rules to obtain a new abstract syntax tree set; The feature extraction and hashing module is used to extract key values ​​and tag strings as code features from the abstract syntax tree of each function based on the new abstract syntax tree set, and convert the tag strings into hash values ​​to obtain vulnerability dictionaries and target dictionaries; The similarity detection module is used to calculate the similarity based on the obtained vulnerability dictionary and the target dictionary, determine whether the target program has reused vulnerable code, and output the vulnerability detection result.

9. A computing device, characterized in that include: Memory, used to store programs; A processor, used for loading the program to execute the steps of the vulnerability detection method based on code similarity analysis as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the steps of the vulnerability detection method based on code similarity analysis as described in any one of claims 1 to 7 are implemented.

Images