A vulnerability mining method, device, equipment and medium

Abstract

The application discloses a vulnerability mining method and device, equipment and medium, and relates to the technical field of computers, which comprises the following steps: acquiring source code files in an operating system; retrieving corresponding target functions from all source code files by using a preset function dictionary based on a reverse index, so as to establish a corresponding relationship between the target functions and the source code files based on the reverse index; wherein the preset function dictionary is constructed based on non-repeating function names; acquiring a pattern string generated based on preset vulnerability keywords, and matching the pattern string with a code segment to be detected corresponding to the target function based on an exact character set algorithm to obtain a matching result, so as to determine whether there is a vulnerability in the target function according to the matching result; if there is a vulnerability, locating a vulnerability code segment from the code segment to be detected, and determining all source code files including the target function based on the corresponding relationship. The application improves the vulnerability mining efficiency by combining the reverse index and the exact character set algorithm.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a vulnerability discovery method, apparatus, device, and medium. Background Technology

[0002] Operating system vulnerabilities refer to flaws in the logical design or errors in the programming of application or operating system software. If attackers discover serious vulnerabilities, they can exploit them to gain unauthorized access to or damage server systems. This allows attackers to infiltrate malware or other malicious programs to attack or control the entire server, stealing important data and information, or even destroying the system itself. Discovering and patching vulnerabilities before attackers can effectively reduce threats from the network. Therefore, proactively discovering and analyzing system security vulnerabilities is of great significance in network attack and defense. Vulnerability discovery technology refers to the exploration of unknown vulnerabilities, comprehensively applying various technologies and tools to find as many potential vulnerabilities as possible in software.

[0003] The commonly used method for vulnerability discovery in existing operating systems is the construction algorithm based on traversal. By traversing the source code files, it detects file structures, naming rules, functions, stack pointers, etc. that do not conform to security rules in the program, and discovers security defects in the program. However, the traversal algorithm has the following disadvantages when processing large-scale text data: (1) There are cases of repeated vulnerability discovery, and it is not possible to quickly determine whether the discovered vulnerability is a known vulnerability or an unknown vulnerability; (2) The vulnerability discovery by the traversal algorithm is blind, and it is difficult to quickly locate the vulnerability within a limited time, which seriously affects the efficiency of vulnerability discovery; (3) Vulnerability discovery methods mainly rely on manual discovery or vulnerability detection tools. However, due to the flexibility of the tools and the limitation of file size, its scalability is low and it cannot be applied to operating system source code detection on a large scale; (4) It has high time complexity and space complexity, and sparse matrices consume a lot of memory space and are limited by scalability, which cannot reach the full performance provided by the hardware, ultimately resulting in poor efficiency.

[0004] In summary, improving the efficiency of vulnerability discovery is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a vulnerability discovery method, apparatus, device, and medium that can improve vulnerability discovery efficiency. The specific solution is as follows:

[0006] Firstly, this application discloses a vulnerability discovery method, including:

[0007] Obtain source code files from the operating system;

[0008] The corresponding target function is retrieved from all the source code files using a preset function dictionary based on inverted indexes, so as to establish a correspondence between the target function and the source code files based on the inverted indexes; wherein, the preset function dictionary is constructed based on non-repeating function names;

[0009] A pattern string generated based on preset vulnerability keywords is obtained, and the pattern string is matched with the code segment to be detected corresponding to the target function based on the precise character set algorithm to obtain the matching result, so as to determine whether there is a vulnerability in the target function based on the matching result;

[0010] If a vulnerability exists, the vulnerable code segment is located from the code segment to be detected, and all source code files containing the target function are determined based on the correspondence.

[0011] Optionally, before retrieving the corresponding target function from all the source code files using a preset function dictionary based on inverted indexes, the method further includes:

[0012] All the source code files are preprocessed to obtain preprocessed source code files; wherein the preprocessing operation includes removing comment statements and unclosed symbols from the source code files.

[0013] Optionally, after locating the vulnerable code segment from the code segment to be detected, the method further includes:

[0014] The vulnerable code snippet is compared with the code data stored in a preset vulnerability database;

[0015] If the preset vulnerability database contains code data that matches the vulnerability code fragment, then the vulnerability type of the vulnerability code fragment is determined to be a known vulnerability;

[0016] If no code data matching the vulnerability code fragment exists in the preset vulnerability database, the vulnerability type of the vulnerability code fragment is determined to be an unknown vulnerability.

[0017] Optionally, the step of matching the pattern string with the code segment to be detected corresponding to the target function based on the precise character set algorithm to obtain the matching result includes:

[0018] Identify the code segment to be detected corresponding to the target function;

[0019] Align the pattern string with the code segment to be detected corresponding to the target function on the left end, and perform character matching with the code segment to be detected starting from the rightmost end of the pattern string;

[0020] If the first character in the pattern string fails to match the second character that is aligned with the code segment to be detected, then the first shift offset corresponding to the first character is calculated based on the pattern string and according to the bad character rule in the precise character set algorithm, and the second shift offset corresponding to the first character is calculated based on the pattern string and according to the good suffix rule in the precise character set algorithm.

[0021] The target moving offset with the larger offset is determined from the first moving offset and the second moving offset, and the pattern string is moved based on the target moving offset to obtain the moved string. Then, the process jumps back to the step of matching characters with the code segment to be detected starting from the rightmost end of the pattern string, until the pattern string matches the code segment to be detected successfully or until the code segment to be detected has been scanned completely.

[0022] Optionally, determining the code segment to be detected corresponding to the target function includes:

[0023] Determine whether the total length of the code corresponding to the target function exceeds the target length;

[0024] If the total length of the code exceeds the target length, the target function is pruned based on the target length to obtain at least two target code segments, and each of the target code segments is used as a code segment to be detected.

[0025] If the total length of the code does not exceed the target length, then the target function is directly used as the code segment to be detected.

[0026] Optionally, the step of calculating the first shift offset corresponding to the first character based on the pattern string and according to the bad character rule in the precise character set algorithm includes:

[0027] Determine the first position number corresponding to the first character in the pattern string, and determine whether the second character exists in the pattern string;

[0028] If the second character exists in the pattern string, then determine the second position number corresponding to the rightmost second character in the pattern string, and use the difference between the first position number and the second position number as the first movement offset corresponding to the first character.

[0029] If the second character is not present in the pattern string, then the length of the pattern string is used as the first movement offset corresponding to the first character.

[0030] Optionally, the step of calculating the second shift offset corresponding to the first character based on the pattern string and according to the good suffix rule in the precise character set algorithm includes:

[0031] Determine good suffixes that match the pattern string from the code segment to be detected, and determine whether there are at least two of the good suffixes in the pattern string;

[0032] If there are at least two good suffixes in the pattern string, then determine the first positional distance between the first good suffix and the second good suffix from right to left, and use the first positional distance as the second movement offset corresponding to the first character;

[0033] If there are no at least two good suffixes in the pattern string, then determine whether the head of the pattern string includes a suffix substring of the good suffixes;

[0034] If the head of the pattern string includes the suffix substring of the good suffix, then the second positional distance between the suffix substring at the head and the suffix substring at the tail of the pattern string is determined, and the second positional distance is used as the second movement offset corresponding to the first character. If the head of the pattern string does not include the suffix substring of the good suffix, then the length of the pattern string is used as the second movement offset corresponding to the first character.

[0035] Secondly, this application discloses a vulnerability discovery device, comprising:

[0036] The source code acquisition module is used to acquire source code files from the operating system.

[0037] The reverse index generation module is used to retrieve the corresponding target function from all the source code files using a preset function dictionary based on the reverse index, so as to establish a correspondence between the target function and the source code file based on the reverse index; wherein, the preset function dictionary is constructed based on non-repeating function names;

[0038] The vulnerability detection module is used to obtain a pattern string generated based on preset vulnerability keywords, and to match the pattern string with the code segment to be detected corresponding to the target function based on the precise character set algorithm to obtain a matching result, so as to determine whether there is a vulnerability in the target function based on the matching result;

[0039] The location module is used to locate the vulnerable code segment from the code segment to be detected if a vulnerability exists, and to determine all source code files that include the target function based on the correspondence.

[0040] Thirdly, this application discloses an electronic device, including:

[0041] Memory, used to store computer programs;

[0042] A processor for executing the computer program to implement the steps of the aforementioned disclosed vulnerability discovery method.

[0043] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed vulnerability discovery method.

[0044] As can be seen, this application obtains source code files from the operating system; uses a preset function dictionary based on inverted indexes to retrieve corresponding target functions from all the source code files, establishing a correspondence between the target functions and the source code files based on the inverted indexes; wherein, the preset function dictionary is constructed based on non-repeating function names; obtains a pattern string generated based on preset vulnerability keywords, and matches the pattern string with the code segment to be detected corresponding to the target function based on a precise character set algorithm to obtain a matching result, thereby determining whether there is a vulnerability in the target function based on the matching result; if a vulnerability exists, the vulnerable code segment is located from the code segment to be detected, and all source code files containing the target function are determined based on the correspondence. Therefore, after obtaining the source code files from the operating system, this application needs to traverse all source code files based on a preset function dictionary with inverted indexes to retrieve the target functions corresponding to the preset function dictionary, wherein the preset function dictionary is specifically constructed based on non-repeating function names. Next, a correspondence is established between the target function and its source code files based on a reverse index. This allows for the location of all source code files containing the target function when a vulnerability exists, enabling rapid identification of duplicate functions and resolving the problem of repeatedly mining vulnerabilities in a specific target function. Furthermore, this application also obtains a pattern string generated based on preset vulnerability keywords and matches it with the corresponding code segment to be detected using a precise character set algorithm. The matching result determines whether a vulnerability exists in the target function. By employing a precise character set algorithm, the pattern string can be moved a considerable distance each time a match fails, reducing the detection time of the code segment to be detected, improving matching efficiency, and enabling rapid location of the vulnerable code segment from the code segment to be detected. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 This is a flowchart of a vulnerability discovery method disclosed in this application;

[0047] Figure 2 This is a flowchart of a specific vulnerability discovery method disclosed in this application;

[0048] Figure 3 This is a schematic diagram of a bad character disclosed in this application;

[0049] Figure 4 This is a schematic diagram illustrating a desirable suffix disclosed in this application;

[0050] Figure 5 This is a schematic diagram illustrating the first type of pattern string mismatch between the disclosed pattern string and the code segment to be detected.

[0051] Figure 6 This is a schematic diagram illustrating the mismatch between the second pattern string disclosed in this application and the code segment to be detected;

[0052] Figure 7 This is a schematic diagram of a moved pattern string disclosed in this application;

[0053] Figure 8 A schematic diagram illustrating the movement of the first pattern disclosed in this application with a suffix appended;

[0054] Figure 9 A schematic diagram illustrating the movement of the second mode disclosed in this application with a suffix appended;

[0055] Figure 10 A schematic diagram illustrating the movement of the third mode disclosed in this application with a suffix appended;

[0056] Figure 11 This is a schematic diagram of the structure of a vulnerability discovery device disclosed in this application;

[0057] Figure 12 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0059] The commonly used method for vulnerability discovery in existing operating systems is the construction algorithm based on traversal. By traversing the source code files, it detects file structures, naming rules, functions, stack pointers, etc. that do not conform to security rules in the program, and discovers security defects in the program. However, the traversal algorithm has the following disadvantages when processing large-scale text data: (1) There are cases of repeated vulnerability discovery, and it is not possible to quickly determine whether the discovered vulnerability is a known vulnerability or an unknown vulnerability; (2) The vulnerability discovery by the traversal algorithm is blind, and it is difficult to quickly locate the vulnerability within a limited time, which seriously affects the efficiency of vulnerability discovery; (3) Vulnerability discovery methods mainly rely on manual discovery or vulnerability detection tools. However, due to the flexibility of the tools and the limitation of file size, its scalability is low and it cannot be applied to operating system source code detection on a large scale; (4) It has high time complexity and space complexity, and sparse matrices consume a lot of memory space and are limited by scalability, which cannot reach the full performance provided by the hardware, ultimately resulting in poor efficiency.

[0060] Therefore, this application discloses a vulnerability discovery method, apparatus, device, and medium that can improve vulnerability discovery efficiency.

[0061] See Figure 1 As shown in the figure, this application discloses a vulnerability discovery method, which includes:

[0062] Step S11: Obtain the source code file from the operating system.

[0063] In this embodiment, the source code files in the collected operating system are first obtained.

[0064] Step S12: Use a preset function dictionary based on inverted index to retrieve the corresponding target function from all the source code files, so as to establish a correspondence between the target function and the source code file based on the inverted index; wherein, the preset function dictionary is constructed based on non-repeating function names.

[0065] In this embodiment, a preset function dictionary based on inverted indexes is used to traverse all source code files to retrieve target functions corresponding to the preset function dictionary. Specifically, the preset function dictionary is constructed based on non-repeating function names. Then, a correspondence is established between target functions and source code files based on the inverted index. This allows for the location of all source code files containing a vulnerability in a target function, enabling rapid identification of duplicate functions and resolving the problem of repeatedly mining vulnerabilities in a specific target function.

[0066] It's worth noting that during the process of establishing the correspondence between target functions and source code files using inverted indexes, information on frequently occurring functions can also be recorded. In other words, the example can also statistically analyze the frequency of function occurrences; frequently occurring functions refer to one or more functions that appear a very large number of times in the source code file. Therefore, when a vulnerability is detected using a frequently occurring function, it can alert users to pay extra attention, enhancing the user experience.

[0067] Before retrieving the corresponding target function from all the source code files using a preset function dictionary based on inverted indexes, the method further includes: preprocessing all the source code files to obtain preprocessed source code files; wherein, the preprocessing operation includes removing comment statements and unclosed symbols from the source code files. In this embodiment, after obtaining the source code files in the operating system, all source code files also need to be preprocessed to obtain preprocessed source code files. The specific preprocessing operation includes removing comment statements, unclosed symbols, and other irrelevant program statements from the source code files, thereby obtaining concise and efficient source code.

[0068] Step S13: Obtain a pattern string generated based on preset vulnerability keywords, and match the pattern string with the code segment to be detected corresponding to the target function based on the precise character set algorithm to obtain a matching result, so as to determine whether there is a vulnerability in the target function based on the matching result.

[0069] In this embodiment, it is also necessary to obtain a pattern string generated based on preset vulnerability keywords, and match the pattern string with the code segment to be detected corresponding to the target function based on the precise character set algorithm, so as to determine whether there is a vulnerability in the target function based on the matching result.

[0070] It should be noted that the Boyer-Moore (BM) algorithm is a string matching algorithm. Given two strings, the shorter string (called the pattern string) is used to match the longer string (the main string, which in this embodiment refers to the code segment to be detected) to see if it is a substring of the longer string. In this embodiment, the pattern string is a keyword of a vulnerability that has not yet been discovered in existing vulnerabilities. If the pattern string is a substring of the code segment to be detected, it indicates that a vulnerability exists in the target function. For example, in a password hard-coded plaintext storage vulnerability, the keywords are password, pass, and jdbc; in an SQL (database) injection vulnerability, the keywords are Select, Dao, from, delete, update, and insert; in a deserialization vulnerability, the keywords are ObjectInputStream.readObject, ObjectInputStream.readUnshared, XMLDecoder.readObject, Yaml.load, XStream.fromXML, ObjectMapper.readValue, and JSON.parseObject; these keywords are the pattern strings used for matching. In other words, this embodiment primarily identifies incorrect function calls and return states, especially function calls that may lack boundary checks or have incorrect boundary checks, functions that could cause buffer overflows, externally called functions, shared memory functions, and function pointers, etc. By detecting functions and stack pointers that do not comply with security rules in the program, security flaws can be discovered.

[0071] The above process, by employing a precise character set algorithm, can move the pattern string a greater distance each time a match fails, thereby reducing the detection time of the code segment to be detected, improving matching efficiency, and enabling the rapid location of the vulnerable code segment from the code segment to be detected.

[0072] Step S14: If a vulnerability exists, locate the vulnerable code segment from the code segment to be detected, and determine all source code files that include the target function based on the correspondence.

[0073] In this embodiment, if the target function has a vulnerability, the vulnerable code segment is located from the code segment to be detected based on the above matching results, and all source code files including the target function are determined based on the correspondence.

[0074] In a specific implementation, after locating the vulnerable code segment from the code segment to be detected, the method further includes: comparing the vulnerable code segment with code data stored in a preset vulnerability database; if code data matching the vulnerable code segment exists in the preset vulnerability database, the vulnerability type of the vulnerable code segment is determined to be a known vulnerability; if code data matching the vulnerable code segment does not exist in the preset vulnerability database, the vulnerability type of the vulnerable code segment is determined to be an unknown vulnerability. This embodiment can also determine whether a discovered vulnerability is a known vulnerability or an unknown vulnerability. Specifically, the vulnerable code segment in the code segment to be detected is compared with code data stored in the preset vulnerability database. If code data matching the vulnerable code segment exists in the preset vulnerability database, the vulnerability type of the vulnerable code segment is determined to be a known vulnerability; otherwise, it is an unknown vulnerability.

[0075] As can be seen, this application obtains source code files from the operating system; uses a preset function dictionary based on inverted indexes to retrieve corresponding target functions from all the source code files, establishing a correspondence between the target functions and the source code files based on the inverted indexes; wherein, the preset function dictionary is constructed based on non-repeating function names; obtains a pattern string generated based on preset vulnerability keywords, and matches the pattern string with the code segment to be detected corresponding to the target function based on a precise character set algorithm to obtain a matching result, thereby determining whether there is a vulnerability in the target function based on the matching result; if a vulnerability exists, the vulnerable code segment is located from the code segment to be detected, and all source code files containing the target function are determined based on the correspondence. Therefore, after obtaining the source code files from the operating system, this application needs to traverse all source code files based on a preset function dictionary with inverted indexes to retrieve the target functions corresponding to the preset function dictionary, wherein the preset function dictionary is specifically constructed based on non-repeating function names. Next, a correspondence is established between the target function and its source code files based on a reverse index. This allows for the location of all source code files containing the target function when a vulnerability exists, enabling rapid identification of duplicate functions and resolving the problem of repeatedly mining vulnerabilities in a specific target function. Furthermore, this application also obtains a pattern string generated based on preset vulnerability keywords and matches it with the corresponding code segment to be detected using a precise character set algorithm. The matching result determines whether a vulnerability exists in the target function. By employing a precise character set algorithm, the pattern string can be moved a considerable distance each time a match fails, reducing the detection time of the code segment to be detected, improving matching efficiency, and enabling rapid location of the vulnerable code segment from the code segment to be detected.

[0076] See Figure 2As shown, this application discloses a specific vulnerability discovery method. Compared to the previous embodiment, this embodiment further explains and optimizes the technical solution. Specifically, it includes:

[0077] Step S21: Obtain the source code file from the operating system.

[0078] Step S22: Use a preset function dictionary based on inverted index to retrieve the corresponding target function from all the source code files, so as to establish a correspondence between the target function and the source code file based on the inverted index; wherein, the preset function dictionary is constructed based on non-repeating function names.

[0079] Step S23: Obtain a pattern string generated based on preset vulnerability keywords, determine the code segment to be detected corresponding to the target function; align the pattern string and the code segment to be detected corresponding to the target function at the left end, and perform character matching between the pattern string and the code segment to be detected starting from the rightmost end of the pattern string.

[0080] In this embodiment, after obtaining the pattern string and the code segment to be detected, the pattern string and the code segment to be detected corresponding to the target function are aligned on the left, and character matching is performed on the pattern string starting from the rightmost end of the pattern string and the code segment to be detected. For example... Figure 3 As shown, Figure 3 In this context, T represents the pattern string, and S represents the code segment to be detected. After S and T are aligned on the left, character matching is performed from right to left.

[0081] In a specific implementation, determining the code segment to be detected corresponding to the target function includes: determining whether the total length of the code corresponding to the target function exceeds a target length; if the total length of the code exceeds the target length, then the target function is pruned based on the target length to obtain at least two target code segments, and each of the target code segments is used as a code segment to be detected; if the total length of the code does not exceed the target length, then the target function is directly used as a code segment to be detected. It is understood that some functions have very long code, and for ease of processing, they can be pruned. Specifically, a target length is preset. If the total length of the code corresponding to the target function exceeds the target length, then the target function is pruned based on the target length to obtain at least two target code segments, and each target code segment is used as a code segment to be detected. For example, assuming the total code length is 100 and the target length is 20, then 5 target code segments can be obtained, i.e., 5 code segments to be detected; if the total length is 118, then 6 code segments to be detected are obtained, with the first 5 code segments having a length of 20 and the 6th code segment having a length of 18.

[0082] Step S24: If the first character in the pattern string and the second character that is aligned in the code segment to be detected fail to match, then calculate the first movement offset corresponding to the first character based on the pattern string and according to the bad character rule in the precise character set algorithm, and calculate the second movement offset corresponding to the first character based on the pattern string and according to the good suffix rule in the precise character set algorithm.

[0083] In this embodiment, if the first character in the pattern string fails to match the corresponding second character in the code segment to be detected, a first shift offset corresponding to the first character is calculated based on the pattern string and according to the bad character rule in the precise character set algorithm, and a second shift offset corresponding to the first character is calculated based on the pattern string and according to the good suffix rule in the precise character set algorithm. It should be noted that the precise character set algorithm has two rules: the bad character rule and the good suffix rule. The purpose of these two rules is to make the pattern string move as far to the right as possible each time, thereby minimizing the string matching time. In this embodiment, reducing matching time means saving vulnerability scanning time. A bad character refers to the first unmatched character in the main string that appears during matching, starting from the last character of the pattern string. A good suffix refers to a character in the main string that matches successfully during matching, starting from the last character of the pattern string.

[0084] Examples of bad characters are as follows Figure 3 As shown in the image, the first character being compared does not match. Therefore, this character in the main string (S) is called the bad character, or f. Examples of good suffixes are shown below. Figure 4 As shown, Figure 4 We can see that the pattern string T and the main string S are matched from right to left. The last three characters are all matched, i.e., cac. The bad character c is only found in the fourth character. Therefore, cac, which has already been matched, is a good suffix.

[0085] In a specific implementation, the Badchar function can be used to calculate the first movement offset, while the Goodsuffix function can be used to calculate the second movement offset.

[0086] Specifically, the above-mentioned calculation of the first movement offset corresponding to the first character based on the pattern string and according to the bad character rule in the precise character set algorithm includes: determining the first position number corresponding to the first character in the pattern string, and determining whether the second character exists in the pattern string; if the second character exists in the pattern string, determining the second position number corresponding to the rightmost second character in the pattern string, and using the difference between the first position number and the second position number as the first movement offset corresponding to the first character; if the second character does not exist in the pattern string, using the length of the pattern string as the first movement offset corresponding to the first character.

[0087] In one specific implementation, when the first character in the pattern string does not match the corresponding second character in the code segment to be detected, it is determined whether the second character exists in the pattern string. If the second character also exists in the pattern string, and there is only one second character in the pattern string, then the second position number corresponding to the second character is determined, and the difference between the first position number and the second position number is used as the first movement offset corresponding to the first character. Figure 5 As shown, Figure 5 The first character A in the pattern string does not match the second character F in the code segment to be detected, meaning the bad character is F. However, the pattern string contains the second character F, and the number of F characters is 1. At this point, the position number difference between F in the pattern string and F in the code segment to be detected is 2. Therefore, the first shift offset is 2. By shifting the pattern string two positions to the right, the F in the pattern string can be aligned with the F in the code segment to be detected.

[0088] In the second specific implementation, if there are multiple second characters in the pattern string, the second position number corresponding to the rightmost second character in the pattern string is determined, thereby aligning the rightmost second character in the pattern string with the second character in the code segment to be detected. For example... Figure 6 As shown, Figure 6 The first character 'c' in the pattern string does not match the second character 'b' in the code segment to be detected, meaning the bad character is 'b'. However, there are two 'b' characters in the pattern string. Therefore, the position number difference between the rightmost 'b' and 'b' in the code segment to be detected is 2. Thus, the first shift offset is 2. By shifting the pattern string by two positions, the 'b' in the pattern string can be aligned with the 'b' in the code segment to be detected.

[0089] In the third specific implementation, when the second character is not present in the pattern string, the length of the pattern string is used as the first shift offset corresponding to the first character; that is, the pattern string is shifted to the right to one position after the bad character (i.e., the second character). Figure 3 and Figure 7 As shown, Figure 3 The pattern string does not contain the bad character f, therefore the shifting operation yields... Figure 7 The example shown.

[0090] When calculating the first shift offset using the Badchar function, for the character 'a' in the code segment to be detected, if character 'a' appears in the pattern string P, then Badchar[a] = min{i|1≤m-1 and P[m-1-i] = a}; if character 'a' does not appear in the pattern string P, then Badchar[a] = m. Here, m is the length of the pattern string, and i is a text pointer pointing to the code segment to be detected.

[0091] Specifically, the above-mentioned calculation of the second movement offset corresponding to the first character based on the pattern string and according to the good suffix rule in the precise character set algorithm includes: determining good suffixes matching the pattern string from the code segment to be detected, and determining whether there are at least two good suffixes in the pattern string; if there are at least two good suffixes in the pattern string, determining the first positional distance between the first good suffix and the second good suffix from right to left, and using the first positional distance as the second movement offset corresponding to the first character; if there are no at least two good suffixes in the pattern string, determining whether the head of the pattern string includes a suffix substring of the good suffix; if the head of the pattern string includes a suffix substring of the good suffix, determining the second positional distance between the suffix substring at the head and the suffix substring at the tail of the pattern string, and using the second positional distance as the second movement offset corresponding to the first character; if the head of the pattern string does not include a suffix substring of the good suffix, using the length of the pattern string as the second movement offset corresponding to the first character.

[0092] Understandably, this embodiment first determines good suffixes that match the pattern string from the code segment to be detected. Understandably, if the code segment to be detected and the pattern string do not match from the rightmost position, then there is no good suffix. If a good suffix exists in the pattern string, it further determines whether there are at least two good suffixes. If there are at least two good suffixes, it determines the first positional distance between the first and second good suffixes from right to left, and uses this first positional distance as the second shift offset corresponding to the first character. For example... Figure 8 As shown, Figure 8 The good suffix is ​​cac, and there are two good suffixes cac in the pattern string T. Therefore, the first positional distance between these two good suffixes is determined, and the first positional distance is used as the second shift offset corresponding to the first character, so that the second good suffix in the pattern string is aligned with the good suffix in the code segment to be detected.

[0093] Furthermore, if the pattern string does not contain at least two good suffixes, it is determined whether the head of the pattern string includes a suffix substring of good suffixes; if the head of the pattern string includes a suffix substring of good suffixes, the second positional distance between the suffix substring at the head and the suffix substring at the tail of the pattern string is determined, and this second positional distance is used as the second shift offset corresponding to the first character. For example... Figure 9 As shown, Figure 9 The good suffix is ​​cac, and the suffix substrings of cac are c and ac. At this time, the head of the pattern string includes the suffix substring ac. Therefore, the second position distance between the suffix substring at the head and the suffix substring at the tail of the pattern string is determined, and the second position distance is used as the second shift offset corresponding to the first character, so that the good suffix ac at the head of the pattern string is aligned with ac in the code segment to be detected.

[0094] Furthermore, if the beginning of the pattern string does not include a suffix substring with a good suffix, then the length of the pattern string is used as the second shift offset corresponding to the first character. It can be understood that if the pattern string contains only one good suffix, and the beginning does not include a suffix substring with a good suffix, then the length of the pattern string is used as the second shift offset corresponding to the first character, such as... Figure 10 As shown in the image.

[0095] When using the Goodsuffix function to calculate the second move offset, its expression is Goodsuffix[j], where j is the position of the character in the pattern string when the code fragment to be detected does not match the pattern string at a certain time.

[0096] Step S25: Determine the target moving offset with the larger offset from the first moving offset and the second moving offset, and move the pattern string based on the target moving offset to obtain the moved string. Then, jump back to the step of matching characters with the code segment to be detected starting from the rightmost end of the pattern string, until the pattern string matches the code segment to be detected successfully or until the code segment to be detected has been scanned completely.

[0097] In this embodiment, when a match fails, the pattern string needs to be moved as far as possible to minimize string matching time and improve vulnerability discovery efficiency. Specifically, this embodiment determines the target offset, which is larger than the first and second offsets, and moves the pattern string based on the target offset to obtain the moved string. That is, according to the precise character set algorithm, the values ​​of Badchar[T[i+j]]-(mj) and Goodsuffix[j] are compared, and the larger one is taken as the offset. After the movement is completed, the process jumps back to the step of matching characters with the code segment to be detected, starting from the rightmost end of the pattern string, until the pattern string matches the code segment to be detected successfully or until the code segment to be detected has been scanned completely.

[0098] Step S26: Determine whether there is a vulnerability in the target function based on the matching result. If there is a vulnerability, locate the vulnerable code segment from the code segment to be detected, and determine all source code files that include the target function based on the correspondence.

[0099] For more detailed processing procedures of steps S21, S22 and S26, please refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0100] As can be seen, in this embodiment, after obtaining the pattern string and the code segment to be detected, the pattern string and the code segment to be detected corresponding to the target function are aligned at the left end, and character matching is performed on the pattern string starting from the rightmost end of the pattern string and the code segment to be detected. When the code of the target function is long, it can be trimmed for easier processing. If the first character in the pattern string and the aligned second character in the code segment to be detected fail to match, a first shift offset corresponding to the first character is calculated based on the pattern string and according to the bad character rule in the precise character set algorithm, and a second shift offset corresponding to the first character is calculated based on the pattern string and according to the good suffix rule in the precise character set algorithm. The purpose of these two rules is to make the pattern string move as far to the right as possible each time, thereby minimizing the string matching time. In this embodiment, reducing the matching time means saving vulnerability scanning time. Therefore, this embodiment determines the target shift offset with the larger offset from the first and second shift offsets, and moves the pattern string based on the target shift offset to obtain the moved string.

[0101] See Figure 11 As shown in the figure, this application discloses a vulnerability discovery device, which includes:

[0102] Source code acquisition module 11 is used to acquire source code files in the operating system;

[0103] The reverse index generation module 12 is used to retrieve the corresponding target function from all the source code files using a preset function dictionary based on the reverse index, so as to establish a correspondence between the target function and the source code file based on the reverse index; wherein, the preset function dictionary is constructed based on non-repeating function names;

[0104] The vulnerability detection module 13 is used to obtain a pattern string generated based on preset vulnerability keywords, and to match the pattern string with the code segment to be detected corresponding to the target function based on the precise character set algorithm to obtain a matching result, so as to determine whether there is a vulnerability in the target function based on the matching result;

[0105] The positioning module 14 is used to locate the vulnerable code segment from the code segment to be detected if a vulnerability exists, and to determine all source code files that include the target function based on the correspondence.

[0106] As can be seen, this application obtains source code files from the operating system; uses a preset function dictionary based on inverted indexes to retrieve corresponding target functions from all the source code files, establishing a correspondence between the target functions and the source code files based on the inverted indexes; wherein, the preset function dictionary is constructed based on non-repeating function names; obtains a pattern string generated based on preset vulnerability keywords, and matches the pattern string with the code segment to be detected corresponding to the target function based on a precise character set algorithm to obtain a matching result, thereby determining whether there is a vulnerability in the target function based on the matching result; if a vulnerability exists, the vulnerable code segment is located from the code segment to be detected, and all source code files containing the target function are determined based on the correspondence. Therefore, after obtaining the source code files from the operating system, this application needs to traverse all source code files based on a preset function dictionary with inverted indexes to retrieve the target functions corresponding to the preset function dictionary, wherein the preset function dictionary is specifically constructed based on non-repeating function names. Next, a correspondence is established between the target function and its source code files based on a reverse index. This allows for the location of all source code files containing the target function when a vulnerability exists, enabling rapid identification of duplicate functions and resolving the problem of repeatedly mining vulnerabilities in a specific target function. Furthermore, this application also obtains a pattern string generated based on preset vulnerability keywords and matches it with the corresponding code segment to be detected using a precise character set algorithm. The matching result determines whether a vulnerability exists in the target function. By employing a precise character set algorithm, the pattern string can be moved a considerable distance each time a match fails, reducing the detection time of the code segment to be detected, improving matching efficiency, and enabling rapid location of the vulnerable code segment from the code segment to be detected.

[0107] In some specific embodiments, before retrieving the corresponding target function from all the source code files using a preset function dictionary based on inverted indexes, the device is further configured to preprocess all the source code files to obtain preprocessed source code files; wherein, the preprocessing operation includes removing comment statements and unclosed symbols from the source code files.

[0108] In some specific embodiments, after locating the vulnerable code segment from the code segment to be detected, the device is further configured to compare the vulnerable code segment with code data stored in a preset vulnerability database; if code data matching the vulnerable code segment exists in the preset vulnerability database, the vulnerability type of the vulnerable code segment is determined to be a known vulnerability; if code data matching the vulnerable code segment does not exist in the preset vulnerability database, the vulnerability type of the vulnerable code segment is determined to be an unknown vulnerability.

[0109] In some specific embodiments, the vulnerability detection module 13 may specifically include:

[0110] A code fragment determination unit is used to determine the code fragment to be detected corresponding to the target function;

[0111] A character matching unit is used to align the pattern string with the code segment to be detected corresponding to the target function on the left end, and to perform character matching with the code segment to be detected starting from the rightmost end of the pattern string;

[0112] The movement offset calculation unit is used to calculate the first movement offset corresponding to the first character based on the pattern string and according to the bad character rule in the precise character set algorithm if the first character in the pattern string and the relatively aligned second character in the code segment to be detected fail to match. It is also used to calculate the second movement offset corresponding to the first character based on the pattern string and according to the good suffix rule in the precise character set algorithm.

[0113] The pattern string moving unit is used to determine the target moving offset with the larger offset from the first moving offset and the second moving offset, and move the pattern string based on the target moving offset to obtain the moved string. Then, it jumps back to the step of matching characters with the code segment to be detected starting from the rightmost end of the pattern string, until the pattern string matches the code segment to be detected successfully or until the code segment to be detected has been scanned completely.

[0114] In some specific embodiments, the code fragment determination unit may specifically include:

[0115] A length determination unit is used to determine whether the total code length corresponding to the target function exceeds the target length;

[0116] The code trimming unit is used to trim the target function based on the target length to obtain at least two target code segments if the total length of the code exceeds the target length, and to use each target code segment as a code segment to be detected.

[0117] The code segment to be detected determination unit is used to directly use the target function as the code segment to be detected if the total length of the code does not exceed the target length.

[0118] In some specific embodiments, the movement offset calculation unit may specifically include:

[0119] The second character determination unit is used to determine the first position number corresponding to the first character in the pattern string, and to determine whether the second character exists in the pattern string;

[0120] The first movement offset determination unit is configured to, if the second character exists in the pattern string, determine the second position number corresponding to the rightmost second character in the pattern string, and use the difference between the first position number and the second position number as the first movement offset corresponding to the first character; if the second character does not exist in the pattern string, use the length of the pattern string as the first movement offset corresponding to the first character.

[0121] In some specific embodiments, the movement offset calculation unit may specifically include:

[0122] A good suffix determination unit is used to determine good suffixes that match the pattern string from the code segment to be detected, and to determine whether there are at least two good suffixes in the pattern string;

[0123] The second movement offset determination unit is configured to: if at least two good suffixes exist in the pattern string, determine a first positional distance between the first good suffix and the second good suffix from right to left, and use the first positional distance as a second movement offset corresponding to the first character; if at least two good suffixes do not exist in the pattern string, determine whether the head of the pattern string includes a suffix substring of the good suffixes; if the head of the pattern string includes a suffix substring of the good suffixes, determine a second positional distance between the suffix substring at the head of the pattern string and the suffix substring at the tail of the pattern string, and use the second positional distance as a second movement offset corresponding to the first character; if the head of the pattern string does not include a suffix substring of the good suffixes, use the length of the pattern string as a second movement offset corresponding to the first character.

[0124] Figure 12This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the vulnerability discovery method executed by the electronic device disclosed in any of the foregoing embodiments.

[0125] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0126] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0127] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0128] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the vulnerability discovery method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0129] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program, which, when loaded and executed by a processor, implements the vulnerability discovery method steps disclosed in any of the foregoing embodiments.

[0130] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0131] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0132] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0133] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0134] The present invention provides a detailed description of a vulnerability discovery method, apparatus, device, and storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A vulnerability discovery method, characterized in that, include: Obtain source code files from the operating system; The corresponding target function is retrieved from all the source code files using a preset function dictionary based on inverted indexes, so as to establish a correspondence between the target function and the source code files based on the inverted indexes; wherein, the preset function dictionary is constructed based on non-repeating function names; A pattern string generated based on preset vulnerability keywords is obtained, and the pattern string is matched with the code segment to be detected corresponding to the target function based on the precise character set algorithm to obtain the matching result, so as to determine whether there is a vulnerability in the target function based on the matching result; If a vulnerability exists, the vulnerable code segment is located from the code segment to be detected, and all source code files containing the target function are determined based on the correspondence.

2. The vulnerability discovery method according to claim 1, characterized in that, Before retrieving the corresponding target function from all the source code files using a preset function dictionary based on inverted indexes, the method further includes: All the source code files are preprocessed to obtain preprocessed source code files; wherein the preprocessing operation includes removing comment statements and unclosed symbols from the source code files.

3. The vulnerability discovery method according to claim 1, characterized in that, After locating the vulnerable code segment from the code segment to be detected, the method further includes: The vulnerable code snippet is compared with the code data stored in a preset vulnerability database; If the preset vulnerability database contains code data that matches the vulnerability code fragment, then the vulnerability type of the vulnerability code fragment is determined to be a known vulnerability. If no code data matching the vulnerability code fragment exists in the preset vulnerability database, the vulnerability type of the vulnerability code fragment is determined to be an unknown vulnerability.

4. The vulnerability discovery method according to any one of claims 1 to 3, characterized in that, The matching result obtained by matching the pattern string with the code segment to be detected corresponding to the target function based on the precise character set algorithm includes: Identify the code segment to be detected corresponding to the target function; Align the pattern string with the code segment to be detected corresponding to the target function on the left, and perform character matching with the code segment to be detected starting from the rightmost end of the pattern string; If the first character in the pattern string fails to match the second character that is aligned with the code segment to be detected, then the first shift offset corresponding to the first character is calculated based on the pattern string and according to the bad character rule in the precise character set algorithm, and the second shift offset corresponding to the first character is calculated based on the pattern string and according to the good suffix rule in the precise character set algorithm. The target moving offset with the larger offset is determined from the first moving offset and the second moving offset, and the pattern string is moved based on the target moving offset to obtain the moved string. Then, the process jumps back to the step of matching characters with the code segment to be detected starting from the rightmost end of the pattern string, until the pattern string matches the code segment to be detected successfully or until the code segment to be detected has been scanned completely.

5. The vulnerability discovery method according to claim 4, characterized in that, The process of determining the code segment to be detected corresponding to the target function includes: Determine whether the total length of the code corresponding to the target function exceeds the target length; If the total length of the code exceeds the target length, the target function is pruned based on the target length to obtain at least two target code segments, and each of the target code segments is used as a code segment to be detected. If the total length of the code does not exceed the target length, then the target function is directly used as the code segment to be detected.

6. The vulnerability discovery method according to claim 4, characterized in that, The step of calculating the first shift offset corresponding to the first character based on the pattern string and according to the bad character rule in the precise character set algorithm includes: Determine the first position number corresponding to the first character in the pattern string, and determine whether the second character exists in the pattern string; If the second character exists in the pattern string, then determine the second position number corresponding to the rightmost second character in the pattern string, and use the difference between the first position number and the second position number as the first movement offset corresponding to the first character. If the second character is not present in the pattern string, then the length of the pattern string is used as the first movement offset corresponding to the first character.

7. The vulnerability discovery method according to claim 4, characterized in that, The step of calculating the second shift offset corresponding to the first character based on the pattern string and according to the good suffix rule in the precise character set algorithm includes: Determine good suffixes that match the pattern string from the code segment to be detected, and determine whether there are at least two of the good suffixes in the pattern string; If there are at least two good suffixes in the pattern string, then determine the first positional distance between the first good suffix and the second good suffix from right to left, and use the first positional distance as the second movement offset corresponding to the first character; If there are no at least two good suffixes in the pattern string, then determine whether the head of the pattern string includes a suffix substring of the good suffixes; If the head of the pattern string includes the suffix substring of the good suffix, then the second positional distance between the suffix substring at the head and the suffix substring at the tail of the pattern string is determined, and the second positional distance is used as the second movement offset corresponding to the first character. If the head of the pattern string does not include the suffix substring of the good suffix, then the length of the pattern string is used as the second movement offset corresponding to the first character.

8. A vulnerability discovery device, characterized in that, include: The source code acquisition module is used to acquire source code files from the operating system. The reverse index generation module is used to retrieve the corresponding target function from all the source code files using a preset function dictionary based on the reverse index, so as to establish a correspondence between the target function and the source code file based on the reverse index; wherein, the preset function dictionary is constructed based on non-repeating function names; The vulnerability detection module is used to obtain a pattern string generated based on preset vulnerability keywords, and to match the pattern string with the code segment to be detected corresponding to the target function based on the precise character set algorithm to obtain a matching result, so as to determine whether there is a vulnerability in the target function based on the matching result; The location module is used to locate the vulnerable code segment from the code segment to be detected if a vulnerability exists, and to determine all source code files that include the target function based on the correspondence.

9. An electronic device, characterized in that, include: Memory is used to store computer programs; A processor for executing the computer program to implement the steps of the vulnerability discovery method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the vulnerability discovery method as described in any one of claims 1 to 7.

Images