A method and apparatus for repeated vulnerability finding

Abstract

The application discloses a repeated vulnerability searching method and device, and relates to the technical field of computers, and mainly aims at improving the efficiency and accuracy of vulnerability searching. The main technical scheme comprises the following steps: determining the vulnerability category of a to-be-classified vulnerability based on vulnerability information of the to-be-classified vulnerability; selecting target sample information matched with the vulnerability information from sample information corresponding to the vulnerability category, wherein each vulnerability category corresponds to at least one sample information, and one sample information corresponds to one sample vulnerability; and determining that the to-be-classified vulnerability and the sample vulnerability corresponding to the target sample information are repeated vulnerabilities.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for finding duplicate vulnerabilities. Background Technology

[0002] A vulnerability is a logical design flaw or an error made during the development of application software or operating system software. The existence of a vulnerability allows an attacker to access or damage the system without authorization. To reduce the possibility of malicious attacks, vulnerability detection is necessary during the design phase of application software or operating system software, and detected vulnerabilities must be eliminated. Duplicate vulnerabilities can be handled using the same or similar vulnerability elimination methods; therefore, it is necessary to identify duplicate vulnerabilities.

[0003] Currently, the common method for identifying duplicate vulnerabilities is to rely on the engineer's subjective judgment to determine whether the root cause of the vulnerability being addressed is the same as that of previously addressed vulnerabilities. If the engineer's subjective judgment indicates that the root cause is the same, no further vulnerability analysis is conducted, and the vulnerability being addressed is identified as a duplicate vulnerability. This method of identifying duplicate vulnerabilities relies entirely on the engineer's subjective judgment, which is prone to misjudgment. On the one hand, it may lead to new vulnerabilities being treated as duplicates, resulting in new vulnerabilities remaining in the software development project. On the other hand, it may cause duplicate vulnerabilities to go undetected, resulting in wasted time analyzing already duplicate vulnerabilities and increasing manpower costs.

[0004] It is evident that this method of finding duplicate vulnerabilities by relying on engineers' subjective perception is not only inefficient but also inaccurate. Summary of the Invention

[0005] In view of this, the present invention proposes a method and apparatus for finding duplicate vulnerabilities, the main purpose of which is to improve the efficiency and accuracy of vulnerability deduplication.

[0006] To achieve the above objectives, the present invention mainly provides the following technical solutions:

[0007] In a first aspect, the present invention provides a method for finding repeated vulnerabilities, the method comprising:

[0008] The vulnerability category of the vulnerability to be classified is determined based on the vulnerability information of the vulnerability to be classified;

[0009] Select target sample information that matches the vulnerability information from the sample information corresponding to the vulnerability category, wherein each vulnerability category corresponds to at least one sample information, and one sample information corresponds to one sample vulnerability;

[0010] The sample vulnerability corresponding to the vulnerability to be classified and the target sample information is determined to be a duplicate vulnerability.

[0011] In a second aspect, the present invention provides a repeat vulnerability detection device, the device comprising:

[0012] The first determining unit is used to determine the vulnerability category of the vulnerability to be classified based on the vulnerability information of the vulnerability to be classified.

[0013] The selection unit is used to select target sample information that matches the vulnerability information from the sample information corresponding to the vulnerability category, wherein each vulnerability category corresponds to at least one sample information, and one sample information corresponds to one sample vulnerability;

[0014] The second determining unit is used to determine that the sample vulnerability corresponding to the vulnerability to be classified and the target sample information is a duplicate vulnerability.

[0015] Thirdly, the present invention provides a computer-readable storage medium comprising a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the duplicate vulnerability finding method described in the first aspect.

[0016] Fourthly, the present invention provides a storage management device, the storage management device comprising:

[0017] Memory, used to store programs;

[0018] A processor, coupled to the memory, is used to run the program to perform the duplicate vulnerability finding method described in the first aspect.

[0019] By employing the above technical solution, the duplicate vulnerability detection method and apparatus provided by this invention, when a vulnerability to be classified exists, firstly determines the vulnerability category of the vulnerability to be classified based on its vulnerability information. Then, target sample information matching the vulnerability information of the vulnerability to be classified is selected from the sample information corresponding to the vulnerability category. Finally, the sample vulnerability corresponding to the vulnerability to be classified and the target sample information is determined to be a duplicate vulnerability. It is evident that the solution provided by this invention does not require manual intervention in determining whether a vulnerability to be classified is a duplicate vulnerability; it only requires matching the vulnerability information of the vulnerability to be classified with the sample information corresponding to the sample vulnerability under the vulnerability category. This avoids situations where human error leads to the failure to identify duplicate vulnerabilities or the misclassification of new vulnerabilities as duplicate vulnerabilities, thus improving the accuracy of vulnerability deduplication detection. Furthermore, since the target sample information is selected only from the sample information corresponding to the vulnerability category of the vulnerability to be classified, the vulnerability category limits the range of target sample information, reducing the computational load of selection, thus improving the efficiency of vulnerability deduplication detection.

[0020] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 The flowchart of a duplicate vulnerability finding method provided by an embodiment of the present invention is shown;

[0023] Figure 2 A flowchart of a duplicate vulnerability finding method provided by another embodiment of the present invention is shown;

[0024] Figure 3 This diagram illustrates a structural schematic of a repeat vulnerability detection device according to an embodiment of the present invention.

[0025] Figure 4 A schematic diagram of a repeat vulnerability finding device provided in another embodiment of the present invention is shown. Detailed Implementation

[0026] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0027] A vulnerability, or bug, is a flaw in the logical design or an error in the writing of application or operating system software that allows an attacker to access or damage the system without authorization. To reduce the possibility of malicious attacks, vulnerability detection is necessary during the design phase of application or operating system software, and detected vulnerabilities must be eliminated.

[0028] Duplicate vulnerabilities can be handled using the same or similar vulnerability removal methods. Therefore, to improve the efficiency of vulnerability removal, it is necessary to identify duplicate vulnerabilities during the removal process. Currently, the common method for finding duplicate vulnerabilities is to rely on the engineer's subjective perception to determine whether the root cause of the vulnerability being addressed is the same as that of previously addressed vulnerabilities. If the engineer's subjective perception of the root cause is consistent, no further vulnerability analysis is conducted, and the vulnerability being addressed is identified as a duplicate. This method of identifying duplicate vulnerabilities has at least two drawbacks: First, engineers have limited personal experience, and relying on their subjective judgment can easily lead to misjudgments, treating new vulnerabilities as duplicates, resulting in new vulnerabilities persisting in the software development project. Second, the engineer's subjective judgment is based on vulnerabilities they have previously recognized or addressed. If the same vulnerability exists in the software development project but the engineer is unaware of it, the duplicate vulnerability will require further time for analysis, increasing the unnecessary human resource costs of the software development project.

[0029] It is evident that relying on engineers' subjective perception to identify duplicate vulnerabilities is not only inefficient but also inaccurate. Therefore, to improve the efficiency and accuracy of vulnerability deduplication detection, this invention proposes a method and apparatus for finding duplicate vulnerabilities, thereby improving both efficiency and accuracy. The following is a detailed description of the method and apparatus for finding duplicate vulnerabilities proposed in this invention.

[0030] like Figure 1 As shown in the figure, this embodiment of the invention provides a method for finding duplicate vulnerabilities, which mainly includes:

[0031] 101. Determine the vulnerability category of the vulnerability to be classified based on the vulnerability information of the vulnerability to be classified.

[0032] Vulnerabilities awaiting classification are those that require remediation. To improve the efficiency of remediation, it's necessary to check if any previously remediated vulnerabilities are identical to those awaiting classification. Two identical vulnerabilities are considered duplicates. Duplicate vulnerabilities involve the same remediation methods. Therefore, when facing duplicate vulnerabilities, there's no need to redesign the remediation methods; simply use the existing methods to remediate them, thus improving efficiency.

[0033] After identifying the vulnerabilities to be classified, it is necessary to obtain their vulnerability information. Vulnerability information describes the specific circumstances of the vulnerabilities to be classified. This information can be obtained by web crawling from the vulnerabilities to be classified. The specific content included in the vulnerability information can be determined based on specific business requirements; this embodiment does not impose specific limitations.

[0034] Illustratively, the vulnerability information includes one or more of the following contents: vulnerability summary, vulnerability description and vulnerability comment. Wherein, the vulnerability summary, namely summary, is used to describe the category information of the corresponding vulnerability. The vulnerability description, namely description, is used to describe the specific content of the corresponding vulnerability in the software development project, for example, which steps of which specific module unit in the software development project the vulnerability involves. The vulnerability comment, namely comments, is the artificial annotation and evaluation made by engineers for the vulnerability.

[0035] After obtaining the vulnerability information of the vulnerability to be classified, it is necessary to determine the vulnerability category of the vulnerability to be classified, and use the vulnerability category as the basic basis for searching duplicate vulnerabilities. The specific process of determining the vulnerability category of the vulnerability to be classified based on the vulnerability information of the vulnerability to be classified is described below, and the process includes the following step 1 to step 2:

[0036] Step 1: performing word segmentation processing on the vulnerability information of the vulnerability to be classified.

[0037] The vulnerability information can characterize the category of the vulnerability to be classified, so the vulnerability information can be used as a basis for determining the vulnerability category of the vulnerability to be classified.

[0038] When performing word segmentation processing on the vulnerability information of the vulnerability to be classified, the following processing is first required to be performed on the vulnerability information: First, remove punctuation marks and Arabic numerals in the vulnerability information, for example, <>, ." 1 2 3 and so on. Second, for vulnerability information containing English, convert words in English descriptions that change according to different persons or tenses into original verb forms, for example, "attacking" and "attacked" can be classified as belonging to the same word, and both are converted to the original form "attack". Third, filter stop words. Stop words refer to words that appear frequently in the text and contribute little or even nothing to the content or classification category of the text information, such as prepositions, articles, auxiliary words, modal verbs, pronouns and conjunctions, etc. For example, the Chinese structural auxiliary words "de", "di" and "de" are all stop words. After the above processing is performed on the vulnerability information, the following two tools can be used to perform word segmentation on the processed vulnerability information: First, use the NLTK (Natural Language Toolkit) tool to perform word segmentation processing on the vulnerability information. Second, perform word segmentation on the vulnerability information by means of matching the vulnerability information with a word segmentation bank, wherein the specific type of the word segmentation bank can be determined based on business requirements, which is not specifically limited in this embodiment. Optionally, the word segmentation bank is the jieba word segmentation bank.

[0039] For example, the vulnerability summary "Test machine restarted." is processed by word segmentation. First, the punctuation mark "." is removed. Since there are no stop words or verbs that need to be converted to their base form in the vulnerability summary, the jieba word segmentation library is used to directly segment "Test machine restarted." The segmentation result is "Test machine, restarted."

[0040] It should be noted that the vulnerability information used to determine the vulnerability category of a vulnerability to be classified can be one or more of the following: vulnerability summary, vulnerability description, and vulnerability annotation. Optionally, to reduce the computational burden of determining the vulnerability category of a vulnerability to be classified, only the vulnerability summary is used, because the vulnerability summary describes the category information of the corresponding vulnerability.

[0041] Step 2: Input the word segmentation results of the vulnerability information into the vulnerability category recognition model to obtain the vulnerability category of the vulnerability to be classified.

[0042] The vulnerability category identification model is a pre-trained model used to identify vulnerability categories. It takes the word segmentation results of vulnerability information as input and the vulnerability category as output. Optionally, the vulnerability category identification model can be a neural network model or a clustering model.

[0043] After obtaining the word segmentation results of the vulnerability information, these results are used as input to the vulnerability category recognition model. The vulnerability category recognition model then processes the word segmentation results, and the output vulnerability category is the vulnerability category of the vulnerability to be classified.

[0044] For example, the word segmentation result of the vulnerability summary of the vulnerability to be classified is "test machine, occurrence, restart". The word segmentation result is input into the clustering model. After the clustering model identifies the word segmentation result, it outputs the vulnerability category of the vulnerability to be classified as restart.

[0045] 102. Select target sample information that matches the vulnerability information from the sample information corresponding to the vulnerability category.

[0046] To improve the efficiency of vulnerability deduplication and reduce the computational burden, multiple vulnerability categories need to be defined. Each vulnerability category corresponds to at least one sample information, and each sample information corresponds to one sample vulnerability. The following explains how to set up vulnerability categories and the sample information under each category. This process can include the following steps one through four:

[0047] Step 1: Perform word segmentation on the vulnerability information of each sample vulnerability in the vulnerability database.

[0048] The vulnerability database is a database deployed on a bug server, which includes a large number of sample vulnerabilities. The vulnerability elimination method for sample vulnerabilities has been determined, and a vulnerability repeated with a sample vulnerability can directly use the corresponding vulnerability elimination method of the sample vulnerability. After the vulnerability database is determined, a web crawler can be used to crawl the key field content of each sample vulnerability from the bug server, and this key field content is vulnerability information. Optionally, the vulnerability information can be one or more of vulnerability summary, vulnerability description and vulnerability annotation.

[0049] The following describes the process of performing word segmentation processing on the vulnerability information of each sample vulnerability in the vulnerability database: firstly, the vulnerability information needs to be processed in the following aspects: first, remove punctuation marks and Arabic numerals in the vulnerability information, for example, <>, "1 2 3, etc. Second, for vulnerability information containing English, convert words in English descriptions that vary according to person or tense to their original verb forms. For example, "attacking" and "attacked" can be classified as belonging to the same word, and both are converted to the original form "attack". Third, filter stop words. Stop words refer to words that appear frequently in the text and contribute little or even nothing to the content or classification of the text information, such as prepositions, articles, auxiliary words, modal verbs, pronouns and conjunctions, etc. For example, the Chinese structural auxiliary words "de", "di" and "de" are all stop words. Then, after the vulnerability information has been processed as described above, the following two tools can be used to perform word segmentation on the processed vulnerability information: first, use the NLTK (Natural Language Toolkit) tool to perform word segmentation processing on the vulnerability information. Second, perform word segmentation on the vulnerability information by means of matching the vulnerability information with a word segmentation library, wherein the specific type of the word segmentation library can be determined based on business requirements, which is not specifically limited in this embodiment. Optionally, the word segmentation library is the jieba word segmentation library.

[0050] For example, the word segmentation processing performed on vulnerability information 1 "the tester crashes" of sample vulnerability 1 results in: test machine, occurrence, crash; the word segmentation processing performed on vulnerability information 2 "the test machine restarts" of sample vulnerability 2 results in: test machine, occurrence, restart; the word segmentation processing performed on vulnerability information 3 "the test machine has anr" of sample vulnerability 3 results in: test machine, occurrence, anr; the word segmentation processing performed on vulnerability information 4 "monkey screen freeze" of sample vulnerability 4 results in: monkey, screen freeze.

[0051] It should be noted that, in order to expand the number of sample vulnerabilities in the vulnerability database, after the vulnerability to be classified is eliminated, the vulnerability to be classified is added to the vulnerability database as a sample vulnerability. When it is necessary to update the setting of vulnerability categories again, the updated vulnerability database can be used, so as to continuously improve the accuracy and comprehensiveness of vulnerability category setting.

[0052] Step 2: Input the word segmentation results of the vulnerability information of each sample vulnerability into the vulnerability category clustering model for clustering.

[0053] The clustering model is used for vulnerability category clustering and identification. It takes the word segmentation results of vulnerability information as input, classifies the word segmentation results of the obtained sample vulnerability information, and outputs the vulnerability category. The specific type of clustering model is not specifically limited in this embodiment and can be determined based on business requirements. Optionally, the clustering model is an unsupervised clustering model.

[0054] The following describes the process of inputting the word segmentation results of the vulnerability information of each sample vulnerability into the vulnerability category clustering model for clustering: First, after segmenting the vulnerability information of each sample vulnerability in the vulnerability database, the segmentation results are compiled into a word list, such as a keyword list. This word list records the keywords extracted from each segmentation result, and which sample vulnerability each keyword appears from. Second, the TF-IDF (term frequency–inverse document frequency) algorithm is used to calculate the TF-IDF value of each keyword in the word segmentation list. Finally, based on the TF-IDF values ​​of each keyword in the word segmentation list, the vector of each sample vulnerability is calculated, and the obtained vectors are input into the vulnerability category clustering model for clustering.

[0055] For example, vulnerability information 1 of sample vulnerability 1 is segmented into: test machine, occurrence, crash; vulnerability information 2 of sample vulnerability 2 is segmented into: test machine, occurrence, restart; vulnerability information 3 of sample vulnerability 3 is segmented into: test machine, occurrence, ANR; vulnerability information 4 of sample vulnerability 4 is segmented into: monkey, screen freeze. A segmentation list is formed based on the segmentation results of the vulnerability information of the above sample vulnerabilities. This segmentation list includes keywords and an explanation of which sample vulnerability each keyword appears in. The segmentation list includes the following keywords: "test machine, occurrence, crash, restart, ANR, screen freeze, monkey" and the source information of each keyword: "Test machine appears in sample vulnerability 1, sample vulnerability 2, and sample vulnerability 3; occurrence appears in sample vulnerability 1, sample vulnerability 2, and sample vulnerability 3; crash appears in sample vulnerability 1; restart appears in sample vulnerability 2; ANR appears in sample vulnerability 3; screen freeze appears in sample vulnerability 4; monkey appears in sample vulnerability 4." Then, the TF-IDF algorithm is used to calculate the TF-IDF value of each keyword in the word segmentation list. The following example illustrates the process of calculating the TF-IDF value of the keyword "test machine": The TF value of "test machine" is 3 / (3+3+3+2), where the numerator 3 indicates that "test machine" appears 3 times in the word segmentation results of vulnerability information 1 to vulnerability information 4, and the denominators 3, 3, 3, and 2 refer to the number of words segmented in the word segmentation results of vulnerability information 1 to vulnerability information 4, respectively. The IDF value of "test machine" is lg(4 / 3), where 4 refers to the total number of sample vulnerabilities, and 3 refers to the appearance of the word "test machine" in the three word segmentation results of vulnerability information 1 to vulnerability information 3. The product of the TF value and the IDF value of "test machine" is then determined as its TF-IDF value. Based on the TF-IDF values ​​of each keyword in the word segmentation list, calculate the vector for each sample vulnerability. For example, the vector for sample vulnerability 1 is [TF-IDF value of "test machine", TF-IDF value of "occurred", TF-IDF value of "crash", 0, 0, 0, 0]. Since "restart", "freeze screen", "anr", and "monkey" do not appear in the word segmentation results of vulnerability information 1 for sample vulnerability 1, they are filled with 0. Similarly, the vector for sample vulnerability 2 is [TF-IDF value of "test machine", TF-IDF value of "occurred", 0, TF-IDF value of "restart", 0, 0, 0], the vector for sample vulnerability 3 is [TF-IDF value of "test machine", TF-IDF value of "occurred", 0, 0, TF-IDF value of "anr", 0, 0], and the vector for sample vulnerability 4 is [0, 0, 0, 0, 0, TF-IDF value of "freeze screen", TF-IDF value of "monkey"]. Finally, the vectors of vulnerabilities in each sample are input into the vulnerability category clustering model for clustering.

[0056] Step 3: Based on the clustering results of the clustering model, determine multiple vulnerability categories and the sample vulnerability corresponding to each vulnerability category.

[0057] The clustering model categorizes the segmented vulnerability information of the acquired sample vulnerabilities. The resulting clustering includes the categorization category and the sample vulnerabilities involved in each category. After obtaining the clustering results, each categorization category is determined as a vulnerability category, and the sample vulnerabilities under each categorization category are determined as sample vulnerabilities under the corresponding vulnerability category.

[0058] For example, the clustering results of the clustering model are as follows: the categories include application issues, restart issues, and ANR issues, with application issues corresponding to sample vulnerabilities 1-3, restart issues to sample vulnerabilities 4-9, and ANR issues to sample vulnerabilities 10-16. Therefore, the vulnerability categories are set as application issue category, restart issue category, and ANR issue category, with application issue category corresponding to sample vulnerabilities 1-3, restart issue category corresponding to sample vulnerabilities 4-9, and ANR issue category corresponding to sample vulnerabilities 10-16.

[0059] Step 4: Based on the word segmentation processing results of the vulnerability information of each sample vulnerability, set the sample information corresponding to each sample vulnerability.

[0060] Sample vulnerabilities are those that have already been eliminated. Vulnerabilities that duplicate these vulnerabilities can be eliminated using the same elimination methods. Therefore, based on the word segmentation results of the vulnerability information of each sample vulnerability, sample information corresponding to each sample vulnerability can be set to find duplicate vulnerabilities based on the sample information. The process of setting sample information is described below, and this process can be as follows:

[0061] For each sample vulnerability, the following steps are performed: Extract target information from the word segmentation results of the vulnerability information for that sample vulnerability, and set this target information as the sample information for that vulnerability. The target information includes the following types: First, extracting the word segmentation results of at least one of the vulnerability summary, vulnerability description, and vulnerability annotations involved in the vulnerability information as the target information. Second, extracting the word segmentation results of the vulnerability description and vulnerability annotations involved in the vulnerability information as the target information.

[0062] For example, vulnerability information 1 of sample vulnerability 1 is segmented into: test machine, occurrence, crash; vulnerability information 2 of sample vulnerability 2 is segmented into: test machine, occurrence, restart; vulnerability information 3 of sample vulnerability 3 is segmented into: test machine, occurrence, ANR; vulnerability information 4 of sample vulnerability 4 is segmented into: monkey, screen freeze. The above segmentation results are all segmentation results of vulnerability summaries. Each segmentation result selects the target information of its corresponding sample vulnerability and sets the target information as the sample information of its corresponding sample vulnerability.

[0063] After determining the vulnerability category of the vulnerability to be classified, target sample information that matches the vulnerability information of the vulnerability to be classified is selected from the sample information corresponding to that vulnerability category. This selection process includes the following steps one and two:

[0064] Step 1: Determine the similarity between the vulnerability information of the vulnerability to be classified and the sample information of the corresponding vulnerability category.

[0065] Specifically, firstly, the vulnerability information to be classified is segmented into words. The word segmentation process is basically the same as that mentioned in step 101, and will not be repeated here. After word segmentation, the word segmentation results of the vulnerabilities to be classified and the words involved in each sample information are compiled into a word segmentation list. This word segmentation list records the keywords extracted from the word segmentation results and each sample information, as well as where each keyword appears. For example, whether a keyword appears in the word segmentation results of the vulnerabilities to be classified or in the sample information; if it appears in the sample information, it needs to record which sample information it comes from. Secondly, the TF-IDF (term frequency–inverse document frequency) algorithm is used to calculate the TF-IDF value of each keyword in the word segmentation list. Finally, based on the TF-IDF values ​​of each keyword in the word segmentation list, the vector of the sample vulnerability corresponding to each sample information and the vector of the vulnerability to be classified are calculated. Finally, the similarity value between the vector of the vulnerability to be classified and the vector of each sample vulnerability is determined. The similarity value can be calculated using any of the following methods: Euclidean distance algorithm, Pearson correlation coefficient algorithm, and cosine similarity algorithm.

[0066] Step 2: If the number of maximum similarities is 1 and the maximum similarity is greater than the similarity threshold, select the sample information corresponding to the maximum similarity as the target sample information.

[0067] The greater the similarity between the vulnerability information of the vulnerability to be classified and the sample information, the higher the probability that the vulnerability to be classified and the corresponding sample vulnerability are duplicate vulnerabilities. Therefore, the sample information corresponding to the maximum similarity can be selected as the target sample information that matches the vulnerability information of the vulnerability to be classified, so as to determine whether the vulnerability to be classified and the sample vulnerability corresponding to the target sample information are duplicate vulnerabilities.

[0068] In practical applications, the selection of target sample information needs to consider the following two factors:

[0069] The first is the numerical value of the maximum similarity.

[0070] Specifically, if the similarity between the vulnerability information of the vulnerability to be classified and the sample information is not high, then the probability that the sample vulnerability corresponding to the sample information with the highest similarity is a duplicate vulnerability of the vulnerability to be classified is also very low. Therefore, to avoid this situation, a similarity threshold needs to be set. The similarity threshold is used to limit the probability that a sample vulnerability and the vulnerability to be classified are duplicate vulnerabilities. When the similarity of a sample vulnerability is greater than the similarity threshold, it indicates that the probability of the sample vulnerability and the vulnerability to be classified being duplicate vulnerabilities is relatively high; conversely, when the similarity of a sample vulnerability is not greater than the similarity threshold, it indicates that the probability of the sample vulnerability and the vulnerability to be classified being duplicate vulnerabilities is relatively low.

[0071] The second is the number of maximum similarities.

[0072] If the maximum similarity score is 1, it indicates a high probability that only one sample vulnerability is a duplicate of the vulnerability to be classified. In other words, the sample information with the highest similarity score has a high probability of being a duplicate of the vulnerability to be classified. If the maximum similarity score is greater than 1, it indicates a high probability that multiple sample vulnerabilities are duplicates of the vulnerability to be classified, suggesting an anomaly. A classification prompt will then be issued for the vulnerability to be classified, allowing engineers to intervene manually based on this prompt.

[0073] Considering both factors above, if, after determining the maximum similarity, the number of maximum similarities is 1, and the maximum similarity is greater than the similarity threshold, it indicates a high probability that the sample vulnerability corresponding to the maximum similarity is a duplicate vulnerability of the vulnerability to be classified. In this case, the sample information corresponding to the maximum similarity is selected as the target sample information. If the number of maximum similarities is greater than 1 or the maximum similarity is not greater than the similarity threshold, a classification prompt is issued for the vulnerability to be classified. If the maximum similarity is not greater than the similarity threshold, it indicates that there are no sample vulnerabilities that duplicate the vulnerability to be classified. In this case, a classification prompt is issued for the vulnerability to be classified so that engineers can intervene manually based on the classification prompt. If the number of maximum similarities is greater than 1, it indicates a high probability that multiple sample vulnerabilities are duplicate vulnerabilities of the vulnerability to be classified, indicating an anomaly. In this case, a classification prompt is issued for the vulnerability to be classified so that engineers can intervene manually based on the classification prompt.

[0074] 103. Determine that the sample vulnerability corresponding to the vulnerability to be classified and the target sample information is a duplicate vulnerability.

[0075] The vulnerability to be classified and the corresponding sample vulnerability in the target sample information are identified as duplicate vulnerabilities. Since duplicate vulnerabilities involve the same vulnerability removal methods, when faced with duplicate vulnerabilities, there is no need to redesign the vulnerability removal method; the vulnerability removal method of the sample vulnerability can be directly used to remove the vulnerability to be classified, thereby improving the efficiency of vulnerability removal.

[0076] The duplicate vulnerability detection method provided in this invention, when a vulnerability to be classified exists, first determines the vulnerability category of the vulnerability to be classified based on its vulnerability information. Then, it selects target sample information that matches the vulnerability information of the vulnerability to be classified from the sample information corresponding to the vulnerability category. Finally, it determines that the sample vulnerability corresponding to the vulnerability to be classified and the target sample information is a duplicate vulnerability. It can be seen that the solution provided by this invention does not require manual intervention in determining whether a vulnerability to be classified is a duplicate vulnerability; it only needs to match the vulnerability information of the vulnerability to be classified with the sample information corresponding to the sample vulnerability under the vulnerability category. This avoids situations where human error leads to the failure to identify duplicate vulnerabilities or the treatment of new vulnerabilities as duplicate vulnerabilities, thus improving the accuracy of vulnerability deduplication detection. Furthermore, since the target sample information is selected only from the sample information corresponding to the vulnerability category of the vulnerability to be classified, the vulnerability category limits the range of target sample information, reducing the computational workload of selection, thus improving the efficiency of vulnerability deduplication detection.

[0077] Furthermore, according to Figure 1 The method shown in the figure, in another embodiment of the present invention, also provides a method for finding repeated vulnerabilities, such as Figure 2 As shown, the method mainly includes:

[0078] 201. Perform word segmentation on the vulnerability information of each sample vulnerability in the vulnerability database.

[0079] In this embodiment, the vulnerability information of each sample vulnerability in the vulnerability database can be segmented in the following two situations: First, when a clustering instruction is received. Second, at the start of a new clustering cycle.

[0080] 202. Input the word segmentation results of the vulnerability information of each sample vulnerability into the vulnerability category clustering model for clustering.

[0081] 203. Based on the clustering results of the clustering model, determine multiple vulnerability categories and the sample vulnerability corresponding to each vulnerability category.

[0082] 204. Based on the word segmentation processing results of the vulnerability information of each sample vulnerability, set the sample information corresponding to each sample vulnerability.

[0083] 205. Determine the vulnerability category of the vulnerability to be classified based on the vulnerability information of the vulnerability to be classified.

[0084] 206. Perform word segmentation on the vulnerability information.

[0085] 207. Based on the word segmentation results of the vulnerability information, calculate the similarity between the vulnerability information and each of the sample information.

[0086] 208. Determine if the maximum similarity is 1. If yes, proceed to 209; otherwise, proceed to 213.

[0087] 209. Determine if the maximum similarity is greater than the similarity threshold. If it is, proceed to step 210; otherwise, proceed to step 213.

[0088] 210. Select the sample information corresponding to the maximum similarity as the target sample information that matches the vulnerability information.

[0089] 211. Determine that the sample vulnerability corresponding to the vulnerability to be classified and the target sample information is a duplicate vulnerability.

[0090] 212. After the vulnerability to be classified has been eliminated, the vulnerability to be classified is added as a sample vulnerability to the vulnerability database, and 201 is executed.

[0091] 213. Issue a classification prompt for the vulnerability to be classified, and execute 212.

[0092] It should be noted that after issuing a classification suggestion for a vulnerability to be classified, if the engineer determines the vulnerability elimination method for the vulnerability to be classified, the vulnerability to be classified can be added to the vulnerability database to enrich the vulnerability database, regardless of whether the vulnerability to be classified is a new category of vulnerability.

[0093] Furthermore, based on the above method embodiments, another embodiment of the present invention also provides a repeat vulnerability finding device, such as... Figure 3 As shown, the device includes:

[0094] The first determining unit 31 is used to determine the vulnerability category of the vulnerability to be classified based on the vulnerability information of the vulnerability to be classified.

[0095] The selection unit 32 is used to select target sample information that matches the vulnerability information from the sample information corresponding to the vulnerability category, wherein each vulnerability category corresponds to at least one sample information, and one sample information corresponds to one sample vulnerability.

[0096] The second determining unit 33 is used to determine that the sample vulnerability corresponding to the vulnerability to be classified and the target sample information is a duplicate vulnerability.

[0097] The duplicate vulnerability detection device provided in this invention, when a vulnerability to be classified exists, first determines the vulnerability category of the vulnerability to be classified based on its vulnerability information. Then, it selects target sample information that matches the vulnerability information of the vulnerability to be classified from the sample information corresponding to the vulnerability category. Finally, it determines that the sample vulnerability corresponding to the vulnerability to be classified and the target sample information is a duplicate vulnerability. It is evident that the solution provided by this invention does not require manual intervention in determining whether a vulnerability to be classified is a duplicate vulnerability; it only requires matching the vulnerability information of the vulnerability to be classified with the sample information corresponding to the sample vulnerability under the vulnerability category. This avoids situations where human error leads to the failure to identify duplicate vulnerabilities or the misclassification of new vulnerabilities as duplicate vulnerabilities, thus improving the accuracy of vulnerability deduplication detection. Furthermore, since the target sample information is selected only from the sample information corresponding to the vulnerability category of the vulnerability to be classified, the vulnerability category limits the range of target sample information, reducing the computational workload of selection, thereby improving the efficiency of vulnerability deduplication detection.

[0098] Optional, such as Figure 4 As shown, the selection unit 32 includes:

[0099] The first determining module 321 is used to determine the similarity between the vulnerability information and the sample information of each vulnerability category;

[0100] The selection module 322 is used to select the sample information corresponding to the maximum similarity as the target sample information if the number of maximum similarities is 1 and the maximum similarity is greater than the similarity threshold.

[0101] Optional, such as Figure 4 As shown, the device further includes:

[0102] The prompting unit 34 is used to issue a classification prompt for the vulnerability to be classified if the number of maximum similarities is greater than 1 or the maximum similarity is not greater than the similarity threshold.

[0103] Optionally, the first determining module 321 is specifically used to perform word segmentation processing on the vulnerability information; and to calculate the similarity between the vulnerability information and each of the sample information based on the word segmentation processing results of the vulnerability information.

[0104] Optional, such as Figure 4 As shown, the device further includes:

[0105] The setting unit 35 is used to perform word segmentation processing on the vulnerability information of each sample vulnerability in the vulnerability database; input the word segmentation processing results of the vulnerability information of each sample vulnerability into the vulnerability category clustering model for clustering; determine multiple vulnerability categories and the sample vulnerability corresponding to each vulnerability category based on the clustering results of the clustering model; and set the sample information corresponding to each sample vulnerability according to the word segmentation processing results of the vulnerability information of each sample vulnerability.

[0106] Optional, such as Figure 4 As shown, the device further includes:

[0107] Adding unit 36 ​​is used to add the vulnerability to be classified as a sample vulnerability to the vulnerability database after the vulnerability to be classified has been eliminated.

[0108] Optional, such as Figure 4 As shown, the first determining unit 31 includes:

[0109] The word segmentation module 311 is used to perform word segmentation processing on the vulnerability information of the vulnerabilities to be classified.

[0110] The second determining module 312 is used to input the word segmentation processing result of the vulnerability information into the vulnerability category recognition model to obtain the vulnerability category of the vulnerability to be classified.

[0111] For a detailed explanation of the methods used in the operation of each functional module in the duplicate vulnerability detection device provided in this embodiment of the invention, please refer to [link to relevant documentation]. Figures 1-2 The detailed explanations of the corresponding methods in the method embodiments will not be repeated here.

[0112] Furthermore, according to the above embodiments, another embodiment of the present invention also provides a computer-readable storage medium, the storage medium including a stored program, wherein the program controls the device where the storage medium is located to execute [the program]. Figure 1 or Figure 2 The method for finding duplicate vulnerabilities.

[0113] Furthermore, according to the above embodiments, another embodiment of the present invention also provides a storage management device, the storage management device comprising:

[0114] Memory, used to store programs;

[0115] A processor, coupled to the memory, is used to run the program to execute... Figure 1 or Figure 2 The method for finding duplicate vulnerabilities.

[0116] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0117] It is understood that the relevant features in the above methods and apparatus can be referenced interchangeably. Furthermore, the terms "first," "second," etc., in the above embodiments are used to distinguish between embodiments and do not represent the superiority or inferiority of any particular embodiment.

[0118] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, embodiments of this disclosure can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this disclosure can take the form of computer program products embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0119] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0120] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0121] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0122] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0123] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0124] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0125] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0126] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, embodiments of this disclosure can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this disclosure can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0127] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for finding duplicate vulnerabilities, characterized in that, The method includes: The vulnerability category of the vulnerability to be classified is determined based on the vulnerability information of the vulnerability to be classified. The vulnerability information includes one or more of the following: vulnerability summary, vulnerability description, and vulnerability annotation; the vulnerability summary is used to describe the category information of the vulnerability to be classified; the vulnerability description is used to describe the module unit involved in the software development project to which the vulnerability to be classified belongs and the steps involved in the module unit; the vulnerability annotation is a human annotation and evaluation of the vulnerability. Selecting target sample information that matches the vulnerability information from sample information corresponding to the vulnerability category in the vulnerability database, wherein each vulnerability category corresponds to at least one sample information, and one sample information corresponds to one sample vulnerability; selecting target sample information that matches the vulnerability information from sample information corresponding to the vulnerability category in the vulnerability database includes: determining the similarity between the vulnerability information and each sample information of the vulnerability category; if the maximum similarity is 1, and the maximum similarity is greater than a similarity threshold, selecting the sample information corresponding to the maximum similarity as the target sample information; wherein, determining the vulnerability information and the vulnerability category... The similarity between the sample information includes: performing word segmentation on the vulnerability information; organizing the word segmentation results and the words involved in each sample information into a word segmentation list, wherein the word segmentation list records the keywords extracted from the word segmentation results and each sample information and the source of each keyword, wherein the source is used to determine whether the corresponding keyword originates from the word segmentation results or from a certain sample information; calculating the term frequency-inverse document frequency value of each keyword in the word segmentation list; calculating the vector of the sample vulnerability corresponding to each sample information and the vector of the vulnerability to be classified based on the term frequency-inverse document frequency value of each keyword; and determining the similarity value between the vector of the vulnerability to be classified and the vector of each sample vulnerability. The sample vulnerability corresponding to the vulnerability to be classified and the target sample information is determined to be a duplicate vulnerability, so that the vulnerability elimination method of the corresponding sample vulnerability is used to perform vulnerability elimination operation on the vulnerability to be classified. The method further includes: after the vulnerability to be classified is eliminated, adding the vulnerability to be classified as a sample vulnerability to the vulnerability database; Upon receiving a clustering instruction or at the start of a new clustering cycle, the vulnerability information of each sample vulnerability in the vulnerability database is segmented into words; the segmentation results of the vulnerability information of each sample vulnerability are input into a vulnerability category clustering model for clustering; based on the clustering results of the clustering model, multiple vulnerability categories are determined, and sample vulnerabilities corresponding to each vulnerability category are determined; according to the segmentation results of the vulnerability information of each sample vulnerability, sample information corresponding to each sample vulnerability is set; the segmentation results of the vulnerability information of each sample vulnerability are input into the vulnerability category clustering model for clustering, including: organizing the segmentation results of the vulnerability information of each sample vulnerability into a segmentation list, the segmentation list recording the keywords extracted from each segmentation result and the sample vulnerabilities in which each keyword appears; calculating the term frequency-inverse document frequency value of each keyword in the segmentation list; calculating the vector of each sample vulnerability according to the term frequency-inverse document frequency value of each keyword; and inputting the obtained vector into the vulnerability category clustering model for clustering.

2. The method according to claim 1, characterized in that, The method further includes: If the number of maximum similarities is greater than 1 or the maximum similarity is not greater than the similarity threshold, a classification prompt is issued for the vulnerability to be classified.

3. The method according to claim 1, characterized in that, Determining the similarity between the vulnerability information and sample information of each vulnerability category includes: The vulnerability information is processed by word segmentation; Based on the word segmentation results of the vulnerability information, the similarity between the vulnerability information and each of the sample information is calculated.

4. The method according to any one of claims 1-3, characterized in that, Determining the vulnerability category of the vulnerability to be classified based on its vulnerability information includes: The vulnerability information of the vulnerabilities to be classified is processed by word segmentation; The word segmentation results of the vulnerability information are input into the vulnerability category recognition model to obtain the vulnerability category of the vulnerability to be classified.

5. A device for finding repeated vulnerabilities, characterized in that, The device includes: The first determining unit is used to determine the vulnerability category of the vulnerability to be classified based on the vulnerability information of the vulnerability to be classified. The vulnerability information includes one or more of the following: vulnerability summary, vulnerability description, and vulnerability annotation; the vulnerability summary is used to describe the category information of the vulnerability to be classified; the vulnerability description is used to describe the module unit involved in the software development project to which the vulnerability to be classified belongs and the steps involved in the module unit; the vulnerability annotation is a human annotation and evaluation of the vulnerability. The selection unit is used to select target sample information that matches the vulnerability information from the sample information corresponding to the vulnerability category in the vulnerability database, wherein each vulnerability category corresponds to at least one sample information, and one sample information corresponds to one sample vulnerability. The second determining unit is used to determine that the sample vulnerability corresponding to the vulnerability to be classified and the target sample information is a duplicate vulnerability, so as to use the vulnerability elimination method of the corresponding sample vulnerability to perform vulnerability elimination operation on the vulnerability to be classified. The selection unit includes: The first determining module is used to determine the similarity between the vulnerability information and each sample information of the vulnerability category; wherein, determining the similarity between the vulnerability information and each sample information of the vulnerability category includes: performing word segmentation on the vulnerability information; organizing the word segmentation results and the words involved in each sample information into a word segmentation list, wherein the word segmentation list records keywords extracted from the word segmentation results and each sample information and the source of each keyword, wherein the source is used to determine whether the corresponding keyword originates from the word segmentation results or from a certain sample information; calculating the term frequency-inverse document frequency value of each keyword in the word segmentation list; calculating the vector of the sample vulnerability corresponding to each sample information and the vector of the vulnerability to be classified based on the term frequency-inverse document frequency value of each keyword; and determining the similarity value between the vector of the vulnerability to be classified and the vector of each sample vulnerability. The selection module is used to select the sample information corresponding to the maximum similarity as the target sample information if the number of maximum similarities is 1 and the maximum similarity is greater than the similarity threshold. The device further includes: An adding unit is used to add the vulnerability to be classified as a sample vulnerability to the vulnerability database after the vulnerability to be classified has been eliminated. A setting unit is configured to, upon receiving a clustering instruction or at the start of a new clustering cycle, perform word segmentation processing on the vulnerability information of each sample vulnerability in the vulnerability database; input the word segmentation processing results of the vulnerability information of each sample vulnerability into a vulnerability category clustering model for clustering; based on the clustering results of the clustering model, determine multiple vulnerability categories and the sample vulnerability corresponding to each vulnerability category; set sample information corresponding to each sample vulnerability according to the word segmentation processing results of the vulnerability information of each sample vulnerability; input the word segmentation processing results of the vulnerability information of each sample vulnerability into the vulnerability category clustering model for clustering, including: organizing the word segmentation results of the vulnerability information of each sample vulnerability into a word segmentation list, the word segmentation list recording the keywords extracted from each word segmentation result and the sample vulnerability in which each keyword appears; calculating the term frequency-inverse document frequency value of each keyword in the word segmentation list; calculating the vector of each sample vulnerability according to the term frequency-inverse document frequency value of each keyword; and inputting the obtained vector into the vulnerability category clustering model for clustering.

6. A computer-readable storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to execute the duplicate vulnerability finding method according to any one of claims 1 to 4.

7. A storage management device, characterized in that, The storage management device includes: Memory, used to store programs; A processor, coupled to the memory, is used to run the program to perform the duplicate vulnerability finding method according to any one of claims 1 to 4.

Images