A zero-day vulnerability identification method, device, equipment and storage medium
By collecting and filtering log data in the cloud, and utilizing hunting strategies and data noise reduction techniques, the problem of low efficiency in zero-day vulnerability identification in existing technologies has been solved, achieving fast and accurate zero-day vulnerability identification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SANGFOR TECH INC
- Filing Date
- 2024-12-02
- Publication Date
- 2026-06-02
AI Technical Summary
Existing technologies for identifying zero-day vulnerabilities are inefficient and resource-limited by manual discovery, while fuzzing requires a large amount of computing resources and has the problem of false alarms, making it difficult to quickly and accurately identify zero-day vulnerabilities.
By continuously collecting log data in the cloud, using a pre-set hunting strategy to filter the first log data suspected of being zero-day vulnerabilities, and performing data noise reduction and labeling, the priority is determined based on the amount of log data for zero-day vulnerability identification.
It enables rapid and accurate identification of zero-day vulnerabilities in the cloud, improving vulnerability hunting efficiency, narrowing the scope of data analysis, reducing false alarms, and improving identification efficiency.
Smart Images
Figure CN122133147A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of vulnerability discovery technology, and in particular to a method, apparatus, device and storage medium for identifying zero-day vulnerabilities. Background Technology
[0002] Zero-day vulnerabilities are those that have not been publicly disclosed and for which there are no official patches. With the normalization of attack and defense, the number of zero-day vulnerabilities exploited in the wild is increasing year by year. Nowadays, enterprises and organizations are paying more attention to identifying zero-day vulnerabilities in addition to how to identify and detect publicly disclosed vulnerabilities.
[0003] Currently, there are two main types of techniques for discovering zero-day vulnerabilities. The first is manual vulnerability discovery. However, manual discovery has a high technical threshold, with only a few security experts possessing the ability to discover vulnerabilities. Furthermore, manual vulnerability discovery is inefficient, often taking a considerable amount of time to discover vulnerabilities in a single software application. Additionally, because vulnerability discovery requires software resources, its scope is relatively limited; for some commercial software, the lack of available software resources makes manual vulnerability discovery impossible. The second method is fuzzing testing. However, fuzzing testing requires significant computing and memory resources, especially when dealing with large and complex software systems. Furthermore, fuzzing testing is prone to false positives.
[0004] Therefore, the aforementioned technical problems urgently need to be solved by those skilled in the art. Summary of the Invention
[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for identifying zero-day vulnerabilities, capable of accurately and quickly identifying zero-day vulnerabilities. The specific solution is as follows:
[0006] The first aspect of this application provides a zero-day vulnerability identification method, including:
[0007] Log data continuously collected from the target server is stored in the cloud, and the first log data suspected of being a zero-day vulnerability is filtered out from the log data based on a pre-set hunting strategy;
[0008] The first log data is subjected to noise reduction processing to remove data unrelated to zero-day vulnerabilities from the first log data, resulting in second log data. The second log data is then analyzed to label the corresponding data tags.
[0009] The amount of log data corresponding to the data tags is continuously counted, and zero-day vulnerabilities are identified in each of the second log data according to the priority determined by the amount of log data.
[0010] Optionally, storing the log data continuously collected from the target server in the cloud includes:
[0011] The data acquisition engine continuously collects log data from the target server, and after anonymizing the log data, it is uploaded to a data lake in the cloud for storage. The log data includes network request data, process behavior data, service status data, and memory data.
[0012] Optionally, the step of filtering suspected zero-day vulnerability data from the log data based on a pre-set hunting strategy includes:
[0013] Using a first hunting strategy based on preset high-risk vulnerability interface characteristics, first log data suspected of being zero-day vulnerabilities is filtered from the log data;
[0014] Alternatively, a second hunting strategy based on common vulnerability attack characteristics can be used to filter out first log data suspected of being zero-day vulnerabilities from the log data;
[0015] Alternatively, a third hunting strategy based on the network request and response characteristics generated after a successful vulnerability attack can be used to filter out the first log data suspected of being a zero-day vulnerability from the log data;
[0016] Alternatively, a fourth hunting strategy based on vulnerability patch repair characteristics can be used to filter out the first log data suspected of being a zero-day vulnerability from the log data;
[0017] Alternatively, a fifth hunting strategy based on external attacker characteristics can be used to filter out the first log data suspected of being a zero-day vulnerability from the log data.
[0018] Optionally, the step of performing data noise reduction processing on the first log data to remove data unrelated to zero-day vulnerabilities from the first log data and obtain the second log data includes:
[0019] The first log data is input into a pre-trained target neural network model to output a target feature vector;
[0020] Each of the target feature vectors is matched with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, respectively, so as to remove data unrelated to zero-day vulnerabilities from the first log data based on the matching results, and obtain the second log data; wherein, the application fingerprint feature vector library is used to store application fingerprint feature information related to the application environment of zero-day vulnerabilities.
[0021] Optionally, the step of matching each of the target feature vectors with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, respectively, to remove data unrelated to zero-day vulnerabilities from the first log data based on the matching results, includes:
[0022] Each target feature vector is compared with each historical known vulnerability feature vector in the historical known vulnerability feature vector library to obtain a corresponding first similarity value. If any first similarity value is greater than a first preset threshold, the corresponding first log data is determined to be data related to historical known vulnerabilities, and the log data corresponding to any first similarity value is cleared from the first log data.
[0023] The similarity between each target feature vector and each application fingerprint feature vector in the application fingerprint feature vector library is calculated to obtain the corresponding second similarity value. If any second similarity value is less than a second preset threshold, the corresponding second log data is determined to be data unrelated to the zero-day vulnerability application environment, and the log data corresponding to any second similarity value is cleared from the first log data.
[0024] Optionally, before matching each of the target feature vectors with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, the method further includes:
[0025] Calculate the similarity between every two target feature vectors to obtain the corresponding third similarity value;
[0026] If any third similarity value is greater than the third preset threshold, the corresponding two target feature vectors are aggregated to obtain the aggregated target feature vector.
[0027] Accordingly, the matching of each target feature vector with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library includes:
[0028] The aggregated target feature vectors are matched with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, respectively.
[0029] Optionally, the training process of the target neural network model includes:
[0030] The data in the historical known vulnerability database and the application fingerprint database are standardized to generate target data that meets the preset format requirements, and each target data is labeled with preset key information; wherein, the preset key information includes Uniform Resource Locator, payload information, vulnerability type and application component information;
[0031] The initial neural network model is trained using the target data to obtain the target neural network model.
[0032] Optionally, after identifying zero-day vulnerabilities in each of the second log data according to the priority determined by the log data volume, the method further includes:
[0033] Based on the identified zero-day vulnerabilities, vulnerability screening and monitoring are conducted on the target user terminals, and warnings are issued for target user terminals with zero-day vulnerability attacks.
[0034] Optionally, the step of identifying zero-day vulnerabilities in each of the second log data according to the priority determined by the log data volume includes:
[0035] The priority of each data tag is determined based on the amount of log data; wherein, there is a positive correlation between the amount of log data and the priority of the corresponding data tag;
[0036] The data tags are sorted in descending order of priority so that the second log data corresponding to each data tag can be identified for zero-day vulnerabilities in turn according to the sorting results.
[0037] A second aspect of this application provides a zero-day vulnerability identification device, comprising:
[0038] The data filtering module is used to store log data continuously collected from the target server in the cloud, and filter out the first log data suspected of being a zero-day vulnerability from the log data based on a pre-set hunting strategy.
[0039] The data noise reduction module is used to perform data noise reduction processing on the first log data to remove data unrelated to zero-day vulnerabilities from the first log data, obtain second log data, and analyze the second log data to label the corresponding data tags.
[0040] The identification module is used to continuously count the amount of log data corresponding to the data tag, and identify zero-day vulnerabilities in each of the second log data according to the priority determined by the amount of log data.
[0041] A third aspect of this application provides an electronic device including a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the aforementioned zero-day vulnerability identification method.
[0042] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the aforementioned zero-day vulnerability identification method.
[0043] In this application, log data continuously collected from the target server is first stored in the cloud. Based on a pre-set hunting strategy, first log data suspected of being zero-day vulnerabilities is selected from the log data. The first log data undergoes noise reduction processing to remove data unrelated to zero-day vulnerabilities, resulting in second log data. This second log data is then analyzed to label it with corresponding data tags. The amount of log data corresponding to each data tag is continuously counted, and zero-day vulnerabilities are identified sequentially for each piece of second log data based on the priority determined by the amount of log data. Therefore, this application first stores log data continuously collected from the target server in real time in the cloud. Then, based on a pre-set hunting strategy, it initially selects first log data suspected of being zero-day vulnerabilities from the massive amount of log data in the cloud, thereby filtering out other irrelevant log data, narrowing the data analysis scope, and improving vulnerability hunting efficiency. Furthermore, the first log data undergoes noise reduction processing to remove data unrelated to zero-day vulnerabilities, resulting in the second log data. That is, the first log data filtered by the hunting strategy still represents a relatively wide range. This application, through noise reduction, can filter out data unrelated to zero-day vulnerabilities from the first log data to obtain the second log data, further improving vulnerability hunting efficiency. Next, the second log data is analyzed to label it with corresponding data tags, and the amount of log data corresponding to the data tags is continuously counted. The purpose is to identify zero-day vulnerabilities in each second log data set according to the priority determined by the amount of log data. That is, after noise reduction, the range of the second log data obtained is already relatively small, so this part of the data needs to be used for final zero-day vulnerability identification. However, considering that the ability to analyze and identify data within the same time frame is also limited, this application sets a priority system. By first labeling the data, and then determining the priority between each second log data set based on the amount of log data corresponding to different data tags, identification is performed sequentially according to priority. This improves the efficiency of zero-day vulnerability identification within a limited time, enabling faster identification of zero-day vulnerabilities. In this way, the above method can identify zero-day vulnerabilities more accurately and quickly. Attached Figure Description
[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0045] Figure 1 This is a flowchart of a zero-day vulnerability identification method disclosed in this application;
[0046] Figure 2 This application discloses a flowchart of a specific zero-day vulnerability identification method.
[0047] Figure 3 This is a schematic diagram of the structure of a zero-day vulnerability identification device disclosed in this application;
[0048] Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0050] In existing technologies, there are two main types of techniques for discovering zero-day vulnerabilities. The first is manual vulnerability discovery. However, manual discovery has a high technical threshold, with only a few security experts possessing the ability to discover vulnerabilities. Furthermore, manual vulnerability discovery is inefficient, often taking a considerable amount of time to discover vulnerabilities in a single software application. Additionally, because vulnerability discovery requires software resources, its scope is limited; for some commercial software, the lack of available software resources prevents manual vulnerability discovery. The second method is fuzzing testing. However, fuzzing testing requires significant computing and memory resources, especially when dealing with large and complex software systems. Furthermore, fuzzing testing is prone to false positives. Therefore, this application discloses a method, apparatus, device, and storage medium for identifying zero-day vulnerabilities, capable of accurately and quickly identifying them.
[0051] Figure 1 A flowchart illustrating a zero-day vulnerability identification method provided in this application embodiment. See also... Figure 1 As shown, this zero-day vulnerability identification method includes:
[0052] Step S11: Store the log data continuously collected from the target server in the cloud, and filter out the first log data suspected of being a zero-day vulnerability from the log data based on the pre-set hunting strategy.
[0053] In this embodiment, log data continuously collected from the target server is stored in the cloud in real time. Then, based on a pre-set hunting strategy, the system initially filters out the first log data suspected of being zero-day vulnerabilities from the massive amount of log data in the cloud, thereby filtering out other irrelevant log data, narrowing the scope of data analysis, and improving vulnerability hunting efficiency. Furthermore, considering that some users are not allowed to transmit data to the cloud due to information security requirements, the solution of this invention also supports local deployment, that is, storing the log data locally on the target server and then identifying zero-day vulnerabilities.
[0054] It should be noted that the log data mentioned above is telemetry data. That is, this embodiment mainly uses telemetry technology to collect security log data from various systems, applications and services. This security log data usually contains information on network traffic, abnormal behavior, system warnings and other potential threats.
[0055] In a specific implementation, the aforementioned cloud storage of log data continuously collected from the target server includes: continuously collecting log data from the target server through a data acquisition engine, and uploading the log data to a cloud-based data lake for storage after anonymizing the log data; wherein the log data includes network request data, process behavior data, service status data, and memory data. That is, in this embodiment, a data acquisition engine is pre-deployed to the target server, and the data acquisition engine continuously collects log data such as network requests and responses, process behavior, service status, and memory from the target server. Furthermore, to further ensure the privacy of user data, the collected log data needs to be anonymized before being uploaded to the cloud-based data lake for storage.
[0056] In a specific implementation, the step of filtering suspected zero-day vulnerability first log data from the log data based on a pre-set hunting strategy includes: filtering suspected zero-day vulnerability first log data from the log data using a first hunting strategy set based on preset high-risk vulnerability interface characteristics; or, filtering suspected zero-day vulnerability first log data from the log data using a second hunting strategy set based on general vulnerability attack characteristics; or, filtering suspected zero-day vulnerability first log data from the log data using a third hunting strategy set based on network request and response characteristics generated after a successful vulnerability attack; or, filtering suspected zero-day vulnerability first log data from the log data using a fourth hunting strategy set based on vulnerability patch repair characteristics; or, filtering suspected zero-day vulnerability first log data from the log data using a fifth hunting strategy set based on external attacker characteristics.
[0057] In other words, this application embodiment has formulated a hunting strategy from multiple dimensions of vulnerability information, including preset high-risk vulnerability interface characteristics, general vulnerability attack characteristics, network request and response characteristics generated after a successful vulnerability attack, vulnerability patch repair characteristics, external attacker characteristics, etc., thereby using these hunting strategies to initially screen out the first log data of suspected zero-day vulnerabilities from massive telemetry data.
[0058] In the first hunting strategy based on the preset high-risk vulnerability interface characteristics, it is necessary to extract the unauthorized, frequently vulnerable, and newly added high-risk URLs (Uniform Resource Locators) exposed by the component, store the URL characteristics in the "component high-risk URL database", and then, based on the extracted URL characteristics, initially screen the log data that matches the characteristics from the cloud data lake to obtain the first log data of suspected zero-day vulnerabilities.
[0059] In the second hunting strategy based on common vulnerability attack characteristics, it is necessary to extract attack characteristics of common vulnerability types, such as the malicious SQL statement characteristics of SQL (Structured Query Language) injection vulnerabilities, the webshell (command execution environment) content characteristics of file upload vulnerabilities, and the function characteristics of code injection vulnerabilities that execute malicious commands. The attack characteristics of common vulnerability types are stored in the "vulnerability attack characteristic database". Then, based on the extracted common vulnerability attack characteristics, log data that matches the characteristics is initially screened from the cloud data lake to obtain the first log data of suspected zero-day vulnerabilities.
[0060] In the third hunting strategy based on the network request and response characteristics generated after a successful vulnerability attack, it is necessary to extract the network request and response characteristics generated after a successful network request attack using a common vulnerability. For example, if a code injection vulnerability is used to execute the `whoami` command (used to view currently valid usernames) on the target server, the response data of a successful vulnerability attack will contain the hostname. The network request and response characteristics generated by a successful vulnerability attack are stored in a "successful vulnerability attack response database." Then, based on the extracted vulnerability attack characteristics, log data matching the characteristics is initially screened from the cloud data lake to obtain the first log data of suspected zero-day vulnerabilities.
[0061] In the fourth hunting strategy based on vulnerability patch repair characteristics, it is necessary to analyze the differences in patch repair characteristics, locate the vulnerability location in the component, extract vulnerability attack characteristics and store them in the "vulnerability attack characteristic database", extract the URL characteristics of the component with vulnerability and store them in the "component high-risk URL database", and then, based on the extracted URL characteristics and vulnerability attack characteristics, initially screen the log data that matches the characteristics from the cloud data lake to obtain the first log data of suspected zero-day vulnerabilities.
[0062] In the fifth hunting strategy based on the characteristics of external attackers, it is necessary to monitor external threat intelligence, extract threat intelligence characteristics such as the IP address (Internet Protocol Address) and domain name of external attackers, store them in the "threat intelligence database", and then, based on the extracted IP address and domain name, initially screen log data that matches the characteristics from the cloud data lake to obtain the first log data of suspected zero-day vulnerabilities.
[0063] Step S12: Perform data noise reduction processing on the first log data to remove data unrelated to zero-day vulnerabilities from the first log data, obtain second log data, and analyze the second log data to label the corresponding data tags.
[0064] In this embodiment, the first log data undergoes noise reduction processing to remove data unrelated to zero-day vulnerabilities, resulting in second log data. The second log data is then analyzed to assign corresponding data tags. That is, the first log data filtered by the hunting strategy still represents a relatively wide range. This application, through noise reduction processing, can filter out data unrelated to zero-day vulnerabilities from the first log data to obtain the second log data, further improving vulnerability hunting efficiency.
[0065] Step S13: Continuously count the amount of log data corresponding to the data tag, and identify zero-day vulnerabilities in each of the second log data according to the priority determined by the amount of log data.
[0066] In this embodiment, after analyzing the second log data to label it with corresponding data tags, the amount of log data corresponding to the data tags is continuously counted. The purpose is to identify zero-day vulnerabilities in each second log data according to the priority determined by the amount of log data. That is, after data noise reduction, the range of the second log data obtained at this time is already small, so it is necessary to perform final zero-day vulnerability identification on this part of the data. However, considering that the ability to analyze and identify data within the same time is also limited, this application sets a priority. By first labeling the data, and then determining the priority between each second log data according to the size of the log data corresponding to different data tags, identification is performed sequentially according to priority. This can improve the efficiency of zero-day vulnerability identification within an effective time, that is, it can identify zero-day vulnerabilities more quickly.
[0067] Furthermore, after identifying zero-day vulnerabilities in each of the second log data according to the priority determined by the log data volume, the process further includes: conducting vulnerability screening and monitoring on the target user terminals based on the identified zero-day vulnerabilities, and issuing warnings to target user terminals with zero-day vulnerability attacks. It can be understood that after identifying a zero-day vulnerability, based on the screening and monitoring of zero-day vulnerabilities on the target user terminals and other user terminals across the entire network, warnings are issued to users with zero-day vulnerabilities or who have already suffered zero-day vulnerability attacks.
[0068] In this application, log data continuously collected from the target server is first stored in the cloud. Based on a pre-set hunting strategy, first log data suspected of being zero-day vulnerabilities is selected from the log data. The first log data undergoes noise reduction processing to remove data unrelated to zero-day vulnerabilities, resulting in second log data. This second log data is then analyzed to label it with corresponding data tags. The amount of log data corresponding to each data tag is continuously counted, and zero-day vulnerabilities are identified sequentially for each piece of second log data based on the priority determined by the amount of log data. Therefore, this application first stores log data continuously collected from the target server in real time in the cloud. Then, based on a pre-set hunting strategy, it initially selects first log data suspected of being zero-day vulnerabilities from the massive amount of log data in the cloud, thereby filtering out other irrelevant log data, narrowing the data analysis scope, and improving vulnerability hunting efficiency. Furthermore, the first log data undergoes noise reduction processing to remove data unrelated to zero-day vulnerabilities, resulting in the second log data. That is, the first log data filtered by the hunting strategy still represents a relatively wide range. This application, through noise reduction, can filter out data unrelated to zero-day vulnerabilities from the first log data to obtain the second log data, further improving vulnerability hunting efficiency. Next, the second log data is analyzed to label it with corresponding data tags, and the amount of log data corresponding to the data tags is continuously counted. The purpose is to identify zero-day vulnerabilities in each second log data set according to the priority determined by the amount of log data. That is, after noise reduction, the range of the second log data obtained is already relatively small, so this part of the data needs to be used for final zero-day vulnerability identification. However, considering that the ability to analyze and identify data within the same time frame is also limited, this application sets a priority system. By first labeling the data, and then determining the priority between each second log data set based on the amount of log data corresponding to different data tags, identification is performed sequentially according to priority. This improves the efficiency of zero-day vulnerability identification within a limited time, enabling faster identification of zero-day vulnerabilities. In this way, the above method can identify zero-day vulnerabilities more accurately and quickly.
[0069] Figure 2This is a flowchart illustrating a specific zero-day vulnerability identification method provided in an embodiment of this application. See also... Figure 2 As shown, this zero-day vulnerability identification method includes:
[0070] Step S21: Store the log data continuously collected from the target server in the cloud, and filter out the first log data suspected of being a zero-day vulnerability from the log data based on the pre-set hunting strategy.
[0071] Step S22: Input the first log data into the pre-trained target neural network model to output the target feature vector.
[0072] In this embodiment, feature vectors are generated from log data using deep learning technology. Specifically, the first log data is input into a pre-trained target neural network model to output the corresponding target feature vector.
[0073] The training process of the target neural network model includes: standardizing data from the historical known vulnerability database and the application fingerprint database to generate target data that meets preset format requirements, and labeling each piece of target data with preset key information; wherein, the preset key information includes Uniform Resource Locator (URL), payload information, vulnerability type, and application component information; and training an initial neural network model using the target data to obtain the target neural network model. Specifically, this embodiment standardizes POCs (Proof of Concept) from the historical known vulnerability database and URLs from the application fingerprint database to form target data in standard HTTP format, and labels it with preset key information related to vulnerability identification, such as Uniform Resource Locator (URL), payload information, and the vulnerability type and application component information corresponding to each piece of standard HTTP (Hypertext Transfer Protocol) format data. Then, the initial neural network model is trained using this target data to obtain the target neural network model.
[0074] Then, by standardizing the POCs in the historical known vulnerability database and the URLs in the application fingerprint database, and then inputting them into the target neural network model, and by standardizing the newly disclosed known vulnerability POCs and URLs and then inputting them into the target neural network model, the historical known vulnerability feature vector database and the application fingerprint feature vector database are obtained.
[0075] Step S23: Match each of the target feature vectors with the feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library respectively, so as to remove data unrelated to zero-day vulnerabilities from the first log data based on the matching results, and obtain the second log data; wherein, the application fingerprint feature vector library is used to store application fingerprint feature information related to the application environment of zero-day vulnerabilities.
[0076] In this embodiment, each target feature vector is matched with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, respectively. Based on the matching results, data unrelated to the zero-day vulnerability is removed from the first log data, resulting in the second log data. It should be noted that the application fingerprint feature vector library primarily stores application fingerprint feature information related to the application environment of the zero-day vulnerability.
[0077] In a specific implementation, each target feature vector is matched with feature vectors in a historical known vulnerability feature vector library and an application fingerprint feature vector library, respectively, to remove data unrelated to zero-day vulnerabilities from the first log data based on the matching results. This includes: calculating the similarity between each target feature vector and each historical known vulnerability feature vector in the historical known vulnerability feature vector library to obtain a corresponding first similarity value; if any first similarity value is greater than a first preset threshold, the corresponding first log data is determined to be data related to historical known vulnerabilities, and log data corresponding to any first similarity value is removed from the first log data; calculating the similarity between each target feature vector and each application fingerprint feature vector in the application fingerprint feature vector library to obtain a corresponding second similarity value; if any second similarity value is less than a second preset threshold, the corresponding second log data is determined to be data unrelated to the zero-day vulnerability application environment, and log data corresponding to any second similarity value is removed from the first log data.
[0078] In other words, the historical known vulnerability feature vector database stores feature vector information of known vulnerabilities. When matching a target feature vector with the historical known vulnerability feature vector database, the main process is to calculate the similarity between each target feature vector and each historical known vulnerability feature vector in the database, obtaining a first similarity value. If any first similarity value is greater than a first preset threshold, it indicates similarity to a known vulnerability. Therefore, the corresponding first log data is determined to be data related to a historical known vulnerability, rather than zero-day vulnerability data. Consequently, log data corresponding to any first similarity value needs to be removed from the first log data. This process effectively removes data related to historical known vulnerabilities from the first log data.
[0079] The application fingerprint feature vector library primarily stores application fingerprint feature information related to the application environment of zero-day vulnerabilities. When matching a target feature vector with the application fingerprint feature vector library, the similarity between each target feature vector and each application fingerprint feature vector in the library is calculated to obtain a second similarity value. If any second similarity value is less than a second preset threshold, the corresponding second log data is determined to be data unrelated to the zero-day vulnerability application environment; for example, it may belong to custom test environment traffic. Therefore, log data corresponding to any second similarity value needs to be removed from the first log data. This allows data related to custom test environments to be removed from the first log data, such as filtering out test traffic from non-component vulnerability testing scenarios like CTF (Capture The Flag) challenges or self-built vulnerability ranges that are unrelated to the component application.
[0080] When calculating similarity, similarity algorithms such as Hamming Distance and Cosine Similarity can be used. This application does not restrict the specific similarity algorithm.
[0081] It should also be noted that before matching each target feature vector with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, the process further includes: calculating the similarity between every two target feature vectors to obtain a corresponding third similarity value; if any third similarity value is greater than a third preset threshold, then the corresponding two target feature vectors are aggregated to obtain an aggregated target feature vector; correspondingly, matching each target feature vector with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library includes: matching each aggregated target feature vector with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library. That is, before matching, the similarity between every two target feature vectors can be calculated to obtain a corresponding third similarity value. If any third similarity value is greater than a third preset threshold, then the two target feature vectors are considered similar or identical. Therefore, the corresponding two target feature vectors can be aggregated to obtain an aggregated target feature vector, thereby reducing the amount of data. Subsequently, the aggregated target feature vectors can be matched with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, thereby saving computational resources.
[0082] Step S24: Analyze the second log data to label the corresponding data tags, continuously count the amount of log data corresponding to the data tags, and determine the priority of each data tag based on the amount of log data; wherein, there is a positive correlation between the amount of log data and the priority of the corresponding data tag.
[0083] In this embodiment, the second log data can be analyzed from multiple dimensions to label corresponding data tags. Data tags can include application name, vulnerability type, attack success status, behavior type, data source, hunting strategy, etc. Then, the amount of log data corresponding to the data tags is continuously counted, thereby determining the priority of each data tag based on the amount of log data. There is a positive correlation between the amount of log data and the priority of the corresponding data tag; that is, the more log data a data tag corresponds to, the higher its priority.
[0084] Step S25: Sort the data tags in descending order of priority so that the second log data corresponding to each data tag can be identified for zero-day vulnerabilities in turn according to the sorting results.
[0085] In this embodiment, the data tags are sorted in descending order of priority, with higher priority tags appearing earlier and lower priority tags appearing later. Then, based on the sorting results, the second log data corresponding to each data tag is sequentially identified for zero-day vulnerabilities. For example, if 80% of the captured log data containing suspected zero-day vulnerabilities contains the tag "Attack Successful Hunting Strategy," 60% contains both the "Attack Successful Hunting Strategy" and "File Upload Attack Type" tags, and 20% contains "SQL Injection Attack Type," then log data containing the "Attack Successful Hunting Strategy" tag will be prioritized for identification.
[0086] When identifying zero-day vulnerabilities, the nature and evidence of the vulnerability can be determined through code auditing, vulnerability reproduction, and other methods. For example, the latest version of the application software environment can be set up in a laboratory environment, and a network attack request for the zero-day vulnerability can be sent. Based on the network request response of the application software, it can be determined whether the zero-day vulnerability attack has been successfully reproduced. If the reproduction is successful, the nature of the zero-day vulnerability is determined.
[0087] In this embodiment, the specific process of step S21 can be referred to the corresponding content disclosed in the previous embodiments, and will not be repeated here.
[0088] As can be seen, this application, after filtering out suspected zero-day vulnerability first log data from massive log data based on a pre-set hunting strategy, generates target feature vectors from the first log data using deep learning technology. It also converts the internal historical known vulnerability database and application fingerprint database into corresponding historical known vulnerability feature vector libraries and application fingerprint feature vector libraries. The target feature vectors are then matched with feature vectors in these libraries for similarity, and data unrelated to zero-day vulnerabilities is removed from the first log data based on the similarity matching results. Furthermore, the similarity value between every two target feature vectors can be calculated, aggregating identical or similar features to reduce data volume. Finally, during zero-day vulnerability identification, priority is determined based on the amount of log data for each data tag, and then the data tags are sorted in descending order of priority. Zero-day vulnerability identification is then performed based on the sorting results, improving the efficiency of zero-day vulnerability identification within a limited time.
[0089] See Figure 3 As shown in the figure, this application also discloses a zero-day vulnerability identification device, including:
[0090] The data filtering module 11 is used to store log data continuously collected from the target server in the cloud, and filter out the first log data suspected of being a zero-day vulnerability from the log data based on a pre-set hunting strategy.
[0091] The data noise reduction module 12 is used to perform data noise reduction processing on the first log data to remove data unrelated to zero-day vulnerabilities from the first log data, obtain second log data, and analyze the second log data to label the corresponding data tags.
[0092] The identification module 13 is used to continuously count the amount of log data corresponding to the data tag, and identify zero-day vulnerabilities in each of the second log data according to the priority determined by the amount of log data.
[0093] This application discloses a data filtering module for storing continuously collected log data from a target server in the cloud and filtering out suspected zero-day vulnerability log data from the log data based on a pre-set hunting strategy; a data denoising module for performing data denoising processing on the first log data to remove data unrelated to zero-day vulnerabilities, obtaining second log data, and analyzing the second log data to label it with corresponding data tags; and an identification module for continuously counting the amount of log data corresponding to the data tags and identifying zero-day vulnerabilities in each second log data according to the priority determined by the amount of log data. As can be seen, this application first stores continuously collected log data from the target server in the cloud in real time, and then pre-filters out suspected zero-day vulnerability log data from the massive amount of log data in the cloud based on a pre-set hunting strategy, thereby filtering out other irrelevant log data, narrowing the scope of data analysis, and improving vulnerability hunting efficiency. Furthermore, the first log data undergoes noise reduction processing to remove data unrelated to zero-day vulnerabilities, resulting in the second log data. That is, the first log data filtered by the hunting strategy still represents a relatively wide range. This application, through noise reduction, can filter out data unrelated to zero-day vulnerabilities from the first log data to obtain the second log data, further improving vulnerability hunting efficiency. Next, the second log data is analyzed to label it with corresponding data tags, and the amount of log data corresponding to the data tags is continuously counted. The purpose is to identify zero-day vulnerabilities in each second log data set according to the priority determined by the amount of log data. That is, after noise reduction, the range of the second log data obtained is already relatively small, so this part of the data needs to be used for final zero-day vulnerability identification. However, considering that the ability to analyze and identify data within the same time frame is also limited, this application sets a priority system. By first labeling the data, and then determining the priority between each second log data set based on the amount of log data corresponding to different data tags, identification is performed sequentially according to priority. This improves the efficiency of zero-day vulnerability identification within a limited time, enabling faster identification of zero-day vulnerabilities. In this way, the above method can identify zero-day vulnerabilities more accurately and quickly.
[0094] In some specific embodiments, the data filtering module 11 may specifically include:
[0095] The data storage unit is used to continuously collect log data from the target server through the data acquisition engine, and after anonymizing the log data, upload it to the data lake in the cloud for storage; wherein, the log data includes network request data, process behavior data, service status data, and memory data.
[0096] In some specific embodiments, the data filtering module 11 is specifically used to filter out first log data suspected of being zero-day vulnerabilities from the log data using a first hunting strategy set based on preset high-risk vulnerability interface characteristics.
[0097] In some specific embodiments, the data filtering module 11 is specifically used to filter out first log data suspected of being zero-day vulnerabilities from the log data using a second hunting strategy set based on common vulnerability attack characteristics.
[0098] In some specific embodiments, the data filtering module 11 is specifically used to filter out first log data suspected of being zero-day vulnerabilities from the log data using a third hunting strategy set based on the network request response characteristics generated after a successful vulnerability attack.
[0099] In some specific embodiments, the data filtering module 11 is specifically used to filter out first log data suspected of being zero-day vulnerabilities from the log data using a fourth hunting strategy based on vulnerability patch repair characteristics.
[0100] In some specific embodiments, the data filtering module 11 is specifically used to filter out first log data suspected of being zero-day vulnerabilities from the log data using a fifth hunting strategy set based on the characteristics of external attackers.
[0101] In some specific embodiments, the data noise reduction module 12 may specifically include:
[0102] The feature vector acquisition unit is used to input the first log data into a pre-trained target neural network model to output a target feature vector;
[0103] The data cleaning unit is used to match each of the target feature vectors with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, respectively, so as to remove data unrelated to zero-day vulnerabilities from the first log data based on the matching results and obtain second log data; wherein, the application fingerprint feature vector library is used to store application fingerprint feature information related to the application environment of zero-day vulnerabilities.
[0104] In some specific embodiments, the data erasure unit may specifically include:
[0105] The first clearing unit is used to calculate the similarity between each target feature vector and each historical known vulnerability feature vector in the historical known vulnerability feature vector library to obtain the corresponding first similarity value. If any first similarity value is greater than a first preset threshold, the corresponding first log data is determined to be data related to historical known vulnerabilities, and the log data corresponding to any first similarity value is cleared from the first log data.
[0106] The second clearing unit is used to calculate the similarity between each target feature vector and each application fingerprint feature vector in the application fingerprint feature vector library to obtain the corresponding second similarity value. If any second similarity value is less than a second preset threshold, the corresponding second log data is determined to be data unrelated to the zero-day vulnerability application environment, and the log data corresponding to any second similarity value is cleared from the first log data.
[0107] In some specific embodiments, before matching each of the target feature vectors with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, the apparatus further includes:
[0108] A similarity calculation unit is used to calculate the similarity between every two target feature vectors to obtain the corresponding third similarity value;
[0109] The aggregation unit is used to aggregate the two corresponding target feature vectors if any third similarity value is greater than the third preset threshold, so as to obtain the aggregated target feature vector.
[0110] Accordingly, the data clearing unit is specifically used to match the aggregated target feature vectors with the feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, respectively.
[0111] In some specific embodiments, the training process of the target neural network model includes: standardizing the data in the historical known vulnerability database and the application fingerprint database to generate target data that meets the preset format requirements, and labeling each target data with preset key information; wherein, the preset key information includes Uniform Resource Locator, payload information, vulnerability type and application component information; and training an initial neural network model using the target data to obtain the target neural network model.
[0112] In some specific embodiments, after identifying zero-day vulnerabilities in each of the second log data according to the priority determined by the log data volume, the device further includes:
[0113] The early warning unit is used to perform vulnerability screening and monitoring on target user terminals based on the identified zero-day vulnerabilities, and to issue early warnings for target user terminals that have zero-day vulnerability attacks.
[0114] In some specific embodiments, the identification module 13 may specifically include:
[0115] A priority determination unit is used to determine the priority of each data tag based on the amount of log data; wherein, there is a positive correlation between the amount of log data and the priority of the corresponding data tag;
[0116] The sorting unit is used to sort the data tags in descending order of priority, so as to identify zero-day vulnerabilities in the second log data corresponding to each data tag in sequence according to the sorting results.
[0117] Furthermore, embodiments of this application also provide an electronic device. Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0118] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the zero-day vulnerability identification method disclosed in any of the foregoing embodiments.
[0119] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0120] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.
[0121] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the zero-day vulnerability identification method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks. The data 223 may include image quality control strategies collected by the electronic device 20, etc.
[0122] Furthermore, this application also discloses a storage medium storing a computer program, which, when loaded and executed by a processor, implements the zero-day vulnerability identification method steps disclosed in any of the foregoing embodiments.
[0123] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0124] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0125] The zero-day vulnerability identification method, apparatus, device, and storage medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for identifying zero-day vulnerabilities, characterized in that, include: Log data continuously collected from the target server is stored in the cloud, and the first log data suspected of being a zero-day vulnerability is filtered out from the log data based on a pre-set hunting strategy; The first log data is subjected to noise reduction processing to remove data unrelated to zero-day vulnerabilities from the first log data, resulting in second log data. The second log data is then analyzed to label the corresponding data tags. The amount of log data corresponding to the data tags is continuously counted, and zero-day vulnerabilities are identified in each of the second log data according to the priority determined by the amount of log data.
2. The zero-day vulnerability identification method according to claim 1, characterized in that, The storage of log data continuously collected from the target server in the cloud includes: The data acquisition engine continuously collects log data from the target server, and after anonymizing the log data, it is uploaded to a data lake in the cloud for storage. The log data includes network request data, process behavior data, service status data, and memory data.
3. The zero-day vulnerability identification method according to claim 1, characterized in that, The first log data suspected of being a zero-day vulnerability is selected from the log data based on a pre-set hunting strategy, including: Using a first hunting strategy based on preset high-risk vulnerability interface characteristics, first log data suspected of being zero-day vulnerabilities is filtered from the log data; Alternatively, a second hunting strategy based on common vulnerability attack characteristics can be used to filter out first log data suspected of being zero-day vulnerabilities from the log data; Alternatively, a third hunting strategy based on the network request and response characteristics generated after a successful vulnerability attack can be used to filter out the first log data suspected of being a zero-day vulnerability from the log data; Alternatively, a fourth hunting strategy based on vulnerability patch repair characteristics can be used to filter out the first log data suspected of being a zero-day vulnerability from the log data; Alternatively, a fifth hunting strategy based on external attacker characteristics can be used to filter out the first log data suspected of being a zero-day vulnerability from the log data.
4. The zero-day vulnerability identification method according to claim 1, characterized in that, The step of performing data noise reduction processing on the first log data to remove data unrelated to zero-day vulnerabilities and obtain second log data includes: The first log data is input into a pre-trained target neural network model to output a target feature vector; Each of the target feature vectors is matched with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, respectively, so as to remove data unrelated to zero-day vulnerabilities from the first log data based on the matching results, and obtain the second log data; wherein, the application fingerprint feature vector library is used to store application fingerprint feature information related to the application environment of zero-day vulnerabilities.
5. The zero-day vulnerability identification method according to claim 4, characterized in that, The step of matching each of the target feature vectors with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, respectively, to remove data unrelated to zero-day vulnerabilities from the first log data based on the matching results, includes: Each target feature vector is compared with each historical known vulnerability feature vector in the historical known vulnerability feature vector library to obtain a corresponding first similarity value. If any first similarity value is greater than a first preset threshold, the corresponding first log data is determined to be data related to historical known vulnerabilities, and the log data corresponding to any first similarity value is cleared from the first log data. The similarity between each target feature vector and each application fingerprint feature vector in the application fingerprint feature vector library is calculated to obtain the corresponding second similarity value. If any second similarity value is less than a second preset threshold, the corresponding second log data is determined to be data unrelated to the zero-day vulnerability application environment, and the log data corresponding to any second similarity value is cleared from the first log data.
6. The zero-day vulnerability identification method according to claim 4, characterized in that, Before matching each of the target feature vectors with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, the process further includes: Calculate the similarity between every two target feature vectors to obtain the corresponding third similarity value; If any third similarity value is greater than the third preset threshold, the corresponding two target feature vectors are aggregated to obtain the aggregated target feature vector. Accordingly, the matching of each target feature vector with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library includes: The aggregated target feature vectors are matched with feature vectors in the historical known vulnerability feature vector library and the application fingerprint feature vector library, respectively.
7. The zero-day vulnerability identification method according to claim 4, characterized in that, The training process of the target neural network model includes: The data in the historical known vulnerability database and the application fingerprint database are standardized to generate target data that meets the preset format requirements, and each target data is labeled with preset key information; wherein, the preset key information includes Uniform Resource Locator, payload information, vulnerability type and application component information; The initial neural network model is trained using the target data to obtain the target neural network model.
8. The zero-day vulnerability identification method according to claim 1, characterized in that, After identifying zero-day vulnerabilities in each of the second log data according to the priority determined by the log data volume, the method further includes: Based on the identified zero-day vulnerabilities, vulnerability screening and monitoring are conducted on the target user terminals, and warnings are issued for target user terminals with zero-day vulnerability attacks.
9. The zero-day vulnerability identification method according to any one of claims 1 to 8, characterized in that, The step of identifying zero-day vulnerabilities in each of the second log data according to the priority determined by the log data volume includes: The priority of each data tag is determined based on the amount of log data; wherein, there is a positive correlation between the amount of log data and the priority of the corresponding data tag; The data tags are sorted in descending order of priority so that the second log data corresponding to each data tag can be identified for zero-day vulnerabilities in turn according to the sorting results.
10. A zero-day vulnerability identification device, characterized in that, include: The data filtering module is used to store log data continuously collected from the target server in the cloud, and filter out the first log data suspected of being a zero-day vulnerability from the log data based on a pre-set hunting strategy. The data noise reduction module is used to perform data noise reduction processing on the first log data to remove data unrelated to zero-day vulnerabilities from the first log data, obtain second log data, and analyze the second log data to label the corresponding data tags. The identification module is used to continuously count the amount of log data corresponding to the data tag, and identify zero-day vulnerabilities in each of the second log data according to the priority determined by the amount of log data.
11. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein: The memory is used to store computer programs; The computer program is loaded and executed by the processor to implement the zero-day vulnerability identification method as described in any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that, Used to store computer-executable instructions, which, when loaded and executed by a processor, implement the zero-day vulnerability identification method as described in any one of claims 1 to 9.