Threat discovery method and device for data, electronic equipment and storage medium

Abstract

The application provides a threat discovery method and device, electronic equipment and a storage medium. The method comprises: performing entity identification on to-be-processed data by using an entity identification model to obtain first entity information contained in the to-be-processed data; the to-be-processed data is asset information of each asset in a to-be-detected system; performing matching based on the first entity information and second entity information in a vulnerability intelligence library to obtain a vulnerability analysis result; wherein the vulnerability intelligence library comprises a plurality of pieces of vulnerability intelligence, and the vulnerability intelligence library is generated by acquiring original intelligence data from a plurality of intelligence sources and performing data standardization on the original intelligence data. According to the embodiment of the application, the entity identification model is used to perform entity identification on to-be-processed data, the accuracy of entity extraction in the to-be-processed data is improved, and the accuracy of vulnerability detection is improved in combination with vulnerability intelligence acquired from a plurality of intelligence sources.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, in particular to a data threat discovery method and device, an electronic device and a storage medium. BACKGROUND

[0002] At present, in order to solve the problems existing in threat intelligence information identification, vulnerability intelligence information and event intelligence information are fused by integrating multiple data sources, so as to generate vulnerability intelligence fusion information and event intelligence fusion information. In this process, for the vulnerability intelligence fusion information, the system extracts key attribute information and combines asset information to determine whether the vulnerability intelligence information related to the vulnerability intelligence fusion information belongs to threat vulnerability information.

[0003] Among them, when extracting key attribute information, a keyword matching method is often used for extraction. The keywords extracted by this method may be limited by data quality and accuracy. If the attribute information extraction is not accurate, it will affect the judgment and identification accuracy of the vulnerability intelligence information. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a data threat discovery method, device, electronic device and storage medium, to improve the judgment and identification accuracy of vulnerability intelligence information.

[0005] In a first aspect, the embodiments of the present application provide a data threat discovery method, comprising:

[0006] performing entity recognition on the to-be-processed data by using an entity recognition model to obtain first entity information contained in the to-be-processed data; the to-be-processed data is asset information of each asset in a to-be-detected system;

[0007] performing matching based on the first entity information and second entity information in a vulnerability intelligence library to obtain a vulnerability analysis result; wherein the vulnerability intelligence library includes multiple vulnerability intelligence, and the vulnerability intelligence library is generated by acquiring original intelligence data from multiple intelligence sources and performing data standardization on the original intelligence data.

[0008] The embodiments of the present application improve the accuracy of entity extraction in the to-be-processed data by using an entity recognition model to perform entity recognition on the to-be-processed data, and improve the accuracy of vulnerability detection by combining the vulnerability intelligence acquired from multiple intelligence sources.

[0009] In any embodiment, the method further comprises:

[0010] receiving original intelligence data from multiple intelligence sources;

[0011] The fields in the original intelligence data are mapped to standard fields by using a pre-constructed field mapping rule.

[0012] The vulnerability intelligence library is generated according to the standard fields.

[0013] The embodiments of the present application can unify data from different data sources into the same data format and standard by using the field mapping rule, thereby reducing the complexity of data processing, improving the comparability and consistency of data, and thus more effectively performing data integration and analysis.

[0014] In any embodiment, the method further comprises:

[0015] Obtaining training samples, and performing word segmentation processing on the training samples by using a GloVe model to obtain a plurality of word vectors;

[0016] The word vectors are labeled by using a BIO format;

[0017] The labeled word vectors are input into a to-be-trained model to obtain predicted entity information output by the to-be-trained model;

[0018] Parameters of the to-be-trained model are optimized based on the predicted entity information and the labeling result to obtain a trained entity recognition model.

[0019] The embodiments of the present application train an entity recognition model, which can improve the accuracy of recognizing and extracting key entity information when extracting attribute information of to-be-processed data, and is helpful to improve the accuracy of attribute information extraction.

[0020] In any embodiment, the vulnerability analysis result is obtained by matching the first entity information with second entity information in the vulnerability intelligence library, and includes:

[0021] The similarity between the second entity information and the first entity information corresponding to each asset in the to-be-detected system is calculated by using a Levenshtein distance algorithm;

[0022] If the similarity is greater than a preset threshold, it is determined that the asset has a vulnerability threat.

[0023] The embodiments of the present application compare the similarity between asset information by using the Levenshtein distance algorithm, thereby improving the accuracy of matching.

[0024] In any embodiment, the method further comprises:

[0025] Obtaining vulnerability event information of the to-be-detected system in a preset historical time period;

[0026] input the vulnerability event information into an event prediction model, and obtain a prediction result output by the event prediction model; the prediction result is used to represent whether a vulnerability occurs in the to-be-detected system in a future preset time period; the event prediction model is obtained by training a long short-term memory network using training vulnerability event information.

[0027] The vulnerability situation of the to-be-detected system in a historical time period is captured by using the event prediction model, so as to predict the probability of occurrence of a vulnerability in the to-be-detected system in the future, and to provide more effective event prediction and decision support for an enterprise.

[0028] In any embodiment, the method is applied to a stream processing platform; the stream processing platform includes a stream processing engine; the matching based on the first entity information and second entity information in a vulnerability intelligence library includes:

[0029] The first entity information is matched with the second entity information in the vulnerability intelligence library by the stream processing engine.

[0030] The stream processing architecture can quickly process data and generate results in real time. Through the stream processing technology, a high-performance data processing system can be realized, and real-time performance and response speed can be improved.

[0031] In a second aspect, an embodiment of the present application provides a data threat discovery device, data entity recognition is performed, and first entity information contained in to-be-processed data is obtained; the to-be-processed data is asset information of each asset in a to-be-detected system.

[0032] An analysis module is configured to match the first entity information with second entity information in a vulnerability intelligence library, and obtain a vulnerability analysis result; the vulnerability intelligence library includes a plurality of vulnerability intelligence, and the vulnerability intelligence library is generated by acquiring original intelligence data from a plurality of intelligence sources and performing data standardization on the original intelligence data.

[0033] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a memory and a bus, wherein,

[0034] The processor and the memory complete mutual communication through the bus;

[0035] The memory stores program instructions executable by the processor, and the processor calling the program instructions can execute the method of the first aspect.

[0036] In a fourth aspect, an embodiment of the present application provides a non-transitory computer readable storage medium, including:

[0037] The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method of the first aspect.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which, when read and executed by a processor, perform the method of the first aspect.

[0039] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or can be learned by practice of the application. The objects and other advantages of the present application will be realized and attained by means of the instrumentalities particularly pointed out in the written description and claims hereof. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0041] Figure 1 A data vulnerability detection method flow diagram provided by an embodiment of the present application;

[0042] Figure 2 A data standardization flow diagram provided by an embodiment of the present application;

[0043] Figure 3 A streaming processing vulnerability method flow diagram provided by an embodiment of the present application;

[0044] Figure 4 A data vulnerability detection device structure diagram provided by an embodiment of the present application;

[0045] Figure 5 An electronic device entity structure diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0046] The embodiments of the technical solutions of the present application will be described in detail below with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of this application; the use of the terms "including," "comprising," "having" and "with" in the specification and claims, along with their derivatives, are meant to be interpreted as specifying inclusion of the referenced elements, but not exclusion of any other elements.

[0048] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0049] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to each other. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0050] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0051] In the description of the embodiments of the present application, the term "a plurality of" refers to two or more (including two), and similarly, "a plurality of groups" refers to two or more groups (including two groups), and "a plurality of pieces" refers to two or more pieces (including two pieces).

[0052] In the description of the embodiments of the present application, unless otherwise explicitly specified and limited, the technical terms "mounting", "connecting", "connecting", "fixing" and the like should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanical connection, or it can be electrical connection; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0053] In order to facilitate understanding, the related terms involved in the embodiments of the present application are explained and described:

[0054] Vulnerability intelligence refers to the intelligence information about information security vulnerabilities of computer systems, network devices, software applications, etc. These intelligence includes known vulnerability descriptions, impact range, possible attack methods, published vulnerability fixes, etc. The collection and analysis of vulnerability intelligence is very important for network security teams and security professionals, which can help them identify potential security threats and take timely measures to prevent and repair, thereby improving the security and defense capability of the system.

[0055] Word2Vec: Word2Vec is a technique for representing words as vectors in a continuous space, which is a word embedding method in the field of natural language processing. Word2Vec model was proposed by Tomas Mikolov et al. of Google in 2013, which maps words to real number vectors in a high-dimensional space by learning the context information of words in a large-scale text corpus.

[0056] GloVe: GloVe (Global Vectors for Word Representation) is a word embedding technique for learning word vector representations, proposed by researchers at Stanford University in 2014. Similar to Word2Vec, GloVe also aims to represent words as vectors in a continuous space to capture the semantic relationships between words.

[0057] The main difference between GloVe and Word2Vec is that GloVe is based on global word frequency statistics, rather than local context window prediction. The core idea of GloVe is to learn the relationship between words by minimizing a loss function, so that the co-occurrence probability information of word vector representation in the global corpus is best.

[0058] NER: Named Entity Recognition (NER) is an important task in natural language processing, which aims to identify entities with specific meanings in text and classify them into predefined categories. NER models are usually used to extract named entities from text, such as names, organization names, places, dates, times, etc., and assign them the correct labels.

[0059] Figure 1 A flowchart of a data vulnerability detection method provided by an embodiment of the present application is shown in Figure 1As shown. The vulnerability detection method for data provided by the embodiments of the application can be applied to an electronic device, which includes a terminal and a server; the terminal can be specifically a smartphone, a tablet computer, a computer, a personal digital assistant (PDA), etc.; and the server can be specifically an application server or a Web server. The method includes the following steps:

[0060] Step 101: performing entity recognition on the to-be-processed data by using an entity recognition model to obtain first entity information contained in the to-be-processed data.

[0061] The to-be-processed data refers to asset information of each asset in the to-be-detected system, wherein the assets include hardware assets and software assets. The hardware assets include computers, servers, network devices, storage devices, etc. The software assets include operating systems, database management systems, middleware, application programs, etc. The assets are taken as the first entity information corresponding to the to-be-processed system in the embodiments of the application.

[0062] The entity recognition model (NER) pre-trained and run in the electronic device can recognize the entity of the to-be-processed data.

[0063] Step 102: performing matching between the first entity information and second entity information in a vulnerability intelligence library to obtain a vulnerability analysis result.

[0064] The electronic device is connected to multiple intelligence sources to obtain original intelligence data from the multiple intelligence sources. Since the original intelligence data corresponding to different intelligence sources are in different formats and the same field means different fields, for example, the vulnerability description field of vulnerability database A is "Description", and the vulnerability description field of vulnerability database B is "Vulnerability_Description". Both are vulnerability description fields, but the actual fields are different. Therefore, the initial intelligence data can be standardized before being stored in the intelligence library to unify the data format and field representation.

[0065] The electronic device matches the first entity information with the second entity information in the vulnerability intelligence library after obtaining the first entity information to determine whether there is a vulnerability in the to-be-detected system. It can be understood that the second entity information corresponding to each vulnerability intelligence in the vulnerability intelligence library can also be pre-recognized by using the above-mentioned entity recognition model. If the first entity information matches the second entity information of a certain vulnerability intelligence, it means that the to-be-detected system has the vulnerability corresponding to this vulnerability intelligence; otherwise, it means that the to-be-detected system does not have the vulnerability corresponding to this vulnerability intelligence.

[0066] The embodiment of the application improves the accuracy of entity extraction in the to-be-processed data by using an entity recognition model to perform entity recognition on the to-be-processed data, and improves the accuracy of vulnerability detection by combining vulnerability intelligence obtained from multiple intelligence sources.

[0067] On the basis of the above embodiment, the method further comprises:

[0068] Receiving original intelligence data from multiple intelligence sources;

[0069] Mapping fields in the original intelligence data into standard fields by using a pre-constructed field mapping rule;

[0070] Generating the vulnerability intelligence library according to the standard fields.

[0071] In the specific implementation process, Figure 2 A data standardization process schematic diagram provided by the embodiment of the application is shown in Figure 2 .

[0072] When standardizing the original intelligence data of multiple intelligence sources, the specific operation steps can include:

[0073] (1) Formulate data standardization specifications: define unified data formats and standards, including field naming, data types, units, coding methods, etc., to ensure that data from different data sources can be effectively connected and integrated.

[0074] (2) Data cleaning and conversion: collect original intelligence data from key products, security media, blogs, forums and other websites through standardization tools, clean and convert the original intelligence data, including removing duplicate data, handling missing values, unifying date formats, etc., so that the data meets the formulated standardization specifications.

[0075] (3) Unified data structure: convert data from different data sources into a unified data structure, such as a unified data table or data model, to facilitate unified data processing and analysis.

[0076] (4) Use data standardization tools: use data standardization tools or platforms to automatically standardize and format the data, improving the efficiency and accuracy of data processing.

[0077] (5) Establish data mapping relationship: extract vulnerability information from the original intelligence data, establish a mapping relationship between different data sources, and map and correspond the data fields of different data sources to ensure the accuracy and integrity of data integration.

[0078] Example: Mapping relationship of integrated vulnerability intelligence data sources.

[0079] 1. Data source description:

[0080] Assume there are two different vulnerability intelligence data sources: Vulnerability Database A and Vulnerability Database B. These two data sources may have differences in the fields of vulnerability information provided, and mapping relationships need to be established for integration.

[0081] 2. Mapping relationship establishment:

[0082] Vulnerability description field mapping: The vulnerability description field of Vulnerability Database A is "Description", and the vulnerability description field of Vulnerability Database B is "Vulnerability_Description". Establish a mapping relationship to map "Description" of Database A to "Vulnerability_Description" of Database B.

[0083] Affected software field mapping: The affected software field of Vulnerability Database A is "Affected_Software", and the affected software field of Vulnerability Database B is "Affected_Product". Establish a mapping relationship to map "Affected_Software" of Database A to "Affected_Product" of Database B.

[0084] Vulnerability level field mapping: The vulnerability level field of Vulnerability Database A is "Severity", and the vulnerability level field of Vulnerability Database B is "Risk_Level". Establish a mapping relationship to map "Severity" of Database A to "Risk_Level" of Database B.

[0085] (6) Generate a work order, after completing the mapping, generate an intelligence data work order on the vulnerability intelligence operation platform. The vulnerability intelligence operation platform judges whether the intelligence data is related to the key product. If it is, the operator can make a judgment on the work order. If it is indeed new vulnerability information, it will be stored in the vulnerability library. If it is not related to the key product, use a one-step task to cache the vulnerability id to redis monitoring, get the monitored vulnerability intelligence data, supplement the vulnerability detail task, and then write the vulnerability intelligence data to the intelligence library.

[0086] For ease of understanding, a specific use case of vulnerability intelligence data standardization and unified formatting is explained:

[0087] Data source description: Assume that the security team subscribes to multiple vulnerability intelligence sources, and each source provides slightly different fields of vulnerability information, which needs to be unified formatted.

[0088] Field mapping: Vulnerability description field: Different data sources may use different field names, and are unified as "Description".

[0089] Affected Software field: unified as "Affected_Software".

[0090] Vulnerability Severity field: unified as "Severity".

[0091] Data cleaning and preprocessing: remove null values and outliers.

[0092] Standardize date format to YYYY-MM-DD.

[0093] Unit standardization: unify all vulnerability fix time units to days.

[0094] Data integration and storage: store the formatted vulnerability information in a unified vulnerability database.

[0095] Example application:

[0096] Vulnerability information query: quickly query and compare different vulnerability information through the unified formatted vulnerability database.

[0097] Risk assessment: score and judge the risk of each different vulnerability information, and assign default values according to the affected product, version, type, etc., to obtain the vulnerability risk level.

[0098] The embodiments of the present application can simplify the complexity of integrating multiple data sources, improve the efficiency and accuracy of data processing, and better meet the integration and processing needs of security threat intelligence information by using data standardization and unified formatting techniques.

[0099] On the basis of the above embodiments, the method further comprises:

[0100] Obtain training samples and perform word segmentation processing on the training samples using a GloVe model to obtain a plurality of word vectors;

[0101] Use BIO format to label the word vectors;

[0102] Input the labeled word vectors into a to-be-trained model to obtain predicted entity information output by the to-be-trained model;

[0103] Optimize the parameters of the to-be-trained model based on the predicted entity information and the labeling results to obtain a trained entity recognition model.

[0104] In the specific implementation process, the training sample can be the obtained historical intelligence data. After obtaining the historical intelligence data, the historical intelligence data is processed by word segmentation, stemming, and removal of stop words, to obtain a plurality of word vectors corresponding to the historical intelligence data, and then the word vectors are labeled by using a BIO format. It should be noted that the BIO format is a label-based prefix to represent the start (B-) and internal (I-) of an entity, and the non-entity (O) mark. Specifically, B (Begin) represents the start of a word block, i.e., the start part of an entity; I (Inside) represents the internal part of the word block, i.e., the non-start part of the entity; and O (Outside) represents not belonging to any word block, i.e., the non-entity part.

[0105] Word embedding vectors are a method of converting words into numerical vectors, enabling these vectors to capture semantic and syntactic relationships between words. We use the GloVe model, which makes the dot product of two word vectors equal to the logarithm of their co-occurrence probability. The following is the training process of GloVe:

[0106] First, construct a co-occurrence matrix, where X ij represents the number of times word j appears in the context of word i. This matrix is constructed by scanning the entire vulnerability intelligence corpus.

[0107] Then, define a loss function that makes the dot product of word vectors close to the logarithm of their co-occurrence probability. Specifically, if V i and V j are the vectors of words i and j, b i and b j are the corresponding bias terms, then the loss function can be defined as

[0108] where V is the size of the vocabulary, f is a weight function that adjusts the contribution of different co-occurrence counts to the loss function. In the original GloVe model, f is defined as where x max and α are hyperparameters.

[0109] Suppose there is a corpus containing the following four sentences:

[0110] "I love reading books."

[0111] "She loves reading too."

[0112] "I love playing football."

[0113] "She loves playing too."

[0114] To build the co-occurrence matrix, only the neighboring words are considered as context. The matrix can be shown as follows (only partial words are shown, in reality all words should be included):

[0115] I Love reading playing she books football too I 0 2 0 1 0 0 0 0 Love 2 0 1 1 0 0 0 0 reading 0 1 0 0 1 1 0 1 playing 1 1 0 0 1 0 1 1 she 0 0 1 1 0 0 0 2 books 0 0 1 0 0 0 0 0 football 0 0 0 1 0 0 0 0 too 0 0 1 1 2 0 0 0

[0116] Then by defining the loss function, assuming that the word vectors and bias terms are randomly initialized, for the words "I" and "love", the following can be derived: V "I” = [0.1, 0.2], V "love” = [0.3, 0.4], b "l” = 0.1, b "love” = 0.2.

[0117] The loss function is calculated as: f(X "I,love" ) = (V "I" · V "love" + b "I" + b "love" - log X "I,love" ) 2 .

[0118] Let x max = 100, α = 0.75, then f(X "I,love" ) = (2 / 100) 0.75 .

[0119] Therefore, the term of this loss function is: (2 / 100) 0.75 * ([0.1, 0.2] · [0.3, 0.4] + 0.1 + 0.2 - log 2) 2 .

[0120] All such terms need to be calculated and then summed up to get the total loss function.

[0121] After training, valuable entity information can be extracted from a large amount of text data, such as identifying CVE (Common Vulnerabilities and Exposures) identifiers, which are unique identifiers used to identify specific vulnerabilities.

[0122] If a vulnerability report mentions "Microsoft Windows 10", then NER can be used to identify "Microsoft" (organization) and "Windows 10" (product).

[0123] In some cases, a vulnerability report may mention threat actors related to the vulnerability. NER can help us identify these threat actors.

[0124] Once these entities are identified, they can be used for further analysis. For example, it can be analyzed which products are most often attacked by vulnerabilities, or which threat actors are most active. These entities can also be used to help understand and classify vulnerability reports, for example, reports can be grouped according to the products or threat actors mentioned in the reports.

[0125] In addition, the identified entities can also be used for information retrieval and vulnerability solution systems. For example, if a user is looking for vulnerabilities related to "Windows 10", NER can be used to find all reports mentioning this product. Or if a user is interested in a certain threat actor, solutions related to that threat actor and their reports can be recommended, etc.

[0126] After completing the labeling, the labeled word vectors are input into the to-be-trained model, which can be BERT, BERT-based model, BERT-CRF, etc. The to-be-trained model outputs the prediction result, i.e. the predicted entity information, and then calculates the loss function according to the predicted entity information and the labeling result, and optimizes the parameters of the to-be-trained model using the loss value obtained by calculation, to obtain the trained entity recognition model. It can be understood that when training the model, it can be trained in batches until the requirements for stopping training are met.

[0127] It should be noted that the entity recognition model can be continuously optimized and adjusted, and through the feedback mechanism and continuous learning, the ability of the model to accurately extract attribute information in vulnerability intelligence text can be improved. The results output by the entity recognition model can also be post-processed and corrected to ensure that the extracted attribute information is accurate and correct, and can be further verified in combination with a rule engine or manual review.

[0128] For example: The vulnerability intelligence is: In the latest version of a certain software, a serious security vulnerability has been found, and attackers can use this vulnerability to execute remote code, affecting all Windows systems installed with the software. It is recommended that users upgrade to version 2.0.1 immediately to fix this vulnerability.

[0129] After the entity recognition model extracts the vulnerability intelligence, the following named entities can be obtained:

[0130] Vulnerability type: security vulnerability;

[0131] Software name: a certain software;

[0132] Vulnerability impact scope: all Windows systems installed with the software;

[0133] Vulnerability repair suggestion: upgrade to version 2.0.1.

[0134] In this example, NER technology can help automatically identify and extract key information in vulnerability descriptions, helping security teams quickly understand vulnerability intelligence content and take appropriate measures to address potential security threats. In this way, the efficiency and accuracy of vulnerability intelligence processing can be improved, helping organizations better protect their systems and data security.

[0135] Based on the above embodiments, the first entity information is matched with the second entity information in the vulnerability intelligence library to obtain a vulnerability analysis result, including:

[0136] The Levenshtein distance algorithm is used to calculate the similarity between the second entity information and the first entity information corresponding to each asset in the system to be detected.

[0137] If the similarity is greater than a preset threshold, it is determined that the asset has a vulnerability threat.

[0138] In the specific implementation process, the Levenshtein distance is an algorithm for measuring the similarity between two strings, which determines the similarity between them by calculating the minimum number of operations required to convert one string to another through insertion, deletion and replacement operations. In the asset information matching process, the Levenshtein distance algorithm can be used to compare the similarity between asset information, thereby achieving accurate matching.

[0139] When matching the first entity information with the second entity information, the Levenshtein distance algorithm can be used to calculate the similarity between the first entity information and the second entity information. The similarity is also the similarity between the assets of the system to be detected and the assets involved in each vulnerability in the vulnerability library. If the similarity is greater than a preset threshold, it is determined that the asset has a vulnerability threat, and otherwise, it is determined that the asset does not have the vulnerability.

[0140] For example: Vulnerability description: "The latest version of a certain software has a vulnerability that may affect the company's server A, server B and database server. It is recommended to upgrade the software immediately to fix the vulnerability." Known asset information in the system to be detected includes: server A, server C, database server and network device.

[0141] The asset information matching process is as follows:

[0142] Extract the assets mentioned in the vulnerability description: server A, server B, database server.

[0143] Use the Levenshtein distance algorithm to calculate the similarity between each asset in the vulnerability description and the known asset information. According to the similarity threshold, determine which known assets have high matching degree with the assets mentioned in the vulnerability description and may be affected.

[0144] Therefore, appropriate security measures can be taken on assets with high matching degrees, such as immediately upgrading software or strengthening monitoring. Of course, increasing manual review to ensure correct operation and avoid false positives leading to abnormal upgrades is also supported.

[0145] In addition, the embodiments of the present application can also combine the Levenshtein distance algorithm with other algorithms, such as weighted Levenshtein distance, n-gram-based similarity matching, etc., to improve the accuracy and efficiency of matching. In addition, the matching results can be verified and post-processed to ensure the accuracy of the matching, and further confirmation can be made in combination with domain knowledge and rule engines and manual review.

[0146] The embodiments of the present application can effectively realize the matching between asset information by applying the string similarity matching algorithm based on the Levenshtein distance, expand the asset vulnerability information and judgment basis, improve the accuracy and reliability of the matching, and provide a more reliable data basis for subsequent threat intelligence analysis and response.

[0147] On the basis of the above-mentioned embodiments, the method further comprises:

[0148] Obtaining vulnerability event information of the system to be detected in a preset historical time period;

[0149] Inputting the vulnerability event information into an event prediction model to obtain a prediction result output by the event prediction model; the prediction result is used to represent whether a vulnerability occurs in the future in the preset time period; the event prediction model is obtained by training a long short-term memory network using training vulnerability event information.

[0150] In the specific implementation process, the long short-term memory network is a deep learning model suitable for processing time series data, which can capture long-term dependencies in time series data and has a memory and forgetting mechanism, and is suitable for strengthening the event prediction model.

[0151] When performing event prediction, the following steps can be implemented:

[0152] 1. Data preparation: prepare historical training vulnerability event information as a training set, including the timestamp and related features of the event. Preprocess and feature engineer the training vulnerability event information to meet the input requirements of the LSTM model. Collect historical training vulnerability event information, including vulnerability description, release time, affected system, etc. Convert the training vulnerability event information into time series data, where each time step represents a vulnerability event. Convert the training vulnerability event information into word vector representation, which can use pre-trained word embedding models such as Word2Vec or GloVe. Build time series features, such as the number of previous vulnerability events, vulnerability type distribution, etc.

[0153] 2. Build LSTM Model: Design and build an LSTM model suitable for event prediction, including input layer, hidden layer, and output layer. Adjust the model's parameters and structure to adapt to specific event prediction tasks.

[0154] 3. Model Training: Train the LSTM model using prepared historical event data, continuously optimize model parameters through backpropagation algorithm, so that it can better learn the patterns and rules of event data.

[0155] 4. Model Validation and Tuning: Validate the trained LSTM model using the validation set, evaluate its prediction performance and tune it to improve prediction accuracy and generalization ability.

[0156] 5. Real-time Prediction and Monitoring: Apply the trained LSTM model to real-time event data for event prediction and monitor model performance. Adjust model parameters and update model in a timely manner to adapt to dynamic changes in event data.

[0157] 6. Continuous Optimization: Continuously collect new event data, retrain the LSTM model to adapt to changing event patterns, and continuously optimize event prediction results.

[0158] By using long short-term memory network (LSTM) to enhance the event prediction model, it can better capture the time dependence of event data, improve the accuracy and reliability of prediction, and provide more effective event prediction and decision support for enterprises.

[0159] The electronic device can obtain vulnerability event information of the to-be-detected system in a preset historical time period, for example, it can be only one week, two weeks or one month. There may be a correlation between vulnerability event information in time, for example, vulnerability A appeared on the to-be-detected system a few days ago, and vulnerability B may appear a few days later.

[0160] After obtaining the vulnerability event information, input the vulnerability event information into the event prediction model to obtain the prediction result output by the event prediction model.

[0161] For example, using vulnerability intelligence data to predict future network security vulnerability events, vulnerability report data including vulnerability description, impact range, repair suggestions, etc.

[0162] Based on historical vulnerability event information, the LSTM model has learned the occurrence rules and impact range of different vulnerability types, and uses the model to predict the vulnerability types that may exist in a certain software version and the affected systems. For example, the model may predict that the specific software version used by the company in the next month may have a high risk of SQL injection attacks, so that vulnerability event monitoring and timely preparation work can be carried out in advance.

[0163] In the embodiments of the present application, the data input into the event prediction model includes time series data: the number of vulnerabilities per month and vulnerability descriptions: keywords describing vulnerabilities.

[0164] Design an LSTM model that combines time series data and vulnerability descriptions to predict future vulnerability events. For example, the model may learn that a certain type of vulnerability occurs in a specific month every year, or that the number of vulnerabilities increases when certain keywords appear. By using an LSTM model combined with vulnerability intelligence data, future potential vulnerability events can be more accurately predicted, allowing timely preventive measures to be taken to enhance network security defenses.

[0165] By combining an LSTM model and vulnerability intelligence data to more accurately predict potential vulnerability events, preventive measures can be taken in advance, thereby improving network security defenses.

[0166] On the basis of the above-mentioned embodiments, the method is applied to a stream processing platform; the stream processing platform includes a stream processing engine; the matching based on the first entity information and the second entity information in the vulnerability intelligence library includes:

[0167] The stream processing engine is used to match the first entity information with the second entity information in the vulnerability intelligence library.

[0168] In the specific implementation process, stream processing is a technology for real-time processing of data streams, which can quickly process data and generate results in real time. Through stream processing technology, a high-performance data processing system can be realized, improving real-time performance and response speed.

[0169] When using a stream architecture, data stream processing architecture design needs to be done in advance, including data source access, data stream transmission, data processing and result output modules. Choose a suitable stream processing engine, such as Apache Kafka, Apache Flink, Apache Storm, etc. The stream processing system can be horizontally expanded to distribute the load to multiple nodes, improving the processing capacity and throughput of the system to meet large-scale data stream processing needs. Fault tolerance mechanisms can also be set up to keep the system running stably in the event of a failure. Manage the state of the stream processing system to recover and maintain the system state.

[0170] After completing the design of the stream architecture, the data to be processed is transmitted to the stream processing system in the form of a stream, and the data stream is processed in real time. The stream processing system can perform real-time computation, filtering, aggregation, association and other operations on the data to quickly generate real-time results.

[0171] The electronic device can monitor the running state and performance indicators of the stream processing system in real time, and timely discover and solve potential problems. According to the monitoring data, the performance is optimized to improve the response speed and efficiency of the system.

[0172] The electronic device can persistently store the processed data (such as vulnerability intelligence, etc.) for subsequent analysis and query. Integrating with other systems, realizing data exchange and sharing, improving the overall performance of the data processing system.

[0173] Figure 3 A flow processing vulnerability method flowchart provided by the embodiment of the application is shown in Figure 3 As shown in the figure, it includes:

[0174] When the electronic device accesses multiple vulnerability intelligence sources, each source will continuously send new vulnerability information. These information includes vulnerability description, affected software, vulnerability level, etc. The electronic device can use a stream processing platform (such as Apache Kafka) to receive data stream from the intelligence source, and the data stream includes vulnerability description, affected software, vulnerability level, etc. Use stream processing engine (such as Apache Flink, Apache Storm) to process and analyze the received vulnerability information in real time. Store the processed data in the database, and display the statistical information, trend and alarm of vulnerability intelligence in real time through the visualization tool (such as Kibana).

[0175] In the stream processing process, the received vulnerability information is cleaned and filtered to remove duplicate data or invalid information. Extract the keywords and features in the vulnerability description, and match them with the known vulnerability database to identify potential new vulnerabilities. According to the vulnerability level and the importance of the affected software, the vulnerability is analyzed and evaluated in real time to determine its threat level to system security. For the case of high-risk vulnerabilities identified, the system can immediately trigger an alarm to push relevant personnel and start an automated response program, such as blocking the affected system or triggering the repair process. This step will use the Levenshtein distance string similarity matching algorithm to match asset information to maximize the matching of assets and automatic response to vulnerabilities.

[0176] In addition, the visualization tool (such as Kibana) can be used to display the statistical information, trend and alarm of vulnerability intelligence in real time.

[0177] Through the above process, the real-time vulnerability detection system can timely discover potential security vulnerabilities and take corresponding measures to protect the system from potential threats. The stream processing technology enables the entire process to be performed in real time as data is continuously generated, improving the efficiency and accuracy of security protection.

[0178] Figure 4A structural schematic diagram of a data vulnerability detection device provided by an embodiment of the present application. The device can be a module, a program segment, or a code on an electronic device. It should be understood that the device is consistent with the above-mentioned Figure 1 The method embodiment corresponds to, and can execute Figure 1 The device corresponds to each step involved in the method embodiment, and the specific functions of the device can be referred to the description in the above. To avoid repetition, the detailed description is appropriately omitted here. The device comprises an identification module 401 and an analysis module 402, wherein:

[0179] The identification module 401 is configured to perform entity identification on the to-be-processed data by using an entity identification model to obtain first entity information contained in the to-be-processed data; the to-be-processed data is asset information of each asset in a to-be-detected system;

[0180] The analysis module 402 is configured to perform matching between the first entity information and second entity information in a vulnerability intelligence library to obtain a vulnerability analysis result; wherein the vulnerability intelligence library comprises a plurality of pieces of vulnerability intelligence, and the vulnerability intelligence library is generated by acquiring original intelligence data from a plurality of intelligence sources and performing data standardization on the original intelligence data.

[0181] On the basis of the above-mentioned embodiment, the device further comprises an intelligence library generation module configured to:

[0182] receive original intelligence data from a plurality of intelligence sources;

[0183] map fields in the original intelligence data into standard fields by using a pre-constructed field mapping rule;

[0184] generate the vulnerability intelligence library according to the standard fields.

[0185] On the basis of the above-mentioned embodiment, the device comprises a model training module configured to:

[0186] obtain training samples and perform word segmentation processing on the training samples by using a GloVe model to obtain a plurality of word vectors;

[0187] label the word vectors by using a BIO format;

[0188] input the labeled word vectors into a to-be-trained model to obtain predicted entity information output by the to-be-trained model;

[0189] optimize parameters of the to-be-trained model based on the predicted entity information and a labeling result to obtain a trained entity identification model.

[0190] On the basis of the above-mentioned embodiment, the analysis module 402 is specifically configured to:

[0191] calculate similarity between the second entity information and first entity information corresponding to each asset in the to-be-detected system by using a Levenshtein distance algorithm;

[0192] If the similarity is greater than a preset threshold, it is determined that the asset has a vulnerability threat.

[0193] On the basis of the above-mentioned embodiments, the device further comprises a prediction module configured to:

[0194] obtain vulnerability event information of the to-be-detected system in a preset historical time period;

[0195] input the vulnerability event information into an event prediction model to obtain a prediction result output by the event prediction model; the prediction result is used to represent whether the to-be-detected system will have a vulnerability in a future preset time period; the event prediction model is obtained by training a long short-term memory network by using training vulnerability event information.

[0196] On the basis of the above-mentioned embodiments, the device is applied to a stream processing platform; the stream processing platform comprises a stream processing engine; the analysis module 402 is specifically configured to:

[0197] match the first entity information with second entity information in a vulnerability intelligence library by using the stream processing engine.

[0198] Figure 5 An electronic device entity structure diagram provided by the embodiments of the present application is shown in FIG. 1, which comprises a processor 501, a memory 502 and a bus 503; wherein, Figure 5

[0199] The processor 501 and the memory 502 communicate with each other through the bus 503;

[0200] The processor 501 is configured to call program instructions in the memory 502 to execute the method provided by each method embodiment, for example, comprising: performing entity recognition on to-be-processed data by using an entity recognition model to obtain first entity information contained in the to-be-processed data; the to-be-processed data is asset information of each asset in a to-be-detected system; performing matching between the first entity information and second entity information corresponding to vulnerability intelligence in a vulnerability intelligence library to obtain a vulnerability analysis result; wherein the vulnerability intelligence library comprises a plurality of pieces of vulnerability intelligence, and the vulnerability intelligence library is generated by obtaining original intelligence data from a plurality of intelligence sources and performing data standardization on the original intelligence data.

[0201] ​The processor 501 can be an integrated circuit chip with a signal processing capability. The processor 501 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), and the like; or can be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 501 can implement or execute the various methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor, or the processor can also be any conventional processor.

[0202] The memory 502 can include, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and the like.

[0203] The embodiment discloses a computer program product, the computer program product includes a computer program stored on a non-transitory computer readable storage medium, the computer program includes program instructions, when the program instructions are executed by a computer, the computer can execute the method provided by each method embodiment, for example, including: performing entity recognition on the to-be-processed data by using an entity recognition model to obtain first entity information contained in the to-be-processed data; the to-be-processed data is asset information of each asset in a to-be-detected system; based on matching the first entity information and second entity information corresponding to vulnerability intelligence in a vulnerability intelligence library, a vulnerability analysis result is obtained; wherein the vulnerability intelligence library includes a plurality of pieces of vulnerability intelligence, and the vulnerability intelligence library is generated by acquiring original intelligence data from a plurality of intelligence sources and performing data standardization on the original intelligence data.

[0204] The embodiment provides a non-transitory computer-readable storage medium storing computer instructions, the computer instructions causing the computer to perform the method provided in each method embodiment, for example comprising: performing entity recognition on to-be-processed data by using an entity recognition model to obtain first entity information contained in the to-be-processed data; the to-be-processed data is asset information of each asset in a to-be-detected system; performing matching based on the first entity information and second entity information corresponding to vulnerability intelligence in a vulnerability intelligence library to obtain a vulnerability analysis result; wherein the vulnerability intelligence library includes a plurality of pieces of vulnerability intelligence, and the vulnerability intelligence library is generated by acquiring original intelligence data from a plurality of intelligence sources and performing data standardization on the original intelligence data.

[0205] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The embodiments described above are merely schematic, for example, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0206] In addition, the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0207] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0208] In this document, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations.

[0209] The above only describes the embodiments of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting threats to data, characterized in that, include: An entity recognition model is used to identify entities in the data to be processed, thereby obtaining the first entity information contained in the data to be processed. The data to be processed is the asset information of each asset in the system to be detected; The vulnerability analysis results are obtained by matching the first entity information with the second entity information corresponding to the vulnerability intelligence in the vulnerability intelligence database; wherein, the vulnerability intelligence database includes multiple vulnerability intelligences, and the vulnerability intelligence database is generated by obtaining raw intelligence data from multiple intelligence sources and standardizing the raw intelligence data; The method further includes: Obtain training samples and use the GloVe model to perform word segmentation on the training samples to obtain multiple word vectors; The word vectors are annotated using the BIO format; The labeled word vectors are input into the model to be trained to obtain the predicted entity information output by the model to be trained. The parameters of the model to be trained are optimized based on the predicted entity information and the annotation results to obtain the trained entity recognition model. The process of matching the first entity information with the second entity information in the vulnerability intelligence database to obtain vulnerability analysis results includes: The similarity between the second entity information and the first entity information corresponding to each asset in the system to be detected is calculated using the Levenshtein distance algorithm. If the similarity is greater than a preset threshold, then the asset is determined to have a vulnerability threat. The method further includes: Obtain vulnerability event information of the system to be tested within a preset historical time period; The vulnerability event information is input into the event prediction model to obtain the prediction result output by the event prediction model; the prediction result is used to characterize whether the system to be detected will have a vulnerability in the future within a preset time period; the event prediction model is obtained by training a long short-term memory network using training vulnerability event information.

2. The method according to claim 1, characterized in that, The method further includes: Receive raw intelligence data from multiple intelligence sources; Using pre-built field mapping rules, the fields in the original intelligence data are mapped to standard fields; The vulnerability intelligence database is generated based on standard fields.

3. The method according to any one of claims 1-2, characterized in that, The method is applied to a streaming processing platform; The streaming processing platform includes a streaming processing engine; the matching based on the first entity information and the second entity information in the vulnerability intelligence database includes: The streaming engine matches the first entity information with the second entity information in the vulnerability intelligence database.

4. A data threat detection device, characterized in that, include: The recognition module is used to perform entity recognition on the data to be processed using an entity recognition model, and to obtain the first entity information contained in the data to be processed. The data to be processed is the asset information of each asset in the system to be detected; The analysis module is used to match the first entity information with the second entity information in the vulnerability intelligence database to obtain vulnerability analysis results; wherein, the vulnerability intelligence database includes multiple vulnerability intelligences, and the vulnerability intelligence database is generated by obtaining raw intelligence data from multiple intelligence sources and standardizing the raw intelligence data; The device further includes a model training module for: Obtain training samples and use the GloVe model to perform word segmentation on the training samples to obtain multiple word vectors; The word vectors are annotated using the BIO format; The labeled word vectors are input into the model to be trained to obtain the predicted entity information output by the model to be trained. The parameters of the model to be trained are optimized based on the predicted entity information and the annotation results to obtain the trained entity recognition model. The analysis module is specifically used for: The similarity between the second entity information and the first entity information corresponding to each asset in the system to be detected is calculated using the Levenshtein distance algorithm. If the similarity is greater than a preset threshold, then the asset is determined to have a vulnerability threat. The device further includes a prediction module for: Obtain vulnerability event information of the system to be tested within a preset historical time period; The vulnerability event information is input into the event prediction model to obtain the prediction result output by the event prediction model; the prediction result is used to characterize whether the system to be detected will have a vulnerability in the future within a preset time period; the event prediction model is obtained by training a long short-term memory network using training vulnerability event information.

5. An electronic device, characterized in that, include: Processor, memory, and bus, among which, The processor and the memory communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor can execute the method as described in any one of claims 1-3 by calling the program instructions.

6. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-3.

7. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-3.

Images