Threat indicator extraction method, device, medium and electronic equipment

Abstract

The present disclosure relates to a threat indicator extraction method, device, medium and electronic device. The method includes: using a classification model based on machine learning and at least one extraction model based on regular expressions to respectively extract threat indicators of the data of threat indicators to be extracted; generating target threat indicators of the above data according to the first extraction result of the classification model and the second extraction result of each extraction model. In this way, machine learning technology and regular expression technology can be combined to form a multi-level, multi-angle IOC extraction system. On the one hand, the extraction system can accurately extract indicators of known threats based on known threat features and patterns, and on the other hand, it can effectively identify and extract indicators of new and unknown threats without relying on known patterns. In this way, it can not only effectively extract various types of IOCs, but also adapt to the dynamic changes and increasing complexity of network threats, and improve the comprehensiveness and foresight of network security protection.

Description

Technical Field

[0001] The present disclosure relates to the field of network security technology, and in particular to a threat indicator extraction method, device, medium and electronic device. Background Art

[0002] With the development and popularization of network technology, network security issues have become increasingly prominent. Network threats such as viruses, malware, and phishing attacks occur frequently, posing a serious threat to the network security of individuals and enterprises. In order to deal with these threats, various network security tools and technologies have emerged, such as network full flow analysis technology, which can be used to detect and prevent potential network attacks and protect network security.

[0003] However, traditional network security tools and technologies mainly rely on known threat features and patterns for detection, and are often unable to effectively respond to new and unknown threats. In addition, traditional network traffic analysis tools can usually only provide information based on the network layer such as IP and port, but cannot go deep into the application layer to extract valuable information from network traffic loads, such as Indicators of Compromise (IOC). Summary of the invention

[0004] This summary is provided to introduce concepts in a brief form that will be described in detail in the detailed description below. This summary is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, the present disclosure provides a threat indicator extraction method, comprising:

[0006] Obtain data on threat indicators to be extracted;

[0007] extracting threat indicators of the data respectively using a classification model based on machine learning and at least one extraction model based on regular expressions;

[0008] A target threat indicator of the data is generated according to the first extraction result of the classification model and the second extraction result of each of the extraction models.

[0009] In a second aspect, the present disclosure provides a threat indicator extraction device, comprising:

[0010] A first acquisition module is used to acquire data of threat indicators to be extracted;

[0011] an extraction module, configured to respectively extract threat indicators of the data using a classification model based on machine learning and at least one extraction model based on regular expressions;

[0012] A generating module is used to generate a target threat indicator of the data according to the first extraction result of the classification model and the second extraction result of each of the extraction models.

[0013] In a third aspect, the present disclosure provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processing device, the steps of the threat indicator extraction method provided in the first aspect of the present disclosure are performed.

[0014] In a fourth aspect, the present disclosure provides an electronic device, including:

[0015] a storage device having a computer program stored thereon;

[0016] A processing device is used to execute the computer program in the storage device to implement the steps of the threat indicator extraction method provided in the first aspect of the present disclosure.

[0017] In the above technical solution, firstly, a classification model based on machine learning and at least one extraction model based on regular expressions are used to extract the IOC of the data of the threat indicators to be extracted respectively; then, according to the first extraction result of the classification model and the second extraction result of each extraction model, the target threat indicator of the data of the threat indicators to be extracted is generated. In this way, the machine learning technology and the regular expression technology can be combined to form a multi-level and multi-angle IOC extraction system. On the one hand, the extraction system can accurately extract the indicators of known threats based on known threat features and patterns, and on the other hand, it can effectively identify and extract the indicators of new and unknown threats without relying on known patterns. In this way, it can not only effectively extract various types of IOC, but also adapt to the dynamic changes and increasingly complex trends of network threats, thereby improving the comprehensiveness and foresight of network security protection. In addition, the classification model based on machine learning can automatically learn and extract IOC, which can reduce the dependence on manually writing and maintaining regular expressions, thereby reducing manpower and time costs while improving the efficiency and accuracy of IOC extraction.

[0018] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale. In the drawings:

[0020] Figure 1 The present invention is a flowchart of a method for extracting threat indicators according to an exemplary embodiment.

[0021] Figure 2 The diagram is a schematic diagram of a threat indicator extraction method according to an exemplary embodiment.

[0022] Figure 3 The figure is a schematic diagram of an attack scenario according to an exemplary embodiment.

[0023] Figure 4 A schematic diagram of data preprocessing is shown according to an exemplary embodiment.

[0024] Figure 5 A schematic diagram of an expert model extracting IOC of threat indicator data to be extracted according to an exemplary embodiment is shown.

[0025] Figure 6 A schematic diagram of an IOC workflow for classifying and extracting data of threat indicators to be extracted according to an exemplary embodiment is shown.

[0026] Figure 7 A schematic diagram of a process for generating a target threat indicator of data to be extracted threat indicators according to a first extraction result of a classification model and a second extraction result of each extraction model is shown according to an exemplary embodiment.

[0027] Figure 8 The diagram is a schematic diagram of an alarm data processing system according to an exemplary embodiment.

[0028] Fig. 9 The present invention is a block diagram of a threat indicator extraction device according to an exemplary embodiment.

[0029] Fig.10 The diagram is a schematic structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] In order to solve the problem mentioned in the background technology that it is impossible to extract IOCs at the application level, regular expressions based on known attack types are usually used to extract IOCs from network traffic. This method usually relies on the experience and knowledge of security experts to write precise regular expressions to match and extract specific patterns in network traffic, such as malicious URLs, suspicious IP addresses and ports, malicious instructions, etc. In the field of network security, IOCs are key data points for identifying and detecting malicious activities. These indicators can be file hash values, malicious IP addresses, registry entries, etc.

[0031] This regular expression-based approach works well when dealing with known threats with fixed patterns and can accurately extract IOCs. However, this approach has some obvious limitations. First, for new and unknown threats, this approach may not be able to effectively detect them because their features are not included in the existing regular expressions. Second, writing and maintaining regular expressions requires a lot of manpower and time costs, and this approach has limited scalability and response capabilities for dynamically changing and increasingly complex network threats.

[0032] In addition, since regular expressions can usually only match specific text patterns, this method may not perform well for complex, non-text network traffic, such as binary data, encrypted data, etc. Therefore, although regular expression-based methods work well in some scenarios, there are still many challenges and problems to be solved.

[0033] Therefore, we need a new technical solution that can effectively extract IOCs from various types of network traffic and adapt to the dynamic changes and increasing complexity of network threats. In view of this, the present disclosure provides a threat indicator extraction method, device, medium and electronic device.

[0034] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0035] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0036] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0037] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0038] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0039] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0040] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0041] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0042] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0043] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0044] At the same time, it is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0045] Figure 1 FIG. 1 is a flowchart of a method for extracting threat indicators according to an exemplary embodiment. Figure 1 As shown, the method may include S101 to S103.

[0046] In S101, data of threat indicators to be extracted are obtained.

[0047] In the present disclosure, the data of threat indicators to be extracted can be network traffic data to be subjected to threat indicator extraction, or network text data to be subjected to threat indicator extraction. Among them, network text data can be various types of text content on the Internet, such as intelligence forums, network social content, attack reports, etc. In this way, the threat indicator extraction method provided by the present disclosure can not only extract IOCs from network traffic data, but also effectively extract IOCs from various types of network texts, providing a more comprehensive and in-depth network threat analysis in addition to traffic load analysis.

[0048] In S102, threat indicators of data of threat indicators to be extracted are respectively extracted using a classification model based on machine learning and at least one extraction model based on regular expressions.

[0049] In the present disclosure, a regular expression uses a single string to describe and match a series of strings that conform to a certain IOC rule. If the regular expression successfully matches the data to be matched, it means that the data to be matched contains IOC.

[0050] The above-mentioned classification model based on machine learning can be a language model such as Bidirectional Encoder Representation from Transformers (BERT), or a deep learning model such as a convolutional neural network or a recurrent neural network.

[0051] In S103, a target threat indicator of the data of the threat indicator to be extracted is generated according to the first extraction result of the classification model and the second extraction result of each extraction model.

[0052] In the present disclosure, the number of extraction models may be one or more. The expert model and each extraction model concurrently execute the IOC extraction operation through a coroutine.

[0053] In one embodiment, there is one extraction model. In this case, a classification model based on machine learning can be used to extract IOCs from the data of threat indicators to be extracted to obtain a first extraction result. At the same time, the one extraction model based on regular expressions can be used to extract IOCs from the data of threat indicators to be extracted to obtain a second extraction result. Thereafter, a target threat indicator of the data of threat indicators to be extracted is generated based on the first extraction result and the second extraction result.

[0054] In another embodiment, there are multiple extraction models. In this case, a classification model based on machine learning can be used to extract IOCs from the data of threat indicators to be extracted to obtain a first extraction result. At the same time, multiple extraction models based on regular expressions can be used to extract IOCs from the data of threat indicators to be extracted to obtain multiple second extraction results. Thereafter, a target threat indicator of the data of threat indicators to be extracted is generated based on the first extraction result and the multiple second extraction results.

[0055] In the above technical solution, firstly, a classification model based on machine learning and at least one extraction model based on regular expressions are used to extract the IOC of the data of the threat indicators to be extracted respectively; then, according to the first extraction result of the classification model and the second extraction result of each extraction model, the target threat indicator of the data of the threat indicators to be extracted is generated. In this way, the machine learning technology and the regular expression technology can be combined to form a multi-level and multi-angle IOC extraction system. On the one hand, the extraction system can accurately extract the indicators of known threats based on known threat features and patterns, and on the other hand, it can effectively identify and extract the indicators of new and unknown threats without relying on known patterns. In this way, it can not only effectively extract various types of IOC, but also adapt to the dynamic changes and increasingly complex trends of network threats, thereby improving the comprehensiveness and foresight of network security protection. In addition, the classification model based on machine learning can automatically learn and extract IOC, which can reduce the dependence on manually writing and maintaining regular expressions, thereby reducing manpower and time costs while improving the efficiency and accuracy of IOC extraction.

[0056] The at least one regular expression-based extraction model may include an expert model based on a regular expression of an attack type (Attack Type).

[0057] Among them, the regular expression based on the attack type is a regular expression summarized by security experts based on known attack types, that is, a regular expression summarized based on the IOC rules related to the attack, and the expert model extracts the attack threat indicators of the data of the threat indicators to be extracted based on these regular expressions. The attack type may include one or more of the following: Structured Query Language (SQL) injection attack, cross-site scripting attack (XSS), information leakage attack, code injection attack, file upload attack, file inclusion attack, weak password attack, etc. Here, the attack type is only exemplarily explained without specific limitation.

[0058] like Figure 2As shown in the figure, the expert model can use the summary of known attack scenarios to cover the detection of various threat indicator types such as network address (IP), port (Port), domain name (Domain), link (URL), command (Command, i.e. binary program, such as python, curl, wget, git, etc.), attack type identification, etc., and provide the attack type identification function to give additional attack scenario semantics to the data of threat indicators to be extracted, which is conducive to understanding attack data and supporting other upstream businesses.

[0059] like Figure 3 As shown in the figure, the known attack scenarios mainly include rebound shell (Reverse Shell, also known as reverse shell), file download, vulnerability verification, domain name query, encoding scenario, host command execution, sandbox escape and other attack scenarios, among which, Figure 3 The adjacent content on the right side of each attack scenario is the command for extracting the attack type in the corresponding attack scenario.

[0060] Specifically, Figure 3 As shown in the figure, for rebound Shell attack scenarios, tools and script commands such as icmp and dnscat2 are usually used to extract attack types. Programming language commands such as python, perl, java, ruby, php, golang, and lua are used to extract attack types. System commands such as bash, whois, wget, curl, socat, nc / netcat, exec, telnet, powershell, and xterm are used to extract attack instructions. For domain name query attack scenarios, commands such as nslookup, dig, and ping are usually used to extract attack types. For file download attack scenarios, commands such as wget, curl, and git are usually used to extract attack types. , ftp, tftp, scp and other commands are used to extract the attack type; for vulnerability verification scenarios, rmi, ldap, jdbc, t3, spring and other commands are usually used to extract the attack type; for encoding scenarios, base64, urlencode, hex, unicode and other commands are usually used to extract the attack type; for host command execution attack scenarios, whoami, id, cat, ifconfig, ls, uname, w, pwd, iwconfig, cd, hostname, ping, lsof, env and other commands are usually used to extract the attack type; for sandbox escape attack scenarios, execSync command is usually used to extract the attack type.

[0061] In one implementation, the number of extraction models is one, and the one extraction model is an expert model based on a regular expression of the attack type.

[0062] In another embodiment, there are multiple extraction models, and the multiple extraction models include: an expert model based on a regular expression of an attack type, and one or more general models based on a general regular expression.

[0063] In the present disclosure, a general regular expression is an expression defined based on a general IOC rule, wherein the general IOC rule is determined based on data features of a corresponding IOC, and a general model is used to extract a general threat indicator.

[0064] Among them, multiple extraction models may include an expert model based on regular expressions of attack types and a general model based on general regular expressions; or, multiple extraction models may include an expert model based on regular expressions of attack types and multiple general models based on general regular expressions. When there are multiple general models, the regular expressions used by the multiple general models are different. Different general models focus on different types of threat indicators, and are irrelevant to the fixed patterns or attack scenarios of the data of the threat indicators to be extracted, and can actively extract all IOCs of the corresponding threat indicator types from the data of the threat indicators to be extracted of unknown threats.

[0065] When there are multiple general models, the user can specify which general model or models to use to extract the IOC of the threat indicator data to be extracted. Different general models may use different regular expressions to process the same threat indicator type, representing extraction strategies of different strictness or standards, so that the extraction results of different general models may be different, that is, different general models provide detection capabilities for different threat indicator types.

[0066] For example, Figure 2 As shown, multiple extraction models include an expert model and three general models, namely general model A, general model B and general model C, wherein general model A and general model B can be used for detecting three types of threat indicators, namely IP, Domain and URL, and model C is used for detecting two types of threat indicators, namely IP and URL.

[0067] In the above implementation, the expert model relies on the experience and knowledge of the expert, focusing on the extraction of threat indicators for known or fixed-pattern attack scenarios, achieving rapid detection efficiency and ensuring basic detection results. At the same time, in order to address the problem that regular expressions can usually only match specific text patterns, an additional extractor for general threat indicators (i.e., a general model) is provided to provide a certain degree of flexibility in detection capabilities, and to provide basic detection results for specific threat indicator types (e.g., IP, domain name, URL, etc.) when expert experience has not been accumulated.

[0068] In addition, the above-mentioned expert model is constructed based on regular expressions of multiple attack types, and the expert model may include multiple extraction plug-ins, wherein each extraction plug-in corresponds to a regular expression of at least one attack type, and the regular expressions of attack types corresponding to different extraction plug-ins do not overlap. In this way, IOC extraction can be performed by calling the extraction plug-in, which will not affect the normal execution of the existing execution process, and the extraction plug-in is configurable, which improves the flexibility of the expert model. At this time, the above-mentioned expert model can extract the IOC of the data of the threat indicator to be extracted through the following steps (11) to (15).

[0069] Step (11): preprocess the data of threat indicators to be extracted to obtain text data.

[0070] In the present disclosure, the data of threat indicators to be extracted may be normal and easy-to-read text, or it may be encoded network traffic data. Attackers usually use obfuscation technology to make the data of threat indicators to be extracted bypass basic detection. In addition, the data of threat indicators to be extracted is unpredictable and has a variety of different text forms, which brings more difficulties to the subsequent IOC extraction. Therefore, the present disclosure sets up a data preprocessing link (such as Figure 2 to facilitate subsequent analysis and extraction.

[0071] like Figure 4 As shown in the figure, the preprocessing process of the data to be extracted for threat indicators can include the following three steps:

[0072] ① Data cleaning: mainly removes invalid, redundant and irrelevant data in the data of threat indicators to be extracted to improve data quality. For example, remove redundant data packets in the data of threat indicators to be extracted, fill in missing values, and process outliers.

[0073] ②Data conversion: mainly converting the data obtained after data cleaning into a format suitable for analysis. This includes converting binary and hexadecimal data into text data, or converting unstructured data into structured data. In addition, it may also include data standardization, such as converting all data to the same scale, converting numerical data into text type, etc., to facilitate subsequent analysis and comparison.

[0074] ③ Data decoding: It mainly decodes the encoded data into its original format through a decoder for subsequent analysis. For example, data encoded by base64 or URL encoding can be decoded into regular text data that can be processed by the classification model.

[0075] Step (12): Perform regular matching on the text data according to the regular expression of the attack type.

[0076] Step (13): If there is a matching regular expression in the text data, for each matching regular expression, a target extraction plug-in corresponding to the matching regular expression is determined from multiple extraction plug-ins.

[0077] Step (14): Use the target extraction plug-in to extract the threat indicators in the matching regular expression.

[0078] Step (15): Merge the threat indicators in each matching regular expression to obtain the second extraction result of the expert model.

[0079] In this disclosure, Figure 5 As shown in , regular expressions for multiple attack types used to construct expert models can exist in the form of a regular expression pattern configuration set. Figure 5 As shown, after obtaining text data through data preprocessing, the text data can be compared with a regular expression whitelist to filter out the content in the regular expression whitelist from the text data; then, the filtered text data is subjected to regular matching with the regular expressions of each attack type in the regular expression mode configuration set; if there is a matching regular expression in the filtered text data, then for each matching regular expression (i.e., each successfully matched regular matching result), a target extraction plug-in corresponding to the matched regular expression is determined from multiple extraction plug-ins (i.e., a regular plug-in corresponding to the matched regular expression is determined); then, the regular configuration (i.e., the configuration of the matched regular expression) and the corresponding extraction plug-in (i.e., the target extraction plug-in) are loaded, and the threat indicator in the matched regular expression is extracted using the target extraction plug-in; finally, the threat indicators in each matched regular expression are merged to remove duplication, and a second extraction result of the expert model is obtained, and the result is returned.

[0080] In addition, if Figure 2 and Figure 5 As shown, the expert model also supports adding new regular expressions to the regular expression whitelist, or adding the configuration of the new regular expression to the regular expression mode configuration set to check suspicious data, thereby improving the scalability of the expert model to adapt to the dynamic changes and increasing complexity of network threats. Specifically, the above threat indicator extraction method can also include the following steps:

[0081] In response to receiving a new regular expression for the expert model, determining an extraction plug-in matching the new regular expression from a plurality of extraction plug-ins;

[0082] Establish a correspondence between the matching extraction plug-in and the newly added regular expression.

[0083] In addition, the above classification model can extract threat indicators of the data of threat indicators to be extracted through the following steps (21) to (25).

[0084] Step (21): preprocess the data of threat indicators to be extracted to obtain text data.

[0085] Step (22): Segment the text data into words to obtain a word segmentation sequence.

[0086] In the present disclosure, a heuristic text segmenter (i.e. Figure 6 The Tokenizer shown in ) performs word segmentation on the text data to obtain a word segmentation sequence, wherein the word segmentation sequence includes a subword of a preset type and carries the first identification information, and the length of the subword is less than the length of the word to which it belongs. For example, the words such as URL and command include subwords such as IP and Port.

[0087] In order to adapt to the special form of network traffic load that is different from regular text, additional word segmentation symbols with special meanings (for example, backslash, hyphen, etc.) can be added to the symbol table of the text word segmenter, so that the granularity of word segmentation can match the symbols and language semantics of the network traffic load and the terminal command line, so as to improve the accuracy of word segmentation and thereby improve the accuracy of the first extraction result.

[0088] Step (23): vectorize the word segmentation sequence to obtain a text vector, input the text vector into the classification model, and obtain the classification result and threat indicator type of each word in the word segmentation sequence.

[0089] After the word segmentation is performed by the text segmenter to obtain the word segmentation sequence, the word segmentation sequence can be vectorized to obtain the text vector, and then the text vector is input into the classification model (for example, Figure 6 The BERT model shown in ). The classification model may include a linear classification layer and multiple encoders connected in sequence, and the linear classification layer is connected to the last encoder of the multiple encoders connected in sequence. The encoder is used to capture the complex relationship between the segmented words in the segmented word sequence and generate a high-dimensional vector representation of the text vector; the linear classification layer is used to generate the classification results and threat indicator types of each segmented word in the segmented word sequence according to the high-dimensional vector output by the last encoder. The classification result is used to characterize whether the corresponding segmented word belongs to a threat indicator.

[0090] Step (24): for each segmented word in the segmented word sequence that carries the first identification information, extract the threat indicator in the subword in the segmented word to obtain a third extraction result.

[0091] In the present disclosure, the subwords in each segmentation may be input into a classification model to extract threat indicators in each subword as a third extraction result.

[0092] Step (25): Determine the word segments in the word segmentation sequence that belong to threat indicators and their threat indicator types, as well as the third extraction result as the first extraction result of the classification model.

[0093] After extracting IOCs and their threat indicator types from the data of threat indicators to be extracted through the classification model, and extracting IOCs from the subwords of each segmented word carrying the first identification information in the segmented word sequence, they are merged to obtain the first extraction result of the classification model. In this way, through the classification model, not only can IOCs and their threat indicator types be extracted from each segmented word of the data of threat indicators to be extracted, but IOCs can also be extracted from subwords of preset types in the segmented words, thereby improving the accuracy and comprehensiveness of IOC extraction.

[0094] In addition, after obtaining the first extraction result of the classification model, a labeling module (such as Figure 6 The ner module shown in the figure automatically assigns relevant labels to each threat indicator in the first extraction result according to the characteristics of the threat indicator in the first extraction result (such as type, source, target, etc.). These labels can help security experts and subsequent processes understand and handle these threat indicators faster and more accurately.

[0095] In addition, the above classification model can be trained through steps (1) and (2):

[0096] Step (1): Get a text sample.

[0097] In the present disclosure, real alarm traffic from network full-flow analysis and network application firewall can be used as the main source of training data, and some text content containing threat indicators can be added to enrich the training data; then, the training data is cleaned and labeled using security expert annotations and heuristic regular automatic annotation methods to obtain text samples.

[0098] The text sample may include multiple sample words and labels for each sample word, wherein the label includes the classification result and threat indicator type used to characterize the sample word. When the sample word implies preset content, the text sample also includes the preset content, as well as the classification result and threat indicator type of the preset content. The classification result is used to characterize whether the corresponding sample word or preset content belongs to a threat indicator. The preset content does not exist in the sample word, but is the content implied by the sample word. In order to ensure the accuracy and comprehensiveness of the IOC extracted by the classification model, the preset content, as well as the classification result and threat indicator type of the preset content are added to the text sample.

[0099] For example, the port number is implied in sample words such as URL. For example, URL is an HTTP protocol, and the HTTP protocol has a default port number, such as 80. The port number does not exist in the HTTP protocol, but in order to ensure the accuracy and comprehensiveness of the IOC extracted by the classification model, the default port number is added to the text sample, as well as the classification result and threat indicator type of the default port number.

[0100] Step (2): Based on the text samples, fine-tune the pre-trained model based on machine learning to obtain a classification model.

[0101] During fine-tuning, the cross entropy loss function can be used to measure the difference between the model's prediction results and the true labels, the Adam optimizer can be used to optimize the model's parameters, and the learning rate decay strategy can be used to adjust the learning rate.

[0102] The following is a detailed description of the specific implementation method of generating the target threat indicator of the data to be extracted based on the first extraction result of the classification model and the second extraction result of each extraction model in S103. Specifically, it can be achieved by the following steps (31) and (32):

[0103] Step (31): Aggregate the first extraction result of the classification model and the second extraction result of each extraction model.

[0104] Step (32): Perform format verification on the extraction results obtained after the aggregation processing to filter out extraction results that do not comply with the format and obtain target threat indicators.

[0105] like Figure 2 and Figure 7 As shown, after extracting a threat indicator set (i.e., the first extraction result) from the data of the threat indicator to be extracted through the classification model, and extracting the threat indicator results (i.e., the second extraction result) through the expert model and the general model respectively, the three threat indicator sets are aggregated to remove duplicate threat indicators (i.e., aggregation and deduplication); then, according to the threat indicator type, the entity legitimacy of each IOC in the extraction result obtained after the aggregation process is verified and filtered to filter out IOCs with non-compliant formats and obtain the final result (i.e., the target threat indicator).

[0106] In addition, the above method may further include the following steps:

[0107] If it is determined based on the target threat indicator that threat intelligence exists in the data of the threat indicator to be extracted, threat alarm information is generated.

[0108] After obtaining the target threat indicator of the data of the threat indicator to be extracted, the target threat indicator is matched with the IOC whitelist. If the target threat indicator contains threat indicators outside the IOC whitelist, it is determined that there is threat intelligence in the data of the threat indicator to be extracted. At this time, threat alarm information can be generated, which can serve as a warning for users to protect against threats in a timely manner.

[0109] The present disclosure also provides an alarm data processing system, such as Figure 8 As shown, the system includes a traffic automatic analysis module and a data association analysis module, wherein the traffic automatic analysis module includes a data preprocessing module and a network traffic data intelligent analysis module. The data preprocessing module is used to filter out some common false alarm data or alarm data of irrelevant business scenarios from the alarm data to be processed. At this time, the data of the threat indicators to be extracted are alarm data. The network traffic data intelligent analysis module mainly distinguishes the alarm data based on knowledge scenarios and natural language technology, automatically identifies IOC information in common high-risk scenarios and provides a unified interface service for structured processing of the extracted threat indicators to obtain structured threat indicators (i.e. Figure 8 The formatted alarm data in the network flow data is used by the data correlation analysis module. The network traffic data intelligent analysis module is the most important and basic link to improve the accuracy of alarm analysis and operation efficiency. After the IOC is identified, the identified IOC can be matched with the IOC whitelist, and the IOCs outside the IOC whitelist can be identified as threat intelligence.

[0110] like Figure 8 As shown, multiple alarm data to be processed can be obtained from the Transmit Load Balancing (TLB) alarm log message queue and the Network Traffic Analysis (NTA) alarm log message queue.

[0111] The data association analysis module is used to associate the Domain Name System (DNS) data, session data, intelligence data and Host-based IDS (HIDS) data (hereinafter referred to as host data) of each structured threat indicator to associate related historical data, thereby obtaining multiple suspected risk information, and conducting alarm analysis based on multiple suspected risk information to identify real risk information, which is then provided to the Security Information and Event Management (SIEM) / Security Orchestration Automation and Response (SOAR) operation platform to generate security work orders based on the real risk information.

[0112] Fig. 9 FIG. 1 is a block diagram of a threat indicator extraction device according to an exemplary embodiment. Fig. 9 As shown, the device 200 includes:

[0113] A first acquisition module 201 is used to acquire data of threat indicators to be extracted;

[0114] An extraction module 202, configured to extract threat indicators of the data using a classification model based on machine learning and at least one extraction model based on regular expressions;

[0115] The generating module 203 is used to generate a target threat indicator of the data according to the first extraction result of the classification model and the second extraction result of each of the extraction models.

[0116] In the above technical solution, firstly, a classification model based on machine learning and at least one extraction model based on regular expressions are used to extract the IOC of the data of the threat indicators to be extracted respectively; then, according to the first extraction result of the classification model and the second extraction result of each extraction model, the target threat indicator of the data of the threat indicators to be extracted is generated. In this way, the machine learning technology and the regular expression technology can be combined to form a multi-level and multi-angle IOC extraction system. On the one hand, the extraction system can accurately extract the indicators of known threats based on known threat features and patterns, and on the other hand, it can effectively identify and extract the indicators of new and unknown threats without relying on known patterns. In this way, it can not only effectively extract various types of IOC, but also adapt to the dynamic changes and increasingly complex trends of network threats, thereby improving the comprehensiveness and foresight of network security protection. In addition, the classification model based on machine learning can automatically learn and extract IOC, which can reduce the dependence on manually writing and maintaining regular expressions, thereby reducing manpower and time costs while improving the efficiency and accuracy of IOC extraction.

[0117] Optionally, the at least one regular expression-based extraction model comprises an expert model based on regular expressions of attack types.

[0118] Optionally, there are multiple extraction models, and the multiple extraction models further include one or more general models based on a general regular expression, wherein when there are multiple general models, the regular expressions adopted by the multiple general models are different.

[0119] Optionally, the expert model includes a plurality of extraction plug-ins, each of which corresponds to a regular expression of at least one of the attack types;

[0120] The extraction module 202 includes:

[0121] A preprocessing submodule, used for performing data preprocessing on the data to obtain text data;

[0122] A regular matching submodule, used for performing regular matching on the text data according to the regular expression of the attack type;

[0123] A first determination submodule, for determining, for each of the matched regular expressions, a target extraction plug-in corresponding to the matched regular expression from the plurality of extraction plug-ins if there is a matched regular expression in the text data; a first extraction submodule, for extracting threat indicators from the matched regular expression using the target extraction plug-in;

[0124] The merging submodule is used to merge the threat indicators in each of the matched regular expressions to obtain the second extraction result of the expert model.

[0125] Optionally, the device 200 further includes:

[0126] a determination module, configured to, in response to receiving a new regular expression for the expert model, determine an extraction plug-in matching the new regular expression from the multiple extraction plug-ins;

[0127] An establishing module is used to establish a corresponding relationship between the matching extraction plug-in and the newly added regular expression.

[0128] Optionally, the extraction module includes:

[0129] A preprocessing submodule, used for performing data preprocessing on the data to obtain text data;

[0130] A word segmentation submodule, configured to segment the text data to obtain a word segmentation sequence, wherein a word segmentation sequence containing a subword of a preset type carries first identification information, and a length of the subword is smaller than a length of the word segmentation to which it belongs;

[0131] A vectorization submodule, used for performing vectorization processing on the segmented word sequence to obtain a text vector, inputting the text vector into the classification model, obtaining a classification result and a threat indicator type of each segmented word in the segmented word sequence, wherein the classification result is used to characterize whether the corresponding segmented word belongs to a threat indicator;

[0132] A second extraction submodule is configured to extract the threat indicator in the subword in each segmented word carrying the first identification information in the segmented word sequence, to obtain a third extraction result;

[0133] The second determination submodule is used to determine the segmented words belonging to threat indicators and their threat indicator types in the segmented word sequence, and the third extraction result as the first extraction result of the classification model.

[0134] Optionally, the classification model is obtained by training through a model training device, wherein the model training device comprises:

[0135] A second acquisition module is used to acquire a text sample, wherein the text sample includes a plurality of sample words and a label of each of the sample words, the label includes a classification result and a threat indicator type used to characterize the sample word, and when the sample word implies a preset content, the text sample also includes the preset content, and a classification result and a threat indicator type of the preset content, wherein the classification result is used to characterize whether the corresponding sample word or the preset content belongs to a threat indicator;

[0136] The fine-tuning module is used to fine-tune the pre-trained model based on machine learning according to the text sample to obtain the classification model.

[0137] Optionally, the classification model is a deep learning model or a language model.

[0138] Optionally, the generating module 203 includes:

[0139] An aggregation processing submodule, used for performing aggregation processing on the first extraction result of the classification model and the second extraction result of each of the extraction models;

[0140] The format check submodule is used to perform format check on the extraction results obtained after the aggregation processing to filter out the extraction results that are not in compliance with the format and obtain the target threat indicators.

[0141] Optionally, the data is network traffic data or network text data.

[0142] Optionally, the device 200 further includes:

[0143] The alarm module is used to generate threat alarm information if it is determined that there is threat intelligence in the data of the threat indicator to be extracted according to the target threat indicator.

[0144] The present disclosure also provides a computer-readable medium on which a computer program is stored. When the program is executed by a processing device, the steps of the above-mentioned threat indicator extraction method provided by the present disclosure are implemented.

[0145] Reference below Fig.10 , which shows a schematic diagram of the structure of an electronic device (such as a terminal device or a server) 600 suitable for implementing the embodiment of the present disclosure. The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.10 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0146] like Fig.10As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0147] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.10 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0148] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0149] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0150] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0151] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0152] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: obtains data of threat indicators to be extracted; uses a classification model based on machine learning and at least one extraction model based on regular expressions to respectively extract threat indicators of the data of threat indicators to be extracted; and generates target threat indicators of the data according to a first extraction result of the classification model and a second extraction result of each of the extraction models.

[0153] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0155] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a module does not limit the module itself in some cases. For example, the first acquisition module may also be described as a "module for acquiring data of threat indicators to be extracted."

[0156] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0157] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] According to one or more embodiments of the present disclosure, Example 1 provides a threat indicator extraction method, including:

[0159] Obtain data on threat indicators to be extracted;

[0160] Using a classification model based on machine learning and at least one extraction model based on regular expressions, respectively extract threat indicators of the data of threat indicators to be extracted;

[0161] A target threat indicator of the data is generated according to the first extraction result of the classification model and the second extraction result of each of the extraction models.

[0162] According to one or more embodiments of the present disclosure, Example 2 provides the method of Example 1, wherein the at least one regular expression-based extraction model includes an expert model based on regular expressions of attack types.

[0163] According to one or more embodiments of the present disclosure, Example 3 provides the method of Example 2, wherein there are multiple extraction models, and the multiple extraction models also include one or more general models based on general regular expressions, wherein when there are multiple general models, the regular expressions used by the multiple general models are different.

[0164] According to one or more embodiments of the present disclosure, Example 4 provides the method of Example 2 or Example 3, wherein the expert model includes a plurality of extraction plug-ins, each of which corresponds to a regular expression of at least one of the attack types;

[0165] The expert model extracts the threat indicators of the data of the threat indicators to be extracted in the following manner:

[0166] Performing data preprocessing on the data to obtain text data;

[0167] According to the regular expression of the attack type, regular matching is performed on the text data;

[0168] If there is a matching regular expression in the text data, for each matching regular expression, determine a target extraction plug-in corresponding to the matching regular expression from the multiple extraction plug-ins; and extract the threat indicator in the matching regular expression using the target extraction plug-in;

[0169] The threat indicators in each of the matched regular expressions are merged to obtain the second extraction result of the expert model.

[0170] According to one or more embodiments of the present disclosure, Example 5 provides the method of Example 4, wherein the method further includes:

[0171] In response to receiving a new regular expression for the expert model, determining an extraction plug-in matching the new regular expression from the multiple extraction plug-ins;

[0172] A correspondence between the matching extraction plug-in and the newly added regular expression is established.

[0173] According to one or more embodiments of the present disclosure, Example 6 provides the method of Example 1, wherein the classification model extracts threat indicators of the data in the following manner:

[0174] Performing data preprocessing on the data to obtain text data;

[0175] Segmenting the text data to obtain a segmentation sequence, wherein a segmentation containing a subword of a preset type in the segmentation sequence carries first identification information, and a length of the subword is smaller than a length of the segmentation to which it belongs;

[0176] Performing vectorization processing on the word segmentation sequence to obtain a text vector, inputting the text vector into the classification model to obtain a classification result and a threat indicator type of each word segmentation in the word segmentation sequence, wherein the classification result is used to characterize whether the corresponding word segmentation belongs to a threat indicator;

[0177] For each segmented word in the segmented word sequence that carries the first identification information, extract the threat indicator in the subword in the segmented word to obtain a third extraction result;

[0178] The segmented words belonging to threat indicators and their threat indicator types in the segmented word sequence, and the third extraction result are determined as the first extraction result of the classification model.

[0179] According to one or more embodiments of the present disclosure, Example 7 provides the method of Example 1, wherein the classification model is trained in the following manner:

[0180] Acquire a text sample, wherein the text sample includes a plurality of sample words and a label of each of the sample words, wherein the label includes a classification result and a threat indicator type used to characterize the sample word, and when the sample word implies a preset content, the text sample also includes the preset content, and a classification result and a threat indicator type of the preset content, wherein the classification result is used to characterize whether the corresponding sample word or the preset content belongs to a threat indicator;

[0181] According to the text sample, the pre-trained model based on machine learning is fine-tuned to obtain the classification model.

[0182] According to one or more embodiments of the present disclosure, Example 8 provides the method of Example 1, wherein the classification model is a deep learning model or a language model.

[0183] According to one or more embodiments of the present disclosure, Example 9 provides the method of Example 1, wherein generating a target threat indicator of the data according to the first extraction result of the classification model and the second extraction result of each of the extraction models includes:

[0184] Aggregating the first extraction result of the classification model and the second extraction result of each of the extraction models;

[0185] The extraction results obtained after the aggregation processing are format checked to filter out extraction results with non-compliant formats and obtain the target threat indicators.

[0186] According to one or more embodiments of the present disclosure, Example 10 provides the method described in any one of Examples 1-3 and 5-9, wherein the data is network traffic data or network text data.

[0187] According to one or more embodiments of the present disclosure, Example 11 provides the method of any one of Examples 1-3 and 5-9, wherein the method further includes:

[0188] If it is determined according to the target threat indicator that threat intelligence exists in the data, threat warning information is generated.

[0189] According to one or more embodiments of the present disclosure, Example 12 provides a threat indicator extraction device, including:

[0190] A first acquisition module is used to acquire data of threat indicators to be extracted;

[0191] an extraction module, configured to respectively extract threat indicators of the data using a classification model based on machine learning and at least one extraction model based on regular expressions;

[0192] A generating module is used to generate a target threat indicator of the data according to the first extraction result of the classification model and the second extraction result of each of the extraction models.

[0193] According to one or more embodiments of the present disclosure, Example 13 provides a computer-readable medium having a computer program stored thereon, which implements the steps of any of the methods described in Examples 1-11 when executed by a processing device.

[0194] According to one or more embodiments of the present disclosure, Example 14 provides an electronic device, including:

[0195] a storage device having a computer program stored thereon;

[0196] A processing device is used to execute the computer program in the storage device to implement the steps of the method described in any one of Examples 1-11.

[0197] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0198] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0199] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. A threat indicator extraction method, characterized in that: include: Obtain data on threat indicators to be extracted; Utilizing a classification model based on machine learning and a plurality of extraction models based on regular expressions, respectively extracting threat indicators of the data, wherein the plurality of extraction models based on regular expressions include: an expert model based on regular expressions of attack types, and one or more general models based on general regular expressions, when there are multiple general models, the regular expressions adopted by the plurality of general models are different, the regular expressions based on attack types are regular expressions defined based on threat indicator rules related to known attack types, the general regular expressions are expressions defined based on general threat indicator rules, the general threat indicator rules are determined based on data features of corresponding threat indicators, and the general models are used to extract general threat indicators; generating a target threat indicator of the data according to the first extraction result of the classification model and the second extraction result of each of the extraction models; Wherein, the expert model includes a plurality of extraction plug-ins, each of which corresponds to a regular expression of at least one of the attack types; The expert model extracts threat indicators from the data in the following ways: Performing data preprocessing on the data to obtain text data; According to the regular expression of the attack type, regular matching is performed on the text data; If there is a matching regular expression in the text data, for each matching regular expression, determine a target extraction plug-in corresponding to the matching regular expression from the multiple extraction plug-ins; and extract the threat indicator in the matching regular expression using the target extraction plug-in; Merging the threat indicators in each of the matched regular expressions to obtain the second extraction result of the expert model; The classification model extracts threat indicators from the data in the following manner: Performing data preprocessing on the data to obtain text data; Segmenting the text data to obtain a segmentation sequence, wherein a segmentation containing a subword of a preset type in the segmentation sequence carries first identification information, and a length of the subword is smaller than a length of the segmentation to which it belongs; Vectorizing the word segmentation sequence to obtain a text vector, inputting the text vector into the classification model to obtain a classification result and a threat indicator type of each word segmentation in the word segmentation sequence, wherein the classification result of each word segmentation in the word segmentation sequence is used to characterize whether the corresponding word segmentation belongs to a threat indicator; For each segmented word in the segmented word sequence that carries the first identification information, extract the threat indicator in the subword in the segmented word to obtain a third extraction result; The segmented words belonging to threat indicators and their threat indicator types in the segmented word sequence, and the third extraction result are determined as the first extraction result of the classification model.

2. The method according to claim 1, characterized in that The method further comprises: In response to receiving a new regular expression for the expert model, determining an extraction plug-in matching the new regular expression from the multiple extraction plug-ins; A correspondence between the matching extraction plug-in and the newly added regular expression is established.

3. The method according to claim 1, characterized in that The classification model is trained in the following way: Acquire a text sample, wherein the text sample includes a plurality of sample words and a label of each of the sample words, wherein the label includes a classification result and a threat indicator type for characterizing the sample word, and when the sample word implies a preset content, the text sample also includes the preset content, and a classification result and a threat indicator type of the preset content, wherein the classification result of the sample word is used to characterize whether the sample word belongs to a threat indicator, and the classification result of the preset content is used to characterize whether the preset content belongs to a threat indicator; According to the text sample, the pre-trained model based on machine learning is fine-tuned to obtain the classification model.

4. The method according to claim 1, characterized in that: The classification model is a deep learning model or a language model.

5. The method according to claim 1, characterized in that Generating a target threat indicator of the data according to the first extraction result of the classification model and the second extraction result of each of the extraction models includes: Aggregating the first extraction result of the classification model and the second extraction result of each of the extraction models; The extraction results obtained after the aggregation processing are format checked to filter out extraction results with non-compliant formats and obtain the target threat indicators.

6. The method according to any one of claims 1 to 5, characterized in that The data is network traffic data or network text data.

7. The method according to any one of claims 1 to 5, characterized in that The method further comprises: If it is determined according to the target threat indicator that threat intelligence exists in the data, threat warning information is generated.

8. A threat indicator extraction device, characterized in that: include: A first acquisition module is used to acquire data of threat indicators to be extracted; An extraction module, used to respectively extract threat indicators of the data using a classification model based on machine learning and a plurality of extraction models based on regular expressions, wherein the plurality of extraction models based on regular expressions include: an expert model based on regular expressions of attack types, and one or more general models based on general regular expressions, when there are multiple general models, the regular expressions used by the multiple general models are different, the regular expressions based on attack types are regular expressions defined based on threat indicator rules related to known attack types, the general regular expressions are expressions defined based on general threat indicator rules, the general threat indicator rules are determined based on data features of corresponding threat indicators, and the general model is used to extract general threat indicators; A generating module, configured to generate a target threat indicator of the data according to the first extraction result of the classification model and the second extraction result of each of the extraction models; Wherein, the expert model includes a plurality of extraction plug-ins, each of which corresponds to a regular expression of at least one of the attack types; The extraction module comprises: A preprocessing submodule, used for performing data preprocessing on the data to obtain text data; A regular matching submodule, used for performing regular matching on the text data according to the regular expression of the attack type; A first determination submodule, for determining, for each of the matched regular expressions, a target extraction plug-in corresponding to the matched regular expression from the plurality of extraction plug-ins if there is a matched regular expression in the text data; a first extraction submodule, for extracting threat indicators from the matched regular expression using the target extraction plug-in; A merging submodule, used for merging the threat indicators in each of the matched regular expressions to obtain the second extraction result of the expert model; The extraction module also includes: A word segmentation submodule, configured to segment the text data to obtain a word segmentation sequence, wherein a word segmentation sequence containing a subword of a preset type carries first identification information, and a length of the subword is smaller than a length of the word segmentation to which it belongs; A vectorization submodule, used for performing vectorization processing on the segmented word sequence to obtain a text vector, inputting the text vector into the classification model, and obtaining a classification result and a threat indicator type of each segmented word in the segmented word sequence, wherein the classification result of each segmented word in the segmented word sequence is used to characterize whether the corresponding segmented word belongs to a threat indicator; A second extraction submodule is configured to extract the threat indicator in the subword in each segmented word carrying the first identification information in the segmented word sequence, to obtain a third extraction result; The second determination submodule is used to determine the segmented words belonging to threat indicators and their threat indicator types in the segmented word sequence, and the third extraction result as the first extraction result of the classification model.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processing device, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a storage device having a computer program stored thereon; A processing device, configured to execute the computer program in the storage device to implement the steps of the method according to any one of claims 1 to 7.

Images