High-value threat intelligence mining method based on feature arrangement and data fusion

Abstract

The invention relates to the technical field of network information security, and discloses a high-value threat intelligence mining method and device based on feature arrangement and data fusion, electronic equipment, a computer readable storage medium and a computer program product. The method and the device are used for solving the technical problem that potential high-value threat intelligence cannot be quickly and accurately mined from massive log data in the prior art. The method comprises the steps of obtaining threat logs and performing screening and aggregation according to basic attributes to obtain threat data; key data features to be arranged are extracted from the threat data; performing arrangement based on a feature arrangement rule, and performing calculation based on an arrangement result to obtain a suspicious host sequence of an attacker angle and a victim angle; obtaining a fusion database, fusing suspicious host information in the suspicious host sequence with intelligence information in the fusion database from an attacker angle and a victim angle, and calculating the value degree of each suspicious host; and sorting the suspicious hosts according to the value degree and generating high-value threat intelligence.

Description

Technical Field

[0001] The present invention relates to the field of network information security technology, and in particular to a high-value threat intelligence mining method, device, electronic device, computer storage medium and computer program product based on feature arrangement and data fusion. Background Art

[0002] With the rapid development of Internet technology and information technology, cyberspace has gradually become an indispensable part of people's production and life, but the resulting cyberspace security issues are becoming increasingly important. Traditional methods such as firewalls, intrusion detection systems, and intrusion prevention systems can capture suspicious network attack data in previous application scenarios. However, with the popularization of big data technology in recent years, the number of threat logs generated by security protection systems has also increased exponentially, resulting in real attacks being overwhelmed by a large amount of false positive data, making it impossible for security personnel to identify real threat events in a timely and effective manner.

[0003] To address this problem, most existing methods use rule matching and machine learning methods to mine effective attacks from threat logs. The rule matching method mainly extracts common features from known attack event intelligence for identification. Although it is simple, efficient and easy to implement, it is difficult to detect unknown and complex attacks, and the extraction of common features requires rich expert experience, with a high false negative rate. The machine learning method mainly uses a data-driven approach to identify suspicious intelligence data from threat logs. Although this method avoids reliance on expert experience, it requires a large amount of labeled data to train the model, and the model's interpretability and real-time performance are poor. Therefore, how to quickly and accurately mine potential high-value threat intelligence from massive log data has become a key issue that needs to be urgently addressed in the current security field. Summary of the invention

[0004] The main purpose of the present invention is to solve the technical problem in the prior art that it is impossible to quickly and accurately mine potential high-value threat intelligence from massive log data.

[0005] The first aspect of the present invention provides a high-value threat intelligence mining method based on feature arrangement and data fusion, comprising:

[0006] Obtain threat logs, filter and aggregate threat logs according to basic attributes, and obtain aggregated threat data;

[0007] Extracting key data features to be compiled from the threat data from the attacker's perspective and the victim's perspective respectively;

[0008] Arrange the key data features based on feature arrangement rules, and calculate suspicious host sequences from the attacker's perspective and the victim's perspective based on the arrangement results;

[0009] Obtaining a fusion database, and fusing the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the perspective of the attacker and the perspective of the victim, and then calculating the value of each suspicious host;

[0010] The suspicious hosts are sorted according to the values, and high-value threat intelligence is generated based on the sorting results.

[0011] Optionally, in a first implementation of the first aspect of the present invention, the feature arrangement rule includes feature serial arrangement and feature parallel arrangement, and the target host sequence includes a suspicious host sequence under serial arrangement and a suspicious host sequence under parallel arrangement;

[0012] The key data features are arranged based on feature arrangement rules, and suspicious host sequences from the attacker's perspective and the victim's perspective are calculated based on the arrangement results, including:

[0013] Randomly select two key data features from all the key features of the threat data and arrange them in series to obtain multiple series arrangement models. Under each series arrangement model, sort them according to the arrangement results to obtain suspicious hosts from the attacker's perspective and the victim's perspective under the series arrangement.

[0014] All key features and basic attributes of threat data are used as sorting criteria to separately arrange multiple parallel arrangement models. In each serial arrangement model, the arrangement results are sorted to obtain suspicious hosts from the attacker's perspective and the victim's perspective under parallel arrangement.

[0015] Based on the weighted fusion algorithm, weighted fusion calculations are performed on the suspicious hosts under serial arrangement and the suspicious hosts under parallel arrangement to obtain the suspicious host sequences from the attacker's perspective and the victim's perspective respectively.

[0016] Optionally, in a second implementation of the first aspect of the present invention,

[0017] The fusion database includes a detection source database, an intelligence source database and an event source database;

[0018] The step of obtaining the fusion database and fusing the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the perspective of the attacker and the perspective of the victim to calculate the value of each suspicious host includes:

[0019] Obtaining the manufacturer information of the security protection system in the detection source database, fusing the manufacturer information of the security protection system with the suspicious host information in the suspicious host sequence, and extracting the associated manufacturer information to obtain a first fusion result;

[0020] Acquire threat intelligence information from an intelligence source database, fuse the threat intelligence information with the suspicious host information in the suspicious host sequence, and determine whether each of the suspicious hosts is a malicious host to obtain a second fusion result;

[0021] Acquire historical event intelligence in an event source database, fuse the historical event intelligence with the suspicious host information in the suspicious host sequence, determine whether there is a host with the same event intelligence, and obtain a third fusion result;

[0022] Based on the first fusion result, the second fusion result and the third fusion result, the value of each suspicious host is calculated from the perspective of the attacker and the perspective of the victim respectively.

[0023] Optionally, in a third implementation of the first aspect of the present invention,

[0024] The suspicious host can be marked by the host's IP address;

[0025] The calculation expression for weighted fusion calculation of suspicious hosts in series arrangement and suspicious hosts in parallel arrangement is:

[0026]

[0027] Where x represents the IP address of the host; A represents the total number of serial orchestration models, and B represents the total number of parallel orchestration models; α i and β i Respectively represent the weights of the i-th series arrangement model and the parallel arrangement model; s ix Indicates whether the host with IP address x is in the serial model s i in;p ix Represents a parallel model, which is similar to the series assignment; C x Indicates the final suspiciousness of the host with IP address x; dec indicates descending order; Top N It indicates the first N host IP addresses in descending order; R indicates the final suspicious host IP address.

[0028] Optionally, in a fourth implementation of the first aspect of the present invention, when calculating the value of each suspicious host, the calculation expression is:

[0029]

[0030] Wherein, x represents the IP address of the host; C x Indicates the suspiciousness of the host with IP address x output in the feature arrangement module; n indicates the number of fusion databases; δ i Respectively represent the credibility weight of the i-th fusion database; l ixrepresents the fusion result of the host with IP address x in the fusion database l; V x Indicates the final value of the host with IP address x.

[0031] Optionally, in a fifth implementation of the first aspect of the present invention, after sorting the suspicious hosts according to the value and generating high-value threat intelligence based on the sorting result, the method further includes:

[0032] The high-value threat intelligence is analyzed and judged, and the weights of each orchestration model and the weights of each fusion database are adjusted based on the suspicious host information included in the analysis and judgment results.

[0033] A second aspect of the present invention provides a high-value threat intelligence mining device based on feature arrangement and data fusion, comprising:

[0034] A screening and aggregation module is used to obtain threat logs and screen and aggregate the threat logs according to basic attributes to obtain aggregated threat data;

[0035] A feature extraction module, used to extract key data features to be compiled from the threat data from the perspective of the attacker and the perspective of the victim respectively;

[0036] A feature arrangement module, used to arrange the key data features based on feature arrangement rules, and calculate suspicious host sequences from the attacker's perspective and the victim's perspective based on the arrangement results;

[0037] An information fusion module is used to obtain a fusion database, and to fuse the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the perspective of the attacker and the perspective of the victim, and then calculate the value of each suspicious host;

[0038] The intelligence generation module is used to sort the suspicious hosts according to the value and generate high-value threat intelligence based on the sorting results.

[0039] The third aspect of the present invention provides a high-value threat intelligence mining device based on feature orchestration and data fusion, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the high-value threat intelligence mining device based on feature orchestration and data fusion performs the steps of the above-mentioned high-value threat intelligence mining method based on feature orchestration and data fusion.

[0040] A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, the computer executes the steps of the above-mentioned high-value threat intelligence mining method based on feature arrangement and data fusion.

[0041] A fifth aspect of the present invention provides a computer program product, including a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the above-mentioned high-value threat intelligence mining method based on feature orchestration and data fusion are implemented.

[0042] In the technical solution provided by the present invention, threat logs are obtained, and the threat logs are screened and aggregated according to basic attributes to obtain aggregated threat data; key data features to be arranged are extracted from the threat data from the attacker's perspective and the victim's perspective respectively; key data features are arranged based on feature arrangement rules, and suspicious host sequences from the attacker's perspective and the victim's perspective are calculated based on the arrangement results; a fusion database is obtained, and the suspicious host information in the suspicious host sequence is fused with the intelligence information in the fusion database from the attacker's perspective and the victim's perspective respectively to calculate the value of each suspicious host; suspicious hosts are sorted according to the value, and high-value threat intelligence is generated based on the sorting results. This method can be based on feature arrangement of the information contained in the threat log, and the arranged feature information is fused according to the intelligence information in the fusion database, so as to quickly and accurately mine potential high-value intelligence in massive threat log data. A device, electronic device, computer-readable storage medium and computer program product provided by the present invention also solve corresponding technical problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0044] Figure 1 It is a flowchart of a first embodiment of a method for mining high-value threat intelligence based on feature arrangement and data fusion in an embodiment of the present invention;

[0045] Figure 2 It is a flow chart of a second embodiment of a high-value threat intelligence mining method based on feature arrangement and data fusion in an embodiment of the present invention;

[0046] Figure 3 It is another flowchart diagram of a second embodiment of a high-value threat intelligence mining method based on feature arrangement and data fusion in an embodiment of the present invention;

[0047] Figure 4 A schematic diagram of an embodiment of a high-value threat intelligence device based on feature arrangement and data fusion in an embodiment of the present invention;

[0048] Figure 5It is a schematic diagram of another embodiment of a high-value threat intelligence device based on feature arrangement and data fusion in an embodiment of the present invention;

[0049] Figure 6 A schematic diagram of an embodiment of a high-value threat intelligence device based on feature arrangement and data fusion in an embodiment of the present invention;

[0050] Figure 7 The figure is a schematic diagram of a computer-readable medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] Exemplary embodiments of the present invention will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in a variety of forms, and should not be construed as limiting the present invention to the embodiments set forth herein. On the contrary, providing these exemplary embodiments enables the present invention to be more comprehensive and complete, and is more convenient for fully conveying the inventive concept to those skilled in the art. The same reference numerals in the figures represent the same or similar elements, components or parts, and thus their repeated description will be omitted.

[0052] Under the premise of being consistent with the technical concept of the present invention, the features, structures, characteristics or other details described in a specific embodiment do not exclude that they can be combined in one or more other embodiments in a suitable manner.

[0053] In the description of specific embodiments, the features, structures, characteristics or other details described in the present invention are intended to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can practice the technical solutions of the present invention without one or more of the specific features, structures, characteristics or other details.

[0054] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0055] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0056] The term "and / or" or "and / or" includes all combinations of any one or more of the associated listed items.

[0057] See also Figure 1A first embodiment of a high-value threat intelligence mining method based on feature arrangement and data fusion in an embodiment of the present invention includes:

[0058] S101, acquiring threat logs, and filtering and aggregating the threat logs according to basic attributes to obtain aggregated threat data;

[0059] It is understandable that the execution subject of the present invention can be a high-value threat intelligence mining device based on feature arrangement and data fusion, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking the server as the execution subject as an example.

[0060] After receiving a high-value threat intelligence mining request, the server first obtains threat logs, where the threat logs mainly come from attack data captured by security protection systems of different manufacturers, and uses the attack data as the original threat logs for data analysis.

[0061] In a specific implementation, after acquiring the threat log, the threat log data is also preprocessed to remove generalized noise logs, such as logs with missing key field values, repeated logs, data format errors, and logs containing whitelists, so as to improve data quality and reduce the impact of invalid logs.

[0062] After obtaining the pre-processed threat logs, the threat logs are filtered and aggregated according to basic attributes. Specifically, based on the log threat level and type, information logs and non-threatening system basic logs are filtered, and then the filtered threat log data is classified and aggregated according to basic attributes to obtain aggregated threat data.

[0063] S102, extracting key data features to be compiled from the threat data from the attacker's perspective and the victim's perspective respectively;

[0064] After obtaining the aggregated threat data in the above steps, key fields are extracted from the attacker's perspective and the victim's perspective respectively, and numerical conversion or data statistics are called to extract key data features to be compiled from the threat data.

[0065] S103, arranging the key data features based on the feature arrangement rules, and calculating the suspicious host sequences from the attacker's perspective and the victim's perspective based on the arrangement results;

[0066] Based on the feature selection result, each feature is arranged in series or in parallel according to its importance, and the suspicious degree of the IP address (Internet Protocol Address) of each attacking host or victim host is sorted from high to low according to the arrangement mode. When recording the host information, the host IP address can be recorded.

[0067] Subsequently, the top N suspicious hosts are extracted for each feature orchestration result, and all suspicious hosts are weightedly fused according to the orchestration model weights to output the top N most suspicious hosts.

[0068] S104, obtaining a fusion database, fusing the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the perspective of the attacker and the perspective of the victim, and then calculating the value of each suspicious host;

[0069] A fusion database is obtained, wherein the fusion database may be constructed before obtaining the threat log, and includes data information from the perspectives of detection source, intelligence source, event source, etc.

[0070] In this step, the top N most suspicious hosts outputted in the previous step are traversed, and multi-dimensional data from the perspectives of detection source, intelligence source, event source, etc. are obtained from the fusion database. From the perspectives of attackers and victims, the information of the top N most suspicious hosts is fused with the multi-dimensional data from the perspectives of detection source, intelligence source, event source, etc. in the fusion database. The value of each suspicious host is calculated based on the fusion result, so as to filter the top N most suspicious hosts and screen out the false positive hosts.

[0071] S105. Sort suspicious hosts according to their value, and generate high-value threat intelligence based on the sorting results.

[0072] According to the value information of each suspicious host after data fusion, high-value suspicious hosts are selected from the perspectives of attackers and victims, and a final high-value threat intelligence report is generated, wherein the high-value threat intelligence report may include the attacking host IP address and all destination host IP addresses and corresponding ports associated with the attacking host IP address, all threat types, transmission protocols, number of logs, hit attack stages, key log names and other information; while high-value victim intelligence mainly includes the victim host IP address and all source IP addresses and corresponding ports associated with the victim host IP address, all threat types, transmission protocols, number of logs, hit attack stages, key log names and other information.

[0073] The method in the embodiment of the present invention can perform feature arrangement based on the information contained in the threat log, and fuse the arranged feature information according to the intelligence information in the fusion database, so as to quickly and accurately mine potential high-value intelligence from massive threat log data.

[0074] Please see Figure 2-3 A second embodiment of a high-value threat intelligence mining method based on feature arrangement and data fusion in an embodiment of the present invention includes:

[0075] S201, obtaining threat logs, and filtering and aggregating the threat logs according to basic attributes to obtain aggregated threat data;

[0076] Please refer to the content of step S101 in the above embodiment. Similarly, this embodiment is described by taking the server as the execution subject. After receiving the high-value threat intelligence mining request, the server first obtains the threat log, wherein the threat log mainly comes from the attack data captured by the security protection systems of different manufacturers, and uses the attack data as the original threat log for data analysis.

[0077] In a specific implementation, since threat logs mainly come from attack data captured by security protection systems of different manufacturers, and considering that the threat log formats generated by protection systems of different manufacturers are different, in order to facilitate subsequent feature selection and data analysis, the threat logs of different manufacturers are generalized into a unified format and stored in the Elastic Search database as the original threat logs for data analysis.

[0078] In a specific implementation, after obtaining the original threat log, the original threat log data is also preprocessed, specifically including eliminating generalized noise logs, such as logs with missing key field values, repeated logs, data format errors, and logs containing whitelists, so as to improve data quality and reduce the impact of invalid logs.

[0079] In a specific implementation, after obtaining the preprocessed threat log, the log is screened and filtered based on the log threat level and type, filtering information logs or non-threatening system basic logs; then the screened log data is aggregated according to 9 basic attributes such as log name, log type, log level, source IP address, destination IP address, source port, destination port, transmission protocol, and log source to obtain aggregated threat data, so as to reduce the amount of subsequent feature analysis calculations.

[0080] S202, extracting key data features to be compiled from the threat data from the attacker's perspective and the victim's perspective respectively;

[0081] In this embodiment, before extracting specific data features, it also includes combining historical expert log analysis experience to extract key fields from the perspective of attackers and victims. After obtaining alternative key fields, key data features to be compiled are extracted based on numerical conversion or statistics.

[0082] In a specific implementation, extracting key data features to be compiled includes calculating 10 pieces of information such as the number of IP addresses associated with each IP address, the number of threat types, the number of log names, the number of logs, the number of ports, the number of protocols, the number of attack stages hit, the number of key types hit, and the number of key log names hit from the perspectives of attackers and victims as key data features.

[0083] S203, randomly selecting two key data features from all the key features of the threat data and arranging them in series to obtain multiple series arrangement models, sorting them according to the arrangement results under each series arrangement model, and obtaining suspicious hosts from the attacker's perspective and the victim's perspective under the series arrangement respectively;

[0084] See also Figure 3 In a specific implementation of this embodiment, after obtaining the key data features, the features will be arranged in series and in parallel respectively. This step specifically describes the specific scheme of the series arrangement. Specifically, the series arrangement mainly mines suspicious hosts through the correlation between different features, and randomly selects two key data features from all the key features to arrange them in series. Specifically, the series arrangement refers to first sorting according to one of the two key data features randomly selected as the sorting index, and then sorting the key data features with equal rankings again according to the other key data feature of the two key data features as the sorting index, to obtain multiple series arrangement models.

[0085] In a specific implementation, when the key data features include 10 categories, randomly selecting two of them for sorting can generate 90 different serial arrangement models. For example, when selecting the number of threat types and the number of logs, the host IP addresses are first sorted in reverse order by the number of threat types, and then sorted in reverse order by the number of logs based on the sorting, and the top several IP addresses are extracted from the final sorting results of the attack host IP addresses and the victim host IP addresses as suspicious hosts detected by the arrangement model, wherein the top several IP addresses (i.e., the top hosts) are extracted. N ) IP address, you can extract the top 100, that is, get the Top 100 IP address information.

[0086] S204, using all key features and basic attributes of the threat data as sorting criteria to separately arrange multiple parallel arrangement models, and sorting according to the arrangement results under each serial arrangement model to obtain suspicious hosts from the attacker's perspective and the victim's perspective under parallel arrangement;

[0087] Similar to the content in S203, the specific scheme of parallel arrangement is specifically described in this step. In a specific implementation of this embodiment, parallel arrangement is to identify suspicious hosts based on the importance of each feature. For example, as in the previous steps, when the key data features contain 10 categories and the logs contain 9 basic attribute fields after aggregation, this information is arranged in parallel, that is, each feature or field is an arrangement model to obtain a total of 19 parallel arrangement models. Further, in a specific embodiment, since the log name, log type, transmission protocol in the basic attributes and the number of threat types, number of log names, and number of protocols in the key features are repeated, the source IP address, destination IP address, source port, and destination port in the basic attributes contain the number of associated IP addresses and ports in the key features; the total of 19 parallel arrangement models obtained will contain 5 repeated parallel arrangement models, so there are actually 14 parallel arrangement models remaining. Then, the host IP addresses in each arrangement model are sorted in reverse order, and the top several (i.e., Top) are extracted from the attacking host IP address and the victim host IP address respectively. N ) as the suspicious host detected by the orchestration model; when extracting the top several IP addresses, the top 100 can be extracted, that is, the Top 100 IP address information.

[0088] Among them, the sorting method of the orchestration model based on basic attribute fields is mainly based on the aggregation of source or destination IP addresses, and then the corresponding field values ​​are deduplicated and counted before being sorted in reverse order.

[0089] S205, based on a weighted fusion algorithm, weighted fusion calculation is performed on the suspicious hosts in the serial arrangement and the suspicious hosts in the parallel arrangement to obtain suspicious host sequences from the attacker's perspective and the victim's perspective respectively;

[0090] In this step, the attack top output of each orchestration model is 100 Host IP address and top victims 100 The host IP addresses are integrated in an adaptive weighted fusion manner, and then the integrated results are sorted in reverse order to extract the attack and victim top 100 The host IP address is used as the suspicious host in the final output of the orchestration policy.

[0091] The specific calculation expression when performing weighted fusion calculation is:

[0092]

[0093] Where x represents the IP address of the host; A represents the total number of serial orchestration models, and B represents the total number of parallel orchestration models; α i and β i Respectively represent the weights of the i-th series arrangement model and the parallel arrangement model; s ix Indicates whether the host with IP address x is in the serial model s i If it exists, the value is 1, otherwise it is 0; p ix Represents a parallel model, which is similar to the series assignment; C x Indicates the final suspiciousness of the host with IP address x; dec (decrement) indicates descending order; Top N It indicates the first N host IP addresses in descending order; R indicates the final suspicious host IP address.

[0094] In a specific embodiment, when the key data features include 10 categories and the logs contain 9 basic attribute fields after aggregation, 90 different serial orchestration models and 14 different parallel orchestration models are generated, and N=100, the expression of the weighted fusion calculation is:

[0095]

[0096] The meanings of the letters in the expression are the same as those of the letters in the above expression in this step.

[0097] S206, obtaining the security protection system manufacturer information in the detection source database, fusing the security protection system manufacturer information with the suspicious host information in the suspicious host sequence, and extracting the associated manufacturer information to obtain a first fusion result;

[0098] S207, acquiring threat intelligence information from the intelligence source database, fusing the threat intelligence information with the suspicious host information in the suspicious host sequence, and determining whether each suspicious host is a malicious host to obtain a second fusion result;

[0099] S208, obtaining historical event intelligence from the event source database, fusing the historical event intelligence with the suspicious host information in the suspicious host sequence, determining whether there is a host with the same event intelligence, and obtaining a third fusion result;

[0100] S209, calculating the value of each suspicious host from the perspective of the attacker and the perspective of the victim respectively based on the first fusion result, the second fusion result and the third fusion result;

[0101] Please continue reading Figure 3 In a specific implementation of this embodiment, the top values ​​from the attacker's perspective and the victim's perspective are obtained respectively. NAfter the suspicious host sequence is obtained, it is necessary to perform multi-dimensional data fusion on the data and the content in the fusion database from the attacker's perspective and the victim's perspective respectively, so as to filter out false positive hosts and adjust their value ranking to enhance the ability to identify complex and hidden threats, reduce the probability of false positives, and thus improve the reliability of intelligence.

[0102] Specifically, in a specific implementation, the fusion database includes a detection source database, an intelligence source database, and an event source database; first, the security protection system manufacturer information in the detection source database is obtained, which mainly integrates the host IP address with the security protection system manufacturer, extracts the associated manufacturer information, and obtains the first fusion result; then, the sources from different threat intelligence libraries in the event source database are obtained for fusion, and it is determined whether each suspicious host is a malicious host to obtain a second fusion result; the historical event intelligence in the event source database and the suspicious host information in the suspicious host sequence are obtained for fusion, and it is determined whether there is a host IP address with the same event intelligence, and finally the value of the IP address is calculated according to the suspicious degree of the IP address in the orchestration model combined with the credibility of the fusion database, and the specific expression includes:

[0103]

[0104] Where x represents the host IP address; C x Indicates that the host with IP address x outputs the suspicious degree in the feature arrangement module; δ i Respectively represent the credibility weight of the i-th fusion database; l ix Indicates the fusion result of the host with IP address x in the fusion database l. If the fusion is successful, it is 1, otherwise it is 0; V x It represents the final value weight of the host with IP address x; n is the number of fusion databases.

[0105] In a specific implementation, when the fusion database includes a detection source database, an intelligence source database, and an event source database, the value of n in the expression is 3.

[0106] S210, sorting suspicious hosts according to their value, and generating high-value threat intelligence based on the sorting results;

[0107] High-value intelligence is mainly based on the results of multi-dimensional data fusion, ranking the value of each suspicious IP address, and extracting the top attackers and victims respectively. 50The IP address of the attacker is used as the high-value intelligence IP address, and the threat log information associated with the IP address is combined to generate the final high-value threat intelligence. Among them, the high-value attacker intelligence mainly includes the attacking IP address and all the destination IP addresses and corresponding ports associated with the attacking IP address, all threat types, transmission protocols, number of logs, hit attack stages, and key log names; while the high-value victim intelligence mainly includes the victim IP address and all the source IP addresses and corresponding ports associated with the victim IP address, all threat types, transmission protocols, number of logs, hit attack stages, and key log names.

[0108] In a specific implementation, after obtaining high-value threat intelligence, it also includes analyzing and judging the high-value threat intelligence, and adjusting the weights of each orchestration model and each fusion database based on the suspicious host information contained in the analysis and judgment results. Before executing the specific high-value threat intelligence mining step, the initial values ​​of the credibility weights of the series orchestration model and the parallel orchestration model during the weighted fusion calculation and the fusion database during multi-dimensional data fusion are all 1. After obtaining the analysis and judgment results, the IP address after the analysis and judgment is adjusted in the opposite direction. The specific methods include: when the high-value intelligence IP address is determined to be a real event, the weights of the orchestration model and the fusion database associated with the IP address are increased; conversely, if it is a false alarm event, the corresponding weights are reduced. The value weights of each module are adaptively adjusted through the analysis and judgment feedback mechanism, thereby continuously improving the accuracy of high-value intelligence.

[0109] The method in the embodiment of the present invention can be based on feature arrangement of information contained in the threat log, and the arranged feature information is integrated according to the intelligence information in the fusion database, so as to quickly and accurately mine potential high-value intelligence in the massive threat log data; and thus can help security personnel to timely and effectively identify real threat events. In addition, the method of this embodiment can, to a certain extent, filter out the impact of false alarm logs on real attacks, enhance the ability to identify complex and hidden threats, and improve the reliability of intelligence; it can alleviate the problem of data offset and further improve the real value of the mined threat intelligence.

[0110] The above describes a high-value threat intelligence mining method based on feature arrangement and data fusion in an embodiment of the present invention. The following describes a high-value threat intelligence mining device based on feature arrangement and data fusion in an embodiment of the present invention. Figure 4 In an embodiment of the present invention, an embodiment of a high-value threat intelligence mining device based on feature arrangement and data fusion includes:

[0111] The screening and aggregation module 401 is used to obtain threat logs, and screen and aggregate the threat logs according to basic attributes to obtain aggregated threat data;

[0112] A feature extraction module 402, used to extract key data features to be compiled from the threat data from the perspective of the attacker and the perspective of the victim;

[0113] A feature arrangement module 403 is used to arrange the key data features based on feature arrangement rules, and calculate suspicious host sequences from the attacker's perspective and the victim's perspective based on the arrangement results;

[0114] The information fusion module 404 is used to obtain a fusion database, and to fuse the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the perspective of the attacker and the perspective of the victim, and then calculate the value of each suspicious host;

[0115] The intelligence generating module 405 is used to sort the suspicious hosts according to the value and generate high-value threat intelligence based on the sorting result.

[0116] The device in the embodiment of the present invention can perform feature arrangement based on the information contained in the threat log, and fuse the arranged feature information according to the intelligence information in the fusion database, so as to quickly and accurately mine potential high-value intelligence from the massive threat log data.

[0117] Please continue reading Figure 5 In another embodiment of the present application, the feature arrangement rule includes feature serial arrangement and feature parallel arrangement, and the target host sequence includes a suspicious host sequence under serial arrangement and a suspicious host sequence under parallel arrangement; the feature arrangement module 403 specifically includes:

[0118] The serial arrangement unit 4031 is used to randomly select two key data features from all the key features of the threat data and arrange them in serial mode to obtain multiple serial arrangement models, and sort them according to the arrangement results under each serial arrangement model to obtain suspicious hosts from the attacker's perspective and the victim's perspective under the serial arrangement respectively;

[0119] The parallel arrangement unit 4032 is used to arrange all the key features and basic attributes of the threat data as sorting criteria to obtain multiple parallel arrangement models, and to sort according to the arrangement results under each serial arrangement model to obtain suspicious hosts from the attacker's perspective and the victim's perspective under the parallel arrangement;

[0120] The sorting and screening unit 4033 is used to perform weighted fusion calculation on the suspicious hosts in the serial arrangement and the suspicious hosts in the parallel arrangement based on the weighted fusion algorithm, and obtain the suspicious host sequences from the attacker's perspective and the victim's perspective respectively.

[0121] In another embodiment of the present application, the fusion database includes a detection source database, an intelligence source database, and an event source database; the information fusion module 404 specifically includes:

[0122] The first fusion unit 4041 is used to obtain the security protection system manufacturer information in the detection source database, fuse the security protection system manufacturer information with the suspicious host information in the suspicious host sequence, and extract the associated manufacturer information to obtain a first fusion result;

[0123] The second fusion unit 4042 is used to obtain threat intelligence information from the intelligence source database, fuse the threat intelligence information with the suspicious host information in the suspicious host sequence, and determine whether each of the suspicious hosts is a malicious host to obtain a second fusion result;

[0124] The third fusion unit 4043 is used to obtain historical event intelligence in the event source database, fuse the historical event intelligence with the suspicious host information in the suspicious host sequence, determine whether there is a host with the same event intelligence, and obtain a third fusion result;

[0125] The value calculation unit 4044 is used to calculate the value of each suspicious host from the perspective of the attacker and the perspective of the victim based on the first fusion result, the second fusion result and the third fusion result.

[0126] In another embodiment of the present application, the suspicious host may be marked by the IP address of the host;

[0127] In the sorting and screening unit 4032, the calculation expression for weighted fusion calculation of suspicious hosts in series arrangement and suspicious hosts in parallel arrangement is:

[0128]

[0129]

[0130] Where x represents the IP address of the host; A represents the total number of serial orchestration models, and B represents the total number of parallel orchestration models; α i and β i Respectively represent the weights of the i-th series arrangement model and the parallel arrangement model; s ix Indicates whether the host with IP address x is in the serial model s i in;p ix Represents a parallel model, which is similar to the series assignment; C x Indicates the final suspiciousness of the host with IP address x; dec indicates descending order; Top N It indicates the first N host IP addresses in descending order; R indicates the final suspicious host IP address.

[0131] In another embodiment of the present application, in the value calculation unit 4044, the calculation expression for calculating the value of each suspicious host is:

[0132]

[0133] Wherein, x represents the IP address of the host; C x Indicates the suspiciousness of the host with IP address x output in the feature arrangement module; n indicates the number of fusion databases; δ i Respectively represent the credibility weight of the i-th fusion database; l ix represents the fusion result of the host with IP address x in the fusion database l; V x Indicates the final value of the host with IP address x.

[0134] In another embodiment of the present application, the high-value threat intelligence mining device based on feature arrangement and data fusion further includes an optimization and adjustment module 406, and the optimization and adjustment module 406 is specifically used to:

[0135] The high-value threat intelligence is analyzed and judged, and the weights of each orchestration model and the weights of each fusion database are adjusted based on the suspicious host information included in the analysis and judgment results.

[0136] In another embodiment of the present application, the specific implementation details of the device for high-value threat intelligence mining based on feature orchestration and data fusion are similar to those of the aforementioned method embodiment during execution, so they will not be repeated here.

[0137] The device in the embodiment of the present invention can perform feature compilation based on the information contained in the threat log, and fuse the compiled feature information according to the intelligence information in the fusion database, so as to quickly and accurately mine potential high-value intelligence in the massive threat log data; and thus can help security personnel to timely and effectively identify real threat events. In addition, the method of this embodiment can, to a certain extent, filter out the impact of false alarm logs on real attacks, enhance the ability to identify complex and hidden threats, and improve the reliability of intelligence; it can alleviate the problem of data offset and further improve the real value of the mined threat intelligence.

[0138] Based on the same inventive concept, an embodiment of this specification also provides an electronic device for high-value threat intelligence mining based on feature arrangement and data fusion. The following is a detailed description of an electronic device for high-value threat intelligence mining based on feature arrangement and data fusion in an embodiment of the present invention from the perspective of hardware processing.

[0139] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this specification. Figure 6The electronic device 600 according to this embodiment of the present invention is described. Figure 6 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0140] like Figure 6 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0141] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above processing method section of this specification. For example, the processing unit 610 can perform the following steps: Figure 1-3 Steps shown.

[0142] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0143] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0144] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0145] The electronic device 600 may also communicate with one or more external devices 100 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Figure 6 Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0146] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the exemplary embodiments described in the present invention can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation method of the present invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (which can be a CD-ROM, USB flash drive, mobile hard disk, etc.) or on a network, including a number of instructions to enable a computing device (which can be a personal computer, server, or network device, etc.) to execute the above method according to the present invention. When the computer program is executed by a data processing device, the computer-readable medium can implement the above method of the present invention, that is: Figure 1-3 The method shown.

[0147] Figure 7 A schematic diagram of a computer-readable medium provided in accordance with an embodiment of the present specification.

[0148] accomplish Figure 1-3The computer program of the method shown can be stored on one or more computer readable media. The computer readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0149] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0150] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0151] In summary, the present invention can be implemented in hardware, or in a software module running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that general data processing devices such as microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0152] In addition, the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements a high-value threat intelligence mining method based on feature orchestration and data fusion as described in any of the above embodiments.

[0153] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0154] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0155] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A high-value threat intelligence mining method based on feature arrangement and data fusion, characterized in that: include: Obtain threat logs, filter and aggregate threat logs according to basic attributes, and obtain aggregated threat data; Extracting key data features to be compiled from the threat data from the attacker's perspective and the victim's perspective respectively; Arrange the key data features based on feature arrangement rules, and calculate suspicious host sequences from the attacker's perspective and the victim's perspective based on the arrangement results; Obtaining a fusion database, and fusing the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the perspective of the attacker and the perspective of the victim, and then calculating the value of each suspicious host; The suspicious hosts are sorted according to the values, and high-value threat intelligence is generated based on the sorting results.

2. According to claim 1, a high-value threat intelligence mining method based on feature arrangement and data fusion is characterized in that: The feature arrangement rule includes feature serial arrangement and feature parallel arrangement, and the target host sequence includes a suspicious host sequence under serial arrangement and a suspicious host sequence under parallel arrangement; The key data features are arranged based on feature arrangement rules, and suspicious host sequences from the attacker's perspective and the victim's perspective are calculated based on the arrangement results, including: Randomly select two key data features from all the key features of the threat data and arrange them in series to obtain multiple series arrangement models. Under each series arrangement model, sort them according to the arrangement results to obtain suspicious hosts from the attacker's perspective and the victim's perspective under the series arrangement. All key features and basic attributes of threat data are used as sorting criteria to separately arrange multiple parallel arrangement models. In each serial arrangement model, the arrangement results are sorted to obtain suspicious hosts from the attacker's perspective and the victim's perspective under parallel arrangement. Based on the weighted fusion algorithm, weighted fusion calculations are performed on the suspicious hosts under serial arrangement and the suspicious hosts under parallel arrangement to obtain the suspicious host sequences from the attacker's perspective and the victim's perspective respectively.

3. According to claim 2, a high-value threat intelligence mining method based on feature arrangement and data fusion is characterized in that: The fusion database includes a detection source database, an intelligence source database and an event source database; The step of obtaining the fusion database and fusing the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the perspective of the attacker and the perspective of the victim to calculate the value of each suspicious host includes: Obtaining the manufacturer information of the security protection system in the detection source database, fusing the manufacturer information of the security protection system with the suspicious host information in the suspicious host sequence, and extracting the associated manufacturer information to obtain a first fusion result; Acquire threat intelligence information from an intelligence source database, fuse the threat intelligence information with the suspicious host information in the suspicious host sequence, and determine whether each of the suspicious hosts is a malicious host to obtain a second fusion result; Acquire historical event intelligence in an event source database, fuse the historical event intelligence with the suspicious host information in the suspicious host sequence, determine whether there is a host with the same event intelligence, and obtain a third fusion result; Based on the first fusion result, the second fusion result and the third fusion result, the value of each suspicious host is calculated from the perspective of the attacker and the perspective of the victim respectively.

4. According to claim 3, a high-value threat intelligence mining method based on feature arrangement and data fusion is characterized in that: The suspicious host can be marked by the host's IP address; The calculation expression for weighted fusion calculation of suspicious hosts in series arrangement and suspicious hosts in parallel arrangement is: Where x represents the IP address of the host; A represents the total number of serial orchestration models, and B represents the total number of parallel orchestration models; α i and β i Respectively represent the weights of the i-th series arrangement model and the parallel arrangement model; s ix Indicates whether the host with IP address x is in the serial model s i in;p ix Represents a parallel model, which is similar to the series assignment; C x Indicates the final suspiciousness of the host with IP address x; dec indicates descending order; Top N It indicates the first N host IP addresses in descending order; R indicates the final suspicious host IP address.

5. According to claim 4, a high-value threat intelligence mining method based on feature arrangement and data fusion is characterized in that: When calculating the value of each suspicious host, the calculation expression is: Wherein, x represents the IP address of the host; C x Indicates the suspiciousness of the host with IP address x output in the feature arrangement module; n indicates the number of fusion databases; δ i Respectively represent the credibility weight of the i-th fusion database; l ix represents the fusion result of the host with IP address x in the fusion database l; V x Indicates the final value of the host with IP address x.

6. A high-value threat intelligence mining method based on feature arrangement and data fusion according to claim 5, characterized in that: After sorting the suspicious hosts according to the values ​​and generating high-value threat intelligence based on the sorting results, the method further includes: The high-value threat intelligence is analyzed and judged, and the weights of each orchestration model and the weights of each fusion database are adjusted based on the suspicious host information included in the analysis and judgment results.

7. A high-value threat intelligence mining device based on feature arrangement and data fusion, characterized in that: The high-value threat intelligence mining device based on feature arrangement and data fusion includes: A screening and aggregation module is used to obtain threat logs and screen and aggregate the threat logs according to basic attributes to obtain aggregated threat data; A feature extraction module, used to extract key data features to be compiled from the threat data from the perspective of the attacker and the perspective of the victim respectively; A feature arrangement module, used to arrange the key data features based on feature arrangement rules, and calculate suspicious host sequences from the attacker's perspective and the victim's perspective based on the arrangement results; An information fusion module is used to obtain a fusion database, and to fuse the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the perspective of the attacker and the perspective of the victim, and then calculate the value of each suspicious host; The intelligence generation module is used to sort the suspicious hosts according to the value and generate high-value threat intelligence based on the sorting results.

8. A high-value threat intelligence mining device based on feature arrangement and data fusion, characterized in that: The high-value threat intelligence mining device based on feature arrangement and data fusion includes: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the high-value threat intelligence mining device based on feature orchestration and data fusion performs the steps of the high-value threat intelligence mining method based on feature orchestration and data fusion as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the program / instructions are executed by the processor, the steps of the high-value threat intelligence mining method based on feature orchestration and data fusion as described in any one of claims 1-6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, the steps of the high-value threat intelligence mining method based on feature orchestration and data fusion as described in any one of claims 1-6 are implemented.

Images