A High-Value Threat Intelligence Mining Method Based on Feature Arrangement and Data Fusion

Abstract

This invention relates to the field of network information security technology, and discloses a method, apparatus, electronic device, computer-readable storage medium, and computer program product for mining high-value threat intelligence based on feature orchestration and data fusion. It addresses the technical problem in existing technologies of being unable to quickly and accurately mine potential high-value threat intelligence from massive amounts of log data. The method includes: acquiring threat logs and filtering and aggregating them according to basic attributes to obtain threat data; extracting key data features to be orchestrated from the threat data; orchestrating based on feature orchestration rules, and calculating a sequence of suspicious hosts from both the attacker's and victim's perspectives based on the orchestration results; acquiring a fusion database, and fusing the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from both the attacker's and victim's perspectives to calculate the value of each suspicious host; and ranking the suspicious hosts according to their value to generate high-value threat intelligence.

Description

Technical Field

[0001] This invention relates to the field of network information security technology, and in particular to a method, apparatus, electronic device, computer storage medium, and computer program product for mining high-value threat intelligence based on feature orchestration and data fusion. Background Technology

[0002] With the rapid development of internet and information technology, cyberspace has gradually become an indispensable part of people's production and life. However, the resulting cyberspace security issues have also become increasingly important. Security incidents such as information theft, cyber extortion, and infrastructure damage are emerging one after another, seriously threatening personal privacy, business operations, and even national security. Traditional security measures such as firewalls, intrusion detection systems, and intrusion prevention systems were effective in capturing suspicious network attack data in the past. However, with the popularization of big data technology in recent years, the number of threat logs generated by security protection systems has increased exponentially. This has led to real attacks being overwhelmed by a large number of false alarms, making it impossible for security personnel to identify real threat events in a timely and effective manner.

[0003] To address this issue, existing methods mostly utilize rule matching and machine learning to extract valid attacks from threat logs. Rule matching primarily extracts common features from known attack events for identification. While simple, efficient, and easy to implement, it struggles to detect unknown or complex attacks, and feature extraction requires extensive expert experience, resulting in a high false negative rate. Machine learning, on the other hand, primarily uses a data-driven approach to identify suspicious intelligence data from threat logs. While avoiding reliance on expert experience, this method requires extensive labeled data to train the model, and the model's interpretability and real-time performance are poor. Therefore, how to quickly and accurately extract potential high-value threat intelligence from massive amounts of log data has become a critical problem that urgently needs to be solved in the current security field. Summary of the Invention

[0004] The main objective of this invention is to solve the technical problem that existing technologies cannot quickly and accurately extract potential high-value threat intelligence from massive amounts of log data.

[0005] The first aspect of this invention provides a method for mining high-value threat intelligence based on feature orchestration and data fusion, comprising:

[0006] Obtain threat logs and filter and aggregate them according to basic attributes to obtain aggregated threat data;

[0007] Key data features to be orchestrated are extracted from the threat data from both the attacker's and victim's perspectives.

[0008] The key data features are arranged based on feature arrangement rules, and a sequence of suspicious hosts from the attacker's perspective and the victim's perspective is calculated based on the arrangement results.

[0009] The fusion database is obtained, and the suspicious host information in the suspicious host sequence is fused with the intelligence information in the fusion database from both the attacker's and victim's perspectives to calculate the value of each suspicious host.

[0010] The suspicious hosts are ranked according to their value, and high-value threat intelligence is generated based on the ranking results.

[0011] Optionally, in a first implementation of the first aspect of the present invention, the feature arrangement rules include feature serial arrangement and feature parallel arrangement, and the suspicious host sequence includes a suspicious host sequence under serial arrangement and a suspicious host sequence under parallel arrangement;

[0012] The key data features are arranged according to the feature arrangement rules, and the suspicious host sequence calculated based on the arrangement results from the attacker's perspective and the victim's perspective includes:

[0013] Two key data features are randomly selected from all the key features of the threat data and arranged in a concatenated manner to obtain multiple concatenated arrangement models. Under each concatenated arrangement model, the arrangement results are sorted to obtain the suspicious hosts under the concatenated arrangement from the perspectives of attackers and victims.

[0014] Multiple parallel orchestration models are obtained by using all the key features and basic attributes of threat data as sorting criteria. Under each parallel orchestration model, the data are sorted according to the orchestration results to obtain suspicious hosts from the perspectives of attackers and victims in the parallel orchestration.

[0015] Based on the weighted fusion algorithm, the suspicious hosts under serial arrangement and the suspicious hosts under parallel arrangement are weighted and fused to obtain the suspicious host sequences from the attacker's perspective and the victim's perspective, respectively.

[0016] Alternatively, in a second implementation of the first aspect of the invention,

[0017] The fusion database includes a detection source database, an intelligence source database, and an event source database;

[0018] The process of acquiring the fusion database, which involves fusing the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from both the attacker's and victim's perspectives, and then calculating the value of each suspicious host, includes:

[0019] Obtain security protection system vendor information from the detection source database, merge the security protection system vendor information with the suspicious host information in the suspicious host sequence, and extract the associated vendor information to obtain the first fusion result;

[0020] The threat intelligence information is obtained from the intelligence source database, and the threat intelligence information is fused with the suspicious host information in the suspicious host sequence. The second fusion result is obtained by determining whether each of the suspicious hosts is a malicious host.

[0021] Historical event intelligence is obtained from the event source database. The historical event intelligence is fused with the suspicious host information in the suspicious host sequence to determine whether there are hosts with the same event intelligence, and a third fusion result is obtained.

[0022] Based on the first, second, and third fusion results, the value of each suspicious host is calculated from both the attacker's and victim's perspectives.

[0023] Alternatively, in a third implementation of the first aspect of the invention,

[0024] The suspicious host can be identified by its IP address;

[0025] The calculation expression for weighted fusion calculation of suspicious hosts under cascaded and parallel orchestration is as follows:

[0026]

[0027]

[0028] in, x A represents the IP address of the host; A represents the total number of serial orchestration models, and B represents the total number of parallel orchestration models. α i and β i They represent the first i The weights of the serial and parallel orchestration models; s ix Indicates that the IP address is x Is the host in the serial model? s i middle; p ix The parallel model is represented in the same way as the series model; C x Indicates that the IP address is x The final suspiciousness of the host; dec Indicates descending order; This indicates taking the first N host IP addresses in descending order; R represents the final suspicious host IP address.

[0029] Optionally, in a fourth implementation of the first aspect of the present invention, the calculation expression for calculating the value of each suspicious host is:

[0030]

[0031] Among them, the x Indicates the host's IP address; C x Indicates that the IP address is x The host outputs the suspiciousness level in the feature orchestration module; n represents the number of fused databases; δ i They represent the first i Credibility weights for each integrated database; l ix Indicates that the IP address is x The host in the merged database l The fusion results in; V x Indicates that the IP address is x The ultimate value of the host.

[0032] Optionally, in a fifth implementation of the first aspect of the present invention, after ranking the suspicious hosts according to the value and generating high-value threat intelligence based on the ranking result, the method further includes:

[0033] The high-value threat intelligence is analyzed, and the weights of each orchestration model and each fusion database are adjusted based on the suspicious host information contained in the analysis results.

[0034] A second aspect of the present invention provides a high-value threat intelligence mining device based on feature orchestration and data fusion, comprising:

[0035] The filtering and aggregation module is used to obtain threat logs and filter and aggregate the threat logs according to basic attributes to obtain aggregated threat data.

[0036] The feature extraction module is used to extract key data features to be orchestrated from the threat data from both the attacker's and victim's perspectives, respectively.

[0037] The feature orchestration module is used to orchestrate the key data features based on feature orchestration rules, and to calculate a sequence of suspicious hosts from the attacker's perspective and the victim's perspective based on the orchestration results.

[0038] The information fusion module is used to acquire the fusion database, and calculate the value of each suspicious host by fusing the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the attacker's perspective and the victim's perspective, respectively.

[0039] The intelligence generation module is used to sort the suspicious hosts according to the value, and generate high-value threat intelligence based on the sorting results.

[0040] A third aspect of the present invention provides a high-value threat intelligence mining device based on feature orchestration and data fusion, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the high-value threat intelligence mining device based on feature orchestration and data fusion to perform the steps of the above-described high-value threat intelligence mining method based on feature orchestration and data fusion.

[0041] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described high-value threat intelligence mining method based on feature orchestration and data fusion.

[0042] A fifth aspect of the present invention provides a computer program product, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, the steps of the above-described high-value threat intelligence mining method based on feature orchestration and data fusion are implemented.

[0043] The technical solution provided by this invention involves acquiring threat logs and filtering and aggregating them according to basic attributes to obtain aggregated threat data. Key data features to be orchestrated are extracted from the threat data from both attacker and victim perspectives. These key data features are then orchestrated based on feature orchestration rules, and a sequence of suspicious hosts from both attacker and victim perspectives is calculated based on the orchestration results. A fusion database is acquired, and the suspicious host information in the suspicious host sequence is fused with the intelligence information in the fusion database from both attacker and victim perspectives to calculate the value of each suspicious host. The suspicious hosts are then ranked according to their value, and high-value threat intelligence is generated based on the ranking results. This method can orchestrate features based on the information contained in threat logs and fuse the orchestrated feature information according to the intelligence information in the fusion database, thereby quickly and accurately mining potential high-value intelligence from massive amounts of threat log data. The device, electronic device, computer-readable storage medium, and computer program product provided by this invention also solve the corresponding technical problems. Attached Figure Description

[0044] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0045] Figure 1This is a flowchart illustrating the first embodiment of a high-value threat intelligence mining method based on feature orchestration and data fusion in this invention.

[0046] Figure 2 This is a flowchart illustrating a second embodiment of a high-value threat intelligence mining method based on feature orchestration and data fusion according to an embodiment of the present invention.

[0047] Figure 3 This is another flowchart illustrating a second embodiment of a high-value threat intelligence mining method based on feature orchestration and data fusion in this invention.

[0048] Figure 4 This is a schematic diagram of an embodiment of a high-value threat intelligence device based on feature orchestration and data fusion according to the present invention;

[0049] Figure 5 This is a schematic diagram of another embodiment of a high-value threat intelligence device based on feature orchestration and data fusion according to the present invention;

[0050] Figure 6 This is a schematic diagram of an embodiment of a high-value threat intelligence device based on feature orchestration and data fusion according to an embodiment of the present invention;

[0051] Figure 7 This is a schematic diagram illustrating the principle of a computer-readable medium according to an embodiment of the present invention. Detailed Implementation

[0052] Exemplary embodiments of the invention will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limiting the invention to the embodiments set forth herein. Rather, these exemplary embodiments are provided to make the invention more comprehensive and complete, and to facilitate a full communication of the inventive concept to those skilled in the art. The same reference numerals in the drawings denote the same or similar elements, components, or parts, and therefore repeated descriptions of them will be omitted.

[0053] Subject to the technical concept of this invention, the features, structures, characteristics or other details described in a particular embodiment may be combined in one or more other embodiments in a suitable manner.

[0054] In the description of specific embodiments, the features, structures, characteristics, or other details described in this invention are intended to enable those skilled in the art to fully understand the embodiments. However, it is not excluded that those skilled in the art can practice the technical solutions of this invention without one or more of the specific features, structures, characteristics, or other details.

[0055] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0056] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0057] The terms “and / or” or “and / or” include all combinations of any one or more of the listed items.

[0058] Please see Figure 1 The first embodiment of a high-value threat intelligence mining method based on feature orchestration and data fusion in this invention includes:

[0059] S101. Obtain threat logs and filter and aggregate the threat logs according to basic attributes to obtain aggregated threat data;

[0060] It is understood that the executing entity of this invention can be a high-value threat intelligence mining device based on feature orchestration and data fusion, or it can be a terminal or a server; the specific implementation is not limited here. This embodiment of the invention will be described using a server as an example.

[0061] After receiving a request to mine high-value threat intelligence, the server first obtains threat logs, which mainly come from attack data captured by security protection systems from different vendors. These attack data are then used as the raw threat logs for data analysis.

[0062] In one specific implementation, after acquiring the threat logs, the process further includes preprocessing the threat log data to remove generalized noisy logs, such as logs with missing key field values, duplicate logs, incorrect data formats, or logs containing whitelists, in order to improve data quality and reduce the impact of invalid logs.

[0063] After obtaining the preprocessed threat logs, the threat logs are filtered and aggregated according to basic attributes. Specifically, the logs are filtered based on threat level and type, filtering out informational logs and non-threatening basic system logs. Then, the filtered threat log data is categorized and aggregated according to basic attributes to obtain aggregated threat data.

[0064] S102. Extract key data features to be orchestrated from the threat data from both the attacker's and victim's perspectives.

[0065] After obtaining the aggregated threat data in the aforementioned steps, key fields are extracted from both the attacker's and victim's perspectives, and numerical transformation or data statistics are used to extract key data features to be orchestrated from the threat data.

[0066] S103. Arrange key data features based on feature arrangement rules, and calculate the suspicious host sequence from the attacker's perspective and the victim's perspective based on the arrangement results;

[0067] Based on the feature selection results, each feature is arranged in series or in parallel according to its importance. Then, according to the arrangement mode, the suspiciousness of the IP address (Internet Protocol Address) of each attacking host or victim host is sorted from high to low. Host information can be recorded by the host IP address.

[0068] Subsequently, for each feature orchestration result, the top N suspicious hosts are extracted, and the top N most suspicious hosts are output after weighted fusion of all suspicious hosts according to the orchestration model weights.

[0069] S104. Obtain the fusion database, and calculate the value of each suspicious host by fusing the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the perspectives of attackers and victims respectively.

[0070] Obtain a fusion database, which can be built before obtaining threat logs and includes data information from perspectives such as detection sources, intelligence sources, and event sources.

[0071] In this step, the most suspicious N hosts output in the previous steps are traversed, and multi-dimensional data from the perspectives of detection source, intelligence source, and event source are obtained from the fusion database. From the perspectives of attacker and victim, the information of the most suspicious N hosts is fused with the multi-dimensional data from the perspectives of detection source, intelligence source, and event source in the fusion database. The value of each suspicious host is calculated based on the fusion result, thereby filtering the most suspicious N hosts and eliminating false positive hosts.

[0072] S105. Sort suspicious hosts according to their value and generate high-value threat intelligence based on the sorting results.

[0073] Based on the value information of each suspicious host after data fusion, high-value suspicious hosts are selected from both the attacker's and victim's perspectives, and a final high-value threat intelligence report is generated. The high-value threat intelligence report may include information such as the attacking host IP address and all destination host IP addresses associated with that attacking host IP address and their corresponding ports, all threat types, transmission protocols, number of logs, the hit attack stage, and key log names. The high-value victim intelligence mainly includes information such as the victim host IP address and all source IP addresses associated with that victim host IP address and their corresponding ports, all threat types, transmission protocols, number of logs, the hit attack stage, and key log names.

[0074] The method in this embodiment of the invention can feature-arrange the information contained in the threat log and fuse the arranged feature information according to the intelligence information in the fusion database, thereby quickly and accurately mining potential high-value intelligence from massive threat log data.

[0075] Please refer to Figure 2-3 A second embodiment of a high-value threat intelligence mining method based on feature orchestration and data fusion in this invention includes:

[0076] S201. Obtain threat logs and filter and aggregate the threat logs according to basic attributes to obtain aggregated threat data;

[0077] Please refer to step S101 in the aforementioned embodiment. Similarly, this embodiment uses a server as the execution subject for explanation. After receiving a high-value threat intelligence mining request, the server first obtains threat logs. These threat logs mainly originate from attack data captured by security protection systems from different vendors, and the attack data is used as the raw threat logs for data analysis.

[0078] In one specific implementation, since threat logs mainly originate from attack data captured by security protection systems from different vendors, and considering that the threat log formats generated by different vendors' protection systems are different, in order to facilitate subsequent feature selection and data analysis, the threat logs from different vendors are generalized into a unified format and stored in the Elastic Search database as the raw threat logs for data analysis.

[0079] In one specific implementation, after obtaining the original threat logs, the data is further preprocessed, specifically by removing generalized noisy logs, such as logs with missing key field values, duplicate logs, incorrect data formats, or logs containing whitelists, in order to improve data quality and reduce the impact of invalid logs.

[0080] In one specific implementation, after obtaining the preprocessed threat logs, the logs are filtered based on the threat level and type, filtering out informational logs or system basic logs without threats; then the filtered log data is aggregated according to nine basic attributes, including log name, log type, log level, source IP address, destination IP address, source port, destination port, transmission protocol, and log source, to obtain aggregated threat data, so as to reduce the computational load of subsequent feature analysis.

[0081] S202. Extract key data features to be orchestrated from the threat data from both the attacker's and victim's perspectives.

[0082] In this embodiment, before extracting specific data features, the method also includes combining historical expert log analysis experience to extract key fields from the perspectives of attackers and victims. After obtaining candidate key fields, the key data features to be arranged are extracted based on numerical transformation or statistical methods.

[0083] In one specific implementation, the key data features to be orchestrated include 10 pieces of information calculated from the perspectives of both attackers and victims, such as the number of IP addresses associated with each IP address, the number of threat types, the number of log names, the number of logs, the number of ports, the number of protocols, the number of attack stages hit, the number of key types hit, and the number of key log names hit.

[0084] S203. Randomly select two key data features from all the key features of the threat data and arrange them in a concatenated manner to obtain multiple concatenated arrangement models. Under each concatenated arrangement model, sort them according to the arrangement results to obtain the suspicious hosts under the concatenated arrangement from the perspective of the attacker and the perspective of the victim respectively.

[0085] Please see Figure 3 In one specific implementation of this embodiment, after obtaining the key data features, the features are arranged in series and in parallel. This step specifically describes the specific scheme of series arrangement. Specifically, series arrangement mainly uses the correlation between different features to discover suspicious hosts. Two key data features are randomly selected from all the key features and arranged in series. Specifically, the series arrangement means that the two key data features are first sorted according to one of the randomly selected key data features as the ranking index, and then those with the same ranking are sorted again according to the other key data feature as the ranking index, resulting in multiple series arrangement models.

[0086] In one specific implementation, when the key data features include 10 categories, randomly selecting two for sorting can generate 90 different concatenated orchestration models. For example, when selecting the threat type quantity and log quantity features, the host IP addresses are first sorted in reverse order by the threat type quantity. Then, based on the sorting, they are sorted in descending order by the log quantity. The top-ranked IP addresses from the final sorting results of the attacking host IP addresses and the victim host IP addresses are extracted as suspicious hosts detected by the orchestration model. Specifically, the top-ranked IP addresses (i.e., the highest-ranked IP addresses) are selected from the final sorting results of the attacking host IP addresses and the victim host IP addresses. When you have an IP address, you can extract the first 100 names, which gives you the IP address. IP address information.

[0087] S204. All key features and basic attributes of the threat data are used as sorting criteria to separately arrange multiple parallel arrangement models. Under each parallel arrangement model, the arrangement results are sorted to obtain the suspicious hosts under the parallel arrangement from the perspectives of attackers and victims.

[0088] Similar to S203, this step details the specific scheme for parallel orchestration. In one specific implementation of this embodiment, parallel orchestration identifies suspicious hosts based on the importance of each feature. For example, as in the aforementioned steps, when the key data features contain 10 categories and the log aggregation contains 9 basic attribute fields, this information is orchestrated in parallel, resulting in a total of 19 parallel orchestration models, where each feature or field is an orchestration model. Further, in one specific embodiment, because the log name, log type, and transmission protocol in the basic attributes are repeated, and the number of threat types, log names, and protocols in the key features are also repeated, and the source IP address, destination IP address, source port, and destination port in the basic attributes include the associated IP address and port number in the key features, the resulting 19 parallel orchestration models will contain 5 duplicate parallel orchestration models, leaving 14 parallel orchestration models remaining. Then, the host IP addresses in each orchestration model are sorted in reverse order, and the top few (i.e., ...) are extracted from the attacking host IP address and the victim host IP address, respectively. The IP addresses of the selected hosts are used as the suspicious hosts detected by this orchestration model; when extracting the top few IP addresses, the top 100 can be extracted, thus obtaining... IP address information.

[0089] Among them, the sorting method of the arrangement model based on basic attribute fields mainly aggregates source or destination IP addresses, and then performs reverse sorting after deduplication and statistical analysis of the corresponding field values.

[0090] S205. Based on the weighted fusion algorithm, perform weighted fusion calculation on suspicious hosts under serial arrangement and suspicious hosts under parallel arrangement to obtain suspicious host sequences from the attacker's perspective and the victim's perspective, respectively.

[0091] In this step, the attacks output by each of the aforementioned orchestration models will be... Host IP address and victim The host IP addresses are combined using an adaptive weighted fusion method, and then the combined results are sorted in reverse order to extract the attacker and victim separately. The host IP address is used as the final output of the orchestration policy to identify suspicious hosts.

[0092] The specific calculation expression for weighted fusion is as follows:

[0093]

[0094]

[0095] in, x A represents the IP address of the host; A represents the total number of serial orchestration models, and B represents the total number of parallel orchestration models. α i and β i They represent the first i The weights of the serial and parallel orchestration models; s ix Indicates that the IP address is x Is the host in the serial model? s i If it exists, the value is 1; otherwise, it is 0. p ix The parallel model is represented in the same way as the series model; C x Indicates that the IP address is x The final suspiciousness of the host; dec (decrement) indicates a descending order; This indicates taking the first N host IP addresses in descending order; R represents the final suspicious host IP address.

[0096] In a specific embodiment, when the key data features include 10 categories and the log aggregation includes 9 basic attribute fields, generating 90 different serial orchestration models and 14 different parallel orchestration models, and N=100, the expression for the weighted fusion calculation is:

[0097]

[0098]

[0099] The meanings of the letters in the expression are the same as those in the expression described above in this step.

[0100] S206. Obtain the security protection system vendor information from the detection source database, merge the security protection system vendor information with the suspicious host information in the suspicious host sequence, and extract the associated vendor information to obtain the first fusion result.

[0101] S207. Obtain threat intelligence information from the intelligence source database, merge the threat intelligence information with the suspicious host information in the suspicious host sequence, and determine whether each suspicious host is a malicious host to obtain a second fusion result.

[0102] S208. Obtain historical event intelligence from the event source database, fuse the historical event intelligence with the suspicious host information in the suspicious host sequence, determine whether there are hosts with the same event intelligence, and obtain the third fusion result;

[0103] S209. Based on the first fusion result, the second fusion result, and the third fusion result, calculate the value of each suspicious host from the perspectives of the attacker and the victim, respectively.

[0104] Please continue reading Figure 3 In one specific implementation of this embodiment, the attacker's angle and the victim's angle are obtained respectively. After identifying a suspicious host sequence, it is necessary to perform multi-dimensional data fusion of the data and the contents of the fusion database from both the attacker's and victim's perspectives. This is to filter out false positive hosts and adjust their value ranking to enhance the ability to identify complex and covert threats, reduce the probability of false alarms, and thus improve the reliability of intelligence.

[0105] Specifically, in one implementation, the fusion database includes a detection source database, an intelligence source database, and an event source database. First, it acquires security system vendor information from the detection source database, primarily fusing host IP addresses with security system vendor information to extract associated vendor information, yielding a first fusion result. Then, it acquires and fuses information from different threat intelligence databases from the event source database, determining whether each suspicious host is malicious, resulting in a second fusion result. Next, it acquires and fuses historical event intelligence and suspicious host information from suspicious host sequences in the event source database, determining if host IP addresses with the same event intelligence exist. Finally, it calculates the value of the IP address based on its suspiciousness in the orchestration model combined with the credibility of the fusion database, with specific expressions including:

[0106]

[0107] in, x Indicates the host IP address;C x Indicates that the IP address is x The host outputs the suspiciousness level in the feature orchestration module; δ i They represent the first i Credibility weights for each integrated database; l ix Indicates that the IP address is x The host in the merged database l The fusion result is 1 if the fusion is successful, and 0 otherwise. V x Indicates that the IP address is x The final value weight of the host; n is the number of merged databases.

[0108] In one specific implementation, when the fusion database includes a detection source database, an intelligence source database, and an event source database, the value of n in the expression is 3.

[0109] S210. Sort suspicious hosts according to their value and generate high-value threat intelligence based on the sorting results;

[0110] High-value intelligence is primarily based on the results of multi-dimensional data fusion, ranking each suspicious IP address by value, and separately extracting attacker and victim information. The IP addresses are used as high-value intelligence IP addresses, and combined with threat log information associated with these IP addresses to generate the final high-value threat intelligence. High-value attacker intelligence mainly includes the attacking IP address and all associated destination IP addresses and corresponding ports, all threat types, transport protocols, number of logs, the attack phase, and key log names; while high-value victim intelligence mainly includes the victim IP address and all associated source IP addresses and corresponding ports, all threat types, transport protocols, number of logs, the attack phase, and key log names.

[0111] In one specific implementation, after obtaining high-value threat intelligence, the process further includes analyzing the high-value threat intelligence and adjusting the weights of each orchestration model and each fusion database based on the suspicious host information contained in the analysis results. Specifically, before executing the specific high-value threat intelligence mining steps, the initial values ​​of the credibility weights of the serial orchestration model, the parallel orchestration model, and the fusion database during multi-dimensional data fusion are all 1. After obtaining the analysis results, the IP addresses are adjusted in reverse. Specifically, when the high-value intelligence IP address is determined to be a real event, the weights of the orchestration model and fusion database associated with the IP address are increased; conversely, if it is a false alarm, the corresponding weights are decreased. This analysis feedback mechanism adaptively adjusts the value weights of each module, thereby continuously improving the accuracy of high-value intelligence.

[0112] The method in this embodiment of the invention can perform feature orchestration on the information contained in threat logs and fuse the orchestrated feature information with intelligence information from a fusion database, thereby quickly and accurately mining potential high-value intelligence from massive threat log data. This helps security personnel to identify real threat events in a timely and effective manner. Furthermore, this method can, to a certain extent, filter out the influence of false positive logs on real attacks, enhance the ability to identify complex and covert threats, and improve the reliability of intelligence; it can alleviate data offset problems and further enhance the true value of the mined threat intelligence.

[0113] The above describes a high-value threat intelligence mining method based on feature orchestration and data fusion in an embodiment of the present invention. The following describes a high-value threat intelligence mining apparatus based on feature orchestration and data fusion in an embodiment of the present invention. Please refer to [link / reference]. Figure 4 One embodiment of the high-value threat intelligence mining device based on feature orchestration and data fusion in this invention includes:

[0114] The filtering and aggregation module 401 is used to obtain threat logs and filter and aggregate the threat logs according to basic attributes to obtain aggregated threat data.

[0115] Feature extraction module 402 is used to extract key data features to be arranged from the threat data from the attacker's perspective and the victim's perspective, respectively;

[0116] The feature orchestration module 403 is used to orchestrate the key data features based on feature orchestration rules, and to calculate a sequence of suspicious hosts from the attacker's perspective and the victim's perspective based on the orchestration results.

[0117] The information fusion module 404 is used to acquire the fusion database and calculate the value of each suspicious host by fusing the suspicious host information in the suspicious host sequence with the intelligence information in the fusion database from the attacker's perspective and the victim's perspective, respectively.

[0118] The intelligence generation module 405 is used to sort the suspicious hosts according to the value and generate high-value threat intelligence based on the sorting results.

[0119] The device in this embodiment of the invention can perform feature arrangement based on the information contained in the threat log, and fuse the arranged feature information according to the intelligence information in the fusion database, thereby quickly and accurately mining potential high-value intelligence from massive threat log data.

[0120] Please continue reading Figure 5 In another embodiment of this application, the feature arrangement rules include feature concatenation arrangement and feature parallel arrangement, and the suspicious host sequence includes a suspicious host sequence under concatenation arrangement and a suspicious host sequence under parallel arrangement; the feature arrangement module 403 specifically includes:

[0121] The cascaded orchestration unit 4031 is used to randomly select two key data features from all the key features of the threat data and orchestrate them in a cascaded manner to obtain multiple cascaded orchestration models. Under each cascaded orchestration model, the suspicious hosts under the cascaded orchestration are sorted according to the orchestration results to obtain the attacker's perspective and the victim's perspective respectively.

[0122] Parallel orchestration unit 4032 is used to separately orchestrate all key features and basic attributes of threat data as sorting criteria to obtain multiple parallel orchestration models. Under each parallel orchestration model, the data is sorted according to the orchestration results to obtain suspicious hosts from the attacker's perspective and the victim's perspective under the parallel orchestration.

[0123] The sorting and filtering unit 4033 is used to perform weighted fusion calculations on suspicious hosts arranged in series and suspicious hosts arranged in parallel based on a weighted fusion algorithm, so as to obtain suspicious host sequences from the attacker's perspective and the victim's perspective, respectively.

[0124] In another embodiment of this application, the fusion database includes a detection source database, an intelligence source database, and an event source database; the information fusion module 404 specifically includes:

[0125] The first fusion unit 4041 is used to obtain security protection system vendor information from the detection source database, fuse the security protection system vendor information with the suspicious host information in the suspicious host sequence, and extract the associated vendor information to obtain the first fusion result.

[0126] The second fusion unit 4042 is used to acquire threat intelligence information from the intelligence source database, fuse the threat intelligence information with the suspicious host information in the suspicious host sequence, and determine whether each of the suspicious hosts is a malicious host to obtain the second fusion result.

[0127] The third fusion unit 4043 is used to acquire historical event intelligence from the event source database, fuse the historical event intelligence with the suspicious host information in the suspicious host sequence, determine whether there are hosts with the same event intelligence, and obtain the third fusion result.

[0128] The value calculation unit 4044 is used to calculate the value of each suspicious host from the perspectives of the attacker and the victim, respectively, based on the first fusion result, the second fusion result, and the third fusion result.

[0129] In another embodiment of this application, the suspicious host can be labeled by the host's IP address;

[0130] In the sorting and filtering unit 4033, the calculation expression for weighted fusion calculation of suspicious hosts under serial arrangement and suspicious hosts under parallel arrangement is as follows:

[0131]

[0132]

[0133] in, x A represents the IP address of the host; A represents the total number of serial orchestration models, and B represents the total number of parallel orchestration models. α i and β i They represent the first i The weights of the serial and parallel orchestration models; s ix Indicates that the IP address is x Is the host in the serial model? s i middle; p ix The parallel model is represented in the same way as the series model; C x Indicates that the IP address is x The final suspiciousness of the host; dec Indicates descending order; This indicates taking the first N host IP addresses in descending order; R represents the final suspicious host IP address.

[0134] In another embodiment of this application, the calculation expression for calculating the value of each suspicious host in the value calculation unit 4044 is as follows:

[0135]

[0136] Among them, the x Indicates the host's IP address; C x Indicates that the IP address is x The host outputs the suspiciousness level in the feature orchestration module; n represents the number of fused databases; δ i They represent the first i Credibility weights for each integrated database; l ix Indicates that the IP address is x The host in the merged database l The fusion results in; V x Indicates that the IP address is x The ultimate value of the host.

[0137] In another embodiment of this application, the high-value threat intelligence mining device based on feature orchestration and data fusion further includes an optimization and adjustment module 406, which is specifically used for:

[0138] The high-value threat intelligence is analyzed, and the weights of each orchestration model and each fusion database are adjusted based on the suspicious host information contained in the analysis results.

[0139] In another embodiment of this application, the specific implementation details of the apparatus for mining high-value threat intelligence based on feature orchestration and data fusion are similar to those of the aforementioned method embodiments, and therefore will not be repeated here.

[0140] The apparatus in this embodiment of the invention can perform feature orchestration on information contained in threat logs and fuse the orchestrated feature information with intelligence information from a fusion database, thereby quickly and accurately mining potential high-value intelligence from massive threat log data. This helps security personnel to identify real threat events in a timely and effective manner. Furthermore, the method in this embodiment can, to a certain extent, filter out the influence of false positive logs on real attacks, enhance the ability to identify complex and covert threats, and improve the reliability of intelligence; it can alleviate data offset problems and further enhance the true value of the mined threat intelligence.

[0141] Based on the same inventive concept, this specification also provides an electronic device for mining high-value threat intelligence based on feature orchestration and data fusion. The following is a detailed description of an electronic device for mining high-value threat intelligence based on feature orchestration and data fusion in this embodiment of the invention from the perspective of hardware processing.

[0142] Figure 6 This is a schematic diagram of an electronic device provided as an embodiment of this specification. Refer to the following... Figure 6 The electronic device 600 according to this embodiment of the present invention will be described. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0143] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0144] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps described in the processing method section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 610 can perform actions such as... Figure 1-3 The steps are shown.

[0145] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.

[0146] The storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0147] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0148] Electronic device 600 can also communicate with one or more external devices 100 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although... Figure 6 As not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0149] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described in this invention can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this invention can be embodied in the form of a software product, which can be stored in a computer-readable storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the method described above according to this invention. When the computer program is executed by a data processing device, it enables the computer-readable medium to implement the method described above, i.e.: as... Figure 1-3 The method shown.

[0150] Figure 7 This is a schematic diagram of a computer-readable medium provided for embodiments of this specification.

[0151] accomplish Figure 1-3The computer program of the method shown can be stored on one or more computer-readable media. A computer-readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0152] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0153] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0154] In summary, the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that in practice, general-purpose data processing devices such as microprocessors or digital signal processors (DSPs) can be used to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0155] In addition, the present invention also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements a high-value threat intelligence mining method based on feature orchestration and data fusion as described in any of the above embodiments.

[0156] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the present invention is not inherently related to any specific computer, virtual device, or electronic device, and various general-purpose devices can also implement the present invention. The above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0157] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0158] The above description is merely an embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application.

Claims

1. A method for mining high-value threat intelligence based on feature orchestration and data fusion, characterized in that, include: Obtain threat logs and filter and aggregate them according to basic attributes to obtain aggregated threat data; Key data features to be orchestrated are extracted from the threat data from both the attacker's and victim's perspectives. Two key data features are randomly selected from all the key features of the threat data and arranged in a concatenated manner to obtain multiple concatenated arrangement models. Under each concatenated arrangement model, the arrangement results are sorted to obtain the suspicious hosts under the concatenated arrangement from the perspectives of attackers and victims. Multiple parallel orchestration models are obtained by using all the key features and basic attributes of threat data as sorting criteria. Under each parallel orchestration model, the data are sorted according to the orchestration results to obtain suspicious hosts from the perspectives of attackers and victims in the parallel orchestration. Based on the weighted fusion algorithm, the suspicious hosts under serial arrangement and the suspicious hosts under parallel arrangement are weighted and fused to obtain the suspicious host sequences from the attacker's perspective and the victim's perspective, respectively. Obtain a fusion database, wherein the fusion database includes a detection source database, an intelligence source database, and an event source database; Obtain security protection system vendor information from the detection source database, merge the security protection system vendor information with the suspicious host information in the suspicious host sequence, and extract the associated vendor information to obtain the first fusion result; The threat intelligence information is obtained from the intelligence source database, and the threat intelligence information is fused with the suspicious host information in the suspicious host sequence. The second fusion result is obtained by determining whether each of the suspicious hosts is a malicious host. Historical event intelligence is obtained from the event source database. The historical event intelligence is fused with the suspicious host information in the suspicious host sequence to determine whether there are hosts with the same event intelligence, and a third fusion result is obtained. Based on the first, second, and third fusion results, the value of each suspicious host is calculated from both the attacker's and victim's perspectives. The suspicious hosts are ranked according to their value, and high-value threat intelligence is generated based on the ranking results.

2. The method for mining high-value threat intelligence based on feature orchestration and data fusion according to claim 1, characterized in that, The suspicious host can be identified by its IP address; The calculation expression for weighted fusion calculation of suspicious hosts under cascaded and parallel orchestration is as follows: in, x A represents the IP address of the host; A represents the total number of serial orchestration models, and B represents the total number of parallel orchestration models. α i and β i They represent the first i The weights of the serial and parallel orchestration models; s ix Indicates that the IP address is x Is the host in the serial model? s i middle; p ix The parallel model is represented in the same way as the series model; C x Indicates that the IP address is x The final suspiciousness of the host; dec Indicates descending order; This indicates taking the first N host IP addresses in descending order; R represents the final suspicious host IP address.

3. The method for mining high-value threat intelligence based on feature orchestration and data fusion according to claim 2, characterized in that, The calculation expression for the value of each suspicious host is as follows: Among them, the x Indicates the host's IP address; C x Indicates that the IP address is x The host outputs the suspiciousness level in the feature orchestration module; n represents the number of fused databases; δ i They represent the first i Credibility weights for each integrated database; l ix Indicates that the IP address is x The host in the merged database l The fusion results in; V x Indicates that the IP address is x The ultimate value of the host.

4. The method for mining high-value threat intelligence based on feature orchestration and data fusion according to claim 3, characterized in that, After ranking the suspicious hosts according to the value and generating high-value threat intelligence based on the ranking results, the method further includes: The high-value threat intelligence is analyzed, and the weights of each orchestration model and each fusion database are adjusted based on the suspicious host information contained in the analysis results.

5. A high-value threat intelligence mining device based on feature orchestration and data fusion, characterized in that, The high-value threat intelligence mining device based on feature orchestration and data fusion includes: The filtering and aggregation module is used to obtain threat logs and filter and aggregate the threat logs according to basic attributes to obtain aggregated threat data. The feature extraction module randomly selects two key data features from all key features of the threat data and arranges them in a concatenated manner to obtain multiple concatenated arrangement models. Under each concatenated arrangement model, the arrangement results are sorted to obtain suspicious hosts from the attacker's perspective and the victim's perspective under the concatenated arrangement. All key features and basic attributes of the threat data are used as sorting criteria to arrange them separately to obtain multiple parallel arrangement models. Under each parallel arrangement model, the arrangement results are sorted to obtain suspicious hosts from the attacker's perspective and the victim's perspective under the parallel arrangement. Based on a weighted fusion algorithm, a weighted fusion calculation is performed on the suspicious hosts under the concatenated arrangement and the suspicious hosts under the parallel arrangement to obtain the suspicious host sequences from the attacker's perspective and the victim's perspective, respectively. The feature orchestration module is used to orchestrate the key data features based on feature orchestration rules, and to calculate a sequence of suspicious hosts from the attacker's perspective and the victim's perspective based on the orchestration results. An information fusion module is used to acquire a fusion database, which includes a detection source database, an intelligence source database, and an event source database; acquire security protection system vendor information from the detection source database, fuse the security protection system vendor information with the suspicious host information in the suspicious host sequence, and extract related vendor information to obtain a first fusion result; acquire threat intelligence information from the intelligence source database, fuse the threat intelligence information with the suspicious host information in the suspicious host sequence, and determine whether each suspicious host is a malicious host to obtain a second fusion result; acquire historical event intelligence from the event source database, fuse the historical event intelligence with the suspicious host information in the suspicious host sequence, and determine whether there are hosts with the same event intelligence to obtain a third fusion result; based on the first fusion result, the second fusion result, and the third fusion result, calculate the value of each suspicious host from the attacker's perspective and the victim's perspective, respectively; The intelligence generation module is used to sort the suspicious hosts according to the value, and generate high-value threat intelligence based on the sorting results.

6. A high-value threat intelligence mining device based on feature orchestration and data fusion, characterized in that, The high-value threat intelligence mining device based on feature orchestration and data fusion includes: a memory and at least one processor, wherein the memory stores instructions; The at least one processor invokes the instructions in the memory to cause the high-value threat intelligence mining device based on feature orchestration and data fusion to perform the steps of the high-value threat intelligence mining method based on feature orchestration and data fusion as described in any one of claims 1-4.

7. A computer-readable storage medium storing a computer program / instructions thereon, characterized in that, When the program / instruction is executed by the processor, it implements the steps of the high-value threat intelligence mining method based on feature orchestration and data fusion as described in any one of claims 1-4.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, the steps of the high-value threat intelligence mining method based on feature orchestration and data fusion as described in any one of claims 1-4 are implemented.

Images