Multi-source information conflict-oriented threat intelligence credibility analysis method

Abstract

The invention provides a threat intelligence credibility analysis method oriented to multi-source information conflicts. The threat intelligence credibility analysis method comprises the following steps: acquiring threat intelligence data of different sources in the same IP; the method comprises the following steps: extracting time, places and attack types for describing attack behaviors adopted by attackers in intelligence data, measuring the credibility of intelligence, coding the attack time, places and attack types, making training materials of the intelligence data by combining the credibility, and importing the training materials into a classification learning model to carry out deep learning. The learning model can finally and accurately classify new information data and judge the credibility of the new information data; scoring the authority of the intelligence source through a large language model, and generating an answer credibility result by using a retrieval enhancement generation technology; and outputting a credibility judgment result which is more credible under the same IP. According to the method, the quality of the multi-source threat intelligence data can be comprehensively evaluated, intelligence conflicts are found by detecting the consistency among multiple intelligence, and low-quality data are eliminated.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligence analysis, and in particular relates to a threat intelligence credibility analysis method oriented to multi-source information conflicts. Background Art

[0002] Faced with increasingly complex and diverse cyber threats, threat intelligence from a single source can no longer meet the needs of modern security defense. Therefore, it is particularly urgent to efficiently integrate multi-source heterogeneous threat intelligence to achieve deep insight into potential threats and rapid response. In the process of collecting multi-source intelligence data, due to problems such as low value density of data, high intelligence duplication rate and short intelligence timeliness, there is a lot of confusion and invalid erroneous intelligence, which makes it difficult to find truly valuable and correct threat intelligence. There are differences in threat levels and confidence levels of intelligence from different sources, especially the false alarm phenomenon of open source website intelligence is more serious. In response to the conflict between intelligence from different sources in terms of threat level and confidence, the existing mechanisms mainly adopt voting, one-vote voting, one-vote veto, etc. Although they solve the problem of multi-source intelligence conflicts to a certain extent, they rely more on the experience and knowledge of experts, have certain subjectivity and limitations, and lack systematic and quantitative standards.

[0003] The information credibility 3S (Semantic, Surface, and Source features) model and the multi-source collaboration of intelligence sharing proposed a multi-dimensional analysis method for the credibility of open source threat intelligence. On this basis, features were extracted from four dimensions: intelligence source, intelligence content, active cycle, and blacklist library matching degree, as the basis for intelligence reputation analysis, and a set of intelligence quality evaluation models based on deep neural network algorithms and Softmax classifiers were designed. However, the extraction of these feature indicators is not comprehensive enough, and only the quantitative evaluation of the intelligence content dimension is considered. Summary of the invention

[0004] The purpose of the present invention is to provide a threat intelligence credibility analysis method for multi-source information conflict, aiming to solve the problem that the existing methods are not comprehensive in the evaluation dimensions of intelligence.

[0005] The present invention is implemented as follows: a threat intelligence credibility analysis method for multi-source information conflict, the method comprising the following steps: Step S1: Obtain threat intelligence data from different sources in the same IP; Step S2: If there is a conflict in the malicious determination fields in the threat intelligence from different sources, execute step S3; if the malicious determination fields in the threat intelligence from different sources are the same, execute step S4; Step S3: extract the time, location and attack type of the attacker's attack behavior described in the intelligence data, measure the credibility of the intelligence, encode the attack time, location and attack type, combine the credibility to make training materials for intelligence data, import them into the classification learning model for deep learning, and the learning model can finally accurately classify the new intelligence data and judge its credibility; Step S4: Use a large language model to score the authority of the intelligence source, use retrieval enhancement generation technology to combine the traditional language model with the external database, and generate a credibility result based on the retrieved information; Step S5: Combine the credibility of step S3 and step S4 to output a more credible credibility determination result under the same IP.

[0006] A further technical solution of the present invention is: in step S2, the intelligence source and the attack type are mapped to different dimensions using onehot encoding.

[0007] A further technical solution of the present invention is: in step S3, in order to enable the classification learning model to learn to distinguish the reliability of intelligence, the training data needs to be labeled with the credibility classification of each piece of intelligence, and the credibility is divided into credible, unreliable and uncertain.

[0008] A further technical solution of the present invention is: in step S4, the large language model retrieves relevant expert knowledge as an evaluation reference, and generates relevant information of the intelligence source according to its own capabilities, and gives a final evaluation result based on these reference knowledge and relevant information.

[0009] A further technical solution of the present invention is: in step S4, a component of the retriever is used in the retrieval enhancement generation to identify and retrieve relevant information or documents from the knowledge base, the knowledge base is an information library used by the searcher to find relevant data or documents, the context conversion module formulates answers that are directly related to the content found in the knowledge base, and the intelligence source authority evaluation module outputs the final credibility evaluation result.

[0010] A further technical solution of the present invention is: if the intelligence content provided by the intelligence source is highly matched with the actual situation, and the intelligence source has a good historical performance, the intelligence credibility score in step S3 will be correspondingly improved.

[0011] A further technical solution of the present invention is: if the intelligence source is considered to be more reliable, the intelligence data content provided by it will be assigned a higher credibility score in step S4.

[0012] The beneficial effects of the present invention are as follows: the present invention can comprehensively evaluate the quality of multi-source threat intelligence data, discover intelligence conflicts by detecting the consistency between multiple intelligences, and eliminate low-quality data. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a main flow chart provided by an embodiment of the present invention; Figure 2 This is the threat intelligence credibility analysis process based on the intelligence content dimension of the present invention; Figure 3 This is the threat intelligence credibility analysis process based on the intelligence source dimension of the present invention; Figure 4 It is a relationship diagram of the threat intelligence credibility analysis stages of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0015] Figure 1 A threat intelligence credibility analysis method for multi-source information conflict provided by the present invention is shown, and the method comprises the following steps: Step S1: Obtain threat intelligence data from different sources in the same IP; Step S2: If there is a conflict in the malicious determination fields in the threat intelligence from different sources, execute step S3; if the malicious determination fields in the threat intelligence from different sources are the same, execute step S4; Step S3: extract the time, location and attack type of the attacker's attack behavior described in the intelligence data, measure the credibility of the intelligence, encode the attack time, location and attack type, combine the credibility to make training materials for intelligence data, import them into the classification learning model for deep learning, and the learning model can finally accurately classify the new intelligence data and judge its credibility; Step S4: Use a large language model to score the authority of the intelligence source, use retrieval enhancement generation technology to combine the traditional language model with the external database, and generate a credibility result based on the retrieved information; Step S5: Combine the credibility of step S3 and step S4 to output a more credible credibility determination result under the same IP.

[0016] Preferably, in step S2, the intelligence source and attack type are mapped to different dimensions using onehot encoding.

[0017] Preferably, in step S3, in order to enable the classification learning model to learn to distinguish the reliability of intelligence, the training data needs to be labeled with the credibility classification of each piece of intelligence, and the credibility is divided into credible, unreliable and uncertain.

[0018] Preferably, in step S4, the large language model retrieves relevant expert knowledge as an evaluation reference, and generates relevant information of the intelligence source according to its own capabilities, and gives a final evaluation result based on these reference knowledge and relevant information.

[0019] Preferably, in step S4, the retrieval enhancement generation utilizes a component of a retriever to identify and retrieve relevant information or documents from a knowledge base, where the knowledge base is an information repository used by the retriever to find relevant data or documents, and the context conversion module formulates answers that are directly related to the content found in the knowledge base, and the intelligence source authority evaluation module outputs the final credibility evaluation result.

[0020] Preferably, if the intelligence content provided by the intelligence source is highly consistent with the actual situation and the intelligence source has a good historical performance, the intelligence credibility score in step S3 will be increased accordingly.

[0021] Preferably, if the intelligence source is considered more reliable, the intelligence data content provided by it will be assigned a higher credibility score in step S4.

[0022] The present invention proposes a threat intelligence credibility analysis method for multi-source information conflicts, which can comprehensively evaluate the quality of multi-source threat intelligence data, discover intelligence conflicts by detecting the consistency between multiple intelligences, and eliminate low-quality data. The method relates to the field of computer technology, especially network information security fields such as threat intelligence and credibility evaluation.

[0023] In order to further improve the reliability of threat intelligence, the invention mainly examines the credibility of threat intelligence from two dimensions: the intelligence content description itself and the intelligence source, combining machine learning and the characteristics of intelligence data.

[0024] From the perspective of intelligence content, the intelligence content describes the time, location and attack method of the attacker's attack behavior. A typical intelligence structure should contain information such as intelligence source, timestamp, category value (IP or domain name). According to the above intelligence structure, multi-faceted characteristic indicators are mined from multiple dimensions to measure the credibility of intelligence content, and a machine learning algorithm is used to train a quality classification model, thereby improving the comprehensiveness of threat intelligence credibility analysis. From the perspective of intelligence source, the source of intelligence can largely reflect the credibility of an intelligence. Generally speaking, multi-source intelligence is of higher quality than single-source intelligence, and well-known threat intelligence vendors and professional intelligence assessment agencies are of higher quality than personal intelligence data. At present, intelligence sources include government security agencies, security organizations or teams, supplier websites, well-known sites or forums, personal blogs, and other unknown sources. The evaluation is weighted and ranked according to its authority. In addition, in addition to being true and correct, threat intelligence must have a customized and comprehensive analysis of the threat. Therefore, the present invention also analyzes its comprehensiveness from the perspective of domain knowledge of the threat intelligence structure provided by the intelligence source.

[0025] Among them, step S3 involves abstractly expressing it by means of feature coding. The intelligence content describes the time, place and attack method of the attacker's attack behavior, which directly reflects the quality of threat intelligence. The release time of intelligence directly affects its timeliness. Timely released intelligence usually reflects the latest attack activities; the attack categories in the intelligence can more accurately assess the potential threats of each attack, which in turn affects the credibility of the intelligence. Geographical location features, such as the latitude and longitude of the IP address, can reveal background information about the source of the attack. If the source of the attack is located in an area where malicious activities are frequent, the credibility increases accordingly. Therefore, based on the content structure of the intelligence, the credibility of the intelligence is measured from multiple aspects such as the release time, attack category, and geographic location. The following table provides an example of feature coding of intelligence content: Symbol (Feature) describe Example Source Intelligence Sources 000100 <![CDATA[T latest ]]> Information latest release time 1 559 638 122 115 Attack_type Attack Types 100000010000 confidence Confidence 0.5 The intelligence source and attack type use onehot encoding to map different types of attack methods to different dimensions. By learning the potential abstract features of multi-dimensional credible features, we train a quality assessment model for intelligence content. Figure 2 As shown, to train the learning model, threat intelligence data sets are collected from multiple sources, which contain detailed content features of each intelligence. Each intelligence record will be feature extracted and encoded based on these features for training the classification model. In order for the model to learn to distinguish the reliability of intelligence, the training data needs to be labeled with the credibility classification of each intelligence (for example, "credible", "uncredible" or "uncertain"). The final evaluation effect of the learning model is to be able to accurately classify new intelligence data and judge it as "credible", "uncredible" or "uncertain". Ideally, the model should have high classification accuracy and recall, be able to capture the relationship between complex features, and achieve effective credibility assessment of intelligence content, thereby helping users identify high-quality threat intelligence information.

[0026] In step S4, based on the source of intelligence, a large language model (LLM) is used to give a score to the authority of the intelligence source. LLM retrieves relevant expert knowledge as an evaluation reference and generates relevant information about the intelligence source based on its own capabilities. The final evaluation result is given based on these reference knowledge and related information. Retrieval-Augmented Generation (RAG) combines traditional language models with external databases. RAG uses a retriever to retrieve relevant information and a generator to generate answers based on the retrieved information. This improves accuracy and relevance, especially for queries in specific areas, such as threat intelligence. Figure 3As shown in the figure, the retriever is the component that identifies and retrieves relevant information or documents from the knowledge base, which is crucial to providing the context and content required for LLM to generate accurate and informed answers; the knowledge base is the information repository used by the retriever to find relevant data or documents; in addition, the model contextual information is used to formulate answers that are accurate, comprehensive, and directly related to the content found in the knowledge base.

[0027] The present invention adopts GPT-4 as the core large-scale language model. GPT-4 has powerful natural language understanding capabilities and can generate high-quality reference information in combination with contextual knowledge. At the same time, in the RAG architecture, the present invention designs a mechanism for multiple retrievers to search multiple knowledge bases in parallel, which significantly improves the retrieval efficiency and evaluation accuracy. Each retriever focuses on different types of knowledge bases (such as paper databases, threat intelligence databases, etc.) and stores relevant intelligence information through ChromaDB. The retriever uses vector retrieval technology to extract highly relevant information from different knowledge bases through multi-threaded parallel processing of query requests. The retrieval results are deduplicated and merged according to the relevance score, and finally GPT-4 integrates to generate a complete context, giving the authority ranking results and analysis conclusions of the intelligence source.

[0028] In addition, the two stages of intelligence source and intelligence content evaluation also have feedback and influence on each other. Figure 4 As shown in the figure, the evaluation results of intelligence content (such as attack type, geographic location, release time, etc.) will affect the credibility of the intelligence source. For example, if the APT attack intelligence provided by an intelligence source is highly consistent with the actual situation, and the source has a good historical performance, its credibility score will be increased accordingly. Then, the credibility of the intelligence source will in turn affect the evaluation of the intelligence content. If a source is considered more reliable, the intelligence it provides will be given a higher credibility score. Through this mutual feedback mechanism, the evaluation system can dynamically adjust and give greater weight to highly credible intelligence sources and intelligence content, thereby improving the quality and accuracy of the overall threat intelligence and realizing a more dynamic evaluation mechanism.

[0029] The credibility evaluation of threat intelligence is essentially to explore the quality of information. The present invention measures its quality by using a credibility evaluation model, resolves possible conflicts and contradictions in intelligence from different sources, and eliminates low-quality intelligence. The evaluation of intelligence sources is currently in a relatively subjective stage. The present invention uses a large language model to score the authority of intelligence sources, retrieves relevant knowledge as an evaluation reference, and generates relevant information about intelligence sources based on its own capabilities, and then gives the final evaluation results based on these reference knowledge and related information.

[0030] The present invention is the first to solve the conflict problem of multi-source threat intelligence by quantitatively evaluating the credibility of threat intelligence. The existing solutions mainly use voting, one-vote voting, one-vote veto, etc., which are more dependent on the experience and knowledge of experts and have certain subjectivity and limitations; the present invention uses different models to evaluate from two dimensions of threat intelligence: intelligence content and intelligence source, and feedback influences each other. Although the existing technology mentions the evaluation of the intelligence source dimension, it is mainly used as part of the intelligence content evaluation or based on expert knowledge, and cannot form a comprehensive and dynamic evaluation; the present invention uses retrieval enhancement generation technology to evaluate the threat intelligence source for the first time. Existing studies give discrete recommended scores based on expert knowledge and directly score the intelligence source. This method is relatively simple and direct, but it will result in low differentiation between different sources. The present invention uses a large language model as a score for the authority of the intelligence source, retrieves relevant knowledge as an evaluation reference, and generates relevant information of the intelligence source according to its own capabilities, and then gives the final evaluation result based on these reference knowledge and related information, reducing the dependence on expert knowledge.

[0031] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A threat intelligence credibility analysis method for multi-source information conflict, characterized in that: The method comprises the following steps: Step S1: Obtain threat intelligence data from different sources in the same IP; Step S2: If there is a conflict in the malicious determination fields in the threat intelligence from different sources, execute step S3; if the malicious determination fields in the threat intelligence from different sources are the same, execute step S4; Step S3: extract the time, location and attack type of the attacker's attack behavior described in the intelligence data, measure the credibility of the intelligence, encode the attack time, location and attack type, combine the credibility to make training materials for intelligence data, import them into the classification learning model for deep learning, and the learning model can finally accurately classify the new intelligence data and judge its credibility; Step S4: Use a large language model to score the authority of the intelligence source, use retrieval enhancement generation technology to combine the traditional language model with the external database, and generate a credibility result based on the retrieved information; Step S5: Combine the credibility of step S3 and step S4 to output a more credible credibility determination result under the same IP.

2. The threat intelligence credibility analysis method for multi-source information conflict according to claim 1 is characterized in that: In step S2, the intelligence sources and attack types are mapped to different dimensions using onehot encoding.

3. The threat intelligence credibility analysis method for multi-source information conflict according to claim 1 is characterized in that: In step S3, in order to enable the classification learning model to learn to distinguish the reliability of intelligence, the training data needs to be labeled with the credibility classification of each piece of intelligence, and the credibility is divided into credible, unreliable, and uncertain.

4. The threat intelligence credibility analysis method for multi-source information conflict according to claim 1 is characterized in that: In step S4, the large language model retrieves relevant expert knowledge as an evaluation reference, and generates relevant information of the intelligence source according to its own capabilities, and gives the final evaluation result based on these reference knowledge and related information.

5. The threat intelligence credibility analysis method for multi-source information conflict according to claim 1 is characterized in that: In step S4, the retriever component is used in the retrieval enhancement generation to identify and retrieve relevant information or documents from the knowledge base. The knowledge base is an information library used by the retriever to find relevant data or documents. The context conversion module formulates answers that are directly related to the content found in the knowledge base, and the intelligence source authority evaluation module outputs the final credibility evaluation result.

6. The threat intelligence credibility analysis method for multi-source information conflict according to claim 1 is characterized in that: If the intelligence content provided by the intelligence source is highly consistent with the actual situation and the intelligence source has a good historical performance, the intelligence credibility score in step S3 will be increased accordingly.

7. The threat intelligence credibility analysis method for multi-source information conflict according to claim 1 is characterized in that: If the intelligence source is considered more reliable, the intelligence data content it provides will be assigned a higher credibility score in step S4.

Images