Threat intelligence information verification method and device, electronic equipment and storage medium

Abstract

The invention discloses a threat intelligence information verification method and device, electronic equipment and a storage medium, which are used for solving the problem of low accuracy of threat intelligence information verification, and the method comprises the following steps: obtaining to-be-verified threat intelligence information; extracting threat elements in the threat intelligence information to be verified; determining a first triple corresponding to the to-be-verified threat intelligence information based on the threat elements, wherein the first triple comprises first threat elements which have association relationships and are of entity types and second threat elements corresponding to the association relationships between the first threat elements; obtaining verification reference information corresponding to the to-be-verified threat intelligence information from at least two data sources according to the first triple, wherein the data sources comprise a network security knowledge graph, a knowledge base and a preset credible threat intelligence website; and generating a threat verification cue word according to the verification reference information obtained from the at least two data sources, and inputting the first triple and the threat verification cue word into the large language model to obtain a verification result of the to-be-verified threat intelligence information.

Description

Technical Field

[0001] This application relates to the field of information security technology, and in particular, to a method, apparatus, electronic device, and storage medium for verifying threat intelligence information. Background Art

[0002] Cyber threat intelligence is information about potential threats to network security and infrastructure collected and analyzed by organizations. It helps predict threats, enhance defense plans, improve incident response, etc. The collection of cyber threat intelligence is an important source for cybersecurity researchers to keep abreast of threat dynamics in a timely manner, and the processing and learning of threat intelligence data are also the main methods for training threat detection models.

[0003] However, with the increasingly frequent attacks by attacking organizations, anti-detection means have also been increasingly improved, resulting in a large number of false and inaccurate threat intelligence information flooding the Internet. If the threat intelligence information obtained from the Internet is received without discrimination, it will not only weaken the recognition accuracy of the threat detection model, but also mislead security analysts to make wrong judgments. And screening all threat intelligence information one by one will impose a great workload on security analysts. The existing Advanced Persistent Threat (APT) data is large in quantity, with a wide variety of attack means and exploited vulnerability types, and it is difficult for security analysts to comprehensively cover all content with their knowledge. Therefore, as a key environment for cybersecurity risk analysis, verifying a large amount of threat intelligence information is not only beneficial for security analysts to quickly analyze threat intelligence information, improve work efficiency, and make effective decisions, but also the verified threat intelligence information can be used as high-quality training data for threat detection models, avoiding the threat detection model from learning wrong information and improving the detection accuracy of the threat detection model.

[0004] In related technologies, threat intelligence information verification methods mainly perform entity and relationship matching or text similarity calculation based on a knowledge graph. However, these methods are limited by the singularity of knowledge sources and have the problem of incomplete knowledge coverage, resulting in low accuracy of threat intelligence information verification. Summary of the Invention

[0005] To solve the problem of low accuracy of threat intelligence information verification, embodiments of this application provide a method, apparatus, electronic device, and storage medium for verifying threat intelligence information.

[0006] In a first aspect, embodiments of this application provide a method for verifying threat intelligence information, including:

[0007] Obtain threat intelligence information to be verified;

[0008] Extract the threat elements included in the threat intelligence information to be verified, where the threat elements are key information for threat analysis of the threat intelligence information to be detected;

[0009] Based on the threat elements, determine a first triple corresponding to the threat intelligence information to be verified, where the first triple includes a first threat element of entity type with an associated relationship and a second threat element corresponding to the associated relationship between the first threat elements of entity type;

[0010] According to the first triple, obtain verification reference information corresponding to the threat intelligence information to be verified from at least two data sources, where the data sources include a cybersecurity knowledge graph, a knowledge base, and a preset trusted threat intelligence website;

[0011] Generate a threat verification prompt word according to the verification reference information obtained from the at least two data sources, and input the first triple and the threat verification prompt word into a large language model to obtain a verification result of the threat intelligence information to be verified, where the large language model is used to generate a verification result of the threat intelligence information to be verified based on the threat verification prompt word.

[0012] In one implementation, the method further includes:

[0013] Generate a verification evaluation report prompt word according to the verification result of the threat intelligence information to be verified;

[0014] Input the threat intelligence information to be verified and the verification evaluation report prompt word into the large language model to obtain a verification evaluation report of the threat intelligence information to be verified, where the large language model is also used to generate a verification evaluation report of the threat intelligence information to be verified based on the verification evaluation report prompt word.

[0015] In one implementation, obtaining the verification reference information corresponding to the threat intelligence information to be verified from the cybersecurity knowledge graph according to the first triple specifically includes:

[0016] Query a subgraph within a preset hop count of the entity from the cybersecurity knowledge graph according to the entity in the first triple;

[0017] Extract each second triple included in the subgraph to obtain a second triple set;

[0018] Use the second triple set as the first verification reference information corresponding to the threat intelligence information to be verified.

[0019] In one embodiment, the knowledge base includes a vector database, and the vector database contains the corresponding relationships of text blocks, text block vectors, text block summaries, text block summary vectors, and metadata, where the text blocks are the text blocks included in the verified threat intelligence text, and the metadata is the key information extracted from the text blocks for threat analysis;

[0020] Obtaining the verification reference information corresponding to the to-be-verified threat intelligence information from the knowledge base according to the first triple specifically includes:

[0021] Converting the first triple into a corresponding to-be-verified natural language description text;

[0022] Inputting the to-be-verified natural language description text into a text embedding model to obtain a to-be-verified natural language description text vector;

[0023] Determining the similarity between the to-be-verified natural language description text vector and each text block vector in the vector database, and determining the similarity between the to-be-verified natural language description text vector and each text block summary vector;

[0024] Taking the text blocks, text block summaries, and metadata corresponding to the preset number of text block vectors or text block summary vectors with the highest similarity to the to-be-verified natural language description text vector as the second verification reference information corresponding to the to-be-verified threat intelligence information.

[0025] In one embodiment, the knowledge base further includes a document database, and the document database contains the corresponding relationships of the text blocks and the metadata;

[0026] Obtaining the verification reference information corresponding to the to-be-verified threat intelligence information from the knowledge base according to the first triple specifically includes:

[0027] Querying and matching metadata from the document database according to the first threat element and the second threat element in the first triple;

[0028] Taking the first metadata and the corresponding text blocks that match the first threat element, and the second metadata and the corresponding text blocks that match the second threat element as the third verification reference information corresponding to the to-be-verified threat intelligence information.

[0029] In one embodiment, obtaining the verification reference information corresponding to the to-be-verified threat intelligence information from the preset trusted threat intelligence website according to the first triple specifically includes:

[0030] Retrieve target threat intelligence information containing the first threat element and / or the second threat element from the preset trusted threat intelligence website according to the first threat element and the second threat element in the first triple;

[0031] Use the target threat intelligence information as the fourth verification reference information corresponding to the threat intelligence information to be verified.

[0032] In one implementation, before obtaining the threat intelligence information to be verified, it further includes:

[0033] Obtain the threat intelligence text;

[0034] For each threat intelligence text, split the threat intelligence text into multiple text blocks;

[0035] For each text block, if it is determined that the text block contains a threat element, extract the threat element in the text block and use the threat element in the text block as the metadata of the text block;

[0036] Input the text block and the preset summary prompt word into the large language model to obtain a text block summary;

[0037] Input the text block and the text block summary into the text embedding model respectively to obtain a text block vector and a text summary vector;

[0038] Store the text block, the text block vector, the text block summary, the text block summary vector, and the corresponding metadata into the vector database;

[0039] Store the text block and the corresponding metadata into the document database.

[0040] In a second aspect, an embodiment of the present application provides a threat intelligence information verification device, including:

[0041] A first acquisition module, configured to acquire threat intelligence information to be verified;

[0042] A first extraction module, configured to extract threat elements included in the threat intelligence information to be verified, where the threat elements are key information for performing threat analysis on the threat intelligence information to be detected;

[0043] A determination module, configured to determine a first triple corresponding to the threat intelligence information to be verified based on the threat elements, where the first triple includes a first threat element of entity type with an association relationship, and a second threat element corresponding to the association relationship between the first threat elements of entity type;

[0044] A second acquisition module, configured to acquire verification reference information corresponding to the to-be-verified threat intelligence information from at least two data sources respectively according to the first triple, wherein the data sources include a cybersecurity knowledge graph, a knowledge base, and a preset trusted threat intelligence website;

[0045] A verification module, configured to generate a threat verification prompt word according to the verification reference information obtained from the at least two data sources, and input the first triple and the threat verification prompt word into a large language model to obtain a verification result of the to-be-verified threat intelligence information, where the large language model is used to generate the verification result of the to-be-verified threat intelligence information based on the threat verification prompt word.

[0046] In one implementation manner, the device further includes:

[0047] A prompt word generation module, configured to generate a verification evaluation report prompt word according to the verification result of the to-be-verified threat intelligence information;

[0048] A verification evaluation report generation module, configured to input the to-be-verified threat intelligence information and the verification evaluation report prompt word into the large language model to obtain a verification evaluation report of the to-be-verified threat intelligence information, where the large language model is further used to generate the verification evaluation report of the to-be-verified threat intelligence information based on the verification evaluation report prompt word.

[0049] In one implementation manner, the second acquisition module is specifically configured to retrieve, from the cybersecurity knowledge graph, the verification reference information corresponding to the to-be-verified threat intelligence information according to the first triple in the following manner: query, from the cybersecurity knowledge graph, a subgraph within a preset hop count of the entity according to the entity in the first triple; extract each second triple included in the subgraph to obtain a second triple set; and use the second triple set as the first verification reference information corresponding to the to-be-verified threat intelligence information.

[0050] In one implementation manner, the knowledge base includes a vector database, and the vector database contains the corresponding relationships among text blocks, text block vectors, text block summaries, text block summary vectors, and metadata, where the text blocks are text blocks included in the verified threat intelligence text, and the metadata is key information extracted from the text blocks for threat analysis;

[0051] The second acquisition module is specifically configured to retrieve the verification reference information corresponding to the to-be-verified threat intelligence information from the knowledge base in the following manner: convert the first triple into a corresponding to-be-verified natural language description text; input the to-be-verified natural language description text into a text embedding model to obtain a to-be-verified natural language description text vector; determine the similarity between the to-be-verified natural language description text vector and each text block vector in the vector database, and determine the similarity between the to-be-verified natural language description text vector and the summary vector of each text block; use the text blocks, text block summaries, and metadata corresponding to the preset number of text block vectors or text block summary vectors with the highest similarity to the to-be-verified natural language description text vector as the second verification reference information corresponding to the to-be-verified threat intelligence information.

[0052] In one implementation manner, the knowledge base further includes a document database, and the document database contains the corresponding relationship between the text blocks and the metadata;

[0053] The second acquisition module is specifically configured to retrieve the verification reference information corresponding to the to-be-verified threat intelligence information from the knowledge base in the following manner: query the matching metadata from the document database according to the first threat element and the second threat element in the first triple; use the first metadata and the corresponding text block that match the first threat element, and the second metadata and the corresponding text block that match the second threat element as the third verification reference information corresponding to the to-be-verified threat intelligence information.

[0054] In one implementation manner, the second acquisition module is specifically configured to retrieve the verification reference information corresponding to the to-be-verified threat intelligence information from the preset trusted threat intelligence website in the following manner: retrieve the target threat intelligence information containing the first threat element and / or the second threat element from the preset trusted threat intelligence website according to the first threat element and the second threat element in the first triple; use the target threat intelligence information as the fourth verification reference information corresponding to the to-be-verified threat intelligence information.

[0055] In one implementation manner, the device further includes:

[0056] A third acquisition module, configured to acquire the threat intelligence text before acquiring the to-be-verified threat intelligence information;

[0057] A segmentation module, configured to segment each threat intelligence text into multiple text blocks;

[0058] A second extraction module, configured to, for each text block, if it is determined that the text block contains threat elements, extract the threat elements in the text block and use the threat elements in the text block as the metadata of the text block;

[0059] An abstract generation module, configured to input the text block and a preset abstract prompt word into the large language model to obtain a text block abstract;

[0060] A text vector extraction module, configured to input the text block and the text block abstract into the text embedding model respectively to obtain a text block vector and a text abstract vector;

[0061] A first storage module, configured to store the text block, the text block vector, the text block abstract, the text block abstract vector and the corresponding metadata into the vector database;

[0062] A second storage module, configured to store the text block and the corresponding metadata into the document database.

[0063] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the threat intelligence information verification method described in the present application is implemented.

[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps in the threat intelligence information verification method described in the present application are implemented.

[0065] The beneficial effects of the present application are as follows:

[0066] The threat intelligence information verification method, device, electronic device, and storage medium provided by the embodiments of this application. The threat intelligence verification platform obtains the threat intelligence information to be verified, extracts the threat elements included in the threat intelligence information to be verified. The threat elements are the key information for threat analysis of the threat intelligence information to be detected. Based on the threat elements, the first triple corresponding to the threat intelligence information to be verified is determined. The first triple includes the first threat element of the entity type with an associated relationship and the second threat element corresponding to the associated relationship between the first threat elements of the entity type. According to the first triple, the verification reference information corresponding to the threat intelligence information to be verified is obtained from at least two data sources respectively. Among them, the data sources include a cybersecurity knowledge graph, a knowledge base, and a preset trusted threat intelligence website. The threat verification prompt words are generated according to the verification reference information obtained from at least two data sources, and the first triple and the threat verification prompt words are input into the large language model to obtain the verification result of the threat intelligence information to be verified. The large language model is used to generate the verification result of the threat intelligence information to be verified based on the threat verification prompt words. In the embodiments of this application, after the threat intelligence verification platform obtains the threat intelligence information to be verified, it extracts the key information for threat analysis included in the threat intelligence information to be verified, that is, the threat elements, and forms the first threat element of the entity type and the second threat element corresponding to the associated relationship between the first threat elements of the entity type into a triple in the form of <entity, relationship, entity> (that is, <first threat element, second threat element, first threat element>) according to the type of the extracted threat elements, which can be denoted as the first triple. The first triple is the triple corresponding to the threat intelligence information to be verified. According to the first triple, the verification reference information corresponding to the threat intelligence information to be verified is obtained from at least two data sources among the cybersecurity knowledge graph, the knowledge base, and the preset trusted threat intelligence website. The verification reference information is used to verify the first triple corresponding to the threat intelligence information to be verified. The threat verification prompt words are generated according to the obtained verification reference information and input into the large language model to assist in verifying the accuracy of the triple corresponding to the threat intelligence information to be verified, and the verification result of the threat intelligence information to be verified is obtained. Compared with the prior art, in order to ensure the comprehensiveness and accuracy of threat intelligence information verification, this application uses the Retrieval Augmented Generation (RAG) technology to automatically verify the threat intelligence information to be verified in combination with the large language model and multiple knowledge retrieval sources. The large language model has strong natural language understanding and generation capabilities, and the addition of multiple knowledge retrieval data sources provides more and more powerful background knowledge for the large language model, greatly reducing the possibility of the large language model outputting hallucinations. Thus, the accuracy and efficiency of threat intelligence information verification are improved.

[0067] Other features and advantages of the present application will be set forth in the following description, and in part will be obvious from the description, or may be learned by practice of the present application. The objectives and other advantages of the present application may be realized and attained by the structure particularly pointed out in the written description, claims, as well as the drawings. Description of the Drawings

[0068] The drawings described herein are provided to further understand the present application and form a part of the present application. The schematic embodiments and descriptions thereof of the present application are used to explain the present application and do not constitute an improper limitation to the present application. In the drawings:

[0069] Figure 1 It is a schematic diagram of an application scenario of the threat intelligence information verification method provided by an embodiment of the present application;

[0070] Figure 2 It is a schematic flowchart of the threat intelligence information verification method provided by an embodiment of the present application;

[0071] Figure 3 It is an example diagram of the threat intelligence information to be verified in JSON format provided by an embodiment of the present application;

[0072] Figure 4 It is an example connection diagram of the network security knowledge graph provided by an embodiment of the present application;

[0073] Figure 5 It is a schematic flowchart of the process for obtaining a vector database and a document database provided by an embodiment of the present application;

[0074] Figure 6 It is an example diagram of the preset summary prompt words provided by an embodiment of the present application;

[0075] Figure 7 It is a schematic flowchart of the process for obtaining verification reference information corresponding to the threat intelligence information to be verified from the network security knowledge graph provided by an embodiment of the present application;

[0076] Figure 8 It is a schematic flowchart of the process for obtaining verification reference information corresponding to the threat intelligence information to be verified from the vector database in the knowledge base provided by an embodiment of the present application;

[0077] Figure 9 It is a schematic flowchart of the process for obtaining verification reference information corresponding to the threat intelligence information to be verified from the document database in the knowledge base provided by an embodiment of the present application;

[0078] Figure 10 It is a schematic flowchart of the process for obtaining verification reference information corresponding to the threat intelligence information to be verified from a preset trusted threat intelligence website provided by an embodiment of the present application;

[0079] Figure 11(a) is a threat verification prompt word template provided by an embodiment of the present application;

[0080] Figure 11(b) is an example diagram of a threat verification prompt word provided by an embodiment of the present application;

[0081] Figure 12 It is a schematic flow diagram of generating a verification evaluation report for threat intelligence information to be verified provided by an embodiment of the present application;

[0082] Figure 13(a) is a verification evaluation report prompt word template provided by an embodiment of the present application;

[0083] Figure 13(b) is an example diagram of a verification evaluation report prompt word provided by an embodiment of the present application;

[0084] Figure 14 It is a schematic structural diagram of a threat intelligence information verification device provided by an embodiment of the present application;

[0085] Figure 15 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0086] To solve the problem of low accuracy in threat intelligence information verification, an embodiment of the present application provides a threat intelligence information verification method, device, electronic device, and storage medium.

[0087] The following describes the preferred embodiments of the present application with reference to the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. And without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0088] In this article, it should be understood that in the technical terms involved in the present application:

[0089] 1. Network security knowledge graph: A knowledge graph is a structured semantic knowledge base that formally describes the entities in the real world and their relationships. A knowledge graph is generally represented by triples, and forms a graphical structure through nodes (entities) and edges (relationships) to form a semantic network, which can effectively represent complex knowledge and information. The basic relationship forms of triples are mainly <concept, attribute, attribute value> and <entity, relationship, entity>, etc.

[0090] The network security knowledge graph targets knowledge in the field of network security, and the described entities and relationships also mainly target the field of network security. The entities include threat actors, attack locations, attack means, vulnerabilities, malware, etc., and the relationships include targeting, attacking, using, exploiting, downloading, etc.

[0091] 2. Knowledge Base: A system for storing and managing knowledge, typically used to support information retrieval and decision-making in a specific domain. By storing various types of information, it aims to help users quickly find the knowledge they need. A knowledge graph can be regarded as a special type of knowledge base.

[0092] 3. Large Language Model (LLM): An artificial intelligence model based on deep learning, used to understand and generate natural language. Large language models typically adopt the Transformer architecture and utilize the self-attention mechanism to effectively capture the context relationships in text. Usually, the number of parameters in large language models is above the billions level.

[0093] 4. Retrieval-Augmented Generation: An artificial intelligence method that combines information retrieval technology with a generation model, aiming to improve the accuracy and relevance of large language models when dealing with knowledge-intensive tasks.

[0094] 5. Large Language Model Hallucination Problem: Refers to the situation where large language models may generate information that is inconsistent with reality or logically incorrect when generating text. This problem not only affects the reliability and credibility of the model but may also cause adverse consequences in practical applications.

[0095] 6. Prompt: In the fields of Natural Language Processing (NLP) and artificial intelligence, it refers to the text input by the user to the model, used to guide the model to generate the required output. It can be a question, a description, or other forms of instructions, aiming to provide context and guidance to the model so that the model can understand the user's intention and generate a response that meets expectations.

[0096] 7. Large Model Fine-tuning: Refers to the process of further training a pre-trained large language model using a specific dataset so that the model can better adapt to a specific task or domain.

[0097] 8. Structured Threat Information Expression (STIX): It is a language used to standardize, acquire, represent, and communicate standardized cyber threat information. It provides support for more effective cyber threat management processes and application automation in a structured manner, enabling different organizations to share cyber threat intelligence in a consistent and machine-readable format. It solves the problems existing in information exchange in the field of cybersecurity, describes threat intelligence in a clear and standardized way, and thus improves the accuracy and efficiency of security analysis. That is, STIX is a structured language for describing cyber threat information, so it can be shared, stored, and analyzed in a consistent manner.

[0098] 9. NVD (National Vulnerability Database): It is a publicly available database maintained by the National Institute of Standards and Technology (NIST) in the United States, aiming to collect, organize, and publish information about computer security vulnerabilities. NVD is one of the most comprehensive and authoritative vulnerability databases globally and is widely used by security researchers, software developers, system administrators, and other relevant personnel to identify and mitigate potential security threats. NVD collects vulnerability information from multiple sources, including but not limited to the CVE (Common Vulnerabilities and Exposures) list, CERT (Computer Emergency Response Team) announcements, and other public security announcements. Each vulnerability entry contains detailed descriptions, scope of impact, severity scores (CVSS, Common Vulnerability Scoring System), known solutions, and repair suggestions, etc. Users can directly browse and search for vulnerability information by accessing the NVD official website. Moreover, NVD provides a RESTful (Representational State Transfer) API (Application Programming Interface) for developers to call to obtain the latest vulnerability data.

[0099] 10. CNVD (China National Vulnerability Database): It is an open information security vulnerability information sharing knowledge base established by the National Computer Network Emergency Response Technical Team / Coordination Center of China (abbreviated as CNCERT / CC in Chinese and CNCERT in English) in conjunction with important domestic information system units, basic telecommunications operators, network security manufacturers, software manufacturers, and Internet enterprises. It aims to collect, organize, and publish information on computer security vulnerabilities to help domestic enterprises, institutions, and individual users identify and mitigate potential security threats.

[0100] CNVD collects vulnerability information from multiple sources, including but not limited to CVE, domestic security bulletins, and other publicly available security reports. Each vulnerability entry contains detailed descriptions, scope of impact, severity scores, known solutions, and repair suggestions, etc. Compared with NVD, in addition to citing international CVE data, CNVD also collects a lot of domestic-specific vulnerability information. CNVD provides an API that allows third-party applications to call and query the latest vulnerability information in real time, which enables developers to integrate vulnerability scanning and management functions into their own tools, improving work efficiency.

[0101] 11. CNNVD (China National Vulnerability Database): It is an open vulnerability database constructed and maintained by the National Information Security Vulnerability Database Management Office. It aims to collect, organize, and publish information on computer security vulnerabilities to help Chinese enterprises, institutions, and individual users identify and mitigate potential security threats. CNNVD collects vulnerability information from multiple sources, including but not limited to CVE, domestic security bulletins, and other publicly available security reports. Each vulnerability entry contains detailed descriptions, scope of impact, severity scores, known solutions, and repair suggestions, etc. CNNVD provides an API that allows third-party applications to call and query the latest vulnerability information in real time, which enables developers to integrate vulnerability scanning and management functions into their own tools, improving work efficiency. CNNVD focuses on the comprehensive collection and publication of information security vulnerabilities, including domestic and foreign vulnerability information. While CNVD mainly focuses on software and hardware products within China, especially some applications and services widely used in the country. In addition to citing international CVE data, CNNVD also collects a lot of domestic-specific vulnerability information, while NVD mainly relies on CVE and other international security bulletins.

[0102] 12. ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge): It is a globally recognized knowledge base and framework based on real-world observed cyber-attack behaviors. It is developed and maintained by MITRE Corporation, aiming to help organizations understand, detect, respond to, and simulate various cyber threats. ATT&CK describes in a structured way the tactics, techniques, and procedures used by attackers at different stages, and provides detailed cases and reference information. Security teams can use the ATT&CK framework to analyze the tactics and techniques of known attackers, understand their behavior patterns, and predict future attack trends. By integrating the ATT&CK framework into the work processes of a Security Operations Center (SOC), it can help security analysts identify and respond to potential security threats more quickly.

[0103] First, refer to Figure 1, which is a schematic diagram of an application scenario of the threat intelligence information verification method provided by an embodiment of the present application. It may include a threat intelligence verification platform 101, a database 102, a knowledge base 103, and a trusted threat intelligence website 104. The threat intelligence verification platform 101 is connected to the database 102, the knowledge base 103, and the trusted threat intelligence website 104 through a network respectively. The threat intelligence verification platform 101 includes a threat intelligence information extraction module 1011, a threat intelligence information verification module 1012, and a threat intelligence information verification and evaluation report generation module 1013. The database 102 stores a network security knowledge graph. The knowledge base 103 includes a vector database 1031 and a document database 1032. The threat intelligence information extraction module 1011 in the threat intelligence verification platform 101 obtains the threat intelligence information to be verified and extracts the threat elements included in the threat intelligence information to be verified. The threat elements are the key information for threat analysis of the threat intelligence information to be detected. The threat intelligence information verification module 1012 determines the first triple corresponding to the threat intelligence information to be verified based on the threat elements included in the threat intelligence information to be verified. The first triple includes the first threat element of the entity type with an association relationship and the second threat element corresponding to the association relationship between the first threat elements of the entity type. According to the first triple, the verification reference information corresponding to the threat intelligence information to be verified is obtained from at least two data sources respectively. Among them, the data sources include a network security knowledge graph, a knowledge base, and a preset trusted threat intelligence website. Furthermore, a threat verification prompt word is generated according to the verification reference information obtained from at least two data sources, and the first triple and the threat verification prompt word are input into a large language model to obtain the verification result of the threat intelligence information to be verified. The large language model is used to generate the verification result of the threat intelligence information to be verified based on the threat verification prompt word. Furthermore, the threat intelligence information verification and evaluation report generation module 1013 generates a verification and evaluation report prompt word according to the verification result of the threat intelligence information to be verified, inputs the first triple and the verification and evaluation report prompt word into the large language model, and obtains the verification and evaluation report of the threat intelligence information to be verified. The large language model is also used to generate the verification and evaluation report of the threat intelligence information to be verified based on the verification and evaluation report prompt word to be provided to security analysts for viewing and analysis, improving the work efficiency of security analysts.

[0104] In an embodiment of the present application, the threat intelligence verification platform 101, the database 102, and the knowledge base 103 may be set in the same server or in different servers. The server may be an independent physical server or a cloud server providing basic cloud computing services such as cloud servers, cloud databases, and cloud storage. The embodiments of the present application do not make any limitations in this regard.

[0105] Based on the above application scenario, the following will refer to the attached Figures 2 to 13(b)To describe the exemplary embodiments of the present application in more detail, it should be noted that the above application scenarios are only shown for the convenience of understanding the spirit and principles of the present application, and the embodiments of the present application are not limited by any of them. On the contrary, the embodiments of the present application can be applied to any applicable scenario.

[0106] As Figure 2 shown, it is a schematic flowchart of the implementation process of the threat intelligence information verification method provided by the embodiments of the present application. The threat intelligence information verification method can be applied to the above-mentioned threat intelligence verification platform 101, and specifically may include the following steps:

[0107] S21. Obtain the threat intelligence information to be verified.

[0108] In specific implementation, the threat intelligence information to be verified can be multi-source heterogeneous threat intelligence data collected by the threat intelligence verification platform from websites containing network threat intelligence data from various sources, such as vendor analysis reports, FreeBuf, Github, technical blogs, etc. Among them, FreeBuf is a Chinese Internet security media platform focusing on the field of network security, which provides a community for network security practitioners and researchers to communicate and share, covering various levels of content from basic security knowledge to advanced security technologies. Github is a hosting platform for open-source and private software projects. It is based on the Git version control system, allowing developers to store, manage, and share code, and providing rich tools to facilitate team cooperation and project management. The threat intelligence information to be verified can also be threat intelligence data sent by external terminal devices to the threat intelligence verification platform, or threat intelligence data extracted from business traffic intercepted by network security devices such as firewalls. The embodiments of the present application do not make any limitations in this regard. The threat intelligence information to be verified includes structured threat intelligence information and unstructured threat intelligence information. Among them, the structured threat intelligence information includes data in formats such as JSON (JavaScript Object Notation) and XML (Extensible Markup Language), and the unstructured threat intelligence information generally includes text data, such as network security research reports.

[0109] S22. Extract the threat elements included in the threat intelligence information to be verified.

[0110] Among them, the threat elements are the key information for threat analysis of the threat intelligence information to be detected.

[0111] In specific implementation, key information for threat analysis of threat intelligence information is predefined, that is, threat elements. Expanding outward with the attacker as the core, threat elements (i.e., key information) can include fine-grained information such as the name, alias, common number, discovery time, active time, etc. of the attacker, as well as coarse-grained information such as the attack location (e.g., the region, industry, company attacked by the attacker), the attack means (or attack technology) used by the attacker, and the vulnerabilities exploited by the attacker. Then, these threat elements (i.e., key information) are mapped to the entity types and relationship types defined by STIX. For example, entity types include: attacker, vulnerability, attack location (i.e., attack site), attack means, etc., and relationship types include: Target, Use, Exploit, Download, etc. Using a unified structure avoids inconsistent structures of the extracted key information.

[0112] In this step, if the threat intelligence information to be verified is structured threat intelligence information to be verified, a corresponding parser is designed according to the data format (JSON, XML, etc.), and based on the entity types and relationship types of the defined threat elements (i.e., key information), the corresponding parser is used to extract the field values of the fields where the threat elements are located from the structured threat intelligence information to be verified. As Figure 3 shown, it is a threat intelligence information to be verified in JSON format, and it is possible but not limited to extract the field values of fields such as the "name" field representing the attacker name, the "threat_actor_types" field representing the type, the "aliases" field representing the alias, and the "goals" field representing the purpose.

[0113] In one implementation, if the threat intelligence information to be verified is unstructured threat intelligence information to be verified, a pre-trained entity relationship extraction model can be used to extract entities and relationships from the unstructured threat intelligence information to be verified. The extracted entities are the threat elements of the type of entity in the threat intelligence information to be verified, and the extracted relationships are the threat elements of the type of relationship in the threat intelligence information to be verified. The entity relationship extraction model can be but not limited to using the Bert (Bidirectional Encoder Representation from Transformers, bidirectional encoder representation based on Transformer) model, and can also use CRF (conditional random field), etc. The embodiments of the present application do not limit this.

[0114] In one implementation, for unstructured threat intelligence information to be verified, a pre-trained large language model can also be used to extract entities and relationships from the unstructured threat intelligence information to be verified. The extracted entities are the threat elements of the type of entity in the threat intelligence information to be verified, and the extracted relationships are the threat elements of the type of relationship in the threat intelligence information to be verified. The large language model can understand and generate natural language text or understand the meaning of language text. The threat intelligence verification platform can be obtained by pre-training with a large amount of natural text data. The large language model can understand and generate natural language by learning the language patterns, grammar, and semantics of the text data. The large language model can be, but is not limited to, the ChatGPT (Chat Generative Pre-trained Transformer) model, the OpenAIo1 model, the DeepSeek model, or the Qwen model, etc. The embodiments of the present application do not limit this. In the embodiments of the present application, the pre-trained large language model is also used for subsequent generation of summaries, verification of threat intelligence information to be verified, and generation of verification evaluation reports.

[0115] In the embodiments of the present application, an entity relationship extraction model or a large language model can be used to automatically extract the information of interest from the text.

[0116] S23. Determine a first triple corresponding to the threat intelligence information to be verified based on the threat elements. The first triple includes a first threat element of the type of entity having an associated relationship and a second threat element corresponding to the associated relationship between the first threat elements of the type of entity.

[0117] In specific implementation, after the threat intelligence verification platform extracts the threat elements included in the threat intelligence information to be verified, the threat elements of entity type can be recorded as the first threat elements, the threat elements of relationship type can be recorded as the second threat elements, and the second threat elements corresponding to the relationship between the first threat elements of entity type with an associated relationship are generated into a number of first triples in the format of <entity, relationship, entity>, that is, <the first threat element, the second threat element, the first threat element> (<subject, predicate, object>). Among them, the two first threat elements in <the first threat element, the second threat element, the first threat element> are different entities, and the second threat element is the relationship between these two entities. For example, taking the attacker "APT32" as an example, <APT32, alias, Hailianhua> and <APT32, uses, T1059.001> can be obtained. Among them, the relationship between "APT32" and "Hailianhua" is: "alias", that is, the alias of "APT32" is "Hailianhua", "T1059.001" is the number of an attack technique, and the relationship between "APT32" and the attack technique "T1059.001" is: "uses", that is, "APT32" uses the attack technique "T1059.001". Furthermore, each first triple can be expressed in natural language form. For example, <APT32, alias, Hailianhua> can be expressed in natural language as "The alias of APT32 is Hailianhua", and <APT32, uses, T1059.001> can be expressed in natural language as "APT32 uses T1059.001". These first triples are the triples corresponding to the threat intelligence information to be verified and can be used as the dataset of information to be verified.

[0118] In the embodiment of the present application, extracting the key information included in the threat intelligence information to be verified and forming it into the form of triples as the dataset of information to be verified can simplify the subsequent verification process and improve the verification efficiency.

[0119] S24. Obtain the verification reference information corresponding to the threat intelligence information to be verified from at least two data sources respectively according to the first triples.

[0120] Among them, the data sources include a cybersecurity knowledge graph, a knowledge base, and a preset trusted threat intelligence website.

[0121] In specific implementation, to ensure the comprehensiveness and accuracy of threat intelligence information verification, the embodiment of the present application uses retrieval-augmented generation technology combined with a large language model and multiple knowledge retrieval sources to verify the threat intelligence information to be verified. The following data sources (that is, multiple knowledge retrieval sources) are preset in advance: a cybersecurity knowledge graph, a knowledge base, and a preset trusted threat intelligence website. The above data sources are knowledge systems constructed or collected in advance, and the knowledge included in the above data sources has been verified to be correct and can be used as the verification basis for the threat intelligence information to be verified.

[0122] The network security knowledge graph contains numerous entity relationships in the field of network security, such as entities like attackers, vulnerabilities, malware, attack techniques, etc., and the relationships between these entities, such as attackers exploiting vulnerabilities, attackers using attack techniques, etc. These entities and relationships are respectively connected in the form of nodes and edges to construct a large semantic network with complex relationships. As Figure 4 shown, it is an example connection diagram of a network security knowledge graph, which shows some entities and the relationships between entities. During implementation, it can be updated in real time according to the entities and the relationships between entities in the collected real attack activities.

[0123] The knowledge base includes a vector database and a document database. The vector database contains the corresponding relationships of text blocks, text block vectors, text block summaries, text block summary vectors, and metadata. Among them, the text block is the text block contained in the verified threat intelligence text, and the metadata is the key information extracted from the text block for threat analysis, that is, the threat elements contained in the text block. The document database contains the corresponding relationships of text blocks and metadata.

[0124] Before obtaining the threat intelligence information to be verified, the vector database and the document database can be obtained according to the process as Figure 5 shown, including the following steps:

[0125] S31. Obtain threat intelligence text.

[0126] Specifically in implementation, the knowledge base is a repository composed of a series of domain-specific documents such as threat intelligence analysis documents, threat advisories, and emergency plans. These documents can be obtained from the Internet and analysis platforms, intelligence platforms, etc. maintained within the enterprise. In order to improve the retrieval efficiency of these documents, in the embodiments of the present application, natural language processing technology is used to process these documents. The threat intelligence verification platform parses these domain-specific documents, extracts the threat intelligence text from the documents, and preprocesses the threat intelligence text to delete useless information such as useless characters and garbled codes in the threat intelligence text.

[0127] S32. For each threat intelligence text, split the threat intelligence text into multiple text blocks.

[0128] Specifically in implementation, for each preprocessed threat intelligence text, the threat intelligence verification platform splits the intelligence text into multiple text blocks in the order from front to back. It can be split by sentences or by a preset character length. The embodiments of the present application do not limit this. Among them, the preset character length can be set by itself, such as any length from 100 to 300 characters, or other lengths can also be set. The embodiments of the present application do not limit this.

[0129] S33. For each text block, if it is determined that the text block contains threat elements, extract the threat elements in the text block and use the threat elements in the text block as the metadata of the text block.

[0130] In specific implementation, the threat intelligence verification platform extracts threat elements from each text block respectively, that is, extracts key information from each text block. If it is determined that the text block contains threat elements, extract the threat elements in the text block and use the threat elements in the text block as the metadata of the text block, that is: the metadata of the text block is the key information related to threats contained in the text block.

[0131] S34. Input the text block and the preset summary prompt word into the large language model to obtain the text block summary.

[0132] In specific implementation, the threat intelligence verification platform inputs each text block and the preset summary prompt word into the large language model respectively to obtain the corresponding text block summary. The preset summary prompt word is used to prompt the large language model to generate the text block summary according to the input text block. As Figure 6 shown, it is an example of a preset summary prompt word, which can be set according to needs during implementation, and the embodiments of the present application do not limit this.

[0133] The embodiments of the present application use the large language model to generate the summary of the text, and the accuracy is relatively high.

[0134] S35. Input the text block and the text block summary into the text embedding model respectively to obtain the text block vector and the text summary vector.

[0135] In specific implementation, for each text block, the threat intelligence verification platform inputs the text block and the text block summary into the text embedding (Embedding) model respectively to embed the text block and the text block summary, and obtains the text block vector and the text block summary vector. The text embedding model is used to generate the vector representation of the text. The text embedding model can be but is not limited to the following models: the text-embedding-large-3 model of OpenAI, the Embedding model of BAAI (Beijing Academy of Artificial Intelligence, Institute of Intelligence) (such as bge-large-en-v1.5, bge-large-zh-v1.5), the BCEmbedding model of Youdao, etc. Any other model that can generate text vectors can also be used, and the embodiments of the present application do not limit this.

[0136] S36. Store the text block, the text block vector, the text block summary, the text block summary vector and the corresponding metadata in the vector database.

[0137] In specific implementation, the threat intelligence verification platform stores the text block, its corresponding text block vector, text block summary, text block vector, and metadata in a vector database for vector retrieval.

[0138] S37. Store the text block and the corresponding metadata in a document database.

[0139] In specific implementation, the threat intelligence verification platform stores the text block and the corresponding metadata in a document database for full-text retrieval.

[0140] In the embodiments of the present application, the trusted threat intelligence website is a verified website related to network threats inside and outside the enterprise. The verified websites related to network threats inside the enterprise include privately maintained security analysis platforms, intelligence platforms, etc. within the enterprise. The verified websites related to network threats outside the enterprise refer to various types of intelligence databases published on the Internet, such as the official websites of NVD, CNVD, and CNNVD for vulnerabilities, and the official website of ATT&CK for attack techniques. The embodiments of the present application do not limit this.

[0141] In implementation, the verification reference information corresponding to the threat intelligence information to be verified can be retrieved from at least two knowledge retrieval sources (i.e., data sources), namely the network security knowledge graph, knowledge base, and preset trusted threat intelligence websites, according to the first triple.

[0142] In specific implementation, it can be in accordance with Figure 7 the process shown to retrieve the verification reference information corresponding to the threat intelligence information to be verified from the network security knowledge graph, including the following steps:

[0143] S41. Query the subgraph within the preset hop count of the entity from the network security knowledge graph according to the entity in the first triple.

[0144] In specific implementation, the first triple includes two entities, namely: two first threat elements. The threat intelligence verification platform can use a graph query language (GQL) to query sub-graphs within a preset number of hops of each first threat element from the cybersecurity knowledge graph. Among them, the sub-graph within the preset number of hops of an entity refers to extracting the relationships and entities connected to it starting from this entity. From the first entity to the last entity, it contains a first preset number of edges, and the first preset number is less than or equal to the preset number of hops. For example, for the sub-graph within m hops of an entity, from the first entity to the last entity, it contains at most m edges (that is, it can include 1 to m edges). The value of m can be set by itself, such as it can be set to 4, or it can be set to any other value. This application example does not limit this. Assuming the number of hops is 4, then 1-hop sub-graph, 2-hop sub-graph, 3-hop sub-graph, and 4-hop sub-graph can be extracted. Extracting multi-hop sub-graphs can obtain richer sub-graph information compared to only extracting a single sub-graph.

[0145] Optionally, the sub-graph formed by the first threat element and the second threat element can also be queried from the cybersecurity knowledge graph using the graph query language. For example, for the first triple <APT32, uses, T1059.001>, the sub-graph that simultaneously satisfies the entity being "APT32" and the relationship being "uses", or the relationship being "uses" and the entity being "T1059.001" can be queried. When there is a relationship restriction, the number of hops can be not set. In this way, the obtained sub-graph information can be more relevant to the meaning of the threat intelligence information to be verified.

[0146] S42. Extract each second triple included in the sub-graph to obtain a second triple set.

[0147] In specific implementation, according to the connection relationships between different entities included in the sub-graph, each triple in the form of <entity, relationship, entity> included in the sub-graph is extracted, which can be recorded as the second triple, to obtain a second triple set. For example, the connection of the sub-graph of a certain first threat element is as follows: entity 1 - relationship 1 - entity 2 - relationship 2 - entity 3 - relationship 3 - entity 4. Among them, each "-relationship-" represents an edge. This sub-graph includes 3 edges, which is the 3-hop sub-graph of entity 1. Entity 1 is the first threat element and includes the following three second triples: <entity 1, relationship 1, entity 2>, <entity 2, relationship 2, entity 3>, <entity 3, relationship 3, entity 4>.

[0148] In this way, the sub-graph within the preset number of hops of the first element in each first triple is extracted, and each second triple included in each sub-graph is extracted to obtain a second triple set.

[0149] S43. Use the second triple set as the first verification reference information corresponding to the threat intelligence information to be verified.

[0150] In specific implementation, the threat intelligence verification platform uses the second triple set as the verification reference information corresponding to the threat intelligence information to be verified, which can be recorded as the first verification reference information.

[0151] During implementation, it can be in accordance with the Figure 8 process shown in the figure to retrieve the verification reference information corresponding to the threat intelligence information to be verified from the vector database in the knowledge base, including the following steps:

[0152] S51. Convert the first triple into the corresponding natural language description text to be verified.

[0153] In specific implementation, for each first triple, the threat intelligence verification platform uses a large language model or a preset template to convert the first triple into the corresponding natural language description text to be verified.

[0154] Specifically, the threat intelligence verification platform can input the first triple into the large language model, and based on the semantic understanding ability of the large language model, output the natural language description text corresponding to the first triple. For example, input <APT32, alias, Hailianhua> into the large language model, and output its corresponding natural language description text: "The alias of APT32 is Hailianhua".

[0155] The preset template can be set according to different entity relationships. Different entity relationships can correspond to different templates. For example, the template corresponding to the relationship "uses" can be set as: "Entity 1 uses Entity 2". During implementation, it can be set according to the actual situation.

[0156] S52. Input the natural language description text to be verified into the text embedding model to obtain the natural language description text vector to be verified.

[0157] In specific implementation, the threat intelligence verification platform inputs the natural language description text corresponding to each first triple into the text embedding model respectively to obtain the natural language description text vectors to be verified.

[0158] S53. Determine the similarity between the natural language description text vector to be verified and each text block vector in the vector database, and determine the similarity between the natural language description text vector to be verified and the summary vector of each text block.

[0159] In specific implementation, for each natural language description text vector to be verified, the threat intelligence verification platform calculates the similarity between the natural language description text vector to be verified and each text block vector in the vector database, and calculates the similarity between the natural language description text vector to be verified and the summary vector of each text block. When calculating the similarity between the natural language description text vector to be verified and the text block vector, and the similarity between the natural language description text vector to be verified and the summary vector of the text block, algorithms such as cosine similarity or Euclidean distance can be used, but are not limited to these, and the embodiments of the present application do not limit this.

[0160] S54. Use the text blocks, text block summaries, and metadata corresponding to the preset number of text block vectors or text block summary vectors with the highest similarity to the natural language description text vector to be verified as the second verification reference information corresponding to the threat intelligence information to be verified.

[0161] In specific implementation, the threat intelligence verification platform extracts the text blocks, text block summaries, and metadata corresponding to the preset number of text block vectors with the highest similarity to the natural language description text vector to be verified, or extracts the text blocks, text block vectors, and metadata corresponding to the preset number of text block summary vectors with the highest similarity to the natural language description text vector to be verified, and uses these text blocks, text block vectors, and metadata as the verification reference information corresponding to the threat intelligence information to be verified, which can be denoted as the second verification reference information.

[0162] During implementation, it can be in accordance with the Figure 9 shown process to retrieve the verification reference information corresponding to the threat intelligence information to be verified from the document database in the knowledge base, including the following steps:

[0163] S61. Query the matching metadata from the document database according to the first threat element and the second threat element in the first triple.

[0164] In specific implementation, for each first triple, the threat intelligence verification platform queries the matching metadata from the document database according to the two entities in the first triple, that is, the two first threat elements, and the relationship between the entities, that is, the second threat element. That is, query the matching key information. The metadata matching the first threat element can be denoted as the first metadata, and the metadata matching the second threat element can be denoted as the second metadata.

[0165] S62. Use the first metadata matching the first threat element and the corresponding text blocks, and the second metadata matching the second threat element and the corresponding text blocks as the third verification reference information corresponding to the threat intelligence information to be verified.

[0166] During specific implementation, the threat intelligence verification platform uses the first metadata and the corresponding text block that match the first threat element, as well as the second metadata and the corresponding text block that match the second threat element, as the verification reference information corresponding to the threat intelligence information to be verified, which can be denoted as the third verification reference information.

[0167] During implementation, it can be carried out according to the Figure 10 shown process to retrieve the verification reference information corresponding to the threat intelligence information to be verified from a preset trusted threat intelligence website, including the following steps:

[0168] S71. Retrieve the target threat intelligence information containing the first threat element and / or the second threat element from the preset trusted threat intelligence website according to the first threat element and the second threat element in the first triple.

[0169] During specific implementation, for each first triple, the threat intelligence verification platform uses the two entities it contains, namely the first threat element and the relationship, namely the second threat element, as keywords to query from the preset trusted threat intelligence website, and retrieves the target threat intelligence information containing the first threat element and / or the target threat intelligence information containing the second threat element. In this way, the target threat intelligence information corresponding to all first triples can be retrieved from the trusted threat intelligence website.

[0170] Specifically, the threat intelligence verification platform can query by calling the APIs of each trusted threat intelligence website. It can also use the large language model agent mode. Regarding the retrieval of each trusted threat intelligence website as a website search tool, the large language model automatically selects the trusted threat intelligence websites and keywords to be searched according to the input and the website search tool information to obtain the target threat intelligence information. The embodiments of the present application do not make limitations on this.

[0171] S72. Use the target threat intelligence information as the fourth verification reference information corresponding to the threat intelligence information to be verified.

[0172] During specific implementation, the threat intelligence verification platform uses the target threat intelligence information retrieved from each trusted threat intelligence website as the verification reference information corresponding to the threat intelligence information to be verified, which can be denoted as the fourth verification reference information.

[0173] S25. Generate a threat verification prompt word according to the verification reference information obtained from at least two data sources, and input the first triple and the threat verification prompt word into the large language model to obtain the verification result of the threat intelligence information to be verified.

[0174] During specific implementation, the threat intelligence verification platform can select any at least two sets of verification reference information from the first verification reference information, the second verification reference information, the third verification reference information, and the fourth verification reference information as the verification reference information for the threat intelligence information to be verified. As a preferred implementation manner, the first verification reference information, the second verification reference information, the third verification reference information, and the fourth verification reference information can be jointly used as the verification reference information for the threat intelligence information to be verified.

[0175] In this step, the threat intelligence verification platform generates threat verification prompt words according to the obtained verification reference information. The threat verification prompt words are used to assist the large language model in verifying the accuracy of the threat intelligence information to be verified. The threat verification prompt words can include task objectives, reference information (i.e., the verification reference information for the threat intelligence to be verified), and information to be verified (i.e., the first triple corresponding to the threat intelligence information to be verified), etc. As shown in Figure 11(a), it is a threat verification prompt word template provided by an embodiment of the present application. In Figure 11(a), "Please check the accuracy of [information to be verified], requirements: 1. Compare for consistency with the provided [reference information]; 2. Identify the content with logical errors or data contradictions; 3. Mark the doubtful points that need manual review; 4. Give the verification conclusion: correct, incorrect, uncertain" is the task objective, the reference information "1.xxxx2.xxxx 3.xxxx" is the verification reference information for the threat intelligence to be verified, the information to be verified "xxxx" is a first triple corresponding to the threat intelligence information to be verified, and "Please output in a structured format, including the verification conclusion, the basis for passing or failing the verification, and the items to be checked for doubts, etc." is the requirement for the output format of the verification result by the large language model. During implementation, it can be set according to needs, and the embodiments of the present application do not limit this. Figure 11(b) is an example of a specific threat verification prompt word provided by an embodiment of the present application.

[0176] During implementation, the threat intelligence verification platform inputs each first triple and the threat verification prompt words into the large language model respectively to obtain the verification results corresponding to each first triple. The verification results corresponding to all first triples form the verification result of the threat intelligence information to be verified.

[0177] In one implementation manner, after obtaining the verification result of the threat intelligence information to be verified, it is also possible to summarize the verification results corresponding to all first triples to generate a verification evaluation report for the threat intelligence information to be verified, so that security analysts can understand the quality of the threat intelligence information to be verified by referring to the verification evaluation report. The high-quality verification evaluation report and the threat intelligence information to be verified can be used as the support for subsequent knowledge graph expansion and threat detection model training data.

[0178] During specific implementation, it can be carried out as follows Figure 12The process shown generates a verification and evaluation report for the threat intelligence information to be verified, including the following steps:

[0179] S81. Generate a prompt for the verification and evaluation report based on the verification result of the threat intelligence information to be verified.

[0180] Specifically, the threat intelligence verification platform generates a prompt for the verification and evaluation report based on the verification result of the threat intelligence information to be verified. Among them, the prompt for the verification and evaluation report is used to assist the large language model in generating a verification and evaluation report corresponding to the threat intelligence information to be verified. The prompt for the verification and evaluation report may include information such as task objectives, report structure, and verified data. As shown in Figure 13(a), it is an example of a prompt for the verification and evaluation report. In Figure 13(a), "Please generate a professional evaluation report based on [completed verified data]" is the task objective, and "The report needs to include the following structured content: 1. Verification overview... 2. Detailed analysis... 3. Data details * Group and summarize the data according to the verification conclusion (provide basis and review suggestions for failed / suspected data)" is the report structure. "The following is the completed verified data: [{"information to be verified": "information to be verified 1", "verification conclusion": "verification conclusion 1", "reference information": "reference information 1", "verification reason": "verification reason 1"},...]" is the verified data and verification reason, where the information to be verified 1 is the threat intelligence information to be verified, and the reference information 1 is the verification reference information corresponding to the threat intelligence information to be verified. Figure 13(b) is an example of a specific prompt for the verification and evaluation report provided by the embodiment of the present application.

[0181] S82. Input the threat intelligence information to be verified and the prompt for the verification and evaluation report into the large language model to obtain a verification and evaluation report for the threat intelligence information to be verified.

[0182] Specifically, the threat intelligence verification platform inputs the threat intelligence information to be verified and the prompt for the verification and evaluation report into the large language model to obtain a verification and evaluation report for the threat intelligence information to be verified.

[0183] The large language model in the embodiment of the present application may be a general large language model or a large language model fine-tuned with data in the field of network security. The embodiment of the present application does not limit this.

[0184] The threat intelligence information verification method provided by the embodiments of this application. The threat intelligence verification platform obtains the threat intelligence information to be verified, extracts the threat elements included in the threat intelligence information to be verified. The threat elements are the key information used for threat analysis of the threat intelligence information to be detected. Based on the threat elements, the first triple corresponding to the threat intelligence information to be verified is determined. The first triple includes the first threat element of the type of entity with an associated relationship and the second threat element corresponding to the associated relationship between the first threat elements of the type of entity. According to the first triple, the verification reference information corresponding to the threat intelligence information to be verified is obtained from at least two data sources respectively. Among them, the data sources include a cybersecurity knowledge graph, a knowledge base, and a preset trusted threat intelligence website. The threat verification prompt words are generated according to the verification reference information obtained from at least two data sources, and the first triple and the threat verification prompt words are input into the large language model to obtain the verification result of the threat intelligence information to be verified. The large language model is used to generate the verification result of the threat intelligence information to be verified based on the threat verification prompt words. In the embodiments of this application, after the threat intelligence verification platform obtains the threat intelligence information to be verified, it extracts the key information for threat analysis included in the threat intelligence information to be verified, that is, the threat elements, and forms the first threat element of the type of entity and the second threat element corresponding to the associated relationship between the first threat elements of the type of entity into a triple in the form of <entity, relationship, entity> (that is, <first threat element, second threat element, first threat element>) according to the type of the extracted threat elements, which can be recorded as the first triple. The first triple is the triple corresponding to the threat intelligence information to be verified. According to the first triple, the verification reference information corresponding to the threat intelligence information to be verified is obtained from at least two data sources among the cybersecurity knowledge graph, the knowledge base, and the preset trusted threat intelligence website. The verification reference information is used to verify the first triple corresponding to the threat intelligence information to be verified. The threat verification prompt words are generated according to the obtained verification reference information and input into the large language model to assist in verifying the accuracy of the triple corresponding to the threat intelligence information to be verified, and the verification result of the threat intelligence information to be verified is obtained. Compared with the prior art, in order to ensure the comprehensiveness and accuracy of threat intelligence information verification, this application uses the retrieval-enhanced generation technology to automatically verify the threat intelligence information to be verified in combination with the large language model and multiple knowledge retrieval sources. The large language model has strong natural language understanding and generation capabilities. The addition of multiple knowledge retrieval data sources provides more and more powerful background knowledge for the large language model, greatly reducing the possibility of the large language model outputting hallucinations. Thus, the accuracy and efficiency of threat intelligence information verification are improved.

[0185] Based on the same inventive concept, an embodiment of the present application further provides a threat intelligence information verification device. Since the principle of solving problems by the above threat intelligence information verification device is similar to that of the above threat intelligence information verification method, the implementation of the above device can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0186] As Figure 14 shown, it is a structural schematic diagram of the threat intelligence information verification device provided by an embodiment of the present application. The threat intelligence information verification device can be applied to Figure 1 the threat intelligence verification platform 101 shown in, and may include:

[0187] A first acquisition module 91, configured to acquire threat intelligence information to be verified;

[0188] A first extraction module 92, configured to extract threat elements included in the threat intelligence information to be verified, where the threat elements are key information for performing threat analysis on the threat intelligence information to be detected;

[0189] A determination module 93, configured to determine a first triple corresponding to the threat intelligence information to be verified based on the threat elements. The first triple includes a first threat element of entity type with an associated relationship, and a second threat element corresponding to the associated relationship between the first threat elements of entity type;

[0190] A second acquisition module 94, configured to respectively acquire verification reference information corresponding to the threat intelligence information to be verified from at least two data sources, where the data sources include a cybersecurity knowledge graph, a knowledge base, and a preset trusted threat intelligence website;

[0191] A verification module 95, configured to generate a threat verification prompt word according to the verification reference information obtained from the at least two data sources, and input the first triple and the threat verification prompt word into a large language model to obtain a verification result of the threat intelligence information to be verified. The large language model is used to generate a verification result of the threat intelligence information to be verified based on the threat verification prompt word.

[0192] In an implementation manner, the device further includes:

[0193] A prompt word generation module, configured to generate a verification evaluation report prompt word according to the verification result of the threat intelligence information to be verified;

[0194] A verification and evaluation report generation module is configured to input the threat intelligence information to be verified and the verification and evaluation report prompt words into the large language model to obtain a verification and evaluation report of the threat intelligence information to be verified. The large language model is also configured to generate a verification and evaluation report of the threat intelligence information to be verified based on the verification and evaluation report prompt words.

[0195] In one implementation, the second acquisition module 94 is specifically configured to retrieve the verification reference information corresponding to the threat intelligence information to be verified from the cybersecurity knowledge graph according to the first triple in the following manner: query the subgraph within a preset hop count of the entity from the cybersecurity knowledge graph according to the entity in the first triple; extract each second triple included in the subgraph to obtain a second triple set; and use the second triple set as the first verification reference information corresponding to the threat intelligence information to be verified.

[0196] In one implementation, the knowledge base includes a vector database, which contains the corresponding relationships among text blocks, text block vectors, text block summaries, text block summary vectors, and metadata. Among them, the text blocks are the text blocks included in the verified threat intelligence texts, and the metadata is the key information extracted from the text blocks for threat analysis.

[0197] The second acquisition module 94 is specifically configured to retrieve the verification reference information corresponding to the threat intelligence information to be verified from the knowledge base according to the first triple in the following manner: convert the first triple into a corresponding natural language description text to be verified; input the natural language description text to be verified into a text embedding model to obtain a natural language description text vector to be verified; determine the similarity between the natural language description text vector to be verified and each text block vector in the vector database, and determine the similarity between the natural language description text vector to be verified and each text block summary vector; and use the text blocks, text block summaries, and metadata corresponding to the preset number of text block vectors or text block summary vectors with the highest similarity to the natural language description text vector to be verified as the second verification reference information corresponding to the threat intelligence information to be verified.

[0198] In one implementation, the knowledge base further includes a document database, which contains the corresponding relationships between the text blocks and the metadata.

[0199] The second acquisition module 94 is specifically configured to retrieve the verification reference information corresponding to the to-be-verified threat intelligence information from the knowledge base in the following manner: query the matching metadata from the document database according to the first threat element and the second threat element in the first triple; use the first metadata and the corresponding text block that match the first threat element, and the second metadata and the corresponding text block that match the second threat element as the third verification reference information corresponding to the to-be-verified threat intelligence information.

[0200] In one implementation manner, the second acquisition module 94 is specifically configured to retrieve the verification reference information corresponding to the to-be-verified threat intelligence information from the preset trusted threat intelligence website in the following manner: retrieve the target threat intelligence information containing the first threat element and / or the second threat element from the preset trusted threat intelligence website according to the first threat element and the second threat element in the first triple; use the target threat intelligence information as the fourth verification reference information corresponding to the to-be-verified threat intelligence information.

[0201] In one implementation manner, the device further includes:

[0202] A third acquisition module, configured to acquire the threat intelligence text before acquiring the to-be-verified threat intelligence information;

[0203] A segmentation module, configured to segment each threat intelligence text into multiple text blocks;

[0204] A second extraction module, configured to, for each text block, if it is determined that the text block contains a threat element, extract the threat element in the text block and use the threat element in the text block as the metadata of the text block;

[0205] An abstract generation module, configured to input the text block and a preset abstract prompt word into the large language model to obtain a text block abstract;

[0206] A text vector extraction module, configured to input the text block and the text block abstract into the text embedding model respectively to obtain a text block vector and a text abstract vector;

[0207] A first storage module, configured to store the text block, the text block vector, the text block abstract, the text block abstract vector, and the corresponding metadata in the vector database;

[0208] A second storage module, configured to store the text block and the corresponding metadata in the document database.

[0209] Based on the same inventive concept, an embodiment of the present application further provides an electronic device 1000. Refer to Figure 15 As shown, the electronic device 1000 is used to implement the threat intelligence information verification method described in the above method embodiment. The electronic device 1000 in this embodiment may include: a memory 1001, a processor 1002, and a computer program stored in the memory and executable on the processor, such as a threat intelligence information verification program. When the processor executes the computer program, the steps in the above various threat intelligence information verification method embodiments are implemented.

[0210] In the embodiment of the present application, the specific connection medium between the above-mentioned memory 1001 and processor 1002 is not limited. In the embodiment of the present application Figure 15 it is shown that the memory 1001 and the processor 1002 are connected by a bus 1003. The bus 1003 is represented by a thick line in Figure 15 For the connection manners between other components, only a schematic illustration is given and is not taken as a limitation. The bus 1003 may be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 15 only one thick line is used to represent it in

[0211] The memory 1001 may be a volatile memory, such as a random-access memory (RAM); the memory 1001 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD), or the memory 1001 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1001 may be a combination of the above memories.

[0212] The processor 1002 is used to implement the threat intelligence information verification method provided by the embodiment of the present application.

[0213] An embodiment of the present application further provides a computer-readable storage medium, storing computer-executable instructions required to be executed by the above-mentioned processor, which includes a program required to be executed by the above-mentioned processor.

[0214] In some possible embodiments, aspects of the threat intelligence information verification method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product runs on an electronic device, the program code is used to cause the electronic device to execute the steps of the threat intelligence information verification method according to various exemplary embodiments of this application described above in this specification.

[0215] Those skilled in the art should understand that the embodiments of this application can be provided as a method, apparatus, or computer program product. Therefore, this application can take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0216] This application is described with reference to the flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to the embodiments of this application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one or more flows and / or Figure 1 blocks.

[0217] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in Figure 1 one or more flows and / or Figure 1 blocks.

[0218] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one or more flows and / or Figure 1 blocks.

[0219] Although the preferred embodiments of the present application have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn of the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

[0220] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these modifications and variations.

Claims

1. A threat intelligence information verification method, characterized in that Including: Obtain threat intelligence information to be verified; Extract threat elements included in the threat intelligence information to be verified, where the threat elements are key information for threat analysis of the threat intelligence information to be detected; Based on the threat elements, determine a first triple corresponding to the threat intelligence information to be verified, where the first triple includes a first threat element of entity type with an associated relationship and a second threat element corresponding to the associated relationship between the first threat elements of entity type; According to the first triple, obtain verification reference information corresponding to the threat intelligence information to be verified from at least two data sources, where the data sources include a cybersecurity knowledge graph, a knowledge base, and a preset trusted threat intelligence website; Generate a threat verification prompt word according to the verification reference information obtained from the at least two data sources, and input the first triple and the threat verification prompt word into a large language model to obtain a verification result of the threat intelligence information to be verified, where the large language model is used to generate a verification result of the threat intelligence information to be verified based on the threat verification prompt word.

2. The method according to claim 1, wherein It also includes: Generate a verification evaluation report prompt word according to the verification result of the threat intelligence information to be verified; Input the threat intelligence information to be verified and the verification evaluation report prompt word into the large language model to obtain a verification evaluation report of the threat intelligence information to be verified, where the large language model is also used to generate a verification evaluation report of the threat intelligence information to be verified based on the verification evaluation report prompt word.

3. The method according to claim 1, wherein Obtain verification reference information corresponding to the threat intelligence information to be verified from the cybersecurity knowledge graph according to the first triple, specifically including: Query a subgraph within a preset hop count of the entity from the cybersecurity knowledge graph according to the entity in the first triple; Extract each second triple included in the subgraph to obtain a second triple set; Use the second triple set as the first verification reference information corresponding to the threat intelligence information to be verified.

4. The method according to claim 1, wherein The knowledge base includes a vector database, where the vector database contains the corresponding relationships of text blocks, text block vectors, text block summaries, text block summary vectors, and metadata. Among them, the text blocks are text blocks included in the verified threat intelligence text, and the metadata is the key information extracted from the text blocks for threat analysis; Obtain verification reference information corresponding to the threat intelligence information to be verified from the knowledge base according to the first triple, specifically including: Convert the first triple into a corresponding natural language description text to be verified; Input the natural language description text to be verified into a text embedding model to obtain a natural language description text vector to be verified; Determine the similarity between the natural language description text vector to be verified and each text block vector in the vector database, and determine the similarity between the natural language description text vector to be verified and each text block summary vector; The text chunks, text chunk summaries, and metadata corresponding to the preset number of text chunk vectors or text chunk summary vectors with the highest similarity to the text vector of the natural language description text to be verified are used as the second verification reference information corresponding to the threat intelligence information to be verified.

5. The method according to claim 4, wherein The knowledge base further includes a document database, and the document database contains the corresponding relationship between the text chunks and the metadata; Obtaining the verification reference information corresponding to the threat intelligence information to be verified from the knowledge base according to the first triple specifically includes: Querying and matching metadata from the document database according to the first threat element and the second threat element in the first triple; The first metadata that matches the first threat element, the corresponding text chunk, the second metadata that matches the second threat element, and the corresponding text chunk are used as the third verification reference information corresponding to the threat intelligence information to be verified.

6. The method according to claim 1, wherein Obtaining the verification reference information corresponding to the threat intelligence information to be verified from the preset trusted threat intelligence website according to the first triple specifically includes: Retrieving target threat intelligence information containing the first threat element and / or the second threat element from the preset trusted threat intelligence website according to the first threat element and the second threat element in the first triple; The target threat intelligence information is used as the fourth verification reference information corresponding to the threat intelligence information to be verified.

7. The method according to claim 5, characterized in that Before obtaining the threat intelligence information to be verified, it further includes: Obtaining the threat intelligence text; For each threat intelligence text, splitting the threat intelligence text into multiple text chunks; For each text chunk, if it is determined that the text chunk contains a threat element, extracting the threat element in the text chunk, and using the threat element in the text chunk as the metadata of the text chunk; Inputting the text chunk and the preset summary prompt word into the large language model to obtain a text chunk summary; Inputting the text chunk and the text chunk summary into the text embedding model respectively to obtain a text chunk vector and a text chunk summary vector; Storing the text chunk, text chunk vector, text chunk summary, text chunk summary vector, and corresponding metadata into the vector database; Storing the text chunk and the corresponding metadata into the document database.

8. A threat intelligence information verification device, characterized in that, Including: A first acquisition module, configured to acquire threat intelligence information to be verified; A first extraction module, configured to extract threat elements included in the threat intelligence information to be verified, and the threat elements are key information for performing threat analysis on the threat intelligence information to be detected; A determination module, configured to determine a first triple corresponding to the threat intelligence information to be verified based on the threat elements, where the first triple includes a first threat element of the type of entity having an association relationship, and a second threat element corresponding to the association relationship between the first threat elements of the type of entity; A second acquisition module, configured to obtain verification reference information corresponding to the threat intelligence information to be verified from at least two data sources according to the first triple, where the data sources include a cybersecurity knowledge graph, a knowledge base, and a preset trusted threat intelligence website; A verification module, configured to generate threat verification prompt words according to verification reference information obtained from the at least two data sources, and input the first triple and the threat verification prompt words into a large language model to obtain a verification result of the threat intelligence information to be verified, where the large language model is configured to generate the verification result of the threat intelligence information to be verified based on the threat verification prompt words.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the threat intelligence information verification method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps in the threat intelligence information verification method according to any one of claims 1 to 7.

Images