Open source threat intelligence acquisition method and system based on large language model

Abstract

The invention discloses an open source threat intelligence acquisition method and system based on a large language model, and the method comprises the steps: receiving a problem proposed by a user through an Agent module, analyzing the problem based on the large language model, and selecting a tool provided by a Toolkits module based on an analysis result; the Toolkits module collects and processes the open source threat intelligence information according to the tool selected by the Agent module; the memory module records an interaction process between the user and the Agent module and an interaction process between the Agent module and the Toolkits module, when a question proposed by the user is not recorded, operation is performed according to the steps 1-2, and when the question proposed by the user is recorded, the memory module participates in interaction between the Agent module and the Toolkits module by providing a historical context. According to the method, threat intelligence can be autonomously obtained based on user requirements without manually scanning tools or querying related websites and platforms, and a large amount of time and labor cost are saved.

Description

Technical Field

[0001] The present invention belongs to the field of information security technology, and specifically relates to a method and system for acquiring open source threat intelligence based on a large language model. Background Art

[0002] Open source intelligence (OSINT) requires processing and analyzing collected information to obtain valuable intelligence information. This process includes not only information collection but also information quality assessment and credibility analysis. Therefore, the research on the acquisition of open source network intelligence covers many aspects, from collection tools and techniques to application value, to intelligence quality and multi-source heterogeneous data processing. These studies not only provide technical support for the effective collection and application of open source intelligence, but also provide a theoretical basis and practical guidance for improving the quality and credibility of intelligence.

[0003] Spiderfoot is an automated open source network intelligence collection tool that has the ability to automatically retrieve more than 100 public data sources, and can centrally collect detailed information about IP addresses, domain names, emails, etc. It can also integrate multiple data sources and adopt multiple data analysis methods, allowing users to efficiently sort out and understand the collected data; Recon-ng can identify Web vulnerabilities and provide functions such as GeoIP positioning, DNS queries, and port scanning; while Chiasmodon focuses on collecting relevant information about target domain names, and its core function revolves around tracking various data related to domain names. In addition, there are a number of open source intelligence tools such as Maltego and MISP, which not only provide interfaces for connecting to different data sources, but also support manual upload of data, helping users to conduct more sophisticated analysis.

[0004] After a lot of pre-training, large language models (LLMs) have the ability to understand and generate complex natural language texts, which are very suitable for analyzing and processing large amounts of threat intelligence text data. This makes large language models play an important role in threat intelligence identification, extraction, and comprehensive evaluation. Large language models are highly scalable and applicable, which means that they can be flexibly applied to various threat intelligence scenarios, such as obtaining threat intelligence from open source information platforms and performing correlation analysis on threat intelligence. In addition, large language models can automatically process large amounts of threat intelligence data, reduce manual intervention, thereby reducing labor costs and improving efficiency. This is particularly important for threat intelligence collection work that requires processing massive amounts of data. However, LLMs have knowledge limitations and timeliness due to their reliance on training data, and it is difficult to obtain the latest vertical field knowledge, and the knowledge depth is insufficient. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a method and system for acquiring open source threat intelligence based on a large language model, and proposes a framework in combination with the thinking chain technology, which aims to help LLMs use tools to acquire OSINT. Based on this framework, an Agent and a toolbox for dynamic use by the Agent are constructed, and users can obtain the required OSINT information by asking questions to the Agent.

[0006] To achieve the above object, the present invention adopts the following technical solution:

[0007] A method for acquiring open source threat intelligence based on a large language model includes the following steps:

[0008] Step 1: The Agent module receives the question raised by the user, parses the question based on the large language model, and selects the tool provided by the Toolkits module based on the parsing result;

[0009] Step 2: The Toolkits module collects and processes open source threat intelligence information based on the tools selected by the Agent module. The Toolkits module includes a search engine tool, a vulnerability information search tool, and a license information search tool.

[0010] Step 3: The memory module records the interaction process between the user and the Agent module, as well as the interaction process between the Agent module and the Toolkits module. When the question raised by the user has not been recorded, follow steps 1 and 2. When the question raised by the user has been recorded, the memory module participates in the interaction between the Agent module and the Toolkits module by providing historical context.

[0011] Furthermore, the step 1 comprises:

[0012] The Agent module receives and parses user input, extracts key information, and selects tools for obtaining the required open source information. The query function of the Agent module sets query parameters to be passed to the Toolkits module based on the key information and selects the tools;

[0013] The Toolkits module receives query parameters and performs open source information collection and processing operations according to the tools selected by the Agent module, and returns the results to the Agent module;

[0014] After receiving each result, the query function of the Agent module evaluates whether the user's problem is solved. If not, the query function of the Agent module will reselect the tool, use the reselected tool to perform open source information collection and processing operations, and observe the new results generated until it is confirmed that the problem has been completely solved.

[0015] Furthermore, the Agent module includes a large language model and a scheduling module. The large language model understands and processes user questions based on the thought chain technology. The scheduling module integrates an open source threat intelligence collection algorithm and is a tool for decision selection based on the analysis results of the large prediction model.

[0016] Furthermore, the search engine tool in step 2 is used to provide a search engine interface, use the Bing Web search API to execute queries and process the returned search results, and the specific implementation steps are as follows:

[0017] Parse query parameters: Receive the query parameters in string form provided by the Agent module, parse them into JSON format, and extract the value with the key "action_input" as the actual query keyword;

[0018] Prepare search request: clean the extracted query keywords to get clean search terms, and set HTTP request headers and request parameters;

[0019] Send search request: Send a request including HTTP request header and request parameters to BingWeb search engine API. If the request is unsuccessful, an exception will be thrown.

[0020] Process search results: If the request is successful, process the obtained search results, organize them and return them.

[0021] Furthermore, the vulnerability information search tool in step 2 is used to obtain vulnerability information of the query function of the Agent module, and the specific implementation steps are as follows:

[0022] Parse parameters and verify the CVE number format: After receiving the query parameters in JSON format passed by the Agent module, extract the target CVE number. If the CVE number format is not met, an error message is thrown;

[0023] Local database query: If the CVE number format is met, query the parsed CVE number information in the local vulnerability information database. If found, return directly, otherwise proceed to the next step;

[0024] Crawling open source vulnerability intelligence information: For CVE numbers with correct format, obtain CVE vulnerability information including vulnerability description and last update time, call specific functions to retrieve CVE vulnerability repair suggestions, and count vulnerability information and repair suggestions;

[0025] Save and return results: Feedback the final results to the Agent module in the designed format and store them in the local vulnerability information database.

[0026] Furthermore, the license information search tool in step 2 is used for the Agent module to query the required license information, and the specific implementation steps are as follows:

[0027] Scrape the license-related information from the web and store it in the local intelligence database;

[0028] Vectorize the crawled license-related information and store it in the vector database;

[0029] Establishing an index table of intelligence information between the local intelligence database and the vector database;

[0030] When the Agent module queries the license information, it searches the vector database based on the index table.

[0031] On the other hand, the present invention also provides an open source threat intelligence acquisition system based on a large language model, comprising:

[0032] The Agent module is used to receive questions raised by users, parse the questions based on the large language model, and select tools provided by the Toolkits module based on the parsing results;

[0033] The Toolkits module is used to collect and process open source threat intelligence information based on the tools selected by the Agent module. The Toolkits module includes a search engine tool, a vulnerability information search tool, and a license information search tool.

[0034] The memory module is used to record the interaction process between the user and the Agent module, as well as the interaction process between the Agent module and the Toolkits module. When the questions raised by the user are recorded, the memory module participates in the interaction between the Agent module and the Toolkits module by providing historical context.

[0035] The advantages of the present invention compared with the prior art are:

[0036] (1) The present invention combines the large language model technology and makes full use of the powerful understanding and generation capabilities of the large language model. It can obtain and process the required open source threat intelligence according to user requirements, which is more flexible and efficient than the existing threat intelligence acquisition technology.

[0037] (2) The present invention combines the thought chain technology to design prompts, so that the large language model can break down logical reasoning problems into multiple steps and solve them step by step, thereby increasing the transparency and interpretability of the model and providing the large language model with tools to obtain open source threat intelligence, so that it can use tools to obtain open source threat intelligence outside its own knowledge base during the reasoning process, thereby breaking through the knowledge limitations and timeliness problems caused by LLMs' reliance on training data, and achieving a higher answer accuracy rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of a method for acquiring open source threat intelligence based on a large language model according to the present invention;

[0039] Figure 2 Prompt template program diagram designed for combining thought chains;

[0040] Figure 3 This is a program logic diagram of an open source threat intelligence acquisition method based on a large language model of the present invention;

[0041] Figure 4 A sample diagram of information integration for vulnerabilities;

[0042] Figure 5 Flowchart of the Agent module processing user questions. DETAILED DESCRIPTION

[0043] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0044] like Figure 1 and Figure 3 As shown, the present invention provides a method for acquiring open source threat intelligence based on a large language model, comprising the following steps:

[0045] Step 1: The Agent module receives the question raised by the user, understands and processes the question through the internal large language model, and selects the tool provided by the Toolkits module;

[0046] like Figure 5 As shown, the Agent module receives and parses user input, extracts key information, and selects tools for obtaining required open source information. The query function of the Agent module sets query parameters to be passed to the tools in the Toolkits based on this information.

[0047] The Agent module passes the query parameters to the selected tool, which will perform the collection and processing of open source information and return the results to the Agent module.

[0048] After receiving each result, the query function of the Agent module will evaluate whether the user's problem has been solved. If not, the query function of the Agent module will think about what tool to choose next, use the tool and observe the results again. This cycle continues until it is confirmed that the problem has been completely solved.

[0049] Finally, when enough information has been collected to answer the user's question, the Agent will compile the final answer and return it to the user.

[0050] The Agent module is the central decision-making unit of the entire framework. It is actually composed of LLM and schduling module, and is the entity that drives decision-making. The Agent is responsible for receiving user input (questions), understanding and processing these questions through the internal large language model (LLM). The scheduling module decides what actions to take, selects appropriate tools to obtain information, and checks the quality of the information obtained until the LLM generates a correct and reasonable answer. The scheduling module integrates the OSINT acquisition algorithm, enabling the Agent to effectively extract the required intelligence from public data sources to respond to user queries. As the schedule module within the Agent module, the algorithm works in conjunction with the large language model to form the core of the system and ensure the efficiency and accuracy of information retrieval.

[0051] In order to improve the output quality of LLMs and enable LLMs to decompose logical reasoning problems into sequential steps and solve them step by step, a prompt for obtaining open source intelligence was designed by combining the thinking chain technology, such as Figure 2 As shown in the figure, tool_descs is used to introduce the tools that can be called by LLM, including the tool name, description, parameters required for calling, and parameter requirements; tool_names is used to explain the names of the tools that can be used by LLM; question is used to explain the problems that LLM needs to solve.

[0052] Step 2: The Toolkits module collects and processes open source threat intelligence information based on the tools selected by the Agent module. The Toolkits module includes a search engine tool, a vulnerability information search tool, and a license information search tool.

[0053] The Toolkits module is a collection of multiple tools, including a set of tools and algorithms related to collecting open source threat intelligence. The functions of the tools can be adjusted according to different task requirements, and it has scalability and flexibility. Since the purpose of the present invention is to obtain open source intelligence, the main function of the tools included in the module is to collect and process open source intelligence information, such as search engines, vulnerability information search tools, and license information search tools. The Agent module works closely with this module to selectively call these tools to search for relevant information in a targeted manner, obtain the required intelligence, or perform specific operations.

[0054] The search engine tool provides a simple search engine interface, uses the Bing Web search API to execute queries and process the returned search results. The specific implementation steps are as follows:

[0055] (1) Parsing query parameters: A function named search_engine is defined for the Agent to call. It receives the query parameters in string form provided by the Agent. The function attempts to parse the data passed by the Agent into JSON format and extract the value with the key "action_input" as the actual query keyword.

[0056] (2) Prepare the search request: Define the URL of the search request as the endpoint address of the Bing Web Search API (https: / / api.bing.microsoft.com / v7.0 / search). Then clean the extracted query keywords, remove the leading and trailing blanks, spaces, and line breaks, and get the clean search term search_term. Then set the HTTP request headers and request parameters params, including the subscription key required to access the Bing API, search keywords, whether to enable text decoration, text format, and market locale settings (here set to the Chinese market zh-CN).

[0057] (3) Send a search request: Use the requests.get() function to send a GET request to the Bing search engine API, carrying the request header information and request parameters prepared above. Then use response.raise_for_status() to check the response status code. If the request is unsuccessful, an exception will be thrown.

[0058] (4) Processing search results: If the request is successful, use the .json() method to convert the response body into a Python dictionary. Then call the _process_response function to process the obtained search results, sort them out and return them.

[0059] The purpose of the vulnerability information search tool is to obtain the vulnerability information queried by the Agent. To this end, the function get_cev_info is defined to search and integrate detailed information of specific CVE vulnerabilities, including but not limited to release time, description, danger level, solution, etc. The specific implementation steps are as follows:

[0060] (1) Parse parameters and verify the CVE number format: After receiving the JSON format parameters passed by the Agent, use json.loads() to convert it into a dictionary and extract the target CVE number from it. Use the regular expression cve_pattern = r"CVE-\d{4}-\d+" to verify whether the input string conforms to the CVE number format. If not, an error message is thrown.

[0061] (2) Local database query: Try to use the db.search_cve_data method to query the parsed CVE information in the local database. If found, return directly; otherwise, proceed to the next step.

[0062] (3) Crawling open source vulnerability intelligence information: Since there are many websites that provide open source vulnerability information, how to filter out the required information from these information sources and build comprehensive and accurate vulnerability intelligence information is the key. In order to ensure the integrity of vulnerability information, a unified data submission and integration template is designed, and crawler technology is used to crawl different field information about a vulnerability from relevant websites; for a CVE number with the correct format, first initiate a request to the URL of the NVD database, parse the response data, and obtain the basic information of the CVE vulnerability including the vulnerability description and the last update time. Then, call a specific function to retrieve the repair suggestions for the CVE vulnerability from the Alibaba Cloud vulnerability library, access the Snyk vulnerability database API, obtain the specific programming language or operating system affected by the vulnerability, and find the Git repair link related to the CVE. Finally, according to the CVSS scoring version (V3 or V2) adopted by CVE, organize the key elements such as the source of the vulnerability, the danger level, and the attack vector. After unified processing and aggregation, all collected information is integrated into a dictionary, including vulnerability name, release time, update time, description, source, type, solution suggestion, affected component, public status, affected language or system, vulnerability URL, etc. These multi-source heterogeneous information are cleaned, screened and integrated, and the final results are fed back to the Agent in a designed format and stored in the local vulnerability information database to form a structured vulnerability intelligence output. Examples of the submitted integrated data are as follows: Figure 4 shown.

[0063] (4) Save and return results: Return the sorted vulnerability information to the Agent and save it to the local intelligence database.

[0064] In summary, we have achieved the goal of collecting CVE information from multiple sources. After integration, we not only provide a detailed overview of the vulnerabilities, but also ensure the local accessibility of the information, which is convenient for subsequent analysis or query. The entire vulnerability information acquisition and integration process.

[0065] The license information search tool is used for Agents to query the license information they want. Since relatively complete license information can be obtained from TLDRLegal (https: / / www.tldrlegal.com / browse), we choose to directly crawl the license-related information on the website and store it in the local intelligence database for Agent use in the form of a plug-in knowledge base. The crawled license-related information is vectorized and stored in the vector database, and an index table of intelligence information between the local intelligence database and the vector database is established; when the Agent queries the license information, the vector database is searched to improve the efficiency of information retrieval. In order to use the Faiss vector database to create and operate vector indexes to process text data, a class named faiss_db is defined. Its methods are centered around text vector index operations based on the Faiss library, covering core functions such as creation, query, save, load, add and delete, and are mainly used to perform information retrieval tasks.

[0066] (1) __init__(): Initialize and load the pre-trained model. Considering that the questions raised by users may involve the simultaneous appearance of Chinese and English, after investigation, we chose to use the text vector model jina-embeddings-v2-base-zh that supports both Chinese and English to generate text feature vectors. It supports text encoding up to 8192 characters. The development of this model is based on the BERT architecture (JinaBERT). JinaBERT is an improvement on the BERT architecture. It is the first to use Applied to the encoder architecture to support longer sequences. Different from previous single-language / multi-language vector models, this bilingual model can better support single-language (Chinese search for Chinese) and cross-language (Chinese search for English) document retrieval.

[0067] (2) get_datas_embedding(): Generates text feature vectors. Receives a text list, uses the loaded model to convert each text into a fixed-length vector representation, and returns a vector list. create_index(): Establishes a Faiss index for storing and retrieving text vectors. First, create an index object using faiss.IndexFlatL2, where L2 represents the use of Euclidean distance as the similarity metric. Accepts a list of data vectors datas_embedding, and determines the dimension of the vector by passing in the parameter datas_embedding.shape to ensure that the index can correctly accommodate vectors of a specific size. Use the .add() method to add the pre-calculated text vector list datas_embedding to the index to complete the index construction.

[0068] (3) data_recall(): For a given query text, find the index of the k most similar text vectors. The received parameters are the Faiss index faiss_index, the query text query, and the number of most similar results k returned. First, the query content query is converted into a vector representation through the pre-trained model, and then the faiss_index.search() method is called to search and return the index of the k most similar data. Since the index of the data in the vector database and the local intelligence database is the same, the corresponding text can be obtained from the local intelligence database according to the index.

[0069] (4) faiss_index_save() and faiss_index_load(): Save the Faiss index to disk and load the previously saved index from disk. faiss_index_save() uses the faiss.write_index() method to save the index object faiss_index to the specified path. And faiss_index_load() uses faiss.read_index() to read and return the index object from the specified path, which is convenient for subsequent query or update operations.

[0070] (5) index_data_add() and index_data_delete(): used to add new data to the Faiss index or delete specified vector data. When adding new data, convert the new data into an embedded vector datas_embedding through get_datas_embedding() and add it to the index through faiss_index.add(datas_embedding). When deleting specified data, use faiss_index.remove_ids() to delete the specified vector data.

[0071] (6) Based on the functions provided by fassi_db, an instance db of the faiss_db class is initialized, and the function get_license_data() is defined to implement the retrieval of license information. After parsing which license information the Agent wants to query, this function is used to obtain the license information most similar to the input content from the license information database. The db.data_recall() function is used to retrieve the index of the license information most similar to the input content from the vector database, read the license information corresponding to the index from the local intelligence database, extract the fields including license name, description, label, user rights and freedoms, rules that must be followed, etc., and splice them into a complete description of the license and return it to the Agent.

[0072] like Figure 1 As shown, the method further includes step 3, the memory module records the interaction process between the user and the Agent, and provides the functions of adding, deleting, obtaining and resetting data (interaction records). It records information related to the user's query. When the user asks a question, the memory module interacts with the Agent and Toolkits modules by providing historical context, which enhances the system's ability to provide accurate responses.

[0073] Corresponding to the above method, refer again Figure 1 The present invention also provides an open source threat intelligence acquisition system based on a large language model, and each module included in the system can execute the aforementioned steps, specifically:

[0074] The Agent module is used to receive questions raised by users, parse the questions based on the large language model, and select tools provided by the Toolkits module based on the parsing results;

[0075] The Toolkits module is used to collect and process open source threat intelligence information based on the tools selected by the Agent module. The Toolkits module includes a search engine tool, a vulnerability information search tool, and a license information search tool.

[0076] The memory module is used to record the interaction process between the user and the Agent module, as well as the interaction process between the Agent module and the Toolkits module. When the questions raised by the user are recorded, the memory module participates in the interaction between the Agent module and the Toolkits module by providing historical context.

[0077] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for acquiring open source threat intelligence based on a large language model, characterized in that: The steps include: Step 1: The Agent module receives the question raised by the user, parses the question based on the large language model, and selects the tool provided by the Toolkits module based on the parsing result; Step 2: The Toolkits module collects and processes open source threat intelligence information based on the tools selected by the Agent module. The Toolkits module includes a search engine tool, a vulnerability information search tool, and a license information search tool. Step 3: The memory module records the interaction process between the user and the Agent module, as well as the interaction process between the Agent module and the Toolkits module. When the question raised by the user has not been recorded, follow steps 1 and 2. When the question raised by the user has been recorded, the memory module participates in the interaction between the Agent module and the Toolkits module by providing historical context.

2. A method for acquiring open source threat intelligence based on a large language model according to 1, characterized in that: The step 1 comprises: The Agent module receives and parses user input, extracts key information, and selects tools for obtaining the required open source information. The query function of the Agent module sets query parameters to be passed to the Toolkits module based on the key information and selects the tools; The Toolkits module receives query parameters and performs open source information collection and processing operations according to the tools selected by the Agent module, and returns the results to the Agent module; After receiving each result, the query function of the Agent module evaluates whether the user's problem is solved. If not, the query function of the Agent module will reselect the tool, use the reselected tool to perform open source information collection and processing operations, and observe the new results generated until it is confirmed that the problem has been completely solved.

3. A method for acquiring open source threat intelligence based on a large language model according to 2, characterized in that: The Agent module includes a large language model and a scheduling module. The large language model understands and processes user questions based on the thought chain technology. The scheduling module integrates an open source threat intelligence collection algorithm and is a tool for decision selection based on the analysis results of the large prediction model.

4. A method for acquiring open source threat intelligence based on a large language model according to 1, characterized in that: The search engine tool in step 2 is used to provide a search engine interface, use the Bing Web search API to execute queries and process the returned search results. The specific implementation steps are as follows: Parse query parameters: Receive the query parameters in string form provided by the Agent module, parse them into JSON format, and extract the value with the key "action_input" as the actual query keyword; Prepare search request: clean the extracted query keywords to get clean search terms, and set HTTP request headers and request parameters; Send search request: Send a request including HTTP request header and request parameters to BingWeb search engine API. If the request is unsuccessful, an exception will be thrown. Process search results: If the request is successful, process the obtained search results, organize them and return them.

5. According to 1, a method for acquiring open source threat intelligence based on a large language model is characterized in that: The vulnerability information search tool in step 2 is used to obtain vulnerability information of the query function of the Agent module. The specific implementation steps are as follows: Parse parameters and verify the CVE number format: After receiving the query parameters in JSON format passed by the Agent module, extract the target CVE number. If the CVE number format is not met, an error message is thrown; Local database query: If the CVE number format is met, query the parsed CVE number information in the local vulnerability information database. If found, return directly, otherwise proceed to the next step; Crawling open source vulnerability intelligence information: For CVE numbers with correct format, obtain CVE vulnerability information including vulnerability description and last update time, call specific functions to retrieve CVE vulnerability repair suggestions, and count vulnerability information and repair suggestions; Save and return results: Feedback the final results to the Agent module in the designed format and store them in the local vulnerability information database.

6. According to 1, a method for acquiring open source threat intelligence based on a large language model is characterized in that: The license information search tool in step 2 is used for the Agent module to query the required license information. The specific implementation steps are as follows: Crawling license-related information from the web and storing it in the local intelligence database; Vectorize the crawled license-related information and store it in the vector database; Establishing an index table of intelligence information between the local intelligence database and the vector database; When the Agent module queries the license information, it searches the vector database based on the index table.

7. An open source threat intelligence acquisition system based on a large language model, characterized in that: include: The Agent module is used to receive questions raised by users, parse the questions based on the large language model, and select tools provided by the Toolkits module based on the parsing results; The Toolkits module is used to collect and process open source threat intelligence information based on the tools selected by the Agent module. The Toolkits module includes a search engine tool, a vulnerability information search tool, and a license information search tool. The memory module is used to record the interaction process between the user and the Agent module, as well as the interaction process between the Agent module and the Toolkits module. When the questions raised by the user are recorded, the memory module participates in the interaction between the Agent module and the Toolkits module by providing historical context.

Images