Threat hunting method based on large language model and reinforcement learning

Abstract

The invention provides a threat hunting method based on a large language model and reinforcement learning, and belongs to the technical field of network security. Firstly, initial threat entities are extracted from threat intelligence, an initial threat node set is formed, and a vector-text retrieval database is constructed; then an optimal threat node is selected from the current threat node set through a reinforcement learning algorithm, an optimal threat hunting method is selected according to a cold start strategy or a variant strategy of an upper confidence boundary in combination with a large language model and a vector-text retrieval database, and the problem that knowledge is scarce in the threat hunting cold start stage is solved; and finally, through a mode of updating node quality through recursion and updating method quality based on similarity, a sparse reward problem existing in a threat node selection process and a threat hunting method selection process is solved. According to the method, the problems of low convergence speed and difficulty in cold start in a threat hunting scene in the prior art are solved, and intelligent and high-efficiency threat hunting is realized.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a threat hunting method based on a large language model and reinforcement learning. Background Art

[0002] With the rapid development and popularization of network technology, the security threats faced by the global network are also increasing. In today's complex and ever-changing network security environment, network attack methods are becoming increasingly diverse and advanced, and complex attack methods such as advanced persistent threats are emerging in an endless stream. Traditional passive defense measures can no longer effectively resist complex attacks such as advanced persistent threats. In this context, threat hunting, as an active defense strategy, has gradually become an important research direction in the field of network security. Threat hunting can detect and respond to threats in a timely manner before they cause serious impacts by actively searching and analyzing abnormal activities and signs in the network. Threat hunting can mine unknown threat entities from limited known threat information. Threat hunting methods based on threat intelligence match unknown threats by performing natural language processing on text, but it is difficult to solve the timeliness problem of threat intelligence text.

[0003] In response to the above problems, reinforcement learning, as an adaptive decision-making method, has begun to be applied to the field of threat hunting. It improves the efficiency of threat hunting by learning the optimal strategy through interaction with the environment. Threat hunting methods based on reinforcement learning can solve the problem of insufficient timeliness of threat intelligence in traditional threat hunting methods based on threat intelligence. However, since this type of method requires the introduction of a large number of external data sources and different data acquisition methods, its query cost is high and the query efficiency is low. At present, threat hunting methods based on reinforcement learning still face the following challenges in practical applications:

[0004] 1) Slow convergence: In threat hunting, there are scenarios where positive feedback is relatively scarce, which leads to inefficient training of reinforcement learning models and slow convergence, resulting in low threat hunting efficiency.

[0005] 2) Cold start difficulty: When using a threat hunting algorithm based on reinforcement learning, in the absence of sufficient initial experience and data, the model needs a long time to interact with the environment to learn effective strategies. This cold start problem makes threat hunting inefficient in the initial stage. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a threat hunting method based on a large language model and reinforcement learning, which utilizes the knowledge base and zero-sample learning ability of the large language model, combined with the strategy optimization and adaptive learning characteristics of reinforcement learning, to solve the problems of slow convergence speed and low cold start efficiency in the prior art, and achieve efficient threat hunting. The technical solution is as follows:

[0007] A threat hunting method based on a large language model and reinforcement learning includes the following steps:

[0008] Step 1: Context Building

[0009] Extract the initial threat entities from the threat intelligence to form an initial threat hunting node set, and convert the threat intelligence text into a high-dimensional vector to establish a vector-text retrieval database;

[0010] Step 2: Threat Hunting Node and Method Selection

[0011] Based on the current threat hunting node set, the optimal threat node is selected through the reinforcement learning algorithm, and then the threat hunting method corresponding to the optimal threat node is selected to obtain the current optimal threat hunting method; if the method selection process of the current node is in the cold start stage, the external knowledge provided by the large language model is used to assist the reinforcement learning algorithm for method selection; if the method selection process of the current node is not in the cold start stage, the variant strategy of the upper confidence bound is used for method selection; finally, the selected threat hunting method is executed to generate a new set of child nodes;

[0012] Step 3: Update node quality and method quality

[0013] By recursively updating the node quality, the method quality is updated using similarity to solve the sparse reward problem, and finally the next threat hunting node set is generated.

[0014] Furthermore, in step 1, the initial threat hunting node set is N 0 ; At the initial stage, let the current threat hunting node set, that is, the Tth threat hunting node set N t is the initial threat hunting node set, that is, N t =N 0 ;

[0015] Preprocess the threat intelligence text and segment it into original text segments T′ by sentence i , each original text segment T′ i The word embedding model is used to convert the text into a high-dimensional vector and represented as V i ; Build vector-text retrieval database DB Context ={ <V i ,T′ i >}.

[0016] Furthermore, the step 2 is specifically as follows:

[0017] Step 2.1: Based on the Tth threat hunting node set N t The expansion is carried out, and the threat hunting node selection is performed according to the following formula to obtain the current optimal threat node n i:

[0018]

[0019] In the formula, Q t (n) represents the node quality corresponding to the threat node n. When initialized, the node quality Q corresponding to all threat nodes t (n) is 0; P(n) represents the parent node of node n; N(n) represents the number of times the corresponding threat node n is selected as the optimal threat node in the historical threat hunting process; c is a hyperparameter for balancing exploration and exploitation;

[0020] Step 2.2: Select the threat hunting method and obtain the current optimal threat hunting method a j ;

[0021] If i ∈N 0 , then the current optimal threat node n i The method selection is set to the cold start stage, otherwise it is the non-cold start stage;

[0022] If the current optimal threat node n i The method selection process is the cold start stage, so the cold start method selection strategy is used to introduce the external knowledge provided by the large language model to assist the reinforcement learning algorithm to select the method, and the current optimal threat hunting method a is obtained. j :

[0023]

[0024] In the formula, k i is the current optimal threat node n i Node type, such as hash, IP, domain name, etc. i is the current optimal threat node n i A collection of alternative threat hunting methods; is the method quality matrix W t The elements in , which represent the node type k i The quality of using method a; the method quality matrix W t is an R×M matrix, where R represents the number of threat node types and M represents the total number of threat hunting methods; W t At initialization, all elements are set to 0; N(a) is the number of times method a is selected, LLMScore(a,n i ) is the current optimal threat node n calculated by the large language model i Applicability score with method a;

[0025] If the current optimal threat node n iThe method selection process is a non-cold start stage, and the upper confidence bound variant strategy is used for method selection, as shown in the following formula:

[0026]

[0027] In the formula, N(A i ) is the current optimal threat node n i The total number of visits to the method collection;

[0028] Step 2.3: Execute the currently selected optimal threat hunting method a through reinforcement learning algorithm j , generate a new threat sub-node set C t+1 .

[0029] Furthermore, in step 2.2, the large language model is used to calculate the current optimal threat node n i The applicability score with method a is as follows:

[0030] LLMScore(a,n i )=LLM(Context(n i ),Description(a))

[0031] In the formula, context Context(n i ) represents the search from vector-text database DB Context The node that is found in i Related contextual information and the corresponding vector representation; Description(a) is the description of method a; LLM(·) represents the reasoning process of the large language model, and outputs method a at the current optimal threat node n i Ratings on

[0032] Extract Context(n i ) is implemented as follows: the vector database FAISS is used for retrieval, cosine similarity is used as the similarity metric, and the Top-K strategy is adopted to select the most relevant content from the top K context fragments with the highest similarity. The default value of K is 3 and can be adjusted according to the algorithm configuration to achieve accurate matching and rapid extraction of context fragments related to threat nodes.

[0033] Furthermore, step 3 is specifically as follows:

[0034] Step 3.1: Update the threat node set N t+1 =N t ∪C t+1 , and record the parent nodes of all newly generated threat nodes n′ in turn, and the current optimal threat node n iUpdate to the parent node of all newly generated threat nodes n′:

[0035]

[0036] Step 3.2: For each newly generated node n′, calculate the incentive ΔQ based on the feedback information t (n′), and traverse all newly generated nodes n′, and update the quality Q of the relevant nodes t (n):

[0037]

[0038] Among them, ΔQ t (n′) is the stimulus calculated based on the feedback information, Ancestors(n′) represents the set of all ancestor nodes of the newly generated node n′;

[0039] Step 3.3: Update the method quality:

[0040] Using the method similarity function Sim method , for the method quality matrix W t To update:

[0041]

[0042] Where A represents the set of all threat hunting methods; a j is the optimal threat node n at the current time i The current best threat hunting method selected on Sim method (a,a j ) is method a and method a j The similarity measure between .

[0043] Furthermore, the node quality Q t In the update of (n), the excitation ΔQ t The strategy of (n′) is: for any newly generated node n′∈C t+1 , if n′ does not belong to the current threat hunting node set N t , then the incentive is 1, otherwise the incentive is 0.

[0044] Furthermore, the method similarity function Sim method (a,a j ) is calculated as follows: for any two methods a, a j If they share the same data source, the similarity is increased by 0.1; if they share the same operation, the similarity is increased by 0.5.

[0045] The beneficial effects of the present invention are:

[0046] 1) Based on the reinforcement learning framework, the present invention utilizes the knowledge base and zero-shot learning capability of a large language model, combined with the strategy optimization and adaptive learning characteristics of reinforcement learning, to improve the efficiency of threat hunting;

[0047] 2) The present invention can solve the problem of sparse rewards by introducing recursive incentives and quality updates based on similarity methods in scenarios with slow convergence speed, and use a large language model to introduce external knowledge in cold start scenarios to improve cold start efficiency, thereby realizing an efficient threat hunting method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The following is a flow chart of the threat hunting method of the present invention.

[0049] Figure 2 A flow diagram is constructed for the context of the present invention.

[0050] Figure 3 A schematic diagram of the threat hunting node and method selection process of the present invention.

[0051] Figure 4 It is a schematic diagram of the node quality and method quality update process of the present invention.

[0052] Figure 5 This is a diagram showing the experimental evaluation effect of the present invention. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] The present invention proposes a threat hunting method based on a large language model and reinforcement learning. Figure 1 As shown, the present invention mainly includes three steps of context construction, threat hunting node and method selection, and node quality and method quality update, as well as the T-th threat hunting node set and the T+1-th threat hunting node set as outputs of different steps.

[0055] 1. Context Building

[0056] The context building steps are as follows Figure 2 As shown, first, the initial threat entity is extracted from the threat intelligence to form the initial threat node set N 0 , at the initial stage, let the current threat node set N t is the initial threat node set, that is, N t =N 0 . For threat intelligence text T 0 Perform preprocessing and split the sentences into original text segments T′ i , each T′ i Use the word embedding model to convert the text into a high-dimensional vector represented as V i, build vector-text retrieval database DB Context ={ <V i ,T′ i >}.

[0057] Preferably, BERT is used as the word embedding model.

[0058] 2. Threat hunting nodes and method selection

[0059] Threat hunting nodes and method selection steps are as follows Figure 3 As shown. Based on the Tth threat hunting node set N t The expansion is carried out, and the threat hunting node selection is performed according to the following formula to obtain the current optimal threat node n i :

[0060]

[0061] In the formula, Q t (n) represents the node quality corresponding to the threat node n. At initialization, all threat nodes correspond to Q t (n) is 0; P(n) represents the parent node of node n; N(n) represents the number of times the corresponding threat node n is selected as the optimal threat node in the historical threat hunting process; c is a hyperparameter for balancing exploration and exploitation.

[0062] Then, the threat hunting method is selected to obtain the current optimal threat hunting method a j If n i ∈N 0 , then the current node n i The method selection is set to the cold start stage, otherwise it is the non-cold start stage.

[0063] If the current node n i The method selection process is the cold start stage, so the cold start method selection strategy is used to introduce the external knowledge provided by the large language model to assist the reinforcement learning algorithm to select the current optimal threat hunting method a j :

[0064]

[0065] In the formula, k i For node n i Node type, such as hash, IP, domain name, etc. i For the current threat node n i For example, the set of optional threat hunting methods, the method quality matrix W t is an R×M matrix, where R represents the number of threat node types and M represents the total number of threat hunting methods; W tAt initialization, all elements are set to 0; the elements of the matrix represents the quality of using method a on node type k. N(a) is the number of times threat hunting method a is selected in historical calls. LLMScore(a,n i ) is the node n calculated by the large language model i Applicability score with method a.

[0066] Use the large language model to calculate the current selected node n i Suitability score with method a:

[0067] LLMScore(a,n i )=LLM(Context(n i ),Description(a))

[0068] In the formula, context Context(n i ) represents the search from vector-text database DB Context The node that is found in i Related contextual information and the corresponding vector representation; Description(a) is the description of method a; LLM(·) represents the reasoning process of the large language model, and outputs method a at the current optimal threat node n i Ratings on;

[0069] Extract Context(n i ) is implemented as follows: use the vector database FAISS for retrieval, use cosine similarity as the similarity metric, adopt the Top-K strategy, select the most relevant content from the top K context fragments with the highest similarity, the K value is 3 by default, and can be adjusted according to the algorithm configuration to achieve accurate matching and rapid extraction of context fragments related to threat nodes. The calculation formula of cosine similarity is as follows:

[0070]

[0071] Where A and B represent the query vector and index vector respectively, · represents the vector dot product, and ∥A∥ and ∥B∥ represent the Euclidean norm of the vector. By calculating the cosine similarity, the system can evaluate the similarity between the query vector and the index vector.

[0072] Preferably, LLM(·) uses ChatGPT4.0 as the inference model.

[0073] If the current node n i The method selection process is a non-cold start phase, and the upper confidence bound variant strategy is used for method selection:

[0074]

[0075] In the formula, N(A i ) is node n i The total number of visits to the method set. Then, the algorithm executes the corresponding selected method a j , generate a new child node set C t+1 .

[0076] 3. Update of node quality and method quality

[0077] The node quality and method quality update steps are as follows: Figure 4 As shown. First, update the threat node set N t+1 =N t ∪C t+1 , and record the parent nodes of all newly generated nodes in turn, and the current optimal threat node n i Update to the parent node of all newly generated threat nodes n′.

[0078]

[0079] According to the expansion results, in order to effectively amplify the incentive, traverse all n′ ancestor nodes and update the quality Q of the relevant nodes t (n):

[0080]

[0081] Among them, ΔQ t (n′) is the stimulus calculated based on the feedback information, and Ancestors(n′) represents the set of all ancestor nodes of the newly generated node n′.

[0082] Preferably, ΔQ t The strategy of (n′) is: for any node If n′ is found not to belong to N t , then the incentive is 1, otherwise the incentive is 0.

[0083] Then, the method quality is updated. Considering the similarity between different threat hunting methods, the method similarity function Sim method , for the method quality matrix W t To update:

[0084]

[0085] Where A represents the set of all threat hunting methods; a j is the optimal threat node n at the current time i The current best threat hunting method selected on Sim method (a,a j) is method a and method a j The similarity measure between .

[0086] Preferably, for any two methods a, a j The default similarity is 0. If they share the same data source, the similarity is increased by 0.1; if they share the same operation, the similarity is increased by 0.5.

[0087] Then, the T+1th threat hunting node set is generated.

[0088] Optionally, the Tth threat hunting node set may be assigned as the T+1th threat hunting node set, and the threat hunting node and method selection step may be re-run to start a new round of threat hunting.

[0089] Based on the reinforcement learning framework, the present invention utilizes the knowledge base and zero-sample learning ability of a large language model, combined with the strategy optimization and adaptive learning characteristics of reinforcement learning, to improve the efficiency of threat hunting. This method can solve the sparse reward problem in scenarios with slow convergence speed by introducing recursive incentive updates and method similarity quality, and use a large language model to introduce external knowledge in cold start scenarios, thus realizing an efficient threat hunting method.

[0090] In order to verify the technical effect of the technical solution proposed by the present invention, the following experimental evaluation was carried out.

[0091] In order to simulate the real threat hunting scenario, the experimental evaluation process of the present invention constructs a dataset for simulating threat hunting, which contains 186 threat intelligence reports. In order to simulate the threat hunting process of experts, the initial threat entity set is extracted from these reports, including IP addresses, domain names, URLs, and file hashes. The data sources are local, VirusTotal, RapidDNS, and ThreatBook, and 21 automatic query methods are implemented. When creating the dataset, the initial threat entity set is expanded to simulate the query process of experts. The dataset is analyzed. Each report in the dataset contains an average of 11.77 initial threat entity nodes. After expansion, the average number of entity nodes increases to 415.80, and the average number of valid queries is 925.47. Different data sources all participate in the contribution, of which VirusTotal accounts for 34.7%, local query data accounts for 34.0%, RapidDNS accounts for 23.3%, and ThreatBook accounts for 8.1%. The dataset can effectively simulate the process of threat hunting response on 186 threat events, and can effectively evaluate the efficiency of the method of the present invention in threat hunting response to threat events.

[0092] The experimental evaluation indicators of the present invention include average recall rate and average number of steps to reach the recall rate.

[0093] Average Recall: measures the proportion of threat nodes correctly identified under a fixed number of search steps. The higher the recall, the better the algorithm's ability to identify threat nodes within a given number of steps.

[0094]

[0095] In the formula, TP represents the number of correctly identified threat nodes, and FN represents the number of actual threat nodes that are not identified.

[0096] The average number of steps to reach recall (S R ): It is used to measure the average number of operation steps required to achieve a specific recall rate, reflecting the efficiency of the algorithm. The fewer the steps, the more efficient the algorithm is in the threat hunting process.

[0097]

[0098] In the formula, is the minimum number of steps required for the ith threat event to reach the recall threshold, and N is the total number of events.

[0099] The performance evaluation process is designed as follows: for each threat event in the data set, start from the context construction step and perform a threat hunting process with an upper limit of 15,000 rounds. Each event is independent of each other in the experiment, and no information is shared during the hunting process to ensure that the results are not affected by other events. The algorithms to be evaluated include: the RE-HUNTER algorithm implemented by the present invention, the MCTS algorithm that uses the MCTS-UCB strategy to implement node and method selection, the SA-UCB algorithm implemented using the SA strategy in MABAT (Dekel et al. 2021), and the SA-UCB-1 algorithm that uses the framework of the present invention to optimize the SA-UCB algorithm and divide the decision-making process into node and method selection processes.

[0100] like Figure 5 As shown, in the interval of 0-2000 steps, compared with different baseline methods, the average recall rate of the present invention is 18.24% higher than that of MCTS, 33.10% higher than that of SA-UCB-1, and 351% higher than that of SA-UCB. From the cold start stage of 0-250 steps, the average recall rate of the present invention is 86.28% higher than that of MCTS, 143.70% higher than that of SA-UCB-1, and 1375% higher than that of SA-UCB. Therefore, according to the experimental data, the algorithm of the present invention can produce a higher recall rate than the comparison method under the same number of threat hunting steps, and therefore has higher performance.

[0101] Table 1. The number of threat hunting decision steps required to achieve different average recall rates

[0102]

[0103] As shown in Table 1, the average number of operation steps required by the present invention is significantly less than that of other methods at the same recall rate, and the overall performance can be divided into three stages for analysis. Specifically, in the initial stage (10%, 20%, 30% recall rate), the present invention only needs 15 steps to achieve the 10% recall rate target, and reduces the number of operation steps by 53.85% and 16.95% at 20% and 30% recall rates, respectively, significantly improving efficiency. Entering the mid-term stage (40%, 50%, 60% recall rate), the method of the present invention continues to maintain a low step requirement, reducing the number of steps by 9.51%, 15.14% and 12.26%, respectively. In the later stage (70%, 80%, 90% recall rate), the present invention still shows obvious advantages at each recall rate level, and the step reduction rate reaches 20.85%, 26.24% and 35.60%, respectively. These results show that the method of the present invention can not only quickly start the threat hunting process, but also remain efficient when higher coverage threat intelligence is needed, effectively reducing the search cost. Specifically, with a recall rate of 90% on the evaluation dataset, the method of the present invention can reduce 1,047 decision and query steps in each threat hunting process on average.

[0104] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the invention.

Claims

1. A threat hunting method based on a large language model and reinforcement learning, characterized in that: The following steps are involved: Step 1: Context Building Extract the initial threat entities from the threat intelligence to form an initial threat hunting node set, and convert the threat intelligence text into a high-dimensional vector to establish a vector-text retrieval database; Step 2: Threat Hunting Node and Method Selection Based on the current threat hunting node set, the optimal threat node is selected through the reinforcement learning algorithm, and then the threat hunting method corresponding to the optimal threat node is selected to obtain the current optimal threat hunting method; if the method selection process of the current node is in the cold start stage, the external knowledge provided by the large language model is used to assist the reinforcement learning algorithm for method selection; if the method selection process of the current node is not in the cold start stage, the variant strategy of the upper confidence bound is used for method selection; finally, the selected threat hunting method is executed to generate a new set of child nodes; Step 3: Update node quality and method quality By recursively updating the node quality, the method quality is updated using similarity to solve the sparse reward problem, and finally the next threat hunting node set is generated.

2. The threat hunting method based on large language model and reinforcement learning according to claim 1, characterized in that: In step 1, the initial threat hunting node set is N0; in the initial stage, let the current threat hunting node set, that is, the Tth threat hunting node set N t is the initial threat hunting node set, that is, N t =N0; Preprocess the threat intelligence text and segment it into original text segments T′ by sentence i , each original text segment T′ i The word embedding model is used to convert the text into a high-dimensional vector and represented as V i ;Build vector-text retrieval database DB Context ={<V i ,T′ i >}。 3. The threat hunting method based on large language model and reinforcement learning according to claim 2, characterized in that: The step 2 is specifically as follows: Step 2.1: Based on the Tth threat hunting node set N t The expansion is carried out, and the threat hunting node selection is performed according to the following formula to obtain the current optimal threat node n i : In the formula, Q t (n) represents the node quality corresponding to the threat node n. When initialized, the node quality Q corresponding to all threat nodes t (n) is 0; P(n) represents the parent node of threat node n; N(n) represents the number of times the corresponding threat node n is selected as the optimal threat node in the historical threat hunting process; c is a hyperparameter for balancing exploration and exploitation; Step 2.2: Select the threat hunting method and obtain the current optimal threat hunting method a j ; If i ∈N0, then the current optimal threat node n i The method selection is set to the cold start stage, otherwise it is the non-cold start stage; If the current optimal threat node n i The method selection process is the cold start stage, so the cold start method selection strategy is used to introduce the external knowledge provided by the large language model to assist the reinforcement learning algorithm to select the method, and the current optimal threat hunting method a is obtained. j : In the formula, k i is the current optimal threat node n i Node type; A i is the current optimal threat node n i A collection of optional threat hunting methods; ki,a is the method quality matrix W t The elements in , which represent the node type k i The quality of using method a; the method quality matrix W t is an R×M matrix, where R represents the number of threat node types and M represents the total number of threat hunting methods; W t At initialization, all elements are set to 0; N(a) is the number of times method a is selected, LLMScore(a,n i ) is the current optimal threat node n calculated by the large language model i Applicability score with method a; If the current optimal threat node n i The method selection process is a non-cold start stage, and the upper confidence bound variant strategy is used for method selection, as shown in the following formula: In the formula, N(A i ) is the current optimal threat node n i The total number of visits to the method collection; Step 2.3: Execute the currently selected optimal threat hunting method a through reinforcement learning algorithm j , generate a new threat sub-node set C t+1 .

4. The threat hunting method based on large language model and reinforcement learning according to claim 3 is characterized in that: In step 2.2, the large language model is used to calculate the current optimal threat node n i The applicability score with method a is as follows: LLMScore(a,n i )=LLM(Context(n i ),Description(a)) In the formula, context Context(n i ) represents the search from vector-text database DB Context The node that is found in i Related contextual information and the corresponding vector representation; Description(a) is the description of method a; LLM(·) represents the reasoning process of the large language model, and outputs method a at the current optimal threat node n i Ratings on Extract Context(n i ) is implemented as follows: the vector database FAISS is used for retrieval, cosine similarity is used as the similarity metric, and the Top-K strategy is adopted to select the most relevant content from the top K context fragments with the highest similarity, so as to achieve accurate matching and rapid extraction of context fragments related to threat nodes.

5. The threat hunting method based on large language model and reinforcement learning according to claim 3, characterized in that: Step 3 is as follows: Step 3.1: Update the threat node set N t+1 =N t ∪C t+1 , and record the parent nodes of all newly generated threat nodes n′ in turn, and the current optimal threat node n i Update to the parent node of all newly generated threat nodes n′: Step 3.2: For each newly generated node n′, calculate the incentive ΔQ based on the feedback information t (n′), and traverse all newly generated nodes n′, and update the quality Q of the relevant nodes t (n): Among them, ΔQ t (n′) is the stimulus calculated based on the feedback information, Ancestors(n′) represents the set of all ancestor nodes of the newly generated node n′; Step 3.3: Update the method quality: Using the method similarity function Sim method , for the method quality matrix W t To update: Where A represents the set of all threat hunting methods; a j is the optimal threat node n at the current time i The current best threat hunting method selected on Sim method (a,a j ) is method a and method a j The similarity measure between .

6. The threat hunting method based on large language model and reinforcement learning according to claim 5, characterized in that: The node quality Q t In the update of (n), the excitation ΔQ t The strategy of (n′) is: for any newly generated node n′∈C t+1 , if n′ does not belong to the current threat hunting node set N t , then the incentive is 1, otherwise the incentive is 0.

7. The threat hunting method based on large language model and reinforcement learning according to claim 5, characterized in that: The method similarity function Sim method (a,a j ) is calculated as follows: for any two methods a, a j If they share the same data source, the similarity is increased by 0.1; if they share the same operation, the similarity is increased by 0.5.

Images