Network security script arrangement method based on LLM enhanced RL

Abstract

A network security script arrangement method based on LLM enhanced RL comprises the following steps: 1) LLM dynamically expands a to-be-selected strategy atom set based on a security threat scene: firstly, performing semantic similarity retrieval on input security threat intelligence by using a vector library, and matching an optimal defense strategy atom in a knowledge library; the context understanding capability of the LLM is then utilized to implement semantic understanding and reasoning on the candidate atom set, generating new policy atoms complementary to the defense. 2) RL refers to LLM atomic action preference to optimize a script generation strategy, and self-adaptive security arrangement of a complex network environment is realized: firstly, a security script graph model based on strategy atoms is adopted, and multiple constraints corresponding to nodes and edges in the graph model are abstracted on the basis of security arrangement definition; and then, a composite reward function is adopted to model a linear programming function for the multi-constrained security script arrangement problem, so that multi-dimensional optimization of defense efficiency and execution cost is realized.

Description

Technical Field

[0001] The present invention relates to the application of artificial intelligence technology in the field of network security, and specifically to a network security script arrangement method based on LLM enhanced RL. Background Art

[0002] With the rapid development of network technology, the global network environment is becoming increasingly complex and highly interconnected, with dynamic evolution. The continuous emergence of new network architectures and applications has blurred the boundaries of the network and expanded the attack surface. [1] The wide variety of network terminal devices and heterogeneous protocols have brought more diverse network security threats and challenges. Network attack methods are more covert and have stronger penetration. [2][3] In this context, how to effectively defend against and respond to new security threats in complex network environments has become a key issue that needs to be addressed in the field of network security. In this context, Security Orchestration, Automation and Response (SOAR) technology has emerged as an important component of network security defense. SOAR technology integrates threat intelligence, situational awareness, and automated response mechanisms to achieve cross-platform coordination and automatic execution of security policies, thereby improving overall security defense capabilities. [4] In recent years, reinforcement learning (RL) has been widely used in security orchestration, breaking through the traditional reliance on manually configured orchestration scripts. However, existing research mostly relies on static script libraries and lacks the ability to dynamically control script atoms in unfamiliar scenarios, limiting their effectiveness in actual emergency response.

[0003] Large Language Model (LLM) has gradually shown great potential in reasoning and decision support in the field of network security. Pternea et al. [5] Combining LLM with RL to analyze the potential for collaboration, in an environment lacking clear reward signals, LLM can effectively assist RL agents in making decisions and accelerate the learning process. Zhang et al. [6] It is proposed that LLM can optimize the performance of RL agents in many situations, especially in dynamic environments, when RL agents need to make decisions based on complex environments. The RLGF method uses LLM as the basic model for generating text, and RL guides LLM to generate content that is more suitable for the target task by providing reward feedback. [7] .

[0004] LLM boasts extensive knowledge transfer and contextual learning capabilities, dynamically expanding the set of policy atoms and addressing the limitations of predefined knowledge bases in unknown threat scenarios. Furthermore, RL, as a data-driven optimization method, enables policy search and reinforcement adjustments in complex environments to improve the effectiveness of playbook execution. By generating high-quality policy atoms through LLM and combining it with RL's optimization capabilities in dynamic environments, a more adaptable and intelligent security orchestration framework can be constructed, empowering security operations with enhanced decision-making and generalizability in responding to new attacks.

[0005] 1.1 Challenging Issues

[0006] Although the introduction of LLM can provide richer atomic extensions for security playbooks and offer more strategic references for RL decision-making, and has the potential to improve security orchestration, it still faces the following challenges:

[0007] 1) Accuracy and reliability of LLM generation strategy atoms: LLM text generation is essentially based on probability distribution, which is prone to hallucination problems. [8] In security operation and maintenance scenarios, LLM may generate logical errors or atomic operations that violate security policies due to hallucination phenomena, reducing the reliability of security script generation. Friel et al. [9] proposed a method for detecting hallucinations in LLM generation, which efficiently identifies hallucinations by analyzing the internal state, attention map, and output prediction probability of a single response. Martino et al.

[10] It is proposed to reduce the hallucination problem in LLM generated content through knowledge injection, and enhance the output accuracy of LLM by mapping the contextual data of task-related entities into the text space.

[0008] 2) LLM-assisted RL strategy interpretability and credibility: RL optimization strategies often lack transparency. In areas such as cybersecurity that require high transparency and verifiability, achieving interpretability in RL algorithms is crucial. By using a hierarchical approach and symbolic representation, the interpretability of some RL models can be significantly improved.

[11] The reasoning generation of LLM relies on a large amount of training data, and its policy preference may be affected by the bias of training data, resulting in unfair selection. Khediri et al.

[12] SHAP (SHapley Additive exPlanations) values ​​are used to explain the model's predictions, and LLM is used to generate easy-to-understand natural language descriptions, thereby helping security analysts understand the reasons behind anomaly detection and improving the transparency and trust of intrusion detection systems. How to incorporate LLM's preference for atom selection into RL training to improve the rationality and interpretability of the final generated security script requires further research. Lin et al.

[13] proposed a hybrid method combining RL and LLM, where RL is used to discover the defects of decision trees and combined with LLM to analyze failure scenarios and generate improved decision trees. Ma et al.

[14] A framework combining LLM and RL agents is proposed to support cybersecurity operations. A pre-trained RL agent acts as a tutor to help the LLM agent accelerate learning, and the LLM can dynamically obtain guidance from the RL agent. Summary of the Invention

[0009] To address the challenges posed by unfamiliar attacks and multi-dimensional, complex network environments, this paper proposes a security orchestration framework based on LLM and RL collaboration to enable interpretable, scenario-adaptive security script generation. Within this framework, LLM knowledge reasoning is combined with RL decision-making to achieve a sustainable, mutually empowering closed loop.

[0010] The present invention provides a network security scenario arrangement method based on LLM enhanced RL, comprising the following steps:

[0011] (1) Using the Large Language Model (LLM) to dynamically expand the set of candidate strategy atoms based on security threat scenarios;

[0012] (2) Using reinforcement learning (RL) algorithms to optimize the script generation strategy based on the LLM atomic action preferences, we can achieve adaptive security orchestration of network security scripts in complex network environments.

[0013] In step (1), the vector library is first used to perform semantic similarity retrieval on the input security threat intelligence and match the best defense strategy atoms in the knowledge base. Subsequently, the contextual understanding capability of the LLM is used to perform semantic understanding and reasoning on the candidate atom set to generate new strategy atoms that complement the defense.

[0014] In step (2), we first use a policy-atom-based security scenario graph model to abstract multiple constraints on nodes and edges in the corresponding graph model based on the security orchestration definition. Then, we use a composite reward function to model the multi-constrained security scenario orchestration problem as a linear programming function to achieve multi-dimensional optimization of defense effectiveness and execution cost.

[0015] In the network security script arrangement method, the security script consists of multiple defense actions, each action is an executable security measure; the security script is divided into a series of minimum executable units, which are called defense strategy atoms or strategy atoms, referred to as atoms.

[0016] The main contributions of this invention include:

[0017] 1) LLM-assisted policy atom augmentation: Analyze semantic matching to preliminarily screen atoms for coverage gaps and indicator shortcomings. Leverage LLM's natural language understanding and generation capabilities to dynamically create and expand security policy atoms during the security script generation process.

[0018] 2) LLM-assisted strategy atom scoring mechanism: Through the Tree of Thoughts (ToT) prompt project, LLM imitates the knowledge base atom scoring criteria and quantitatively evaluates newly generated strategy atoms based on four dimensions: processing time, resource usage, manual participation, and operation and maintenance effect, to ensure their feasibility and effectiveness.

[0019] 3) LLM and RL Policy Co-Optimization and Alignment: In the proposed LLM and RL co-optimization decision-making framework, RL is responsible for policy optimization, while LLM provides the prior probability for policy generation. Safety policy trajectories generated through multiple rounds of RL agent training are used to guide LLM fine-tuning. Through fine-tuning and merging, LLM is aligned with the RL policy, ensuring that the resulting safety scenario is both strategically sound and efficient.

[0020] In experimental validation, the present invention established a real enterprise network experimental system based on 5G-IoT to test the performance of scenarios generated by the proposed method in various security threat scenarios. The experiment compared the security scenarios generated by the proposed method with those generated solely using LLM and RL methods. The results confirmed that the scenarios generated by the proposed method have greater adaptability and effectiveness in complex and unfamiliar network scenarios, providing new insights for intelligent security operations and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 represents the atom matching and augmentation scheme.

[0022] Figure 2 Represents a security scenario graph model for LLM-enhanced atoms.

[0023] Figure 3 A functional framework representing the LLM-enhanced RL scheme.

[0024] Figure 4 Represents the LoRA fine-tuning and merging framework based on gradient clipping.

[0025] Figure 5 Represents the 5G-IoT experimental network topology diagram.

[0026] Figure 6 Represents a preprocessed data sample.

[0027] Figure 7 This indicates that RAG assists RL in implementing the secure orchestration process.

[0028] Figure 8Indicates the ToT prompt word of the design.

[0029] Figure 9 Indicates that ToT prompts LLM security orchestration process.

[0030] Figure 10(a) and Figure 10(b) show the performance scores of the script atoms, respectively, where:

[0031] Figure 10(a) shows the script performance scores before fine-tuning;

[0032] Figure 10(b) shows the performance scores of the scripts after fine-tuning.

[0033] Figure 11(a) and Figure 11(b) show the reward value and probability density distribution respectively, where:

[0034] Figure 11(a) shows the average total reward value.

[0035] Figure 11(b) shows the probability density distribution of reward values.

[0036] Figure 12(a) and Figure 12(b) show the failure rate and probability density distribution of the script operation and maintenance, respectively, where:

[0037] Figure 12(a) shows the average script operation and maintenance failure rate.

[0038] Figure 12(b) shows the probability density distribution of failure rate.

[0039] Figure 13(a) and Figure 13(b) show the operation and maintenance failure rate and script generation time for different numbers of atoms, respectively.

[0040] Figure 13(a) shows the average script operation and maintenance failure rate.

[0041] Figure 13(b) shows the average script generation time.

[0042] Figures 14(a) to 14(c) Represent the quality scores of security scripts, where:

[0043] Figure 14(a) shows the script BLEU score.

[0044] Figure 14(b) shows the ROUGE score of the script.

[0045] Figure 14(c) shows the GPT-4 score of the script. DETAILED DESCRIPTION 1. Summary of the Invention

[0047] With the continuous evolution of cyberattack technologies, security orchestration and response systems struggle to cope with heterogeneous devices, multi-source attacks, and rapidly changing network dynamics. However, existing security orchestration methods mostly rely on static script libraries and single reinforcement learning strategies, lacking dynamic control over script atoms in unfamiliar scenarios. This invention integrates semantic retrieval and large language model reasoning into the state-action space construction and decision-making process of reinforcement learning, constructing an orchestration framework with a complementary closed loop of "atomic knowledge base enhancement and decision optimization" to achieve adaptive generation of security scripts in complex network dynamics. The contextual learning and knowledge transfer capabilities of the large language model (LLM) are utilized to dynamically expand the set of candidate policy atoms based on the scenario. Combined with the mind tree prompting project, the large language model policy reasoning process is broken down step by step, increasing the interpretability and controllability of its reasoning process. The reinforcement learning (RL) algorithm optimizes the decision-making of the script generation strategy based on the action preferences of the large model atoms, achieving adaptive security orchestration for complex network environments.

[0048] 2 System Model

[0049] This section constructs a system model for a cybersecurity scenario orchestration method based on LLM-enhanced RL. This system model generates security scenarios through LLM-assisted retrieval enhancement and RL decision-making. This method achieves security scenario generation under multiple constraints by synergizing an LLM-driven policy atom expansion mechanism with an RL optimization algorithm. The core module consists of two components: dynamic matching and augmentation of policy atoms, and LLM-enhanced RL decision optimization.

[0050] In the dynamic matching and augmentation of strategy atoms, in order to improve the coverage and scenario adaptability of the script strategy, a dual mechanism of atomic semantic matching and LLM augmentation is proposed.

[15] The vector library searches for semantic similarity in the input security threat intelligence and matches it with the best defense strategy atoms in the knowledge base. The contextual understanding capabilities of the LLM are then leveraged to perform semantic understanding and reasoning on the candidate atom set, generating new strategy atoms that complement the defense logic.

[0051] In the LLM-enhanced RL decision optimization part, this paper constructs a security scenario graph model based on policy atoms, and abstracts multiple constraints on nodes and edges in the corresponding graph model based on the security orchestration definition. In addition, a composite reward function is designed to model the multi-constrained security scenario orchestration problem as a linear programming function, with the goal of achieving multi-dimensional optimization of defense effectiveness and execution cost. 2.1 Policy Atom Matching and Augmentation

[0052] Figure 1This paper demonstrates the design of an LLM-assisted policy atom matching and augmentation scheme. This scheme first uses a semantic embedding model to embed input security threat intelligence online, generating threat vectors. These vectors are then matched against static defense policy atoms in the FAISS vector library. The optimal policy atom is then selected through semantic similarity retrieval.

[0053] Building on cosine similarity, the Top-K atom matching method incorporates enhanced index retrieval technology from the FAISS vector library to further improve the accuracy of matching results. Matched policy atoms are incorporated into the ToT prompts in the mind tree as contextual references. LLM generates new policy atoms based on contextual semantic understanding and reasoning expansion, thereby enhancing the performance of defense strategies and attack coverage.

[0054] A security script consists of multiple defensive actions, each of which is an executable security measure, such as "isolating the host," "blocking traffic," or "modifying access control policies." To facilitate modeling and calculation, the security script is broken down into a series of minimal executable units, namely, policy atoms. Assume that each script in the security script knowledge base is broken down, and the resulting set of policy atom index numbers is represented as Constitute the policy atom knowledge base. The corresponding defense operation specific text description is recorded as d n .d n Describes specific defensive actions such as "limiting the transmission of credentials to a limited set of relying parties" and "suspending the processes running on the computer system". The network attack type corresponding to the policy atom n is marked as y n , used for subsequent index markers. and These are defined as the response time, process occupancy, manual participation, and defense effectiveness score of policy atom n. To ensure the rationality of the script, the selection of policy atom sets should consider the correlation between the corresponding defense action text description and threat intelligence.

[0055] Considering only the correlation between the text description of the policy atom defense action and the text description of the threat scenario, K policy atoms are selected from the policy atom knowledge base (see formula (1)) to obtain the selected policy atom index number set In addition, suppose the threat scenario of the current network environment is Its text description is recorded as is the correlation function between the policy atom n and the threat scenario text description. The target atom is filtered by the correlation function value, which is expressed as:

[0056]

[0057] Matched Finally, LLM combined with contextual learning and knowledge transfer capabilities is utilized to expand the atomic set to cover missing threat types or optimize the performance indicators of matched atoms. The set of possible attack types is represented as The set of attack types covered is Then the candidate atomic attack type coverage can be expressed as:

[0058]

[0059] Ω∈[0,1] is the preset attack type coverage threshold. The candidate atom set is judged to have insufficient type coverage. The coverage gap of the attack type is

[0060] Defining binary variables is the optimization judgment index of the strategy atom n. If the strategy atom If the defense effect score of is lower than the mean score of the candidate atoms, the atom is marked as an object to be optimized:

[0061]

[0062] in, It is the average score of all candidate atomic defense effects.

[0063] LLM supplements the selected policy atom set in two main aspects: one is to supplement the attack types with insufficient coverage, and the other is to supplement and replace security policy atoms with shortcomings in defense effect indicators. is used as a context reference to prompt LLM to generate the corresponding supplementary atom index set, which is expressed as:

[0064]

[0065] For any LLM-generated supplementary strategy atom h∈H, it must satisfy

[0066]

[0067]

[0068] The supplementary policy atoms generated by LLM can achieve the missing coverage of threat types and ensure that the defense effect scores of all policy atoms are higher than the mean of the candidate policy atom set. As the input of subsequent security orchestration, the final candidate atom index set is expressed as

[0069] 2.2LLM Enhanced Security Script Graph Model

[0070] like Figure 2 As shown, for a given This section constructs a graph model, represented as G = (V, E), to describe the connection relationship between all candidate security policy atoms. In G, Identified as node v in the graph n ∈V. v0 is specially defined as the security event alarm trigger atom, indicating the start of security orchestration. m With v n The directed edge between them is denoted as e m,n ∈E, reflecting the existence of a directed connection between strategy atoms m and n. After the atomic space is expanded by LLM, a type of atom is no longer unique and fixed, and there may be multiple choices, which means that node v n There may be multiple forward and backward directed edges.

[0071] Based on the proposed graph model, a security orchestration strategy selects the most appropriate node combination from the starting node to form a tree-like path graph. All vertices and directed edges on this tree-like path constitute a security scenario.

[0072] Definition 1 (Security Orchestration Strategy)

[0073] A security orchestration scenario can be modeled as a rooted tree, represented as Z = (V′, E′), where is the set of selected nodes, is the selected edge set that satisfies the following three conditions:

[0074] 1) The root node v0 must be selected:

[0075] v0∈V′(7)

[0076] 2) If node v n ∈V′, then there is at least one entry into v n Edges (except the root node v0) are also selected:

[0077]

[0078] 3) If edge e m,n ∈E′, then node v m and v n Must also be selected:

[0079]

[0080] 2.3 Modeling of Multi-Constrained Security Orchestration Problems

[0081] Binary decision variable y m,n Indicates whether the security orchestration script path passes through the directed edge e m,n, that is, whether to choose to continue executing policy atom n after executing policy atom m, satisfying:

[0082]

[0083] binary variable x n Indicates whether the security operation and maintenance policy atom n is selected for orchestration and satisfies:

[0084]

[0085] Each strategy selection step in the security script must refer to the real-time network situation, system configuration, load status and operation and maintenance records. These four network elements are represented as v n The function g1(v n )、g2(v n )、g3(v n ) and g4(v n ). Strategy atom n and are expressed as n The function f1(v n )、f2(v n )、f3(v n ) and f4(v n The four scoring items are standardized by formula (13), and all scoring indicators are mapped to the same [0,1] interval, which makes them comparable in the final objective function.

[0086]

[0087] By analyzing the threat intelligence report, LLM evaluates the total threat score of the current network environment. The total threat score is expressed as Used to characterize the overall index loss value of environmental security. In a security scenario path, the sum of all selected atomic defense effect scores should meet the same Comparative goals:

[0088]

[0089] The goal of security orchestration is to minimize security O&M costs while addressing security attack scenarios, thereby maximizing relative O&M benefits. Weight coefficients λ1, λ2, and λ3 are defined to balance the contribution of response time, process occupancy, and human participation score to O&M costs for selected atoms in a security scenario. Minimizing costs while ensuring successful network O&M—that is, maximizing relative O&M benefits—is modeled as a linear programming function:

[0090]

[0091] st

[0092] x0=1 (14)

[0093]

[0094]

[0095] The three constraints of the linear programming problem are respectively for the conditions in Definition 1. Equation (14) indicates from the data level that the root node v0 must be selected. n ∈V′, that is, x n =1, then as long as formula (15) is satisfied, all the entries into v in the safe scenario path n The decision variables of the edges are greater than or equal to 1, which proves that there is at least one entry into v n (Except the root node v0) edges are also selected. According to formula (16), for any e m,n ∈E′, that is, y m,n =1, the condition is established as y m,n ≤x m And y m,n ≤x n This means that x m ,x n ≥1,x m and x n It is a binary variable 0,1, then we can definitely get x m ,x n =1, that is, node v m and v n Definitely selected.

[0096] 3LLM Enhanced RL Scheme Design

[0097] like Figure 3 This section proposes an LLM-enhanced RL framework that leverages LLM's natural language understanding capabilities and its extensive cybersecurity knowledge storage to address the shortcomings of RL strategies and enhance the ability to intelligently orchestrate security scenarios in dynamic network environments. From a functional perspective, the proposed framework can be divided into four parts:

[0098] 1) Security situation analysis: Threat intelligence analysis is injected as a context prompt word, and LLM is prompted to understand the state of the network security environment and identify potential threats in the environment, and generate specific situation analysis information quantitatively processed to obtain For each RL decision, LLM predicts the real-time network situation based on the current atomic combination.

[0099] 2) Strategy Orchestration and Reward Setting: In this module, a scenario strategy graph model is generated. LLM augments the atomic set and assists in security situation analysis, further enriching the action space and state space of RL.

[0100] 3) LLM Assists RL Script Decision-Making: LLM provides prior probabilities for action preferences, helping RL algorithms train and generate safe scripts. LLM guidance improves the efficiency of RL algorithm script generation, making safe scripts more flexible and adaptable.

[0101] 4) Script Trajectory Feedback and Optimization: The safety policy trajectory data from RL training is used for LLM fine-tuning to align the LLM action selection preference strategy with the RL strategy, further adjusting and optimizing the safety script generation process.

[0102] 3.1 Security Situation Analysis

[0103] The goal of the security situation analysis part is to provide environment status information for subsequent RL decision-making. Threat intelligence in the network environment is extracted and used as input context. It is injected into the LLM in the form of prompt words to guide the model to identify security risks and potential threats in the current environment. LLM calculates the total threat score of the environment by analyzing various types of information in the threat intelligence. It reflects the degree of deviation of the current network environment from the ideal state and the impact of potential threats on environmental security.

[0104] During the RL training decision-making process, LLM predicts the network status in real time based on the selected scenario atom combination. The status data includes network status, system configuration, load status and operation and maintenance records, which are quantified by LLM to represent the g1(v n )、g2(v n )、g3(v n ) and g4(v n ) score. Based on the data scores, LLM assesses potential changes in the network environment after the script is executed and predicts the security situation at the next time. This process ensures that during each RL action decision training, the network environment change predictions from LLM are referenced in real time and can be used to predict future security threats.

[0105] 3.2 State and Action Space Enhancement

[0106] LLM, powered by its natural language understanding capabilities and extensive cybersecurity knowledge, is prompted to generate new operational strategy atoms that are tailored to the scenario. Furthermore, LLM analyzes and predicts network situations based on threat intelligence reports and selected scenario atoms. This enhanced situational awareness further enhances the adaptability and flexibility of the RL framework. On this basis, the LLM-assisted RL scenario generation problem is modeled as a discrete-time Markov decision process (MDP).

[16] , represented as a four-tuple

[0107] State space In a high-dimensional state space, the corresponding state s in round t t Includes the currently selected script atomic group

[0108] The combined graph model and the network status, system configuration, load status and operation and maintenance record score corresponding to the finally selected atom j.

[0109]

[0110] Action space Contains security policy atoms filtered from the knowledge base and new policy atoms generated by LLM.

[0111] Action a t is in state s t The security policy atom selected below includes scores for the policy atom itself and its relative processing time, resource usage, manual participation, and operation and maintenance effect.

[0112]

[0113] ●State transition probability P: represents the transition from the current state s t Transfer to next state t+1 The probability of choosing the strategy

[0114] Make a t Then select strategy action a t+1 probability.

[0115] ●Cumulative reward R: a collection of reward values, where r t Represents a given state s t and action a t When, from state s t Transfer to next

[0116] One-step state s t+1 The reward value obtained from the feedback.

[0117] LLM will g1(v n )、g2(v n )、g3(v n ) and g4(v n ) and other network state elements into high-dimensional vector representations, enhancing the state space Helps RL agents better understand the task environment. Combined with the policy atoms matched by the knowledge base, LLM generates some new policy atoms that fit the network scenario and enhances the action space. This allows the agent to have more action options. Based on the assistance of these two dimensions of LLM, the action and state space of RL are enhanced.

[0118] 3.3LLM-assisted RL offline training

[0119] LLM is used to provide the prior probability of action preference for RL algorithm to improve the efficiency and accuracy of RL training to generate safe scripts. t Under the prompt word instruction, predict the next state s t+1 , and for optional Predict action preference probability distribution p LLM (a t |τ t ). Among them, the RL training trajectory τ t ={(s1,a1,r1),...,(s t ,a t ,r t )} consists of state, action, and reward.

[0120] Strategy π θ (a t |s t ) indicates that the RL algorithm is in state s t Next take action a t The probability of. At time t, the algorithm combines p LLM and strategy π θ (a t |s t ) Decision selection action a t , the immediate reward is expressed as:

[0121]

[0122] in, f1 is the normalized value of the operation and maintenance effect score of the selected action, i.e., the atom. norm (a t )、f2 norm (a t ), Represents the normalized response time, process occupancy, and human participation score corresponding to the atom. For each trajectory τ, its cumulative reward R(τ) is calculated in the form of discounted reward and used to update the parameters of the RL strategy. The cumulative reward is expressed as:

[0123]

[0124] Where γ∈[0,1] is a discount factor that controls the importance of future rewards. During offline training, the goal of the RL algorithm is to optimize the policy π θ , maximize the expected cumulative reward, that is:

[0125]

[0126] To maximize J(θ), the gradient strategy method

[17] is used to update the RL policy parameters θ. The policy gradient is calculated as follows:

[0127]

[0128] By estimating the gradient of the policy, the algorithm can clearly determine the direction of the optimal policy atom in each state and optimize the policy parameters accordingly. Finally, the policy parameters are iteratively updated using the gradient ascent method, and the policy parameters are updated to:

[0129]

[0130] Where η is the learning rate, which controls the magnitude of each parameter update. Through repeated training, the RL strategy tends to be optimal, generating adaptable and efficient safety scenarios.

[0131]

[0132] The LLM-assisted RL offline training process is summarized as Algorithm 1. Assume that the atomic action selection balance coefficient β. In each round of offline training, RL selects action a by adjusting the strategy with β and the LLM preference probability. t (See row 5). The cumulative operation and maintenance scores are expressed as Contains the operational effectiveness scores of all selected orchestration atoms. The comparison of X and S values ​​serves as an evaluation of the script's effectiveness in addressing security threats. Policy updates are achieved by adjusting the learning rate and reward function weight coefficients (see line 9). When the policy gradient change falls below a set threshold of 0.01, the algorithm stops training and returns the current policy as the optimal policy. Through repeated training and parameter updates, the RL policy ultimately achieves efficient convergence, generating a stable and highly adaptable security script.

[0133] 3.4 Fine-tuning alignment of LLM to RL policy

[0134] For the arrangement decision of the safety script, LLM provides the probability of atomic selection preference as a reference for RL strategy update. In order to improve the alignment of LLM with RL strategy, this section designs a LoRA

[18] The fine-tuning and parameter merging framework ensures that the fine-tuned LLM can not only learn the RL-generated policy but also maintain the stability and generalization ability of the original model.

[0135] like Figure 4 The framework includes two stages: LoRA fine-tuning training and gradient clipping-based LoRA adapter LLM parameter merging, as follows:

[0136] (1) LoRA fine-tuning training

[0137] The core idea is to freeze the pre-trained model weights W and introduce low-rank matrices A and B for parameter updates:

[0138] W′=W+ΔW, ΔW=BA (24)

[0139] in, are the original weights of the pre-trained model, It is a trainable low-rank matrix, r<<d, which greatly reduces the cost of parameter update.

[0140] At the end of the nth round of RL training, the generated security policy trajectory is expressed as To optimize the fine-tuning data for LLM, we filter all time-step reward values ​​above the reward threshold Ω + High reward trajectory and reward values ​​are both lower than the reward threshold Ω - Low reward trajectory The final fine-tuning dataset is updated to by Fine-tuning the LLM for the training sample, the loss function is expressed as:

[0141]

[0142] Among them, π LLM (a|s) is the probability distribution of LLM choosing action a in state s, π RL (a|s) is the policy distribution of the RL agent in the same state. KL divergence

[19] is calculated to measure the difference between the policy generated by the LLM and the policy of the RL agent, expressed as:

[0143]

[0144] When the policy generated by the LLM perfectly matches the RL policy, the KL divergence is 0; otherwise, the KL is positively correlated with the policy deviation between the two, and the loss function will guide the LLM to update.

[0145] The purpose of fine-tuning is to gradually align the action selection preferences of LLM with the decision requirements of RL, that is, to minimize The gradient of fine-tuning is calculated as:

[0146]

[0147] g ΔW Represents the parameter gradient update direction of A and B, ensuring that gradient descent helps reduce KL divergence.

[0148] (2) Parameter fusion based on gradient clipping

[0149] Although LoRA fine-tuning only optimizes the low-rank matrices A and B, the gradient may change drastically during training, especially when the KL divergence is large, which may cause gradient explosion.

[20] is introduced to constrain the magnitude of gradient updates and ensure training stability. At each step of LoRA training, the gradient g ΔW is clipped to implement the gradient norm limit, expressed as:

[0150]

[0151] Among them, ‖g ΔW ‖2 is the L2 of the gradient

[21] norm, c is the preset gradient clipping threshold. According to the clipped gradient, the LoRA low-rank matrix is ​​updated as follows:

[0152]

[0153] The fine-tuned LLM parameters cannot directly replace the original model parameters to avoid overfitting the model or loss of its original capabilities. LLM and the fine-tuned LoRA model parameters θ′ LLM Fusion can retain the LLM capability while introducing the fine-tuned policy preferences. A weighted fusion strategy is adopted, and the LoRA fine-tuned parameters and the original LLM parameters are merged into:

[0154] θ″ LLM =ωθ′ LLM +(1-ω)θ LLM (30)

[0155] Among them, ω is the fusion parameter, which determines the weight distribution of the original model and the fine-tuned model. In order to dynamically adjust the fusion parameter, an adaptive fusion strategy based on KL divergence is designed

[22] , ω is adjusted to

[0156]

[0157] This ensures that when D KL When D is small (LLM is well aligned with RL strategy), ω is close to 1, and more fine-tuned model parameters are retained. KL If ω is large (the policy still has large deviations after fine-tuning), ω is reduced to retain more original LLM information and prevent overfitting. The fine-tuning-based RL alignment process is summarized as Algorithm 2.

[0158]

[0159] After fusion and merging, the LLM forms a new parameter configuration that makes it more favorable for RL-generated security policies while also ensuring generalization capabilities. The final model can be directly applied to security decision-making tasks, ensuring the generated security policies are efficient and reliable.

[0160] 4 Experimental Preparation

[0161] 4.1 Experimental Environment Construction

[0162] The integration of 5G and IoT has become a trend in network technology development. Compared to previous communication networks, 5G networks are significantly more complex and open, making them more vulnerable to various network security threats and attacks.

[0163] like Figure 5 A complex experimental network environment simulating real enterprise networks and 5G-IoT applications was built to test the effectiveness of the generated security protection strategies in real-world scenarios. The perimeter firewall and internal firewall separated the environment into network zones with different security levels, specifically:

[0164] 1) C2 penetration zone: Contains attacker hosts (Linux and Windows), simulating external threat sources and connected to the network through the Internet.

[0165] 2) DMZ: Contains database, web, and FTP servers. This area serves as a buffer zone for server deployment, primarily for risk isolation, access control, and server isolation.

[0166] 3) 5G core network area: includes 5GC network elements (AMF / SMF / UPF) and 5GC servers, which are used for signal processing and data management of 5G networks and are connected to other areas through multiple gateways.

[0167] 4) Security Management Area: This area includes the bastion host and SOC security management equipment, providing overall network security management and policy implementation.

[0168] 5) Internal Application and IoT Zone: This zone includes IoT devices such as PCs, mobile phones, and sensors, as well as application server clusters. These devices are connected to the core network via multiple base stations, involving multiple wireless communication nodes.

[0169] For the proposed 5G-IoT experimental network scenario, this section uses the NVD general vulnerability database

[23] Targeted vulnerabilities are deployed for each area. Table 1 gives the specific information of some vulnerabilities.

[0170] Table 1 Vulnerability information

[0171]

[0172] 4.2 Knowledge Base Construction

[0173] ATT&CK

[24] Data related to defensive mitigation measures in the cybersecurity knowledge base was collected and used to build the experimental knowledge base. These measures include strategies and technologies that reduce and prevent attackers from performing malicious actions in the network environment. Table 2 summarizes the contents of the knowledge base and shows the distribution and proportion of strategy and technology categories in three security scenarios. Each strategy may correspond to many different technology categories. For example, the data backup strategy corresponds to multiple technologies such as regular backup plans, backup encryption, and off-site backup storage. The three scenarios are enterprise environments (Enterprise), mobile environments (Mobile), and industrial control system (ICS) environments. ICS refers to networks and systems used to control and monitor industrial processes.

[0174] Table 2 Experimental knowledge base categories and distribution

[0175]

[0176] In the knowledge base, each strategy in each scenario contains multiple corresponding specific technologies. To facilitate searching, all technologies are categorized and stored according to their corresponding strategy IDs. Many strategies are applicable to multiple scenarios, but the corresponding technologies differ. Therefore, the strategy ID is used as the file name to store all corresponding technologies. Technologies for different scenarios are further divided into different dictionaries to achieve structured storage in the knowledge base. Figure 6 This section presents examples of knowledge base data preprocessed using scoring criteria with the assistance of GPT-4. GPT-4's scoring method offers advantages in data evaluation and processing, comprehensively evaluating atomic, multi-dimensional metrics to improve script interpretability. Table 3 lists the specific scoring criteria. The scoring criteria primarily consider the strategy or technique's importance and applicability within the environment, its implementation complexity and potential system impact, its impact on daily operations and system performance, and its expected effectiveness in improving overall security.

[0177] Table 3 Strategy and technology scoring criteria

[0178]

[0179] 4.3 Method Implementation

[0180] Figure 7 Demonstration based on LangChain framework

[25] and RAG

[26] The security orchestration system process constructed by the technology is designed to enhance the relevance and accuracy of the content generated by LLM in security orchestration by plugging in the security knowledge base.

[0181] The system process is divided into three stages according to the processing logic: data processing, retrieval enhancement, and script generation:

[0182] (1) Data processing stage

[0183] The system loads data from preprocessed, structured security operations policy atom JSON files. LangChain's JSON loader batch-loads nested policy files and converts them into a Python dictionary format. To address the nested nature of policy atoms, this study designed a logically based dictionary item splitter that recursively splits each policy atom and generates independent item blocks by scenario classification. For policy items containing multiple levels of classification (such as "System Security / Patch Management"), the splitter retains the complete classification path as metadata and extracts the "Description" field as the core semantic content.

[0184] To support semantic similarity retrieval, the description text of the policy atom is transformed by sentence-transformers / all-mpnet-base-v2

[27] Pre-trained language model vector embedding. All description texts are batch-converted into 768-dimensional normalized vectors and injected into the FAISS vector database along with metadata such as policy ID and priority.

[0185] (2) Retrieval enhancement stage

[0186] Retrieval enhancement in this section is mainly reflected in RAG and ToT

[28] There are two aspects to this. The network attack threat intelligence is mapped into a vector through the embedding model. The system integrates structured filtering and retrieval, and returns the most relevant strategy atom set in the vector knowledge base based on semantic similarity. Figure 8 ,The retrieved policy atoms are combined with threat intelligence and ,embedded in the prompt words for constructing LLM.

[0187] Step 1: LLM builds a corresponding security orchestration script template based on threat intelligence analysis application scenarios, clarifying the basic structure such as trigger conditions, execution actions, and verification steps.

[0188] Step 2: The model selects appropriate content from the retrieved security policy atoms to populate the script template. If the atomic attack coverage is insufficient or there are shortcomings in the defense effectiveness indicators, the model imitates the existing atomic style to generate corresponding atomic actions.

[0189] Step ③: LLM scores the additional atomic blocks based on the scoring criteria of existing atoms to ensure that they meet the existing standards.

[0190] Step 4: LLM sets a preference value (ranging from 0 to 1) for each action in the script. Atoms with a value exceeding 0.8 are marked as highly recommended atoms, and the complete script is finally output.

[0191] Figure 9This figure illustrates the multi-round, iterative prompting structure of the ToT model. Complex security orchestration tasks are broken down into multiple thinking rounds, each with specific sub-questions and guidance to ensure the model systematically completes the task. The green blocks correspond to atoms matched from the knowledge base, the pink blocks correspond to supplementary atoms generated by the LLM through prompting, and the arrays represent the four scores of the atoms. Finally, in the fourth round, the LLM is prompted to analyze a combination of cyber threat intelligence and atom-related attribute analysis scenarios, providing a preferred probability for atom selection. Atoms highlighted in yellow in the figure indicate atoms with a higher probability of preference.

[0192] (3) Script generation stage

[0193] The scenario generated by the LLM is split into an RL action space, and the preference value of each action serves as a reference indicator for RL reward training. After multiple rounds of RL training, the RL converges to a stable policy, ultimately generating a scenario-specific safety scenario.

[0194] 5 Experimental design and result analysis

[0195] In order to comprehensively and objectively evaluate the performance of the proposed solution, a solution that relies solely on LLM and RL to generate scripts is selected as a benchmark.

[29] This method uses a limited number of behavioral strategies to achieve flexible control of the threshold of dynamic security scripts, which is set as Baseline-1.

[30] The method adjusts the three-stage process (initial script generation → vulnerability repair → cross-validation and optimization of script logic) to gradually generate a tree-like security script response process, which is set as Baseline-2.

[0196] Table 4 Experimental method settings

[0197]

[0198] Table 4 shows the specific experimental configuration details of the proposed method, and conducts evaluation from three dimensions: knowledge enhancement mechanism, LLM reasoning and RL decision collaboration, and fine-tuning. More specifically, the knowledge enhancement mechanism is subdivided into static knowledge base, dynamic knowledge base, and ToT prompts; LLM reasoning and RL decision collaboration are refined according to model scale and RL type, and the RL algorithms are optimized for security orchestration scenarios. This section uses the LLM-PGRL method with dynamic knowledge base and ToT reasoning as Proposed-1. Proposed-2-7 respectively compare the ablation of Proposed-1 from four dimensions: knowledge enhancement method, base model scale, RL algorithm, and fine-tuning rounds. Qwen

[31] It is selected as the base model and experimentally compared with Qwen7B and Qwen13B to study the impact of LLMs with different parameter amounts on security orchestration.

[0199] In this section, we select five attack types to conduct attack and defense experiments in a network environment for the five vulnerabilities listed in Table 1. The LLM fine-tuning and alignment settings are set at the end of the 40th round of RL training in the experiment.

[0200] 5.1 Performance Analysis of Knowledge Enhancement Mechanism

[0201] The first set of experiments evaluates the impact of different knowledge enhancement mechanisms in the proposed method on the script generation performance, specifically focusing on the number of atoms in the generated script, as well as indicators such as precision and recall compared to the knowledge base.

[0202] Table 5. Experimental results of performance comparison of knowledge enhancement mechanism

[0203]

[0204] As shown in Table 5, in the untuned phase, Proposed-1 achieved the highest precision of 85.2% and recall of 78.5%, with an F1 score of 81.7%, and generated scripts with an average of 14.7 atoms. In contrast, Proposed-2, which only used static knowledge base augmentation, performed the worst in this phase and generated scripts with the highest average number of atoms. The combination of dynamic knowledge augmentation and the ToT reasoning mechanism effectively improved the quality of atomic action generation in security scripts. In the static knowledge base augmentation method, the high-quality atoms added by the LLM in the training-generated scripts were not added to the knowledge base, necessitating the orchestration of more atoms to achieve the script operation and maintenance goals.

[0205] Figure 10(b) shows that after fine-tuning, the performance of each method improved. Proposed-1's F1 score further increased to 86.7%, achieving the best balance between precision and recall. Although Proposed-3 and Proposed-4 each separately introduced a dynamic knowledge base or ToT mechanism, their combined effect was still not as outstanding as the two. In addition, fine-tuning effectively reduced the average number of atoms generated in the script, reducing the redundancy of script atoms and demonstrating a more refined and accurate structure. Proposed-2, Proposed-3, and Proposed-4 also achieved significant improvements in F1 scores after fine-tuning. These results indicate that introducing LLM fine-tuning helps improve metrics such as the knowledge recall rate of script atoms, and that more atoms in the knowledge base provide support for stabilizing the script results.

[0206] 5.2 Performance Analysis of Security Orchestration under Different RL Methods

[0207] The second set of experiments evaluated the security orchestration performance analysis of different RL methods, specifically focusing on the draw reward value and the probability density distribution of the reward value during RL training. Furthermore, the failure rate and probability density of the generated scripts during attack and defense testing in the experimental environment were also used as reference indicators for the security orchestration performance analysis.

[0208] Figure 11 shows the reward trends and final reward distribution densities of different RL methods during training. Judging from the average total reward values ​​in Figure 11(a), the performance of all methods shows an improving trend with increasing training rounds. Judging from the training convergence curves, Proposed-1 and Proposed-6 training are relatively stable, with the final rewards converging to above 70 after around 50 rounds. Proposed-7 training fluctuates significantly, with the final reward slightly lower than the previous two. This is because the LLM strategy was not fine-tuned to align with RL, and RL training was significantly affected by the LLM's own preference values ​​and difficult to converge. Baseline-1 performed the worst, with violent oscillations during training, and the reward value stabilized between approximately 60 and 62 in the later stages of training.

[0209] Judging from the reward probability density distribution in Figure 11(b), the reward density distributions for Proposed-1 and Proposed-6 are concentrated in the higher range, with peaks significantly to the right and more concentrated. This indicates a more stable distribution of higher reward returns and a more consistent quality of generated scripts. The rewards for Proposed-7 shift slightly to around 65, while the reward distribution for Baseline-1 is the lowest and more dispersed, indicating weaker script optimization capabilities and a lack of stable performance. Overall, relying solely on RL algorithms underperforms the combined LLM logical reasoning and RL strategy in terms of script reward values ​​and probability density distribution, and fine-tuning can further enhance the coordination between LLM reasoning and RL strategies.

[0210] Figure 12 shows the failure rate trends and probability distributions of operational execution in an experimental environment guided by scripts generated by training using several methods. The average script failure rate curve in Figure 12(a) shows that Proposed-1's failure rate decreased the fastest, ultimately stabilizing below 4%, demonstrating that its optimization strategy effectively reduces the likelihood of script execution failure. Proposed-6's failure rate was slightly higher than Proposed-1's, while Proposed-7's failure rate steadily decreased to around 5%. Baseline-1's failure rate remained consistently high, ultimately remaining above 7%, highlighting the limitations of relying solely on RL-generated scripts in real-world operational environments. The failure rate probability density distribution in Figure 12(b) shows that the failure rates of Proposed-1 and Proposed-6 are concentrated around 4%, with high peak density, indicating the strongest operational stability. Baseline-1's failure rate distribution is far to the right, indicating unstable script quality and poor overall operational stability. The above analysis demonstrates that Proposed-1 significantly reduces script failure rates by integrating a more effective LLM reasoning and RL decision-making synergy. In addition, the fine-tuning alignment of LLM to RL strategy effectively improves the reliability and stability of script maintenance.

[0211] 5.3 Performance Analysis of Scripts under Different LLM Settings

[0212] The third set of experiments evaluated script performance analysis under different LLM settings, specifically focusing on the operational failure rate and script generation time of several script generation methods under different atom number settings. Furthermore, this section used commonly used LLM text generation quality evaluation metrics to assess script quality. The GPT-4 model was also used to score the generated scripts based on completeness, interpretability, and scenario-specificity.

[0213] (1) Script performance analysis under different atomic numbers

[0214] To verify the impact of the number of atoms on the script operation failure rate and generation time, this section's experimental design adjusts the number of Top-K atoms in semantic retrieval matching recall. Figure 13 shows the impact of different methods on the failure rate and script generation time as the number of script atoms changes. Judging from the average script operation failure rate in Figure 13(a), the failure rate of each method shows a gradual downward trend as the number of atoms increases. Among them, Proposed-5 exhibits the lowest failure rate under different numbers of atoms, especially when the number of atoms exceeds 25, the failure rate stabilizes at around 3.5%. Baseline-2 has the highest overall failure rate, maintaining above 6%, demonstrating its lack of stability in complex script generation environments.

[0215] As shown in Figure 13(b), the average script generation time for all three methods increases significantly with increasing atom count. Baseline-2 has the shortest generation time, ranging from approximately 50 to 80 seconds. Relying solely on the LLM method eliminates the time required for RL decision-making, resulting in an advantage in generation speed. Proposed-5 has the longest generation time, reaching approximately 130 seconds, indicating that script generation efficiency is significantly impacted by increasing model size. The above analysis shows that increasing the size of the base model reduces the script operation and maintenance failure rate to a certain extent, but also results in a corresponding decrease in script generation efficiency. Proposed-1, based on the 7B model, achieves a better balance between script generation performance and efficiency.

[0216] (2) Script quality assessment

[0217] For the five predefined attack types, this set of experiments selected corresponding policy atoms from the knowledge base and, with the assistance of the GPT-4 model and security experts, generated five standard reference scripts to serve as a benchmark for subsequent evaluation. To comprehensively and objectively evaluate the quality of security scripts generated by each method, this section introduces a variety of subjective and objective evaluation metrics, including the BLEU metric for measuring the exact match between the generated scripts and the standard scripts, and the ROUGE metric for assessing recall and structural similarity to the standard scripts. GPT-4-based manual scoring subjectively evaluates script quality across three dimensions: completeness, interpretability, and scenario-specificity, enabling a comprehensive quality analysis of the generated scripts.

[0218] 1) BLEU

[32] It is an objective evaluation method based on n-gram word sequence matching, which mainly measures the degree of exact matching between the generated text and the reference text, and is expressed as:

[0219]

[0220] Among them, w n is the weight of the n-gram of the nth order, usually distributed equally. The exact match rate of the n-gram of the nth order is expressed as:

[0221]

[0222] Among them, C * is the candidate text set, N max (n-gram) is the maximum number of times the n-gram word sequence in the candidate text appears in the reference text. BP is the length penalty. If the generated text is too short, the BLEU score will be penalized. The specific rules are summarized as follows:

[0223]

[0224] Where c is the length of the candidate text and r is the length of the reference text. In this experiment, the BLEU-1, BLEU-2, and BLEU-4 indicator rules are summarized as follows:

[0225] BLEU-1: Calculates word-level matching of script text, reflecting basic vocabulary consistency.

[0226] BLEU-2: Calculates the matching degree of two-word phrases in the script text and evaluates the consistency of phrase-level expressions.

[0227] BLEU-4: Calculates the matching degree of four-gram phrases in the script text and measures the matching of long sentence structures.

[0228] 2) ROUGE

[32] The recall rate used to evaluate the generated text measures the text structure similarity by calculating the degree of matching at the n-gram level and the longest common subsequence between the reference text and the candidate text. In this experiment, the three indicator rules of ROUGE-1, ROUGE-2, and ROUGE-L are summarized as follows:

[0229] ●ROUGE-1: Calculates word-level recall of script text and evaluates basic vocabulary matching.

[0230] ●ROUGE-2: Calculates the recall rate of two-word phrases in script text and measures phrase-level similarity.

[0231] ●ROUGE-L: Calculates the longest common subsequence of the script text to measure the similarity of the overall structure.

[0232] The calculation method of ROUGE-1 and ROUGE-2 is:

[0233]

[0234] Among them, N match (n-gram) represents the number of n-grams in the reference text that match the candidate text.

[0235] The calculation formula for ROUGE-L is:

[0236]

[0237] Among them, R LCS is the recall rate, which indicates the matching degree of the longest common subsequence. LCS is the precision, which represents the proportion of the longest common subsequence. β adjusts the relationship between recall and precision.

[0238] 3) GPT-4 scoring is used to subjectively evaluate the completeness, interpretability, and context-specificity of generated text. The specific criteria are set as follows:

[0239] ●Completeness (A): Evaluate whether the generated scenario covers the necessary defense steps and action logic.

[0240] ● Explainability (I): Evaluate the clarity of the generated script, the rationality of the logical branches, and the

[0241] effectiveness of action guidance.

[0242] Scenario-specific (T): Evaluate whether the generated script meets the needs of specific security scenarios and can solve actual scenarios

[0243] The problem in .

[0244] GPT-4 uses a 5-point scoring system, with 1 indicating the lowest quality and 5 indicating the highest quality. The quality evaluation results of the generated scripts are shown in Table 6.

[0245] Table 6 Safety script quality evaluation results

[0246]

[0247]

[0248] Figure 14(a) shows that Proposed-5 performs best in terms of BLEU scores, with BLEU-1, BLEU-2, and BLEU-4 scores of 56.82%, 42.37%, and 29.81%, respectively. This demonstrates the high accuracy of the generated scripts in both short and long sentences. Proposed-1 follows closely behind, with a slightly lower BLEU score than Proposed-5, but still higher than Baseline-2. Baseline-2 performs the worst, with a BLEU-4 score of only 21.70%, indicating that the accuracy of large-scale models alone in script text representation is relatively low. The ROUGE scores shown in Figure 14(b) show that the rankings of the methods are consistent with their BLEU performance. Proposed-5 leads the other methods overall, with ROUGE-1, ROUGE-2, and L scores of 60.73%, 44.97%, and 49.31%, respectively. This indicates that the generated safe scripts have the highest similarity in overall logical structure with the reference scripts. The ROUGE values ​​of Proposed-1 and Proposed-7 are slightly lower than Proposed-5, but both are significantly better than Baseline-2, further demonstrating that the introduction of RL and knowledge augmentation mechanisms can effectively improve script quality. Figure 14(c) shows the human scores of GPT-4, with Baseline-2 receiving the lowest scores, including 3.54 for completeness, 2.89 for explainability, and 3.27 for scenario-specificity. This indicates that the single LLM approach suffers from significant logical flaws and insufficient usability when generating scripts in real-world scenarios. In summary, Proposed-5, based on the 13B model and incorporating knowledge augmentation and RL mechanisms, performs best across all evaluation dimensions of script generation quality. Baseline-2, a pure LLM approach, performs significantly worse, demonstrating that LLM approaches combined with RL and knowledge augmentation have advantages in secure script generation scenarios.

[0249] 6. Summary

[0250] To address the limited adaptability of static security knowledge bases in complex dynamic threat scenarios, this paper proposes a security script orchestration method that collaborates with LLM and RL. This method leverages the contextual learning and knowledge transfer capabilities of LLM to dynamically expand the set of policy atoms. Furthermore, a ToT reasoning method is used to prompt the LLM to perform multi-dimensional quantitative scoring of new atoms to ensure the feasibility and security of the policy. The RL algorithm optimizes the policy based on the reference LLM action selection preferences. The model is fine-tuned and merged through policy trajectory parameters to achieve alignment with the RL policy, enhancing decision interpretability. Experiments demonstrate that the proposed method improves adaptability to new attacks in simulated enterprise networks, 5G core networks, and IoT environments, confirming the efficiency and superiority of the LLM and RL collaborative framework for security orchestration in complex network environments.

[0251] References

[0252] [1]Pirbhulal S, Chockalingam S, Shukla A, et al. IoT cybersecurity in 5Gand beyond: a systematic literature review [J]. International Journal of Information Security, 2024, 23(4): 2827-2879.

[0253] [2]Baniya P,Agrawal A.The Internet of Things: Security Challenges andOpportunities[C] / / 2024 3rd International conference on Power Electronics and IoT Applications in Renewable Energy and its Control(PARC).IEEE,2024:153-158.

[0254] [3]Cook J,Rehman SU,Khan M A.Security and Privacy for Low Power IoTDevices on 5G and Beyond Networks:Challenges and Future Directions[J].IEEEAccess,2023,11:39295-39317.

[0255] [4] Zhao Yuezheng, Ye Jianwei, Yun Shan, et al. Construction of key technologies for security operation automation based on SOAR and future evolution direction [J]. Information Technology and Network Security, 2021, 40(3): 19-27.

[0256] [5]Pternea M, Singh P, Chakraborty A, et al. The rl / llm taxonomy tree: Reviewing synergies between reinforcement learning and large language models [J]. Journal of Artificial Intelligence Research, 2024, 80: 1525-1573.

[0257] [6]Zhang S, Zheng S, Ke S, et al. How can llm guide rl? a value-basedapproach[J].arXiv preprint arXiv:2402.16181,2024.

[0258] [7]Chang JD,Brantley K,Ramamurthy R,et al.Learning to generate better than your llm[J].arXiv preprint arXiv:2306.11816,2023.

[0259] [8] Huang L, Yu W, Ma W, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions [J]. ACM Transactions on Information Systems, 2025, 43(2): 1-55.

[0260] [9]Friel R,Sanyal A.Chainpoll:A high efficacy method for llmhallucination detection[J].arXiv preprint arXiv:2310.18344,2023.

[0261]

[10] Martino A,Iannelli M,Truong C.Knowledge injection to counterlarge language model(LLM)hallucination[C] / / European Semantic WebConference.Cham:Springer Nature Switzerland,2023:182-185.

[0262]

[11] Glanois C,Weng P,Zimmer M,et al.A survey on interpretablereinforcement learning[J].Machine Learning,2024,113(8):5847-5890.

[0263]

[12] Khediri A,Slimi H.Enhancing machine learning modelinterpretability in intrusion detection systems through shap explanations andllm-generated descriptions[C] / / 2024 6th International Conference on PatternAnalysis and Intelligent Systems(PAIS).IEEE,2024:1-6.

[0264]

[13] Lin J,Zhao J,Liu L,et al.RL-LLM-DT:An Automatic Decision TreeGeneration Method Based on RL Evaluation and LLM Enhancement[J].arXivpreprint arXiv:2412.11417,2024.

[0265]

[14] Ma H,Hu T,Pu Z,et al.Coevolving with the other you:Fine-tuningllm with sequential cooperative multi-agent reinforcement learning[J].Advances in Neural Information Processing Systems,2024,37:15497-15525.

[0266]

[15] Douze M,Guzhva A,Deng C.The faiss library[J].arXiv preprintarXiv:2401.08281,2024.

[0267]

[16] Wei Z,Xu J.Reinforcement learning to rank with Markov decisionprocess[C] / / Proceedings of the 40th international ACM SIGIR conference onresearch and development in information retrieval.2017:945-948.

[0268]

[17] Khadka S,Tumer K.Evolution-guided policy gradient inreinforcement learning[J].Advances in Neural Information Processing Systems,2018,31.

[0269]

[18] Hu E J,Shen Y,Wallis P,et al.Lora:Low-rank adaptation of largelanguage models[J].ICLR,2022,1(2):3.

[0270]

[19] Van Erven T,Harremos P.Rényi divergence and Kullback-Leiblerdivergence[J].IEEE Transactions on Information Theory,2014,60(7):3797-3820.

[0271]

[20] Pascanu R,Mikolov T,Bengio Y.On the difficulty of trainingrecurrent neural networks[C] / / Proceedings of the 30th InternationalConference on Machine Learning.2013:1310–1318.

[0272]

[21] Zhao B.Clinical data extraction and normalization of cyrillicelectronic health records via deep-learning natural language processing[J].JCO Clinical Cancer Informatics,2019,3:1-9.

[0273]

[22] Gong T,Lee T,Stephenson C,et al.A comparison of loss weightingstrategies for multi task learning in deep neural networks[J].IEEE Access,2019,7:141627-141632.

[0274]

[23] National Institute of Standards and Technology.NationalVulnerability Database(NVD)[EB / OL].https: / / nvd.nist.gov / ,2025-04-05.

[0275]

[24] MITRE Corporation.MITRE ATT&CK Framework[EB / OL].https: / / attack.mitre.org / ,2025-04-05.

[0276]

[25] Chase Kocher.LangChain:Building LLM applications withcomposability[EB / OL].https: / / docs.langchain.com / ,2023.

[0277]

[26] Lewis P,Perez E,Piktus A,et al.Retrieval-augmented generation forknowledge-intensive nlp tasks[J].Advances in neural information processingsystems,2020,33:9459-9474.

[0278]

[27] Sentence-transformers / all-mpnet-base-v2[EB / OL].https: / / huggingface.co / ,2025-04-05.

[0279]

[28] Yao S,Yu D,Zhao J,et al.Tree of thoughts:Deliberate problemsolving with large language models[J].Advances in neural informationprocessing systems,2023,36:11809-11822.

[0280]

[29] Yao Y,Liu Z,Cen Z,et al.Constraint-conditioned policyoptimization for versatile safe reinforcement learning[J].Advances in NeuralInformation Processing Systems,2023,36:12555-12568.

[0281]

[30] Liu S,Sabir B,Jang S I,et al.From solitary directives tointeractive encouragement!llm secure code generation by natural languageprompting[J].arXiv preprint arXiv:2410.14321,2024.

[0282]

[31] Qwen:A family of open-source large language models[EB / OL].https: / / huggingface.co / Qwen,2023.

[0283]

[32] Reiter E.A structured review of the validity of BLEU[J].Computational Linguistics,2018,44(3):393-401.

[0284]

[33] Lin C Y,Och F J.Looking for a few good metrics:ROUGE and itsevaluation[C] / / Ntcir workshop.2004:1-8.

Claims

1. A cybersecurity script arrangement method based on LLM-enhanced RL, characterized by the following steps: include: (1) Using the Large Language Model (LLM) to dynamically expand the set of candidate strategy atoms based on security threat scenarios; (2) Using reinforcement learning (RL) algorithms to optimize the script generation strategy based on the LLM atomic action preferences, we can achieve adaptive security orchestration of network security scripts in complex network environments. In step (1), the vector library is first used to perform semantic similarity retrieval on the input security threat intelligence and match the best defense strategy atoms in the knowledge base. Subsequently, the contextual understanding capability of the LLM is used to perform semantic understanding and reasoning on the candidate atom set to generate new strategy atoms that complement the defense. In step (2), we first use a policy-atom-based security scenario graph model to abstract multiple constraints on nodes and edges in the corresponding graph model based on the security orchestration definition. Then, we use a composite reward function to model the multi-constrained security scenario orchestration problem as a linear programming function to achieve multi-dimensional optimization of defense effectiveness and execution cost. In the network security script arrangement method, the security script consists of multiple defense actions, each action is an executable security measure; the security script is divided into a series of minimum executable units, which are called defense strategy atoms or strategy atoms, referred to as atoms.

2. The network security script arrangement method according to claim 1, wherein In step (1): First, the input security threat intelligence is embedded online using a semantic embedding model to generate a threat vector. Then, the threat vector is matched with the static defense policy atoms in the vector library. The matching is done by searching for semantic similarity and selecting the best policy atom. Finally, the matched policy atoms are arranged into the ToT prompt words of the thinking tree as context reference; LLM generates new policy atoms based on context semantic understanding and reasoning expansion.

3. The network security script arrangement method according to claim 1, wherein In step (1), the matching method is to use Top-K atom matching based on cosine similarity combined with index retrieval of the vector library.

4. The method for arranging network security scripts according to claim 1, wherein In step (1), 1.1) Each script in the security script knowledge base is split, and the resulting policy atom index number set is represented as Constructing a strategic atomic knowledge base; Policy Atoms The corresponding defense operation text description is recorded as d n , d n Describes the specific content of defensive actions; The network attack type corresponding to the policy atom n is marked as y n , used for index marking; and They are defined as the response time score, process occupancy score, manual participation score and defense effect score of the policy atom n respectively; Based on the correlation between the text description of the policy atom defense action and the text description of the threat situation, K policy atoms are screened from the policy atom knowledge base, and the target atom is obtained, which is the set of selected policy atom index numbers. Among them, the threat scenario of the current network environment The text description is recorded as It's about d n and The correlation function of 1.2) Matching Afterwards, LLM combines contextual learning and knowledge transfer capabilities to expand the atomic set; Threat Scenario The set of potential attack types is expressed as The set of attack types covered is The attack type coverage of the candidate atom is expressed as: Let Ω∈[0,1] be the preset attack type coverage threshold, if The candidate atom set is judged to have insufficient attack type coverage, and the attack type coverage gap is Defining binary variables is the optimization judgment index of the strategy atom n. If the strategy atom If the defense effect score of is lower than the mean defense effect score of the candidate atoms, the atom is marked as an object to be optimized: in, is the mean defense effect score of all candidate atoms; is used as a context reference to prompt LLM to generate the corresponding supplementary atom index set, which is expressed as: For any LLM-generated supplementary strategy atom h∈H, it must satisfy As the input of subsequent security orchestration, the final candidate atomic index set is expressed as 5. The method for arranging network security scripts according to claim 4, wherein In step (2), First, build the graph model: For a given Construct a graph model, represented as G = (V, E), to describe the connection relationship between all candidate security policy atoms; In G, Identified as node v in the graph n ∈V; v0 is defined as the security event alarm trigger atom, indicating the start of security orchestration; Node v m With v n The directed edge between them is denoted as e m,n ∈E, indicating that there is a directed connection between strategy atoms m and n; After the candidate atom space is expanded by LLM, there are multiple potential choices for atoms to defend against a certain attack type. Then the node v n There are potentially multiple forward and backward directed edges; In the graph model, a security orchestration policy corresponds to a path that starts from the starting node and forms a tree-structured path using the most appropriate node combination. All vertices and directed edges on this tree-structured path constitute a security scenario. A security orchestration scenario is modeled as a rooted tree, represented as Z = (V′, E′), where is the set of selected nodes, is the selected edge set, which satisfies the following constraints: The root node v0 must be selected: v0∈V′(7) If the node v n ∈V′, then except the root node v0, there is at least one node that enters v n The edges of are also selected: If the edge e m,n ∈E′, then node v m and v n Must also be selected: Then, based on the constraints, the security script orchestration problem is modeled: Binary decision variable y m,n Indicates whether the path of the security script passes through the directed edge e m,n , that is, whether to choose to continue executing policy atom n after executing policy atom m, satisfying: binary variable x n Indicates whether the policy atom n is selected for orchestration, satisfying: Each strategy selection step in the security script must refer to the real-time network situation, system configuration, load status and operation and maintenance records. These four network elements are represented as v n The function g1(v n )、g2(v n )、g3(v n ) and g4(v n ); Strategy Atom n and are expressed as n The function f1(v n )、f2(v n )、f3(v n ) and f4(v n ), the four scores are standardized by formula (13), and all scoring indicators are mapped to the same [0,1] interval, which makes them comparable in the final objective function; By analyzing the threat intelligence report, LLM evaluates the total threat score of the current network environment. The total threat score is expressed as Used to characterize the overall environmental safety index loss value; In a security scenario path, the sum of the defense effect scores of all selected strategy atoms satisfies Comparative goals: The goal of security orchestration is to minimize the cost of security operations and maintenance while resolving security attack issues, thereby maximizing relative operational benefits. Define weight coefficients λ1, λ2, and λ3 to balance the contribution of the response time score, process occupancy score, and manual participation score of the selected atoms in a security script to the operation and maintenance cost. The problem of minimizing the cost under the condition of successful operation and maintenance of the environmental network, that is, maximizing the relative operation and maintenance benefits, is modeled as a linear programming function: The three constraints of the linear programming problem are respectively directed to the conditions in equations (7) to (9); Formula (14) indicates from the data level that the root node v0 must be selected; if the node v n ∈V′, that is, x n =1, then as long as formula (15) is satisfied, all the entries into v in the safe scenario path n The sum of the decision variables of the edges is greater than or equal to 1, which proves that there is at least one entry into v except the root node v0. n The edges of are also selected; according to formula (16), for any e m,n ∈E′, that is, y m,n =1, the condition is established as y m,n ≤x m And y m,n ≤x n ; This means x m ,x n ≥1,x m and x n is a binary variable 0,1, then x m ,x n =1, that is, node v m and v n Definitely selected.

6. The method for arranging network security scripts according to claim 5, wherein In step (2), LLM provides the prior probability for strategy generation, and RL is responsible for strategy optimization. The specific method is: 2.1) Security Situation Analysis LLM performs security situation analysis and provides environment status information for subsequent RL decision-making; Threat intelligence in the network environment is extracted and used as input context, which is injected into the LLM in the form of prompt words to guide the LLM to identify security risks and potential threats in the current environment; LLM calculates the total threat score of the environment by analyzing the information in the threat intelligence. Reflects the degree to which the current network environment deviates from the ideal state, and the impact of potential threats on environmental security; During the RL training decision-making process, LLM predicts the network status in real time based on the selected scenario atom combination; the status data includes network status, system configuration, load status and operation and maintenance records. These status data are quantified by LLM to represent the g1(v n )、g2(v n )、g3(v n ) and g4(v n ) score; Based on the score of the status data, LLM evaluates the possible changes in the network environment after the script is executed and predicts the security situation of the next state; 2.2) State and action space enhancement The LLM is prompted to generate new policy atoms that match the scenario. At the same time, the LLM analyzes and predicts the network situation based on threat intelligence and the selected atom combination. Based on this, the problem of LLM-assisted RL script generation is modeled as a discrete-time Markov decision process MDP, represented as a four-tuple State space In a high-dimensional state space, the corresponding state s in round t t Including the currently selected script atom combination graph model and the network situation score, system configuration score, load status score and operation and maintenance record score corresponding to the last selected atom j; Action Space Including security policy atoms filtered from the knowledge base and new policy atoms generated by LLM; action a t is in state s t The security policy atom selected in the next step is scored based on the policy atom itself and its relative processing time, resource usage, manual involvement, and operational effectiveness. ●State transition probability P: represents the transition from the current state s t Transfer to next state t+1 The probability of selecting the strategy action a t Then select strategy action a t+1 probability; Cumulative reward R: is the collection of reward values, r t Represents a given state s t and action a t When, from state s t Transfer to next state t+1 The value of the reward received from the feedback; LLM will g1(v n )、g2(v n )、g3(v n ) and g4(v n ) and other network state elements into high-dimensional vector representations, enhancing the state space Helps RL agents better understand the task environment. Combined with the policy atoms matched by the knowledge base, LLM generates some new policy atoms that fit the network scenario and enhances the action space. This allows the agent to have more action options. Based on the assistance of these two dimensions of LLM, the action and state space of RL are enhanced; 2.3) LLM-assisted RL offline training LLM is used to provide RL algorithms with prior probabilities of action preferences to improve the efficiency and accuracy of RL training in generating safe scenarios. LLM based on state t , under the prompt word instruction, predict the next state s t+1 , and for optional Predict action preference probability distribution p LLM (a t |τ t ); where RL training trajectory τ t ={(s1,a1,r1),...,(s t ,a t ,r t )} consists of state, action and reward; Strategy π θ (a t |s t ) indicates that the RL algorithm is in state s t Next take action a t The probability of; at time t, the algorithm combines p LLM and strategy π θ (a t |s t ) Decision selection action a t , the immediate reward is expressed as: in, It is the normalized value of the operation and maintenance effect score of the selected action, i.e. the atom. Represents the normalized response time, process occupancy, and manual participation score corresponding to the atom; For each trajectory τ, its cumulative reward R(τ) is calculated in the form of discounted reward, which is used to update the parameters of the RL strategy; The cumulative reward return is expressed as: Among them, γ∈[0,1] is a discount factor used to control the importance of future rewards; During offline training, the goal of the RL algorithm is to optimize the policy π θ , maximize the expected cumulative reward, that is: To maximize J(θ), the RL policy parameters θ are updated using a gradient strategy. The policy gradient is calculated as follows: By estimating the gradient of the policy, the algorithm clearly determines the direction of the optimal policy atom in each state and optimizes the policy parameters accordingly; Finally, the policy parameters are iteratively updated using the gradient ascent method, and the policy parameters are updated to: Among them, η is the learning rate, which is used to control the amplitude of each parameter update; Through multiple training sessions, RL strategies tend to be optimal, generating adaptable and efficient safety scenarios. 2.4) Fine-tuning alignment of LLM to RL policy The LoRA fine-tuning and parameter merging method is used to improve the alignment of LLM with RL strategy. This method includes two stages: LoRA fine-tuning training and parameter fusion based on gradient clipping: Phase 1: LoRA fine-tuning training Freeze the pre-trained LLM model weights W and introduce low-rank matrices A and B for parameter update: W′=W+ΔW,ΔW=BA(24) in, are the original weights of the pre-trained model, is a trainable low-rank matrix, r< <d; At the end of the nth round of RL training, the generated security policy trajectory is expressed as Filter all time-step reward values ​​from the safe policy track to be higher than the reward threshold Ω + High reward trajectory and reward values ​​are both lower than the reward threshold Ω - Low reward trajectory Then, the final fine-tuning dataset is updated to by Fine-tuning the LLM for the training sample, the loss function is expressed as: Among them, π LLM (a|s) is the probability distribution of LLM choosing action a in state s, π RL (a|s) is the policy distribution of the RL agent in the same state; The KL divergence is calculated to measure the difference between the policy generated by the LLM and the RL agent policy, expressed as: When the policy generated by the LLM perfectly matches the RL policy, the KL divergence is 0. Otherwise, the KL is positively correlated with the policy deviation between the two, and the loss function will guide the LLM to update; In order to gradually align the action selection preferences of LLM with the decision requirements of RL, that is, to minimize The gradient of fine-tuning is calculated as: g ΔW Represents the parameter gradient update direction of low-rank matrices A and B; Stage 2: Parameter Fusion Based on Gradient Clipping At each step of LoRA training, the gradient g ΔW is clipped to implement the gradient norm limit, expressed as: Among them, ‖g ΔW ‖2 is the L2 norm of the gradient, and c is the preset gradient clipping threshold; According to the clipped gradient, the LoRA low-rank matrix is ​​updated as: The original LLM parameters θ LLM and the LLM parameters θ′ after LoRA fine-tuning LLM The fusion is performed by adopting a weighted fusion strategy, and the fused parameters are expressed as: θ″ LLM =push′ LLM +(1-ω)θ LLM (30) Among them, ω is the fusion parameter, which determines the weight distribution of the original LLM and the fine-tuned LLM model; An adaptive fusion strategy based on KL divergence is used to adjust ω: When D KL When ω is less than the threshold, the value of ω is close to 1, and more parameters of the fine-tuned LLM model are retained; When D KL When it is not less than the threshold, ω becomes smaller and more original LLM information is retained.

7. The network security script arrangement method according to claim 1 is characterized in that the network The security script orchestration method is built based on the LangChain framework and the Retrieval Enhanced Generation (RAG) technology, where the vector library is the FAISS vector database. The steps in the cybersecurity playbook orchestration methodology include: (1) Data processing stage Load data from the pre-processed structured security operation and maintenance policy atomic JSON file, use LangChain's JSON loader to batch load nested structured policy files, and convert them into Python dictionary format; A dictionary item splitter based on logical rules is used to recursively split each policy atom and generate independent item blocks according to scenario classification; For policy items containing multi-level classifications, the segmenter retains its complete classification path as metadata and extracts the "Description" field as the core semantic content; The description text of the policy atom is embedded into the sentence-transformers / all-mpnet-base-v2 pre-trained language model vector. All description texts are batch-converted into 768-dimensional normalized vectors and injected into the FAISS vector database along with metadata such as policy ID and priority. (2) Retrieval enhancement stage Cyber ​​attack threat intelligence is mapped into vectors through the embedding model, and then retrieved through fusion structured filtering. The most relevant policy atom set in the vector knowledge base is returned based on semantic similarity sorting. The retrieved policy atoms are combined with threat intelligence and embedded in the prompt words for constructing LLM. Step 1: LLM builds a corresponding security orchestration script template based on threat intelligence analysis application scenarios, clearly defining trigger conditions, execution actions, and verification steps; Step 2: Select appropriate content from the retrieved security policy atomic set to fill the script template. If the atomic attack coverage is insufficient or there are shortcomings in the defense effectiveness indicators, imitate the existing atomic style to supplement and generate corresponding atomic actions. Step 3: LLM scores the generated atomic block based on the scoring criteria of the existing atoms to ensure that it meets the existing standards; Step 4: LLM sets a preference value for each action in the script, ranging from 0 to 1; those exceeding 0.8 are marked as highly recommended atoms, and the complete script is finally output; (3) Script generation stage The script content generated by LLM is split into the RL action space, and the preference value of each action is used as a reference indicator for RL reward training. After multiple rounds of RL training, the RL converges to a stable strategy and ultimately generates a scenario-specific safety script.

Images