Large language model security decision agent driven by security reinforcement learning

Abstract

The invention discloses a security reinforcement learning-driven large language model security decision agent, and the decision agent comprises a high-level semantic planner which is used for receiving a target and constraint instruction in a text form, receiving a language or visual observation signal of an environment at the same time, and outputting text formatted security risk information and suggested action planning; the low-layer action actuator is used for receiving low-dimensional observation and semantic codes of the environment, and the semantic codes are output by the high-layer semantic planner after text embedding conversion; the strategy network of the low-layer action actuator outputs a final safety action; the training alignment module is used for optimizing the strategy network and the value network; a high-level semantic planner is fed back and prompted through reward and cost signals collected through environment interaction, and parameters of a strategy network and a value network are trained through a security reinforcement learning algorithm. According to the method, the decision cannot violate the given text security constraint while the decision of the given text target is completed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence and machine learning technologies, and more particularly to a large language model security decision-making intelligent agent driven by security reinforcement learning. Background Art

[0002] Decision-making agents are a key concept in the fields of artificial intelligence (AI) and machine learning (ML). They refer to systems or network models that can autonomously make decisions and execute actions to achieve specific goals or optimize certain performance metrics. The mainstream approach to implementing decision-making intelligence is reinforcement learning (RL). RL focuses on training agents to continuously learn optimal decisions through interaction and training with their environment. In this process, the agent adjusts its behavior based on feedback from the environment, typically with the goal of maximizing cumulative rewards.

[0003] Large language models (LLMs), on the other hand, are beginning to demonstrate significant potential and success in the field of decision-making intelligence. Based on deep learning-based natural language processing (NLP) technology, they learn the complex patterns and structures of language by training on massive amounts of text data. These models utilize billions or even trillions of parameters to capture the richness, diversity, and nuance of language, enabling them to perform a variety of language-related tasks such as text generation, translation, summarization, and question-answering. With increasing computing power and advancements in algorithms, research has shown that large language models possess the logical reasoning and zero-shot generalization capabilities required for downstream decision-making tasks. Therefore, they can manage other AI models and tools to tackle complex multimodal tasks, assist or participate in complex programs such as TextWorld, Handbi, and MineCraft, or be deployed on robots to interact with the real world.

[0004] From a unified perspective, reinforcement learning agents and large language models can both be viewed as interactive sequential modeling problems: LLMs generate text within a sequential decision framework, selecting the most likely next word or phrase based on known textual information, and looping until the end of the text; RL agents make decisions within an interactive environment, selecting the most appropriate next action based on known environmental observations, and looping until the goal is achieved. Based on this similarity, researchers have begun combining reinforcement learning techniques with large language models for decision-making tasks, aiming to build more powerful decision-making agents and accelerate their training.

[0005] This technical approach typically replaces or supplements one of the fundamental components of reinforcement learning loops with a large language model in various ways. This includes constructing the LLM as a reward function, using the LLM to decompose complex task objectives, and using the LLM to process environment representations. Among these, the LLM as a decision-making agent or policy itself (LLM as Policy) has been widely researched recently.

[0006] In this setting, LLM is trained or adapted through reinforcement learning methods to generate action or policy recommendations given a description of the environment state. This approach shows great potential in scenarios that require complex decision-making and understanding of natural language instructions. Specifically, LLM-based methods input observations described in text, output actions in text form to form a policy, and then fine-tune the training through reinforcement learning algorithms. This route has been proven to be effective in low-level navigation tasks and high-level decision-making tasks described in text. In addition, the output of LLM can also be mapped to non-text actions by constructing additional learnable layers. In summary, LLM as Policy technology is considered a potential path to achieving general intelligence in the real world.

[0007] However, current research focused on LLM as Policy rarely considers the safety of policy actions, hindering the further development of this technology in the real world. Specifically, these deployment schemes based on a given goal and following the traditional reinforcement learning paradigm primarily focus on maximizing cumulative rewards, sometimes overlooking safety issues, leading the agent to adopt dangerous or undesirable behaviors when exploring the environment. In fields such as autonomous driving, robotic control, medical decision support, and spacecraft control, achieving goals while maintaining safety is crucial, as unsafe decisions can lead to serious consequences.

[0008] Therefore, deploying large language models as safe agents (LLMs as safe policies) in decision-making tasks is essential, yet research in this area has made little progress, both domestically and internationally. While LLMs can respond to language instructions, simply adding constraints directly to the model's prompts makes it difficult to achieve a good trade-off between adhering to constraints and maximizing cumulative rewards due to the complexity of the constraints and the misalignment of the decision-making environment. Furthermore, because LLMs inherently harbor a certain degree of knowledge illusion, overly complex coupled descriptions of objectives and constraints can lead to incorrect outputs. Therefore, online adjustments through policy interaction are essential.

[0009] In view of this, the present invention considers providing a large language model security decision-making intelligent agent driven by security reinforcement learning, and is prepared to solve the above problems from the two perspectives of security prompt planning and security reinforcement learning. Summary of the Invention

[0010] In response to the above research status and existing problems, the present invention provides a large language model security decision-making intelligent agent driven by security reinforcement learning. It combines the ideas of security reinforcement learning and model layering, and mainly includes three modules: a high-level semantic planner based on a large language model, a low-level action executor based on semantic encoding, and a training alignment module driven by security reinforcement learning methods. It helps to achieve the decision-making of a given text target while ensuring that the decision does not violate the given text security constraints, providing a scalable and explainable technical foundation for intelligent decision-making scenarios with high security requirements.

[0011] In order to achieve the above object, the present invention adopts the following technical solutions:

[0012] In a first aspect, an embodiment of the present invention provides a large language model security decision-making agent driven by security reinforcement learning, which includes: a high-level semantic planner, a low-level action executor, and a training alignment module; wherein:

[0013] A high-level semantic planner that receives textual goals and constraints, along with linguistic or visual observations of the environment, and outputs textually formatted safety risk information and recommended action plans.

[0014] The low-level action executor receives low-dimensional observations of the environment and semantic encodings, where the semantic encodings come from the output of the high-level semantic planner after text embedding conversion. The policy network of the low-level action executor outputs the final safe action.

[0015] The training alignment module is used to optimize the policy network and the value network. The reward and cost signals collected through environmental interactions are used as feedback to prompt the high-level semantic planner, and the parameters of the policy network and the value network are trained through a secure reinforcement learning algorithm. This realizes the alignment process of the decision-making agent from the large language model to the environmental safety action output.

[0016] In a preferred embodiment, the high-level semantic planner is an LLM or VLM, the text embedding conversion is applicable to any encoding language model, the training alignment module adopts a secure reinforcement learning algorithm based on a value method, and the policy network and the value network are MLP or LSTM cascade network structures.

[0017] In a preferred embodiment, the high-level semantic planner is implemented through a predefined context planning dialogue framework to achieve end-to-end output between texts, which is formulated as follows:

[0018]

[0019] in, Represents the text output of the high-level semantic planner, containing and , It represents the text description of whether there is a security risk in the environment or whether the security risk has been triggered after analyzing the high-dimensional observation at the current moment, as well as the corresponding location. It represents the next coarse-grained action plan proposed based on the analysis results of security risks; Represents a high-level semantic planner; Indicates the target instruction text description; Represents the text description of the security constraint instruction; Represents the high-dimensional observations provided by the environment, which are constructed text descriptions or visual picture descriptions, depending on the settings of the high-level semantic planner; Indicates safety reminder words; Indicates feedback prompt words.

[0020] In a preferred embodiment, the low-level action executor extracts knowledge from the LLM or VLM and converts it into executable actions; specifically, the text conversion embedding framework SimCSE-BERT is used to convert and Map to a vector of fixed length and concatenate it in the feature dimension, recorded as a semantic encoding vector :

[0021]

[0022] This vector contains the security risk analysis knowledge of the environment extracted from the LLM / VLM and is used as the output of the low-level action executor;

[0023] The current low-dimensional observation provided by the environment and semantic encoding vector Concatenate on the feature dimension and feed it to the policy network To generate the action:

[0024]

[0025] in, represents the action vector.

[0026] In a preferred embodiment, the policy network Constructed as an MLP neural network with more than two layers.

[0027] In a preferred embodiment, the training alignment module uses the reward signal and cost signal provided by the environment to train and adjust the neural network part of the agent architecture, specifically:

[0028] First, construct the value network , the network and the policy network Using the same input, Output separately and , corresponding to the time Expected cumulative reward and expected cumulative cost after starting:

[0029]

[0030] Then, the Lagrangian-based improved gradient ascent training method is used for optimization, and the original optimization objective is:

[0031]

[0032] The safety constraints are converted into an unconstrained problem using the Lagrangian method:

[0033]

[0034] in, Represents the neural network parameters involved in optimization; Indicates cumulative rewards; represents the cumulative cost; represents the budget threshold; represents the hyperparameters used to balance the loss function for guiding the goal with the loss function for penalizing unsafe actions; is the overall loss function.

[0035] In a preferred embodiment, by continuously adjusting The dynamic adjustment of safety constraint learning intensity is as follows:

[0036]

[0037] in, Indicates the The hyperparameters of the sub-tuning, Indicates the last time The hyperparameters of times, Display Control Update amplitude coefficient.

[0038] In a second aspect, an embodiment of the present invention further provides a method for training a large language model security decision-making agent driven by security reinforcement learning, which is applied to the above-mentioned large language model security decision-making agent driven by security reinforcement learning. The training method includes:

[0039] Step 1: Select and load the high-level semantic planner to be used, select and load the text embedding pre-trained model, and randomly initialize the policy network and value network;

[0040] Step 2: Give the required goals and constraints:

[0041] Step 3: In the single-step environment interaction, according to the forward process, the low-level action executor collects low-dimensional observations, semantic encodings, actions and environment signals;

[0042] Step 4: Integrate the collected signals into an overall trajectory and use it to construct a sample experience pool;

[0043] Step 5: Randomly select trajectories from the sample experience pool and use secure reinforcement learning to train the policy network and value network;

[0044] Step 6: Determine whether the cumulative cost is within the constraint threshold by testing in the test environment. If so, proceed to Step 7. If not, adjust the secure reinforcement learning hyperparameters and return to Step 3.

[0045] Step 7: Determine whether the learning has converged by testing on the test environment. If so, terminate the training; if not, return to step 3.

[0046] It can be seen from the above technical solutions that compared with the prior art, the present invention has the following technical advantages:

[0047] The present invention provides a large language model security decision-making agent driven by secure reinforcement learning. This technical solution systematically addresses the shortcomings of LLM decision-making strategies in terms of security, executability, and environmental adaptability, providing an innovative solution for intelligent decision-making in security-sensitive areas and a scalable and explainable technical foundation for intelligent decision-making scenarios with high security requirements.

[0048] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0049] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0051] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0052] Figure 1 Schematic diagram of the large language model security decision-making agent framework driven by secure reinforcement learning provided in an embodiment of the present invention.

[0053] Figure 2 Schematic diagram of the large language model security decision-making agent training method driven by security reinforcement learning provided in an embodiment of the present invention.

[0054] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0056] In describing the present invention, it should be noted that some processes described in this specification and accompanying drawings include multiple operations that appear in a specific order. However, it should be understood that these operations may be performed in a different order than the order in which they appear, or may be performed in parallel. Furthermore, the use of various sequence numbers is for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0057] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0058] This paper addresses the potential risks posed by Large Language Model (LLM)-based decision-making agents in real-world scenarios due to safety deficiencies. It proposes a safety reinforcement learning-driven solution, aiming to build an agent system that combines efficient decision-making capabilities with safety constraint compliance. By leveraging safety prompt planning techniques and a safety reinforcement learning algorithm, this approach enables online safety calibration of policies. This effectively addresses safety hazards inherent in LLM decision-making tasks caused by knowledge illusions, misaligned rules, and environmental dynamics. This enables agents to efficiently achieve goals (such as precise robotic grasping and autonomous driving path planning) while proactively avoiding risks (such as avoiding robotic arm collisions and illegal lane changes).

[0059] See also Figure 1 As shown, the present invention mainly includes a main framework to achieve safe decision-making of intelligent agents in the environment based on a large language model. The framework can be divided into three modules: a high-level semantic planner based on the large language model, a low-level action executor based on semantic encoding, and a training alignment module driven by secure reinforcement learning. Among them:

[0060] A high-level semantic planner based on a large language model is used to receive textual goals and constraint instructions, as well as language or visual observation signals from the environment. Through a security-sensitive contextual planning dialogue framework, the high-level semantic planner can output text-formatted security-sensitive outputs, including potential security risk information and recommended action plans.

[0061] The low-level action executor based on semantic encoding accepts low-dimensional observations and semantic encoding of the environment, where the semantic encoding comes from the output of the high-level semantic planner after text embedding conversion; the policy network of the low-level action executor outputs the final safe action.

[0062] The training alignment module, driven by secure reinforcement learning, optimizes the policy and value networks, aligning the decision-making agent's output from the large language model to safe environmental actions. Reward and cost signals collected through environmental interactions are used to provide feedback to the high-level semantic planner and train the parameters of the policy and value networks using secure reinforcement learning algorithms.

[0063] In the present invention, the high-level semantic planner is applicable to any LLM or VLM, the text embedding conversion is applicable to any encoding language model, the secure reinforcement learning training method is applicable to any value-based secure reinforcement learning algorithm, and the policy network and value network are applicable to any MLP or LSTM cascade network structure.

[0064] The working principle and specific implementation of each module of the present invention are described in detail below:

[0065] First, the parameter variables involved in the embodiment of the present invention are introduced:

[0066] LLM: Large Language Model, which accepts free-form text input and outputs free-form text answers;

[0067] VLM: Visual Language Model, which accepts both image and free-form text input and outputs free-form text answers;

[0068] : One-dimensional observation vector provided by the environment;

[0069] : The high-dimensional observations provided by the environment can be constructed as textual descriptions or visual picture descriptions, depending on the settings of the high-level semantic planner;

[0070] : A text description of the target instruction provided by the user to the agent;

[0071] : A text description of the safety constraint instructions provided by the user to the agent;

[0072] : High-level semantic planner, suitable for large language models or visual language models;

[0073] : Safety prompt words, text description;

[0074] : Feedback prompt words, text description;

[0075] : text output of the high-level semantic planner;

[0076] : The converted semantic encoding vector;

[0077] : Low-level action executor, i.e., policy network;

[0078] : one-dimensional motion vector;

[0079] : Value Network;

[0080] : target reward value, scalar;

[0081] : safety constraint value, scalar;

[0082] : Environmental reward signal, scalar, represents the reward that the agent can harvest from its current action on the environment. The goal is to maximize the cumulative reward ;

[0083] : Environmental cost signal, scalar, represents the cost of the agent's current action on the environment if it violates the safety constraints. The goal is to maximize the cumulative reward while making the cumulative cost Stay within budget .

[0084] In this embodiment, the large language model security decision-making intelligent agent framework driven by security reinforcement learning mainly combines security reinforcement learning and model layering ideas to achieve the goal of completing the decision of a given text while ensuring that the decision does not violate the given text security constraints.

[0085] Figure 1 Describes the main framework of the decision-making agent, which accepts the goal description provided by the user during initialization and safety constraints , which requires the agent to not violate safety constraints and achieve the ultimate goal in the process of interacting with the environment. , the agent will receive low-dimensional observations from the environment and high-dimensional observations , where the high-dimensional observation can be a textual description of the environment state, or an image observation from a certain perspective of the agent. The output of the framework is the decision execution time Action , this action will gradually align to a safe action through secure reinforcement learning training. The following is the specific execution process of the decision-making agent framework:

[0086] (1) High-Level Semantic Planner Based on Large Language Model: Since large-scale pre-trained LLM / VLM have universal logical reasoning and world knowledge understanding capabilities, this paper proposes to achieve end-to-end output of high-level semantic planner across texts through a pre-defined dialogue framework without fine-tuning the LLM / VLM. This is called the security-sensitive contextual dialogue framework. The framework is formulated as follows:

[0087]

[0088] in, It is the text output of the high-level semantic planner, which consists of two parts: one is the high-dimensional observation of the current moment After analyzing the security risk in the environment, determine whether there is a security risk, or whether the security risk has been triggered, and the corresponding location (if the risk is contact with a certain area or object), recorded as Taking VLM as an example, through the high-dimensional observation dimension first-person perspective image, VLM receives the image, the prompt word of the next paragraph description, and the user instruction, and can directly output whether there are security risk items or areas in the image that meet the description of the user instruction, and their approximate location information relative to the intelligent agent; secondly, based on the analysis results of the security risk, the proposed next coarse-grained action plan (such as the forward direction, etc.) is recorded as .

[0089] In this embodiment, the security-sensitive context dialogue frame includes two parts of prompt words: Through multiple questions, the LLM / VLM is guided to identify potential safety risks in the scenario and to plan the next steps to avoid constraint violations and achieve the goal. Combined with the previous moment Environmental signal feedback and , prompts LLM / VLM the result of the action plan execution of the previous step, to help generate the action plan at this moment .

[0090] (2) Low-level action executor based on semantic encoding: Since the output of LLM / VLM is in text form, and the actions of interacting with the environment are usually numerical values ​​or vectors, a low-level action executor is designed to extract knowledge from LLM / VLM and convert it into executable actions.

[0091] Specifically, this embodiment first adopts the text conversion embedding framework SimCSE-BERT. This pre-training model is based on the language encoding model Bert. An unsupervised contrastive learning method is used to enable the Bert model to have obvious vector representation distinctions for words and sentences of different lengths and meanings. Through SimCSE-BERT, and Map to a vector of fixed length and concatenate it in the feature dimension, recorded as a semantic encoding vector :

[0092]

[0093] This vector contains the security risk analysis knowledge of the environment extracted from LLM / VLM and can be used as the output of the low-level action executor. and semantic encoding vector Splicing on the feature dimension can be fed into the policy network To generate actions :

[0094]

[0095] in, It can be constructed as an MLP neural network with more than two layers.

[0096] (3) Training alignment module driven by security reinforcement learning: Although LLM / VLM can examine the security risk information in the environment to a certain extent, due to its insufficient alignment with the environment, on the one hand, LLM / VLM can only generate coarse-grained action planning and cannot directly implement the execution action, which requires the policy network to convert it into executable action. On the other hand, LLM / VLM may not be able to achieve absolute compliance with security constraints, requiring the policy network to accept Able to generate corresponding safe actions, therefore, it is necessary to utilize the reward signal provided by the environment and cost signal The neural network part of the framework is trained and adjusted to achieve the above two requirements.

[0097] Specifically, in this embodiment, the value network is first constructed , the network and the policy network Using the same input, Output separately and , corresponding to the time Expected cumulative reward and expected cumulative cost after starting:

[0098]

[0099] Subsequently, this module uses the Lagrangian-based improved gradient ascent training method for optimization, and the original optimization objective is:

[0100]

[0101] The safety constraints are converted into an unconstrained problem using the Lagrangian method:

[0102]

[0103] in, represents the hyperparameter used to balance the loss function for guiding the goal and the loss function for penalizing unsafe actions, when The larger it is, the more emphasis this learning puts on punishing unsafe actions; Represents the neural network parameters involved in optimization, including and Parameters in the two neural networks; is the overall loss function. This shows that by maintaining , you can use the inner target To update the policy network In this embodiment, and Based on the generalized advantage calculation method, it can be obtained by 、 as well as and Calculated. Value Network It can be trained using traditional temporal difference methods.

[0104] On the other hand, by constantly adjusting Implement dynamic adjustment of the safety constraint learning intensity. The adjustment process is as follows:

[0105]

[0106] in, Indicates the The hyperparameters of the sub-tuning, Indicates the last time The hyperparameters of times, Display Control Update amplitude coefficient.

[0107] This process shows that the cumulative cost of violating safety constraints during the agent training process is Exceeding the threshold When we increase , to the inner target Increase versus decrease level of learning.

[0108] The overall algorithm flow of training alignment is as follows Figure 2 As shown, it can be broken down into the following steps:

[0109] Step 1: Select and load the LLM / VLM pre-trained model to be used (i.e., high-level semantic planner), select and load the text embedding pre-trained model, and randomly initialize the policy network and value network.

[0110] Step 2: Give the required goals and constraints;

[0111] Step 3: In a single-step environment interaction, low-dimensional observations, semantic encodings, actions, and environment signals are collected according to the framework forward process;

[0112] Step 4: Integrate the collected signals into an overall trajectory and use it to construct a sample experience pool;

[0113] Step 5: Randomly select trajectories from the sample experience pool and use secure reinforcement learning to train the policy network and value network;

[0114] Step 6: Determine whether the cumulative cost is within the constraint threshold by testing in the test environment. If so, proceed to Step 7. If not, adjust the secure reinforcement learning hyperparameters and return to Step 3.

[0115] Step 7: Determine whether the learning has converged by testing on the test environment. If so, terminate the training; if not, return to step 3.

[0116] The following two practical application examples illustrate the specific application of the present invention:

[0117] Application Example 1:

[0118] The implementation of a large language model safety decision-making agent driven by safety reinforcement learning in a simulation environment on the safety-gymnasium, using safety-vision-fading as an example:

[0119] Step 1: Install the safety-gymnasium simulation call Python package, initialize multiple safety-vision-fading environments in parallel, and configure the observations to be first-person images and low-dimensional input observations.

[0120] Step 2: Select and load the LLM / VLM pre-trained model to be used. Here, CogVLM is used. Select and load the text embedding pre-trained model. Here, SimCSE-Bert is used. Randomly initialize the policy network and value network.

[0121] Step 3: Set the environmental goal to reach the green light beam area (except for the purple area, which is the dangerous area, the rest are safe areas, and the green area is the navigation target location), and constrain the number of times the purple dangerous area is touched less than three times during the process.

[0122] Step 4: After initializing multiple safety-vision-fading environments, obtain the current first-person image and low-dimensional input observations. Return the first-person image and textual descriptions of the goal and constraints to the high-level semantic planner. The plan output and low-dimensional input observations are returned to the low-level action controller. The controller's output action is returned to the environment, which updates its state and returns reward and cost signals. Repeat this process until the simulation ends, and collect the simulation trajectories as training samples.

[0123] Step 5: Build a fixed-size experience replay pool, choosing a pool size of 5000. Initially, collect 5000 corresponding trajectory samples from random actions and store them in the pool. Then, sample the pool for network training. For new trajectory samples, discard the first one to enter the pool, and place the new one at the end of the queue.

[0124] Step 6: Each time, 32 samples are extracted from the experience replay pool as training samples, and the policy network and value network are trained using the secure reinforcement learning method.

[0125] Step 7: After a certain number of training steps, initialize a new safety-vision-fading environment and test it. If the safety constraint threshold is violated after testing, correct the hyperparameters in the safe reinforcement learning algorithm.

[0126] Step 8: After a certain number of training steps, initialize a new safety-vision-fading environment and test it. If the total reward curve converges, terminate the training. If not, return to step 4.

[0127] Application Example 2:

[0128] The implementation of a large language model safety decision-making agent driven by safety reinforcement learning in autonomous driving tasks is illustrated by taking the urban road scenario of the CARLA simulation platform as an example:

[0129] Step 1: Use CARLA version 0.9.14 to build a city road scene, including dynamic elements such as intersections, traffic lights, pedestrians, and vehicles. Configure the vehicle's sensors: High-dimensional observations: A forward-looking RGB camera (resolution 640×480) captures road images; low-dimensional observations: Vehicle state information (speed, acceleration, steering angle), and radar point cloud data (the direction and distance to the nearest obstacle).

[0130] Step 2: Select and load the pre-trained LLM / VLM model to be used. Here, CogVLM is used. Select and load the pre-trained text embedding model. Here, SimCSE-Bert is used. Randomly initialize the policy network and value network. The policy network is a three-layer MLP (input: low-dimensional observations + semantic encoding, dimensionality from 256 → 128 → 3; output: throttle, brake, and steering angle commands).

[0131] Step 3: Set the target instruction: Safely reach the target location 3 kilometers away. Set safety constraints: No red light running, no collision, and maintain a distance of ≥ 2 meters. Configure the safety-sensitive contextual dialogue framework:

[0132] in The specific content is:

[0133] "You are an autonomous driving planning expert. Please answer the following questions based on the current camera image: 1. Is there a red light or pedestrian ahead? If so, where are they?

[0134] 2. Are there any obstacles or lane change risks in the current lane?

[0135] 3. Based on the above risks, recommend the next action (go straight, turn left, turn right, slow down, stop)."

[0136] in The specific content is:

[0137] "After the last action was performed, the vehicle was penalized for [running a red light / collision]. Please adjust your plan to avoid further violations."

[0138] Step 4: Initialize and load multiple CARLA instances simultaneously, randomly generate different weather conditions (sunny, rainy) and traffic density (low, medium, high) in each instance, and start making single-step decisions. The single-step decision process includes:

[0139] (1) High-level semantic planner input: high-dimensional observation: current frame RGB image; text instructions: description of goals and safety constraints.

[0140] (2) Planning output example:

[0141] = "A red light is detected 50 meters ahead, and there is a risk of a vehicle cutting into the left lane.";

[0142] ="It is recommended to slow down to 20km / h and prepare to stop.".

[0143] (3) Semantic encoding: The above text is encoded by SimCSE-BERT into vector.

[0144] (4) Low-level action generation: Input splicing: low-dimensional observations (speed, radar data) + →Policy network.

[0145] (5) Output action: Throttle = 0.2, Brake = 0.5, Steering angle = -0.1 (slight left turn to avoid).

[0146] (6) Signal feedback: the environment returns rewards (Positively correlated with shorter target distance) and cost (Running a red light = +1, collision = +2, distance between vehicles <2 meters = +0.5).

[0147] Step 5: Construct a fixed-size experience replay pool, selecting a pool size of 5000. Initially, collect 5000 corresponding trajectory samples using random actions and store them in the pool. Then, sample the pool for network training. For new trajectory samples, prioritize high-cost samples (such as collision events). The first trajectory sample to enter the pool is discarded, and new samples are placed at the end of the queue.

[0148] Step 6: Each time, 32 samples are extracted from the experience replay pool as training samples, and the policy network and value network are trained using the secure reinforcement learning method.

[0149] Step 7: After each training step, initialize a new CARLA environment and test it. If the safety constraint threshold is violated after testing, correct the hyperparameters in the safe reinforcement learning algorithm.

[0150] Step 8: After a certain number of training steps, initialize a new CARLA environment and test it. If the total reward curve tends to converge, terminate the training. If not, return to step 4.

[0151] From the description of the above embodiments, those skilled in the art will appreciate that current large language model (LLM)-based decision-making agents present significant security risks in real-world deployments. This stems from the traditional reinforcement learning paradigm's single-reward maximization approach, which lacks the ability to proactively perceive and avoid dynamic safety constraints. Furthermore, LLMs inherently struggle to directly generate safe and reliable low-level actions due to issues such as knowledge illusions and misaligned rules. To address this technical bottleneck, this paper proposes a secure reinforcement learning-driven large language model-based secure decision-making agent (an agent framework that hierarchically integrates secure reinforcement learning and semantic planning). This agent incorporates three key innovations: a high-level semantic planner based on the LLM, which integrates a security-sensitive contextual dialogue framework to achieve high-level secure action planning; a low-level action executor based on semantic encoding, which incorporates a semantic transformation embedding to enable the migration from high-level actions to executable actions; and a secure reinforcement learning-driven training alignment method, which utilizes actual feedback signals to align fine-tuning strategies, resulting in safer generated low-level actions. This technical solution systematically addresses the shortcomings of LLM decision-making strategies in terms of security, executability, and environmental adaptability, providing innovative solutions for intelligent decision-making in security-sensitive areas and a scalable and explainable technical foundation for intelligent decision-making scenarios with high security requirements.

[0152] Further, refer to Figure 3 As shown, an embodiment of the present invention also provides an electronic device, which may include a processor 10, a memory 11, a communication bus 12 and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, and the processor executes the computer program to execute the above-mentioned intelligent agent and the above-mentioned training method.

[0153] In some embodiments, the processor 10 may be comprised of an integrated circuit, such as a single packaged integrated circuit or a combination of multiple packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core of the electronic device, connecting the various components of the entire electronic device using various interfaces and circuits. It executes programs or modules stored in the memory 11 and accesses data stored in the memory 11 to perform various functions of the electronic device and process data.

[0154] Those skilled in the art will appreciate that embodiments of the present invention may be provided as systems, methods, electronic devices, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0155] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in a claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer.

[0156] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0157] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large language model security decision-making agent driven by security reinforcement learning, characterized by: The decision-making agent includes: a high-level semantic planner, a low-level action executor, and a training alignment module; among which: A high-level semantic planner that receives textual goals and constraints, along with linguistic or visual observations of the environment, and outputs textually formatted safety risk information and recommended action plans. The low-level action executor receives low-dimensional observations of the environment and semantic encodings, where the semantic encodings come from the output of the high-level semantic planner after text embedding conversion. The policy network of the low-level action executor outputs the final safe action. The training alignment module is used to optimize the policy network and the value network. The reward and cost signals collected through environmental interactions are used as feedback to prompt the high-level semantic planner, and the parameters of the policy network and the value network are trained through a secure reinforcement learning algorithm. This realizes the alignment process of the decision-making agent from the large language model to the environmental safety action output.

2. The large language model security decision-making agent driven by security reinforcement learning according to claim 1 is characterized in that: The high-level semantic planner is LLM or VLM, the text embedding conversion is applicable to any encoding language model, the training alignment module adopts a secure reinforcement learning algorithm based on the value method, and the policy network and value network are MLP or LSTM cascade network structures.

3. The large language model security decision-making agent driven by security reinforcement learning according to claim 2 is characterized in that: The high-level semantic planner is implemented through a predefined context planning dialogue framework to achieve end-to-end output between texts, which is formulated as follows: ； in, Represents the text output of the high-level semantic planner, containing and , It represents the text description of whether there is a security risk in the environment or whether the security risk has been triggered after analyzing the high-dimensional observation at the current moment, as well as the corresponding location. It represents the next coarse-grained action plan proposed based on the analysis results of security risks; Represents a high-level semantic planner; Indicates the target instruction text description; Represents the text description of the security constraint instruction; Represents the high-dimensional observations provided by the environment, which are constructed text descriptions or visual picture descriptions, depending on the settings of the high-level semantic planner; Indicates safety reminder words; Indicates feedback prompt words.

4. The large language model security decision-making agent driven by security reinforcement learning according to claim 3 is characterized in that: The low-level action executor extracts knowledge from the LLM or VLM and converts it into executable actions; specifically, it adopts the text conversion embedding framework SimCSE-BERT, and uses SimCSE-BERT to convert and Map to a vector of fixed length and concatenate it in the feature dimension, recorded as a semantic encoding vector : ； This vector contains the security risk analysis knowledge of the environment extracted from the LLM / VLM and is used as the output of the low-level action executor; The current low-dimensional observation provided by the environment and semantic encoding vector Concatenate on the feature dimension and feed it to the policy network To generate the action: ； in, represents the action vector.

5. The large language model security decision-making agent driven by security reinforcement learning according to claim 4 is characterized in that: The policy network Constructed as an MLP neural network with more than two layers.

6. The large language model security decision-making agent driven by security reinforcement learning according to claim 4 is characterized in that: The training alignment module uses the reward and cost signals provided by the environment to train and adjust the neural network part of the agent architecture. Specifically: First, construct the value network , the network and the policy network Using the same input, Output separately and , corresponding to the time Expected cumulative reward and expected cumulative cost after starting: ； Then, the Lagrangian-based improved gradient ascent training method is used for optimization, and the original optimization objective is: ； The safety constraints are converted into an unconstrained problem using the Lagrangian method: ； in, Represents the neural network parameters involved in optimization; Indicates cumulative rewards; represents the cumulative cost; represents the budget threshold; represents the hyperparameters used to balance the loss function for guiding the goal with the loss function for penalizing unsafe actions; is the overall loss function.

7. The large language model security decision-making agent driven by security reinforcement learning according to claim 6 is characterized in that: By constantly adjusting The dynamic adjustment of safety constraint learning intensity is as follows: ； in, Indicates the The hyperparameters of the sub-tuning, Indicates the last time The hyperparameters of times, Display Control Update amplitude coefficient.

8. A large language model security decision-making intelligent agent training method driven by security reinforcement learning, characterized by: The large language model security decision-making agent driven by security reinforcement learning according to any one of claims 1 to 7 is trained using the following method: Step 1: Select and load the high-level semantic planner to be used, select and load the text embedding pre-trained model, and randomly initialize the policy network and value network; Step 2: Give the required goals and constraints: Step 3: In the single-step environment interaction, according to the forward process, the low-level action executor collects low-dimensional observations, semantic encodings, actions and environment signals; Step 4: Integrate the collected signals into an overall trajectory and use it to construct a sample experience pool; Step 5: Randomly select trajectories from the sample experience pool and use secure reinforcement learning to train the policy network and value network; Step 6: Determine whether the cumulative cost is within the constraint threshold by testing in the test environment. If so, proceed to Step 7. If not, adjust the secure reinforcement learning hyperparameters and return to Step 3. Step 7: Determine whether the learning has converged by testing on the test environment. If so, terminate the training; if not, return to step 3.

Images