Safety reinforcement learning driven large language model safety decision-making agent

Abstract

The application discloses a safe decision-making agent of a large language model driven by safe reinforcement learning, and the decision-making agent comprises: a high-level semantic planner, which is used for receiving target and constraint instructions in a text form, receiving language or visual observation signals of an environment, and outputting safe risk information and a suggestion action plan in a text format; a low-level action executor, which is used for receiving low-dimensional observation and semantic encoding of the environment, wherein the semantic encoding is output by the high-level semantic planner after text embedding conversion; a strategy network of the low-level action executor, which outputs a final safe action; a training alignment module, which is used for optimizing the strategy network and a value network; reward and cost signals collected through environment interaction are fed back to the high-level semantic planner, and parameters of the strategy network and the value network are trained through a safe reinforcement learning algorithm. The application is convenient for realizing the decision-making of a given text target while ensuring that the decision-making does not violate a given text safety constraint.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and machine learning, and more particularly to a safe decision-making agent driven by safe reinforcement learning of a large language model. BACKGROUND

[0002] A decision-making agent is an important concept in the field of artificial intelligence (AI) and machine learning (ML), referring to a system or network model that can autonomously make decisions and perform actions to achieve specific goals or optimize certain performance indicators. The mainstream approach to achieving decision-making intelligence is reinforcement learning (RL) technology, which focuses on training agents to continuously learn optimal decisions through interaction and training with the environment. In this process, the agent adjusts its behavior based on feedback from the environment, usually with the goal of maximizing cumulative rewards.

[0003] On the other hand, large language models (LLM) have begun to show significant potential and effectiveness in the field of decision-making intelligence. Based on deep learning natural language processing (NLP) technology, these models are trained on vast amounts of text data to learn the complex patterns and structures of language. These models use billions or even tens of billions of parameters to capture the richness, diversity, and subtle differences of language, enabling them to perform various language-related tasks such as text generation, translation, summarization, and question answering. With the increase in computing power and algorithmic progress, research has shown that large language models have the ability to perform logical reasoning and zero-shot generalization required for downstream decision-making tasks, and therefore can manage other AI models and tools to handle complex multi-modal tasks, assist or participate in complex programs such as TextWorld, Handbi, and MineCraft, or be deployed on robots for interaction with the real world.

[0004] From a unified perspective, both reinforcement learning agents and large language models can be seen as an interactive sequential modeling problem: LLM generates text in a sequential decision-making framework, selecting the most likely next word or phrase based on known text information, and repeating until the end of the text; RL agent generates decisions in an interactive environment, selecting the most appropriate next action based on known environmental observations, and repeating until the goal is achieved. Based on this similarity, researchers have begun to combine reinforcement learning techniques with large language models to apply them to decision-making tasks, in order to build more powerful decision-making agents and accelerate the training process of agents.

[0005] This technical route usually replaces or assists one of the basic components of the reinforcement learning loop in different ways, including building LLM as a reward function, using LLM to decompose complex task objectives, and using LLM to process environment representations. Among them, the technical route of LLM as Policy has been widely studied in recent years.

[0006] In this setting, LLM is trained or adapted by reinforcement learning methods to generate actions or policy suggestions given a description of the environment state, which shows great potential in scenarios that require complex decision-making and understanding of natural language instructions. Specifically, LLM-based methods output policies in the form of text actions by inputting text descriptions of observations, and then fine-tune them through reinforcement learning algorithms. This route has been proven to be effective in low-level navigation tasks and high-level decision-making tasks described in text. In addition, the output of LLM can also be mapped to non-text actions through additional learnable layers. In summary, LLM as Policy is considered a potential approach to achieving general intelligence in the real world.

[0007] However, current research focusing on the LLM as Policy block rarely considers the safety of policy actions, which hinders the further development of this technology in the real world. Specifically, these deployment schemes based on given objectives and following the traditional reinforcement learning paradigm mainly focus on maximizing cumulative rewards, sometimes ignoring safety issues, leading to dangerous or undesirable behavior by agents exploring the environment. In the fields of autonomous driving, robot control, medical decision support, spacecraft control, etc., it is crucial to meet safety requirements while achieving objectives, as unsafe decisions can have serious consequences.

[0008] Therefore, it is essential to deploy large language models as safe agents in decision-making tasks (LLM as Safe Policy), and there has been little progress in this area both domestically and internationally. Although LLM itself can respond to language instructions, simply adding constraints directly to the model's prompts makes it difficult for LLM to achieve a good balance between adhering to constraints and maximizing cumulative rewards due to the complexity of constraint rules and the misalignment of decision-making environments. At the same time, due to the LLM's own knowledge illusion, overly complex objective and constraint coupling descriptions can directly cause LLM to output incorrect behavior. Therefore, online adjustment through policy interaction is necessary.

[0009] In view of this, the present application considers providing a safe reinforcement learning-driven large language model safe decision-making agent to address the above problems from the perspectives of safe prompt planning and safe reinforcement learning. SUMMARY

[0010] The present application aims at the above research status and existing problems, and provides a safe reinforcement learning driven large language model safe decision-making agent, which combines the ideas of safe reinforcement learning and model layering, and mainly includes three modules: a high-level semantic planner based on a large language model, a low-level action executor based on semantic coding, and a training alignment module driven by a safe reinforcement learning method; it is helpful to ensure that the decision-making does not violate the given text safety constraints while completing the decision-making of the given text target, and provides an extensible and interpretable technical foundation for intelligent decision-making scenarios with high safety requirements.

[0011] In order to achieve the above object, the present application adopts the following technical scheme:

[0012] In a first aspect, the present application provides a safe reinforcement learning driven large language model safe decision-making agent, which includes: a high-level semantic planner, a low-level action executor, and a training alignment module; wherein:

[0013] The high-level semantic planner is used for receiving target and constraint instructions in text form, and simultaneously receiving language or visual observation signals of the environment, and outputting safety risk information and recommended action planning in text format;

[0014] The low-level action executor is used for receiving low-dimensional observation and semantic coding of the environment, wherein the semantic coding is from the output of the high-level semantic planner after text embedding conversion; the policy network of the low-level action executor outputs the final safety action;

[0015] The training alignment module is used for optimizing the policy network and the value network; the reward and cost signals collected through environment interaction are fed back to prompt the high-level semantic planner, and the parameters of the policy network and the value network are trained through a safe reinforcement learning algorithm; the alignment process of the decision-making agent from the large language model to the safety action output of the environment is realized.

[0016] In a preferred embodiment, the high-level semantic planner is an LLM or a VLM, the text embedding conversion is applicable to any coding language model, the training alignment module adopts a safe reinforcement learning algorithm based on a value method, and the policy network and the value network are MLP or LSTM cascade network structures.

[0017] In a preferred embodiment, the high-level semantic planner realizes end-to-end output between texts through a pre-defined context planning dialogue framework, which is formalized as:

[0018]

[0019] wherein, represents the text output of the high-level semantic planner, including and , represents the text description of the judgment of whether there is a safety risk in the environment or whether a safety risk has been triggered after analyzing the high-dimensional observation at the current moment, and the corresponding position, represents the proposed next-step coarse-grained action plan according to the analysis result of the safety risk; represents a high-level semantic planner; represents a target instruction text description; represents a safety constraint instruction text description; represents the high-dimensional observation provided by the environment, which is a text description or a visual picture description, depending on the setting of the high-level semantic planner; represents a safety prompt word; represents a feedback prompt word.

[0020] In a preferred embodiment, the low-level action executor extracts knowledge from the LLM or VLM and converts it into executable actions; specifically: using the text conversion embedding framework SimCSE-BERT, respectively mapping and to fixed-length vectors, and concatenating them in the feature dimension, denoted as semantic encoding vector :

[0021]

[0022] The vector contains safety risk analysis knowledge for the environment extracted from the LLM / VLM, which is used as the output of the low-level action executor;

[0023] concatenating the current moment low-dimensional observation provided by the environment with the semantic encoding vector in the feature dimension, and feeding it to the policy network to generate actions:

[0024]

[0025] wherein, represents an action vector.

[0026] In a preferred embodiment, the policy network is constructed as a two-layer or more MLP neural network.

[0027] In a preferred embodiment, the training alignment module uses the reward signal and the cost signal provided by the environment to train and adjust the neural network part in the agent architecture, specifically:

[0028] First, construct a value network The network and the policy network Using the same input, Output respectively And Corresponding to the expected cumulative reward and expected cumulative cost respectively after time :

[0029]

[0030] Then, using the gradient ascent training method based on Lagrange improvement, the original optimization target is optimized:

[0031]

[0032] By Lagrange method, the safety constraint condition is converted into a non-constrained problem:

[0033]

[0034] Wherein, Indicates the neural network parameter participating in optimization; Indicates the cumulative reward; Indicates the cumulative cost; Indicates the budget threshold; Indicates the hyperparameter for balancing the loss function of the goal-oriented target and the loss function of punishing unsafe actions; Is the total loss function.

[0035] In a preferred embodiment, the safety constraint learning strength is dynamically adjusted by continuously adjusting The adjustment process is as follows:

[0036]

[0037] Wherein, Indicates the hyperparameter of the Adjustment, Indicates the hyperparameter of the last, i.e. the Adjustment, Indicates the coefficient for controlling the Updating amplitude.

[0038] Secondly, the embodiments of the present application also provide a safe reinforcement learning driven large language model safe decision-making agent training method, applied to the safe reinforcement learning driven large language model safe decision-making agent described above, the training method comprising:

[0039] Step one: select and load the high-level semantic planner to be used, select and load the text embedding pre-training model, randomly initialize the policy network and the value network;

[0040] Step two: given the target to be achieved and the constraint instructions:

[0041] Step three: in single-step environment interaction, according to the forward process, the low-level action executor collects low-dimensional observations, semantic encoding, actions and environment signals;

[0042] Step four: integrate the collected signals into an overall trajectory and use it to build a sample experience pool;

[0043] Step five: randomly select trajectories from the sample experience pool, and train the policy network and value network using safe reinforcement learning;

[0044] Step six: test on the test environment to determine whether the cumulative cost is within the constraint threshold, if yes, go to step seven, if no, adjust the safe reinforcement learning hyperparameters and return to step three;

[0045] Step seven: test on the test environment to determine whether the learning has converged, if yes, terminate training, if no, return to step three.

[0046] According to the technical solution described above, compared with the prior art, the present application has the following technical advantages:

[0047] The present application provides a safe reinforcement learning driven large language model safe decision-making agent, which solves the deficiencies of LLM decision-making strategy in safety, executability and environmental adaptability from a systematic level, provides an innovative solution for intelligent decision-making in safety-sensitive fields, and provides an extensible and interpretable technical foundation for intelligent decision-making scenarios with high safety requirements.

[0048] Other features and advantages of the present application will be described in the following description, and some will become apparent from the description, or will be understood by those skilled in the art through implementation of the present application. The purpose and other advantages of the present application can be achieved and obtained by the structure specifically pointed out in the written description and the accompanying drawings.

[0049] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments or prior art of the present application, the accompanying drawings needed in the embodiment or prior art description will be briefly introduced. Obviously, the accompanying drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other accompanying drawings according to these accompanying drawings without creative labor.

[0051] The accompanying drawings are included to provide a further understanding of the present application, and are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and are used to explain the present application, but do not limit the present application.

[0052] Figure 1 A schematic diagram of a safe reinforcement learning driven large language model safe decision-making agent framework is provided for the embodiments of the present application.

[0053] Figure 2 A schematic diagram of a safe reinforcement learning driven large language model safe decision-making agent training method is provided for the embodiments of the present application.

[0054] Figure 3 A schematic diagram of an electronic device structure is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, but not all embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0056] In the description of the present application, it should be noted that in some processes described in the specification and drawings, a plurality of operations appear in a specific order, but it should be clearly understood that these operations can be executed or performed in parallel or not in the order they appear in this text. In addition, various serial numbers and the like are only for description purposes and cannot be understood as indicating or implying relative importance.

[0057] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0058] The present application aims at the problem that the decision-making agent based on large language model (LLM) lacks security in real scenarios, leading to potential risks, and proposes a solution driven by safe reinforcement learning, aiming to build an agent system with both efficient decision-making ability and safety constraint compliance ability. Through safe prompt planning technology and safe reinforcement learning algorithm, online safety calibration of the strategy is realized, which can effectively solve the safety hazards caused by knowledge illusion, rule misalignment or environmental dynamics in the decision-making task of LLM, enabling the agent to efficiently complete the target (such as precise grasping of robots, path planning of autonomous driving) and actively avoid risks (such as avoiding collision of robotic arms, illegal lane changing of vehicles) in complex interactive scenarios.

[0059] Referring to Figure 1 The present application mainly includes a main framework to realize safe decision-making of agents based on large language models in the environment, which can be divided into three modules: a high-level semantic planner based on a large language model, a low-level action executor based on semantic encoding, and a training alignment module driven by safe reinforcement learning. Among them:

[0060] The high-level semantic planner based on a large language model is used to receive target and constraint instructions in text form, as well as language or visual observation signals of the environment. Through a safe and sensitive context planning dialogue framework, the high-level semantic planner can output safe and sensitive outputs in text format, including potential safety risk information and recommended action planning.

[0061] The low-level action executor based on semantic encoding accepts low-dimensional observations and semantic encodings of the environment, where the semantic encodings come from the output of the high-level semantic planner after text embedding conversion; the policy network of the low-level action executor outputs the final safe action.

[0062] The training alignment module driven by safe reinforcement learning is used to optimize the policy network and the value network, realizing the alignment process of the decision-making agent from the large language model to the safe action output in the environment. The reward and cost signals collected through environmental interaction are used to feedback to the high-level semantic planner on the one hand, and to train the parameters of the policy and value networks through the safe reinforcement learning algorithm on the other hand.

[0063] In the present application, the high-level semantic planner is applicable to any LLM or VLM, the text embedding conversion is applicable to any encoding language model, the safe reinforcement learning training method is applicable to any value-based safe reinforcement learning algorithm, and the policy network and the value network are applicable to any structure of MLP or LSTM cascade network structure.

[0064] The working principles and specific implementation modes of each module of the present application are described in detail as follows:

[0065] First, the parameter variables involved in the embodiment of the present application are introduced:

[0066] LLM: Large Language Model, accepts free-form text input, outputs free-form text answer;

[0067] VLM: Visual Language Model, accepts both picture and free-form text input, outputs free-form text answer;

[0068] : One-dimensional observation vector provided by the environment;

[0069] : High-dimensional observation provided by the environment, could be a constructed text description, or a visual picture description, depending on the setup of the high-level semantic planner;

[0070] : Goal instruction text description provided by the user for the agent;

[0071] : Safety constraint instruction text description provided by the user for the agent;

[0072] : High-level semantic planner, adapted to either a large language model or a visual language model;

[0073] : Safety cue word, text description;

[0074] : Feedback cue word, text description;

[0075] : Text output of the high-level semantic planner;

[0076] : Transformed semantic encoding vector;

[0077] : Low-level action executor, i.e., policy network;

[0078] : One-dimensional action vector;

[0079] : Value network;

[0080] : Goal reward value, scalar;

[0081] : Safety constraint value, scalar;

[0082] : Environment reward signal, scalar, represents the reward the agent can harvest from the environment for its current action, the goal is to maximize the cumulative reward ;

[0083] : environmental cost signal, scalar, represents the cost that the agent's current action will incur if it violates the safety constraint, the goal is to maximize the cumulative reward while keeping the cumulative cost not exceeding the budget .

[0084] In this embodiment, the safety reinforcement learning driven large language model safety decision-making agent framework mainly combines safety reinforcement learning and model hierarchical thinking to ensure that the decision does not violate the given text safety constraints while completing the decision of the given text target.

[0085] Figure 1 The main framework of the decision-making agent is described, which accepts the user-provided target description and safety constraint requirements when initialized, that is, requires the agent to not violate the safety constraints and achieve the final goal in the process of interacting with the environment. At the time of decision execution , the agent will accept low-dimensional observations and high-dimensional observations from the environment, where the high-dimensional observations can be text-form environment state descriptions or image observations from the agent's perspective. The output of the framework is the action at the time of decision execution , which will gradually align to the safety action through safety reinforcement learning training. The following is the specific execution process of the decision-making agent framework:

[0086] (1) Large language model-based high-level semantic planner: since the large-scale pre-trained LLM / VLM has the ability of general logical reasoning and world knowledge understanding, the invention plans to realize the end-to-end output of the high-level semantic planner between texts through a pre-defined dialogue framework, without fine-tuning the LLM / VLM, which is called a safety-sensitive context dialogue framework. The framework is formulated as follows:

[0087]

[0088] wherein, is the text output of the high-level semantic planner, which contains two parts: one is the text description of whether there is a safety risk in the environment or whether a safety risk has been triggered after analyzing the high-dimensional observation at the current time, and the corresponding position (if the risk is to touch a certain area or object), denoted as Taking VLM as an example, through a high-dimensional first-person perspective image, VLM receives the image, the prompt for the next paragraph, and user commands. It can directly output whether there are any security risk items or areas in the image that match the description in the user commands, and their approximate location information relative to the agent. Secondly, based on the analysis results of the security risks, the proposed next coarse-grained action plan (e.g., direction of movement) is denoted as... .

[0089] In this embodiment, the security-sensitive contextual dialogue framework includes two parts of prompts. Through multiple questions, the LLM / VLM is guided to identify potential security risks in the scenario, and suggestions are made for the next stage of action planning to avoid violating constraints and achieve the goal. Combined with the previous moment Environmental signal feedback and This indicates the quality of the previous action planning result in the LLM / VLM process, helping to generate the action planning for the current moment. .

[0090] (2) Low-level action executor based on semantic encoding: Since the output of LLM / VLM is in text form, while the actions that interact with the environment are usually numerical or vector, a low-level action executor was designed to extract knowledge from LLM / VLM and convert it into executable actions.

[0091] Specifically, this embodiment first employs the text transformation embedding framework SimCSE-BERT. This pre-trained model is built upon the language encoding model BERT and utilizes an unsupervised contrastive learning method to enable the BERT model to clearly distinguish the vector representations of words and sentences of different lengths and meanings. Through SimCSE-BERT, the text transformation embedding framework SimCSE-BERT is used to transform the text into a text encoding model. and Mapping to a fixed-length vector and concatenating them along the feature dimension, denoted as the semantic encoding vector. :

[0092]

[0093] This vector contains safety risk analysis knowledge about the environment extracted from LLM / VLM and can be used as the output of low-level action actuators. It incorporates low-dimensional observations of the environment at the current moment. With semantic encoding vector By concatenating the features along the feature dimension, they can be fed into the policy network. To generate actions :

[0094]

[0095] wherein, The MLP neural network can be configured to have two or more layers.

[0096] (3) Safety reinforcement learning driven training alignment module: although the LLM / VLM can examine the safety risk information in the environment to some extent, due to the insufficient alignment with the environment, on the one hand, the LLM / VLM can only produce a coarse-grained action plan and cannot directly implement the action, and needs the policy network to convert into an executable action On the other hand, the LLM / VLM may not be able to complete the absolute compliance of the safety constraint, and the policy network needs to be accepted to generate corresponding safety actions, therefore, the reward signal and the cost signal provided by the environment need to be used to train and adjust the neural network part in the framework to achieve the above two requirements.

[0097] Specifically, in the present embodiment, a value network is first constructed, which uses the same input as the policy network and outputs and respectively, corresponding to the expected cumulative reward and the expected cumulative cost after time :

[0098]

[0099] Subsequently, the gradient ascent training method based on Lagrange improvement is used in this module for optimization, and the original optimization objective is:

[0100]

[0101] Through the Lagrange method, the safety constraint condition is converted into a non-constrained problem:

[0102]

[0103] wherein, represents a hyperparameter for balancing the loss function of the goal and the loss function of punishing unsafe actions, when is larger, the current learning is more focused on punishing unsafe action behavior; represents the neural network parameters participating in optimization, which includes the parameters in and two neural networks; is the overall loss function. This shows that by maintaining , the policy network can be updated by using the above inner goal ​In this embodiment, With Based on the generalized advantage calculation method, it can be obtained by , and With Calculation. The value network Can be trained by traditional time difference method.

[0104] On the other hand, by continuously adjusting The dynamic adjustment of learning strength of safety constraints is realized, and the adjustment process is as follows:

[0105]

[0106] Among them, Indicates the hyperparameter of the Adjustment, the Indicates the last hyperparameter, that is, the Adjustment, and Indicates the coefficient of controlling Update amplitude.

[0107] The process shows that when the cumulative cost Caused by the violation of safety constraints in the training process of the agent Exceeds the threshold , we increase To increase the learning degree of reducing In the inner target.

[0108] The overall algorithm flow of training alignment is shown in Figure 2 It can be divided into the following steps:

[0109] Step 1: Select and load the LLM / VLM pre-training model to be used (i.e. high-level semantic planner), select and load the text embedding pre-training model, randomly initialize the strategy network, and the value network;

[0110] Step 2: Given the target and constraint instructions required to be achieved;

[0111] Step 3: In single-step environment interaction, according to the framework forward process, collect low-dimensional observation, semantic encoding, action and environment signal;

[0112] Step 4: Integrate the collected signals into the overall trajectory and use them to build a sample experience pool;

[0113] Step 5: Randomly select trajectories in the sample experience pool, and train the strategy network and value network using safety reinforcement learning;

[0114] Step six: judge whether the cumulative cost is within the constraint threshold by testing on the test environment, if yes, go to step seven, if no, adjust the safety reinforcement learning hyperparameters and return to step three;

[0115] Step seven: judge whether the learning converges by testing on the test environment, if yes, terminate the training, if no, return to step three.

[0116] The following two practical application examples demonstrate the specific application of the present application:

[0117] Application example 1:

[0118] Implementation of a safety reinforcement learning driven large language model safety decision-making agent in a simulation environment of safety-gymnasium, taking safety-vision-fading as an example:

[0119] Step one: install the simulation calling python package of safety-gymnasium, and initialize multiple safety-vision-fading environments in parallel, and configure the observation as the first-person image and low-dimensional input observation.

[0120] Step two: select and load the LLM / VLM pre-training model to be used, here CogVLM is used, select and load the text embedding pre-training model, here SimCSE-Bert is used, and randomly initialize the policy network and value network.

[0121] Step three: set the environment goal to reach the green light column area (except for the purple area which is a dangerous area, the rest are safe areas, and the green area is the target position of navigation), and the constraint is that the number of times of touching the purple dangerous area in the process is less than three.

[0122] Step four: after initializing multiple safety-vision-fading environments, obtain the first-person image and low-dimensional input observation at the current time, return the first-person image, the target and the text description of the constraint to the high-level semantic planner, and return the planning output and low-dimensional input observation to the low-level action controller, and return the action output by the controller to the environment, and the environment updates the state and returns the reward and cost signal. Repeat the above operation until the simulation is completed, and collect the simulation trajectory as a training sample.

[0123] Step five: build an experience replay pool with a fixed size, and select an experience pool size of 5000. At the beginning, collect 5000 corresponding trajectory samples in the form of random actions and store them in the experience replay pool. Then, sample the experience replay pool for network training. For new trajectory samples, remove the first trajectory sample in the experience replay pool, and place the new sample at the end of the queue.

[0124] Step six: Extract 32 samples from the experience replay pool as training samples each time, and train the policy network and value network using the safety reinforcement learning method.

[0125] Step seven: After training for a certain number of steps, initialize a new safety-vision-fading environment and test it. If the test violates the safety constraint threshold, adjust the hyperparameters in the safety reinforcement learning algorithm.

[0126] Step eight: After training for a certain number of steps, initialize a new safety-vision-fading environment and test it. If the total reward curve tends to converge, terminate the training. If not, return to step four.

[0127] Application example 2:

[0128] Implementation of safety reinforcement learning-driven large language model safety decision-making agents in autonomous driving tasks, taking the city road scene of CARLA simulation platform as an example:

[0129] Step one: Use CARLA version 0.9.14 to build a city road scene, including dynamic elements such as intersections, traffic lights, pedestrians, and vehicles. Configure the agent vehicle sensors: high-dimensional observation: front-view RGB camera (resolution 640×480) captures road images; low-dimensional observation: vehicle state information (speed, acceleration, steering angle), radar point cloud data (direction and distance of the closest obstacle).

[0130] Step two: Select and load the LLM / VLM pre-trained model to be used, here we use CogVLM, select and load the text embedding pre-trained model, here we use SimCSE-Bert, and randomly initialize the policy network and value network. The policy network is a 3-layer MLP (input: low-dimensional observation + semantic encoding, dimension from 256→128→3; output includes: throttle, brake, steering angle instruction).

[0131] Step three: Set the target instruction: safely reach the target location 3 kilometers away. Set the safety constraints: no red light running, no collision, and keep the distance ≥2 meters. Configure the safety-sensitive context dialogue framework:

[0132] The specific content of is:

[0133] "You are an autonomous driving planning expert. Please answer the following questions based on the current camera image: 1. Are there any red lights or pedestrians in front? If so, where are they located?

[0134] 2. Is there any obstacle or lane change risk in the current lane?

[0135] 3. Next action suggestion (go straight, turn left, turn right, slow down, stop) based on the above risks.

[0136] where The specific content is:

[0137] "Last action execution, the vehicle was punished due to [running a red light / collision]. Please adjust the planning to avoid violating the rules again."

[0138] Step four: Initialize and load multiple CARLA instances at the same time, randomly generate different weather (sunny, rainy) and traffic flow density (low, medium, high) in each instance. Start step-by-step decision-making, and the single-step decision-making process includes:

[0139] (1) High-level semantic planner input: high-dimensional observation: current frame RGB image; text instructions: target and safety constraint description.

[0140] (2) Planning output example:

[0141] "Red light detected 50 meters ahead, risk of vehicle cutting into left lane."

[0142] "Suggest reducing speed to 20 km / h and preparing to stop."

[0143] (3) Semantic encoding: encode the above text into vector through SimCSE-BERT.

[0144] (4) Low-level action generation: input splicing: low-dimensional observation (speed, radar data) + → strategy network.

[0145] (5) Output action: throttle = 0.2, brake = 0.5, steering angle = -0.1 (slight left turn to avoid).

[0146] (6) Signal feedback: environment returns reward (positively correlated with target distance reduction) and cost (red light = +1, collision = +2, vehicle distance < 2 meters = +0.5).

[0147] Step five: Build an experience replay pool of fixed size, choose experience pool size 5000. At the beginning, collect 5000 corresponding trajectory samples in the form of random actions into the experience replay pool. Then, sample the experience replay pool for network training. For new trajectory samples, prefer to keep high-cost samples (such as collision events). Remove the first trajectory sample in the experience replay pool, and place the new sample at the end of the queue.

[0148] Step six: use the safe reinforcement learning method to train the policy network and value network by extracting 32 samples as training samples from the experience replay pool each time.

[0149] Step seven: after training for a certain number of steps, initialize a new CARLA environment and test it. If the test violates the safety constraint threshold, adjust the hyperparameters in the safe reinforcement learning algorithm.

[0150] Step eight: after training for a certain number of steps, initialize a new CARLA environment and test it. If the total reward curve converges, terminate the training. If not, return to step four.

[0151] From the description of the above embodiments, those skilled in the art can know that the current decision-making agent based on large language model (LLM) has significant safety risks in real scene deployment, the root cause of which lies in the fact that the traditional reinforcement learning paradigm is oriented towards single reward maximization, lacking active perception and avoidance ability for dynamic safety constraints, and the LLM itself is difficult to directly generate safe and reliable underlying actions due to knowledge illusion, rule misalignment and other problems. In view of this technical bottleneck, the present application proposes a safe reinforcement learning driven large language model safety decision-making agent (an agent framework integrating safety reinforcement learning and semantic planning), which mainly includes three innovative designs. The first is a high-level semantic planner based on a large language model, which combines a safety-sensitive context dialogue framework to realize high-level safety action planning. The second is a low-level action executor based on semantic encoding, which combines a semantic conversion embedding to realize the migration from high-level actions to executable actions. The third is a safe reinforcement learning driven training alignment method, which uses actual feedback signals to fine-tune the policy, making the generated low-level actions more secure. This technical solution addresses the shortcomings of LLM decision-making strategies in safety, executability and environmental adaptability from a systematic level, providing an innovative solution for intelligent decision-making in safety-sensitive fields and an extensible and interpretable technical foundation for intelligent decision-making scenarios with high safety requirements.

[0152] Further, referring to Figure 3 The electronic device can include a processor 10, a memory 11, a communication bus 12, and a communication interface 13, and can further include a computer program stored in the memory 11 and executable on the processor 10, and the processor executes the computer program to implement the above-mentioned agent and execute the above-mentioned training method.

[0153] The processor 10 can be composed of integrated circuits in some embodiments, for example, can be composed of a single packaged integrated circuit, or can be composed of multiple packaged integrated circuits with same or different functions, including one or more combinations of central processing units, microprocessors, digital processing chips, graphics processors, and various control chips, etc. The processor 10 is the control core of the electronic device, connects various components of the electronic device through various interfaces and lines, and executes programs or modules stored in the memory 11 and calls data stored in the memory 11 to perform various functions and process data of the electronic device.

[0154] Those skilled in the art should understand that the embodiments of the present application can be provided as a system, a method, an electronic device or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media having computer-usable program code contained therein.

[0155] It should be noted that the word "comprising" does not exclude the presence of other elements or steps than those listed in a claim. The word "a" or "an" preceding the disclosure of a plurality of elements does not exclude the presence of multiple such elements. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer.

[0156] Various embodiments are described in the specification with progressive development of the application, each embodiment highlighting a different aspect of the application. The same or similar parts between various embodiments are referenced to each other.

[0157] The above description of disclosed embodiments enables a person skilled in the art to implement or use the application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A security reinforcement learning-driven large language model security decision-making intelligent agent system, characterized in that, The decision-making intelligent agent system includes: a high-level semantic planner, a low-level action executor, and a training alignment module; wherein: The high-level semantic planner receives textual objectives and constraints, as well as linguistic or visual observation signals from the environment, and outputs text-formatted safety risk information and suggested action plans. The low-level action executor receives low-dimensional observations and semantic encodings of the environment, where the semantic encodings are derived from the output of the high-level semantic planner after text embedding transformation; the policy network of the low-level action executor outputs the final safety action. The training alignment module is used to optimize the policy network and value network; it uses reward and cost signals collected through environmental interactions to provide feedback prompts to the high-level semantic planner, and trains the parameters of the policy network and value network through a safety reinforcement learning algorithm; it realizes the alignment process of the decision-making agent from the large language model to the environmental safety action output. The high-level semantic planner achieves end-to-end output between texts through a predefined contextual planning dialogue framework, which can be formulated as follows: ； in, The text output of the high-level semantic planner contains and , This text description, along with its corresponding location, indicates the determination of whether a security risk exists or has been triggered in the environment based on the analysis of high-dimensional observations at the current moment. This indicates the proposed next coarse-grained action plan based on the analysis of security risks. This represents a high-level semantic planner; This indicates the target instruction text description; This represents a textual description of the safety constraint instructions; The high-dimensional observations provided by the environment can be constructed as textual or visual image descriptions, depending on the settings of the high-level semantic planner. Indicates safety warning words; Indicates feedback prompts; The low-level action executor extracts knowledge from the high-level semantic planner and converts it into executable actions; specifically, it uses the text transformation and embedding framework SimCSE-BERT to transform the text into executable actions. and Mapping to a fixed-length vector and concatenating them along the feature dimension, denoted as the semantic encoding vector. : ； This vector contains safety risk analysis knowledge about the environment extracted from the high-level semantic planner, which is used as the output of the low-level action executor. The low-dimensional observations provided by the environment at the current moment With semantic encoding vector The features are concatenated along the feature dimension and then fed into the policy network. To generate actions: ； in, Represents the action vector; The training alignment module uses the reward and cost signals provided by the environment to train and adjust the neural network part of the agent architecture, specifically as follows: First, construct a value network. This network and the policy network Using the same input, Output respectively and , respectively corresponding to time Expected cumulative reward and expected cumulative cost after the start: ； Then, optimization is performed using a gradient ascent training method based on the Lagrange multiplication principle. The original optimization objective is: ； By using the Lagrange method, the safety constraints are transformed into an unconstrained problem: ； in, This represents the parameters of the neural network involved in the optimization; Indicates cumulative rewards; Indicates the cumulative cost; Indicates the budget threshold; The hyperparameters represent the loss function used to balance the target-oriented loss function and the loss function used to punish unsafe actions; This is the overall loss function; Through continuous adjustment The learning intensity of safety constraints is dynamically adjusted, and the adjustment process is as follows: ； in, Indicates the first Hyperparameters adjusted in this step Indicates the previous time, i.e., the number Hyperparameters of the order, Indicates control The coefficient for updating the amplitude.

2. The security reinforcement learning-driven large language model security decision-making intelligent agent system according to claim 1, characterized in that, The high-level semantic planner is an LLM or VLM, the text embedding transformation is applicable to any encoding language model, the training alignment module adopts a value-based secure reinforcement learning algorithm, and the policy network and value network are MLP or LSTM cascaded network structures.

3. The security reinforcement learning-driven large language model security decision-making intelligent agent system according to claim 1, characterized in that, The policy network It is constructed as an MLP neural network with two or more layers.

4. A training method for a security decision-making intelligent agent system based on a large language model driven by security reinforcement learning, characterized in that, The training method, applied to the security reinforcement learning-driven large language model security decision-making intelligent agent system according to any one of claims 1-3, comprises: Step 1: Select and load the high-level semantic planner to be used, select and load the text embedding pre-trained model, and randomly initialize the policy network and value network; Step Two: Specify the required goals and constraints: Step 3: In single-step environmental interaction, based on the forward process, the low-level action actuator collects low-dimensional observations, semantic encodings, and action and environment signals; Step 4: Integrate the collected signals into a complete trajectory and use it to construct a sample experience pool; Step 5: Randomly select trajectories from the sample experience pool and train the policy network and value network using secure reinforcement learning; Step Six: Test on the test environment to determine if the cumulative cost is within the constraint threshold. If yes, proceed to Step Seven; otherwise, adjust the hyperparameters of the security reinforcement learning and return to Step Three. Step 7: Determine whether the learning has converged by testing in the test environment. If it has, terminate the training; otherwise, return to Step 3.

Images