Training method, device, computer equipment and storage medium for strategy model

Abstract

The present application provides a training method, apparatus, computer device, and storage medium for a policy model. The method comprises: obtaining a first environmental state and inputting the first environmental state into a policy model to be trained to obtain a first action corresponding to the first environmental state; processing the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action; the safety indication signal is used to indicate whether it is safe to perform the first action under the first environmental state; and interacting with the environment based on the first action to obtain a second environmental state and a reward; constructing interaction data based on the first environmental state, the first action, the safety indication signal, the second environmental state, and the reward, and training the policy model to be trained based on the interaction data to obtain a target policy model.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and more specifically, to a training method, apparatus, computer equipment, and storage medium for a strategy model. Background Art

[0002] Reinforcement learning (RL) is a machine learning method that learns optimal policies through the interaction of an intelligent agent with its environment. In recent years, RL has made significant progress in fields such as game playing, robotic control, and autonomous driving. However, traditional RL faces numerous challenges in practical applications, particularly during training. RL agents may take dangerous actions during exploration, violating the safety constraints of the environment. This not only increases training costs but can also cause irreversible damage to the system in which the agent is deployed.

[0003] Safe Reinforcement Learning (Safe RL) introduces safety constraints during the training process to ensure that the agent does not violate the safety rules of the environment when exploring and optimizing strategies. Summary of the Invention

[0004] In view of this, the present application provides a training method, apparatus, computer device and storage medium for a policy model.

[0005] Specifically, this application is implemented through the following technical solutions:

[0006] In a first aspect, an embodiment of the present disclosure provides a method for training a policy model, the method comprising:

[0007] Acquire a first environmental state, and input the first environmental state into a policy model to be trained to obtain a first action corresponding to the first environmental state;

[0008] Processing the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action; the safety indication signal is used to indicate whether it is safe to perform the first action under the first environmental state; and

[0009] Interacting with the environment based on the first action to obtain a second environment state and a reward;

[0010] Interaction data is formed based on the first environmental state, the first action, the safety indication signal, the second environmental state, and the reward, and the policy model to be trained is trained based on the interaction data to obtain a target policy model.

[0011] Optionally, the processing the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action includes:

[0012] converting the first environmental state and the first action into a natural language description;

[0013] The first environment state and the first action described in natural language are input into the large language model to obtain the safety indication signal.

[0014] Optionally, the training the to-be-trained policy model based on the interaction data includes:

[0015] Using the interaction data generated in the current training cycle as target interaction data, and training the strategy model to be trained in the current training cycle based on the target interaction data; and / or,

[0016] Store the interaction data determined in the current training cycle into the experience replay pool;

[0017] The interaction data stored in the experience replay pool is sampled to obtain target interaction data corresponding to the current training cycle; and the strategy model to be trained is trained for the current training cycle based on the target interaction data.

[0018] Optionally, the training of the policy model to be trained in a current training cycle based on the target interaction data includes:

[0019] Determine the policy model, action value network, and target network to be trained in the current training cycle; wherein the policy model, action value network, and target network to be trained in the current training cycle are, respectively, an initialized policy model, an initialized action value network, and an initialized target network; and the network architecture and network internal parameters of the initialized action value network and the initialized target network are the same; or, they are the policy model to be trained, the action value network, and the target network updated in the previous training cycle, which were updated in the previous training cycle, and the target network updated in the previous training cycle;

[0020] Based on the target interaction data, the action value network of the current training cycle is updated by minimizing the Bellman residual, the policy model of the current training cycle is updated by maximizing the objective function, and the target network of the current training cycle is updated through a slow update mechanism.

[0021] Optionally, the action value network includes: a first action value network and a second action value network; the target network includes: a first target network and a second target network; the first action value network and the second action value network have the same network architecture but different network internal parameters; the network architecture and network internal parameters of the initialized first action value network and the initialized first target network are the same; the network internal parameters of the initialized second action value network and the initialized second target network are the same.

[0022] Optionally, based on the target interaction data, updating the action value network of the current training cycle by minimizing the Bellman residual, updating the policy model of the current training cycle by maximizing the objective function, and updating the target network of the current training cycle by a slow update mechanism, including:

[0023] Based on the preset weights, the network parameters of the target network in the current training cycle and the network parameters of the action value network in the current training cycle are weighted and summed to obtain the target network parameters of the target network in the current training cycle; based on the target network parameters, the target network of the current training cycle is updated;

[0024] Using the action-value network of the current training cycle, the first environment state and the first action in the target interaction data are processed to obtain a first cumulative reward expectation; and using the target network of the current training cycle or the updated target network, the second environment state in the target interaction data and the second action corresponding to the second environment state are processed to obtain a second cumulative reward expectation; the first cumulative reward expectation, the second cumulative reward expectation, the safety indication information in the target interaction data, and the reward are substituted into a pre-constructed Bellman residual function, and the action-value network of the current training cycle is updated with the goal of minimizing the Bellman residual;

[0025] Substitute the first cumulative reward expectation into the pre-built objective function, and update the policy model of the current training cycle with the goal of maximizing the objective function.

[0026] Optionally, the interaction data further includes a completion indication signal; the completion indication signal is used to indicate the execution status of the task corresponding to the policy network after the first action is performed in the first environment state;

[0027] When the task execution status indicates that no other actions can be performed, updating the action value network of the current training cycle by minimizing the Bellman residual includes:

[0028] The action-value network of the current training cycle is used to process the first environmental state and the first action in the target interaction data to obtain a first cumulative reward expectation. The first cumulative reward expectation and the safety indication information in the target interaction data are substituted into a pre-constructed Bellman residual function, and the action-value network of the current training cycle is updated with the goal of minimizing the Bellman residual.

[0029] Optionally, the method further includes:

[0030] Determining a security entropy weight parameter for the current training cycle; the security entropy weight parameter is used to adjust the weight of the security information in the target interaction information when updating the action value network; the security entropy weight parameter for the current training cycle is a preset parameter, or is determined based on the security information in the target interaction data determined in the previous training cycle;

[0031] Update the action-value network of the current training cycle by minimizing the Bellman residual, including:

[0032] Based on the security entropy weight parameter, the action value network of the current training cycle is updated by minimizing the Bellman residual.

[0033] In a second aspect, an embodiment of the present disclosure further provides a training device for a policy model, the device comprising:

[0034] an acquisition module, configured to acquire a first environmental state, and input the first environmental state into a policy model to be trained to obtain a first action corresponding to the first environmental state;

[0035] a processing module, configured to process the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action; the safety indication signal is used to indicate whether it is safe to perform the first action under the first environmental state; and

[0036] Interacting with the environment based on the first action to obtain a second environment state and a reward;

[0037] A training module is used to form interaction data based on the first environmental state, the first action, the safety indication signal, the second environmental state, and the reward, and to train the policy model to be trained based on the interaction data to obtain a target policy model.

[0038] Optionally, the processing module, when processing the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action, is configured to:

[0039] converting the first environmental state and the first action into a natural language description;

[0040] The first environment state and the first action described in natural language are input into the large language model to obtain the safety indication signal.

[0041] Optionally, the training module, when training the policy model to be trained based on the interaction data, is configured to:

[0042] Using the interaction data generated in the current training cycle as target interaction data, and training the strategy model to be trained in the current training cycle based on the target interaction data; and / or,

[0043] Store the interaction data determined in the current training cycle into the experience replay pool;

[0044] The interaction data stored in the experience replay pool is sampled to obtain target interaction data corresponding to the current training cycle; and the strategy model to be trained is trained for the current training cycle based on the target interaction data.

[0045] Optionally, the training module, when training the policy model to be trained in a current training cycle based on the target interaction data, is configured to:

[0046] Determine the policy model, action value network, and target network to be trained in the current training cycle; wherein the policy model, action value network, and target network to be trained in the current training cycle are, respectively, an initialized policy model, an initialized action value network, and an initialized target network; and the network architecture and network internal parameters of the initialized action value network and the initialized target network are the same; or, they are the policy model to be trained, the action value network, and the target network updated in the previous training cycle, which were updated in the previous training cycle, and the target network updated in the previous training cycle;

[0047] Based on the target interaction data, the action value network of the current training cycle is updated by minimizing the Bellman residual, the policy model of the current training cycle is updated by maximizing the objective function, and the target network of the current training cycle is updated through a slow update mechanism.

[0048] Optionally, the action value network includes: a first action value network and a second action value network; the target network includes: a first target network and a second target network; the first action value network and the second action value network have the same network architecture but different network internal parameters; the network architecture and network internal parameters of the initialized first action value network and the initialized first target network are the same; the network internal parameters of the initialized second action value network and the initialized second target network are the same.

[0049] Optionally, the training module, when updating the action-value network of the current training cycle by minimizing the Bellman residual based on the target interaction data, updating the policy model of the current training cycle by maximizing the objective function, and updating the target network of the current training cycle by a slow update mechanism, is configured to:

[0050] Based on the preset weights, the network parameters of the target network in the current training cycle and the network parameters of the action value network in the current training cycle are weighted and summed to obtain the target network parameters of the target network in the current training cycle; based on the target network parameters, the target network of the current training cycle is updated;

[0051] Using the action-value network of the current training cycle, the first environment state and the first action in the target interaction data are processed to obtain a first cumulative reward expectation; and using the target network of the current training cycle or the updated target network, the second environment state in the target interaction data and the second action corresponding to the second environment state are processed to obtain a second cumulative reward expectation; the first cumulative reward expectation, the second cumulative reward expectation, the safety indication information in the target interaction data, and the reward are substituted into a pre-constructed Bellman residual function, and the action-value network of the current training cycle is updated with the goal of minimizing the Bellman residual;

[0052] Substitute the first cumulative reward expectation into the pre-built objective function, and update the policy model of the current training cycle with the goal of maximizing the objective function.

[0053] Optionally, the interaction data further includes a completion indication signal; the completion indication signal is used to indicate the execution status of the task corresponding to the policy network after the first action is performed in the first environment state;

[0054] When the task execution status indicates that no other actions can be performed, the training module 53, when updating the action value network of the current training cycle by minimizing the Bellman residual, is configured to:

[0055] The action-value network of the current training cycle is used to process the first environmental state and the first action in the target interaction data to obtain a first cumulative reward expectation. The first cumulative reward expectation and the safety indication information in the target interaction data are substituted into a pre-constructed Bellman residual function, and the action-value network of the current training cycle is updated with the goal of minimizing the Bellman residual.

[0056] Optionally, the determining module is further configured to:

[0057] Determining a security entropy weight parameter for the current training cycle; the security entropy weight parameter is used to adjust the weight of the security information in the target interaction information when updating the action value network; the security entropy weight parameter for the current training cycle is a preset parameter, or is determined based on the security information in the target interaction data determined in the previous training cycle;

[0058] The training module, when updating the action-value network of the current training cycle by minimizing the Bellman residual, is used to:

[0059] Based on the security entropy weight parameter, the action value network of the current training cycle is updated by minimizing the Bellman residual.

[0060] In a third aspect, an optional implementation of the present disclosure further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the first aspect or any possible implementation of the first aspect are implemented.

[0061] In a fourth aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0062] In a fifth aspect, an optional implementation of the present disclosure further provides a computer program product, which carries a program code, and the instructions included in the program code can be used to execute the steps of the model training method as described in the first aspect or any one of the first aspects.

[0063] The training method for a policy model provided in an embodiment of the present disclosure obtains a first environmental state, inputs the first environmental state into a policy model to be trained, obtains an action corresponding to the first environmental state, then uses a pre-trained large language model to process the first environmental state and action to obtain a safety indication signal corresponding to the action, and interacts with the environment based on the action to obtain a second environmental state, a reward, and a completion signal. Subsequently, the first environmental state, action, safety indication signal, second environmental state, reward, and completion signal are used as interaction data, and the policy model to be trained is trained based on the interaction data. The large language model then performs a safety assessment on the environmental state-action pair, generating a dense safety indication signal as a safety constraint during the training process. The large language model, leveraging its rich prior knowledge, can quickly adapt to different task scenarios and provide accurate safety feedback, thereby ensuring that sufficient safety information is obtained during the training process.

[0064] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present disclosure.

[0065] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A flowchart of a method for training a policy model provided by some embodiments of the present disclosure is shown;

[0067] Figure 2 A flowchart illustrating a specific method for training a policy model to be trained in a current training cycle provided by some embodiments of the present disclosure is shown;

[0068] Figure 3 A flowchart showing a specific example of a training method for a policy model provided by some embodiments of the present disclosure is shown;

[0069] Figure 4 A schematic diagram showing the structure of a computer device provided by some embodiments of the present disclosure is shown;

[0070] Figure 5 A schematic diagram of the structure of a training device for a policy model provided in some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0071] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0072] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0073] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0074] In related technologies, secure reinforcement learning has the following problems:

[0075] 1. In many real-world scenarios, the environment's safety constraint signals are sparse, meaning that corresponding feedback is triggered only when the agent takes dangerous actions. This sparsity of safety constraint signals makes it difficult for the agent to obtain sufficient safety information early in training, increasing the number of safety constraint violations.

[0076] 2: Safety constraints in safety reinforcement learning rely on manually designed constraint functions, which are difficult to adapt to dynamically changing environments and task requirements and have limited generalization capabilities.

[0077] 3. Current safety reinforcement learning algorithms also have sparse rewards, meaning that, typically, the agent must interact with the environment multiple times before receiving a reward. In environments with sparse rewards, safety reinforcement learning algorithms may overly focus on safety constraints, resulting in overly conservative strategies and inability to effectively complete the task. This problem is particularly prominent in complex task scenarios.

[0078] 4. Traditional safety reinforcement learning algorithms typically use on-policy training, which has low data utilization efficiency and leads to excessively long training times. In addition, frequent interactions with the environment increase the risk of violating safety constraints.

[0079] To address the above-mentioned issues, the disclosed embodiments provide a method for training a policy model. After obtaining a first environmental state, the first environmental state is input into a policy model to be trained to obtain an action corresponding to the first environmental state. The first environmental state and action are then processed using a pre-trained large language model to obtain a safety indication signal corresponding to the action. The second environmental state and reward are obtained by interacting with the action and the environment. Subsequently, the first environmental state, action, safety indication signal, second environmental state, and reward are used as interaction data, and the policy model to be trained is trained based on the interaction data. The large language model is used to perform a safety assessment on the environmental state-action pair, generating a dense safety indication signal as a safety constraint during the training process. The large language model utilizes its rich prior knowledge to quickly adapt to different task scenarios and provide accurate safety feedback, thereby ensuring that sufficient safety information is obtained during the training process.

[0080] In addition, the disclosed embodiments utilize an adaptive safety entropy mechanism as a safety metric to dynamically adjust the safety of the strategy under different environmental conditions. In the early stages of training, the adaptive safety entropy reduces the frequency of visits to dangerous states by evenly distributing action probabilities. Later in training, it dynamically adjusts based on the strategy's learning to increase the frequency of visits to safe states. This dynamically balances exploration and safety, ensuring that strategy optimization during training maximizes cumulative rewards while meeting safety constraints.

[0081] The disclosed embodiment also adopts an off-policy training method, storing historical interaction data through an experience replay pool to improve data utilization efficiency.

[0082] In addition, when training the policy model, the embodiment of the present disclosure uses two action value networks and two target networks as auxiliary to dynamically balance exploration and security, ensuring that the policy optimization during training maximizes the cumulative reward while satisfying safety constraints.

[0083] The defects in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by this disclosure for the above problems below should be the contributions made by the inventors to this disclosure during the disclosure process.

[0084] To facilitate understanding of the technical solution of the present disclosure, the technical terms used in the embodiments of the present disclosure are first explained:

[0085] Reinforcement learning (RL) is a key branch of machine learning. Its core concept is to enable an agent to learn optimal behavior strategies through trial-and-error interaction with its environment, thereby maximizing long-term cumulative rewards. RL is widely used in areas such as game AI (such as AlphaGo), robotic control, autonomous driving, and recommendation systems.

[0086] An agent, the subject that makes decisions, interacts with the environment by sensing the state of the environment and selecting actions.

[0087] The environment refers to everything outside the agent, including the state space, action space, and dynamic rules. When the agent performs an action, the environment transitions from one state to another and returns a reward. For example, in the field of autonomous driving, the environment includes the road conditions near the autonomous vehicle (such as road grade and whether it is an intersection), obstacle conditions (such as the direction, speed, and acceleration of pedestrians or other vehicles), and road driving rules (such as whether lane changes and overtaking are allowed).

[0088] State (S) is a set of features that describe the state of the environment. Agents determine their actions based on observing the state of the environment. For example, in the field of autonomous driving, agents use radar and image sensors to identify the types of obstacles, their movement directions, speeds, and accelerations, or GPS to identify information about the current road section.

[0089] Actions (A) are the set of operations that an agent can perform in a given state, representing the legal action space under the environment state s. For example, in the field of autonomous driving, actions can be defined as the area within which an agent can move, the speed range within which an agent can move, the acceleration range within which an agent can move, etc.

[0090] Reward (R), the immediate feedback from the environment to the agent's actions, is the core goal of driving learning. A properly defined reward function is needed to guide the agent in learning the desired behavior. For example, in Go, "winning" is +1 and "losing" is -1. In autonomous driving, "reaching the destination" is +1 and "not reaching the destination" is -1.

[0091] Policy (π): The policy model of the agent, which describes the mapping from state to action and is divided into:

[0092] Deterministic strategies: , which means selecting action a in the environment state s.

[0093] Randomness strategy: , which represents the probability of selecting action a in the environment state s.

[0094] Value Function: Evaluates the long-term reward expectation of the state-action pair and is used to guide policy optimization. It includes: Action Value Function (Q Function):

[0095]

[0096] It represents the expected cumulative reward after executing action a from state s and following policy π.

[0097] To facilitate understanding of this embodiment, we first provide a detailed introduction to the training of a policy model disclosed in this embodiment. The policy model training method provided in this embodiment is generally executed by a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, an in-vehicle device, a wearable device, etc. In some possible implementations, the policy model training method may be implemented by a processor invoking computer-readable instructions stored in a memory.

[0098] It should be noted that the training method of the strategy model provided in the embodiments of the present disclosure can be used in a variety of scenarios, such as game and gambling scenarios, robot control, autonomous driving, recommendation systems, financial transactions, etc., and the specific embodiments of the present disclosure are not limited.

[0099] The following describes the training method of the policy model provided by the embodiment of the present disclosure.

[0100] See also Figure 1 FIG. 1 is a flowchart of a method for training a policy model according to an embodiment of the present disclosure, wherein the method includes steps S101 to S104, wherein:

[0101] S101: Acquire a first environmental state, and input the first environmental state into a policy model to be trained to obtain a first action corresponding to the first environmental state;

[0102] S102: Processing the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action; the safety indication signal is used to indicate whether it is safe to perform the first action under the first environmental state; and

[0103] S103: Interacting with the environment based on the first action to obtain a second environment state and a reward;

[0104] S104: Constructing interaction data based on the first environmental state, the first action, the safety indication signal, the second environmental state, and the reward, and training the policy model to be trained based on the interaction data to obtain a target policy model.

[0105] There is no logical order for executing S102 and S103 above. S102 may be executed first and then S103, or S103 may be executed first and then S102. S102 and S103 may also be executed in parallel.

[0106] The above S101 to S104 are described in detail below.

[0107] Regarding S101 above:

[0108] In a specific implementation, when obtaining the first environmental state, for example, the first environmental state of the working environment of the agent in which the target strategy model is deployed is obtained. This working environment is, for example, related to the agent's application scenario. When the agent's application scenario is a game scenario, the agent's working environment is the game scenario. The game scenario can be a virtual game scenario, a real game scenario, an augmented reality (AR) game scenario, or a virtual reality (VR) game scenario, depending on the actual application needs. When the agent's application scenario is a robot control scenario, the agent's working environment is the robot's specific working environment, such as the sorting site for an automatic sorting robot or the search and rescue site for an automatic search and rescue robot. When the agent's application scenario is in the field of autonomous driving, the agent's working environment is the working environment corresponding to the autonomous driving road, such as the road on which the autonomous driving vehicle on which the agent is deployed is currently traveling.

[0109] When the intelligent agent is used in different scenarios, the method of obtaining the first environment state is different, and the obtained first environment state is also different.

[0110] For example, when an intelligent agent is used in a game, the first environmental state may be the state of the game scene. For example, in the case of a strategic game (e.g., Go), the first environmental state may be the layout of the chess pieces on the board. In a real-life game between a human and an intelligent agent, the first environmental state may be acquired by, for example, capturing an image of the current chessboard using an image sensor and, based on this image, determining the current state of the board, such as the position of the chess pieces and the player to whom they belong. In a game between a human and an intelligent agent using a computer, the first environmental state may be acquired by, for example, reading the game progress information currently recorded by the computer device to determine the state of the board. Alternatively, when an intelligent agent is used to control a robot, the first environmental state may be the state of the robot's operating scene. For example, if the robot is used to sort items on a conveyor belt, the first environmental state may be the distribution of the items on the conveyor belt. The first environmental state may be acquired by, for example, capturing an image of the conveyor belt using an image sensor and, based on this image, determining the distribution of the items on the conveyor belt and the conveying status. When an intelligent agent is used in the field of autonomous driving, the first environmental state is, for example, the current road state of the autonomous vehicle, such as obstacle distribution, obstacle movement, and road conditions. For example, the obstacle state and road conditions within a certain range of the vehicle can be obtained using onboard radar or onboard image sensors.

[0111] After obtaining the first environment status After that, the first environment state can be Input to the strategy model to be trained, use the strategy model to be trained to process the first environment state to output the first action corresponding to the first environment state .

[0112] Specifically, for different application scenarios, the first action corresponding to the first environmental state is also different. For example, when the intelligent agent is used in a game scenario, the first action corresponding to the first environmental state is the next game operation; in a game of chess, the first action is, for example, placing a chess piece at a specific landing point. When the intelligent agent is used for robot control, the first action corresponding to the first environmental state is a specific control action for the robot; when the intelligent agent is used in the field of autonomous driving, the first action corresponding to the first environmental state is, for example, the next driving action, such as moving forward, backward, accelerating, decelerating, changing lanes, controlling the high beam to flash, etc. Specifically, according to different application scenarios, the first action is also different.

[0113] Regarding the above S102, the pre-trained large language model is used, for example, to perform a safety assessment on the environment state-action pair, to assess whether it is safe to perform a certain action under a specific environment state, and to output a corresponding safety indication signal. In the embodiment of the present disclosure, the safety indication signal and the signal for indicating the safety of the action under the first environment state are Execute the first action Is it safe?

[0114] In a possible implementation, the safety indication signal is, for example, a binary signal; when the value of the safety indication signal is a first value, the indication is in the first environmental state. Execute the first action Safety; when the safety indication signal takes the second value, the indication is in the first environmental state Execute the first action Not safe.

[0115] For example, it is assumed that when the value of the safety indication signal is 0, it indicates safety, and when the value of the safety indication signal is 1, it indicates unsafe.

[0116] In different scenarios, whether it is safe or not can be set according to the specific application scenario; for example, in a game scenario, safety can be, for example, being attacked, having chess pieces eaten, losing the game, etc., and the content corresponding to safety will be different depending on the different games. For another example, in a scenario where a robot is performing item sorting control, safety can mean, for example, that items to be sorted will not be missed; unsafe means that the probability of missing items to be sorted reaches a certain probability threshold. In the field of autonomous driving, safety can mean, for example, that there will be no collision; unsafe means that the probability of colliding with nearby obstacles, or driving out of the current lane reaches a certain probability threshold, etc. It can be set according to actual needs, and the embodiments of this disclosure are not limited thereto.

[0117] In an optional embodiment, a specific method of processing the first environmental state and the action using a pre-trained large language model to obtain a safety indication signal corresponding to the action is further provided, including:

[0118] converting the first environmental state and the first action into a natural language description;

[0119] The first environment state and the first action described in natural language are input into the large language model to obtain the safety indication signal.

[0120] In a specific implementation, for example, a state conversion module can be used to convert the first environmental state and the first action into a natural language description. The state conversion module may include two parts: state translation and action translation. State translation converts the numerical or symbolic environmental state encoding into a natural language with clear semantics. Action translation can discretize continuous actions into different options and convert them into natural language descriptions.

[0121] After the first environment state and the first action are converted into natural language descriptions, the first environment state and the first action described in the natural language are input into the large language model to obtain a safety indication signal.

[0122] The safety indication signal is, for example, represented as: .

[0123] Regarding S103 above:

[0124] When interacting with the environment based on the first action, for example, in the first environment state, the intelligent agent deployed with the policy model to be trained is controlled to perform the first action to obtain the second environment state and reward.

[0125] Specifically, the second environmental state, for example, is in the first environmental state Next, the first action is performed At some point after the first action The impact of the first environmental state Change to another new state, which is the second environment state, which can be expressed as: .

[0126] award For example, the environment is the first action Instant feedback signal.

[0127] Taking autonomous driving as an example, the first environmental state at a certain moment is , the first environmental state It is represented as a vector, which represents the position, speed and surrounding environment (such as the distribution, movement and type of obstacles) of the autonomous driving vehicle. The policy model outputs the distribution of different actions based on this vector, and obtains a driving action by sampling this distribution. , the action is assumed to be walking forward. The agent controls the vehicle to perform this action. At this time, the vehicle's environment, position, speed, etc. have changed, and a new environmental state is obtained. If the task is completed, such as moving to the desired location, a positive reward value (scalar) is returned, otherwise if the first action is executed If a collision occurs after the movement, a negative reward value is returned; if neither the desired position is reached nor a collision occurs, 0 is returned, that is, the reward value is 0.

[0128] Regarding S104 above:

[0129] Based on the above S101~S103, the first environmental state is obtained. , the corresponding first action , safety indication signal , the second environmental state ,award After that, the first environmental state can be , the corresponding action , safety indication signal , the second environmental state ,award Constitute a set of interaction data, , and use the interaction data to train the policy model to be trained to obtain the target policy model.

[0130] In another embodiment of the present disclosure, for example, at least one of the following methods a1 and a2 may be used to train the policy model to be trained using the interaction data:

[0131] a1: Use the interaction data generated in the current training cycle to train the policy model in the current training cycle.

[0132] In a specific implementation, the policy model to be trained can be trained for multiple training cycles. During each training cycle, steps S101 to S103 can be performed at least once. Specifically, the agent can be used to obtain interaction data corresponding to at least one moment in time. In other words, assuming that during a training cycle, the agent obtains interaction data corresponding to n moments in time, there will be n sets of interaction data obtained, where n is an integer greater than 0.

[0133] In this case, n sets of interaction data may be used to train the decision model to be trained in the current training cycle.

[0134] a2: Storing the interaction data determined in the current training cycle in the experience replay pool; sampling the interaction data stored in the experience replay pool to obtain target interaction data corresponding to the current training cycle; and training the strategy model to be trained in the current training cycle based on the target interaction data.

[0135] In a specific implementation, similar to step a1 above, the policy model to be trained can also be trained for multiple training cycles. In each training cycle, n sets of interaction data are obtained according to the method described in step a1 above. These n sets of interaction data can then be stored in an experience replay pool. Subsequently, at least one set of target interaction data corresponding to the current training cycle can be sampled from the experience replay pool according to a certain sampling rate and / or sampling method. Based on this target interaction data, the policy model to be trained can then be trained for the current training cycle.

[0136] In this case, if the current training cycle is the first training cycle, the experience replay pool only includes the interaction data generated in the previous training cycle. Therefore, in general, when the policy model to be trained is trained in the first training cycle, the target interaction data includes at least part of the interaction data generated in the first training cycle.

[0137] If the current training cycle is the kth training cycle, and it is a training cycle other than the first training cycle, then the experience replay pool includes not only the interaction data generated in the current training cycle, but also the interaction data stored in the experience replay pool from the 1st to the k-1th training cycles. Therefore, when the policy model is trained in the kth training cycle, the target interaction data may include not only the interaction data generated in the kth training cycle, but also the interaction data generated in at least part of the training cycles from the 1st to the k-1th training cycles.

[0138] In this way, the experience replay pool is used to store the interaction data of each training cycle, and the experience replay pool is sampled in each training cycle, so that the target interaction data obtained by sampling is used to train the policy model to be trained. This adopts the off-policy training method. By storing historical interaction data in the experience replay pool, the utilization efficiency of the interaction data can be improved, so that sufficient target interaction data can be obtained in each training cycle, thereby improving the training efficiency of the decision network.

[0139] See also Figure 2 As shown, the embodiment of the present disclosure further provides a specific method for training the policy model to be trained in the current training cycle based on the target interaction data, including:

[0140] S201: Determine a policy model, an action-value network, and a target network to be trained in a current training cycle; wherein the target network and the action-value network have the same network architecture;

[0141] S202: Based on the target interaction data, updating the action value network by minimizing the Bellman residual, updating the policy model by maximizing the objective function, and updating the target network by a slow update mechanism.

[0142] In a specific implementation, when the current training cycle is the first training cycle, the policy model, action value network, and target network to be trained in the current training cycle are respectively the initialized policy model, the initialized action value network, and the initialized target network; and the network architecture and network internal parameters of the initialized action value network and the initialized target network are the same; when the current training cycle is a training cycle other than a training cycle, the policy model, action value network, and target network to be trained in the current training cycle are the policy model to be trained, the action value network, and the target network after being updated in the previous training cycle of the current training cycle, respectively.

[0143] In a specific implementation, there can be two action value networks, including a first action value network and a second action value network; there can also be two target networks, including a first target network and a second target network.

[0144] Among them, the network architecture of the first action value network and the second action value network are the same, but the network internal parameters are different; the network architecture and network internal parameters of the initialized first action value network and the initialized first target network are the same; the network internal parameters of the initialized second action value network and the initialized second target network are the same.

[0145] In a specific implementation, when initializing the action-value network, the network internal parameters of the first action-value network and the second action-value network can be randomly initialized to obtain the first action-value network and the second action-value network. Furthermore, when initializing the first target network, the first target network is initialized according to the network internal parameters of the first action-value network; when initializing the second target network, the second target network is initialized according to the network internal parameters of the second action-value network.

[0146] During subsequent training, the action-value network and target network are updated during each training cycle. Consequently, the internal parameters of the first action-value network and the first target network will differ, and the internal parameters of the second action-value network and the second target network will also gradually differ. The first and second target networks can automatically adjust the degree of optimization during each round of action-value network optimization to address the potential overestimation of an action by a single action-value network, making the training process more stable.

[0147] In another embodiment of the present disclosure, a specific method is provided for updating the action value network of the current training cycle by minimizing the Bellman residual based on the target interaction data, updating the policy model of the current training cycle by maximizing the objective function, and updating the target network of the current training cycle by a slow update mechanism, including:

[0148] (1): Based on the preset weights, the network parameters of the target network in the current training cycle and the network parameters of the action value network in the current training cycle are weighted and summed to obtain the target network parameters in the current training cycle; based on the target network parameters, the target network in the current training cycle is updated;

[0149] (2): Using the action-value network of the current training cycle, the first environment state and the first action in the target interaction data are processed to obtain a first cumulative reward expectation; and

[0150] Using the target network of the current training cycle or the updated target network, processing the second environment state in the target interaction data and the second action corresponding to the second environment state to obtain a second cumulative reward expectation;

[0151] Substituting the first cumulative reward expectation, the second cumulative reward expectation, the safety indication information in the target interaction data, and the reward into a pre-constructed Bellman residual function, and updating the action value network of the current training cycle with the goal of minimizing the Bellman residual;

[0152] (3): Substitute the first cumulative reward expectation into the pre-built objective function, and update the policy model of the current training cycle with the goal of maximizing the objective function.

[0153] In a specific implementation, the action value network includes a first action value network and the second action-value network , the target network includes the first target network and the second target network For example, the policy model is expressed as: .in, Represents a policy, which consists of an action sequence consisting of multiple actions.

[0154] The loss function is expressed as:

[0155] (1)

[0156] (2)

[0157] (3)

[0158] Among them, the above formula (3) is used to update the parameters of the target network, represents the weighting coefficient; represents the network parameters of the action-value network; Represents the network parameters of the target network. Generally, the update speed of the network parameters of the target network can be controlled by controlling the weight coefficient.

[0159] The above formula (2) is the Bellman residual function, where Indicates security information; represents the second cumulative reward expectation; represents the first cumulative reward expectation; It represents the minimum value of the second cumulative reward expectation obtained by the first target network and the second target network respectively; Indicates reward; Represents the weight parameter, which is a preset constant.

[0160] The above formula (1) is the objective function. Represents the first cumulative reward expectation; when there are two action-value networks, it can be the mean, maximum or minimum value of the first cumulative reward expectations corresponding to the first action-value network and the second action-value network respectively. Indicates that in the strategy Under the environmental state Execute action when probability.

[0161] In another embodiment of the present disclosure, in order to be able to flexibly control the impact of security information on the training process, the embodiment of the present disclosure may also include: determining a security entropy weight parameter of the current training cycle; the security entropy weight parameter is used to adjust the weight of the security information in the target interaction information when updating the action value network; when the current training cycle is the first training cycle, the security entropy weight parameter of the current training cycle is a preset parameter; when the current training cycle is a training cycle other than the first training cycle, the security entropy weight parameter of the current training cycle is determined based on the security information in the target interaction data determined in the previous training cycle;

[0162] In this case, in the process of training the policy model, for example, based on the target interaction data and the security entropy weight parameter, the action value network of the current training cycle is updated by minimizing the Bellman residual, the policy model of the current training cycle is updated by maximizing the objective function, and the target network of the current training cycle is updated through a slow update mechanism.

[0163] In the embodiment of the present disclosure, the adaptive security entropy represents the security expectation value S of the policy when executing different actions under the environment state s, and is used to measure the security of the policy. For example, it is defined as:

[0164]

[0165] When determining the safety entropy temperature coefficient of the current training cycle, the optimization objective is the following formula (4):

[0166] (4)

[0167] Indicates the target security entropy; it is a preset constant. Represents the security entropy weight parameter.

[0168] Use this formula as the loss function and update it using gradient descent That is, in the early stage of training, the adaptive safety entropy reduces the frequency of visits to dangerous states through uniformly distributed action probabilities; as training progresses, the adaptive safety entropy dynamically adjusts according to the strategy's learning of external safety knowledge, increasing the frequency of visits to safe states; by automatically adjusting the safety entropy weight parameters, it dynamically balances exploration and safety.

[0169] In this case, when updating the action value network, the above formula (2) can be expressed as:

[0170]

[0171] Therefore, by adaptively adjusting the security entropy weight parameter based on the above formula (4) in different training cycles, the weight of security information when updating the action value network is dynamically controlled to dynamically balance exploration and security.

[0172] In another embodiment of the present disclosure, the interactive data may further include: a completion indication signal .

[0173] The completion indication signal is used to indicate the task execution status corresponding to the policy network after executing the first action in the first environmental state; wherein, the task execution status includes: other actions can be executed after the action is executed; or other actions cannot be executed after the action is executed. If the task execution status indicates that there are still subsequent actions to be executed, the completion indication signal is "false", that is, it is not completed; if the task execution status indicates that other actions cannot be executed subsequently, such as in the Go game scenario, "failure" or "victory" are both considered as not being able to execute other actions; in the autonomous driving scenario, "collision occurred", "reached the autonomous driving destination", and "did not reach the destination within the specified time" are all considered as not being able to execute other subsequent actions. At this time, the completion indication signal can be set to: "true".

[0174] When the task execution status indicates that no other subsequent actions can be taken, that is, In the case of , the updating of the action value network of the current training cycle by minimizing the Bellman residual may include, for example:

[0175] The action-value network of the current training cycle is used to process the first environmental state and the first action in the target interaction data to obtain a first cumulative reward expectation. The first cumulative reward expectation and the safety indication information in the target interaction data are substituted into a pre-constructed Bellman residual function, and the action-value network of the current training cycle is updated with the goal of minimizing the Bellman residual.

[0176] For example, in this case, the formula (2) can be expressed as:

[0177]

[0178] That is, since there are no other subsequent actions that can be performed, there will be no second environmental state corresponding to the next moment, and there will be no reward corresponding to the current moment.

[0179] See also Figure 3 As shown, the embodiment of the present disclosure also provides a specific example of a training decision model, including:

[0180] (1): Preliminary preparation stage:

[0181] A large language model is pre-trained to generate a specific binary safety feedback signal, i.e., a safety signal, based on the environment state and the corresponding action.

[0182] (2) Training phase:

[0183] S301: Initialize the policy network to be trained , two action-value networks and , two target networks and , and experience replay pool B.

[0184] in, and The parameters are exactly the same. and The parameters are the same, and The model structures are the same, but the internal parameters of the models are different.

[0185] S302: Execute the kth round of training and determine the first environment state in the kth round of training , Represents the environmental state at a certain moment. Input the policy model to be trained and get the first action .

[0186] S303: Using the first environmental state and the corresponding first action , interact with the environment and get the second environment state at the next moment ,award , and completion indication signals .

[0187] S304: The first environmental state and the first action Convert the natural language description into a description, and input the current environment state and action described in natural language into the pre-trained large language model to obtain a safety indication signal .

[0188] S305: Based on the first environmental state , first action , Second environmental state ,award , completion indication signal , and security signals constitute interactive data , the interaction data Stored in experience replay pool B.

[0189] S306: Randomly sample from the experience replay pool B to obtain target interaction data b corresponding to the first round of training. The target interaction data b includes at least one set of interaction data.

[0190] S307: Using target interaction data and security entropy temperature coefficient, update the two action value networks by minimizing the Bellman residual and , update the policy network by maximizing the objective function , and update the two target networks through the slow update mechanism and .

[0191] The loss function is expressed as:

[0192]

[0193]

[0194] S308: Based on the adaptive security entropy mechanism, the security entropy temperature coefficient is updated using the target interaction data b. , and returns to S302 to perform the next round of training.

[0195] The training method of the policy model provided by the embodiments of the present disclosure has the following beneficial effects:

[0196] Large Language Model Feedback Mechanism: A large language model (LLM) is used to assess the safety of state-action pairs, generating dense safety constraint signals. Leveraging its rich prior knowledge, the LLM can quickly adapt to different task scenarios and provide accurate safety feedback.

[0197] Adaptive Safety Entropy Mechanism: Adaptive Safety Entropy is introduced as a safety metric to dynamically adjust the safety of the policy under different states. In the early stages of training, Adaptive Safety Entropy reduces the frequency of visits to dangerous states by evenly distributing action probabilities. Later in training, it dynamically adjusts based on policy learning to increase the frequency of visits to safe states.

[0198] Off-policy training: Using an off-policy training method, the experience replay pool stores historical interaction data to improve data utilization efficiency. Combining the dual action value network and target network architecture reduces fluctuations during training.

[0199] Optimization objectives and convergence: The safety constraint problem is transformed into an unconstrained optimization problem through the Lagrange multiplier method, and the policy gradient method is used to optimize the strategy to accelerate the optimization of the objectives and convergence.

[0200] Corresponding to the aforementioned embodiments of the method for training a policy model, the present application also provides embodiments of an apparatus for training a policy model.

[0201] The embodiment of the training device of the strategy model of the present application can be applied to a computer device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the computer device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, such as Figure 4 The figure shows a hardware structure diagram of the computer equipment where the training device of the strategy model of this application is located, except Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, the computer device where the apparatus in the embodiment is located may also include other hardware according to the actual function of the training apparatus of the policy model, which will not be described in detail.

[0202] Please refer to Figure 5 The training device of the policy model provided in the embodiment of the present disclosure includes:

[0203] An acquisition module 51 is configured to acquire a first environmental state and input the first environmental state into a policy model to be trained to obtain a first action corresponding to the first environmental state;

[0204] a processing module 52 configured to process the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action; the safety indication signal being used to indicate whether it is safe to perform the first action under the first environmental state; and

[0205] Interacting with the environment based on the first action to obtain a second environment state and a reward;

[0206] The training module 53 is used to form interaction data based on the first environmental state, the first action, the safety indication signal, the second environmental state, and the reward, and to train the policy model to be trained based on the interaction data to obtain a target policy model.

[0207] Optionally, the processing module 52, when processing the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action, is configured to:

[0208] converting the first environmental state and the first action into a natural language description;

[0209] The first environment state and the first action described in natural language are input into the large language model to obtain the safety indication signal.

[0210] Optionally, the training module 53, when training the policy model to be trained based on the interaction data, is configured to:

[0211] Using the interaction data generated in the current training cycle as target interaction data, and training the strategy model to be trained in the current training cycle based on the target interaction data; and / or,

[0212] Store the interaction data determined in the current training cycle into the experience replay pool;

[0213] The interaction data stored in the experience replay pool is sampled to obtain target interaction data corresponding to the current training cycle; and the strategy model to be trained is trained for the current training cycle based on the target interaction data.

[0214] Optionally, the training module 53, when training the policy model to be trained in the current training cycle based on the target interaction data, is configured to:

[0215] Determine the policy model, action value network, and target network to be trained in the current training cycle; wherein the policy model, action value network, and target network to be trained in the current training cycle are, respectively, an initialized policy model, an initialized action value network, and an initialized target network; and the network architecture and network internal parameters of the initialized action value network and the initialized target network are the same; or, they are the policy model to be trained, the action value network, and the target network updated in the previous training cycle, which were updated in the previous training cycle, and the target network updated in the previous training cycle;

[0216] Based on the target interaction data, the action value network of the current training cycle is updated by minimizing the Bellman residual, the policy model of the current training cycle is updated by maximizing the objective function, and the target network of the current training cycle is updated through a slow update mechanism.

[0217] Optionally, the action value network includes: a first action value network and a second action value network; the target network includes: a first target network and a second target network; the first action value network and the second action value network have the same network architecture but different network internal parameters; the network architecture and network internal parameters of the initialized first action value network and the initialized first target network are the same; the network internal parameters of the initialized second action value network and the initialized second target network are the same.

[0218] Optionally, the training module 53, when updating the action-value network of the current training cycle by minimizing the Bellman residual based on the target interaction data, updating the policy model of the current training cycle by maximizing the objective function, and updating the target network of the current training cycle by the slow update mechanism, is configured to:

[0219] Based on the preset weights, the network parameters of the target network in the current training cycle and the network parameters of the action value network in the current training cycle are weighted and summed to obtain the target network parameters of the target network in the current training cycle; based on the target network parameters, the target network of the current training cycle is updated;

[0220] Using the action-value network of the current training cycle, the first environment state and the first action in the target interaction data are processed to obtain a first cumulative reward expectation; and using the target network of the current training cycle or the updated target network, the second environment state in the target interaction data and the second action corresponding to the second environment state are processed to obtain a second cumulative reward expectation; the first cumulative reward expectation, the second cumulative reward expectation, the safety indication information in the target interaction data, and the reward are substituted into a pre-constructed Bellman residual function, and the action-value network of the current training cycle is updated with the goal of minimizing the Bellman residual;

[0221] Substitute the first cumulative reward expectation into the pre-built objective function, and update the policy model of the current training cycle with the goal of maximizing the objective function.

[0222] Optionally, the interaction data further includes a completion indication signal; the completion indication signal is used to indicate the execution status of the task corresponding to the policy network after the first action is performed in the first environment state;

[0223] When the task execution status indicates that no other actions can be performed, the training module 53, when updating the action value network of the current training cycle by minimizing the Bellman residual, is configured to:

[0224] The action-value network of the current training cycle is used to process the first environmental state and the first action in the target interaction data to obtain a first cumulative reward expectation. The first cumulative reward expectation and the safety indication information in the target interaction data are substituted into a pre-constructed Bellman residual function, and the action-value network of the current training cycle is updated with the goal of minimizing the Bellman residual.

[0225] Optionally, the method further includes a determination module for determining a security entropy weight parameter for the current training cycle; the security entropy weight parameter is used to adjust the weight of the security information in the target interaction information when updating the action value network; the security entropy weight parameter for the current training cycle is a preset parameter, or is determined based on the security information in the target interaction data determined in the previous training cycle; the training module, when updating the action value network of the current training cycle by minimizing the Bellman residual, is used to:

[0226] Based on the security entropy weight parameter, the action value network of the current training cycle is updated by minimizing the Bellman residual.

[0227] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0228] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the policy model training method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0229] The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the training method of the policy model described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0230] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0231] The computer program or instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; an optical medium, such as a digital video disk; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0232] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0233] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for a strategy model, characterized in that: The method comprises: Obtaining a first environmental state and inputting the first environmental state into a policy model to be trained to obtain a first action corresponding to the first environmental state; the first environmental state includes: a current road state on which the autonomous driving vehicle is traveling; and the first action includes a next driving action; Processing the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action; the safety indication signal is used to indicate whether it is safe to perform the first action under the first environmental state; and Interacting with the environment based on the first action to obtain a second environment state and a reward; Constructing interaction data based on the first environmental state, the first action, the safety indication signal, the second environmental state, and the reward, and training the policy model to be trained based on the interaction data to obtain a target policy model; The training of the strategy model to be trained based on the interaction data includes: Using the interaction data generated in the current training cycle as target interaction data, and training the strategy model to be trained in the current training cycle based on the target interaction data; and / or, Store the interaction data determined in the current training cycle into the experience replay pool; Sampling the interaction data stored in the experience replay pool to obtain target interaction data corresponding to the current training cycle; training the policy model to be trained for the current training cycle based on the target interaction data; The training of the strategy model to be trained in a current training cycle based on the target interaction data includes: Determine the policy model, action-value network, and target network to be trained for the current training cycle; Based on the target interaction data, updating the action-value network of the current training cycle by minimizing the Bellman residual, updating the policy model of the current training cycle by maximizing the objective function, and updating the target network of the current training cycle by a slow update mechanism; The method further comprises: Determining a security entropy weight parameter for the current training cycle; the security entropy weight parameter is used to adjust the weight of the security information in the target interaction data when updating the action-value network; the security entropy weight parameter for the current training cycle is a preset parameter, or is determined based on the security information in the target interaction data determined in the previous training cycle; Update the action-value network of the current training cycle by minimizing the Bellman residual, including: Based on the security entropy weight parameter, the action value network of the current training cycle is updated by minimizing the Bellman residual.

2. The training method according to claim 1, characterized in that The using a pre-trained large language model to process the first environmental state and the first action to obtain a safety indication signal corresponding to the first action includes: converting the first environmental state and the first action into a natural language description; The first environment state and the first action described in natural language are input into the large language model to obtain the safety indication signal.

3. The method according to claim 1, characterized in that The strategy model, action value network, and target network to be trained in the current training cycle are respectively the initialized strategy model, the initialized action value network, and the initialized target network; and the network architecture and network internal parameters of the initialized action value network and the initialized target network are the same; or, the strategy model, action value network, and target network to be trained in the current training cycle are respectively the strategy model to be trained, the action value network, and the target network after being updated in the previous training cycle of the current training cycle.

4. The method according to claim 3, characterized in that The action value network includes: a first action value network and a second action value network; the target network includes: a first target network and a second target network; the first action value network and the second action value network have the same network architecture but different network internal parameters; the network architecture and network internal parameters of the initialized first action value network and the initialized first target network are the same; the network internal parameters of the initialized second action value network and the initialized second target network are the same.

5. The method according to claim 3 or 4, characterized in that Based on the target interaction data, the action value network of the current training cycle is updated by minimizing the Bellman residual, the policy model of the current training cycle is updated by maximizing the objective function, and the target network of the current training cycle is updated by a slow update mechanism, including: Based on the preset weights, the network parameters of the target network in the current training cycle and the network parameters of the action value network in the current training cycle are weighted and summed to obtain the target network parameters of the target network in the current training cycle; based on the target network parameters, the target network of the current training cycle is updated; Using the action-value network of the current training cycle, the first environment state and the first action in the target interaction data are processed to obtain a first cumulative reward expectation; and using the target network of the current training cycle or the updated target network, the second environment state in the target interaction data and the second action corresponding to the second environment state are processed to obtain a second cumulative reward expectation; the first cumulative reward expectation, the second cumulative reward expectation, the safety indication information in the target interaction data, and the reward are substituted into a pre-constructed Bellman residual function, and the action-value network of the current training cycle is updated with the goal of minimizing the Bellman residual; Substitute the first cumulative reward expectation into the pre-built objective function, and update the policy model of the current training cycle with the goal of maximizing the objective function.

6. The method according to claim 5, characterized in that The interaction data further includes a completion indication signal; the completion indication signal is used to indicate the execution status of the task corresponding to the policy network after the first action is performed in the first environment state; When the task execution status indicates that no other actions can be performed, updating the action value network of the current training cycle by minimizing the Bellman residual includes: Processing the first environment state and the first action in the target interaction data using the action-value network of the current training cycle to obtain a first cumulative reward expectation; The first cumulative reward expectation and the safety indication information in the target interaction data are substituted into a pre-constructed Bellman residual function, and the action value network of the current training cycle is updated with the goal of minimizing the Bellman residual.

7. A training device for a strategy model, characterized in that: The device comprises: an acquisition module, configured to acquire a first environmental state and input the first environmental state into a policy model to be trained to obtain a first action corresponding to the first environmental state; the first environmental state includes: a current road state on which the autonomous driving vehicle is traveling; and the first action includes a next driving action; a processing module, configured to process the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action; the safety indication signal is used to indicate whether it is safe to perform the first action under the first environmental state; and Interacting with the environment based on the first action to obtain a second environment state and a reward; a training module, configured to construct interaction data based on the first environmental state, the first action, the safety indication signal, the second environmental state, and the reward, and train the policy model to be trained based on the interaction data to obtain a target policy model; The training module is configured to: Using the interaction data generated in the current training cycle as target interaction data, and training the strategy model to be trained in the current training cycle based on the target interaction data; and / or, Store the interaction data determined in the current training cycle into the experience replay pool; Sampling the interaction data stored in the experience replay pool to obtain target interaction data corresponding to the current training cycle; training the policy model to be trained for the current training cycle based on the target interaction data; The training module is configured to, when training the policy model to be trained in a current training cycle based on the target interaction data,: Determine the policy model, action-value network, and target network to be trained for the current training cycle; Based on the target interaction data, updating the action-value network of the current training cycle by minimizing the Bellman residual, updating the policy model of the current training cycle by maximizing the objective function, and updating the target network of the current training cycle by a slow update mechanism; Also included is a determination module, configured to: Determining a security entropy weight parameter for the current training cycle; the security entropy weight parameter is used to adjust the weight of the security information in the target interaction data when updating the action-value network; the security entropy weight parameter for the current training cycle is a preset parameter, or is determined based on the security information in the target interaction data determined in the previous training cycle; The training module, when updating the action-value network of the current training cycle by minimizing the Bellman residual, is used to: Based on the security entropy weight parameter, the action value network of the current training cycle is updated by minimizing the Bellman residual.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When executing the program, the processor implements the following steps: obtaining a first environmental state, inputting the first environmental state into a policy model to be trained, and obtaining a first action corresponding to the first environmental state; the first environmental state includes: a current road state on which the autonomous driving vehicle is traveling; the first action includes a next driving action; Processing the first environmental state and the first action using a pre-trained large language model to obtain a safety indication signal corresponding to the first action; the safety indication signal is used to indicate whether it is safe to perform the first action under the first environmental state; and Interacting with the environment based on the first action to obtain a second environment state and a reward; Constructing interaction data based on the first environmental state, the first action, the safety indication signal, the second environmental state, and the reward, and training the policy model to be trained based on the interaction data to obtain a target policy model; The training of the strategy model to be trained based on the interaction data includes: Using the interaction data generated in the current training cycle as target interaction data, and training the strategy model to be trained in the current training cycle based on the target interaction data; and / or, Store the interaction data determined in the current training cycle into the experience replay pool; Sampling the interaction data stored in the experience replay pool to obtain target interaction data corresponding to the current training cycle; training the policy model to be trained for the current training cycle based on the target interaction data; The training of the strategy model to be trained in a current training cycle based on the target interaction data includes: Determine the policy model, action-value network, and target network to be trained for the current training cycle; Based on the target interaction data, updating the action-value network of the current training cycle by minimizing the Bellman residual, updating the policy model of the current training cycle by maximizing the objective function, and updating the target network of the current training cycle by a slow update mechanism; Also includes: Determining a security entropy weight parameter for the current training cycle; the security entropy weight parameter is used to adjust the weight of the security information in the target interaction data when updating the action-value network; the security entropy weight parameter for the current training cycle is a preset parameter, or is determined based on the security information in the target interaction data determined in the previous training cycle; Update the action-value network of the current training cycle by minimizing the Bellman residual, including: Based on the security entropy weight parameter, the action value network of the current training cycle is updated by minimizing the Bellman residual.

Images