Safety reinforcement learning intelligent vehicle afflux decision-making method and system based on prediction trajectory

By introducing predicted trajectory calculation safety action sets in reinforcement learning autonomous driving decisions, the problems of poor safety and long cycles during training are solved, and more efficient and safe autonomous driving decisions are achieved, suitable for complex traffic scenarios.

CN118928463APending Publication Date: 2024-11-12YANSHAN UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410990980.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing reinforcement learning decision-making methods for autonomous driving are poor in safety, long cycles, high costs, and difficult to put into practical applications during training, and fail to effectively consider changes in traffic situations.

Method used

Using a safe reinforcement learning method based on predicted trajectory, build a reinforcement learning input decision model, define state space and action space, design reward function and cost function, calculate the total discounted cost through predicted trajectory, determine the safe action set, and replace unsafe actions. , improve training safety and efficiency.

Benefits of technology

It significantly reduces the number of collisions during the training process, shortens the training cycle, reduces costs, improves the safety and efficiency of autonomous driving decisions, and takes into account the evolution of future traffic situations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118928463A_ABST
    Figure CN118928463A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of automatic driving behavior decision-making, and discloses a safety reinforcement learning intelligent vehicle afflux decision-making method and system based on a prediction track. And constructing a reinforcement learning afflux decision model, obtaining a prediction trajectory corresponding to each action by using the prediction model, calculating the cost of each step state in the prediction trajectory based on a cost function, and calculating the total discount cost of the prediction trajectory by using a total discount cost calculation method. And determining a safety action set according to the discount total cost of each action. If the original decision action of the intelligent agent is in the safety action set or the safety action set is empty, outputting the original decision action; and if not in the safety action set, outputting the action with the maximum value in the safety action set. According to the method, the predicted trajectory is fused into safe reinforcement learning, and unsafe actions are replaced with safe actions, so that the collision frequency in the training process is effectively reduced, the training period is shortened, the training cost is reduced, and the safety of afflux decision making is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of autonomous driving behavior decision-making, and specifically to a smart car merging decision-making method and system based on safe reinforcement learning of predicted trajectories. Background Art

[0002] As a key core part of autonomous driving, the decision layer takes over the perception and prediction layer, and makes high-level behavioral decisions including lane changing and acceleration and deceleration according to the current environmental traffic situation, which plays an important role in the safety and traffic efficiency of autonomous driving vehicles. However, in complex traffic scenarios where human drivers and autonomous driving vehicles are mixed, especially when merging into ramps in merging areas, the safety decision-making of autonomous driving still faces challenges.

[0003] On urban roads, merging into a merging area is one of the most common driving tasks. At present, the decision-making methods used for merging into a merging area ramp are mainly divided into two categories: rule-based methods and learning-based methods. Among them, the rule-based decision-making method pre-sets rules and can cope with common and simple traffic scenarios, with the advantages of simplicity and efficiency. However, the rule-based method often requires sufficient prior knowledge, and the actual traffic environment is complex and changeable. The set rules are difficult to cope with complex traffic scenarios, and thus it is impossible to make safe and efficient decisions. The decision-making method based on learning uses a data-driven method through continuous training to enable the decision-making model to cope with more situations and make corresponding optimal decisions under different traffic situations. The decision-making method based on deep reinforcement learning explores the environment and accepts feedback rewards from the environment, thereby continuously optimizing the decision-making model. This makes the reinforcement learning decision-making model more versatile to the environment and is considered to have great potential when facing complex traffic scenarios.

[0004] However, reinforcement learning requires training to obtain the optimal decision model, and the decisions made by the agent before training convergence are often less safe. At the same time, the training cycle is long and the cost is high, which makes it difficult to put the autonomous driving decision-making method based on reinforcement learning into practical application. Safety reinforcement learning is an effective means to improve the safety of the training process and reduce the cost of training, but existing safety reinforcement learning often only focuses on the current traffic situation and fails to take into account the changes in the traffic situation. Summary of the invention

[0005] The present application provides a safe reinforcement learning intelligent vehicle convergence decision-making method based on predicted trajectory to solve the problem in the prior art that reinforcement learning requires training to obtain the optimal decision-making model, and the decisions made by the intelligent agent in the process before training convergence are often less safe. At the same time, the training cycle is long and the cost is high, making it difficult to put the autonomous driving decision-making method based on reinforcement learning into practical application.

[0006] Correspondingly, the present application also provides a safe reinforcement learning intelligent vehicle integration decision system based on predicted trajectory, an electronic device, and a computer-readable storage medium to ensure the implementation and application of the above method.

[0007] In order to solve the above technical problems, the present application discloses a safe reinforcement learning intelligent vehicle merging decision method based on predicted trajectory, the method comprising:

[0008] Construct a reinforcement learning merging decision model, define the state space and action space of the agent, and design a reward function that considers merging success and traffic efficiency, as well as a cost function that considers collision and safety;

[0009] Use the pre-built prediction model to obtain the predicted trajectory corresponding to each action in the action space, and calculate the cost of each step state in the predicted trajectory based on the cost function;

[0010] Based on the cost of each state step in the predicted trajectory, the discounted total cost of the predicted trajectory is calculated using the discounted total cost calculation method;

[0011] Determine a safe action set according to the discounted total cost of the predicted trajectory corresponding to each action; the safe action set is a set of actions whose discounted total cost satisfies the discounted total cost threshold constraint;

[0012] Use the value network to calculate the value of each action in the action space;

[0013] According to the value of each action, determine the original decision action of the agent. If the original decision action is in the safe action set, or the safe action set is empty, output the original decision action; if the original decision action is not in the safe action set, output the action with the largest value in the safe action set;

[0014] The training reinforcement learning is fed into the decision-making model, and the optimal strategy is obtained after convergence.

[0015] The present application also discloses a safe reinforcement learning intelligent vehicle merging decision system based on predicted trajectory, the system comprising:

[0016] The model building module is used to build a reinforcement learning merging decision model, define the state space and action space of the agent, and design a reward function that considers merging success and traffic efficiency, as well as a cost function that considers collision and safety;

[0017] The cost calculation module is used to obtain the predicted trajectory corresponding to each action in the action space using the pre-built prediction model, and calculate the cost of each step state in the predicted trajectory based on the cost function;

[0018] The cost calculation module is further used to calculate the discounted total cost of the predicted trajectory based on the cost of each step state in the predicted trajectory using a discounted total cost calculation method;

[0019] A safety action constraint module is used to determine a safety action set according to the discounted total cost of the predicted trajectory corresponding to each action; the safety action set is a set of actions whose discounted total cost satisfies the discounted total cost threshold constraint;

[0020] The value evaluation module is used to calculate the value of each action in the action space using the value network;

[0021] The safety action output module is used to determine the original decision action of the agent according to the value of each action. If the original decision action is in the safety action set or the safety action set is empty, the original decision action is output; if the original decision action is not in the safety action set, the action with the largest value in the safety action set is output;

[0022] The model training module is used to train the reinforcement learning input decision model and obtain the optimal strategy after convergence.

[0023] The present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, one or more methods described in the present application are implemented.

[0024] The present application also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, one or more methods described in the present application are implemented.

[0025] In this application, a reinforcement learning influx decision model is constructed, the state space and action space of the agent are defined, and a reward function that considers the success of the influx and traffic efficiency, as well as a cost function that considers collision and safety, is designed. The prediction model is used to obtain the predicted trajectory corresponding to each action in the action space, and the cost of each step state in the predicted trajectory is calculated based on the cost function. Based on the cost of each step state in the predicted trajectory, the discounted total cost of the predicted trajectory is calculated using the discounted total cost calculation method, and then the safe action set is determined according to the discounted total cost of the predicted trajectory corresponding to each action. According to whether the original decision action of the agent is in the safe action set, if it is in the safe action set, or the safe action set is empty, the original decision action is output; if it is not in the safe action set, the action with the highest value in the safe action set is output. In this application, the predicted trajectory is integrated into safe reinforcement learning, and the number of collisions in the training process is effectively reduced by replacing unsafe actions with safe actions, thereby shortening the training cycle, reducing training costs, and improving the safety of influx decisions.

[0026] In addition, the decision-making in this application takes trajectory prediction into consideration, realizes the combination of prediction layer and decision-making layer, and makes the decision-making take into account the evolution of future traffic situation, which has great potential in autonomous driving safety decision-making. In addition, this application takes into account the unsafe factors of predicted trajectory.

[0027] Additional aspects and advantages of the present application will be given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0029] Figure 1 A flowchart of a smart car merging decision-making method based on safety reinforcement learning of predicted trajectories provided in an embodiment of the present application;

[0030] Figure 2 A schematic diagram of a ramp merging environment for a two-lane merging area provided in an embodiment of the present application;

[0031] Figure 3 A schematic diagram of a straight trajectory prediction method provided in an embodiment of the present application;

[0032] Figure 4 A schematic diagram of lane change trajectory prediction of a trajectory prediction method provided in an embodiment of the present application;

[0033] Figure 5 A schematic diagram of the accumulated cost of each round in the training process of the method provided in the embodiment of the present application and the basic input decision algorithm;

[0034] Figure 6 A schematic diagram of the cumulative rewards of each round in the training process of the basic import decision algorithm provided in an embodiment of the present application;

[0035] Figure 7 A schematic diagram of the structure of a smart car integration decision system based on safety reinforcement learning of predicted trajectories provided in an embodiment of the present application;

[0036] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0038] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0039] Those skilled in the art will appreciate that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0040] The solution provided in the embodiments of the present application can be executed by any electronic device, such as a terminal device or a server, wherein the server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this. With regard to the technical problems existing in the prior art, the smart car entry decision method and system based on safe reinforcement learning of predicted trajectories provided in this application is intended to solve at least one of the technical problems of the prior art.

[0041] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0042] The merging area ramp merging environment consists of a main road and a ramp. The main road is a two-lane road. The vehicle is an intelligent connected car, which can obtain the status information of the vehicle and surrounding vehicles through vehicle wireless communication technology (V2X) and sensors. The vehicle needs to consider the surrounding traffic situation and complete the merging at the optimal merging time before driving to the merging point. Stopping is not allowed during the period. The vehicle should drive as fast as possible while completing the merging without collision.

[0043] In the embodiment of the present application, a safe reinforcement learning intelligent vehicle merging decision method based on predicted trajectory is proposed for the merging environment in the merging area, the intersection of the ramp and the main road is set as the merging point, and a certain range thereof is set as the control area.

[0044] In the control area, the intelligent agent obtains vehicle status information; based on the vehicle status information, reinforcement learning is used to import the decision model to make discrete high-level behavior decisions, thereby improving the safety of reinforcement learning training.

[0045] like Figure 1 As shown in , the method may include the following steps:

[0046] Step 101, construct a reinforcement learning merging decision model, define the state space and action space of the agent, and design a reward function that considers merging success and traffic efficiency, as well as a cost function that considers collision and safety.

[0047] The reinforcement learning import decision model in the embodiment of the present application can be constructed based on any reinforcement learning algorithm, such as the deep Q network (DQN). The embodiment of the present application constructs the reinforcement learning import decision model based on the DQN algorithm.

[0048] Step 102: Use the pre-built prediction model to obtain the predicted trajectory corresponding to each action in the action space, and calculate the cost of each step state in the predicted trajectory based on the cost function.

[0049] Step 103: Based on the cost of each state step in the predicted trajectory, the discounted total cost of the predicted trajectory is calculated using a discounted total cost calculation method.

[0050] In the embodiment of the present application, since the cost function takes collision and safety into consideration, the discounted total cost of the action obtained takes safety into consideration.

[0051] Step 104 , determining a safe action set according to the discounted total cost of the predicted trajectory corresponding to each action; the safe action set is a set of actions whose discounted total cost satisfies the discounted total cost threshold constraint.

[0052] The predicted trajectory can reflect the evolution of future traffic conditions. By calculating the discounted total cost of the predicted trajectory and taking unsafe factors into account, the safe action set is determined. It can be considered that when executing the actions in the safe action set, it is safe in the predicted future period.

[0053] Step 105, using the value network to calculate the value of each action in the action space;

[0054] Step 106, determine the original decision action of the intelligent agent according to the value of each action. If the original decision action is in the safe action set, or the safe action set is empty, output the original decision action; if the original decision action is not in the safe action set, output the action with the greatest value in the safe action set.

[0055] In the embodiment of the present application, security can be improved by judging whether the original decision action of the intelligent agent belongs to the safe action set, processing the original decision action according to the above method, and outputting a relatively safe action.

[0056] Step 107, training reinforcement learning is fed into the decision model, and the optimal strategy is obtained after convergence.

[0057] In the embodiment of the present application, a reinforcement learning influx decision model is constructed, the state space and action space of the agent are defined, and a reward function that considers the success of the influx and traffic efficiency, as well as a cost function that considers collision and safety, is designed. The prediction model is used to obtain the predicted trajectory corresponding to each action in the action space, and the cost of each step state in the predicted trajectory is calculated based on the cost function; based on the cost of each step state in the predicted trajectory, the discounted total cost calculation method is used to calculate the discounted total cost of the predicted trajectory, and then the safe action set is determined according to the discounted total cost of the predicted trajectory corresponding to each action. According to whether the original decision action of the agent is in the safe action set, if it is in the safe action set, or the safe action set is empty, the original decision action is output; if it is not in the safe action set, the action with the highest value in the safe action set is output. In the embodiment of the present application, the predicted trajectory is integrated into safe reinforcement learning, and the number of collisions in the training process is effectively reduced by replacing unsafe actions with safe actions, thereby shortening the training cycle, reducing training costs, and improving the safety of influx decisions.

[0058] In an optional embodiment, before constructing a reinforcement learning merging decision model, defining the state space and action space of the agent, and designing a reward function that considers merging success and traffic efficiency, and a cost function that considers collision and safety, the method further includes:

[0059] Build an autonomous driving simulation environment for merging into the merging area ramp, and obtain current status information based on the autonomous driving simulation environment for merging into the merging area ramp;

[0060] The current status information includes the vehicle status information and surrounding vehicle status information. The surrounding vehicle status information is the status information of other vehicles around the vehicle within the sensing range.

[0061] In the embodiment of the present application, a simulation environment for merging into the merging area is built based on the highway_env simulator, specifically based on the highway_env1.5 version. The number of lanes and the location of the merging point are set; the initial state of the intelligent agent is set and manually driven vehicles are added, and different traffic flow densities are defined; each intelligent agent can obtain the state information of the vehicle and the surrounding vehicles within the perception range through the simulation environment.

[0062] For example, the lane information of the simulated merging environment is set as follows: Figure 2 As shown, the total length of the lane is 350m, of which the merging point is located at 310m; the lane width is 4 meters, the main road is a double lane, the ramp is a single lane, and the shortest lateral distance between the main road and the ramp is 10.5m;

[0063] The total number of vehicles (Vehicle_env) is 5, with an initial speed range of 18-22m / s. They are randomly distributed in two lanes, and the initial spacing between two vehicles in the same lane is not less than 20m.

[0064] Set the vertical decision model of Zhouche to IDM model and the horizontal decision model to MOBIL model;

[0065] When the ego vehicle (Vehicle_ego) collides, or fails to merge successfully after driving to the end of the ramp, or successfully merges and leaves the control area, the simulation is considered to be over. The end of a simulation can be regarded as a round of training. Once the simulation is over, the environment is terminated. To simulate again, the environment and vehicle must be regenerated.

[0066] The refresh frequency of the simulation environment is 50Hz, that is, the environment is refreshed 50 times in 1 second, ensuring that the state information in the environment is updated in real time; the frequency of the agent's decision-making is 2Hz, that is, the agent can make two decisions in 1 second;

[0067] After the agent makes a decision, the underlying control model executes the decision command, which is a vertical IDM combined with a horizontal MOBIL model;

[0068] Figure 2 The part shown in the figure is the control area. When the smart car enters the control area, the reinforcement learning decision model takes over the control, and when it leaves the control area, the control ends.

[0069] In an optional embodiment, defining the state space and action space of the agent includes:

[0070] The state space S of the intelligent agent is defined as the state information of the vehicle itself and the state information of the observable surrounding vehicles within the perception range.

[0071] S is a matrix of shape N×F, where N is the number of vehicles observed by the agent (including itself), F is the number of state features of each vehicle, and the feature vector of the ego vehicle, i.e. the 0th row of S, is expressed as:

[0072]

[0073] Among them, exist0 represents whether the vehicle is within the observation range of the ego vehicle. Here, it is the ego vehicle, so this quantity is always 1; represents the longitudinal position coordinate of the vehicle, Represents the horizontal position coordinate; represents the longitudinal velocity of the vehicle, Represents the lateral velocity; cosθ0 represents the cosine value of the x-coordinate direction of the vehicle's heading angle.

[0074] The eigenvector of Zhouche i, i.e., the i-th row of S, is expressed as:

[0075]

[0076] Among them, exist i Indicates whether the pericycle is within the observation range; represents the longitudinal position coordinate of the pericycle i, Represents the horizontal position coordinate; represents the longitudinal velocity of the circumvehicle i, represents the lateral velocity of the circumvehicle i; cosθ i Represents the x-coordinate direction cosine value of the heading angle of vehicle i in the surrounding environment.

[0077] The complete state space S is defined as the combination of the feature vectors of each vehicle, expressed as:

[0078] S=[S0,S1...S N ]

[0079] Where N represents the upper limit of the total number of observed weekly vehicles.

[0080] The action space A of the agent is defined as discrete high-level behavior decisions, expressed as:

[0081] A={0,1,2,3,4}

[0082] Among them, action "0" represents changing lanes to the left, action "1" represents keeping the lane straight, action "2" represents changing lanes to the right, action "3" represents accelerating, and action "4" represents decelerating.

[0083] In an optional embodiment, the reward function is:

[0084]

[0085] Among them, r is the reward obtained by the agent when it outputs action a in state s, β action , β effi They are the sparse rewards for successful remittance Rewards for successful remittance Action Reward action and traffic efficiency reward r effi The weight coefficient of .

[0086] Sinking Sparse Rewards

[0087]

[0088] Successful remittance intensive rewards

[0089]

[0090] Action Reward action :

[0091]

[0092] Traffic efficiency reward effi :

[0093]

[0094] Where v represents the current vehicle speed, v high Represents the speed threshold for high-speed driving, v low Speed ​​threshold representing low speed driving.

[0095] In an optional embodiment, the cost function is:

[0096] c=β TTC ×c TTC +β TDTM ×c TDTM +β crash ×c crash

[0097] Among them, c is the cost obtained when the agent outputs action a in state s, β TTC , β TDTM , β crash They are respectively the collision distance cost c TTC 、Import time difference cost c TDTM and collision cost c crash The weight coefficient of .

[0098] Collision time cost c:

[0099] TTC

[0100]

[0101] Where TTC is the time to collision (if there is a preceding vehicle in the same lane), which is defined as follows:

[0102]

[0103] in, Represents the longitudinal position coordinate of the closest preceding vehicle in the same lane as the vehicle itself, represents the longitudinal position coordinate of the vehicle, Represents the longitudinal speed of the closest preceding vehicle in the same lane as the vehicle itself, represents the longitudinal velocity of the vehicle;

[0104] Import time difference cost c TDTM :

[0105]

[0106] Where TDTM is the time to collision (if there is a preceding vehicle in the same lane), which is defined as follows:

[0107]

[0108] in, Represents the vertical position coordinate of the entry point. Represents the longitudinal position coordinate of the vehicle in front of the vehicle in the same lane. represents the longitudinal position coordinate of the vehicle, represents the longitudinal speed of the vehicle in front of the vehicle in the same lane, represents the longitudinal velocity of the vehicle;

[0109] Collision cost c crash :

[0110]

[0111] In an optional embodiment, a pre-built prediction model is used to obtain a prediction trajectory corresponding to each action in the action space, and the cost of each step state in the prediction trajectory is calculated based on a cost function, including:

[0112] Input the current state information into the prediction model, and output the predicted trajectory corresponding to each action in the action space; the predicted trajectory includes the state of multiple steps in the future period of time;

[0113] The cost function is used to calculate the cost of each state in the predicted trajectory.

[0114] In an optional embodiment, the discounted total cost calculation method includes:

[0115]

[0116] Among them, c future_based is the discounted total cost of the predicted trajectory, k is the number of steps of the predicted point, K is the total number of steps of the predicted trajectory, and β future is the trajectory cost discount factor, c k is the cost of the state corresponding to the predicted point in the kth step.

[0117] In the embodiment of the present application, considering that the impact of unsafe factors decays with time, the cost weight of each step state in the predicted trajectory is allocated according to the exponential decay property. Then, the cost of each state is weighted based on the cost weight of each state to obtain the discounted total cost of the predicted trajectory. Each action in the action space corresponds to a predicted trajectory, so each action in the action space obtains the corresponding discounted total cost.

[0118] In an optional embodiment, the security action set can be expressed as:

[0119]

[0120] Among them, among them, represents the set of safe actions for state s, represents the discounted total cost of taking action a corresponding to the predicted trajectory, c safe is the discounted total cost threshold, and is the set hyperparameter.

[0121] According to the above formula, for each action in the action space, the corresponding discounted total cost is less than or equal to the discounted total cost threshold c safe The actions generate a safe action set.

[0122] In an optional embodiment, the parameters of the value network include input dimension, number and type of hidden layers, output dimension and activation function.

[0123] Among them, the input dimension of the value network is the same as the number of elements in the state s. Assuming there are 6 cars in total, based on the 6 elements defined in the state space, each car has 6 features, so the input dimension of the value network is 36.

[0124] In order to increase the representation ability of the value network, the number of hidden layers in the embodiment of the present application is 3, the number of neurons in each layer is 256, and the type of each layer is a fully connected layer.

[0125] The output of the value network is the value of each action in the action space, so the output dimension is the number of elements in the action space A, which is 5 here;

[0126] In order to enhance the nonlinear representation capability of the value network, an activation function is designed between every two layers of the value network. The activation function in the embodiment of the present application may be a tanh function.

[0127] In the embodiment of the present application, the original decision action of the intelligent agent is determined according to the value of each action, and then it is judged whether the original decision action is in the safe action set of the current state, that is, it is judged whether it is safe to take the current decision action.

[0128] If the original action is in the safe action set of the current state, that is, it is safe to take the current decision action, then output the original decision action;

[0129] If the original action is not in the safe action set of the current state, that is, it is unsafe to take the current decision action, then action replacement is performed to replace the original action with a safe action. Specifically, the action with the highest value in the safe action set is used to replace the original action for output;

[0130] If the safe action set is empty, output the original decision action.

[0131] To increase the exploratory nature, the actions output by the above process are greedily explored and the final execution actions are output. The specific method is as follows:

[0132]

[0133] Among them, a final represents the final action to be executed, a represents the secure action after security, random unif represents the generated random number in the range of 0 to 1, epsilon represents the greedy exploration rate, and a random Represents a randomly selected action.

[0134] In an optional embodiment, training reinforcement learning is integrated into the decision model, and obtaining the optimal strategy after convergence includes:

[0135] Taking the expectation of maximizing the total return under the cost constraint as the optimization goal, training reinforcement learning is integrated into the decision model to converge to the optimal strategy;

[0136] Among them, the total return is the difference between the reward and the cost. The reward is obtained through the reward function, and the cost is obtained through the cost function.

[0137] The prediction model in the embodiment of the present application uses a rule-based prediction method, the prediction step length is not less than 2 time steps, and the prediction step length of all vehicles should remain the same. The specific prediction rules are as follows:

[0138] (1) For the ego vehicle, a constant acceleration prediction is used to predict the position and velocity of the ego vehicle at the next time step. During the prediction, the actions of the second time step and subsequent time steps are considered to keep the lane straight.

[0139] (2) For the pericycle, constant speed prediction is used to predict the position and speed of the next time step at a constant speed, without involving decision making;

[0140] (3) Each vehicle can only change lanes once in the predicted trajectory;

[0141] When the vehicle is moving straight, the predicted position and speed are calculated according to the above rules. Figure 3 The scene 1 shown is a schematic diagram showing the first time step t=0, the second time step t=1, and the third time step t=3 when the vehicle is moving straight.

[0142] Figure 4 The scene 2 shown in FIG. 2 shows schematic diagrams of the first time step t=0, the second time step t=1, and the third time step t=3 during the vehicle lane change process. Figure 4 As shown in , the vehicle is currently changing lanes. The predicted position and speed are considered that the vehicle is changing lanes only once, and will keep going straight after the lane change is completed. Figure 4 In the scenario 2 shown, the lane is changed to the right at the first time step t=0, and the speed direction is to the right during the right lane change. The lane change is completed at the second time step t=1, and the lane is kept straight until the third time step t=3, and the speed direction is forward during the straight lane.

[0143] The above rule-based prediction model needs to provide the predicted trajectory generated by each action a under a given state s in real time.

[0144] The training process is to merge into the autonomous driving simulation environment through the built merging area ramp, and train the prediction model and reinforcement learning into the decision model; the intelligent agent outputs the original decision action by obtaining the state in the environment, and then makes it safe through the above process, executes the final action and obtains feedback and the state of the next moment from the environment; and continuously updates the value network until it converges completely.

[0145] The reinforcement learning input decision model in the embodiment of the present application is provided with two value networks, namely the main value network and the target value network, which are used to evaluate the value of the action, design an experience revisit pool, and adopt a delayed update method for optimization and update; it continuously iterates and updates with the expectation of maximizing the total return as the optimization goal, and finally converges to obtain the optimal strategy;

[0146] The specific process of training in the embodiment of the present application is as follows:

[0147] (1) Initialize the main value network Q with random network parameters θθ (s,a);

[0148] (2) Copy the same parameters θ' and initialize the target value network Q θ' (s,a);

[0149] (3) Initialize the experience replay pool with a maximum capacity of 100 and a sampling batch size of 80, that is, 80 groups of samples are extracted for each update;

[0150] (4) Obtain the current state s from the simulation environment t , which contains the position, attitude and speed information of the ego vehicle and surrounding vehicles;

[0151] (5) Based on the current main value network Q θ (s,a) Get the value of each action;

[0152] (6) Using the above trajectory prediction model, predict the trajectories of the ego vehicle and surrounding vehicles within a certain time domain;

[0153] (7) Calculate the discounted total cost of each action using the above discounted total cost calculation method and determine the safe action set under the current state;

[0154] (8) According to the main value network Q θ (s, a) outputs the value, selects the original decision action; and performs corresponding processing according to whether it is in the safe action set, greedily explores the secured action, and obtains the final execution action;

[0155] (9) Execute the final action and obtain reward feedback from the environment to obtain the state s of the next time step t+1 ; will (s t ,a final ,(rc),s t+1 ) The four-tuple is stored in the experience replay pool; if a simulation has not terminated at this time, the state is updated and continued until a simulation terminates;

[0156] After the above process is completed, if there are enough samples in the experience replay pool, 80 groups of samples are drawn from it, which can be expressed as:

[0157]

[0158] The decay rate is 0.9, and for each set of data, the target value network is used to calculate the temporal difference residual:

[0159]

[0160] The loss function is calculated:

[0161]

[0162] Use the gradient descent method to minimize the loss function L and update the current main value network Q θ (s, a); update the main value network every C times, and update the target value network synchronously, where C is 10; the initial learning rate of the update is 0.001, which decays linearly to 0 during the training process; the training process lasts for a maximum of 20,000 times;

[0163] The above training process is repeated until the reinforcement learning-integrated decision model is fully converged or the maximum number of training times is reached.

[0164] Among them, the parameters used in the training process are shown in Table 1 below:

[0165] Table 1

[0166]

[0167]

[0168] When training the reinforcement learning import decision model in the embodiment of the present application, the cumulative cost and cumulative reward of each training are recorded; in addition, the basic import decision algorithm without step 102, step 103 and step 105 in the above method is compared with the method in the embodiment of the present application:

[0169] The method in the embodiment of the present application and the cost of the basic input decision algorithm are as follows: Figure 5 As shown. Among them, DQN-baseline represents the basic DQN input decision algorithm, and DQN-safety represents the method in the embodiment of the present application. Figure 5 It can be seen that the cost obtained by the method in the embodiment of the present application is lower than the cost of the basic DQN input decision algorithm.

[0170] The method in the embodiment of the present application and the basic DQN input decision algorithm return Return, that is, the difference between the reward and the cost, are as follows Figure 6 As shown. Among them, DQN-baseline represents the basic DQN input decision algorithm, and DQN-safety represents the method in the embodiment of the present application. Figure 6 It can be seen that the reward of the method in the embodiment of the present application is higher than the reward of the basic DQN input decision algorithm.

[0171] The method in the embodiment of the present application has a total collision number of 6154 during the training process, and the basic DQN merged into the decision algorithm has a total collision number of 8862. By comparison, the method in the embodiment of the present application can significantly reduce the number of collisions during the training process and reduce the training cost.

[0172] The trained models were tested 100 times respectively. The method in the embodiment of the present application successfully imported 85 times, and the basic DQN imported 72 times into the decision algorithm. Therefore, the method in the embodiment of the present application improves the security of decision-making.

[0173] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides a smart car entry decision system based on safety reinforcement learning of predicted trajectories, such as Figure 7 As shown, the system comprises:

[0174] The model building module 701 is used to build a reinforcement learning merging decision model, define the state space and action space of the agent, and design a reward function that considers the merging success and traffic efficiency, as well as a cost function that considers collision and safety;

[0175] The cost calculation module 702 is used to obtain the predicted trajectory corresponding to each action in the action space by using the pre-built prediction model, and calculate the cost of each step state in the predicted trajectory based on the cost function;

[0176] The cost calculation module 702 is further used to calculate the discounted total cost of the predicted trajectory based on the cost of each step state in the predicted trajectory using a discounted total cost calculation method;

[0177] The safety action constraint module 703 is used to determine a safety action set according to the discounted total cost of the predicted trajectory corresponding to each action; the safety action set is a set of actions whose discounted total cost satisfies the discounted total cost threshold constraint;

[0178] A value evaluation module 704, for calculating the value of each action in the action space using a value network;

[0179] The safety action output module 705 is used to determine the original decision action of the agent according to the value of each action. If the original decision action is in the safety action set or the safety action set is empty, the original decision action is output; if the original decision action is not in the safety action set, the action with the greatest value in the safety action set is output;

[0180] The model training module 706 is used to train the reinforcement learning input decision model and obtain the optimal strategy after convergence.

[0181] In the embodiment of the present application, a reinforcement learning inflow decision model is constructed, the state space and action space of the agent are defined, and a reward function that considers the success of the inflow and traffic efficiency, as well as a cost function that considers collision and safety, is designed. The prediction model is used to obtain the predicted trajectory corresponding to each action in the action space, and the cost of each step state in the predicted trajectory is calculated based on the cost function. Based on the cost of each step state in the predicted trajectory, the discounted total cost of the predicted trajectory is calculated using the discounted total cost calculation method, and then the safe action set is determined according to the discounted total cost of the predicted trajectory corresponding to each action. According to whether the original decision action of the agent is in the safe action set, if it is in the safe action set, or the safe action set is empty, the original decision action is output; if it is not in the safe action set, the action with the highest value in the safe action set is output. In the embodiment of the present application, the predicted trajectory is integrated into safe reinforcement learning, and the number of collisions in the training process is effectively reduced by replacing unsafe actions with safe actions, thereby shortening the training cycle, reducing training costs, and improving the safety of inflow decisions.

[0182] The smart car integration decision system based on safety reinforcement learning of predicted trajectories provided in the embodiment of the present application can achieve Figures 1 to 6 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0183] The safety reinforcement learning smart car integration decision system based on predicted trajectory of the embodiment of the present application can execute the safety reinforcement learning smart car integration decision method based on predicted trajectory provided by the embodiment of the present application, and the implementation principle is similar. The actions performed by each module and unit in the safety reinforcement learning smart car integration decision system based on predicted trajectory in each embodiment of the present application correspond to the steps in the safety reinforcement learning smart car integration decision method based on predicted trajectory in each embodiment of the present application. For the detailed functional description of each module of the safety reinforcement learning smart car integration decision system based on predicted trajectory, please refer to the description of the corresponding safety reinforcement learning smart car integration decision method based on predicted trajectory shown in the previous text, which will not be repeated here.

[0184] Based on the same principle as the method shown in the embodiment of the present application, the embodiment of the present application also provides an electronic device, which may include but is not limited to: a processor and a memory; a memory for storing a computer program; a processor for executing the safe reinforcement learning intelligent car merging decision method based on the predicted trajectory shown in any optional embodiment of the present application by calling a computer program. Compared with the prior art, the safe reinforcement learning intelligent car merging decision method based on the predicted trajectory provided by the present application constructs a reinforcement learning merging decision model, defines the state space and action space of the intelligent agent, and designs a reward function that considers the merging success and traffic efficiency, as well as a cost function that considers collision and safety. The predicted trajectory corresponding to each action in the action space is obtained by using the prediction model, and the cost of each step state in the predicted trajectory is calculated based on the cost function. Based on the cost of each step state in the predicted trajectory, the discounted total cost of the predicted trajectory is calculated using the discounted total cost calculation method, and then the safe action set is determined according to the discounted total cost of the predicted trajectory corresponding to each action. According to whether the original decision action of the intelligent agent is in the safe action set, if it is in the safe action set, or the safe action set is empty, the original decision action is output; if it is not in the safe action set, the action with the highest value in the safe action set is output. In the embodiment of the present application, the predicted trajectory is integrated into safety reinforcement learning, and the number of collisions during the training process is effectively reduced by replacing unsafe actions with safe actions, thereby shortening the training cycle, reducing training costs, and improving the safety of input decisions.

[0185] In an optional embodiment, an electronic device is also provided, such as Figure 8 As shown, Figure 8 The electronic device 800 shown may be a server, including: a processor 801 and a memory 803. The processor 801 and the memory 803 are connected, such as through a bus 802. Optionally, the electronic device 800 may further include a transceiver 804. It should be noted that in actual applications, the transceiver 804 is not limited to one, and the structure of the electronic device 800 does not constitute a limitation on the embodiments of the present application.

[0186] The processor 801 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 801 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0187] The bus 802 may include a path to transmit information between the above components. The bus 802 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 802 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0188] The memory 803 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0189] The memory 803 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 801. The processor 801 is used to execute the application code stored in the memory 803 to implement the content shown in the above method embodiment.

[0190] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0191] The server provided in this application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this.

[0192] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment.

[0193] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0194] It should be noted that the computer-readable storage medium mentioned above in the present application can also be a computer-readable signal medium or a combination of a computer-readable storage medium and a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0195] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0196] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0197] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the safe reinforcement learning intelligent vehicle merging decision method and system based on predicted trajectory provided in the above-mentioned various optional implementations.

[0198] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0199] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0200] The modules involved in the embodiments described in this application can be implemented by software or hardware. The name of the module does not limit the module itself in some cases. For example, the model building module can also be described as "a model building module for building a reinforcement learning merging decision model, defining the state space and action space of the agent, and designing a reward function that considers the merging success and traffic efficiency, as well as a cost function that considers collision and safety."

[0201] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. A safe reinforcement learning intelligent vehicle merging decision method based on predicted trajectory, characterized in that: The method comprises: Construct a reinforcement learning merging decision model, define the state space and action space of the agent, and design a reward function that considers merging success and traffic efficiency, as well as a cost function that considers collision and safety; Using a pre-built prediction model to obtain a prediction trajectory corresponding to each action in the action space, and calculating the cost of each step state in the prediction trajectory based on the cost function; Based on the cost of each step state in the predicted trajectory, a discounted total cost of the predicted trajectory is calculated using a discounted total cost calculation method; Determine a safe action set according to the discounted total cost of the predicted trajectory corresponding to each action; the safe action set is a set of actions whose discounted total cost satisfies a discounted total cost threshold constraint; Calculate the value of each action in the action space using a value network; According to the value of each action, the original decision action of the intelligent agent is determined. If the original decision action is in the safe action set, or the safe action set is empty, the original decision action is output; if the original decision action is not in the safe action set, the action with the largest value in the safe action set is output; The reinforcement learning training is integrated into the decision model, and the optimal strategy is obtained after convergence.

2. The method for intelligent vehicle merging decision-making based on safe reinforcement learning of predicted trajectories according to claim 1 is characterized in that: Before constructing the reinforcement learning merging decision model, defining the state space and action space of the intelligent agent, and designing a reward function that considers merging success and traffic efficiency, and a cost function that considers collision and safety, the method further includes: Building a merging area ramp merging automatic driving simulation environment, and obtaining current state information based on the merging area ramp merging automatic driving simulation environment; The current status information includes the vehicle status information and surrounding vehicle status information, and the surrounding vehicle status information is status information of other vehicles around the vehicle within the sensing range.

3. The method for intelligent vehicle merging decision-making based on safe reinforcement learning of predicted trajectories according to claim 2 is characterized in that: The method of using the pre-built prediction model to obtain the predicted trajectory corresponding to each action in the action space, and calculating the cost of each step state in the predicted trajectory based on the cost function, includes: Input the current state information into the prediction model, and output the prediction trajectory corresponding to each action in the action space; the prediction trajectory includes the states of multiple steps in the future period of time; The cost function is used to calculate the cost of each step state in the predicted trajectory.

4. The method for intelligent vehicle merging decision-making based on safe reinforcement learning of predicted trajectories according to claim 1 or 3, characterized in that: The cost function is: c=β TTC ×c TTC +b TDTM ×c TDTM +b crash ×c crash Among them, c is the cost obtained when the agent outputs action a in state s, β TTC , β TDTM , β crash are respectively the collision distance cost c TTC 、Import time difference cost c TDTM and collision cost c crash The weight coefficient of .

5. The method for intelligent vehicle merging decision-making based on safe reinforcement learning of predicted trajectories according to claim 1 is characterized in that: The reward function is: Among them, r is the reward obtained by the agent when it outputs action a in state s, β action , β effi They are the sparse rewards for successful remittance Rewards for successful remittance Action Reward action and traffic efficiency reward r effi The weight coefficient of .

6. The method for intelligent vehicle merging decision-making based on safe reinforcement learning of predicted trajectories according to claim 1, characterized in that: The discounted total cost is calculated as follows: Among them, c future_based is the discounted total cost of the predicted trajectory, k is the number of steps of the predicted point, K is the total number of steps of the predicted trajectory, β future is the trajectory cost discount factor, c k is the cost of the state corresponding to the predicted point in the kth step.

7. The method for intelligent vehicle merging decision-making based on safe reinforcement learning of predicted trajectories according to claim 1 is characterized in that: The training reinforcement learning is integrated into the decision model, and the optimal strategy is obtained after convergence, including: Taking maximizing the expectation of total return under the cost constraint as the optimization goal, training the reinforcement learning into the decision model, and converging to obtain the optimal strategy; The total return is the difference between the reward and the cost, the reward is obtained through the reward function, and the cost is obtained through the cost function.

8. A safe reinforcement learning intelligent vehicle integration decision system based on predicted trajectory, characterized in that: The system comprises: The model building module is used to build a reinforcement learning merging decision model, define the state space and action space of the agent, and design a reward function that considers merging success and traffic efficiency, as well as a cost function that considers collision and safety; A cost calculation module, used to obtain the predicted trajectory corresponding to each action in the action space by using a pre-built prediction model, and the cost function calculates the cost of each step state in the predicted trajectory; The cost calculation module is further used to calculate the discounted total cost of the predicted trajectory based on the cost of each step state in the predicted trajectory using a discounted total cost calculation method; A safety action constraint module, used to determine a safety action set according to the discounted total cost of the predicted trajectory corresponding to each action; the safety action set is a set of actions whose discounted total cost satisfies a discounted total cost threshold constraint; A value evaluation module, used for calculating the value of each action in the action space by using a value network; A safety action output module is used to determine the original decision action of the intelligent agent according to the value of each action, and output the original decision action if the original decision action is in the safety action set or the safety action set is empty; if the original decision action is not in the safety action set, output the action with the greatest value in the safety action set; The model training module is used to train the reinforcement learning input decision model and obtain the optimal strategy after convergence.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Intelligent agent behavior decision method, device and electronic equipment

    CN119740602A