An automatic driving decision-making method and system based on risk-sensitive reinforcement learning, a terminal and a storage medium

Abstract

The application relates to the technical field of machine learning and automatic driving, and discloses an automatic driving decision-making method and system based on risk-sensitive reinforcement learning, a terminal and a storage medium.The method comprises the following steps: a generalized risk function corresponding to the risk of automatic driving is constructed, the negative value of the cumulative cost in the generalized risk function is taken as an augmented state, an original state space of automatic driving is obtained, an augmented state space is constructed according to the augmented state and the original state space, a risk-sensitive driving strategy model is constructed, the risk-sensitive driving strategy model is optimized based on an augmented state space vector, and a target model is obtained; a current augmented state space vector is constructed according to a current original state space and a current augmented state, the current augmented state space vector is input into the target model, a control instruction is output, and automatic driving control is completed according to the control instruction.The application guarantees the time consistency of the decision-making process, significantly improves the driving safety, and effectively avoids high-risk events such as collisions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning and automatic driving, and particularly relates to an automatic driving decision method and system based on risk-sensitive reinforcement learning, a terminal and a storage medium. BACKGROUND

[0002] In an automatic driving system, an intelligent vehicle needs to learn a safe driving strategy through continuous interaction with a complex traffic environment. Traditional reinforcement learning algorithms usually aim to maximize the expected value of cumulative rewards (such as travel efficiency and passenger comfort), which reflects a risk-neutral decision-making attitude. However, in real road driving scenarios, the decision-maker (automatic driving system) must exhibit a strong risk-averse tendency, because extreme adverse events such as collisions, violations, and sudden obstacles can lead to disastrous consequences. Therefore, simply optimizing expected performance is not enough. A good decision under a risk-neutral attitude may be accompanied by extremely high variance or tail risk of serious loss, which is unacceptable in practice.

[0003] To solve this problem, risk-sensitive reinforcement learning has emerged. Its core idea is to integrate the risk preference of the decision-maker into the learning process, thereby learning a strategy that can avoid high-risk events. However, it faces a fundamental challenge: time inconsistency. This inconsistency makes standard dynamic programming and Bellman equations cannot be directly applied, making policy evaluation and optimization extremely difficult.

[0004] However, to solve the problem of time inconsistency, existing methods are usually designed for specific risk measures, cannot uniformly handle multiple risk measures, and are computationally inefficient, resulting in traditional reinforcement learning methods being unable to effectively avoid high-risk events in automatic driving systems.

[0005] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0006] The main purpose of the present application is to provide an automatic driving decision method and system based on risk-sensitive reinforcement learning, a terminal and a computer readable storage medium, which aims to solve the problem that the existing automatic driving strategy is designed for specific risk measures, cannot uniformly handle multiple risk measures, and is computationally inefficient, resulting in traditional reinforcement learning methods being unable to effectively avoid high-risk events in automatic driving systems.

[0007] To achieve the above purpose, the present application provides an automatic driving decision method based on risk-sensitive reinforcement learning, which comprises the following steps:

[0008] Construct a generalized risk function corresponding to autonomous driving risks, take the negative value of the cumulative cost in the generalized risk function as the augmented state, obtain the original state space of autonomous driving, and construct an augmented state space vector based on the augmented state and the original state space.

[0009] A risk-sensitive driving strategy model is constructed, and the risk-sensitive driving strategy model is subjected to inner-layer dynamic optimization and outer-layer static optimization based on the augmented state space vector to obtain the target model.

[0010] Obtain the current original state space and the current augmented state, construct the current augmented state space vector based on the current original state space and the current augmented state, input the current augmented state space vector into the target model, output control commands, and complete autonomous driving control according to the control commands.

[0011] Construct a generalized risk function corresponding to autonomous driving risks, take the negative value of the cumulative cost in the generalized risk function as the augmented state, obtain the original state space of autonomous driving, and construct an augmented state space based on the augmented state and the original state space.

[0012] A risk-sensitive driving strategy model is constructed. The risk-sensitive driving strategy model is dynamically optimized at the inner layer based on the augmented state space vector. After the inner layer dynamic optimization is completed, the outer layer static optimization is performed to obtain the target model.

[0013] Obtain the current original state space and the current augmented state, construct the current augmented state space vector based on the current original state space and the current augmented state, input the current augmented state space vector into the target model, output control commands, and complete autonomous driving control according to the control commands.

[0014] Optionally, the autonomous driving decision-making method based on risk-sensitive reinforcement learning, wherein the step of using the negative value of the cumulative cost in the generalized risk function as the augmented state to obtain the original state space of the autonomous driving system, and constructing an augmented state space vector based on the augmented state and the original state, specifically includes:

[0015] Obtain the negative value of the cumulative cost in the generalized risk function, and define the negative value of the cumulative cost in the generalized risk function as an augmented state:

[0016] ;

[0017] in, Indicates an augmented state. This represents the initial state. Indicates the current moment. Indicates the initial time;

[0018] Obtain the original state space for autonomous driving The original state space includes at least one of the vehicle state, the state of surrounding traffic participants, and the road environment state, which are sensed and fused by vehicle sensors.

[0019] The augmented state and the original state space Together, they form the augmented state space vector. .

[0020] Optionally, in the aforementioned risk-sensitive reinforcement learning-based autonomous driving decision-making method, the cumulative cost is determined by at least one autonomous driving risk event among collision probability, degree of traffic rule violation, magnitude of rapid acceleration, magnitude of rapid acceleration, and magnitude of rapid steering.

[0021] Optionally, in the aforementioned autonomous driving decision-making method based on risk-sensitive reinforcement learning, the risk-sensitive driving strategy model is a two-stage network composed of an Actor network and a Critic network.

[0022] Optionally, the autonomous driving decision-making method based on risk-sensitive reinforcement learning, wherein the inner-layer dynamic optimization of the risk-sensitive driving strategy model based on the augmented state space vector specifically includes:

[0023] Under the current strategy and fixed auxiliary variables Under the condition of the augmented state space vector, The estimated value of the Critic network is calculated:

[0024] ;

[0025] in, This represents the estimated value of the Critic network. This indicates the expected outcome under the current strategy. Representation Strategy The parameters, Indicates future accumulated costs. Indicates from the current moment To the final moment Future accumulated costs This represents the total cumulative cost from the initial time to the final time. This represents a convex scoring function used to quantify the total cumulative cost relative to the fixed auxiliary variable. Deviation between;

[0026] Based on the estimated value, the Critic network is updated through temporal difference learning until the estimated value approaches the Bellman target;

[0027] Under the current strategy Under the condition of the augmented state space vector, Calculate the policy gradient used to update the Actor network:

[0028] ;

[0029] in, This represents the policy gradient used to update the Actor network. The scoring function represents the policy, used to measure the policy parameters in the current state. Changes in the selection of the current action The effect of logarithmic probability, The vector in the augmented state space of the Critic network represents... The next state vector ( The assessed advantage value, Indicates the next augmented state. Represents the next primitive state space;

[0030] Based on the policy gradient, the Critic network is updated using an alternating minimization sampling strategy, with dynamic optimization of the inner layers.

[0031] Optionally, the autonomous driving decision-making method based on risk-sensitive reinforcement learning, wherein the step of performing outer-layer static optimization after inner-layer dynamic optimization to obtain the target model specifically includes:

[0032] After the inner dynamic optimization is completed, the fixed auxiliary variable is optimized by stochastic gradient descent with the goal of minimizing the generalized risk function, and the target auxiliary variable is obtained.

[0033] Output the target auxiliary variable, add the target auxiliary variable to the inner layer dynamically optimized risk-sensitive driving strategy model, complete the outer layer static optimization, and obtain the target model.

[0034] Optionally, the autonomous driving decision-making method based on risk-sensitive reinforcement learning, wherein obtaining the current original state space and the current augmented state, constructing a current augmented state space vector based on the current original state space and the current augmented state, inputting the current augmented state space vector into the target model, and outputting control commands specifically includes:

[0035] The current original state space is obtained by vehicle sensors, and the current original state space includes at least one of the current vehicle state, the current state of surrounding traffic participants, and the current road environment state.

[0036] The current augmented state is calculated based on historical cost data. The current original state space is combined with the current augmented state to form the current augmented state space vector.

[0037] The current augmented state space vector is input into the target model for prediction, and control commands are output. The control commands include left turn, right turn, acceleration, deceleration, reversing, or braking.

[0038] Furthermore, to achieve the above objectives, the present invention also provides an autonomous driving decision-making system based on risk-sensitive reinforcement learning, wherein the autonomous driving decision-making system based on risk-sensitive reinforcement learning includes:

[0039] The problem reconstruction module is used to construct a generalized risk function corresponding to the risks of autonomous driving, take the negative value of the cumulative cost in the generalized risk function as the augmented state, obtain the original state space of autonomous driving, and construct an augmented state space vector based on the augmented state and the original state space.

[0040] The model optimization module is used to construct a risk-sensitive driving strategy model. It performs inner-layer dynamic optimization on the risk-sensitive driving strategy model based on the augmented state space vector, and performs outer-layer static optimization after the inner-layer dynamic optimization is completed to obtain the target model.

[0041] The control execution module is used to acquire the current original state space and the current augmented state, construct the current augmented state space vector based on the current original state space and the current augmented state, input the current augmented state space vector into the target model, output control commands, and complete autonomous driving control according to the control commands.

[0042] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an autonomous driving decision-making program based on risk-sensitive reinforcement learning stored in the memory and executable on the processor, wherein when the autonomous driving decision-making program based on risk-sensitive reinforcement learning is executed by the processor, it implements the steps of the autonomous driving decision-making method based on risk-sensitive reinforcement learning as described above.

[0043] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an autonomous driving decision-making program based on risk-sensitive reinforcement learning, and the autonomous driving decision-making program based on risk-sensitive reinforcement learning, when executed by a processor, implements the steps of the autonomous driving decision-making method based on risk-sensitive reinforcement learning as described above.

[0044] In this invention, a generalized risk function corresponding to autonomous driving risks is constructed. The negative value of the cumulative cost in the generalized risk function is used as the augmented state to obtain the original state space of autonomous driving. An augmented state space is constructed based on the augmented state and the original state space. A risk-sensitive driving strategy model is then constructed. The risk-sensitive driving strategy model is optimized based on the augmented state space vector to obtain the target model. The current original state space and the current augmented state are obtained. A current augmented state space vector is constructed based on the current original state space and the current augmented state. The current augmented state space vector is input into the target model, and control commands are output. Autonomous driving control is completed according to the control commands. This invention ensures the temporal consistency of the decision-making process, significantly improves driving safety, and effectively avoids high-risk events such as collisions. Attached Figure Description

[0045] Figure 1 This is a flowchart of a preferred embodiment of the autonomous driving decision-making method based on risk-sensitive reinforcement learning of the present invention;

[0046] Figure 2 This is a flowchart of the two-stage optimization of the risk-sensitive driving strategy model in the autonomous driving decision-making method based on risk-sensitive reinforcement learning of this invention;

[0047] Figure 3 This is a structural diagram of a preferred embodiment of the autonomous driving decision-making system based on risk-sensitive reinforcement learning of the present invention;

[0048] Figure 4 This is a structural diagram of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0049] This application provides an autonomous driving decision-making method, system, terminal, and storage medium based on risk-sensitive reinforcement learning. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0050] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0051] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0052] The preferred embodiment of the autonomous driving decision-making method based on risk-sensitive reinforcement learning described in this invention, such as... Figure 1 As shown, the autonomous driving decision-making method based on risk-sensitive reinforcement learning includes the following steps:

[0053] Step S10: Construct a generalized risk function corresponding to autonomous driving risks, take the negative value of the cumulative cost in the generalized risk function as the augmented state, obtain the original state space of autonomous driving, and construct an augmented state space vector based on the augmented state and the original state space.

[0054] It is understandable that the core of this invention is to transform the risk-sensitive decision-making problem in autonomous driving into a time-consistent optimization problem that can be applied to the principles of dynamic programming through mathematical reconstruction.

[0055] The core challenge in quantifying autonomous driving risks is that risks are not only the probability of a single event, but also the dynamic accumulation of costs over long-term operation. This application first constructs a generalized risk function for autonomous driving with cumulative costs, which couples instantaneous risk events with the cumulative costs over the entire life cycle to form a quantitative model that evolves dynamically over time.

[0056] Furthermore, the step of using the negative value of the cumulative cost in the generalized risk function as the augmented state to obtain the original state space of autonomous driving, and constructing an augmented state space vector based on the augmented state and the original state, specifically includes:

[0057] Obtain the negative value of the cumulative cost in the generalized risk function, and define the negative value of the cumulative cost in the generalized risk function as an augmented state:

[0058] ;

[0059] in, Indicates an augmented state. This represents the initial state. Indicates the current moment. Indicates the initial time;

[0060] Obtain the original state space for autonomous driving The original state space includes at least one of the vehicle state, the state of surrounding traffic participants, and the road environment state, which are sensed and fused by vehicle sensors.

[0061] The augmented state and the original state space Together, they form the augmented state space vector. .

[0062] In this embodiment, the augmented state The design incorporates cumulative costs (such as cumulative collision risk and number of violations), defined as the cost from the initial moment to the current moment. The negative value of the cumulative cost, augmented state As new state variables, they are "augmented" into the original state space. In this process, a new state vector is formed. In this way, decision-makers (autonomous driving systems) can perceive the total cost accumulated to date at every moment, thereby breaking down the risk assessment that was originally for the entire future trajectory into an assessment of the current augmented state, which paves the way for achieving time consistency.

[0063] Furthermore, the cumulative cost is determined by at least one of the following autonomous driving risk events: collision probability, degree of traffic rule violation, magnitude of rapid acceleration, magnitude of rapid acceleration, and magnitude of rapid steering.

[0064] It is understood that the accumulated cost is the core signal driving the convergence of the entire risk-sensitive reinforcement learning model, and its design directly determines the final safety tendency of the autonomous driving strategy. In this invention, the value of the accumulated cost at each time step is determined by at least one of the following autonomous driving risk events:

[0065] (1) Collision risk probability with surrounding vehicles or pedestrians. This is the most critical safety cost. In a preferred embodiment, this cost can be quantified by calculating the reciprocal or negative exponential function of the collision time between the vehicle and surrounding dynamic traffic participants (vehicles, pedestrians, etc.).

[0066] (2) The degree of traffic rule violation. This cost is used to ensure the compliance of the decision. It is not a simple binary switch (violation or no violation), but a quantifiable "degree". For example: Cost of crossing the lane line: When a vehicle crosses the lane line, the cost is proportional to the distance the tires cross the lane line. Cost of speeding: Proportional to the extent that the current speed exceeds the speed limit. Cost of running a red light: At an intersection, when the traffic light is red or yellow, the cost is related to the distance and speed at which the vehicle enters the stop line. This design allows the strategy to gradually learn compliant behavior in the early stages of training, rather than being penalized only when a serious violation occurs.

[0067] (3) Lateral offset from the lane boundary. This cost primarily ensures smooth and centered driving, improves passenger comfort, and reduces the risk of collisions with lateral obstacles. In one implementation, this cost can be defined as the square of the lateral distance between the vehicle center and the lane centerline.

[0068] (4) The magnitude of rapid acceleration, deceleration, or steering. This cost is directly related to passenger comfort and the smoothness of vehicle control. It is achieved by penalizing drastic changes in the control commands themselves (such as acceleration and steering angular velocity). This cost effectively suppresses the strategy from outputting overly aggressive or jerky control commands, guiding it to learn a smooth and comfortable driving style.

[0069] Finally, a comprehensive autonomous driving risk assessment system can be constructed by linearly or non-linearly combining one or more of the above costs into a total cost. This enables reinforcement learning strategies to not only focus on instantaneous risks but also make long-term risk avoidance decisions based on historically accumulated risks, ultimately achieving safe, comfortable, and compliant autonomous driving behavior.

[0070] Step S20: Construct a risk-sensitive driving strategy model, perform inner-layer dynamic optimization on the risk-sensitive driving strategy model based on the augmented state space vector, and perform outer-layer static optimization after the inner-layer dynamic optimization is completed to obtain the target model.

[0071] The risk-sensitive driving strategy model is a two-stage network consisting of an Actor network and a Critic network. The optimization of the risk-sensitive driving strategy model is divided into two stages: inner-layer dynamic optimization and outer-layer static optimization. In the inner-layer dynamic optimization, auxiliary variables are fixed. Strategy optimization is performed, and in the outer static optimization, auxiliary variables are optimized. Obtain the optimal auxiliary variable (target auxiliary variable). To minimize risk objectives.

[0072] The inner-layer dynamic optimization of the risk-sensitive driving strategy model based on the augmented state space vector specifically includes:

[0073] Under the current strategy and fixed auxiliary variables Under the condition of the augmented state space vector, The estimated value of the Critic network is calculated:

[0074] ;

[0075] in, This represents the estimated value of the Critic network. This indicates the expected outcome under the current strategy. Representation Strategy The parameters, Indicates future accumulated costs. Indicates from the current moment To the final moment Future accumulated costs This represents the total cumulative cost from the initial time to the final time. This represents a convex scoring function used to quantify the total cumulative cost relative to the fixed auxiliary variable. Deviation between;

[0076] Based on the estimated value, the Critic network is updated through temporal difference learning until the estimated value approaches the Bellman target;

[0077] Under the current strategy Under the condition of the augmented state space vector, Calculate the policy gradient used to update the Actor network:

[0078] ;

[0079] in, This represents the policy gradient used to update the Actor network. The scoring function represents the policy, used to measure the policy parameters in the current state. Changes in the selection of the current action The effect of logarithmic probability, The vector in the augmented state space of the Critic network represents... The next state vector ( The assessed advantage value, Indicates the next augmented state. Represents the next primitive state space;

[0080] Based on the policy gradient, the Critic network is updated using an alternating minimization sampling strategy, with dynamic optimization of the inner layers.

[0081] like Figure 2 As shown, it is understandable that the Critic network The input is the policy Fixed auxiliary variables and the augmented state space vector Its output is an estimate of the value function, i.e., the value under the current policy. and fixed auxiliary variables Starting from the current augmented state, the future cumulative costs and The expected deviation.

[0082] Furthermore, the formula for calculating the estimated value of the Critic network defines the learning objective of the Critic network. (Function) It is a convex score function used to quantify the total cumulative cost and auxiliary variables. The task of the Critic network is to predict the expected value of this deviation. The Critic network is updated through temporal difference (TD) learning, with the goal of improving the estimated value. Approaching Bellman's target.

[0083] Furthermore, the input to the Actor network is similar to that of the Critic network, and its output is the probability of selecting each action. The Actor network is updated based on the policy gradient, the direction of which is determined by the value function evaluated by the Critic network. The specific gradient formula is as follows:

[0084] ;

[0085] It is understandable that this formula is a specific application of the Policy Gradient Theorem within the framework of this invention, guiding the Actor network on how to update its parameters. To obtain a better strategy. The expected value on the right side of the formula contains two key parts: This is the policy's score function. It measures the policy parameters in the current state. Small changes affect the selection of the current action The effect of the logarithmic probability. This gradient points to the potential to increase... The probability of being selected Update direction. This is the next state of the Critic network. The assessed advantage value, or valuation estimate, represents the cumulative costs and benefits from the next state onwards. The expected value of the deviation. Its value serves as the reward signal used to weight the policy gradient. The entire formula means: Actor network parameters The update direction is a weighted average of the "policy gradient" and the "future value brought by the action" of all possible actions. If an action... If an action leads to a high-value (i.e., low-risk bias) next state, its policy gradient will be weighted with a large positive number, strongly encouraging the policy to choose this action in similar future states. Conversely, if an action leads to a low-value next state, it will suppress the tendency to choose that action. In this way, the Actor network is guided to learn a policy that minimizes the risk objective.

[0086] It should be noted that, in another implementation, a Q-Learning algorithm can be designed to replace the Actor-Critic network of this invention. In this case, a function needs to be learned, which represents the expected deviation of the future cumulative cost from υ after performing an action in a certain state. The update rule is based on the Bellman optimality equation. Q-Learning can theoretically solve this problem, but it performs poorly in continuous action spaces or high-dimensional action spaces and requires expensive argmax operations. In contrast, the Actor-Critic network can handle continuous actions more naturally and optimizes the policy directly through gradient ascent, which is generally more efficient.

[0087] Furthermore, the step of performing outer-layer static optimization after inner-layer dynamic optimization to obtain the target model specifically includes:

[0088] After the inner dynamic optimization is completed, the fixed auxiliary variable is optimized by stochastic gradient descent with the goal of minimizing the generalized risk function, and the target auxiliary variable is obtained.

[0089] Output the target auxiliary variable, add the target auxiliary variable to the inner layer dynamically optimized risk-sensitive driving strategy model, complete the outer layer static optimization, and obtain the target model.

[0090] In this embodiment, stochastic gradient descent (SGD) is used to optimize the auxiliary variables. Thus, the outer optimization problem can be solved. It is understandable that... The optimization is independent of the updates of the Actor and Critic networks. Specifically, with the goal of minimizing the generalized risk function, an alternating minimization sampling method is proposed, whereby auxiliary variables are minimized during the optimization process. The sampling distribution is based on the current optimal target auxiliary variable. Centered on this, the sampling variance is gradually reduced as iterations proceed. This allows the algorithm to fully explore the early stages. The space can quickly converge to the optimal value in the later stages.

[0091] It should be noted that, besides SGD, other optimization algorithms, such as Bayesian optimization, can be used for the outer optimization problem in this invention. These algorithms may be more robust when dealing with non-convex or noisy objective functions. The difference lies in the fact that SGD's advantages are its computational simplicity, ease of integration with backpropagation in neural networks, and suitability for large-scale data. Bayesian optimization and other methods, on the other hand, have higher computational costs and are more suitable for low-dimensional scenarios with high evaluation costs. This invention chooses SGD to ensure the overall computational efficiency and scalability of the algorithm.

[0092] Step S30: Obtain the current original state space and the current augmented state; construct the current augmented state space vector based on the current original state space and the current augmented state; input the current augmented state space vector into the target model; output control commands; and complete autonomous driving control based on the control commands.

[0093] Specifically, the current original state space is obtained through vehicle sensors, and the current original state space includes at least one of the current vehicle state, the current state of surrounding traffic participants, and the current road environment state.

[0094] In this embodiment, raw, high-dimensional environmental data is acquired through the fusion of multiple sensors mounted on the vehicle. This system typically includes: visual sensors (cameras) for identifying lane lines, traffic lights, and traffic signs; lidar and millimeter-wave radar for detecting the precise distance, location, and speed of surrounding vehicles, pedestrians, and other obstacles; and a positioning and mapping unit (combining a navigation system and a high-precision map) for providing the vehicle's absolute position, heading, and lane-level prior information. This raw data is then processed to obtain the current raw state space.

[0095] The current augmented state is calculated based on historical cost data, and the current original state space is combined with the current augmented state to form the current augmented state space vector.

[0096] The current augmented state space vector is input into the target model for prediction, and control commands are output. The control commands include left turn, right turn, acceleration, deceleration, reversing, or braking.

[0097] In this embodiment, the current original state space is combined with the current augmented state to form a current augmented state space vector, which is then input into the loaded policy model. This model, acting as a forward inference network, quickly outputs the optimal action probability distribution in the current state. Finally, the system generates specific, low-level vehicle control commands based on this distribution (e.g., selecting the action with the highest probability, or sampling based on the distribution). These control commands include left turn, right turn, acceleration, deceleration, reversing, or braking. These commands are typically standardized control quantities, such as: steering commands: target steering wheel angle or steering curvature; longitudinal commands: target acceleration, or directly converted to throttle opening.

[0098] As can be seen, this invention addresses the problem that traditional reinforcement learning methods in autonomous driving systems cannot effectively avoid high-risk events, proposing a risk-sensitive reinforcement learning framework based on convex scoring functions. By constructing an augmented state space and a two-stage optimization architecture, the time-inconsistent risk optimization problem is transformed into a time-consistent problem applicable to dynamic programming. The technical solution of this invention can bring the following overall technical effects to autonomous driving systems: significantly improve driving safety and effectively avoid high-risk events such as collisions; ensure the time consistency of the decision-making process, ensuring that the strategy remains optimal throughout the driving process; have high computational efficiency, suitable for the real-time decision-making needs of onboard computing platforms; and support flexible configuration of multiple risk metrics to adapt to different safety standards and requirements.

[0099] Furthermore, such as Figure 3 As shown, based on the above-mentioned autonomous driving decision-making method based on risk-sensitive reinforcement learning, the present invention also provides an autonomous driving decision-making system based on risk-sensitive reinforcement learning, wherein the autonomous driving decision-making system based on risk-sensitive reinforcement learning includes:

[0100] Problem reconstruction module 51 is used to construct a generalized risk function corresponding to autonomous driving risks, take the negative value of the cumulative cost in the generalized risk function as the augmented state, obtain the original state space of autonomous driving, and construct an augmented state space vector based on the augmented state and the original state space.

[0101] Model optimization module 52 is used to construct a risk-sensitive driving strategy model, perform inner-layer dynamic optimization on the risk-sensitive driving strategy model based on the augmented state space vector, and perform outer-layer static optimization after the inner-layer dynamic optimization is completed to obtain the target model;

[0102] The control execution module 53 is used to obtain the current original state space and the current augmented state, construct the current augmented state space vector based on the current original state space and the current augmented state, input the current augmented state space vector into the target model, output control commands, and complete the autonomous driving control according to the control commands.

[0103] Furthermore, such as Figure 4 As shown, based on the above-mentioned autonomous driving decision-making method and system based on risk-sensitive reinforcement learning, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 4 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0104] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores an autonomous driving decision-making program 40 based on risk-sensitive reinforcement learning, which can be executed by the processor 10 to implement the autonomous driving decision-making method based on risk-sensitive reinforcement learning in this application.

[0105] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the risk-sensitive reinforcement learning-based autonomous driving decision-making method.

[0106] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information about the vehicle and to display a visual user interface. The components of the vehicle communicate with each other via a system bus.

[0107] In one embodiment, when the processor 10 executes the risk-sensitive reinforcement learning-based autonomous driving decision-making program 40 stored in the memory 20, the following steps are performed:

[0108] Construct a generalized risk function corresponding to autonomous driving risks, take the negative value of the cumulative cost in the generalized risk function as the augmented state, obtain the original state space of autonomous driving, and construct an augmented state space vector based on the augmented state and the original state space.

[0109] A risk-sensitive driving strategy model is constructed. The risk-sensitive driving strategy model is dynamically optimized at the inner layer based on the augmented state space vector. After the inner layer dynamic optimization is completed, the outer layer static optimization is performed to obtain the target model.

[0110] Obtain the current original state space and the current augmented state, construct the current augmented state space vector based on the current original state space and the current augmented state, input the current augmented state space vector into the target model, output control commands, and complete autonomous driving control according to the control commands.

[0111] Specifically, the step of using the negative value of the cumulative cost in the generalized risk function as the augmented state to obtain the original state space of autonomous driving, and constructing an augmented state space vector based on the augmented state and the original state, includes:

[0112] Obtain the negative value of the cumulative cost in the generalized risk function, and define the negative value of the cumulative cost in the generalized risk function as an augmented state:

[0113] ;

[0114] in, Indicates an augmented state. This represents the initial state. Indicates the current moment. Indicates the initial time;

[0115] Obtain the original state space for autonomous driving The original state space includes at least one of the vehicle state, the state of surrounding traffic participants, and the road environment state, which are sensed and fused by vehicle sensors.

[0116] The augmented state and the original state space Together, they form the augmented state space vector. .

[0117] The cumulative cost is determined by at least one of the following autonomous driving risk events: collision probability, degree of traffic rule violation, magnitude of rapid acceleration, magnitude of rapid acceleration, and magnitude of rapid steering.

[0118] The risk-sensitive driving strategy model is a two-stage network consisting of an Actor network and a Critic network.

[0119] Specifically, the inner-layer dynamic optimization of the risk-sensitive driving strategy model based on the augmented state space vector includes:

[0120] Under the current strategy and fixed auxiliary variables Under the condition of the augmented state space vector, The estimated value of the Critic network is calculated:

[0121] ;

[0122] in, This represents the estimated value of the Critic network. This indicates the expected outcome under the current strategy. Representation Strategy The parameters, Indicates future accumulated costs. Indicates from the current moment To the final moment Future accumulated costs This represents the total cumulative cost from the initial time to the final time. This represents a convex scoring function used to quantify the total cumulative cost relative to the fixed auxiliary variable. Deviation between;

[0123] Based on the estimated value, the Critic network is updated through temporal difference learning until the estimated value approaches the Bellman target;

[0124] Under the current strategy Under the condition of the augmented state space vector, Calculate the policy gradient used to update the Actor network:

[0125] ;

[0126] in, This represents the policy gradient used to update the Actor network. The scoring function represents the policy, used to measure the policy parameters in the current state. Changes in the selection of the current action The effect of logarithmic probability, The vector in the augmented state space of the Critic network represents... The next state vector ( The assessed advantage value, Indicates the next augmented state. Represents the next primitive state space;

[0127] Based on the policy gradient, the Critic network is updated using an alternating minimization sampling strategy, with dynamic optimization of the inner layers.

[0128] The step of performing outer-layer static optimization after inner-layer dynamic optimization to obtain the target model specifically includes:

[0129] After the inner dynamic optimization is completed, the fixed auxiliary variable is optimized by stochastic gradient descent with the goal of minimizing the generalized risk function, and the target auxiliary variable is obtained.

[0130] Output the target auxiliary variable, add the target auxiliary variable to the inner layer dynamically optimized risk-sensitive driving strategy model, complete the outer layer static optimization, and obtain the target model.

[0131] Specifically, the steps of obtaining the current original state space and the current augmented state, constructing a current augmented state space vector based on the current original state space and the current augmented state, inputting the current augmented state space vector into the target model, and outputting control commands include:

[0132] The current original state space is obtained by vehicle sensors, and the current original state space includes at least one of the current vehicle state, the current state of surrounding traffic participants, and the current road environment state.

[0133] The current augmented state is calculated based on historical cost data. The current original state space is combined with the current augmented state to form the current augmented state space vector.

[0134] The current augmented state space vector is input into the target model for prediction, and control commands are output. The control commands include left turn, right turn, acceleration, deceleration, reversing, or braking.

[0135] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an autonomous driving decision-making program based on risk-sensitive reinforcement learning, and the autonomous driving decision-making program based on risk-sensitive reinforcement learning, when executed by a processor, implements the steps of the autonomous driving decision-making method based on risk-sensitive reinforcement learning as described above.

[0136] In summary, this invention provides an autonomous driving decision-making method, system, terminal, and storage medium based on risk-sensitive reinforcement learning. The method includes: constructing a generalized risk function corresponding to autonomous driving risks; using the negative value of the cumulative cost in the generalized risk function as an augmented state to obtain the original state space of autonomous driving; constructing an augmented state space based on the augmented state and the original state space; constructing a risk-sensitive driving strategy model; performing inner-layer dynamic optimization on the risk-sensitive driving strategy model based on the augmented state space vector; and performing outer-layer static optimization after the inner-layer dynamic optimization is completed to obtain a target model; obtaining the current original state space and the current augmented state; constructing a current augmented state space vector based on the current original state space and the current augmented state; inputting the current augmented state space vector into the target model; outputting control commands; and completing autonomous driving control according to the control commands. This invention ensures the temporal consistency of the decision-making process, significantly improves driving safety, and effectively avoids high-risk events such as collisions.

[0137] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0138] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0139] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A risk-sensitive reinforcement learning-based automatic driving decision method, characterized by, The automatic driving decision-making method based on risk-sensitive reinforcement learning comprises the following steps: A generalized risk function corresponding to the risk of automatic driving is constructed, a negative value of accumulated cost in the generalized risk function is taken as an augmented state, an original state space of automatic driving is obtained, and an augmented state space vector is constructed according to the augmented state and the original state space; A risk-sensitive driving strategy model is constructed, the risk-sensitive driving strategy model is dynamically optimized in an inner layer based on the augmented state space vector, and static optimization in an outer layer is performed after the dynamic optimization in the inner layer is completed to obtain a target model; A current original state space and a current augmented state are obtained, a current augmented state space vector is constructed according to the current original state space and the current augmented state, the current augmented state space vector is input into the target model, a control instruction is output, and automatic driving control is completed according to the control instruction; The negative value of the accumulated cost in the generalized risk function is taken as the augmented state, the original state space of the automatic driving is obtained, and the augmented state space vector is constructed according to the augmented state and the original state, specifically comprising: The negative value of the accumulated cost in the generalized risk function is obtained, and the negative value of the accumulated cost in the generalized risk function is defined as the augmented state: ； wherein, denotes the augmented state, denotes the state at the initial time, denotes the current time, denotes the initial time; Acquiring a raw state space for autonomous driving , the raw state space including at least one of a self-vehicle state, a surrounding traffic participant state, and a road environment state perceived and fused by vehicle sensors The augmented state and the original state space together form an augmented state space vector ; The risk-sensitive driving strategy model is a two-stage network composed of an Actor network and a Critic network.

2. The risk-sensitive reinforcement learning based automatic driving decision method according to claim 1, characterized in that, The accumulated cost is determined by at least one of the automatic driving risk events of collision probability, traffic rule violation degree, sudden acceleration amplitude, sudden acceleration amplitude and sudden turning amplitude.

3. The risk-sensitive reinforcement learning based automatic driving decision method according to claim 1, characterized in that, The risk-sensitive driving strategy model is dynamically optimized in the inner layer based on the augmented state space vector, specifically comprising: In the current policy and the fixed auxiliary variable , based on the augmented state space vector , the estimated value of the Critic network is calculated: ； wherein, represents an estimated value of the Critic network, represents an expectation under the current policy, represents a policy parameter, represents a future cumulative cost, represents a future cumulative cost from a current time to a final time , represents a total cumulative cost from an initial time to a final time, represents a convex score function for quantifying a deviation between the total cumulative cost and the fixed auxiliary variable . According to the estimated value, the Critic network is updated through time difference learning until the estimated value approximates the Bellman target; Under the current policy , a policy gradient for updating the Actor network is calculated based on the augmented state space vector ​ ； wherein, denotes a policy gradient for updating the Actor network, denotes a score function of the policy, measuring the influence of a change in the policy parameters on the log probability of selecting the current action , denotes the advantage value evaluated by the Critic network at the next state vector of the augmented state space vector , , denotes the next augmented state, denotes the next original state space; According to the policy gradient, the Critic network is updated by combining the alternating minimization sampling strategy, and the inner layer dynamic optimization is performed.

4. The risk-sensitive reinforcement learning based automatic driving decision method according to claim 3, characterized in that, After the inner layer dynamic optimization is completed, the outer layer static optimization is performed to obtain the target model, specifically comprising: After the inner layer dynamic optimization is completed, the fixed auxiliary variable is optimized by random gradient descent to minimize the generalized risk function to obtain a target auxiliary variable; The target auxiliary variable is output, the target auxiliary variable is added to the risk-sensitive driving strategy model after the inner layer dynamic optimization, the outer layer static optimization is completed, and the target model is obtained.

5. The risk-sensitive reinforcement learning based automatic driving decision method according to claim 1, characterized in that, The current original state space and the current augmented state are obtained, the current augmented state space vector is constructed according to the current original state space and the current augmented state, the current augmented state space vector is input into the target model, and the control instruction is output, specifically comprising: The current original state space is obtained through a vehicle sensor, and the current original state space comprises at least one of a current ego state, a current surrounding traffic participant state and a current road environment state; The current augmented state is calculated based on historical cost data, and the current original state space and the current augmented state are combined to form the current augmented state space vector; inputting the current augmented state space vector into the target model for prediction, outputting a control instruction, the control instruction including left steering, right steering, accelerating, releasing the accelerator, reversing, or braking.

6. An automatic driving decision system based on risk-sensitive reinforcement learning, characterized in that, The automatic driving decision system based on risk-sensitive reinforcement learning is applied to the automatic driving decision method based on risk-sensitive reinforcement learning in any one of claims 1-5, and the automatic driving decision system based on risk-sensitive reinforcement learning comprises: a problem reconstruction module configured to construct a generalized risk function corresponding to the risk of autonomous driving, take a negative value of an accumulated cost in the generalized risk function as an augmented state, obtain an original state space of autonomous driving, and construct an augmented state space vector according to the augmented state and the original state space; a model optimization module configured to construct a risk-sensitive driving strategy model, perform inner-layer dynamic optimization on the risk-sensitive driving strategy model based on the augmented state space vector, and perform outer-layer static optimization after the inner-layer dynamic optimization is completed to obtain a target model; a control execution module configured to obtain a current original state space and a current augmented state, construct a current augmented state space vector according to the current original state space and the current augmented state, input the current augmented state space vector into the target model, output a control instruction, and complete autonomous driving control according to the control instruction.

7. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a risk-sensitive reinforcement learning-based automatic driving decision program stored on the memory and executable on the processor, and the risk-sensitive reinforcement learning-based automatic driving decision program, when executed by the processor, implements the steps of the risk-sensitive reinforcement learning-based automatic driving decision method in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a risk-sensitive reinforcement learning-based automatic driving decision program, and the risk-sensitive reinforcement learning-based automatic driving decision program, when executed by a processor, implements the steps of the risk-sensitive reinforcement learning-based automatic driving decision method in any one of claims 1-5.

Images