Self-evolution method and system for autonomous driving human-like safety based on data mechanism fusion

Abstract

The application relates to an automatic driving human-like safety self-evolution method and system based on data mechanism fusion, and the method comprises the following steps: extracting real human driving data features from historical experience data, iteratively extracting a target function of a driver decision habit and a planning habit through a maximum entropy inverse reinforcement learning algorithm; sampling from a traffic environment in real time to obtain environment information, constructing an experience revisit pool comprising a current state, an action, a reward and a next moment state, constructing a Q value neural network, extracting data from the experience revisit pool, iteratively updating the Q value neural network, and obtaining humanization constraints; establishing a vehicle model, substituting environment information of a current moment into the vehicle model, constructing a vehicle actuator constraint, combining the vehicle model, the humanization target function and the humanization constraint to search and solve, and obtaining vehicle control information. Compared with the prior art, the application enables the automatic driving vehicle to have self-learning and adaptability, and can realize safe, efficient and comfortable driving.

Description

Technical Field

[0001] This invention relates to the field of autonomous vehicle technology, and in particular to an autonomous driving human-like safety self-evolution method and system based on data mechanism fusion. Background Technology

[0002] The development of autonomous vehicle technology is progressing rapidly, with a hierarchical structure of perception, decision-making, planning, and control becoming the mainstream in commercially available autonomous vehicles. However, due to insufficient information transmission and high latency between layers, the hierarchical architecture often faces constraints in functional integration. For example, limitations in vehicle actuator capabilities lead to incomplete trajectory following, and decision delays in highly time-varying environments cause planning failures. Therefore, designing an integrated decision-making, planning, and control framework has gradually become a research hotspot in this field. Furthermore, mixed traffic environments involving human and autonomous drivers place higher demands on autonomous driving: autonomous driving functions need to conform to human drivers' driving habits and maintain a consistent style. This is crucial for human / autonomous drivers to judge the behavior of surrounding vehicles in highly interactive environments with mixed traffic flow. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a self-evolving method and system for autonomous driving human-like safety based on data mechanism fusion, so that autonomous vehicles have self-learning and adaptability.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] A self-evolving method for human-like safety in autonomous driving based on data mechanism fusion includes the following steps:

[0006] The steps for learning the anthropomorphic objective function are as follows: Extract real human driving data features from historical experience data, and iteratively extract the objective function that is most similar to the driver's decision-making and planning habits through the maximum entropy inverse reinforcement learning algorithm. During the iteration process, multiple candidate trajectories are generated by changing the values ​​of actions in the time domain of the real human driving data features. After extracting features from the trajectories through inverse reinforcement learning, the trajectory that is most similar to the distribution of real human driving data features and its corresponding objective function are extracted through the maximum entropy principle.

[0007] Anthropomorphic constraint learning steps: Sample from the traffic environment in real time to obtain environmental information, construct an experience backdoor pool including the current state, action, reward and the state at the next moment, construct a Q-value neural network, extract data from the experience backdoor pool, iteratively update the Q-value neural network, and use the updated Q-value neural network to obtain anthropomorphic constraints;

[0008] Continuous decision-making, planning, and control steps: Establish a vehicle model and substitute the environmental information at the current moment. Obtain the anthropomorphic objective function through the anthropomorphic objective function learning step. Obtain the anthropomorphic constraints through the anthropomorphic constraint learning step. Construct vehicle actuator constraints. Combine the vehicle model, the anthropomorphic objective function, and the anthropomorphic constraints to perform a search and solution process to obtain vehicle control information.

[0009] Furthermore, the anthropomorphic objective function learning step specifically includes:

[0010] Suppose a discrete-time system has a finite time length L, and a trajectory ζ is formed by organizing the states and actions at each moment in the decision-making field of view:

[0011] ζ=[s1,a1,s2,a2…s L ,a L ]

[0012] The historical experience data is a human driving dataset containing N trajectories:

[0013] D = {ζ1,ζ2,…,ζ} N}

[0014] When performing trajectory evaluation, a linear reward function is selected, which is a weighted sum of the selected trajectory features:

[0015] r(s t )=θ T f(s t )

[0016] In the formula, r(s) t f(s) represents the reward at time t, θ represents the reward weight, and f(s) represents the reward weight at time t. t () represents the trajectory characteristics at time t;

[0017] The reward R(ζ) for trajectory ζ is expressed as:

[0018]

[0019] According to maximum entropy inverse reinforcement learning, the probability of each trajectory is represented as:

[0020]

[0021] In the formula, P(ζ|θ) is the probability of trajectory ζ with reward weight θ, and Z(θ) is the partition function with reward weight θ.

[0022] The maximum entropy inverse reinforcement learning algorithm maximizes the probability of expert demonstrations in the trajectory distribution by adjusting the reward weight θ; thereby iteratively extracting the objective function related to the driver's decision-making and planning habits.

[0023] Furthermore, the driver's lane-changing process is discretized, and a finite number of lane-changing strategy trajectories are generated during trajectory generation to approximate the partition function. The expression of the partition function is as follows:

[0024]

[0025] In the formula, Let M be the i-th lane-changing strategy trajectory, and M be the total number of lane-changing strategy trajectories.

[0026] The objective function of the maximum entropy inverse reinforcement learning is:

[0027]

[0028] In the formula, j(θ) is the objective function of maximum entropy inverse reinforcement learning with reward weight θ.

[0029] Furthermore, the trajectory features include efficiency features, comfort features, risk features, interaction features, and decision features, and the expression for the efficiency feature is:

[0030] f efficient (s t )=v(t)

[0031] The expression for the comfort feature is:

[0032] f comfort,ax (s t )=|a x (t)|

[0033] f comfort,ay (s t )=|a y (t)|

[0034]

[0035]

[0036] The expression for the risk characteristic is:

[0037]

[0038]

[0039]

[0040] The expression for the interaction feature is:

[0041] when a i (t)<0

[0042] The expression for the decision feature is:

[0043] f follow,x (s t )=|s(t)-s ref (t)|

[0044] f follow,y (s t )=|l(t)-l ref (t)|

[0045] In the formula, v(t) and a x (t), a y (t) represents the longitudinal velocity, longitudinal acceleration, and lateral acceleration in the vehicle's coordinate system, respectively, and x represents the x-axis. front (t) represents the longitudinal position of the nearest preceding vehicle, x rear (t) represents the longitudinal position of the nearest following vehicle, a i (t) represents the deceleration of the i-th environmental vehicle affected by the vehicle's movement, s ref (t) and l ref (t) is the reference trajectory.

[0046] Furthermore, the iterative update process of the Q-value neural network is specifically as follows:

[0047] Select state s and action a, calculate Q(s,a) through Q-value neural network, select position, velocity and rotation angle constraints and output them to MPC for solution, and obtain the system state s' and reward R at the next time step, so as to update the gradient of the weights of Q-value neural network.

[0048] Furthermore, the Q-value neural network includes a value function network and a target value function network. The gradient update process for the weights of the Q-value neural network includes: randomly sampling N data points (s, a, R, s') from the experience replay pool, determining whether the endpoint has been reached; if reached, the estimated value of the target value function network is targetQ = R; otherwise, targetQ = R + γmax. a′ Q, where γ is the discount factor, which gradually decreases as the trajectory lengthens. max a′ Q is the largest Q value in the current value function network, which is obtained when the action is a′;

[0049] Calculate the mean squared error loss Loss(θ) = E[(targetQ - Q]] 2 The algorithm initializes the value function network Q and the target value function network targetQ. The parameters of the value function network Q are updated according to the mean squared error loss, while the targetQ remains unchanged. After multiple iterations, all the parameters of the value function network are copied to the target value function network, and this process is repeated to update the algorithm.

[0050] Furthermore, the selection range of the state s is:

[0051] s = [slv] x v y Δs front Δs rear Δl right Δl left Δv x,front Δv x,rear ]

[0052] In the formula, s,l represent the longitudinal and lateral displacements of the vehicle in the Frenet coordinate system, and v x ,v y Let Δs be the vehicle's speed. front ,Δs rear ,Δl right ,Δl left Δv is the relative distance between the vehicle and the nearest surrounding vehicles in all directions. x,front Δv x,rear The relative speed between the vehicle and the nearest surrounding vehicles;

[0053] The selection range of action a is:

[0054] a=[Δs max Δs min Δv max Δv min ,δ min δ max ]

[0055] In the formula, Δs max Δs min The positional constraint input to the MPC represents the maximum / minimum vehicle position difference between the next time step and the current time step, Δv. max Δv min For vehicle speed constraints, δ represents the limit value for the speed increment at the next moment. min δ max This represents the limit value of the vehicle actuator's rotation angle at the next moment.

[0056] Furthermore, the longitudinal kinematic model of the vehicle model is as follows:

[0057]

[0058]

[0059] Where s(t) is the longitudinal displacement of the vehicle in the Frenet coordinate system, l(s) is the lateral displacement at s, and κ(s) is the curvature of the road at point s. v represents the longitudinal velocity of the vehicle in the Frenet coordinate system. x (t), a x (t) represents the longitudinal velocity and acceleration in the vehicle's coordinate system, respectively. The yaw angle of the vehicle relative to the road. The acceleration is in the vehicle's coordinate system;

[0060] The lateral dynamics model of the vehicle model is as follows:

[0061]

[0062]

[0063]

[0064]

[0065] In the formula, Let r be the yaw angle of the vehicle relative to the road, and l be the yaw velocity of the vehicle at its center of gravity. f and l r C represents the distance from the center of gravity to the front and rear axles. f and C r Indicates the lateral stiffness of the front and rear tires, m represents the total vehicle mass, and I represents the lateral stiffness of the front and rear tires. zz Let be the moment of inertia of the vehicle about the z-axis;

[0066] In the continuous decision-making and planning control steps, state variables and action variables are selected, and a vertical objective function and a horizontal objective function are constructed based on the anthropomorphic objective function and anthropomorphic constraints. The expression of the vertical objective function is as follows:

[0067]

[0068] The constraints corresponding to the vertical objective function are:

[0069]

[0070]

[0071]

[0072]

[0073] In the formula, s x,min and s x,max v x,min and v x,max The value of is obtained through a humanized constraint learning step, and its value is the current value plus the position difference and velocity difference constraints obtained through the humanized constraint learning step. x,min ,ax,max It is a constant value determined by the capability of the vehicle actuator;

[0074] The expression for the lateral objective function is:

[0075]

[0076] The constraints corresponding to the lateral objective function are:

[0077]

[0078]

[0079]

[0080] In the formula, and The value is obtained through anthropomorphic constraint learning steps. It is a constant value determined by the vehicle's actuator capability.

[0081] Furthermore, in the continuous decision-making planning and control step, a decision instruction reference curve coefficient is added as a continuous decision reference, and the continuous decision reference corresponds to the input value s of the longitudinal objective function. ref The expression for (t) is:

[0082] s ref (t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5

[0083] In the formula, t is the time value, and a0, a1, a2, a3, a4 and a5 are all polynomial coefficients;

[0084] The continuous decision reference is given by the input value l of the lateral objective function. ref The expression for (t) is:

[0085] l ref (t)=b0+b1t+b2t 2 +b3t 3 +b4t 4 +b5t 5

[0086] In the formula, b0, b1, b2, b3, b4 and b5 are all polynomial coefficients.

[0087] This invention also provides a system based on the above-described data mechanism fusion-based self-evolution method for human-like safety in autonomous driving, comprising:

[0088] An anthropomorphic objective function learning module is used to execute the anthropomorphic objective function learning steps;

[0089] An anthropomorphic constraint learning module is used to execute the anthropomorphic constraint learning steps;

[0090] The continuous decision planning and control module is used to execute the continuous decision planning and control steps.

[0091] Compared with the prior art, the present invention has the following advantages:

[0092] This invention proposes a data-mechanism fusion-based human-like safety self-evolutionary framework for autonomous driving. It adopts an integrated decision-making, planning, and control structure. Under the premise that the mechanistic model meets safety requirements, it simulates human driving strategies from empirical data as much as possible and automatically updates adjustments to driving habits during data flow input. This invention uses a constrained model predictive control (MPC) framework to ensure driving safety in complex scenarios. Simultaneously, it combines inverse reinforcement learning and reinforcement learning algorithms to continuously simulate and adjust the driver's potential reward function and constraints, enabling autonomous vehicles to possess self-learning and adaptive capabilities.

[0093] This invention employs a data mechanism fusion approach, enabling autonomous vehicles to extract human-like driving strategies from real-world driving experiences. This allows the car to mimic personalized driving behaviors in complex and ever-changing traffic environments, achieving safe, efficient, and comfortable driving. Attached Figure Description

[0094] Figure 1 This is a schematic diagram of the processing flow of an autonomous driving humanoid safety self-evolution system based on data mechanism fusion provided in an embodiment of the present invention;

[0095] Figure 2 This is a flowchart illustrating an anthropomorphic objective function learning process based on IRL-MPC, provided in an embodiment of the present invention.

[0096] Figure 3 This is a flowchart of an anthropomorphic constraint learning process based on DQN-MPC provided in an embodiment of the present invention;

[0097] Figure 4 This is a flowchart of an integrated decision-making, planning, and control process based on MPC provided in an embodiment of the present invention. Detailed Implementation

[0098] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0099] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0100] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0101] Example 1

[0102] This embodiment provides a self-evolving method for human-like safety in autonomous driving based on data mechanism fusion, including the following steps:

[0103] The steps for learning the anthropomorphic objective function are as follows: Extract real human driving data features from historical experience data, and iteratively extract the objective function that is most similar to the driver's decision-making and planning habits through the maximum entropy inverse reinforcement learning algorithm. During the iteration process, multiple candidate trajectories are generated by changing the values ​​of actions in the time domain of the real human driving data features. After extracting features from the trajectories through inverse reinforcement learning, the trajectory that is most similar to the distribution of real human driving data features and its corresponding objective function are extracted through the maximum entropy principle.

[0104] Anthropomorphic constraint learning steps: Sample from the traffic environment in real time to obtain environmental information, construct an experience backdoor pool including the current state, action, reward and the state at the next moment, construct a Q-value neural network, extract data from the experience backdoor pool, iteratively update the Q-value neural network, and use the updated Q-value neural network to obtain anthropomorphic constraints;

[0105] Continuous decision-making, planning, and control steps: Establish a vehicle model and substitute the environmental information at the current moment. Obtain the anthropomorphic objective function through the anthropomorphic objective function learning step. Obtain the anthropomorphic constraints through the anthropomorphic constraint learning step. Construct vehicle actuator constraints. Combine the vehicle model, the anthropomorphic objective function, and the anthropomorphic constraints to perform a search and solution process to obtain vehicle control information.

[0106] Specifically, this scheme is based on the concept of model predictive control, and is described as a comprehensive optimization problem consisting of three parts: the model, constraints, and the objective function. The input to this framework is the environmental information at the current moment, and the output is the steering wheel angle and longitudinal acceleration of the controlled autonomous vehicle. The method consists of three steps:

[0107] Step 1: Construct a self-learning objective function algorithm based on inverse reinforcement learning and model predictive control (IRL-MPC). Using maximum entropy inverse reinforcement learning, an objective function representing the driver's decision-making and planning habits is extracted from real data. During the iteration of the objective function, MPC generates a large number of curve clusters by changing the values ​​of actions in the control time domain. After extracting features from the trajectory, inverse reinforcement learning uses the maximum entropy principle to extract the trajectory most similar to the feature distribution of the real driving trajectory and its corresponding objective function.

[0108] Step 2: Construct a self-learning constraint algorithm based on reinforcement learning. By establishing a DQN network and an experience backtracking pool, and using the ε-greedy principle to select the action with the largest Q value and output it to the MPC as position, velocity, and actuator constraints.

[0109] Step 3: Establish an integrated model framework for continuous decision-making, planning, and control based on model predictive control. To incorporate the decision-making component into the overall framework, this invention introduces polynomial coefficients of the vehicle's driving curves to represent decision variables in the action and state spaces of the model construction, thereby enabling continuous decision-making. Simultaneously, by combining the objective function from Step 1 and the constraints from Step 2, the goal of simultaneously mimicking the driver's decision-making logic and route planning from real driving data is achieved.

[0110] The overall logic of the self-learning anthropomorphic objective function algorithm of IRL-MPC mentioned in step 1 is as follows.

[0111] First, the reward parameters, i.e., the reward function weights, are randomly initialized, and the expected features of human driver trajectories in the real dataset are calculated. For each driving scenario provided in the demonstration data, a set of candidate trajectories is generated using the MPC algorithm, and simulation is performed in the environment model to obtain the feature vector of each candidate trajectory. For a given driving scenario, the size of the generated candidate trajectory space is determined by the size of the action space and the control time domain. After the candidate trajectory generation is completed, the gradient is calculated, and the reward parameters are iteratively updated using the gradient ascent method to match the expected features of the generated candidate trajectories with those of a human driver.

[0112] Suppose a discrete-time system has a finite time length L, and a trajectory ζ is formed by organizing the states and actions at each moment in the decision-making field of view:

[0113] ζ=[s1,a1,s2,a2…sL ,a L ]

[0114] Given a human driving dataset containing N trajectories:

[0115] D = {ζ1,ζ2,…,ζ} N}

[0116] When performing trajectory evaluation, a linear reward function is selected, which is a weighted sum of the selected trajectory features:

[0117] r(s t )=θ T f(s t )

[0118] Among them, trajectory features f9s t The selection is mainly categorized into five aspects: efficiency, comfort, risk, interaction, and decision-making, thereby reflecting the main considerations of human drivers when driving.

[0119] efficiency

[0120] f efficient (s t )=v(t)

[0121] comfort

[0122] f comfort,ax (s t )=|a x (t)|

[0123] f comfort,ay (s t )=|a y (t)|

[0124]

[0125]

[0126] risk

[0127]

[0128]

[0129]

[0130] Interaction

[0131]

[0132] Decision

[0133] f follow,x (st )=|s(t)-s ref (t)|

[0134] f follow,y (s t )=|l(t)-l ref (t)|

[0135] Where, v(t), a x (t), a y (t) represents the longitudinal velocity, longitudinal acceleration, and lateral acceleration in the vehicle coordinate system, respectively. Among the risk considerations, x... front (t) represents the longitudinal position of the nearest preceding vehicle, x rear (t) represents the longitudinal position of the nearest following vehicle. Due to the difference between the initial learning strategy and the actual driving strategy, the vehicle's trajectory differs significantly from the actual trajectory in the early stages of self-learning. To ensure the realism of the surrounding environment, the interaction factors between the vehicle and the environment vehicles are incorporated into the modeling. When the distance between the vehicle and the environment vehicles approaches the danger boundary, the environment vehicles will take interactive actions to avoid a collision. Here, the IDM driving model is used to predict the actions of the environment vehicles caused by the vehicle. i (t) represents the deceleration of the i-th environmental vehicle affected by the vehicle's movement. y,ref and The reference trajectory is given by continuous decision-making, and its value is given by the relevant lane-changing decisions and road conditions, expressed as a fifth-degree polynomial.

[0136] The reward R(ζ) for trajectory ζ is:

[0137]

[0138] According to maximum entropy inverse reinforcement learning, the probability of each trajectory can be expressed as:

[0139]

[0140] The partition function Z(θ) is difficult to handle in continuous high-dimensional spaces because it requires integration over all possible trajectories. This patent discretizes the driver's lane-changing process, generating a finite number of lane-changing strategy trajectories during trajectory generation. To approximate the partition function:

[0141]

[0142] The goal of maximum entropy inverse reinforcement learning is to adjust the reward weights θ to maximize the probability of expert demonstrations in the trajectory distribution. Therefore, its objective function is:

[0143]

[0144] The reward function weights θ are obtained by using the Adam optimization algorithm and gradient ascent method for iterative updates.

[0145] The logic of the self-learning constraint algorithm based on reinforcement learning in step 2 is as follows.

[0146] The DQN reinforcement learning algorithm is used to generate anthropomorphic constraints. First, a DQN algorithm model is constructed, mainly consisting of a Q-value neural network and an experience replay pool. The former is divided into a value function network and a target value function network, with network weights θ randomly selected during initialization and subsequently updated using gradient descent. Additionally, a reward function R needs to be designed. It is important to note that the reward function setting cannot contradict the reward function in inverse reinforcement learning; in this embodiment, speed reward, safety reward, and comfort reward are selected.

[0147] Secondly, an experience revisit pool is constructed, consisting of state s, action a, reward R, and the next time-step state s'. State s is selected as follows, where s,l represent the longitudinal and lateral displacements of the vehicle in the Frenet coordinate system, and v... x ,v y Let Δs be the vehicle's speed. front ,Δs rear ,Δl right ,Δl left Δv is the relative distance between the vehicle and the nearest surrounding vehicles in all directions. x,front Δv x,rear The relative speed between the vehicle and the nearest surrounding vehicles.

[0148] s = [slv] x v y Δs front Δs rear Δl right Δl left Δv x,front Δv x,rear ]

[0149] The selected actions are as follows, where Δs max Δs min The positional constraint input to the MPC represents the maximum / minimum vehicle position difference between the next time step and the current time step, Δv. max Δv min For vehicle speed constraints, δ represents the limit value for the speed increment at the next moment. min δ max This represents the limit value of the vehicle actuator's rotation angle at the next moment.

[0150] a=[Δs max Δs min Δv max Δv min ,δmin δ max ]

[0151] The system uses a neural network to calculate Q(s,a), selects appropriate position, velocity and rotation constraints according to the ε-greedy algorithm, outputs them to the MPC, and obtains the system state s' and reward R at the next moment.

[0152] Finally, the gradients of the network weights θ are updated. N data points (s, a, R, s') are randomly drawn from the experience replay pool. It is determined whether the endpoint has been reached. If reached, the estimated value is targetQ = R; otherwise, targetQ = R + γmax. a′ Q. To make Q(s,a) as close as possible to targetQ, calculate the mean squared error loss Loss(θ) = E[(targetQ - Q]. 2 The algorithm initializes a value function network Q and a target value function network targetQ. The parameters of the value function network Q are updated according to the loss function, while targetQ remains constant. After several iterations, all parameters of Q are copied to the targetQ network, and this process is repeated iteratively. This keeps targetQ constant for a period of time, making the algorithm's updates more stable.

[0153] The logic for establishing the integrated model framework for continuous decision-making, planning, and control based on model predictive control in step 3 is as follows.

[0154] First, to ensure safe driving on curved roads, a vehicle model with decoupled lateral and longitudinal axes is established in the Frenet coordinate system.

[0155] The longitudinal kinematic model is as follows:

[0156]

[0157]

[0158] Where s(t) is the longitudinal displacement of the vehicle in the Frenet coordinate system, l(s) is the lateral displacement at point s, and κ(s) is the curvature of the road at point s. v represents the longitudinal velocity of the vehicle in the Frenet coordinate system. x (t), a x (t) represents the longitudinal velocity and acceleration in the vehicle's coordinate system, respectively. This is the yaw angle of the vehicle relative to the road.

[0159] To integrate planning and decision-making, decision instruction reference curve coefficients are added to the action space as continuous decision references. The decision-making process is continuously represented as a fifth-degree polynomial, serving as the reference input for planning. The coordinates of each point are:

[0160] s ref (t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 +a5t 5

[0161] Choose state variable x lon =[sv x The action variable is u. lon =[a x a i＝0～5 The cost function is defined by combining the anthropomorphic weight coefficients obtained from the IRL self-learning algorithm in step 1. The longitudinal objective function considers longitudinally relevant features such as efficiency, comfort, risk, interaction, and decision-making.

[0162]

[0163] The constraints are set as follows:

[0164]

[0165]

[0166]

[0167]

[0168] Where s x,min and s x,max v x,min and v x,max The value is obtained through the reinforcement learning anthropomorphic constraint algorithm in step 2. Its value is the current time value plus the position difference and velocity difference constraints obtained from the DQN algorithm. x,min ,a x,max It is a constant value determined by the vehicle's actuator capability.

[0169] The lateral dynamics model is as follows:

[0170]

[0171]

[0172]

[0173]

[0174] in, Let r be the yaw angle of the vehicle relative to the road. Let r represent the yaw velocity of the vehicle at its center of gravity, and l be the yaw rate. f and l r C represents the distance from the center of gravity to the front and rear axles.f and C r This indicates the lateral stiffness of the front and rear tires. m represents the vehicle mass, and I... zz Let be the moment of inertia of the vehicle about the z-axis.

[0175] Consistent with the horizontal approach, to integrate planning and decision-making, reference curve coefficients are added to the action space as continuous decision-making references. The decision-making process is continuously represented as a fifth-degree polynomial, serving as the reference input for planning. The coordinates of each point are:

[0176] l ref (t)=b0+b1t+b2t 2 +b3t 3 +b4t 4 +b5t 5

[0177] Select the state variable as The action variable is u lat =[δ f b i＝0～5 The horizontal objective function is selected as follows.

[0178]

[0179] The constraints are set as follows:

[0180]

[0181]

[0182]

[0183] in and The value is obtained through the reinforcement learning anthropomorphic constraint algorithm in step 2. It is a constant value determined by the vehicle's actuator capability.

[0184] This embodiment also provides a human-like safety self-evolutionary system for autonomous driving based on data mechanism fusion. This invention enables autonomous vehicles to extract human-like driving strategies from real-world driving experience, allowing the car to mimic personalized driving behavior in complex and ever-changing traffic environments, achieving safe, efficient, and comfortable driving. Its algorithm framework is as follows: Figure 1 As shown, the system includes an anthropomorphic objective function learning module of IRL-MPC and an anthropomorphic constraint learning module of DQN-MPC, and combines the two modules to construct an integrated self-evolutionary algorithm for decision-making, planning, and control. Module one (IRL-MPC algorithm) provides partial weights of the anthropomorphic objective function for module three (integrated self-evolutionary algorithm for decision-making, planning, and control), while module two (DQN-MPC algorithm) provides constraint parameters for module three.

[0185] The IRL-MPC anthropomorphic objective function learning module, such as... Figure 2 As shown, this module consists of two parts: candidate trajectory sampling and evaluation, and gradient iteration. The first step extracts features from real human driving data. The second step uses the MPC method to simulate in an environment model, generating a candidate trajectory set and extracting the feature vector for each candidate trajectory. The features are divided into two categories: planning features and continuous decision-making features. The third step uses gradient descent and combines the expected features of candidate trajectories and the expected features of real driving trajectories to update the weight coefficients of each feature in the reward function. The final reward function is then passed to the integrated decision-planning MPC framework. Here, ζ represents the human driver demonstration trajectory, and f(ζ) i () represents the trajectory features extracted from the i-th trajectory. Each candidate trajectory generated by the MPC trajectory generator... Demonstrating trajectory with human drivers ζ i They have the same initial state.

[0186] DQN-MPC anthropomorphic constraint learning module, such as Figure 3 As shown, this module consists of an experience pool and an objective function network. The first step involves sampling from the traffic environment to construct an experience backtracking pool, comprising state s, action a, reward R, and the next-time state s'. The second step uses a neural network to calculate Q(s,a), selects appropriate position, velocity, and turning angle constraints according to the ε-greedy algorithm, outputs them to the MPC, and obtains the next-time system state s' and reward R. The third step updates the network weights θ using gradients.

[0187] MPC continuous decision planning and control integrated module, such as Figure 4 As shown, this module consists of three parts: a vehicle model, an objective function, and constraints. The first step is to establish a vehicle model with horizontal and vertical decoupling in the Frenet coordinate system. The second step is to use an inverse reinforcement learning algorithm to obtain anthropomorphic weights and construct an integrated objective function combining planning and decision-making. The third step is to use the anthropomorphic constraints obtained from the reinforcement learning algorithm to construct vehicle actuator constraints. The fourth step is to perform a search to find the solution.

[0188] Each module may include a memory and a processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of the method corresponding to each module.

[0189] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A self-evolving method for human-like safety in autonomous driving based on data mechanism fusion, characterized in that, Includes the following steps: The steps for learning the anthropomorphic objective function are as follows: Extract real human driving data features from historical experience data, and iteratively extract the objective function that is most similar to the current driver's decision-making and planning habits through the maximum entropy inverse reinforcement learning algorithm; during the iteration process, generate multiple candidate trajectories by changing the values ​​of actions in the time domain of real human driving data features; after extracting features from the trajectories through inverse reinforcement learning, extract the trajectory that is most similar to the distribution of real human driving data features and its corresponding objective function through the maximum entropy principle; Anthropomorphic constraint learning steps: Sample from the traffic environment in real time to obtain environmental information, construct an experience backdoor pool including the current state, action, reward and the state at the next moment, construct a Q-value neural network, extract data from the experience backdoor pool, iteratively update the Q-value neural network, and use the updated Q-value neural network to obtain anthropomorphic constraints; Continuous decision-making, planning, and control steps: Establish a vehicle model and substitute it with the environmental information at the current moment. Obtain the anthropomorphic objective function through the anthropomorphic objective function learning step, obtain the anthropomorphic constraints through the anthropomorphic constraint learning step, construct vehicle actuator constraints, and search and solve the vehicle model, anthropomorphic objective function, and anthropomorphic constraints to obtain vehicle control information. The iterative update process of the Q-value neural network is as follows: Select state s and action a, and calculate using Q-value neural network. The system selects position, velocity, and rotation angle constraints, outputs them to the MPC for solving, and obtains the system state at the next moment. s’ and rewards R This allows for gradient updates of the weights in the Q-value neural network. The selection range of the state s is: In the formula, Let these be the longitudinal and lateral displacements of the vehicle in the Frenet coordinate system. For the vehicle's speed, This represents the relative distance between the vehicle and the nearest surrounding vehicles in all directions. The relative speed between the vehicle and the nearest surrounding vehicles; The selection range of action a is: In the formula, where The positional constraints input to the MPC represent the maximum / minimum vehicle position difference between the next time step and the current time step. For vehicle speed constraints, it represents the limit of the speed increment at the next moment. This represents the limit value of the vehicle actuator's rotation angle at the next moment.

2. The self-evolutionary method for humanoid safety in autonomous driving based on data mechanism fusion as described in claim 1, characterized in that, The specific steps for learning the anthropomorphic objective function are as follows: Suppose a discrete-time system has a finite time length. Trajectory is formed by organizing the states and actions at each moment in the decision-making field of vision. : The historical experience data is a human driving dataset containing N trajectories: When performing trajectory evaluation, a linear reward function is selected, which is a weighted sum of the selected trajectory features: In the formula, The reward at time t, As a reward weight, The trajectory characteristics at time t; trajectory Rewards Represented as: According to maximum entropy inverse reinforcement learning, the probability of each trajectory is represented as: In the formula, For the trajectory In reward weight The probability of that time. For reward weight The partition function at time; The maximum entropy inverse reinforcement learning algorithm maximizes the probability of expert demonstrations in the trajectory distribution by adjusting the reward weight θ; thereby iteratively extracting the objective function that aligns with the current driver's decision-making and planning habits.

3. The self-evolutionary method for humanoid safety in autonomous driving based on data mechanism fusion as described in claim 2, characterized in that, The driver's lane-changing process is discretized, and a finite number of lane-changing strategy trajectories are generated during trajectory generation to approximate the partition function. The expression of the partition function is as follows: In the formula, Let M be the i-th lane-changing strategy trajectory, and M be the total number of lane-changing strategy trajectories. The objective function of the maximum entropy inverse reinforcement learning is: In the formula, Let θ be the objective function for maximum entropy inverse reinforcement learning.

4. The self-evolutionary method for humanoid safety in autonomous driving based on data mechanism fusion as described in claim 1, characterized in that, The Q-value neural network includes a value function network and a target value function network. The gradient update process for the weights of the Q-value neural network includes: randomly selecting N data points from the experience replay pool. Determine whether the endpoint has been reached. If it has, then the estimated value of the objective value function network is determined. ,otherwise ,in, The discount factor gradually decreases as the trajectory lengthens. The maximum Q-value in the current value function network is the value of the action. Obtained in time; Calculate the mean square error loss The algorithm initializes the value function network Q and the target value function network targetQ. The parameters of the value function network Q are updated according to the mean squared error loss, while the targetQ remains unchanged. After multiple iterations, all the parameters of the value function network are copied to the target value function network, and this process is repeated to update the algorithm.

5. The self-evolutionary method for humanoid safety in autonomous driving based on data mechanism fusion according to claim 1, characterized in that, In the continuous decision-making, planning, and control steps, state variables and action variables are selected, and vertical and horizontal objective functions are constructed based on the anthropomorphic objective function and anthropomorphic constraints. In the continuous decision-making planning and control step, a decision instruction reference curve coefficient is added as a continuous decision reference. This continuous decision reference corresponds to the input value of the longitudinal objective function. The expression is: In the formula, t is the time value. , , , , and All are polynomial coefficients; The continuous decision reference is given by the input value of the lateral objective function. The expression is: In the formula, , , , , and All are polynomial coefficients.

6. A self-evolving safety system for autonomous driving based on data mechanism fusion, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor calling the computer program to perform the steps of the method as described in any one of claims 1 to 5.

Images