Human feedback based interactive adaptive decision control method for autonomous vehicle

Abstract

The present application relates to a kind of human feedback's automatic driving vehicle interactive adaptive decision control method, comprising the following steps: the state perception information that automatic driving car is collected, using the upper layer reinforcement learning DDQN network model trained, determine the weight coefficient of lower layer MPC controller optimization problem, the DDQN network model is based on the interactive feedback information and scene risk assessment result of human-in-the-loop realizes update, the scene risk assessment result that human evaluation and scene risk assessment model output of imitating human evaluation feedback network model is acted in reward function of DDQN network model, trains DDQN network model and updates parameter;MPC controller is based on weight coefficient and determines optimization problem, and carries out horizontal and longitudinal planning tracking, solves the control amount of next time.Compared with prior art, the present application has the advantages of being able to adaptively control model parameters and learn human driving evaluation, improve driving safety and scene adaptability.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to an interactive adaptive decision control method for autonomous vehicles based on human feedback. Background Technology

[0002] In recent years, with the continuous development of artificial intelligence algorithms, autonomous vehicle technology has advanced rapidly. In the field of autonomous vehicle control, Model Predictive Control (MPC) methods can efficiently handle constrained multi-objective optimization problems. By constructing optimization problems, it ensures that vehicles avoid collisions and improves their performance in dynamic environments, leading to its increasingly widespread application in the development of autonomous driving control systems. While existing MPC algorithms can improve reliability, fault tolerance, and adaptability to specific scenarios through dynamic weight adjustments, the diverse road types and complex, ever-changing traffic flows in real-world scenarios mean that MPC cannot effectively adapt to various complex dynamic environments.

[0003] Meanwhile, reinforcement learning algorithms, by constructing reasonable reward functions, enable autonomous vehicles to autonomously search for feasible control measures and optimize decision-making strategies in the driving environment, making them suitable for handling problems involving scene changes and performance optimization. However, pure reinforcement learning methods suffer from convergence difficulties and limited learning capabilities. Interactive reinforcement learning methods, through reward shaping, policy shaping, exploration guidance, and value function expansion, learn how to perform tasks from real-time evaluations or feedback from human supervisors. This can improve sampling efficiency, accelerate learning speed, and enhance learning credibility, making it a highly promising reinforcement learning method. However, for safety-critical systems, vehicles must explore under the premise of ensuring safety, and reinforcement learning lacks the necessary safety prerequisites. Summary of the Invention

[0004] The purpose of this invention is to provide an interactive adaptive decision control method for autonomous vehicles with human feedback, which combines model prediction methods with interactive reinforcement learning methods to simultaneously satisfy safety, robustness and dynamic scene adaptability.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] An interactive adaptive decision-making control method for autonomous vehicles with human feedback includes the following steps:

[0007] The system acquires state perception information collected by autonomous vehicles, uses a pre-trained upper-layer reinforcement learning Double-DQN (DDQN) network model to determine the weight coefficients of the lower-layer MPC controller optimization problem, and updates the DDQN network model based on human-in-the-loop interaction feedback information and scene risk assessment results. It pre-trains a human evaluation feedback network model using offline data to evaluate the control performance of autonomous vehicles, and uses a scene risk assessment model to assess scene risks. The human evaluation and scene risk assessment results are applied to the reward function of the DDQN network model to train the DDQN network model and update its parameters.

[0008] The MPC controller determines the optimization problem based on the weight coefficients transmitted from the upper layer, and performs horizontal and vertical planning and tracking to solve the control quantity at the next moment.

[0009] The simulated human evaluation feedback network model uses pre-collected diverse continuous driving trajectory information to obtain human evaluation results for driving segments, and saves the trajectory information and evaluation results to the dataset for training. The diverse continuous driving trajectory information includes state information and control actions adopted by the autonomous vehicle.

[0010] The evaluation results output by the human-mimicking evaluation feedback network model are expressed as scores or the degree of difference between sample control actions and human driver control actions.

[0011] The scenario risk assessment model categorizes scenarios into four risk levels:

[0012] Departure from the scenario: The relative distance is greater than the maximum following distance defined in the scenario. In this scenario, the vehicle should shorten the distance between itself and the vehicle in front to ensure the following task is completed.

[0013] Safety scenario: The relative distance is within the range of the maximum following distance and the safe driving distance; or the relative distance is between the safe driving distance and the dangerous stopping distance, but the relative speed (the difference between the speed of the vehicle in front and the speed of the vehicle itself) is greater than the safe relative speed. In this scenario, the vehicle should maintain a constant speed or accelerate slowly.

[0014] Low-risk scenario: The relative distance is within the safe driving distance and dangerous stopping distance range, and the relative speed is less than the safe relative speed. There is a risk of collision, and the vehicle should prioritize safety.

[0015] High-risk scenario: When the relative distance is within the dangerous parking distance range, the vehicles must immediately slow down to increase the distance between the two vehicles and ensure that a collision does not occur.

[0016] The reward function of the DDQN network model is expressed as:

[0017] r(s,a)=r collision +rU +r ΔU +r D +r D

[0018] Among them, the first term r collision For collisions, the negative reward applies; the second item r U For acceleration rewards, if the vehicle meets the acceleration constraints of the MPC problem, it receives a reward; the third term r ΔU For acceleration bonuses, vehicles that fail to meet acceleration constraints will be penalized; the fourth item, r D This is a reward for the accompanying mission; the last item is r. H The reward is an anthropomorphic reward, representing the evaluation result of the human evaluation feedback network model. This reward encourages the agent to pursue high evaluations from human drivers during the exploration process.

[0019] Incorporate the evaluation results of the scenario risk assessment model into the acceleration reward r U middle:

[0020]

[0021] In longitudinal motion control, the control variable u represents the longitudinal acceleration a. X In security scenarios, r u1 As a preset positive value, k1 is a preset negative value; in high-risk scenarios, r u2 r is a preset negative value, and k2 is a preset positive value; in low-risk scenarios, r u3 =0; outside the context, r u4 The default value is negative.

[0022] The training process of the DDQN network model includes:

[0023] Initialize the experience replay pool, with a total capacity of N;

[0024] Randomly initialize the Q parameters θ of the value network and the target value network parameters

[0025] Vehicle state perception information is collected using onboard equipment or sensors. The output value of the DDQN network model is used as the weight coefficient of the MPC controller to optimize the problem. The control quantity is calculated by the MPC controller, and the observed state and reward value at the next moment are obtained. The data pairs are formed by combining the network output value and stored in the experience replay pool.

[0026] The training dataset is constructed by sampling from the experience replay pool, and the Q-network parameters θ are updated using gradient descent.

[0027] During the training and exploration process of autonomous vehicles, the driving is evaluated using the simulated human evaluation feedback network model, and the reward value of the DDQN network model is updated. If the simulated human evaluation feedback network model score is unreasonable or the vehicle exploration is unreasonable, an online human score is obtained, the score of the simulated human evaluation feedback network model is replaced with the online human score, and the reward value of the DDQN network model is updated.

[0028] The MPC controller calculates the expected acceleration of the autonomous vehicle at the next moment based on the weighting coefficients passed from the upper layer, and uses the PID controller for acceleration tracking.

[0029] In the MPC controller, the objective function of the longitudinal motion planning problem is:

[0030]

[0031] Where a=[Q,R u ,R du [x] represents the weight coefficients passed from the upper-level model to the MPC controller. lon,t This represents the state variable in the longitudinal motion control at time t. ΔU represents the control input at the previous moment. lon,t Indicates the increment of the control input; N p Indicates the prediction time domain, and These represent the predicted and reference values ​​of the control output, respectively, reflecting the safety requirements for longitudinal following and the ability to track the reference trajectory; N c Indicates control of the time domain, It indicates the amount of control input, reflecting fuel economy requirements; This indicates the increment of control input, reflecting the requirements for driving and riding comfort.

[0032] This invention addresses driving tasks and road environments under different driving scenarios by using a risk assessment model to evaluate scenario constraints; it utilizes a reinforcement learning DDQN network model to obtain weight coefficient information for the MPC controller optimization problem, and incorporates human driver rating information to train a safe and reliable neural network; the lower-level MPC controller performs lateral and vertical planning based on scenario conditions and weight information to obtain the desired control command, thereby achieving adaptive vehicle safe driving.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) This invention uses the reinforcement learning DDQN network model to solve the sequence decision problem, obtains the weight coefficient information of the MPC controller optimization problem, constructs different MPC optimization problems based on the scenario risk information, and realizes the scenario adaptive control of autonomous vehicles.

[0035] (2) Based on the vehicle exploration situation and human evaluation results, the present invention designs a corresponding human evaluation feedback network model, which incorporates the driving preferences and characteristics of human drivers into the network training process, making full use of the strong robustness and adaptability of human decision-making; at the same time, as the training deepens, the human evaluation feedback network model is gradually optimized, gradually reducing the number of human driver ratings and gradually simplifying the human task.

[0036] (3) The MPC controller performs horizontal and vertical planning based on the scenario risk and the weight information from the upper layer. Under the constraints, it realizes the adaptive control of autonomous vehicles in multiple scenarios, improves control efficiency, and at the same time ensures safety and comfort.

[0037] (4) The scenario risk assessment model of the present invention can analyze the risk level and priority requirements of vehicles in different scenarios and improve the adaptive adjustment capability of multi-weight MPC. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0040] like Figure 1 As shown, this embodiment provides an interactive adaptive decision-making control method for autonomous vehicles based on human feedback, including the following steps:

[0041] S1. Obtain the state perception information collected by the autonomous vehicle, and use the trained upper-layer reinforcement learning DDQN network model to determine the weight coefficients of the lower-layer MPC controller optimization problem.

[0042] The S2 and MPC controllers determine the optimization problem based on the weight coefficients transmitted from the upper layer, and perform horizontal and vertical planning and tracking to solve the control quantity at the next moment.

[0043] Meanwhile, this invention proposes a human evaluation feedback network model that incorporates human driving preferences and characteristics into network training, fully leveraging the robustness and adaptability of human decision-making, and using human ratings to update the experience replay pool, thereby improving the training speed of the DDQN network.

[0044] Specifically, the DDQN network model of this invention is updated based on human-in-the-loop interactive feedback information and scene risk assessment results. It utilizes the adaptability and robustness of humans in handling complex high-dimensional problems, sets up a human evaluation mechanism, and pre-trains a human evaluation feedback network model using offline data to evaluate the control performance of autonomous vehicles. It also uses a scene risk assessment model to assess scene risks, and applies the human evaluation and scene risk assessment results to the reward function of the DDQN network model to train the DDQN network model and update its parameters.

[0045] Specifically, the above methods include:

[0046] (1) Based on the scenario risk assessment model, the scenario risk is assessed by analyzing the scenario environment characteristics, which guides the construction of MPC optimization problems and the updating of DDQN network.

[0047] Autonomous vehicles encounter a variety of complex driving scenarios, each with its own characteristics. In actual driving, autonomous vehicles must strike a balance between different performance parameters, including following distance, safety, fuel economy, and ride comfort. The scenario risk assessment model mentioned in this invention can analyze the risk level and priority requirements of vehicles in different scenarios, improving the adaptive adjustment capability of multi-weighted MPC (Multi-Level Marketing). For example, in a following distance task, based on the relative distance between the vehicle and the vehicle in front, the scenario is divided into four risk levels:

[0048] Departure from the scenario: The relative distance is greater than the maximum following distance defined in the scenario. In this scenario, the vehicle should shorten the distance between itself and the vehicle in front to ensure the following task is completed.

[0049] Safety scenario: The relative distance is within the range of the maximum following distance and the safe driving distance; or the relative distance is between the range of the safe driving distance and the dangerous stopping distance, but the relative speed is greater than the safe relative speed. In this scenario, the vehicle should maintain a constant speed or accelerate slowly to reduce unnecessary vibration and pursue better energy utilization and driving comfort.

[0050] Low-risk scenario: The relative distance is within the safe driving distance and dangerous stopping distance range, and the relative speed is less than the safe relative speed. There is a certain risk of collision, and vehicles should prioritize safety.

[0051] High-risk scenario: When the relative distance is within the dangerous parking distance range, the vehicles must immediately slow down to increase the distance between the two vehicles and ensure that a collision does not occur.

[0052] According to this model, the input is the relative distance and relative speed in the car-following task, and the output is the risk level of the scene. It is applied to the design of reward functions in reinforcement learning and the construction of MPC optimization problems.

[0053] (2) Obtain the weight coefficients of the MPC layer based on the environmental conditions, and use the MPC layer to obtain the desired control instructions to realize adaptive control of autonomous vehicles.

[0054] During driving, the autonomous vehicle collects the current state set s, and in the following task, selects the relative distance and relative speed between the vehicle and the vehicle in front, as well as the longitudinal speed of the vehicle s = [R]. d ,R v ,v ego The information from the state set is passed to the reinforcement learning model, which calculates the weight coefficients a = [Q, R] of the MPC controller in the following vehicle task. u ,R du The weighting coefficients are assigned to the MPC layer to obtain the expected acceleration of the autonomous vehicle in the next moment, and the PID controller is used for acceleration tracking. The MPC controller not only acts as the lower-level controller executing the decisions of the upper layer, but also ensures the safety and comfort of the autonomous vehicle. Its specific structural design is as follows:

[0055] The longitudinal kinematic model is as follows:

[0056]

[0057]

[0058] Where X, v X a X Let x represent longitudinal displacement, longitudinal velocity, and longitudinal acceleration, respectively; select longitudinal displacement and longitudinal velocity as state variable x and output variable y, respectively, and longitudinal acceleration as control variable u.

[0059] u = a X

[0060] The MPC longitudinal motion planning problem can be defined as follows, with the objective function being:

[0061]

[0062] The constraints are set as follows:

[0063]

[0064] st

[0065]

[0066]

[0067]

[0068]

[0069] 0≤ε≤ε max

[0070] Where, x lon,t This represents the state variable in the longitudinal motion control at time t. ΔU represents the control input at the previous moment. lon,t Indicates the increment of the control input; N p Indicates the prediction time domain, and These represent the predicted and reference values ​​of the control output, respectively; N c Indicates control of the time domain, Indicates the control input quantity. This indicates the control input increment.

[0071] In the objective function of MPC, the first term reflects the safety requirements for longitudinal following and the ability to track the reference trajectory; the second term reflects the fuel economy requirements; the third term reflects the requirements for driving comfort; and the addition of the fourth relaxation factor avoids situations where the optimization problem is unsolvable. In this embodiment, the weights of the first term in the MPC objective function are trained using reinforcement learning, thereby dynamically adjusting the desired longitudinal following distance and improving the adaptability of the autonomous vehicle to different scenarios.

[0072] (3) Based on the human evaluation feedback network model that mimics the online rating of the driver, the reinforcement learning DDQN model is updated to achieve interactive vehicle control.

[0073] In real-world training of autonomous vehicle systems, reinforcement learning suffers from drawbacks such as low sampling efficiency, slow learning speed, difficulty in ensuring safety, and low reliability of learning results, hindering its ability to achieve online self-evolution of vehicles in the real world. In this embodiment of the invention, a pre-trained human evaluation feedback network model can be used to score vehicle trajectory segments, and this network model can be trained through real-time human monitoring and intervention.

[0074] Before training the autonomous vehicle, trajectory data training and validation sets are constructed by collecting data on the vehicle's observed states and corresponding control actions. These data are then divided into trajectory segments of equal size, with each segment having a length of K. Human drivers then evaluate the trajectory scores. To ensure greater representativeness of the data, the sources and performance of the trajectory data training and validation sets are diverse, resulting in high variance in human scores. This ensures that the samples are more universal and representative, and also guarantees the effectiveness of network training.

[0075] The human-like evaluation feedback network model uses pre-collected diverse continuous driving trajectory information (including state information and control actions taken by the autonomous vehicle) to obtain human evaluation results for driving segments, and saves the trajectory information and evaluation results to the dataset for training.

[0076] Meanwhile, the human-like evaluation feedback network model allows for real-time interaction with human drivers through the control of devices. If the network model's evaluation is unreasonable or the vehicle's exploration is unreasonable, humans will provide online evaluations, replacing the network model's evaluation results and assigning them to the DDQN network. The network model is also updated synchronously.

[0077] In this embodiment, the evaluation result output by the human evaluation feedback network model is represented by a score. In another embodiment, it can also be represented by measuring the degree of difference between the control actions in the sample and the control actions of a human driver.

[0078] By using trajectory segments as input to a neural network and human driver ratings as output, the neural network is trained to obtain a human evaluation feedback network model, which is then used to guide the update of the reinforcement learning DDQN model.

[0079] During the training and exploration process of autonomous vehicles, the trajectory of each K steps is first stored in a buffer. A score is then obtained using the aforementioned human evaluation feedback network model. This score is processed and added to the reward value; it can be simply processed as the average, or a cumulative discount reward can be used. Meanwhile, humans monitor the training and scoring of the autonomous vehicle in real time. If the network output score is unreasonable, or if human intervention occurs, the human score is used to replace the network score and update the reinforcement learning network. The evaluation network is also updated using human parameters. To prevent excessive fluctuations in network parameters due to data changes, or to address potential unreasonable human scoring, a network fine-tuning method is employed, as follows:

[0080] 1) The neural network model before the update is called the source model. A new neural network model, the target model, is created based on this model; it replicates all the model design and parameters of the source model except for the output layer.

[0081] 2) Set the target model output layer, with the same output size as the source model, and set it to a scalar score value. Randomly initialize the model parameters of this output layer;

[0082] 3) Sample from all trajectory segments and human ratings (new data must be included) to form the training dataset, train the output layer from scratch, and fine-tune the parameters of the remaining layers based on the parameters of the source model.

[0083] Furthermore, the network training can be updated after each human intervention and scoring M times, ensuring sufficient data quantity and saving training computing power and resource consumption.

[0084] By incorporating human feedback into the model training process through the aforementioned human-computer interaction methods, unreasonable or even erroneous decisions during training can be corrected in a timely manner. This allows for continuous adjustment and correction of the model parameter update process, enabling the system to learn the optimal strategy quickly and efficiently, thereby improving the reliability and safety of the autonomous driving system.

[0085] (4) Based on the Double-DQN method, the scenario risk assessment and human evaluation feedback are integrated to output the weight coefficients of the optimization problem in the MPC layer.

[0086] In this embodiment, the DDQN network model will be trained using the scenario risk assessment model obtained from the scenario risk assessment model and the ratings from the human evaluation feedback network model or the human driver in the vicinity, and will output the weight parameters of the lower-level MPC controller.

[0087] In this embodiment, the reinforcement learning layer uses the Double-DQN algorithm, including a Q-value network and an experience replay pool. Compared to the DQN method, it can avoid the network performance degradation caused by overfitting. The training process includes:

[0088] 1) Initialize the experience replay pool, with a total capacity of N;

[0089] 2) Randomly initialize the Q parameters θ of the value network and the target value network. parameters

[0090] 3) Collect vehicle state perception information over a period of time using onboard equipment or sensors. Use the output value of the DDQN network model as the weight coefficients for the MPC controller to optimize the problem. Calculate the control quantity through the MPC controller and obtain the observed state and reward value at the next moment, forming a state-action-reward data pair (s, a, r, s). ′ The data is stored in the experience replay pool, where s represents the state, a represents the action (i.e., the weight coefficients of the MPC layer optimization problem), and r represents the reward. ′ This indicates the state at the next moment.

[0091] Specifically, onboard sensors are used to observe the vehicle's current lane and the situation of the vehicle in front, and the relative distance R between the vehicle and the vehicle in front is calculated. d Relative vehicle speed R v And the longitudinal speed v of the vehicle ego Assuming the state s at time t, the action value Q(s,a) is calculated using the network, which is the weight coefficient value of the following task in the MPC layer. Then, the control quantity a from time t to t+1 is calculated through the MPC layer. MThe system obtains the next observation state s' and the corresponding reward value r, and stores the data pair into the experience replay pool.

[0092] Furthermore, before storing the data in the experience replay pool, the data is first stored in a buffer. After obtaining human evaluation feedback scores, the reward value is updated and then put into the experience replay pool.

[0093] The reward value r in the data pair is calculated using a reward function, defined as r(s,a):

[0094] r(s,a)=r collision +r U +r ΔU +r D +r H

[0095] Among them, the first term r collision For collisions, the negative reward applies; the second item r U For acceleration rewards, if the vehicle meets the acceleration constraints of the MPC problem, it receives a reward; the third term r ΔU For acceleration bonuses, vehicles that fail to meet acceleration constraints will be penalized; the fourth item, r D This is a reward for following vehicles, designed to encourage vehicles to enter the following vehicle mode; the last item is r H The reward is an anthropomorphic reward, representing the evaluation result of the human evaluation feedback network model. This reward encourages the agent to pursue high evaluations from human drivers during exploration and punishes actions with low scores from human evaluations.

[0096] Note that in the rewards of this embodiment, human feedback experience is represented by r. H Introduced in the form of acceleration reward r. U In this embodiment, a scenario risk assessment model is introduced, and a reward construction method is presented:

[0097]

[0098] Wherein, the control variable u represents the longitudinal acceleration a X When the vehicle is in a safe scenario, the reward objective is to control the vehicle's driving behavior within a safe range, keeping acceleration within a low range, and correspondingly, r u1 Take positive, k1 takes negative (r) u1 =30, k1=-10). In high-risk scenarios, this reward encourages the vehicle to decelerate rapidly to avoid a collision, at which point t... u2 Take the negative value, and take the positive value of k2 (t). u2 =-10, k2=1). A constant reward is assigned both in low-risk scenarios and outside of those scenarios, r. u3 =0, r u4=-100, the latter is to impose a larger penalty on behavior in order to make the vehicle adapt to the MPC constraints as soon as possible.

[0099] 4) Using the conditional sampling mechanism mentioned above, sample from two empirical replay pools to construct a training dataset, and update the parameters using gradient descent:

[0100]

[0101] Meanwhile, due to cognitive biases or driver fatigue, especially in complex real-world traffic, human feedback during interactive learning may not always be reasonable and safe. Even if the human driver's intervention is correct, it may conflict with the autonomous vehicle itself. Without strict safety guarantees, these factors can endanger the vehicle and its occupants. In this embodiment of the invention, by introducing a lower-level MPC controller, multi-dimensional safety constraints are incorporated, thereby ensuring safe exploration and accelerated model convergence.

[0102] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. An interactive adaptive decision-making control method for autonomous vehicles based on human feedback, characterized in that, Includes the following steps: The system acquires state perception information collected by autonomous vehicles, uses a pre-trained upper-layer reinforcement learning DDQN network model to determine the weight coefficients of the lower-layer MPC controller optimization problem, and updates the DDQN network model based on human-in-the-loop interaction feedback information and scene risk assessment results. It pre-trains a human evaluation feedback network model using offline data to evaluate the control performance of autonomous vehicles and uses a scene risk assessment model to assess scene risks. The human evaluation and scene risk assessment results are applied to the reward function of the DDQN network model to train the DDQN network model and update its parameters. The MPC controller determines the optimization problem based on the weight coefficients transmitted from the upper layer, and performs horizontal and vertical planning and tracking to solve the control quantity at the next moment. The human-mimicking evaluation feedback network model uses pre-collected diverse continuous driving trajectory information to obtain human evaluation results for driving segments, and saves the trajectory information and evaluation results to the dataset for training. The diverse continuous driving trajectory information includes state information and control actions adopted by the autonomous vehicle. The evaluation results output by the human-mimicking evaluation feedback network model are expressed as scores or the degree of difference between sample control actions and human driver control actions. The reward function of the DDQN network model is expressed as: Among them, the first item For collisions, the negative reward applies; the second item For acceleration rewards, if a vehicle meets the acceleration constraints of the MPC problem, it receives a reward; the third item To provide acceleration bonuses, vehicles that fail to meet acceleration constraints will be penalized; (Fourth item) This is a reward for the accompanying mission; the last item. The reward is an anthropomorphic reward, representing the evaluation result of the human evaluation feedback network model. This reward encourages the agent to pursue high evaluations from human drivers during the exploration process. The training process of the DDQN network model includes: Initialize the experience replay pool, setting the total capacity to be [value missing]. ; Randomly initialize value network parameter and target value network parameters ; Vehicle state perception information is collected using onboard equipment or sensors. The output value of the DDQN network model is used as the weight coefficient of the MPC controller to optimize the problem. The control quantity is calculated by the MPC controller, and the observed state and reward value at the next moment are obtained. The data pairs are formed by combining the network output value and stored in the experience replay pool. A training dataset is constructed by sampling from the experience replay pool and updated using gradient descent. Network parameters .

2. The interactive adaptive decision-making control method for autonomous vehicles based on human feedback according to claim 1, characterized in that, The scenario risk assessment model categorizes scenarios into four risk levels: Departure from the scenario: The relative distance is greater than the maximum following distance defined in the scenario. In this scenario, the vehicle should shorten the distance between itself and the vehicle in front to ensure the following task is completed. Safety scenario: The relative distance is within the range of the maximum following distance and the safe driving distance; or the relative distance is between the safe driving distance and the dangerous stopping distance, but the relative speed is greater than the safe relative speed. In this scenario, the vehicle should maintain a constant speed or accelerate slowly. The relative speed is the difference between the speed of the vehicle in front and the speed of the vehicle itself. Low-risk scenario: The relative distance is within the safe driving distance and dangerous stopping distance range, and the relative speed is less than the safe relative speed. There is a risk of collision, and the vehicle should prioritize safety. High-risk scenario: When the relative distance is within the dangerous parking distance range, the vehicles must immediately slow down to increase the distance between the two vehicles and ensure that a collision does not occur.

3. The interactive adaptive decision-making control method for autonomous vehicles with human feedback as described in claim 1, characterized in that, Incorporate the assessment results of the scenario risk assessment model into the acceleration reward system. middle: In longitudinal motion control, the control variable is... Represents longitudinal acceleration In a security scenario, As a preset positive value, The default value is negative; in high-risk scenarios, The default value is negative. The preset positive value; in low-risk scenarios, ; out of context The default value is negative.

4. The interactive adaptive decision-making control method for autonomous vehicles with human feedback as described in claim 1, characterized in that, During the training and exploration process of autonomous vehicles, the driving is evaluated using the simulated human evaluation feedback network model, and the reward value of the DDQN network model is updated. If the simulated human evaluation feedback network model score is unreasonable or the vehicle exploration is unreasonable, an online human score is obtained, the score of the simulated human evaluation feedback network model is replaced with the online human score, and the reward value of the DDQN network model is updated.

5. The interactive adaptive decision-making control method for autonomous vehicles with human feedback according to claim 1, characterized in that, The MPC controller calculates the expected acceleration of the autonomous vehicle at the next moment based on the weighting coefficients passed from the upper layer, and uses the PID controller for acceleration tracking.

6. The interactive adaptive decision-making control method for autonomous vehicles with human feedback according to claim 1, characterized in that, In the MPC controller, the objective function of the longitudinal motion planning problem is: in, These are the weight coefficients passed from the upper-level model to the MPC controller; This represents the state variable in the longitudinal motion control at time t. This indicates the control input from the previous moment. Indicates the increment of the control input; Indicates the prediction time domain; and These represent the predicted and reference values ​​of the control output, respectively, reflecting the safety requirements for longitudinal following and the ability to track the reference trajectory. Indicates control of the time domain, It indicates the amount of control input, reflecting fuel economy requirements; This indicates the increment of control input, reflecting the requirements for driving and riding comfort.

Images