A control parameter determination method, device, equipment, storage medium and vehicle

Abstract

The application relates to the technical field of automatic driving control, in particular to a control parameter determination method and device, equipment, a storage medium and a vehicle. The method comprises the following steps: in the case of driving object auxiliary control, state data of a vehicle during automatic driving is acquired; the state data is input into a control parameter determination model to obtain control parameters output by the control parameter determination model; wherein the control parameter determination model is obtained by iteratively training an initial control parameter determination model based on updated training sample data; control error data corresponding to the control parameters is determined; in the case that the control error data meets a preset control condition, the automatic driving control parameters of the vehicle are determined according to the control parameters. The automatic driving control parameters are automatically calibrated in real time online through the method of reinforcement learning. According to the scheme, the control parameters meeting the control precision requirement can be obtained without manual parameter adjustment in multiple scenes.

Description

Technical Field

[0001] This application relates to the field of autonomous driving control technology, and in particular to a method, apparatus, device, storage medium, and vehicle for determining control parameters. Background Technology

[0002] Currently, various control algorithms are used in the field of autonomous driving control, such as PID (Proportional Integral Derivative) control, linear quadratic regulator (LQR), and model predictive control (MPC). In these algorithms, the control parameters are calibrated. Real-vehicle testing is required during the algorithm development phase to determine an optimal set of control parameters for application in subsequent real-vehicle operation.

[0003] Typically, when calibrating control parameters for a control algorithm, the parameters are manually adjusted within the autonomous vehicle, with each adjustment followed by driving through a specific road condition. Once the control performance under the current condition is optimal, parameter calibration is performed for the next specific condition, continuously optimizing the parameters to meet most driving conditions. This calibration process often requires a significant amount of repetitive work and parameter trials, consuming considerable manpower and resources. Even so, manually adjusted parameters may never achieve the ideal control accuracy. Furthermore, the control parameters determined after calibration are not fully adaptable to all vehicle models, requiring repeated parameter adjustments and calibrations for different models. Moreover, once an autonomous driving control algorithm is determined for a particular vehicle type, it cannot be replaced with a new algorithm, resulting in low scalability of the vehicle's autonomous driving control. Summary of the Invention

[0004] This application provides a method, apparatus, device, storage medium, and vehicle for determining autonomous driving control parameters. By using reinforcement learning, the autonomous driving control parameters are automatically calibrated online in real time. This solution eliminates the need for manual parameter tuning in various scenarios and can obtain control parameters that meet the control accuracy requirements.

[0005] In a first aspect, embodiments of this application disclose a method for determining autonomous driving control parameters, the method comprising:

[0006] In the case of driver assistance control, acquire the vehicle's state data during autonomous driving;

[0007] The state data is input into the control parameter determination model to obtain the control parameters output by the model. The control parameter determination model is obtained by iteratively training an initial control parameter determination model based on updated training sample data. The updated training sample data is determined based on sample state data, the corresponding sample control parameters, and sample optimized control data. The sample optimized control data is used to evaluate the vehicle's autonomous driving performance under the sample predicted control parameters, which are predicted by the initial control parameter determination model based on the updated training sample data.

[0008] Determine the control error data corresponding to the control parameters;

[0009] If the control error data meets the preset control conditions, the vehicle's automatic driving control parameters are determined based on the control parameters.

[0010] Furthermore, determine the control error data corresponding to the control parameters, including:

[0011] Based on the control parameters, determine the vehicle control data for controlling the vehicle;

[0012] Control the vehicle based on vehicle control data to obtain the vehicle's position data;

[0013] Obtain the vehicle's desired trajectory data;

[0014] Based on the location data and the desired trajectory data, determine the control error data corresponding to the control parameters.

[0015] Furthermore, the method also includes:

[0016] Acquire corrective control data; the corrective control data is used to correct vehicle control data;

[0017] Determine the optimized control data corresponding to the control parameters based on the corrective control data and control error data;

[0018] The state data, the corresponding control parameters, and the optimized control data corresponding to the control parameters are added to the initial training sample data to obtain the updated training sample data.

[0019] The updated training sample data is input into the initial control parameters to determine the model for control parameter prediction training, and the sample predicted control parameters are obtained.

[0020] Based on the difference information between the sample prediction control parameters and the sample control parameters, the initial control parameters are adjusted to determine the model parameters until the preset training termination condition is met, thus obtaining the control parameter determined model.

[0021] Furthermore, the control error data includes the current control error data corresponding to the current control parameter; the time corresponding to the current control parameter is the current time.

[0022] If the control error data meets the preset control conditions, the vehicle's autonomous driving control parameters are determined based on the control parameters, including:

[0023] If the current control error data meets the preset control conditions, obtain the set of control parameters from the current time to a preset time period in the future; the set of control parameters includes multiple control parameters.

[0024] Determine the control error data corresponding to each control parameter;

[0025] When each control error data meets the preset control conditions, the vehicle's autonomous driving control parameters are determined based on the control parameters in the control parameter set.

[0026] Furthermore, if each control error data point meets the preset control conditions, the vehicle's autonomous driving control parameters are determined based on the control parameters in the control parameter set, including:

[0027] If each control error data satisfies the preset control conditions, determine the control parameter in the control parameter set whose control error data is less than the error threshold.

[0028] Control parameters whose control error data is less than the error threshold are used as the autonomous driving control parameters for the vehicle.

[0029] Furthermore, if each control error data point meets the preset control conditions, the vehicle's autonomous driving control parameters are determined based on the control parameters in the control parameter set, including:

[0030] Under the condition that each control error data meets the preset control conditions, determine the dispersion value of the control parameters in the control parameter set;

[0031] When the dispersion value is less than the dispersion threshold, the autonomous driving control parameters of the vehicle are determined based on the control parameters in the control parameter set.

[0032] Furthermore, when the dispersion value is less than the dispersion threshold, the autonomous driving control parameters of the vehicle are determined based on the control parameters in the control parameter set, including:

[0033] When the dispersion value is less than the dispersion threshold, determine the average value of the control parameters in the control parameter set;

[0034] The average value of the control parameters is used as the autonomous driving control parameters for the vehicle.

[0035] Secondly, embodiments of this application disclose an autonomous driving control parameter determination device, the device comprising:

[0036] The status data acquisition module is used to acquire the vehicle's status data during autonomous driving under the assistance of the driving object;

[0037] The control parameter determination module is used to input state data into the control parameter determination model and obtain the control parameters output by the model. The control parameter determination model is obtained by iteratively training an initial control parameter determination model based on updated training sample data. The updated training sample data is determined based on sample state data, the corresponding sample control parameters, and sample optimized control data. The sample optimized control data is used to evaluate the autonomous driving performance of the vehicle under the sample predicted control parameters, which are predicted by the initial control parameter determination model based on the updated training sample data.

[0038] The control error data determination module is used to determine the control error data corresponding to the control parameters;

[0039] The autonomous driving control parameter determination module is used to determine the autonomous driving control parameters of the vehicle based on the control parameters when the control error data meets the preset control conditions.

[0040] In some optional implementations, the control error data determination module includes:

[0041] The vehicle control data determination unit is used to determine the vehicle control data for controlling the vehicle based on the control parameters.

[0042] The position data determination unit is used to control the vehicle based on the vehicle control data to obtain the vehicle's position data.

[0043] Location data acquisition unit, used to acquire vehicle location data;

[0044] The control error data determination unit is used to determine the control error data corresponding to the control parameters based on the position data and the desired trajectory data.

[0045] In some alternative embodiments, the device further includes:

[0046] The correction control data acquisition module is used to acquire correction control data; the correction control data is used to correct vehicle control data.

[0047] The optimized control data determination module is used to determine the optimized control data corresponding to the control parameters based on the corrected control data and control error data.

[0048] The module for determining updated training sample data is used to add state data, the corresponding control parameters, and the optimized control data corresponding to the control parameters to the initial training sample data to obtain updated training sample data.

[0049] The sample prediction control parameter determination module is used to input updated training sample data into the initial control parameter determination model to perform control parameter prediction training and obtain the sample prediction control parameters.

[0050] The model training module is used to adjust the initial control parameters to determine the model parameters based on the difference information between the sample predicted control parameters and the sample control parameters, until the preset training termination condition is met, and the control parameter determined model is obtained.

[0051] In some optional implementations, the control error data includes the current control error data corresponding to the current control parameters; the time corresponding to the current control parameters is the current time; the autonomous driving control parameter determination module includes:

[0052] The control parameter set acquisition unit is used to acquire a set of control parameters from the current time to a future preset time period when the current control error data meets the preset control conditions; the control parameter set includes multiple control parameters.

[0053] The control error data determination unit is used to determine the control error data corresponding to each control parameter.

[0054] The autonomous driving control parameter determination unit is used to determine the autonomous driving control parameters of the vehicle based on the control parameters in the control parameter set, provided that each control error data meets the preset control conditions.

[0055] As an optional implementation, the autonomous driving control parameter determination unit includes:

[0056] The control parameter determination subunit is used to determine the control parameters in the control parameter set whose control error data is less than the error threshold, provided that each control error data meets the preset control conditions.

[0057] The autonomous driving control parameter determination subunit is used to determine the control parameters whose control error data is less than the error threshold as the autonomous driving control parameters of the vehicle.

[0058] As another optional implementation, the autonomous driving control parameter determination unit includes:

[0059] The discreteness value determination subunit is used to determine the discreteness value of the control parameters in the control parameter set when each control error data meets the preset control conditions.

[0060] The autonomous driving control parameter determination subunit is used to determine the autonomous driving control parameters of the vehicle based on the control parameters in the control parameter set when the dispersion value is less than the dispersion threshold.

[0061] In some optional implementations, the autonomous driving control parameter determination subunit includes:

[0062] The average value of the control parameters determines the molecular unit, which is used to determine the average value of the control parameters in the control parameter set when the dispersion value is less than the dispersion threshold.

[0063] The autonomous driving control parameter determination molecule is used to take the average value of the control parameters as the autonomous driving control parameters of the vehicle.

[0064] Thirdly, embodiments of this application disclose an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The processor loads and executes the autonomous driving control parameter determination method as described above.

[0065] Fourthly, embodiments of this application disclose a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the autonomous driving control parameter determination method as described above.

[0066] Fifthly, embodiments of this application disclose a vehicle, which includes the autonomous driving control parameter determination device described above.

[0067] The technical solution provided in this application has the following technical effects:

[0068] This method for determining autonomous driving control parameters utilizes reinforcement learning to automatically calibrate control parameters during the autonomous driving process, reducing the cost and time of manual calibration. Compared to manual calibration, the control parameters calibrated using this method achieve higher accuracy and smaller control errors. Furthermore, this parameter determination method is applicable to various control algorithms and can quickly adapt to the parameter calibration of new control algorithms and vehicle models, improving the scalability of autonomous driving algorithms. Attached Figure Description

[0069] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a schematic diagram of the application environment of an autonomous driving control parameter determination method provided in an embodiment of this application;

[0071] Figure 2 This is a flowchart illustrating a method for determining autonomous driving control parameters provided in an embodiment of this application;

[0072] Figure 3 This is a schematic diagram of a linear two-degree-of-freedom vehicle model provided in an embodiment of this application;

[0073] Figure 4 This is a schematic diagram of a reinforcement learning architecture provided in an embodiment of this application;

[0074] Figure 5 This is a schematic diagram of a neural network structure provided in an embodiment of this application;

[0075] Figure 6 This is a schematic diagram of the structure of an automatic driving control parameter determination device provided in an embodiment of this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0077] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0078] To make the objectives, technical solutions, and advantages disclosed in the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application.

[0079] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0080] Currently, the control algorithms used in autonomous driving undergo real-vehicle debugging during the algorithm development phase to determine a set of optimal control parameters for subsequent real-vehicle operation. This debugging process requires numerous repetitive real-vehicle tests, repeatedly examining the performance of all parameters on real vehicles. Different algorithms, vehicle models, and even different road conditions require individual testing, consuming significant time and manpower. The control parameter calibration of the aforementioned control algorithms suffers from the following problems: 1. Control algorithm calibration is difficult, requiring extensive real-vehicle testing and parameter tuning. 2. A set of control parameters cannot fully adapt to all vehicle models, requiring repeated parameter tuning and calibration. 3. Replacing with a new control algorithm requires recalibrating and re-tuning. 4. Manual parameter tuning may consistently fail to achieve the desired control accuracy.

[0081] In view of this, embodiments of this application provide a method for determining autonomous driving control parameters. This method does not employ manual calibration and does not rely on any calibration experience or theoretical support, ensuring that the vehicle obtains control parameters that meet the required control accuracy during autonomous driving. This method can satisfy the calibration of control parameters for any vehicle model and multiple control algorithms. On any vehicle model, any control algorithm can be deployed, and autonomous driving can be initiated based on the method described in this application. As the mileage increases, the control algorithm parameters are automatically calibrated. This calibration process does not require intervention from developers or testers, thus eliminating reliance on development and testing personnel for control parameter calibration and significantly reducing manpower and time costs.

[0082] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating the application environment of an autonomous driving control parameter determination method provided in an embodiment of this application, such as... Figure 1 As shown, the application environment may include an in-vehicle computer terminal.

[0083] In this embodiment, an on-board computer terminal is installed in a vehicle equipped with autonomous driving capabilities. Optionally, the on-board computer terminal includes an autonomous driving control module and a control parameter calibration module. The autonomous driving control module includes an autonomous driving control algorithm, which may be PID, LQR, or MPC, etc. The control algorithm determines the control quantity for controlling the vehicle based on dynamic errors. The control algorithm includes control parameters, which are calibration parameters. The control parameter calibration module is used to calibrate the control parameters in the autonomous driving control algorithm.

[0084] The following describes a specific embodiment of an autonomous driving control parameter determination method according to this application. Figure 2 This is a flowchart illustrating a method for determining autonomous driving control parameters according to an embodiment of this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server products, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 2 As shown, the method may include:

[0085] S201: Acquire vehicle status data during autonomous driving under driver assistance control.

[0086] In this embodiment, before calibrating the control parameters in the autonomous driving control algorithm, it is necessary to first construct an agent for reinforcement learning. During the autonomous driving process, the agent continuously interacts with the environment and learns based on the reward function to obtain highly accurate autonomous driving control parameters.

[0087] In this embodiment, before constructing the reinforcement learning agent, it is necessary to construct a vehicle dynamics error model for autonomous driving. When a vehicle is performing autonomous driving, the dynamics error models generally involved can be divided into a longitudinal dynamics error model and a lateral dynamics error model. The autonomous driving control parameter determination method described in this embodiment mainly calibrates the control parameters in the autonomous driving control algorithm. Therefore, the control parameters in the control algorithms used for both longitudinal and lateral control of the vehicle during autonomous driving can be calibrated using the autonomous driving control parameter determination method described in this embodiment. The following uses the lateral control of the vehicle during autonomous driving as an example to illustrate the control parameter calibration process of the adopted control algorithm. For the specific implementation process of the control parameter calibration process of the control algorithm used for longitudinal control of the vehicle during autonomous driving, please refer to the control parameter calibration process of the control algorithm used for lateral control; it will not be repeated here.

[0088] In this embodiment of the application, when calibrating the control parameters in the control algorithm used for the lateral control of the vehicle during autonomous driving, it is first necessary to establish a lateral dynamics error model of the vehicle. Specifically, Figure 3 This is a schematic diagram of a linear two-degree-of-freedom vehicle model provided in an embodiment of this application, as shown below. Figure 3 As shown, a linear two-degree-of-freedom model of the vehicle is established based on its motion characteristics. The vehicle's forward direction is defined as the x-axis, and the direction perpendicular to the forward direction as the y-axis. The resultant force along the y-axis and the torque about the center of mass acting on the vehicle satisfy the following equation:

[0089]

[0090] In equation (1), m is the vehicle mass; a y F is the lateral acceleration. Y1 F Y2 The lateral force of the front and rear wheels is δ; the front wheel steering angle is I. Z Let ω be the vehicle's moment of inertia about its center of mass; ω be the vehicle's yaw rate; and a and b be the distances from the center of mass to the front and rear axles.

[0091] Since the front wheel steering angle δ is relatively small, and the tire lateral force is proportional to the lateral stiffness and the lateral angle, equation (1) can be transformed into the following equation:

[0092]

[0093] In equation (2), c f c r The lateral stiffness of the front and rear wheels; α f α r The slip angle is the side slip angle of the front and rear wheels.

[0094] like Figure 3As shown, the vehicle's sideslip angle is the angle between the plane of the wheel center and the velocity directions of the midpoints of the front and rear axles, satisfying the following formula:

[0095]

[0096] In equation (3), v is the velocity of the center of mass. com The component along the y-axis is the lateral velocity; u is the velocity of the center of mass v. com The component in the x-axis direction is the longitudinal velocity.

[0097] lateral acceleration a of the vehicle y The sum of the differential of the lateral velocity v and the centripetal acceleration, i.e., a y Satisfy the following formula:

[0098]

[0099] Combining equations (2), (3), and (4), we can obtain the following linear two-degree-of-freedom motion differential equations for the vehicle:

[0100]

[0101] When performing lateral control for autonomous driving, the vehicle is generally required to follow the desired position and desired heading angle of the target point on the planned trajectory. Therefore, equation (5) above is rewritten as an error model. The following two error quantities are selected as control state quantities:

[0102] e1: The lateral distance between the vehicle's center of gravity and the desired position of the target point on the planned trajectory;

[0103] e2: The difference between the vehicle's heading angle and the expected heading angle of the target point on the planned trajectory.

[0104] According to the definition of heading angle error e2, the following formula can be obtained:

[0105] e2=θ-θ′

[0106] Equation (6);

[0107]

[0108]

[0109] In equation (6), θ is the vehicle heading angle, and θ′ is the desired heading angle of the target point on the planned trajectory. Let the lateral velocity v of the vehicle along the y-axis be equal to the differential of the lateral displacement y. The vehicle's yaw rate ω is equal to the differential of the vehicle's heading angle θ. The desired yaw rate is the differential of the desired heading angle θ′. Assuming the vehicle's longitudinal velocity u and trajectory radius R are constants during the control cycle, then Satisfy the following formula:

[0110]

[0111] The desired lateral acceleration of the vehicle satisfies the following formula:

[0112]

[0113] Based on equations (4) and (10) and the definition of lateral error e1, the lateral acceleration error can be obtained as:

[0114]

[0115] From equation (11), the lateral velocity error can be obtained as:

[0116]

[0117] Substituting equations (6), (7), (8), (11), and (12) into equation (5), we obtain the vehicle dynamics error model as follows, which satisfies the following equation:

[0118]

[0119] Let e1, e2 and their first derivatives be defined as state variables, and let equation (13) be transformed into matrix form to satisfy the following equation:

[0120]

[0121]

[0122] Equation (14) is the vehicle lateral dynamics error model used in the autonomous driving control algorithm.

[0123] In this embodiment, after obtaining the vehicle lateral dynamics error model for the autonomous driving control algorithm, it is also necessary to design an autonomous vehicle lateral control algorithm. Algorithms applied to autonomous driving lateral control generally include PID control, LQR control, and MPC control. The principle of each control algorithm is to solve for the control quantity from the state matrix composed of the lateral error, and then apply the control quantity to the vehicle lateral control system to form closed-loop feedback control, gradually reducing the vehicle's lateral error. Since the control principles of each control algorithm are similar, and the autonomous driving control parameter determination method described in this embodiment is mainly used to calibrate the control parameters in the control algorithm, the following uses model predictive control (MPC) as an example of the autonomous vehicle lateral control algorithm to illustrate the calibration process of the control parameters in the control algorithm. For the specific implementation process of the control parameter calibration process of other control algorithms, please refer to the control parameter calibration process of the MPC control algorithm, which will not be repeated here.

[0124] The system state equation based on the MPC algorithm, equation (14), can be expressed as follows:

[0125]

[0126] Solving the MPC problem can often be transformed into solving a quadratic optimization problem, with the objective function being the following:

[0127]

[0128] In equation (16), Q is the state weight matrix; R is the control weight matrix; x contains all state variables within the prediction step size P; and u contains all control variables within the prediction step size P.

[0129] Solving the above quadratic optimization problem yields a set of optimal control sequences U = [u t ,u t+1 ,…,u t+P ], the first item u t The control output at the current time t is applied to the control system. The system state matrix is ​​updated at the next time step, and the above process of solving the quadratic optimization problem is repeated to continuously control the vehicle's lateral control system.

[0130] As can be seen from equation (16), for the MPC algorithm, the control sequence U is the control quantity that directly acts on the vehicle, while the control parameters P, Q, and R are calibration quantities. After calibration, P, Q, and R generally do not need to be changed if the system has not changed.

[0131] In this embodiment of the application, after the control algorithm is determined, it is also necessary to construct a deep reinforcement learning agent. The agent continuously learns and optimizes, and finally completes the calibration of the control parameters in the control algorithm. Figure 4 This is a schematic diagram of a reinforcement learning architecture provided in an embodiment of this application, such as... Figure 4 As shown, the reinforcement learning agent receives state s from the environment and outputs action a. Here, state s is the state variable in the vehicle lateral error dynamics model. Action 'a' refers to the control parameters in the lateral control algorithm for autonomous vehicles. When the control algorithm uses the MPC algorithm, the output action 'a' is a 6-dimensional vector [P, Q, R]. When the control algorithm uses the PID algorithm, the output action 'a' is a 3-dimensional vector [P, I, D].

[0132] As an optional implementation, the agent is constructed using a deep Q-network (DQN) algorithm. DQN is a value-based rather than policy-based approach; it learns a critic rather than a policy. The critic does not directly take action but evaluates the quality of the action. The DQN algorithm uses two identical neural networks: an action network and a target network. These two networks initially have the same structure and parameter configuration. During training, the parameters of the action network are continuously updated and optimized, and these parameters are only assigned to the target network at regular intervals. Optionally, both the action network and the target network can be multi-layered fully connected layers. Figure 5 This is a schematic diagram of a neural network structure provided in an embodiment of this application, such as... Figure 5 As shown, the action network and target network structures contain three fully connected layers, with 300 neurons in the hidden layers.

[0133] It should be noted that, Figure 4 This is only one optional implementation method. The number of fully connected layers in the action network and the target network, as well as the number of neurons in the hidden layers, can be flexibly adjusted according to actual needs.

[0134] In this embodiment, after the reinforcement learning agent is constructed, the reward function for reinforcement learning also needs to be designed. Since autonomous vehicles operating on open roads currently involve certain uncertainties and dangers, a safety driver is required to monitor the vehicle's status and take over at any time. When designing the reward function for this method, in addition to considering the impact of the agent's output control parameters on the vehicle's control performance, the correction of the control input from the driver also needs to be considered. Furthermore, to optimize the autonomous driving control performance and reduce human intervention, the reward / penalty for the driver's correction is set to the maximum. In summary, the reward function is set as follows, satisfying the following formula:

[0135]

[0136] In equation (17), e is the natural logarithm; δ is the front wheel angle of the control output; ψ is the steering wheel angle of the driving object correction; μ and η are the coefficients of the control quantity and the correction quantity; ε is the allowable error range. The reward is the largest when both the lateral error and the heading angle error are less than ε, the reward is the second largest when only the lateral error is less than ε, the reward is the smallest when only the heading angle error is less than ε, and there is no reward in other cases.

[0137] In this embodiment, the autonomous vehicle differs from other controlled systems. Currently, the autonomous driving system is in the human-machine co-driving stage, where the machine generates the control inputs, which are then corrected by the driver. This ensures the safety and reliability of the autonomous vehicle. During the development of the autonomous driving algorithm, it is even more crucial for the driver to be able to monitor road conditions in real time to avoid dangerous situations. Therefore, the steering wheel correction input of the driving object is added to the reinforcement learning reward function, making the reinforcement learning system not a completely closed-loop system. The goal is for the agent to obtain optimal control parameters through its own learning, while also mimicking the driving behavior of a human driver under certain specific conditions to achieve optimal control accuracy and maximum safety.

[0138] In this embodiment, the DQN algorithm, during training, needs to establish a memory bank to store training samples in order to weaken the correlation between adjacent states, and randomly selects samples for training each time. Therefore, the initial memory bank size can be set according to actual needs. The memory bank stores several records composed of [s] t ,a t ,r t ,s t+1 A memory tuple composed of s. t As the current state, a t The action output by the DQN algorithm based on the current state, i.e., the control parameters [P, Q, R] corresponding to the current state, r t To optimize control data, specifically by applying a reward function to the current action 'a'... t The rating, s t+1 This represents the state at the next moment.

[0139] In this embodiment, during the initial stage, two neural networks in the agent are initially trained to obtain an initial control parameter determination model. When training the neural networks to obtain the initial control parameter determination model, initial training sample data is first required. This initial training sample data can be obtained during the driving process. For example, in the initial state, a set of experience-based control parameters can be configured into the autonomous driving control module, and the driver manually drives the vehicle for a certain distance. Since the control parameters are configured based on experience, they remain constant during this process. Then, the optimized control data output by the reward function corresponding to different state data during the driving process, as well as the state data at the next moment after each state is controlled, are extracted to obtain a set of memory tuples. These memory tuples are added to the memory bank until the number of memories reaches a preset number, such as 1000. These memory tuples are then used as initial training sample data for the initial training of the network to obtain the initial control parameter determination model.

[0140] S203: Input the state data into the control parameter determination model to obtain the control parameters output by the control parameter determination model.

[0141] In this embodiment, state data is input into a control parameter determination model, which then outputs corresponding control parameters. The control parameter determination model is obtained by iteratively training an initial control parameter determination model based on updated training sample data. The updated training sample data is determined based on sample state data, the corresponding sample control parameters, and optimized sample control data. The optimized sample control data is used to evaluate the vehicle's autonomous driving performance under the predicted sample control parameters; these predicted parameters are predicted based on the updated training sample data from the initial control parameter determination model. Specifically, corrective control data is obtained and used to correct the vehicle control data determined based on the control parameters. Optimized control data corresponding to the control parameters is determined based on the corrected control data and control error data. The state data, the corresponding control parameters, and the optimized control data are added to the initial training sample data to obtain updated training sample data.

[0142] As an optional implementation, after obtaining the initial control parameter determination model, automatic driving can be initiated under monitoring to calibrate the control parameters. For example... Figure 4 As shown, during this process, the agent receives the acquired state data s t and output the corresponding action a. t The autonomous driving control module, based on action a t To control the vehicle, output control quantity u t The driver inputs a corrective control quantity ψ to the vehicle based on whether the vehicle is safe to drive. t Correct the control input, u t With ψ t These factors work together to update the vehicle's state, thus obtaining the state data s for the next moment. t+1 Furthermore, in u t With ψ t When these forces act together on the vehicle, the corrected control quantity u' is obtained. t The reward function is based on u' t Action a output by the agent t Scoring is performed to obtain optimized control data r. t Therefore, a complete memory tuple [s] can be obtained. t ,a t ,r t ,s t+1 This memory tuple can be stored in the memory bank to update the initial training sample data in the memory bank and obtain the updated training sample data.

[0143] In this embodiment, several sample data points are randomly selected from the updated training sample data in the memory bank, and the selected sample data is used to train the initial control parameter determination model, thereby continuously updating the initial control parameter determination model. After multiple training iterations, the control parameter determination model can be obtained. Specifically, the updated training sample data is input into the initial control parameter determination model for control parameter prediction training to obtain sample predicted control parameters. Based on the difference information between the sample predicted control parameters and the sample control parameters, the model parameters of the initial control parameter determination model are adjusted until the preset training termination condition is met, thus obtaining the control parameter determination model.

[0144] As an example, during autonomous driving, the neural network in the agent is trained online. Specifically, autonomous driving is initiated, and lateral control parameters are configured. Based on these parameters, control inputs are output, and the driver adjusts the steering wheel according to road conditions. After one control cycle, a reward function is calculated, and a set of memories is added to the memory bank. Simultaneously, the state of the next time step is input to the agent. The agent generates new actions based on the state through the target network and outputs them to the control module for parameter configuration. Every time a preset number of memories are added to the memory bank (e.g., 100), several memories are randomly selected for one round of online training of the action network. The action network parameters are updated using backpropagation based on the error. After several training iterations (e.g., 300 iterations), the action network parameters are hard-updated to the target network.

[0145] In this embodiment, during the autonomous driving process, control parameters are automatically calibrated. Using reinforcement learning, the reward function and network structure are rationally set, and the correlation between consecutive state variables is weakened, thus obtaining a set of optimal control parameters, reducing the cost and time of manual calibration. Furthermore, in the DQN algorithm, each action output of the agent affects subsequent decision actions. However, the autonomous driving lateral control system is a feedback control based on an error model. The control output, calculated from a set of control parameters, only affects the change in the state variable at the current moment. A new set of error state variables is obtained at the next moment, but it does not affect the control parameters, i.e., the controller itself. Therefore, DQN can be combined with autonomous driving control parameter calibration, and the agent's network parameters can be learned online. Testers only need to continuously run the autonomous vehicle, and the agent can continuously learn and update the network, continuously optimizing the control parameters. The testing process is continuous and does not require repeatedly driving the same test road.

[0146] S205: Determine the control error data corresponding to the control parameters.

[0147] In this embodiment, whether the control parameters output by the control parameter determination model meet the calibration accuracy requirements can be judged based on the control error data of the control parameters. Specifically, vehicle control data is determined based on the control parameters. The vehicle is controlled according to the vehicle control data to obtain the vehicle's position data. The desired trajectory data of the vehicle is obtained, and the control error data corresponding to the control parameters is determined based on the position data and the desired trajectory data. The calculation of the control error data is the error value between the vehicle's position after control is applied and the desired trajectory before control is applied. Optionally, control error data e1 and e2 are used. Figure 4 As shown, for the control parameter a, the control parameter a is determined by the model output. t The autonomous driving control module, based on control parameter a t Determine the vehicle control data u for controlling the vehicle t , get u t The position data of the vehicle behind it is applied. Furthermore, the vehicle's position is determined based on the state data, specifically the output control parameter a. t The desired trajectory data is obtained, and then the control error data when using this control parameter can be determined based on the vehicle's position data and the desired trajectory data.

[0148] S207: If the control error data meets the preset control conditions, determine the vehicle's automatic driving control parameters based on the control parameters.

[0149] In this embodiment, as the lateral control algorithm parameters for autonomous driving are continuously updated, the target network used to solve for the control parameters is also continuously updated along with the update of the memory bank and the online training of the action network until the optimal lateral control effect is achieved. The last set of control parameters is then fixed to the autonomous vehicle, thereby realizing the calibration of the vehicle's autonomous driving lateral control parameters. Specifically, the control error data includes the current control error data corresponding to the current control parameter, and the time corresponding to the current control parameter is the current time. When the current control error data meets preset control conditions, a set of control parameters from the current time to a preset future time period is obtained. This set of control parameters includes multiple control parameters. The control error data corresponding to each control parameter is determined. When each control error data meets the preset control conditions, the autonomous driving control parameters of the vehicle are determined based on the control parameters in the control parameter set. In the above process, the preset control condition is that there is no correction for the driving object, i.e., the correction control data ψ is always 0, and e1 < and e2 <. At the current moment, the control parameter determination model outputs the current control parameters based on the current state data. When the corrected control data ψ is 0, it determines the current control error data corresponding to the current control parameters and checks whether the current control error data satisfies e1 < and e2 < 0. If the current control error data satisfies e1 < and e2 < 0, the model then checks the control error data corresponding to each control parameter within a future preset time period T. If all control error data within time period T satisfy the condition that the corrected control data ψ is always 0 and e1 < and e2 < 0, then a set of control parameters can be determined as the vehicle's autonomous driving control parameters based on all control parameters within time period T.

[0150] As an optional implementation, if each control error data point satisfies a preset control condition, the control parameter whose control error data is less than an error threshold is determined from the control parameter set. This control parameter, whose control error data is less than the error threshold, is then used as the vehicle's autonomous driving control parameter. Specifically, if all control error data within time period T satisfies the condition that the correction control data ψ is always 0, e1 < e2 < 0, then the control error data corresponding to all control parameters within time period T can be sorted, and the control parameter with the smallest control error data is determined as the vehicle's autonomous driving control parameter.

[0151] As another optional implementation, if each control error data satisfies a preset control condition, the dispersion value of the control parameters in the control parameter set is determined. If the dispersion value is less than a dispersion threshold, the autonomous driving control parameters of the vehicle are determined based on the control parameters in the control parameter set. Optionally, if the dispersion value is less than the dispersion threshold, the average value of the control parameters in the control parameter set is determined, and this average value is used as the autonomous driving control parameter of the vehicle. Specifically, if all control error data within time period T satisfies the condition that the correction control data ψ is always 0, e1 < and e2 <, then the dispersion value of all control parameters within time period T is calculated. Optionally, the dispersion value can be one of the mean, variance, or standard deviation. If the dispersion value is less than the threshold, the mean of all control parameters within time period T is calculated, and this mean is used as the autonomous driving control parameter of the vehicle.

[0152] The autonomous driving control parameter determination method described in this application can automatically calibrate control parameters during the autonomous driving process. Utilizing reinforcement learning, it rationally sets the reward function and network structure, and weakens the correlation between continuous state variables, thereby obtaining a set of optimal control parameters, reducing the cost and time of manual calibration. Compared to manually calibrated lateral control parameters for autonomous driving, reinforcement learning methods have advantages such as large training volume and quantifiable error tolerance in the reward function. It can fully explore the parameter space of control parameters. Theoretically, after sufficient test mileage and collection of sufficient training samples, the control parameters calibrated by this method can improve the accuracy of lateral control and achieve smaller control errors compared to manual calibration. Furthermore, since the control principles of each control algorithm are the same, but the dimensions and meanings of the control parameters differ, this method can quickly adapt to different control algorithms by simply changing the output dimension of the agent. At the same time, different autonomous vehicles typically cannot be fitted with the same set of control parameters due to differences in their drive-by-wire chassis characteristics and vehicle electronic and electrical hardware parameters. Manual calibration requires calibrating control parameters individually for each vehicle, which is very time-consuming and labor-intensive. This method allows the calibration algorithm to be run directly on different vehicle models, saving time.

[0153] This application also provides an autonomous driving control parameter determination device. Figure 6 This is a schematic diagram of the structure of an automatic driving control parameter determination device provided in an embodiment of this application, as shown below. Figure 6 As shown, the device includes:

[0154] The state data acquisition module 601 is used to acquire the state data of the vehicle during autonomous driving under the condition of driver assistance control;

[0155] The control parameter determination module 603 is used to input state data into the control parameter determination model to obtain the control parameters output by the control parameter determination model. The control parameter determination model is obtained by iteratively training an initial control parameter determination model based on updated training sample data. The updated training sample data is determined based on sample state data, the corresponding sample control parameters, and sample optimized control data. The sample optimized control data is used to evaluate the autonomous driving performance of the vehicle under the sample predicted control parameters, and the sample predicted control parameters are predicted based on the updated training sample data from the initial control parameter determination model.

[0156] The control error data determination module 605 is used to determine the control error data corresponding to the control parameters;

[0157] The autonomous driving control parameter determination module 607 is used to determine the autonomous driving control parameters of the vehicle based on the control parameters when the control error data meets the preset control conditions.

[0158] In some optional implementations, the control error data determination module includes:

[0159] The vehicle control data determination unit is used to determine the vehicle control data for controlling the vehicle based on the control parameters.

[0160] The position data determination unit is used to control the vehicle based on the vehicle control data to obtain the vehicle's position data.

[0161] Location data acquisition unit, used to acquire vehicle location data;

[0162] The control error data determination unit is used to determine the control error data corresponding to the control parameters based on the position data and the desired trajectory data.

[0163] In some alternative embodiments, the device further includes:

[0164] The correction control data acquisition module is used to acquire correction control data; the correction control data is used to correct vehicle control data.

[0165] The optimized control data determination module is used to determine the optimized control data corresponding to the control parameters based on the corrected control data and control error data.

[0166] The module for determining updated training sample data is used to add state data, the corresponding control parameters, and the optimized control data corresponding to the control parameters to the initial training sample data to obtain updated training sample data.

[0167] The sample prediction control parameter determination module is used to input updated training sample data into the initial control parameter determination model to perform control parameter prediction training and obtain the sample prediction control parameters.

[0168] The model training module is used to adjust the initial control parameters to determine the model parameters based on the difference information between the sample predicted control parameters and the sample control parameters, until the preset training termination condition is met, and the control parameter determined model is obtained.

[0169] In some optional implementations, the control error data includes the current control error data corresponding to the current control parameters; the time corresponding to the current control parameters is the current time; the autonomous driving control parameter determination module includes:

[0170] The control parameter set acquisition unit is used to acquire a set of control parameters from the current time to a future preset time period when the current control error data meets the preset control conditions; the control parameter set includes multiple control parameters.

[0171] The control error data determination unit is used to determine the control error data corresponding to each control parameter.

[0172] The autonomous driving control parameter determination unit is used to determine the autonomous driving control parameters of the vehicle based on the control parameters in the control parameter set, provided that each control error data meets the preset control conditions.

[0173] As an optional implementation, the autonomous driving control parameter determination unit includes:

[0174] The control parameter determination subunit is used to determine the control parameters in the control parameter set whose control error data is less than the error threshold, provided that each control error data meets the preset control conditions.

[0175] The autonomous driving control parameter determination subunit is used to determine the control parameters whose control error data is less than the error threshold as the autonomous driving control parameters of the vehicle.

[0176] As another optional implementation, the autonomous driving control parameter determination unit includes:

[0177] The discreteness value determination subunit is used to determine the discreteness value of the control parameters in the control parameter set when each control error data meets the preset control conditions.

[0178] The autonomous driving control parameter determination subunit is used to determine the autonomous driving control parameters of the vehicle based on the control parameters in the control parameter set when the dispersion value is less than the dispersion threshold.

[0179] In some optional implementations, the autonomous driving control parameter determination subunit includes:

[0180] The average value of the control parameters determines the molecular unit, which is used to determine the average value of the control parameters in the control parameter set when the dispersion value is less than the dispersion threshold.

[0181] The autonomous driving control parameter determination molecule is used to take the average value of the control parameters as the autonomous driving control parameters of the vehicle.

[0182] The autonomous driving control parameter determination device and the autonomous driving control parameter determination method embodiments in this application are based on the same application concept. For the specific implementation of the autonomous driving control parameter determination device, please refer to the implementation method of the autonomous driving control parameter determination method, which will not be repeated here.

[0183] This application also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction or at least one program. The processor loads and executes the autonomous driving control parameter determination method as described above.

[0184] The methods and embodiments provided in this application can be executed in an in-vehicle computer terminal, such as an ECU (Electronic Control Unit). The in-vehicle computer terminal includes a processor and a memory. Optionally, the processor of the in-vehicle computer terminal includes, but is not limited to, a microcontroller unit (MCU), a system-on-chip (SOC), etc. Optionally, the memory of the in-vehicle computer terminal can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for functions, etc.; the data storage area may store data created according to the use of the terminal, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0185] This application also provides a computer-readable storage medium storing at least one instruction or at least one program, wherein the at least one instruction or at least one program is loaded and executed by a processor to implement the autonomous driving control parameter determination method as described above.

[0186] In this embodiment, the computer storage medium can be located in at least one of multiple network servers in a computer network. Optionally, the computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), solid-state drives (SSDs), or optical discs, etc. The random access memory may include resistive random access memory (ReRAM) and dynamic random access memory (DRAM).

[0187] This application also provides a vehicle, which includes the autonomous driving control parameter determination device described above.

[0188] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0189] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0190] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0191] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An automatic driving control parameter determination method characterized by comprising: The method comprises: In the case of driving object auxiliary control, acquiring state data of the vehicle in automatic driving; inputting the state data into a control parameter determination model to obtain a control parameter output by the control parameter determination model; wherein the control parameter determination model is obtained by iteratively training an initial control parameter determination model based on updated training sample data; the updated training sample data is determined based on sample state data, sample control parameters corresponding to the sample state data, and sample optimization control data; the sample optimization control data is used to evaluate the automatic driving performance of the vehicle under sample predicted control parameters, which are obtained by predicting the updated training sample data based on the initial control parameter determination model; determining vehicle control data for controlling the vehicle according to the control parameter; controlling the vehicle according to the vehicle control data to obtain position data of the vehicle; acquiring expected trajectory data of the vehicle; determining control error data corresponding to the control parameter according to the position data and the expected trajectory data; in the case that the control error data meets a preset control condition, determining an automatic driving control parameter of the vehicle according to the control parameter.

2. The method of claim 1, wherein, The method further comprises: acquiring correction control data; the correction control data is used to correct the vehicle control data; determining optimization control data corresponding to the control parameter based on the correction control data and the control error data; adding the state data, the control parameter corresponding to the state data, and the optimization control data corresponding to the control parameter to initial training sample data to obtain updated training sample data; inputting the updated training sample data into the initial control parameter determination model for control parameter prediction training to obtain the sample predicted control parameter; adjusting model parameters of the initial control parameter determination model based on the difference information between the sample predicted control parameter and the sample control parameter until a preset training end condition is met to obtain the control parameter determination model.

3. The method of claim 1, wherein, The control error data comprises current control error data corresponding to a current control parameter; the time corresponding to the current control parameter is a current time; in the case that the control error data meets a preset control condition, determining an automatic driving control parameter of the vehicle according to the control parameter comprises: in the case that the current control error data meets a preset control condition, acquiring a control parameter set from the current time to a future preset time period; the control parameter set comprises a plurality of control parameters; determining the control error data corresponding to each control parameter; in the case that each control error data meets a preset control condition, determining an automatic driving control parameter of the vehicle according to the control parameters in the control parameter set.

4. The method of claim 3, wherein, in the case that each control error data meets a preset control condition, determining an automatic driving control parameter of the vehicle according to the control parameters in the control parameter set comprises: In a case where each of the control error data satisfies a preset control condition, determining, from the control parameter set, the control parameter with the control error data less than an error threshold value; determining, from the control parameter set, the control parameter with the control error data less than an error threshold value as the automatic driving control parameter of the vehicle.

5. The method of claim 3, wherein, In a case where each of the control error data satisfies a preset control condition, determining, from the control parameter set, the control parameter with the control error data less than an error threshold value as the automatic driving control parameter of the vehicle. In a case where each of the control error data satisfies a preset control condition, determining, from the control parameter set, the control parameter with the control error data less than an error threshold value as the automatic driving control parameter of the vehicle. In a case where each of the control error data satisfies a preset control condition, determining, from the control parameter set, the control parameter with the control error data less than an error threshold value as the automatic driving control parameter of the vehicle.

6. The method of claim 5, wherein, In a case where each of the control error data satisfies a preset control condition, determining, from the control parameter set, the control parameter with the control error data less than an error threshold value as the automatic driving control parameter of the vehicle. The device comprises: a state data acquisition module configured to acquire state data of a vehicle in automatic driving in a case of driving object auxiliary control; 7. An automatic driving control parameter determination apparatus characterized by comprising: a control parameter determination module configured to input the state data into a control parameter determination model to obtain control parameters output by the control parameter determination model, wherein the control parameter determination model is obtained by iteratively training an initial control parameter determination model based on updated training sample data, the updated training sample data is determined based on sample state data, sample control parameters corresponding to the sample state data, and sample optimization control data, the sample optimization control data is used to evaluate automatic driving performance of the vehicle under sample prediction control parameters, the sample prediction control parameters are obtained by predicting the updated training sample data based on the initial control parameter determination model; a control error data determination module configured to determine vehicle control data for controlling the vehicle based on the control parameters, control the vehicle based on the vehicle control data to obtain position data of the vehicle, acquire expected trajectory data of the vehicle, and determine control error data corresponding to the control parameters based on the position data and the expected trajectory data; an automatic driving control parameter determination module configured to determine, in a case where the control error data satisfies a preset control condition, the automatic driving control parameter of the vehicle based on the control parameters. The device comprises a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor to determine the automatic driving control parameter as claimed in any one of claims 1-6. ​ 8. An electronic device, comprising: ​ 9. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the automatic driving control parameter determination method according to any one of claims 1-6.

10. A vehicle characterized by comprising: The vehicle comprises the automatic driving control parameter determination apparatus according to claim 7.

Images