A vehicle driving state estimation system and method based on deep reinforcement learning

Abstract

The present invention relates to the field of vehicle dynamics technology, and discloses a vehicle driving state estimation system and method based on deep reinforcement learning, wherein a data source module of the system obtains the vehicle driving state quantity, and outputs the obtained Class I state quantity to a deep reinforcement learning agent, and outputs the Class II state quantity to an estimation error evaluator; the deep reinforcement learning agent receives the Class I state quantity, calculates the Class II state quantity estimation value through a neural network, and outputs it to the estimation error evaluator, the estimation error evaluator compares the received Class II state quantity with the Class II state quantity estimation value, generates a reward value and outputs it to the deep reinforcement learning agent; the deep reinforcement learning agent adjusts and optimizes the internal deep neural network parameters according to the reward value, and adjusts the calculation of the Class II state quantity estimation value. The present invention can estimate the driving state quantity that is difficult to directly measure on the vehicle, and the function is reliable and easy to implement.

Description

Technical Field

[0001] The present invention relates to the cross-technical field of vehicle dynamics and artificial intelligence, and more specifically, to a vehicle driving state estimation system and method based on deep reinforcement learning. Background Art

[0002] The driving state of a vehicle is the basis of vehicle motion control, and the estimation of key driving states has always been a research hotspot in the field of vehicle dynamics. With the increasing application of advanced driver assistance systems (ADAS) and the increasing importance of vehicle safety in autonomous driving technology, obtaining accurate driving state quantities of the vehicle is particularly important for vehicle active safety research. There are many driving states of real vehicles, which can usually be obtained using on-board sensors. However, due to the cost of sensors, the use environment, and the complexity of the measurement method, there are still many driving state quantities that are difficult to obtain directly using existing conventional mass-produced on-board sensors.

[0003] In the planar motion state of the vehicle, the longitudinal speed, lateral speed and yaw rate are the key factors affecting the vehicle's handling performance and motion stability. Currently, the commonly used vehicle-mounted measuring instruments mainly include inertial measurement units (IMUs) and vehicle-mounted GPS. Among them, IMU can accurately calculate the vehicle's speed information by detecting the longitudinal acceleration, lateral acceleration, yaw rate and other information of the vehicle, but the existence of cumulative errors will lead to a decrease in accuracy. In theory, the vehicle-mounted GPS module can continuously measure the longitudinal speed and lateral speed with high precision, but due to factors such as terrain and weather, the vehicle-mounted GPS module usually cannot guarantee a continuous and good satellite signal quality, and there are unstable factors; on the other hand, the refresh rate of the vehicle-mounted GPS module signal is too low, and the vehicle speed update cycle is generally in the second level, which is difficult to meet the requirements of high real-time control, especially under extreme working conditions that affect the safety of vehicle driving, it is often difficult to provide timely state feedback for vehicle body stability control. Therefore, obtaining limited vehicle state quantities through conventional mass-produced vehicle-mounted sensors and making real-time and accurate estimates of vehicle driving states such as longitudinal speed and lateral speed are research and application fields that have been widely concerned by the automotive industry.

[0004] At present, in the field of vehicle driving state estimation, experts and scholars have carried out a lot of research work on the accurate estimation of the longitudinal velocity and lateral velocity (or sideslip angle) of the vehicle's center of mass, and proposed some estimation methods. The traditional method is based on the vehicle's dynamic model (model-driven method), and estimates the unknown state quantity using the state quantity that can be collected by mass-produced vehicle sensors. This estimation method often relies on the accuracy of the model, and most models are difficult to guarantee the accuracy of the estimation under extreme conditions. Among them, the more representative technology is the Kalman filter and its variants (such as extended Kalman filter, unscented Kalman filter, cubature Kalman filter, federal Kalman filter, etc.). However, the conditions for the full applicability of Kalman filtering are: linear system, Gaussian white noise. For the typical nonlinear system of vehicle dynamics, it can only be compensated by improved methods such as extended Kalman filtering and unscented Kalman filtering. The more complex the vehicle's driving conditions and the more variable the road conditions, the worse the accuracy of the Kalman filtering method. If the vehicle system state and the noise in the measurement process do not meet the Gaussian white noise conditions, the Kalman filtering method may fail completely.

[0005] Another method is the estimation method based on measured data (data-driven method), which usually uses neural networks and other means, based on deep learning, supervised learning and other methods, to fit the mapping relationship between the vehicle state quantity that needs to be estimated and the vehicle state quantity that is easy to directly measure, and obtain the empirical calculation formula for vehicle driving state estimation. The main process is to first train the neural network based on the test data obtained from the actual vehicle test, and then verify the effect of the trained neural network through the actual vehicle. This training mode is mainly offline, and has high requirements and strong dependence on the existing measured database. In addition, with the changes in vehicle parameters (load, tire pressure, tire wear, etc.), the fitting relationship based on the original measured data may change, resulting in inaccurate estimation of the vehicle driving state. Summary of the invention

[0006] The purpose of the present invention is to address the technical problems existing in the prior art and to provide a vehicle driving state estimation system and method based on deep reinforcement learning, which can estimate the driving state quantities that are difficult to measure directly on the vehicle, as well as estimate the driving state quantities of special vehicles under other complex road conditions and special operating conditions.

[0007] In order to solve the above problems, the technical solution adopted by the present invention is:

[0008] The present invention provides a vehicle driving state estimation system based on deep reinforcement learning, the system comprising: a deep reinforcement learning agent, an estimation error evaluator and a data source module;

[0009] The data source module obtains the vehicle driving state quantity, and outputs the obtained type I state quantity to the deep reinforcement learning agent, and outputs the type II state quantity to the estimation error evaluator;

[0010] The deep reinforcement learning agent receives the type I state quantity, calculates through a neural network to obtain an estimated value of the type II state quantity, and outputs it to the estimation error evaluator;

[0011] The estimation error evaluator compares the received Class II state quantity with the Class II state quantity estimate, generates a reward value and outputs it to the deep reinforcement learning agent; the deep reinforcement learning agent adjusts and optimizes the internal deep neural network parameters according to the reward value, and then adjusts the calculation of the Class II state quantity estimate.

[0012] Furthermore, the data source module includes a vehicle dynamics simulation model, an experimental test prototype vehicle and a mass-produced vehicle, and the simulation results of the vehicle dynamics simulation model output Class I state quantities and Class II state quantities; Class I state quantities are obtained through sensors on the experimental test prototype vehicle, and Class II state quantities are obtained through testing equipment; Class I state quantities are obtained through sensors on the mass-produced vehicle, and Class II state quantities are obtained through a vehicle-mounted GPS positioning system.

[0013] Furthermore, the deep reinforcement learning agent includes a policy function neural network, a value function neural network and a comparator;

[0014] The policy function neural network receives the type I state quantity output by the data source module, calculates the estimated value of the type II state quantity, and outputs it to the error evaluator and the value function neural network respectively;

[0015] The value function neural network receives the Class I state quantity output by the data source module and the Class II state quantity estimation value output by the strategy function neural network, calculates to obtain the estimated effect evaluation value, and outputs it to the comparator;

[0016] The comparator compares the reward value output by the estimation error evaluator with the estimated effect evaluation value output by the value function neural network to form a first correction action acting on the value function neural network and a second correction action acting on the policy function neural network.

[0017] Furthermore, the estimation error evaluator generates an evaluation function expression of the reward value, as shown in formula (1) or formula (2):

[0018] (1)

[0019] (2)

[0020] in, is the estimated value of the type II state quantity, x i is the type II state quantity; k is the number of all state quantities to be estimated; ε i is the weight coefficient of the estimation error of each state quantity to be estimated, ε i The larger the value is, the lower the tolerance for the estimation error of the i-th state quantity to be estimated is.

[0021] Further, the policy function neural network includes a normalization processing layer, an input layer, a first fully connected layer, a first activation function layer, a recurrent neural network layer, a second fully connected layer, a second activation function layer and a scaling layer connected in sequence;

[0022] The normalization processing layer receives the type I state quantity, performs normalization processing on it and then inputs it into the input layer. After sequentially passing through the linear transformation of the first fully connected layer and the nonlinear calculation of the first activation function layer, it is input into the recurrent neural network layer for simulation learning. After passing through the linear change of the second fully connected layer and the nonlinear calculation of the second activation function layer, the scaling layer outputs the estimated value of the type II state quantity.

[0023] Further, the value function neural network includes two groups of normalization processing layers, an input layer, a first fully connected layer and a first activation function layer, and an addition layer, a recurrent neural network layer, a second fully connected layer, a second activation function layer and a scaling layer;

[0024] The two groups of normalization processing layers receive the estimated values ​​of the type I state quantity and the type II state quantity respectively, and are input by the corresponding input layer after normalization processing respectively. After the linear transformation of the first fully connected layer and the nonlinear calculation of the first activation function layer, they are summarized by the addition layer and output to the recurrent neural network for simulation learning. After the linear transformation of the second fully connected layer and the nonlinear calculation of the second activation function layer, the scaling layer outputs the estimated effect evaluation value.

[0025] Furthermore, the expression of the comparator is as follows:

[0026]

[0027] Among them, Reward represents the reward value output by the estimation error evaluator, J represents the estimated effect evaluation value output by the value function neural network, and y is the output result of the comparator.

[0028] The present invention also provides a vehicle driving state estimation method based on deep reinforcement learning, and the specific steps of the estimation method include the following:

[0029] The data source module obtains the vehicle driving state quantity, including the Class I state quantity and the Class II state quantity;

[0030] Output the type I state quantity to the policy function neural network and the value function neural network respectively, and output the type II state quantity to the error evaluator;

[0031] The policy function neural network calculates the estimated value of the type II state quantity according to the type I state quantity, and outputs it to the error evaluator and the value function neural network respectively;

[0032] The value function neural network receives the estimated values ​​of the type I state quantity and the type II state quantity, calculates to obtain an estimated effect evaluation value, and outputs the estimated effect evaluation value to the comparator;

[0033] The comparator compares the reward value and the estimated effect evaluation value to form a first correction action acting on the value function neural network and a second correction action acting on the strategy function neural network, thereby completing the adjustment of the internal parameters of the strategy function neural network and the value function neural network.

[0034] Furthermore, the data source module obtains the vehicle driving state quantity, specifically including:

[0035] In the vehicle dynamics model simulation stage, a vehicle dynamics model of the target vehicle is constructed, and the first type state quantity and the second type state quantity of the vehicle dynamics model are generated;

[0036] In the test vehicle prototype stage, the Class I state quantity is acquired through the sensors on the mass-produced vehicle, and the Class II state quantity is acquired through the test equipment test;

[0037] During the use stage of mass-produced vehicles, Class I state quantities are obtained through the collection of sensors on the mass-produced vehicles, and Class II state quantities are obtained through the positioning system.

[0038] Furthermore, in the vehicle dynamics model simulation stage and the experimental test vehicle stage, thresholds of the reward values ​​are set respectively. When the output reward value of the Class II state quantity estimated value and the Class II state quantity is calculated by the estimation error evaluator and is greater than the corresponding threshold, the current stage of learning and training is terminated.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The present invention is based on the framework of deep reinforcement learning. Through the deep reinforcement learning agent, the estimation error evaluator and the data source module, it realizes the sustainable strong real-time interaction between the deep neural network learning process and the real vehicle environment and the usage process. Through online interaction and feedback, it accelerates the training process of the deep reinforcement learning agent and realizes the estimation of driving state quantities that are difficult to measure on mass-produced vehicles. The overall structure is simple, the function is reliable and easy to implement.

[0041] In addition, the present invention can reduce the demand and dependence of traditional machine learning paradigms such as pure deep learning and supervised learning on the amount of existing data samples. With a relatively small real-vehicle training database, better driving state estimation training effects can be obtained, and the vehicle driving state estimation can be continuously updated and optimized during the long-term use of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the scheme of the present invention, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0043] Figure 1 This is a schematic diagram of the vehicle driving state estimation system based on deep reinforcement learning of the present invention.

[0044] Figure 2 It is a schematic diagram of the vehicle dynamics model simulation stage system of the present invention.

[0045] Figure 3 This is a schematic diagram of the system for the test prototype stage of the present invention.

[0046] Figure 4 It is a schematic diagram of the system for the use stage of mass-produced vehicles of the present invention.

[0047] Figure 5 It is the principle diagram of the strategy function neural network of the present invention.

[0048] Figure 6 Schematic diagram of the value function neural network of the present invention.

[0049] Figure 7 This is a flow chart of a vehicle driving state estimation method based on deep reinforcement learning of the present invention.

[0050] Figure 8 It is a flow chart of the learning and training in three stages in the present invention. DETAILED DESCRIPTION

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field of the present invention; the terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. For example, the directions or positions indicated by the terms "length", "width", "up", "down", "left", "right", "front", "back", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. are based on the directions or positions shown in the drawings, which are only for the convenience of description and should not be understood as limitations on the present technical solution.

[0052] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions; the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects rather than to describe a specific order. In the specification and claims of the present invention and the above-mentioned drawings, when an element is referred to as being "fixed to" or "mounted on" or "disposed on" or "connected to" another element, it may be directly or indirectly located on the other element. For example, when an element is referred to as being "connected to" another element, it may be directly or indirectly connected to the other element.

[0053] In addition, reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0054] See also Figure 1 As shown, the present invention provides a vehicle driving state estimation system based on deep reinforcement learning, the estimation system comprising: a deep reinforcement learning agent A, an estimation error evaluator B and a data source module C;

[0055] The data source module C obtains the vehicle driving state quantity, and outputs the observable state quantity, i.e., the Class I state quantity, to the deep reinforcement learning agent A, and outputs the unmeasurable state quantity, i.e., the Class II state quantity, to the estimation error evaluator B; specifically, the Class I state quantity is the state quantity that is easy to directly obtain on a mass-produced vehicle, such as longitudinal acceleration, lateral acceleration, yaw angular velocity, wheel speed, steering wheel angle (or wheel steering angle), etc.; the Class II state quantity is the state quantity that is difficult to measure accurately and in real time on a mass-produced vehicle through conventional on-board sensors, such as longitudinal vehicle speed, lateral vehicle speed, wheel slip rate, tire sideslip angle, etc., but can be directly obtained in vehicle dynamics simulation, or measured on a test prototype vehicle through professional test equipment.

[0056] The deep reinforcement learning agent A receives the type I state quantity provided by the data source module C, performs network learning to obtain the estimated value of the type II state quantity, and outputs it to the estimation error evaluator B;

[0057] The estimation error evaluator B compares the received Class II state quantity with the estimated value of the Class II state quantity, generates a reward value 4 and outputs it to the deep reinforcement learning agent A; the deep reinforcement learning agent A adjusts and optimizes the internal deep neural network parameters according to the reward value 4, so that the output estimated value of the Class II state quantity is as close as possible to the Class II state quantity, i.e., the true value.

[0058] For further information, see Figures 2 to 4 As shown, the data source module C includes a vehicle dynamics simulation model C1, a test sample vehicle C2, and a mass-produced vehicle C3, wherein the Class I state quantity and the Class II state quantity provided by the vehicle dynamics simulation model C1 are both derived from the simulation results of the simulation model. The Class I state quantity provided by the test sample vehicle C2 is derived from conventional mass-produced sensors and controllers installed on the test sample vehicle C2, and the Class II state quantity is derived from the dedicated high-precision test equipment installed on the test sample vehicle C2 during the test phase, such as high-precision GPS inertial navigation measurement equipment, five-wheel instrument, etc. The Class I state quantity provided by the mass-produced vehicle C3 is derived from conventional mass-produced sensors and controllers installed on the mass-produced vehicle C3, and the Class II state quantity is derived from a conventional vehicle-mounted GPS module. When the vehicle-mounted GPS module has a good signal, it can provide a Class II state quantity with a lower update frequency.

[0059] In this embodiment, the deep reinforcement learning agent A can be implemented by a variety of existing algorithms, including: deep Q network algorithm (DQN), deep deterministic policy gradient algorithm (DDPG), proximal policy optimization algorithm (PPO), Actor-Critic algorithm, etc. The above algorithms can all be used for vehicle driving state estimation in the present invention.

[0060] Furthermore, the deep reinforcement learning agent A includes a policy function neural network A1, a value function neural network A2 and a comparator A3, and the policy function neural network A1 and the value function neural network A2 are both built based on a recurrent neural network (RNN). The recurrent neural network is a commonly used network for time series prediction, which has data memory characteristics and can eliminate the time lag characteristics of vehicle status data in the real-time system acquisition process.

[0061] The strategy function neural network A1 receives the type I state quantity output by the data source module C, calculates the estimated value of the state quantity to be estimated, that is, the estimated value of the type II state quantity, and outputs it to the error evaluator B and the value function neural network A2 respectively.

[0062] The value function neural network A2 receives the estimated values ​​of the Class I state quantity and the Class II state quantity output by the data source module C, calculates to obtain the estimated effect evaluation value 5, and outputs the estimated effect evaluation value 5 to the comparator A3.

[0063] The comparator A3 compares the reward value 4 output by the estimation error evaluator B and the estimated effect evaluation value 5 output by the value function neural network A2, thereby forming a first correction action 6 acting on the value function neural network A2 and a second correction action 7 acting on the policy function neural network A1, thereby adjusting the internal parameters of the policy function neural network A1 and the value function neural network A2.

[0064] Furthermore, the estimation error evaluator B generates a reward value 4 according to the evaluation function and feeds it back to the deep reinforcement learning agent A to evaluate the estimation accuracy of the trained neural network, so that the deep reinforcement learning agent A can adjust and correct the internal neural network parameters and continuously improve the estimation accuracy. Specifically, the evaluation function expression for calculating the reward value 4 can be designed as formula (1) or formula (2):

[0065] (1)

[0066] (2)

[0067] in, is the estimated value of the type II state quantity, calculated by the deep reinforcement learning agent A; x i is the true value of the type II state quantity, provided by the data source module C; k is the number of all state quantities to be estimated; ε i is the weight coefficient of the estimation error of each state quantity to be estimated, ε i The larger the value is, the lower the tolerance for the estimation error of the i-th state quantity to be estimated is, that is, the higher the estimation accuracy requirement is. From formula (1) and formula (2), it can be seen that the larger the estimation error is, the smaller the reward value is.

[0068] For further information, see Figure 5 As shown, the strategy function neural network A1 includes a normalization processing layer, an input layer, a first fully connected layer, a first activation function layer, a recurrent neural network layer, a second fully connected layer, a second activation function layer and a scaling layer connected in sequence. The normalization processing layer receives the type I state quantity, which is input by the input layer after normalization processing, and is sequentially input into the recurrent neural network layer after linear transformation of the first fully connected layer and nonlinear calculation of the first activation function layer for simulation learning. After linear transformation of the second fully connected layer and nonlinear calculation of the second activation function layer, the scaling layer outputs the estimated value of the type II state quantity.

[0069] See also Figure 6As shown, the value function neural network A2 includes two groups of normalization processing layers, input layers, first fully connected layers and first activation function layers connected in sequence, addition layers connected to the two groups of first activation function layers, recurrent neural network layers connected in sequence to the addition layers, second fully connected layers, second activation function layers and scaling layers. The two groups of normalization processing layers receive the estimated values ​​of the type I state quantity and the type II state quantity respectively, and are input by the corresponding input layers after normalization processing respectively. After the linear transformation of the first fully connected layer and the nonlinear calculation of the first activation function layer respectively, they are summarized by the addition layer and output to the recurrent neural network for simulation learning. After the linear transformation of the second fully connected layer and the nonlinear calculation of the second activation function layer, the scaling layer outputs the estimated effect evaluation value.

[0070] Specifically, the normalization processing layer of the strategy function neural network A1 receives the state quantity of the first category, normalizes the values ​​of each variable therein to improve the calculation accuracy and training efficiency of the neural network, and inputs it to the first fully connected layer through the corresponding input layer. Since the input of the neural network is the state quantity of the vehicle during driving, the driving state quantity observed by conventional on-board sensors, and its value range can be pre-estimated according to the driving limit of the vehicle, the normalization processing is performed based on the reasonable upper and lower limits of each state quantity.

[0071] The two groups of normalization processing layers of the value function neural network A2 receive the output values ​​of the type I state quantity and the strategy function neural network A1, i.e., the estimated value of the type II state quantity, respectively, perform normalization processing, and input them into the first fully connected layer through the corresponding input layer. The configuration of the input layer mainly depends on the number of input quantities of its neural network, i.e., the number of input layers can be adjusted according to actual needs.

[0072] The first fully connected layer performs linear transformation on all input data through full connection with the input layer, and outputs the calculation result to the first activation function layer.

[0073] The function of the first activation function layer is to introduce nonlinear computational features into the neural network and perform nonlinear computational processing on the linear computational results of the fully connected layer, so that the neural network can perform a preliminary simulation of the complex nonlinear computational features of the vehicle dynamics system.

[0074] The function of the addition layer is to summarize the two input values ​​of the value function neural network A2, namely the type I state quantity and the type II state quantity estimation value, and output them to the recurrent neural network.

[0075] The recurrent neural network layer (RNN) is a neural network that takes sequence data as input and performs recursion in the direction of sequence evolution. Its memory enables it to achieve better results when learning the nonlinear characteristics of the sequence, that is, to simulate the nonlinear calculation characteristics with time characteristics to obtain a nonlinear mapping relationship between the input sequence and the output sequence. Since all state quantities of vehicle dynamics are sequence data that change continuously over time, RNN is used to describe the nonlinear state equations and state transfer functions of vehicle dynamics. Based on the above advantages of RNN, recurrent neural network layers are added to the strategy function neural network A1 and the value function neural network A2 respectively. Each layer contains a number of nodes, and the number of nodes can be selected according to the specific application scenario.

[0076] The second fully connected layer performs a linear transformation on all data output by the recurrent neural network, and outputs the calculation results to the second activation function layer.

[0077] The second activation function layer performs nonlinear feature calculation on the data output by the second fully connected layer to obtain a deep simulation result of the vehicle dynamics system characteristics.

[0078] The main function of the scaling layer is to scale the output value of the second activation function layer according to the actual possible value range of the state quantity to be estimated that is finally output by the second activation function layer. Its main parameters include the proportional gain and bias of the scaling layer. The scaling layer of the strategy function neural network A1 outputs the estimated value of the second type of state quantity, and the scaling layer of the value function neural network A2 outputs the estimated effect evaluation value.

[0079] In this embodiment, the main bodies of the strategy function neural network A1 and the value function neural network A2 are fully connected layers and activation function layers, and several fully connected layers are used in each of them, and each fully connected layer contains several nodes. The number of fully connected layers and the number of nodes in each layer can be selected according to the specific application. The strategy function neural network A1 and the value function neural network A2 both use the advantages of recurrent neural networks in processing multi-input and multi-output nonlinear time series problems to achieve high-accuracy estimation of the vehicle's driving state, and through the combination of multi-layer neural networks and estimation error evaluator B, the algorithm training speed is accelerated, the accuracy of the estimation is improved, and continuous online learning and training can be achieved in the mass production stage.

[0080] Furthermore, the comparator A3 is used to evaluate the error between two inputs, namely, the reward value 4 output by the estimation error evaluator B and the estimated effect evaluation value 5 output by the value function neural network A2. In this embodiment, the expression of the comparator A3 is as follows:

[0081]

[0082] Among them, Reward represents the reward value 4 output by the estimation error evaluator B, J represents the estimated effect evaluation value 5 output by the value function neural network A2, and y is the output result of the comparator A3. The output result of the comparator A3 represents the deviation between the reward value 4 and the estimated effect evaluation value 5. The smaller the output result, the closer the result between the reward value 4 and the estimated effect evaluation value 5 is, and the more ideal the learning result of the deep reinforcement learning agent A is.

[0083] See also Figure 7 As shown, the present invention also provides a vehicle driving state estimation method based on deep reinforcement learning, which is respectively applied to the vehicle dynamics model simulation stage, the test prototype stage, and the mass production vehicle use stage, and specifically includes the following:

[0084] Step S1: The data source module C obtains the vehicle driving state quantity, including the Class I state quantity and the Class II state quantity, specifically including:

[0085] a. In the vehicle dynamics model simulation stage, a vehicle dynamics model C1 of the target vehicle is constructed according to simulation software such as Matlab / Simulink and CarSim. The construction parameters of the vehicle dynamics model C1 are selected based on the actual parameters of the target vehicle and the training database, and the first type state quantity and the second type state quantity of the vehicle dynamics model C1 are generated. Specifically, the training database provides a large amount of driving conditions, road conditions and driver operation information of the target vehicle.

[0086] b. During the test vehicle prototype phase, the actual vehicle data of the test vehicle prototype C2 is collected, and signal processing and calculation are performed on the actual vehicle data to obtain the Class I state quantity and Class II state quantity of the test vehicle prototype C2 respectively.

[0087] In this embodiment, since the test sample vehicle C2 is equipped with sensors such as a vehicle-mounted IMU, a wheel speed sensor, a steering wheel angle sensor, and professional test equipment such as a high-precision GPS inertial navigation measurement device and a five-wheel instrument, the test driver operates the test sample vehicle C2 to collect real vehicle data under actual test road conditions. The sensors provide Class I state quantities, and the professional test equipment provides accurate measurement values ​​of Class II state quantities, which are used as true values.

[0088] c. During the mass-production vehicle use stage, the driving data of the mass-production vehicle C3 is collected during use, and the driving data is preliminarily processed and solved to obtain the Class I state quantity and Class II state quantity of the mass-production vehicle stage respectively.

[0089] In this embodiment, since the mass-produced car C3 is equipped with sensors such as an on-board IMU, a wheel speed sensor, a steering wheel angle, and a conventional on-board GPS module and other positioning systems, the driving data of the mass-produced car C3 is collected during the user's driving operation, and the sensor provides the Class I state quantity, and the conventional on-board GPS positioning system provides the Class II state quantity. However, since the conventional on-board GPS module has a long update cycle for the longitudinal speed and the lateral speed (usually in the order of seconds), and is easily affected by the terrain, weather, etc. during driving, the GPS signal is lost. Therefore, when the on-board GPS module signal is good, the Class II state quantity provided is accurate, that is, the true value, which can be used for the training of the deep reinforcement learning agent, while when the GPS signal is weak, the Class II state quantity provided is inaccurate and needs to be estimated by the deep reinforcement learning agent A.

[0090] In this embodiment, the vehicle driving state quantity provided by the data source module C can be derived from the vehicle dynamics model or from real vehicle collection. Since the parameter collection of the vehicle physical model often depends on real vehicle testing, limited by the test equipment and test costs, the amount of real vehicle data is often difficult to meet the training sample volume requirements of deep reinforcement learning. The vehicle dynamics model C1 of this embodiment is simple and fast to build, and simulation data can be obtained according to demand, but it is difficult to accurately simulate the state of the real vehicle by relying solely on simulation data. Therefore, combining simulation data with real vehicle data can achieve complementary advantages and improve the credibility of the model.

[0091] Step S2: Output the type I state quantity to the policy function neural network A1 and the value function neural network A2 in the deep reinforcement learning agent A, and output the type II state quantity to the error evaluator B. Specifically, in the vehicle dynamics model simulation stage and the experimental test prototype stage, the corresponding type II state quantity is the true value.

[0092] Step S3: The strategy function neural network A1 calculates the estimated value of the state quantity to be estimated, i.e., the estimated value of the type II state quantity, based on the type I state quantity, and outputs it to the error evaluator B and the value function neural network A2 respectively.

[0093] Step S4: The value function neural network A2 receives the estimated values ​​of the Class I state quantity and the Class II state quantity, calculates to obtain the estimated effect evaluation value, and outputs the estimated effect evaluation value to the comparator A3.

[0094] Step S5: the error evaluator B calculates and generates a reward value based on the received type II state quantity and the type II state quantity estimation value, and outputs the reward value to the comparator A3.

[0095] Step S6: The comparator A3 compares the reward value output by the error evaluator B with the estimated effect evaluation value output by the value function neural network A2, thereby forming a first correction action acting on the value function neural network A2 and a second correction action acting on the strategy function neural network A1, thereby adjusting the internal parameters of the strategy function neural network A1 and the value function neural network A2.

[0096] In this embodiment, in the vehicle dynamics model simulation stage and the test sample stage, reward value thresholds Qth1 and Qth2 are set respectively. When the estimated value of the Class II state quantity obtained by the deep reinforcement learning agent A and the Class II state quantity are calculated by the estimation error evaluator B, and the output reward value is greater than the threshold Qth, the current stage of learning and training is terminated and the next stage of learning and training is entered. That is, when the reward value output in the vehicle dynamics model simulation stage is greater than the threshold Qth1, the training process is terminated and the test sample stage is entered. After completing the training of the test sample stage, the deep reinforcement learning agent A achieves a higher estimation accuracy of the vehicle driving state quantity; when the reward value output in the test sample stage is greater than the threshold Qth2, the training process is terminated and the mass production vehicle use stage is entered. Since the learning and training of the deep reinforcement learning agent A in the mass production vehicle stage is a long-term and continuous process, no reward value threshold is set for the end.

[0097] In this embodiment, since the vehicle operating conditions that can be covered by the vehicle dynamics model C1 are more flexible and complex, the vehicle driving state estimation is mainly based on the learning and training in the vehicle dynamics model simulation stage. For special conditions that cannot be achieved by the test sample vehicle C2, the training method in the vehicle dynamics model simulation stage is used for supplementary learning. This can accelerate the learning and training process of the deep reinforcement learning agent A and complete a reliable estimation of the vehicle driving state.

[0098] In this embodiment, refer to Figure 8As shown in the figure, the training process of the vehicle dynamics model simulation stage and the test vehicle stage needs to go through multiple iterations until a satisfactory training result is achieved. Through multiple rounds of iterative training of the vehicle dynamics model C1 and the test vehicle C2 data, that is, the Class I state quantity and Class II state quantity are obtained through the simulation results of the vehicle dynamics model C1, and then the accurate Class I state quantity and Class II state quantity are obtained through the sensors and test equipment of the test vehicle C2. The deep reinforcement learning agent A is trained and optimized according to the corresponding Class I state quantity and Class II state quantity, so that the vehicle driving state quantity estimation accuracy of the optimized deep reinforcement learning agent A can meet the requirements of the mass production vehicle C3 stage, that is, the deep reinforcement learning agent A that has undergone multiple iterations of training has an estimated value of the Class II state quantity close to the actual value in the mass production vehicle C3 stage, which ensures the accuracy of the estimation accuracy. During the mass-production vehicle usage phase, the deep reinforcement learning agent A is continuously trained based on the vehicle operation data. Through real-time online training and updating of the deep reinforcement learning agent A, the learning accuracy of the estimation system for low-probability conditions and small sample data during actual vehicle operation is continuously improved.

[0099] The present invention constructs a vehicle driving state estimation system based on the framework of deep reinforcement learning, that is, based on the deep reinforcement learning intelligent agent A, the estimation error evaluator B and the data source module C, the vehicle driving state quantities that are difficult to directly measure, such as the longitudinal vehicle speed, lateral vehicle speed, vehicle center of mass sideslip angle, wheel slip rate, tire sideslip angle, etc., are accurately estimated, and continuous self-learning and self-optimization can be performed in the three stages of vehicle dynamics simulation model simulation, test sample vehicle testing, and mass-produced vehicle user use. When the mass-produced vehicle cannot obtain the accurate Class II state quantity, a relatively accurate estimate of the Class II state quantity can be calculated, and the simulation data of the vehicle dynamics model and the actual vehicle data are fully utilized to improve the estimation accuracy and practical effect.

[0100] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A vehicle driving state estimation system based on deep reinforcement learning, characterized in that: The system includes: a deep reinforcement learning agent, an estimation error evaluator and a data source module; The data source module obtains the vehicle driving state quantity, and outputs the obtained type I state quantity to the deep reinforcement learning agent, and outputs the type II state quantity to the estimation error evaluator; The deep reinforcement learning agent receives the type I state quantity, calculates the type II state quantity through a neural network, and outputs the estimated value to the estimation error evaluator; The estimation error evaluator compares the received Class II state quantity with the Class II state quantity estimate, generates a reward value and outputs it to the deep reinforcement learning agent; the deep reinforcement learning agent adjusts and optimizes the internal deep neural network parameters according to the reward value, and then adjusts the calculation of the Class II state quantity estimate; The deep reinforcement learning agent includes a policy function neural network, a value function neural network and a comparator; The strategy function neural network receives the type I state quantity output by the data source module, calculates the estimated value of the type II state quantity, and outputs it to the error evaluator and the value function neural network respectively; The value function neural network receives the estimated values ​​of the type I state quantity and the type II state quantity output by the data source module, calculates to obtain the estimated effect evaluation value, and outputs it to the comparator; The comparator compares the reward value output by the estimation error evaluator with the estimated effect evaluation value output by the value function neural network to form a first correction action acting on the value function neural network and a second correction action acting on the policy function neural network.

2. The vehicle driving state estimation system based on deep reinforcement learning according to claim 1, characterized in that: The data source module includes a vehicle dynamics simulation model, an experimental test prototype vehicle and a mass-produced vehicle. The simulation results of the vehicle dynamics simulation model output a Class I state quantity and a Class II state quantity. The Class I state quantity of the experimental test prototype vehicle is obtained through the sensor on the experimental test prototype vehicle, and the Class II state quantity is obtained through the test equipment; the Class I state quantity is obtained through the sensor on the mass-produced vehicle, and the Class II state quantity is obtained through the vehicle-mounted positioning system.

3. The vehicle driving state estimation system based on deep reinforcement learning according to claim 1, characterized in that: The estimation error evaluator generates an evaluation function expression of the reward value, as shown in formula (1) or formula (2): (1) (2) in, is the estimated value of the second type of state quantity, It is the type II state quantity; is the number of all state quantities to be estimated; is the weight coefficient of the estimation error of each state quantity to be estimated, The larger the value, the more The lower the tolerance of the estimation error of the state quantity to be estimated.

4. The vehicle driving state estimation system based on deep reinforcement learning according to claim 1, characterized in that: The strategy function neural network includes a normalization processing layer, an input layer, a first fully connected layer, a first activation function layer, a recurrent neural network layer, a second fully connected layer, a second activation function layer and a scaling layer connected in sequence; The normalization processing layer receives the type I state quantity, performs normalization processing on it and then inputs it into the input layer. After sequentially passing through the linear transformation of the first fully connected layer and the nonlinear calculation of the first activation function layer, it is input into the recurrent neural network layer for simulation learning. After passing through the linear transformation of the second fully connected layer and the nonlinear calculation of the second activation function layer, the scaling layer outputs the estimated value of the type II state quantity.

5. The vehicle driving state estimation system based on deep reinforcement learning according to claim 4, characterized in that: The value function neural network includes two groups of normalization processing layers, an input layer, a first fully connected layer and a first activation function layer, an addition layer, a recurrent neural network layer, a second fully connected layer, a second activation function layer and a scaling layer; The two groups of normalization processing layers receive the estimated values ​​of the first type of state quantity and the second type of state quantity respectively, and are input by the corresponding input layer after normalization processing respectively. After linear transformation of the first fully connected layer and nonlinear calculation of the first activation function layer, they are summarized by the addition layer and output to the recurrent neural network for simulation learning. After linear transformation of the second fully connected layer and nonlinear calculation of the second activation function layer, the scaling layer outputs the estimated effect evaluation value.

6. The vehicle driving state estimation system based on deep reinforcement learning according to claim 1, characterized in that: The expression of the comparator is as follows: in, It represents the reward value output by the estimation error evaluator, J represents the estimated effect evaluation value output by the value function neural network, and y is the output result of the comparator.

7. An estimation method for a vehicle driving state estimation system based on deep reinforcement learning according to any one of claims 1 to 6, characterized in that: The specific steps of the estimation method include the following: The data source module obtains the vehicle driving state quantity, including the first category state quantity and the second category state quantity; Output the state quantity of type I to the strategy function neural network and the value function neural network respectively, and output the state quantity of type II to the error evaluator; The policy function neural network calculates the estimated value of the type II state quantity according to the type I state quantity, and outputs it to the error evaluator and the value function neural network respectively; The value function neural network receives the estimated values ​​of the first type state quantity and the second type state quantity, calculates to obtain an estimated effect evaluation value, and outputs the estimated effect evaluation value to the comparator; The comparator compares the reward value and the estimated effect evaluation value to form a first correction action acting on the value function neural network and a second correction action acting on the policy function neural network, thereby completing the adjustment of the internal parameters of the policy function neural network and the value function neural network.

8. The method according to claim 7, characterized in that: The data source module obtains the vehicle driving state quantity, specifically including: In the vehicle dynamics model simulation stage, the vehicle dynamics model of the target vehicle is constructed, and the simulation results output the first type state quantity and the second type state quantity of the vehicle dynamics model; In the test vehicle stage, the Class I state quantity is acquired through the sensors on the test vehicle, and the Class II state quantity is acquired through the test equipment test; During the use stage of mass-produced vehicles, the first type of state quantity is obtained through the collection of sensors on the mass-produced vehicles, and the second type of state quantity is obtained through the positioning system.

9. The method according to claim 8, characterized in that: In the vehicle dynamics model simulation stage and the experimental test vehicle stage, the thresholds of the reward values ​​are set respectively. When the output reward value of the Class II state quantity estimated value and the Class II state quantity is calculated by the estimation error evaluator and is greater than the corresponding threshold value, the current stage of learning and training is terminated.

Images