An adaptive cruise control method and system based on deep reinforcement learning

Abstract

This invention provides an adaptive cruise control method and system based on deep reinforcement learning; a driver following style identification module based on a BP neural network; and an ACC control model based on deep reinforcement learning. The ACC control model includes a strategy module and a deep Q-network module. Driver following behavior is categorized into three following styles and then labeled. The following style identification module uses a BP neural network to train a driver following style classification model, and outputs the driver's following style based on the classification model. The strategy module establishes a Q-value network, which outputs the desired acceleration of the vehicle based on the effective characteristic performance data of the acquired real-time following behavior. This achieves an adaptive cruise control method that is closer to the driver's individual needs and offers high safety, following accuracy, comfort, and economy.

Description

Technical Field

[0001] This invention relates to the field of cruise control method technology, and more specifically, to an adaptive cruise control method and system based on deep reinforcement learning. Background Technology

[0002] Adaptive Cruise Control (ACC) is increasingly used in society. ACC systems effectively reduce the driver's workload, optimize operational precision, and thus improve driving comfort and safety. With the increasing intelligence of vehicles, designing personalized ACC systems has become a key research focus.

[0003] How to implement a personalized ACC system has become a technical challenge in this field. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an adaptive cruise control method and system based on deep reinforcement learning. It utilizes an identification module built upon a BP neural network and establishes an ACC control model based on deep reinforcement learning. This achieves a more personalized adaptive cruise control method based on deep reinforcement learning that is closer to the driver's experience and offers high safety, responsiveness, comfort, and economy.

[0005] One aspect of the present invention provides an adaptive cruise control method based on deep reinforcement learning, the method comprising:

[0006] A pre-defined vehicle following behavior database containing feature performance data of vehicle following behavior is prepared; feature performance data of m vehicle following behaviors are obtained from the vehicle following behavior database and preprocessed to obtain effective feature performance data of m vehicle following behaviors.

[0007] Based on the partial effective feature performance data of the m following behaviors, factor analysis was used to process the data to obtain 7 cluster analysis data corresponding to the m following behaviors.

[0008] Based on the 7 clustering analysis data corresponding to m following behaviors, the K-means algorithm was used to cluster the behaviors to obtain three types of following behaviors, and each of the three types of following behaviors was labeled with a label representing the following style.

[0009] The effective feature performance data of the three types of car-following behaviors after clustering are processed to obtain the expected headway deviation k of the three car-following styles. d The expected range and critical range, and the expected following speed deviation k of the three types of following styles. v The expected range and critical range;

[0010] A following style identification module is established based on a BP neural network. Through the following style identification module, a driver following style classification model is obtained by training a BP neural network based on the labeled following behavior.

[0011] Acquire 7 cluster analysis data corresponding to the real-time vehicle following behavior during vehicle driving, and output the driver's following style through the following style identification module based on the driver following style classification model.

[0012] Preferably, the identification module acquires 7 clustering analysis data points in real time during vehicle operation, and outputs the following style during vehicle operation based on the driver's following style classification model. The specific process is as follows:

[0013] The system acquires and processes real-time vehicle following behavior characteristic performance data during vehicle operation to obtain 7 cluster analysis data corresponding to real-time vehicle following behavior.

[0014] Preferably, factor analysis is used to process the statistical features of the effective performance data of real-time following behavior, extracting the top 7 common factors of the effective performance data of real-time following behavior and the statistical feature coefficients corresponding to each statistical feature in the top 7 common factors of the effective performance data of real-time following behavior; the statistical feature information corresponding to the effective performance data of real-time following behavior is multiplied by the statistical feature coefficients in the common factors corresponding to the effective performance data of real-time following behavior, and the summation is performed to obtain the 7 cluster analysis data for each following behavior;

[0015] Preferably, the driver following style classification model has 2 hidden layers with 10 nodes in each layer. The activation function of the neural network is the sigmoid function, the optimizer is Adam parameter optimization, the loss function is the cross-entropy loss function, and the learning rate is 0.01.

[0016] An ACC control model is established based on deep reinforcement learning. The ACC control model includes a policy module and a deep Q-network module.

[0017] The strategy module establishes a Q-value network based on a pre-learned driving behavior database. Through the Q-value network, the desired acceleration of the vehicle is adjusted based on the effective characteristic performance data of the real-time vehicle following behavior.

[0018] Using a deep Q-network module, the expected headway deviation k is calculated based on three types of following behaviors. d The expected range and critical range, and the expected following speed deviation k of the three types of following styles. vThe expected range and critical range are used to process the effective characteristic performance data of the driver's following style and the real-time vehicle following behavior to obtain the vehicle following process reward. The Q value is deeply optimized based on the vehicle following process reward, and the Q value network is updated to achieve the following behavior to meet the safety, following, comfort and economy goals.

[0019] The three categories of following styles are: aggressive, normal, and cautious.

[0020] Preferably, the establishment of the pre-learning driving behavior database involves processing the feature performance data of different styles of following behavior after clustering and labeling through a strategy module, setting a state space and an action space, and setting the range of the vehicle's speed v0, the range of the vehicle's acceleration a, and the range of the distance between the vehicle and the front of the vehicle d.

[0021] The state space includes the distance between vehicle heads (d) and the desired safe distance (d). des The vehicle's speed v0 and the speed of the vehicle in front v p The motion space includes the vehicle's acceleration 'a';

[0022] The spatial range of each item is discretized; the following behavior is trained to obtain a pre-learning driving behavior database; further optimized, the working conditions of the preceding vehicle are set as three types: constant speed, constant acceleration, and constant deceleration, and training is performed on the three different types of working conditions.

[0023] The Q-value network includes an agent's action set, state set, and a neural network that processes the action set and state set; the action set is the desired acceleration a′; the state set includes the front-end distance d and the desired safety distance d. des The vehicle's speed v0 and the speed of the vehicle in front v p ;

[0024] The Q-value network takes the real-time state and actions of the effective feature performance data of the following behavior as input, and outputs the Q-values ​​corresponding to the real-time state and actions. The strategy module uses the action with the maximum Q-value corresponding to the real-time state of the effective feature performance data of the following behavior as the expected acceleration to be output.

[0025] Furthermore, the effective characteristic performance data includes the vehicle's speed v0 and the speed of the preceding vehicle v. p , vehicle acceleration a, vehicle jerk, front-to-front distance d, front-to-front time distance d t ;

[0026] The partially effective feature performance data in processing m following behaviors are the vehicle's speed v0 and the speed of the vehicle in front v. p , vehicle frontage distance d, vehicle frontage distance d t .

[0027] Furthermore, the 7 cluster analysis data corresponding to the m following behaviors were obtained through the following steps:

[0028] Process the partial effective feature performance data of m following behaviors to obtain the statistical feature information corresponding to the effective feature performance data;

[0029] Factor analysis was used to process the statistical features of m following behaviors, and the first 7 common factors were extracted to obtain the initial component matrix representing the performance of effective features.

[0030] The initial component matrix of the first 7 common factors is rotated using the variance maximization method to obtain the feature matrix representing the performance of the effective features; the statistical feature coefficients corresponding to each statistical feature in each of the 7 common factors are obtained through the feature matrix.

[0031] The statistical feature information corresponding to each following behavior is processed by the statistical feature coefficients of the feature matrix to obtain 7 cluster analysis data corresponding to each following behavior;

[0032] The statistical characteristics include: maximum value, minimum value, mean, median, variance, and standard deviation;

[0033] The statistical feature information corresponding to each following behavior is multiplied by the statistical feature coefficient in the common factor corresponding to each following behavior, and the sum of the data is obtained to obtain the 7 cluster analysis data for each following behavior.

[0034] The preferred method is to label the categorized following behavior and assign physical meaning to each public factor;

[0035] The radical category parameter of each car-following behavior is obtained by multiplying the statistical feature information corresponding to each car-following behavior with the statistical feature coefficient in the common factor representing the radical physical meaning, and then summing the data.

[0036] Arrange all the aggressive category parameters of the driving behavior in descending order to obtain the aggressive sequence. The first 30% of the aggressive sequence corresponds to the aggressive type of driving behavior, the 30%-70% of the aggressive sequence corresponds to the general type of driving behavior, and the 70%-100% of the aggressive sequence corresponds to the cautious type of driving behavior.

[0037] or

[0038] The caution category parameter for each following behavior is obtained by multiplying the statistical feature information corresponding to each following behavior with the statistical feature coefficient in the common factor representing the cautious physical meaning, and then summing the data.

[0039] Arrange all caution category parameters of the driving behavior in descending order to obtain a caution sequence. The first 30% of the caution sequence corresponds to the cautious driving behavior, the 30%-70% of the caution sequence corresponds to the general driving behavior, and the 70%-100% of the aggressive sequence corresponds to the aggressive driving behavior.

[0040] The specific method for assigning physical meaning to each public factor is as follows: when the statistical characteristic corresponding to the statistical characteristic coefficient greater than or equal to 0.7 in each public factor is one or more of the following: maximum velocity, minimum velocity, average velocity, median velocity, maximum acceleration, minimum acceleration, average acceleration, and median acceleration, this public factor is a radical public factor.

[0041] A public factor is considered a cautious public factor when the statistical characteristic corresponding to the statistical characteristic coefficient of ≥0.7 in each public factor is one or more of the following: maximum headway, minimum headway, average headway, median headway, maximum headway distance, minimum headway distance, average headway distance, and median headway distance.

[0042] Furthermore, based on the effective feature performance data of the clustered following behavior, the expected headway deviation k of the following behavior for the three types of following styles is calculated using Formula 1. d The expected following speed deviation k for the three types of following behaviors is calculated using Formula 2. v ;

[0043] The expected headway deviation k for the three types of following behaviors d Fit the distribution to obtain k. d The normal distribution curve fitted by the distribution represents the expected following speed deviation k for the three types of following behaviors. v Fit the distribution to obtain k. v The normal distribution curve fitted to the distribution;

[0044] k d The expected interval is k d The (μ) normal distribution curve d -σ d ,μ d +σ d ) interval, k d The critical interval is k d The (μ) normal distribution curve d -3σ d ,μ d +3σ d ) interval;

[0045] k v The expected interval is kv The (μ) normal distribution curve v -σ v ,μ v +σ v ) interval, k v The critical interval is k v The (μ) normal distribution curve v -3σ v ,μ v +3σ v ) interval.

[0046] Furthermore, effective characteristic performance data during vehicle operation is acquired in real time. The expected headway deviation k during vehicle operation is obtained from the effective characteristic performance data at each moment using Formula 1 and Formula 2. d Expected following speed deviation k v .

[0047] Furthermore, according to claim 4, the adaptive cruise control method based on deep reinforcement learning is characterized in that, formula one is:

[0048] d des =v0·τ+d0;

[0049]

[0050] In the formula, v0 is the vehicle speed, τ is the desired headway, d0 is the minimum safe distance after stopping, preferably d0 = 2m; i = 1, 2, 3 correspond to the three following styles: aggressive, normal, and cautious, respectively. These are speed-related coefficients corresponding to three types of following styles: aggressive, moderate, and cautious; preferably... See the table below for specific values:

[0051] i <![CDATA[α1]]> <![CDATA[α2]]> <![CDATA[α3]]> <![CDATA[α4]]> 1 -4.25e-5 0.0095 -0.2851 3.143 2 -2.13e-5 0.0081 -0.3151 3.871 3 -1.39e-5 0.0158 -0.4657 4.732

[0052]

[0053] In the formula, d is the actual distance between the front ends of the vehicles. des Desired following distance;

[0054] And / or Formula 2 is:

[0055]

[0056] Furthermore, based on the deep Q-network module, ACC system control objectives are constructed, including safety objectives, following objectives, comfort objectives, and economy objectives. Safety objectives and following objectives directly reflect the motion state of the vehicle and are the core objectives of control. Comfort objectives and economy objectives reflect the driver's riding experience and are the constraining objectives of control.

[0057] Furthermore, the safety targets for the three types of following styles are: k corresponding to the three types of following styles. d The expected range;

[0058] The following objectives for the three types of car-following styles are: k corresponding to the three types of car-following styles. v The expected range;

[0059] The comfort targets for the three following styles are as follows: using acceleration jerk as the evaluation index for the comfort of the ACC system, and obtaining the jerk size range corresponding to the three following styles according to the ISO 15622 (2018) standard;

[0060] The economic targets for the three types of following styles are as follows: the vehicle's acceleration 'a' is used as the evaluation index for the economic performance of the ACC system, and the magnitude range of acceleration is specified with reference to the provisions of ISO 15622 (2018) standard.

[0061] Furthermore, the deep Q-network module uses the D3QN algorithm to design the ACC control algorithm. The deep Q-network module includes a reward function based on deep reinforcement learning, which consists of four parts: safety reward, following performance reward, comfort reward, and economy reward. The deep Q-network module processes the effective feature performance data during the vehicle following process using the reward function formula to obtain the vehicle following process reward data.

[0062] The reward function formula is as follows:

[0063]

[0064] In the formula, ω=[ω1,ω2,ω3,ω4] T ω1, ω2, ω3, ω4 are the weighting coefficients for safety, responsiveness, comfort, and economic rewards; r0 = [r safety ,r following ,r comfort ,r economy ] T It is a matrix of reward functions for safety, responsiveness, comfort, and economy;

[0065] The security reward function is:

[0066]

[0067] In the formula, a2, a1, b1, and b2 are k corresponding to the driver's following style, respectively. d The expected interval endpoint μ d -σ d μ d +σ d The driver's following style corresponds to k d The critical interval endpoint μ d -3σ d μ d +3σ d ;

[0068] The following reward function is:

[0069]

[0070] In the formula, c2, c1, d1, and d2 are k corresponding to the driver's following style, respectively. v The expected interval endpoint μ v -σ v μ v +σ v The driver's following style corresponds to k v The critical interval endpoint μ v -3σ v μ v +3σ v ;

[0071] The comfort reward function is:

[0072]

[0073] In the formula, jerk max The maximum value specified for jerk size; the jerk corresponding to the driver's following style. max The maximum value specified for the jerk magnitude of the ACC system was obtained according to ISO 15622 (2018) standard;

[0074] The economic reward function is as follows:

[0075]

[0076] In the formula, a is the vehicle's acceleration at the current moment. max The maximum value specified for size 'a'; the driver's following style corresponding to 'a'. max The maximum value specified for the magnitude of acceleration 'a' in the ACC system is obtained according to ISO 15622 (2018) standard;

[0077] ω = [ω1, ω2, ω3, ω4] can be obtained using the following method. T :

[0078] According to k during the vehicle following process d k during vehicle following v The cruise operation is divided into five types: steady-state following, safe approach, safe distance, dangerous distance and dangerous approach.

[0079] Fuzzy rules are used to switch the weight coefficients under different operating conditions; the fuzzy rules obtain k during the vehicle's driving process. d k during vehicle following v The output is the four weight coefficients of the reward function;

[0080] Preferably, in each simulation step, k during the vehicle following process is first included. d k during vehicle following v Discretized with four weight coefficients, the dataset is divided into seven fuzzy subsets: PB, PM, PS, ZO, NS, NM, and NB. Fuzzy rules are designed according to the S7 design method, and the resulting fuzzy rules are shown in the table below:

[0081] (a) Security weight table

[0082]

[0083]

[0084] (b) Followership weighting table

[0085]

[0086] (c) Comfort weighting table

[0087]

[0088]

[0089] (d) Comfort weighting table

[0090]

[0091] The triangle membership function (trimf function) is selected to transform the input into a fuzzy output. Finally, the centroid method is used to defuzzify the data and obtain specific weight coefficients. The obtained weight coefficients are then standardized to obtain the weight coefficients for safety, responsiveness, comfort, and economic rewards.

[0092] Another aspect of the present invention provides an adaptive cruise control system based on deep reinforcement learning, comprising:

[0093] The database preset and data extraction module is used to preset a following behavior database including the characteristic performance data of following behavior and to obtain the characteristic performance data of m following behaviors from the following behavior database.

[0094] The preprocessing module is used to preprocess the feature performance data of m following behaviors to obtain the effective feature performance data of m following behaviors.

[0095] The cluster analysis data acquisition module is used to process the partial effective feature performance data of the m following behaviors through factor analysis to obtain 7 cluster analysis data corresponding to the m following behaviors.

[0096] The clustering module is used to cluster based on 7 clustering analysis data corresponding to m following behaviors using the K-means algorithm to obtain three types of following behaviors, and then label the three types of following behaviors after clustering with labels representing following styles.

[0097] The module for obtaining the expected headway deviation and the expected following speed deviation is used to process the effective feature performance data of the three types of following behaviors after clustering, and to obtain the expected range and critical range of the expected headway deviation and the expected following speed deviation of the three types of following behaviors.

[0098] The driver following style classification model acquisition module is used to establish a following style identification module based on a BP neural network. Through the following style identification module, a driver following style classification model is obtained by training a BP neural network based on the labeled following behavior.

[0099] The following style identification module is used to acquire 7 cluster analysis data corresponding to the vehicle following behavior in real time during vehicle driving, and output the driver's following style based on the driver following style classification model.

[0100] The strategy module is used to build a Q-value network based on the pre-learned driving behavior database. Through the Q-value network, the desired acceleration of the vehicle is output based on the effective feature performance data of the real-time vehicle following behavior.

[0101] The deep Q-network module is used to deeply optimize the Q-value based on the reward for the vehicle following process and update the Q-value network, thereby achieving the following behavior to meet the goals of safety, following performance, comfort, and economy.

[0102] The control unit is used to issue commands to control the execution of various modules of the system.

[0103] Preferably, the identification module is trained offline on seven clustering analysis data of different styles of following behavior after clustering and labeling based on a BP neural network to establish a driver following style classification model.

[0104] The identification module acquires seven clustering analysis data points in real time during vehicle operation. Based on the driver's following style classification model, it outputs the following style during vehicle operation. The specific process is as follows:

[0105] The clustering analysis data acquisition module acquires and processes real-time characteristic performance data of the vehicle following behavior during the vehicle's driving process, and obtains 7 clustering analysis data corresponding to the real-time following behavior.

[0106] Preferably, the cluster analysis data acquisition module uses factor analysis to process the statistical features of the effective characteristic performance data of real-time vehicle following behavior, extracting the top 7 common factors of the characteristic performance data of real-time vehicle following behavior and the statistical feature coefficients corresponding to each statistical feature in the top 7 common factors of the characteristic performance data of real-time vehicle following behavior; the statistical feature information corresponding to the effective characteristic performance data of real-time vehicle following behavior is multiplied by the statistical feature coefficients in the common factors corresponding to the effective characteristic performance data of real-time vehicle following behavior, and the resulting data are summed to obtain the 7 cluster analysis data for each vehicle following behavior;

[0107] Preferably, the driver following style classification model has 2 hidden layers with 10 nodes in each layer. The activation function of the neural network is the sigmoid function, the optimizer is Adam parameter optimization, the loss function is set to the cross-entropy loss function, and the learning rate is set to 0.01.

[0108] Compared with the prior art, the present invention has the following beneficial effects: based on the following behavior database, it analyzes and processes the following behavior of a large number of driving behaviors, classifies them into three categories, and labels the classified following behaviors to give them physical meaning;

[0109] The identification module uses a BP neural network to train a driver following style classification model based on the tagged following behavior; the identification module acquires 7 clustering analysis data in real time during the vehicle's driving process and outputs the following style during the vehicle's driving process based on the driver following style classification model.

[0110] By identifying different following styles based on driving behavior, the ACC control model identifies different safety performance goals, following goals, comfort goals, and economic goals for different following styles. This results in different parameters for the deep Q-network modules corresponding to different following styles, which is more conducive to adjusting the vehicle's autonomous driving behavior commands to achieve the safety, following, comfort, and economic goals expected by the driver.

[0111] By using a deep Q-network module to process rewards based on real-time following style and effective feature performance data, the Q-value is deeply optimized, enabling each driver to achieve their desired safety, following performance, comfort, and economy goals in a real-time driving environment.

[0112] Different driving behaviors correspond to different rewards and different Q values, and different commands are given for different real-time states of different following styles.

[0113] This enables a more personalized adaptive cruise control method based on deep reinforcement learning, which offers high safety, following accuracy, comfort, and economy.

[0114] Furthermore, the reward function formula not only realizes a more personalized deep reinforcement learning-based adaptive cruise control method for drivers, but also prioritizes safety and following performance during the process, thus ensuring safety and following performance even in the event of anomalies. Detailed Implementation

[0115] To make the objectives, technical solutions and advantages of the present invention clearer, the various aspects of the present invention will be described in detail below with reference to specific embodiments. However, these specific embodiments are only used to illustrate the present invention and do not constitute any limitation on the scope of protection and the substantive content of the present invention.

[0116] Example 1

[0117] One aspect of this embodiment provides an adaptive cruise control method based on deep reinforcement learning. The method includes: a pre-set following behavior database containing feature performance data of following behavior; an identification module based on a BP neural network; and an ACC control model based on deep reinforcement learning.

[0118] The feature performance data of m following behaviors are obtained from the following behavior database, and the feature performance data are preprocessed to obtain effective feature performance data. The preprocessing uses wavelet analysis to filter and reconstruct the feature performance data of following behaviors to obtain effective feature performance data.

[0119] By processing some effective characteristic performance data of m following behaviors, the effective characteristic performance data includes the vehicle's speed v0 and the speed of the vehicle in front v p , vehicle frontage distance d, vehicle frontage distance d t ;

[0120] The effective feature performance data of m following behaviors are processed to obtain 7 cluster analysis data corresponding to the m following behaviors; the identification module performs cluster analysis on the 7 cluster analysis data of the m following behaviors using the K-means algorithm, and divides the m following behaviors into three following styles; the classified following behaviors are labeled; the three following style categories are aggressive, general, and cautious.

[0121] Methods for classifying and tagging following behavior include:

[0122] The effective characteristic performance data of m following behaviors are processed to obtain the statistical characteristic information corresponding to the effective characteristic performance data; the statistical characteristics include: maximum value, minimum value, average value, median, variance, and standard deviation;

[0123] Factor analysis was used to process the statistical features corresponding to m following behaviors, and the first 7 common factors were extracted to obtain the initial component matrix representing the effective feature performance; the initial component matrix is ​​shown in the table below:

[0124]

[0125]

[0126] The initial component matrix of the first 7 common factors is factor-rotated using the variance-maximization method to obtain the feature matrix representing the performance of the effective features. The statistical feature coefficients corresponding to each statistical feature in each of the 7 common factors are then obtained from the feature matrix. The feature matrix is ​​shown in the table below:

[0127]

[0128]

[0129] The statistical feature coefficients of the feature matrix are used to multiply and sum the statistical feature information corresponding to each following behavior to obtain 7 cluster analysis data corresponding to each following behavior;

[0130] The K-means algorithm was used to perform cluster analysis on the seven cluster analysis data of m car-following behaviors, and the m car-following behaviors were divided into three car-following styles.

[0131] The specific method for tagging the categorized car-following behavior assigns physical meaning to each public factor;

[0132] The radical category parameter of each car-following behavior is obtained by multiplying the statistical feature information corresponding to each car-following behavior with the statistical feature coefficient in the common factor representing the radical physical meaning, and then summing the data.

[0133] Arrange all the aggressive category parameters of the driving behavior in descending order to obtain the aggressive sequence. The first 30% of the aggressive sequence corresponds to the aggressive driving behavior, the 30%-70% corresponds to the general driving behavior, and the 70%-100% corresponds to the cautious driving behavior. The specific method for assigning physical meaning to each common factor is as follows: when the statistical characteristic coefficient of each common factor is greater than or equal to 0.7, the statistical characteristic corresponding to one or more of the following is: maximum speed, minimum speed, average speed, median speed, maximum acceleration, minimum acceleration, average acceleration, and median acceleration, then this common factor is an aggressive common factor.

[0134] The identification module is trained offline on seven clustering analysis data of different styles of following behavior after clustering and labeling using a BP neural network to establish a driver following style classification model.

[0135] The identification module acquires seven clustering analysis data points in real time during vehicle operation. Based on the driver's following style classification model, it outputs the following style during vehicle operation. The specific process is as follows:

[0136] Real-time characteristic performance data of vehicle following behavior is acquired and processed to obtain 7 cluster analysis data corresponding to real-time following behavior.

[0137] Factor analysis was used to process the statistical features of the effective performance data of real-time vehicle following behavior, extracting the top 7 common factors and the statistical feature coefficients corresponding to each statistical feature in the top 7 common factors. The statistical feature information corresponding to the real-time effective performance data of real-time vehicle following behavior was multiplied by the statistical feature coefficients in the common factors corresponding to the real-time effective performance data of real-time vehicle following behavior, and the summation of the data obtained was used to obtain the 7 cluster analysis data for each vehicle following behavior.

[0138] The driver following style classification model has two hidden layers with 10 nodes in each layer. The activation function of the neural network is the sigmoid function, the optimizer is Adam with parameters, the loss function is cross-entropy loss, and the learning rate is 0.01.

[0139] The ACC control model includes a strategy module and a deep Q-network module;

[0140] By processing the effective feature performance data of the classified following behaviors, the expected headway deviation k of the three types of following behaviors is obtained. d The expected interval, k d The critical range, the expected following speed deviation k for the three types of following behaviors. vThe expected interval, k v The critical interval; specific methods include:

[0141] After obtaining the effective feature performance data of the categorized following behavior, the expected headway deviation k of the three following styles is calculated using Formula 1. d The expected following speed deviation k for the three types of following behaviors is calculated using Formula 2. v ;

[0142] The expected headway deviation k for the three types of following behaviors d Fit the distribution to obtain k. d The distribution fits the normal distribution curve, and the expected following speed deviation k corresponds to the three types of following behaviors. v Fit the distribution to obtain k. v The normal distribution curve fitted by the distribution; k d The expected interval is k d The (μ) normal distribution curve d -σ d ,μ d +σ d ) interval, k d The critical interval is k d The (μ) normal distribution curve d -3σ d ,μ d +3σ d ) interval;

[0143] k v The expected interval is k v The (μ) normal distribution curve v -σ v ,μ v +σ v ) interval, k v The critical interval is k v The (μ) normal distribution curve v -3σ v ,μ v +3σ v ) interval.

[0144] Real-time acquisition of effective characteristic performance data during vehicle operation; and using Formula 1 and Formula 2 to obtain the real-time expected headway deviation k during vehicle operation from the effective characteristic performance data at each moment. d Expected following speed deviation k v ;

[0145] Formula 1 is:

[0146] d des =v0·τ+d0;

[0147]

[0148] In the formula, v0 is the vehicle speed, τ is the desired headway, d0 is the minimum safe distance after stopping, d0 = 2m; i = 1, 2, 3 correspond to the three following styles: aggressive, normal, and cautious, respectively. These are speed-related coefficients corresponding to three types of following styles: aggressive, moderate, and cautious; preferably... See the table below for specific values:

[0149] i <![CDATA[α1]]> <![CDATA[α2]]> <![CDATA[α3]]> <![CDATA[α4]]> 1 -4.25e-5 0.0095 -0.2851 3.143 2 -2.13e-5 0.0081 -0.3151 3.871 3 -1.39e-5 0.0158 -0.4657 4.732

[0150]

[0151] In the formula, d is the actual distance between the front ends of the vehicles. des Desired following distance;

[0152] Formula 2 is as follows:

[0153]

[0154] The strategy module establishes a Q-value network based on a pre-learned driving behavior database. The Q-value network outputs the desired acceleration of the vehicle based on the effective feature performance data of the acquired real-time following behavior.

[0155] The specific methods for establishing the pre-learning driving behavior database include:

[0156] A deep reinforcement learning framework was built in a Python environment, and multi-round iterative computation was performed. The neural network size was set to 32*64*128*64*32, and the hyperparameters for reinforcement learning are shown in the table below:

[0157] hyperparameters effect numerical values lr Neural network learning rate 0.001 γ Discount coefficient 0.99 <![CDATA[ε0]]> Greedy strategy initial value 0 <![CDATA[ε increment ]]> Greedy strategy increases value 0.000001 ε Greedy strategy termination value 0.000001 memory_size Memory pool capacity 1000000 batch_size Batch update quantity 128 replace_step Value network and target network update interval 500

[0158] In the training of the upper-level controller, the self-vehicle and the preceding vehicle are simplified as two point masses, and the kinematic state of the vehicle is represented by the law of uniformly accelerated motion. The velocity range of the self-vehicle is [0m / s, 20m / s], the acceleration range is [-3m / s, 3m / s], and the distance to the preceding vehicle ranges from [0m, 150m].

[0159] The strategy module processes the feature performance data of different styles of following behavior after clustering and labeling, sets up the state space and action space, and sets the range of the vehicle speed v0, the range of the vehicle acceleration a, and the range of the distance between the vehicles d.

[0160] The state space includes the distance between vehicle heads (d) and the desired safe distance (d). des The vehicle's speed v0 and the speed of the vehicle in front v pThe motion space includes the vehicle's acceleration 'a';

[0161] The spatial range of each element is discretized with a discretization accuracy of 0.1. Following behavior is trained to obtain a pre-learning driving behavior database. Three driving conditions are set for the preceding vehicle: constant speed, constant acceleration, and constant deceleration. Training is performed for each of these three conditions separately. The parameter settings for the three conditions are shown in the table below.

[0162]

[0163] Each training round consists of 1000 steps with a step size of 0.1; the training iterations are 2000 rounds.

[0164] The termination conditions are: the vehicle speed is less than or equal to 0 m / s, the time difference between the vehicle and the vehicle in front is greater than 6 seconds, the distance between the vehicle and the vehicle in front is less than or equal to 0 m, and the number of rounds reaches the set value.

[0165] The Q-value network includes an agent's action set, state set, and a neural network that processes the action set and state set; the action set is the desired acceleration a′; the state set includes the front-end distance d and the desired safety distance d. des The vehicle's speed v0 and the speed of the vehicle in front v p ;

[0166] The Q-value network takes the real-time state and actions of the effective feature performance data of the following behavior as input, and outputs the Q-values ​​corresponding to the real-time state and actions. The strategy module uses the action with the maximum Q-value corresponding to the real-time state of the effective feature performance data of the following behavior as the expected acceleration to be output.

[0167] The deep Q-network module calculates the expected headway deviation k based on three types of following behaviors. d The expected range and critical range, and the expected following speed deviation k for the three types of following behaviors. v The expected range and critical range are defined. Effective characteristic performance data of the acquired driver's following style and real-time following behavior are processed to obtain a vehicle following process reward. Based on the reward, the Q-value is deeply optimized, and the Q-value network is updated, thereby achieving the following behavior goals of safety, following performance, comfort, and economy. After the strategy module outputs an adjustment command for the vehicle's expected acceleration, it acquires the characteristic performance data of the real-time following behavior at the moment the expected acceleration command is executed during vehicle operation, processes it to obtain a corresponding reward, processes the reward to obtain the corresponding Q-value, and updates the Q-value corresponding to the real-time characteristic performance data of the following behavior in the state when the expected acceleration command is output and the output expected acceleration information in the Q-value network.

[0168] The formula for updating the Q value is:

[0169] The formula for strategy selection is:

[0170] The specific method by which the deep Q-network module obtains vehicle driving process rewards includes: the deep Q-network module constructs ACC system control objectives, which include safety objectives, following objectives, comfort objectives, and economy objectives; safety objectives and following objectives directly reflect the motion state of the vehicle and are the core objectives of control; comfort objectives and economy objectives reflect the driver's riding experience and are the constraining objectives of control;

[0171] The safety targets for the three following styles are: k corresponding to the three following styles. d The expected range;

[0172] The following objectives for the three types of car-following styles are: k corresponding to the three types of car-following styles. v The expected range;

[0173] The comfort targets for the three following styles are as follows: using acceleration jerk as the evaluation index for the comfort of the ACC system, and obtaining the jerk size range corresponding to the three following styles according to the ISO 15622 (2018) standard;

[0174] The economic targets for the three types of following styles are as follows: the vehicle's acceleration 'a' is used as the evaluation index for the economic performance of the ACC system, and the magnitude range of acceleration is specified with reference to the provisions of ISO 15622 (2018) standard.

[0175] The deep Q-network module uses the D3QN algorithm to design the ACC control algorithm. The deep Q-network module includes a reward function based on deep reinforcement learning, which consists of four parts: safety reward, following performance reward, comfort reward, and economy reward. The deep Q-network module processes the effective feature performance data during vehicle driving through the reward function formula to obtain the vehicle following process reward data.

[0176] The reward function formula is as follows:

[0177]

[0178] In the formula, ω=[ω1,ω2,ω3,ω4] T ω1, ω2, ω3, and ω4 represent the weights of rewards for safety, responsiveness, comfort, and economy.

[0179] Multiple coefficient; r0 = [r safety ,r following ,r comfort ,r economy ] TIt is a matrix of reward functions for safety, responsiveness, comfort, and economy;

[0180] The security reward function is:

[0181]

[0182] In the formula, a2, a1, b1, and b2 are k corresponding to the driver's following style, respectively. d The expected interval endpoint μ d -σ d μ d +σ d The driver's following style corresponds to k d The critical interval endpoint μ d -3σ d μ d +3σ d ;

[0183] The following reward function is:

[0184]

[0185] In the formula, c2, c1, d1, and d2 are k corresponding to the driver's following style, respectively. v The expected interval endpoint μ v -σ v μ v +σ v The driver's following style corresponds to k v The critical interval endpoint μ v -3σ v μ v +3σ v The comfort reward function is:

[0186]

[0187] In the formula, jerk max The maximum value specified for jerk size; the jerk corresponding to the driver's following style. max The maximum value specified for the jerk magnitude of the ACC system was obtained according to ISO 15622 (2018) standard;

[0188] The economic reward function is as follows:

[0189]

[0190] In the formula, a is the vehicle's acceleration at the current moment. max The maximum value specified for size 'a'; the driver's following style corresponding to 'a'. maxThe maximum value specified for the jerk 'a' in the ACC system is obtained according to ISO 15622 (2018) standard;

[0191] ω = [ω1, ω2, ω3, ω4] can be obtained using the following method. T :

[0192] According to k during the vehicle following process d k during vehicle following v The cruise operation is divided into five types: steady-state following, safe approach, safe distance, dangerous distance and dangerous approach.

[0193] Fuzzy rules are used to switch the weight coefficients under different operating conditions; the fuzzy rules obtain k during the vehicle's driving process. d k during vehicle following v The output is the four weight coefficients of the reward function;

[0194] In each simulation step, k during the vehicle's movement is first... d k during vehicle movement v Discretized with four weight coefficients, the dataset is divided into seven fuzzy subsets: PB, PM, PS, ZO, NS, NM, and NB. Fuzzy rules are designed according to the S7 design method, and the resulting fuzzy rules are shown in the table below:

[0195] (a) Security weight table

[0196]

[0197] (b) Followership weighting table

[0198]

[0199] (c) Comfort weighting table

[0200]

[0201] (d) Comfort weighting table

[0202]

[0203] The triangle membership function (trimf function) is selected to transform the input into a fuzzy output. Finally, the centroid method is used to defuzzify the data and obtain specific weight coefficients. The obtained weight coefficients are then standardized to obtain the weight coefficients for safety, responsiveness, comfort, and economic rewards.

[0204] This embodiment also provides an adaptive cruise control system based on deep reinforcement learning, including:

[0205] The database preset and data extraction module is used to preset a following behavior database including the characteristic performance data of following behavior and to obtain the characteristic performance data of m following behaviors from the following behavior database.

[0206] The preprocessing module is used to preprocess the feature performance data of m following behaviors to obtain the effective feature performance data of m following behaviors.

[0207] The cluster analysis data acquisition module is used to process the partial effective feature performance data of the m following behaviors through factor analysis to obtain 7 cluster analysis data corresponding to the m following behaviors.

[0208] The clustering module is used to cluster based on 7 clustering analysis data corresponding to m following behaviors using the K-means algorithm to obtain three types of following behaviors, and then label the three types of following behaviors after clustering with labels representing following styles.

[0209] Desired frontage deviation k d and the deviation of the expected following speed k v The acquisition module processes the effective feature performance data of the three types of following behaviors after clustering to obtain the expected headway deviation k of the three following styles. d The expected range and critical range, and the expected following speed deviation k of the three types of following styles. v The expected range and critical range;

[0210] The driver following style classification model acquisition module is used to establish a following style identification module based on a BP neural network. Through the following style identification module, a driver following style classification model is obtained by training a BP neural network based on the labeled following behavior.

[0211] The following style identification module is used to acquire 7 cluster analysis data corresponding to the vehicle following behavior in real time during vehicle driving, and output the driver's following style based on the driver following style classification model.

[0212] The strategy module is used to build a Q-value network based on the pre-learned driving behavior database. Through the Q-value network, the desired acceleration of the vehicle is output based on the effective feature performance data of the real-time vehicle following behavior.

[0213] The deep Q-network module is used to deeply optimize the Q-value based on the reward for the vehicle following process and update the Q-value network, thereby achieving the following behavior to meet the goals of safety, following performance, comfort, and economy.

[0214] The control unit is used to issue commands to control the execution of various modules of the system.

[0215] The present invention has been described above with reference to specific embodiments. These specific embodiments are merely exemplary and should not be construed as limiting the scope of protection of the present invention. Those skilled in the art can make various modifications, changes, or substitutions without departing from the essence of the present invention. Therefore, various equivalent variations made according to the present invention still fall within the scope of the present invention.

Claims

1. A method of adaptive cruise control based on deep reinforcement learning, characterized in that, The method comprises: a following behavior database including characteristic performance data of following behaviors; obtaining characteristic performance data of m following behaviors from the following behavior database and preprocessing to obtain effective characteristic performance data of the m following behaviors; based on part of the effective characteristic performance data of the m following behaviors, processing by a factor analysis method to obtain 7 clustering analysis data corresponding to the m following behaviors; based on the 7 clustering analysis data corresponding to the m following behaviors, clustering by a K-means algorithm to obtain three types of following behaviors, and marking the three types of following behaviors after clustering with labels representing following styles respectively; The effective characteristic performance data of the three types of following behaviors after clustering is processed to obtain the expected headway deviation k of the following behavior of the three types of following styles d , the expected interval and critical interval of the expected following speed deviation k of the following behavior of the three types of following styles v ; establishing a following style recognition module based on a BP neural network, training a driver following style classification model by the BP neural network based on the following behaviors after marking by the following style recognition module; obtaining 7 clustering analysis data corresponding to real-time following behaviors of a vehicle during vehicle driving, outputting a following style of a driver by the following style recognition module based on the driver following style classification model; establishing an ACC control model based on deep reinforcement learning, the ACC control model comprising a policy module and a deep Q network module; establishing a Q value network based on a pre-learned driving behavior database by the policy module, outputting an expected acceleration for adjusting a vehicle according to the obtained effective characteristic performance data of real-time following behaviors of the vehicle by the Q value network; The expected headway deviation degree k of the following behavior of the three types of following styles is obtained through the deep Q network module d The expected headway deviation degree k of the following behavior of the three types of following styles is obtained through the deep Q network module v The expected headway deviation degree k of the following behavior of the three types of following styles is obtained through the deep Q network module The obtained driving style and the effective characteristic performance data of the real-time vehicle following behavior are processed to obtain the vehicle following process reward, the Q value is deep optimized according to the vehicle following process reward, and the Q value network is updated, so that the following behavior meets the safety target, the following target, the comfort target and the economy target. 2.The adaptive cruise control method based on deep reinforcement learning of claim 1, wherein, The effective feature performance data includes self-vehicle speed v0, front-vehicle speed v p , self-vehicle acceleration a, self-vehicle jerk, headway d, and headway time d t ; The part of the effective feature performance data in processing m car-following behaviors is the part of the effective feature performance data of the ego vehicle speed v0, the front vehicle speed v p , the headway d, and the headway time d t . 3.The adaptive cruise control method based on deep reinforcement learning of claim 1, wherein, the 7 clustering analysis data corresponding to the m following behaviors are obtained by the following steps: processing part of the effective characteristic performance data of the m following behaviors to obtain statistical characteristic information corresponding to the effective characteristic performance data; processing the statistical characteristic information corresponding to the m following behaviors by a factor analysis method to extract the first 7 common factors to obtain an initial component matrix representing effective characteristic performance; rotating the initial component matrix of the first 7 common factors by a variance maximization method to obtain a characteristic matrix representing effective characteristic performance, and obtaining statistical characteristic coefficients corresponding to each statistical characteristic in each common factor among the 7 common factors by the characteristic matrix; processing the statistical characteristic information corresponding to each following behavior by the statistical characteristic coefficients of the characteristic matrix to obtain 7 clustering analysis data corresponding to each following behavior; the statistical characteristics include maximum value, minimum value, average value, median, variance and standard deviation. 4.The adaptive cruise control method based on deep reinforcement learning of claim 2, wherein, Based on the effective feature performance data of the clustered following behaviors, the expected headway deviation k of the following behaviors of the three following styles is calculated by formula one d The expected following speed deviation k of the following behaviors of the three following styles is calculated by formula two v ; The expected headway deviation k for the three types of following behaviors d Fit the distribution to obtain k. d The normal distribution curve fitted by the distribution represents the expected following speed deviation k corresponding to the three types of following behaviors. v Fit the distribution to obtain k. v The normal distribution curve fitted to the distribution; k d The expected interval is k d The (μ) normal distribution curve d -σ d ,μ d +σ d ) interval, k d The critical interval is k d The (μ) normal distribution curve d -3σ d ,μ d +3σ d ) interval; k v The expected interval is k v The (μ) normal distribution curve v -σ v ,μ v +σ v ) interval, k v The critical interval is k v The (μ) normal distribution curve v -3σ v ,μ v +3σ v ) interval.

5. The method of claim 4, wherein, Real-time acquisition of the effective characteristic performance data of the vehicle during driving, and the expected headway deviation degree k of the vehicle during driving is obtained through formula one and formula two for each time of the effective characteristic performance data d , the expected following speed deviation degree k v . 6.The adaptive cruise control method based on deep reinforcement learning of claim 4, wherein, the first formula is: d des = v0τ + d0; In the formula, v0 is the vehicle speed, τ is the desired vehicle headway, d0 is the minimum safety distance after stopping; i = 1, 2, 3 respectively correspond to the aggressive type, the general type, and the cautious type of three types of following styles, is the speed-related coefficient corresponding to the following behavior of the aggressive type, the general type, and the cautious type of three types of following styles. where d is the actual headway, d des is the desired headway; and / or the second formula is:

7. The method of claim 4, wherein the adaptive cruise control is based on deep reinforcement learning. based on the deep Q network module, constructing an ACC system control target, the ACC system control target comprising safety target, following target, comfort target and economy target; the safety target and the following target directly reflect the motion state of a driving vehicle and are core targets of control; the comfort target and the economy target reflect the riding experience of a driver and are constraint targets of control. 8.The adaptive cruise control method based on deep reinforcement learning of claim 7, wherein, The safety targets of the three types of following styles are: the expected interval of k corresponding to the three types of following styles d ; The following properties are defined for each of the three types of following styles: v The following properties are defined for each of the three types of following styles: the comfort target of the three types of following styles is that jerk is used as an evaluation index of ACC system comfort; the economy target of the three types of following styles is that vehicle acceleration a is used as an evaluation index of ACC system economy. 9.The adaptive cruise control method based on deep reinforcement learning of claim 4, wherein, The deep Q network module comprises a reward function of deep reinforcement learning, which is composed of four parts of safety reward, following reward, comfort reward and economy reward; the deep Q network module processes the effective feature performance data in the vehicle following process through the reward function formula to obtain vehicle following process reward data; The reward function formula is: In the formula, ω=[ω1,ω2,ω3,ω4] T ω1, ω2, ω3, ω4 are the weighting coefficients for safety, responsiveness, comfort, and economic rewards; r0 = [r safety ,r following ,r comfort ,r economy ] T It is a matrix of reward functions for safety, responsiveness, comfort, and economy; The safety reward function is: In the formula, a2, a1, b1, b1, respectively, are k d of the driver's following style d -σ d , μ d +σ d , the critical interval end point μ d of the driver's following style d -3σ d , μ d +3σ d ; The following reward function is: In the formula, c2, c1, d1, d1, respectively, are the k v of the driver's following style v -σ v , μ v +σ v corresponding to the driver's following style v of the critical interval interval end point μ v -3σ v , μ v +3σ v ; The comfort reward function is: where jerk max is a maximum value prescribed for the jerk magnitude; The economy reward function is: In the formula, a is the current time self-vehicle acceleration, a max is the maximum value of a size; According to the following method ω = [ω1, ω2, ω3, ω4] T : According to the k d With the k v The cruise working condition is divided into five working conditions: stable following, safe approaching, safe leaving, dangerous leaving and dangerous approaching. The weight coefficients under different working conditions are switched using fuzzy rules; the fuzzy rules obtain k d k in the vehicle following process of the vehicle v , and the output is 4 weight coefficients of the reward function.

10. A deep reinforcement learning based adaptive cruise control system, characterized in that, Comprise: The database preset and data extraction module is used for presetting the following behavior database comprising the feature performance data of the following behavior and obtaining the feature performance data of m following behaviors from the following behavior database; The preprocessing module is used for preprocessing the feature performance data of m following behaviors to obtain the effective feature performance data of m following behaviors; The cluster analysis data acquisition module is used for processing based on part of the effective feature performance data of m following behaviors through factor analysis method to obtain 7 cluster analysis data corresponding to m following behaviors; The clustering module is used for clustering based on the 7 cluster analysis data corresponding to m following behaviors through K-means algorithm to obtain three types of following behaviors, and respectively marking the three types of following behaviors after clustering with labels representing following styles; Expected headway deviation k d and expected following speed deviation k v An acquisition module is configured to process the effective feature performance data of the clustered three types of following behaviors to obtain expected headway deviation k d of the three types of following behaviors, and expected headway deviation k v of the three types of following behaviors. The driver following style classification model acquisition module is used for establishing a following style recognition module based on BP neural network, and obtaining a driver following style classification model through the following style recognition module based on the following behaviors after marking labels and using BP neural network training; The following style recognition module is used for obtaining 7 cluster analysis data corresponding to real-time vehicle following behavior in vehicle driving process, outputting the following style of the driver based on the driver following style classification model; The strategy module is used for establishing a Q value network based on the pre-learned driving behavior database through the strategy module, and outputting the expected acceleration of adjusting the vehicle according to the obtained effective feature performance data of real-time vehicle following behavior through the Q value network; The deep Q network module is used for deep optimization of Q value according to the vehicle following process reward, and updating the Q value network, so as to realize that the following behavior meets the safety target, the following target, the comfort target and the economy target; The control unit is used for issuing instructions to control the execution of each module of the system.

Images