An individualized adaptive cruise system based on deep reinforcement learning and a control method thereof

Abstract

The application discloses a kind of individualized adaptive cruise system and control method based on deep reinforcement learning, and individualized multi-objective adaptive cruise control system is designed based on deep reinforcement learning, based on the improved deep deterministic gradient strategy, by designing the control module of three different control styles, and the safety, the design of modular reward function of car-following and comfort of this system meets the demand of different style drivers for the performance of ACC system, by constructing similarity function, it is used for the update of online action network, so that it is more in line with the driving habits of human driver, by experience update module, the reuse rate of high-value experience samples of each type of driver is improved, effectively improve the convergence speed of network, overall improve the individualization and comprehensive performance of ACC system.

Description

Technical Field

[0001] This invention relates to an automatic vehicle control system, and in particular provides a personalized adaptive cruise system and its control method based on deep reinforcement learning, belonging to the field of vehicle driver assistance technology. Background Technology

[0002] With the continuous advancement of science and technology, automobiles are becoming increasingly intelligent, and autonomous driving is one of the main trends in future automotive development. Adaptive cruise control (ACC) is an important component of advanced driving assistance systems (ADAS) and represents a relatively basic stage of autonomous driving. ACC initially evolved from cruise control, and later added distance-keeping functionality. It uses onboard sensors to perceive the driving environment ahead, and the controller makes decisions based on this information, providing corresponding control inputs to automatically adjust the vehicle speed by controlling the throttle and braking system, thus improving driving safety and comfort.

[0003] The core of adaptive cruise control (ACC) system research lies in the design of control methods, which can generally be divided into two categories: rule-based and learning-based control methods. Rule-based ACC methods respond to the environment according to fixed rules, but their complexity is high in real-world driving conditions. Human-designed rules are difficult to encompass all situations, resulting in poor generalization. Furthermore, because decision-making is based on fixed rules, it cannot meet the personalized needs of human drivers, leading to low acceptance of ACC systems. Classical PID control theory, sliding mode control theory, fuzzy control theory, and model predictive control methods are widely used in rule-based ACC control methods. Model predictive control, in particular, can design multi-objective ACC systems by combining predetermined rules with performance indicators, but it lacks a causal relationship between state and action.

[0004] Learning-based adaptive cruise control methods, which do not require specific rules for particular operating conditions, have a natural advantage in solving the aforementioned problems. Currently, many researchers favor using deep reinforcement learning to design adaptive cruise control methods. Deep learning possesses powerful perception capabilities and can serve as a tool for feature extraction in reinforcement learning. Furthermore, reinforcement learning treats the vehicle as an intelligent agent, continuously interacting with the environment through trial and error to seek the optimal strategy, thereby maximizing cumulative rewards in specific states. Deep reinforcement learning, which combines deep learning and reinforcement learning, possesses the ability to make reasonable decisions in complex environments, thus exhibiting good generalization ability.

[0005] Currently, many researchers are conducting studies on driver driving styles based on a large amount of driver behavior data in order to improve the market acceptance of ACC systems. Many scholars have also conducted in-depth research on the multi-objective coordination problem of ACC. However, when designing multi-objective ACC systems, the performance requirements of different driving styles for various ACC systems have been overlooked. How to effectively balance the various performance aspects of the ACC system while considering driving styles, and achieve personalized and comprehensive performance improvements, is the focus of ACC research. Summary of the Invention

[0006] Objective: To address the shortcomings of existing technologies, this invention provides a personalized adaptive cruise control system and its control method based on deep reinforcement learning. This invention designs a personalized multi-objective adaptive cruise control system based on deep reinforcement learning. Based on an improved deep deterministic gradient strategy, it designs modular reward functions for the safety, following performance, and comfort of the adaptive cruise control system. Furthermore, it designs three control modules with different control styles according to specific driving styles to meet the different performance needs of drivers with different driving styles.

[0007] Technical solution: A personalized adaptive cruise system based on deep reinforcement learning, comprising an environmental perception layer, an upper-level controller, a lower-level controller, and an actuator. The signals acquired by the environmental perception layer are input to the upper-level controller, which converts the input signals into control signals and outputs them to the lower-level controller. The lower-level controller converts the control signals output by the upper-level controller into action commands and sends them to the actuator, which then operates according to the execution commands.

[0008] The environmental perception layer includes millimeter-wave radar and a vehicle speed sensor. The millimeter-wave radar acquires the actual distance d between the vehicle and the vehicle in front within the detection range, as well as the relative vehicle speed. Signal; vehicle speed sensor is used to obtain the vehicle's speed. Signal;

[0009] The upper-level controller includes a driver style recognizer, a motion input calculation module, and a deep reinforcement learning module;

[0010] The driver style recognizer includes a signal parameter recording module, an impact calculation module, a driving state recognition coefficient calculation module, and a driving style recognition module. The signal parameter recording module records the vehicle speed transmitted from the environmental perception layer. Relative distance d, relative vehicle speed Based on the signals indicating whether there are vehicles ahead, the impact calculation module calculates the impact intensity, and the driving state recognition coefficient calculation module calculates the driver's driving state recognition coefficient based on the calculated impact intensity. The driving style recognition module calculates the driver's driving state recognition coefficient. Determine the driver's style, which includes three types: aggressive, average, and conservative.

[0011] The action input calculation module includes an aggressive action input calculation module, a general action input calculation module, and a conservative action input calculation module. The aggressive action input calculation module, the general action input calculation module, and the conservative action input calculation module each include a desired distance calculation module, a parameter calculation module, and a parameter recording module, respectively.

[0012] The deep reinforcement learning module includes an aggressive DDPG_aggressive control module, a normal DDPG_normal control module, and a conservative DDPG_calm control module. Based on the driving style identified by the driving style recognizer, the corresponding action input calculation module and control module are selected.

[0013] The lower-level controller includes a drive / brake switching module and a desired throttle / brake pressure calculation module. The drive / brake switching module converts the desired acceleration value output by the upper-level controller into an acceleration / braking signal and sends the acceleration or braking signal to the desired throttle / brake pressure calculation module. The desired throttle / brake pressure calculation module calculates the throttle opening or desired braking pressure and sends it to the actuator.

[0014] The actuator includes a power unit control unit, an electronic throttle, a brake control unit, and a brake cylinder. The power unit control unit controls the electronic throttle according to the calculated throttle opening, and the brake control unit controls the brake cylinder according to the desired brake pressure.

[0015] This invention collects the state signals of the intelligent agent through the environmental perception layer and transmits them to the upper-level controller. The upper-level controller identifies the corresponding driving style based on the signals and switches to the corresponding driving style control mode. After training, the expected acceleration for the corresponding driving style is obtained. The lower-level controller calculates the corresponding expected throttle opening or braking pressure based on the expected acceleration and inputs it to the actuator for execution. This achieves the goal of improving the personalization and overall performance of the system on the basis of its basic performance and meeting the needs of various driving styles.

[0016] A control method for a personalized adaptive cruise system based on deep reinforcement learning includes the following steps:

[0017] Step 1: Start the system. The millimeter-wave radar will detect whether there is a vehicle in front within the detection range. If there is no vehicle in front, maintain the set speed. If there is a vehicle in front, proceed to Step 2.

[0018] Step 2: Acquire signal parameters. Millimeter-wave radar and vehicle speed sensor respectively detect and obtain the relative distance d and relative speed with the vehicle in front. and vehicle speed Signal;

[0019] Step 3: Input signal parameters. Input the signal parameters obtained in Step 2 into the signal parameter recording module and save them.

[0020] Step 4: Identify driving style. The driving style identification module determines the current driver's style based on the signal parameters in the parameter recording module. If it is determined to be aggressive, proceed to step 5. If it is not aggressive, further determine whether it is moderate. If it is moderate, proceed to step 6. If it is not moderate, determine whether it is conservative, proceed to step 7. Store the driver's identification result in the driving style identification module.

[0021] Step 5: Switch to aggressive control mode. If the driving style is determined to be aggressive by the driving style identification module, the signal parameters stored in the signal parameter recording module will be input into the aggressive action input calculation module. The output is the state quantity between the vehicle and the vehicle in front. The aggressive DDPG_aggressive control module is connected to the aggressive action input calculation module. It takes the state quantity as input and outputs the desired acceleration of the aggressive driver.

[0022] Step 6: Switch to the general control mode. If the driving style is determined to be general by the driving style identification module, the signal parameters stored in the signal parameter recording module will be input into the general action input calculation module. The output is the state quantity between the vehicle and the vehicle in front. The general DDPG_normal control module is connected to the general action input calculation module. It takes the state quantity as input and outputs the expected acceleration of the general driver.

[0023] Step 7: Switch to conservative control mode. If the driving style is determined to be conservative by the driving style identification module, the signal parameters stored in the signal parameter recording module will be input into the conservative action input calculation module. The output is the state quantity between the vehicle and the vehicle in front. The conservative DDPG_calm control module is connected to the conservative action input calculation module. It takes the state quantity as input and outputs the desired acceleration of the conservative driver.

[0024] Step 8: Adjust acceleration. The drive / brake switching module converts the desired acceleration value output by the upper controller into an acceleration / braking signal and sends the acceleration / or braking signal to the desired throttle / brake pressure calculation module. The desired throttle / brake pressure calculation module calculates the throttle opening or desired braking pressure and sends it to the actuator. The power unit control unit controls the electronic throttle according to the throttle opening, or the brake control unit controls the brake cylinder or actuator to maintain the current state according to the desired braking pressure. If the system continues to run, return to step 2 and repeat the loop to step 8. If the system is shut down, proceed to step 9.

[0025] Step Nine: End. When shutting down or parking the system, the system will stop running after step eight.

[0026] This invention designs three control modules with different control styles based on specific driving styles, and designs modular reward functions for the system's safety, following performance, and comfort. By using control modules with different styles for strategy training and with the assistance of reward functions, the system can continuously learn and update its strategies while ensuring that the strategies are all high-value experiences and making the learning of the agent more stable. This meets the performance requirements of different driving styles for the ACC system and improves the system's personalization and overall performance.

[0027] Preferably, the method for identifying driving style in step four is as follows:

[0028] S1. Calculate the impact intensity and read the vehicle speed transmitted from the environmental perception layer by the signal parameter recording module. Relative distance d, relative vehicle speed The impact calculation module receives the vehicle speed from the vehicle speed sensor, along with signals indicating whether there are vehicles ahead. The signal's impact J(t) is calculated using the formula:

[0029] J(t) = / (1)

[0030] In the formula, Let t be the speed of the vehicle at time t;

[0031] S2, Calculate the driver's driving state recognition coefficient. The driver state recognition coefficient calculation module calculates the driver's driving state recognition coefficient based on the calculated impact intensity. And input it into the driving style recognition module.

[0032] Driver driving status recognition coefficient :

[0033] = = (2)

[0034] In the formula, To identify the standard deviation of the impact within the period, This represents the average impact force experienced by a standard driver. Let be the impact intensity at time i, and T be the recognition period;

[0035] S3. Determine the driver's style. In the driving style recognition module, based on the driver's driving state recognition coefficient calculated in real time. To determine the driver's style, if > Then the driver's style is aggressive. This is the critical value for the driving state recognition coefficient of aggressive drivers; if < If so, the driver's style is conservative; if < < If the driver's style is determined to be "general", the driver's identification result will be stored in the driving style identification module.

[0036] Preferably, the state quantities between the vehicle and the vehicle in front in steps five, six, and seven include distance error. Bicycle speed and relative velocity The parameter calculation module calculates the distance based on the formula for the desired distance. = * Calculate the expected distance In the formula, Let be the speed of the vehicle at time t. For following distance, To maintain a safe following distance, the parameter calculation module combines the actual distance d and the vehicle's speed. and relative velocity The signal is used to calculate the distance error. Distance error Bicycle speed and relative velocity Signal parameters are saved in the parameter recording module.

[0037] Select an aggressive following distance For 1s and A standard following distance of 2 meters. For 1.5s and The following distance is 3m for the conservative type and 2s for the conservative type. It is 4m.

[0038] In a preferred embodiment, the control strategies of the aggressive DDPG_aggressive control module, the normal DDPG_normal control module, and the conservative DDPG_calm control module in steps five, six, and seven are obtained through training, and after the training converges, they are stored in the corresponding modules respectively. The desired acceleration is output according to the control strategy based on the state variables as input.

[0039] In a preferred embodiment, the training method includes a state update module, an experience update module, a network parameter update module, and a reward function module. The state update module converts the data collected by the environment perception layer into state variables and inputs them into the experience update module to update high-value experience samples and store them. The network parameter update module randomly selects experience samples from the experience update module for training and updates them in a way that maximizes the reward according to the reward given by the reward function module.

[0040] Preferably, the state update module updates the relative speed between the vehicle and the vehicle in front. Bicycle speed Distance error between actual spacing and expected spacing As the state input, the network parameter update module outputs the desired acceleration as the state output. Given the state quantity at time t, the state quantity at the next time is updated. This state input and the actual acceleration are defined as a sample empirical sequence. ),in This is the state variable of the current sample. The actual acceleration of the current sample. The reward for the current sample. This represents the state of the current sample at the next time step.

[0041] (3)

[0042] In the formula, For the update time step, The speed of the vehicle in front. For the vehicle's speed, The relative speed between the vehicle and the vehicle in front. This refers to the actual distance between your vehicle and the vehicle in front. This represents the distance error between the actual vehicle spacing and the desired vehicle spacing. The desired vehicle spacing;

[0043] Initialize the state variables before training begins:

[0044] ; (4)

[0045] ; (5)

[0046] ; (6)

[0047] In the formula, It is the vehicle speed at time t=0, the start of training. It is the speed of the vehicle at time t=0 in a randomly sampled following video clip. It is the distance between the two cars at the start of training, t=0. It is the actual distance between the two vehicles at time t=0 in a randomly sampled following segment. It represents the relative speed between the vehicle and the vehicle in front at time t=0, the start of training. It is the actual relative speed between the two vehicles in a randomly sampled following segment.

[0048] Preferably, the experience update module selects high-value sample experiences and adds them to the corresponding style experience pool, collecting sample experiences from drivers of the corresponding style. ),

[0049] Define the average reward of training samples of aggressive-style drivers for:

[0050] (7)

[0051] In the formula, n is the number of iterations from the beginning of training to the current training. It is the reward obtained during the m-th training session;

[0052] Reward for gaining current experience (r) actual When r actual ≥ When the experience is zero, it is stored in the experience pool. <r actual < When, calculate the probability p=r actual / The experience is stored in the experience pool with probability p. When r actual When the value is less than 0, the experience is discarded.

[0053] For both average and conservative style drivers, the same update method as for aggressive style drivers' experience update module is applied. The average reward for average style drivers is defined as follows: The average reward for a conservative-style driver is defined as .

[0054] Preferably, the network parameter update module includes an action network and a policy evaluation network. The action network includes an online action network and a target action network. The policy evaluation network includes an online action evaluation network and a target action evaluation network. The action network outputs a deterministic expected acceleration based on the state variables of the vehicle and the preceding vehicle. The policy evaluation network outputs an action value evaluation function based on the quality of the actions taken by the online action network. ,

[0055] Define the parameters of the online action network as follows The target action network parameters are: The parameters of the online motion evaluation network are: The target action evaluation network parameters are: ;

[0056] Action Network Update:

[0057] Random collection The online action network is trained on a sample, and then uses the data from the sample... Output To online motion assessment networks;

[0058] Construct a similarity function:

[0059] (8)

[0060] In the formula, For actual acceleration, For sample state quantities, For online action network parameters, A deterministic policy for output by the online action network;

[0061] Online motion evaluation networks based on samples and similarity function Calculate the new action value evaluation function:

[0062] (9)

[0063] In the formula, For the action value evaluation function, For similarity functions;

[0064] Update via the optimizer of the actor network The optimizer updates the parameters of the online action network using gradient ascent. :

[0065] (10)

[0067] In the formula, For online action network parameters, For the evaluation function of the value of the new action, For the action value evaluation function, For sample state quantities, For deterministic policies output by online action networks, Where N is the actual acceleration and N is the number of samples;

[0068] Action evaluation network update:

[0069] The target action network is based on the samples Output The target action evaluation network is then used to evaluate the action based on the samples. and the output of the target action network Predicting future rewards and :

[0070] = (11)

[0071] In the formula, The reward for the current sample. This represents the state of the current sample at the next time step. Let the target action evaluation function be... For the target action network parameters, The network parameters are used to evaluate the target action. As a discount factor, A deterministic strategy for the target action network;

[0072] Update the parameters of the online action evaluation network using the optimizer of the critic network. The optimizer is based on rewards and and the motion value evaluation function output by the online motion evaluation network The parameters of the online action evaluation network are updated by minimizing the loss function. :

[0073] L( )= = (12)

[0074] In the formula, For future rewards and, This is the state variable of the current sample. The actual acceleration of the current sample. The reward for the current sample. This represents the state of the current sample at the next time step. Here, N is the discount factor, and N is the sample size. For the action value evaluation function, A deterministic strategy for the target action network;

[0075] Both the target action network and the target action evaluation network update their parameters using a soft update method. and Updated periodically:

[0076] (13)

[0077] In the formula, For the target action network parameters, The network parameters are used to evaluate the target action. For online action network parameters, For online motion evaluation network parameters, This refers to the following distance.

[0078] The tanh activation function is used for the output of the action network. For an aggressive driving style, the function is mapped to the range [-1, 1], thus limiting the output acceleration to [-4, 4] m / s². For a typical driving style, the function is mapped to the range [-1, 1], thus limiting the output acceleration to [-3.5, 3.5] m / s². For a conservative driving style, mapping the function to the range [-1, 1] allows the output acceleration to be limited to [-3, 3] m / s. .

[0079] In a preferred embodiment, the reward function module is divided into a safety reward function, a following performance reward function, and a comfort reward function, which are linearly integrated, and the weights of each performance are continuously adjusted during training.

[0080] Safety is mainly related to the actual distance d between the vehicle and the vehicle in front. act and the relative speed between the two vehicles Related, combining the minimum safe distance d between the vehicle and the vehicle in front. safe The security reward function R1 is as follows:

[0081] R1 (14)

[0082] Where, d act d represents the actual distance between the vehicle and the vehicle in front. safe The minimum safe distance between the vehicle and the vehicle in front is set. If the actual distance between the vehicles is less than the minimum safe distance and the vehicle is still accelerating to approach, a penalty of -10 is imposed.

[0083] Following performance is measured by the deviation between the actual distance and the expected distance between the two vehicles. The larger the deviation, the greater the penalty. The following performance reward function R2 is as follows:

[0084] R2 = -| | (15)

[0085] In the formula, This represents the actual distance between your vehicle and the vehicle in front. This represents the expected distance between the vehicle and the vehicle in front.

[0086] The comfort reward function R3 is as follows:

[0087] R3 = - (16)

[0088] In the formula, Jerk represents the rate of change of acceleration. It is often used to measure comfort; the smaller the Jerk value, the better the comfort.

[0089] For aggressive driving styles, following distance is prioritized over comfort; therefore, the following time distance is included in the expected following distance. When set to 1 second, its reward function is linearly integrated as R = + + ;

[0090] For a typical driving style, the required following distance and comfort are between those of the other two driving styles. Therefore, the following distance in the expected following distance is considered as follows. Set to 1.5s, its reward function linearly integrated is R= + + ;

[0091] For a conservative driving style, comfort is prioritized over following distance; therefore, the following time distance is included in the expected following distance. When set to 2s, its reward function is linearly integrated as R= + + .

[0092] Beneficial effects: This invention designs control modules with three different control styles and modularizes reward functions for the system's safety, following ability, and comfort to meet the performance requirements of different driving styles of ACC system. By constructing a similarity function and using it to update the online action network, it makes the system more in line with human driving habits. Through the experience update module, it improves the reuse rate of high-value experience samples of various types of drivers, effectively improves the convergence speed of the network, and improves the overall personalization and comprehensive performance of the system. Attached Figure Description

[0093] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0094] Figure 1 This is a diagram of the overall architecture of the ACC system of this invention;

[0095] Figure 2 This is a structural diagram of the driver style recognizer of the present invention;

[0096] Figure 3 This is a flowchart of the ACC system workflow of the present invention;

[0097] Figure 4 This is a diagram illustrating the processing of the experience update module in this invention.

[0098] Figure 5 This is a network update diagram for the present invention. Detailed Implementation

[0099] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0100] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0101] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0102] like Figure 1 As shown, a personalized adaptive cruise system based on deep reinforcement learning includes an environmental perception layer, an upper-level controller, a lower-level controller, and an actuator. The signals acquired by the environmental perception layer are input to the upper-level controller. The upper-level controller converts the input signals into control signals and outputs them to the lower-level controller. The lower-level controller converts the control signals output by the upper-level controller into action commands and sends them to the actuator. The actuator operates according to the execution commands.

[0103] The environmental perception layer includes millimeter-wave radar and a vehicle speed sensor. The millimeter-wave radar acquires the actual distance d between the vehicle and the vehicle in front within the detection range, as well as the relative vehicle speed. Signal; vehicle speed sensor is used to obtain the vehicle's speed. Signal;

[0104] The upper-level controller includes a driver style recognizer, a motion input calculation module, and a deep reinforcement learning module;

[0105] like Figure 2 As shown, the driver style recognizer includes a signal parameter recording module, an impact calculation module, a driving state recognition coefficient calculation module, and a driving style recognition module. The signal parameter recording module records the vehicle speed transmitted from the environmental perception layer. Relative distance d, relative vehicle speed Based on the signals indicating whether there are vehicles ahead, the impact calculation module calculates the impact intensity, and the driving state recognition coefficient calculation module calculates the driver's driving state recognition coefficient based on the calculated impact intensity. The driving style recognition module calculates the driver's driving state recognition coefficient. Determine the driver's style, which includes three types: aggressive, average, and conservative.

[0106] The action input calculation module includes an aggressive action input calculation module, a general action input calculation module, and a conservative action input calculation module. The aggressive action input calculation module, the general action input calculation module, and the conservative action input calculation module each include a desired distance calculation module, a parameter calculation module, and a parameter recording module, respectively.

[0107] The deep reinforcement learning module includes an aggressive DDPG_aggressive control module, a normal DDPG_normal control module, and a conservative DDPG_calm control module. Based on the driving style identified by the driving style recognizer, the corresponding action input calculation module and control module are selected.

[0108] The lower-level controller includes a drive / brake switching module and a desired throttle / brake pressure calculation module. The drive / brake switching module converts the desired acceleration value output by the upper-level controller into an acceleration / braking signal and sends the acceleration or braking signal to the desired throttle / brake pressure calculation module. The desired throttle / brake pressure calculation module calculates the throttle opening or desired braking pressure and sends it to the actuator.

[0109] To avoid frequent switching between drive and brake, three modes—drive control, brake control, and no operation—are defined by setting thresholds above and below the drive / brake curves. Drive control is activated when the desired acceleration is above the drive / brake threshold, brake control is activated when the desired acceleration is below the drive / brake threshold, and no drive or brake control is required when the desired acceleration is between the two thresholds. This improves driving comfort and fuel economy.

[0110] The actuator includes a power unit control unit, an electronic throttle, a brake control unit, and a brake cylinder. The power unit control unit controls the electronic throttle according to the calculated throttle opening, and the brake control unit controls the brake cylinder according to the desired brake pressure.

[0111] This invention collects the state signals of the intelligent agent through the environmental perception layer and transmits them to the upper-level controller. The upper-level controller identifies the corresponding driving style based on the signals and switches to the corresponding driving style control mode. After training, the expected acceleration for the corresponding driving style is obtained. The lower-level controller calculates the corresponding expected throttle opening or braking pressure based on the expected acceleration and inputs it to the actuator for execution. This achieves the goal of improving the personalization and overall performance of the system on the basis of its basic performance and meeting the needs of various driving styles.

[0112] like Figure 3 As shown, a control method for a personalized adaptive cruise system based on deep reinforcement learning includes the following steps:

[0113] Step 1: Start the system. The millimeter-wave radar will detect whether there is a vehicle in front within the detection range. If there is no vehicle in front, maintain the set speed. If there is a vehicle in front, proceed to Step 2.

[0114] Step 2: Acquire signal parameters. Millimeter-wave radar and vehicle speed sensor respectively detect and obtain the relative distance d and relative speed with the vehicle in front. and vehicle speed Signal;

[0115] Step 3: Input signal parameters. Input the signal parameters obtained in Step 2 into the signal parameter recording module and save them.

[0116] Step 4: Identify driving style. The driving style identification module determines the current driver's style based on the signal parameters in the parameter recording module. If it is determined to be aggressive, proceed to step 5. If it is not aggressive, further determine whether it is moderate. If it is moderate, proceed to step 6. If it is not moderate, determine whether it is conservative, proceed to step 7. Store the driver's identification result in the driving style identification module.

[0117] Step 5: Switch to aggressive control mode. If the driving style is determined to be aggressive by the driving style identification module, the signal parameters stored in the signal parameter recording module will be input into the aggressive action input calculation module. The output is the state quantity between the vehicle and the vehicle in front. The aggressive DDPG_aggressive control module is connected to the aggressive action input calculation module. It takes the state quantity as input and outputs the desired acceleration of the aggressive driver.

[0118] Step 6: Switch to the general control mode. If the driving style is determined to be general by the driving style identification module, the signal parameters stored in the signal parameter recording module will be input into the general action input calculation module. The output is the state quantity between the vehicle and the vehicle in front. The general DDPG_normal control module is connected to the general action input calculation module. It takes the state quantity as input and outputs the expected acceleration of the general driver.

[0119] Step 7: Switch to conservative control mode. If the driving style is determined to be conservative by the driving style identification module, the signal parameters stored in the signal parameter recording module will be input into the conservative action input calculation module. The output is the state quantity between the vehicle and the vehicle in front. The conservative DDPG_calm control module is connected to the conservative action input calculation module. It takes the state quantity as input and outputs the desired acceleration of the conservative driver.

[0120] Step 8: Adjust acceleration. The drive / brake switching module converts the desired acceleration value output by the upper controller into an acceleration / braking signal and sends the acceleration / or braking signal to the desired throttle / brake pressure calculation module. The desired throttle / brake pressure calculation module calculates the throttle opening or desired braking pressure and sends it to the actuator. The power unit control unit controls the electronic throttle according to the throttle opening, or the brake control unit controls the brake cylinder or actuator to maintain the current state according to the desired braking pressure. If the system continues to run, return to step 2 and repeat the loop to step 8. If the system is shut down, proceed to step 9.

[0121] Step Nine: End. When shutting down or parking the system, the system will stop running after step eight.

[0122] This invention designs three control modules with different control styles based on specific driving styles, and modularizes the reward function design for the system's safety, following ability, and comfort. By using control modules with different control styles for strategy training and with the assistance of reward functions, the system can continuously learn and update its strategies while ensuring that the strategies are high-value experiences and making the agent's learning more stable. The strategies of the three modes are stored in the corresponding control modules, so that each mode's strategy can make decisions that conform to driving habits based on the state information between the two vehicles collected by sensors. This meets the performance requirements of different driving styles for the ACC system and improves the system's personalization and overall performance.

[0123] The method for identifying driving style in step four is as follows:

[0124] S1. Calculate the impact intensity and read the vehicle speed transmitted from the environmental perception layer by the signal parameter recording module. Relative distance d, relative vehicle speed The impact calculation module receives the vehicle speed from the vehicle speed sensor, along with signals indicating whether there are vehicles ahead. The signal's impact J(t) is calculated using the formula:

[0125] J(t) = / (1)

[0126] In the formula, Let t be the speed of the vehicle at time t;

[0127] S2, Calculate the driver's driving state recognition coefficient. The driver state recognition coefficient calculation module calculates the driver state recognition coefficient based on the calculated impact intensity. And input it into the driving style recognition module.

[0128] Driver driving status recognition coefficient :

[0129] = = (2)

[0130] In the formula, To identify the standard deviation of the impact within the period, This represents the average impact force experienced by a standard driver. Let be the impact intensity at time i, and T be the recognition period;

[0131] S3. Determine the driver's style. In the driving style recognition module, based on the real-time calculated driver driving state recognition coefficient... To determine the driver's style, if > Then the driver's style is aggressive. This is the critical value for the driving state recognition coefficient of aggressive drivers; if < If so, the driver's style is conservative; if < < If the driver's style is determined to be "general", the driver's identification result will be stored in the driving style identification module.

[0132] In steps five, six, and seven, the state parameters between the vehicle and the vehicle in front include distance error. Bicycle speed and relative velocity The parameter calculation module calculates the distance based on the formula for the desired distance. = * Calculate the expected distance In the formula, Let be the speed of the vehicle at time t. For following distance, To maintain a safe following distance, the parameter calculation module combines the actual distance d and the vehicle's speed. and relative velocity The signal is used to calculate the distance error. Distance error Bicycle speed and relative velocity Signal parameters are saved in the parameter recording module.

[0133] Select an aggressive following distance For 1s and A standard following distance of 2 meters. For 1.5s and The following distance is 3m for the conservative type and 2s for the conservative type. It is 4m.

[0134] The control strategies of the aggressive DDPG_aggressive control module, the normal DDPG_normal control module, and the conservative DDPG_calm control module in steps five, six, and seven are obtained through training. After the training converges, they are stored in the corresponding modules. The desired acceleration is output according to the control strategy, with the state variable as input.

[0135] The training method includes a state update module, an experience update module, a network parameter update module, and a reward function module. The state update module converts the data collected by the environment perception layer into state variables and inputs them into the experience update module to update high-value experience samples and store them. The network parameter update module randomly selects experience samples from the experience update module for training and updates them in a way that maximizes the reward according to the reward given by the reward function module.

[0136] The state update module updates the relative speed between the vehicle and the vehicle in front. Bicycle speed Distance error between actual spacing and expected spacing As the state input, the network parameter update module outputs the desired acceleration as the state output. Given the state quantity at time t, the state quantity at the next time is updated. This state input and the actual acceleration are defined as a sample empirical sequence. ),in This is the state variable of the current sample. The actual acceleration of the current sample. The reward for the current sample. This represents the state of the current sample at the next time step.

[0137] (3)

[0138] In the formula, For the update time step, The speed of the vehicle in front. For the vehicle's speed, The relative speed between the vehicle and the vehicle in front. This refers to the actual distance between your vehicle and the vehicle in front. This represents the distance error between the actual vehicle spacing and the desired vehicle spacing. The desired vehicle spacing;

[0139] Initialize the state variables before training begins:

[0140] ; (4)

[0141] ; (5)

[0142] ; (6)

[0143] In the formula, It is the vehicle speed at time t=0, the start of training. It is the speed of the vehicle at time t=0 in a randomly sampled following video clip. It is the distance between the two cars at the start of training, t=0. It is the actual distance between the two vehicles at time t=0 in a randomly sampled following segment. It represents the relative speed between the vehicle and the vehicle in front at time t=0, the start of training. It is the actual relative speed between the two vehicles in a randomly sampled following segment.

[0144] like Figure 4 As shown, since the experience pool is finite, new samples continuously replace old samples to achieve the purpose of updating. However, during the agent's exploration process, low-value new sample experience may replace the experience of high-value old samples, which will lead to slow network convergence. Therefore, in order to improve the reuse rate of high-value experience samples, high-value experience samples are selected, and an experience update module is designed.

[0145] The experience update module selects high-value sample experiences and adds them to the corresponding style experience pool, collecting sample experiences from drivers of the corresponding style. ),

[0146] Define the average reward of training samples of aggressive-style drivers for:

[0147] (7)

[0148] In the formula, n is the number of iterations from the beginning of training to the current training. It is the reward obtained during the m-th training session;

[0149] Reward for gaining current experience (r) actual When r actual ≥ When the experience is zero, it is stored in the experience pool. <r actual < When, calculate the probability p=r actual / The experience is stored in the experience pool with probability p. When r actual When the value is less than 0, the experience is discarded.

[0150] For both average and conservative style drivers, the same update method as for aggressive style drivers' experience update module is applied. The average reward for average style drivers is defined as follows: The average reward for a conservative-style driver is defined as .

[0151] like Figure 5As shown, the network parameter update module includes an action network and a policy evaluation network. The action network includes an online action network and a target action network, and the policy evaluation network includes an online action evaluation network and a target action evaluation network. The action network outputs a deterministic expected acceleration based on the state variables of the vehicle and the preceding vehicle. The policy evaluation network outputs an action value evaluation function based on the quality of the actions taken by the online action network. ,

[0152] Define the parameters of the online action network as follows The target action network parameters are: The parameters of the online motion evaluation network are: The target action evaluation network parameters are: The goal of online action networks is to continuously optimize the parameters of the action network. This allows the network to output increasingly accurate and predictable expected accelerations based on the conditions of the two workshops, conforming more and more to the behavior of a human driver. The goal of the online motion evaluation network is to update its own network parameters. This makes the evaluation of the value of actions output by action networks increasingly accurate.

[0153] Action Network Update:

[0154] Random collection The online action network is trained on a sample, and then uses the data from the sample... Output To online motion assessment networks;

[0155] Construct a similarity function where the greater the deviation between the actual acceleration and the expected acceleration, the lower the value assigned.

[0156] (8)

[0157] In the formula, For actual acceleration, For sample state quantities, For online action network parameters, A deterministic policy for output by the online action network;

[0158] Online motion evaluation networks based on samples and similarity function Calculate the new action value evaluation function:

[0159] (9)

[0160] In the formula, For the action value evaluation function, For similarity functions;

[0161] To maximize This method of updating the parameters of the action network not only caters to the policy evaluation network, but also makes the expected acceleration output by the action network more consistent with human driving habits.

[0162] Update via the optimizer of the actor network The optimizer updates the parameters of the online action network using gradient ascent. :

[0163] (10)

[0165] In the formula, For online action network parameters, For the evaluation function of the value of the new action, For the action value evaluation function, For sample state quantities, For deterministic policies output by online action networks, Where N is the actual acceleration and N is the number of samples;

[0166] Action evaluation network update:

[0167] The target action network is based on the samples Output The target action evaluation network is then used to evaluate the action based on the samples. and the output of the target action network Predicting future rewards and :

[0168] = (11)

[0169] In the formula, The reward for the current sample. This represents the state of the current sample at the next time step. Let the target action evaluation function be... For the target action network parameters, The network parameters are used to evaluate the target action. As a discount factor, A deterministic strategy for the target action network;

[0170] Update the parameters of the online action evaluation network using the optimizer of the critic network. The optimizer is based on rewards and and the motion value evaluation function output by the online motion evaluation network The parameters of the online action evaluation network are updated by minimizing the loss function. :

[0171] L( )= = (12)

[0172] In the formula, For future rewards and, This is the state variable of the current sample. The actual acceleration of the current sample. The reward for the current sample. This represents the state of the current sample at the next time step. Here, N is the discount factor, and N is the sample size. For the action value evaluation function, A deterministic strategy for the target action network;

[0173] Both the target action network and the target action evaluation network update their parameters using a soft update method. and The parameters are updated periodically, allowing the target network to slowly follow the main network in updating its parameters, which greatly enhances the stability of the learning process.

[0174] (13)

[0175] In the formula, For the target action network parameters, The network parameters are used to evaluate the target action. For online action network parameters, For online motion evaluation network parameters, This refers to the following distance.

[0176] The tanh activation function is used for the output of the action network. For an aggressive driving style, the function is mapped to the range [-1, 1], thus limiting the output acceleration to [-4, 4] m / s². For a typical driving style, the function is mapped to the range [-1, 1], thus limiting the output acceleration to [-3.5, 3.5] m / s². For a conservative driving style, mapping the function to the range [-1, 1] allows the output acceleration to be limited to [-3, 3] m / s. .

[0177] Since the design of the reward function is crucial to the effectiveness of the control method, it guides the updating of parameters in the action network and policy evaluation network, playing a vital role in reinforcement learning tasks. The reward function is a function of state and action, and its design is key to reinforcement learning methods, requiring continuous optimization and experimentation to prevent opportunistic behavior by the agent. The reward function module is divided into a safety reward function, a following performance reward function, and a comfort reward function, which are linearly integrated, and their weights are continuously adjusted during training.

[0178] Safety is mainly related to the actual distance d between the vehicle and the vehicle in front. act and the relative speed between the two vehicles Related, combining the minimum safe distance d between the vehicle and the vehicle in front. safe The security reward function R1 is as follows:

[0179] R1 (14)

[0180] Where, d act d represents the actual distance between the vehicle and the vehicle in front. safe The minimum safe distance between the vehicle and the vehicle in front is set. If the actual distance between the vehicles is less than the minimum safe distance and the vehicle is still accelerating to approach, a penalty of -10 is imposed.

[0181] Following performance is measured by the deviation between the actual distance and the expected distance between the two vehicles. The larger the deviation, the greater the penalty. The following performance reward function R2 is as follows:

[0182] R2 = -| | (15)

[0183] In the formula, This represents the actual distance between your vehicle and the vehicle in front. This represents the expected distance between the vehicle and the vehicle in front.

[0184] The comfort reward function R3 is as follows:

[0185] R3 = - (16)

[0186] In the formula, Jerk represents the rate of change of acceleration. It is often used to measure comfort; the smaller the Jerk value, the better the comfort.

[0187] For aggressive driving styles, following distance is prioritized over comfort; therefore, the following time distance is included in the expected following distance. When set to 1 second, its reward function is linearly integrated as R = + + ;

[0188] For a typical driving style, the required following distance and comfort are between those of the other two driving styles. Therefore, the following distance in the expected following distance is considered as follows. Set to 1.5s, its reward function linearly integrated is R= + + ;

[0189] For a conservative driving style, comfort is prioritized over following distance; therefore, the following time distance is included in the expected following distance. When set to 2s, its reward function is linearly integrated as R= + + .

[0190] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0191] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A personalized adaptive cruise system based on deep reinforcement learning, comprising an environmental perception layer, an upper-level controller, a lower-level controller, and an actuator, wherein signals acquired by the environmental perception layer are input to the upper-level controller, the upper-level controller converts the input signals into control signals and outputs them to the lower-level controller, the lower-level controller converts the control signals output by the upper-level controller into action commands and sends them to the actuator, and the actuator operates according to the execution commands; characterized in that: The environmental perception layer includes millimeter-wave radar and a vehicle speed sensor. The millimeter-wave radar acquires the actual distance d between the vehicle and the vehicle in front within the detection range, as well as the relative vehicle speed. Signal; vehicle speed sensor is used to obtain the vehicle's speed. Signal; The upper-level controller includes a driver style recognizer, a motion input calculation module, and a deep reinforcement learning module; The driver style recognizer includes a signal parameter recording module, an impact calculation module, a driving state recognition coefficient calculation module, and a driving style recognition module. The signal parameter recording module records the vehicle speed transmitted from the environmental perception layer. Relative distance d, relative vehicle speed Based on the signals indicating whether there are vehicles ahead, the impact calculation module calculates the impact intensity, and the driving state recognition coefficient calculation module calculates the driver's driving state recognition coefficient based on the calculated impact intensity. The driving style recognition module calculates the driver's driving state recognition coefficient. Determine the driver's style, which includes three types: aggressive, average, and conservative. The action input calculation module includes an aggressive action input calculation module, a general action input calculation module, and a conservative action input calculation module. The aggressive action input calculation module, the general action input calculation module, and the conservative action input calculation module each include a desired distance calculation module, a parameter calculation module, and a parameter recording module, respectively. The deep reinforcement learning module includes an aggressive DDPG_aggressive control module, a normal DDPG_normal control module, and a conservative DDPG_calm control module. Based on the driving style identified by the driving style recognizer, the corresponding action input calculation module and control module are selected. The control strategies of the aggressive DDPG_aggressive control module, the normal DDPG_normal control module, and the conservative DDPG_calm control module are obtained through training and stored in their respective modules after training convergence. The desired acceleration is output according to the control strategy based on the state variables as input. The training includes a state update module, an experience update module, a network parameter update module, and a reward function module. The state update module converts the data collected by the environment perception layer into state variables and inputs them into the experience update module to update high-value experience samples and store them. The network parameter update module randomly selects experience samples from the experience update module for training and updates them in a way that maximizes the reward according to the reward given by the reward function module. The network parameter update module includes an action network and a policy evaluation network. The action network includes an online action network and a target action network. The policy evaluation network includes an online action evaluation network and a target action evaluation network. The action network outputs a deterministic expected acceleration based on the state variables of the vehicle and the preceding vehicle. The policy evaluation network outputs an action value evaluation function based on the quality of the actions taken by the online action network. , Define the parameters of the online action network as follows The target action network parameters are: ; The parameters of the online motion evaluation network are: The target action evaluation network parameters are: ; Action Network Update: Random collection The online action network is trained on a sample, and then uses the data from the sample... Output To online motion assessment networks; Construct a similarity function: （8） In the formula, For actual acceleration, For sample state quantities, For online action network parameters, A deterministic policy for output by the online action network; Online motion evaluation networks based on samples and similarity function Calculate the new action value evaluation function: （9） In the formula, For the action value evaluation function, For similarity functions; Update via the optimizer of the actor network The optimizer updates the parameters of the online action network using gradient ascent. : （10） In the formula, For online action network parameters, For the evaluation function of the value of the new action, For the action value evaluation function, For sample state quantities, For deterministic policies output by online action networks, Where N is the actual acceleration and N is the number of samples; Action evaluation network update: The target action network is based on the samples Output The target action evaluation network is then used to evaluate the action based on the samples. and the output of the target action network Predicting future rewards and : = （11） In the formula, The reward for the current sample. This represents the state of the current sample at the next time step. Let the target action evaluation function be... For the target action network parameters, The network parameters are used to evaluate the target action. As a discount factor, A deterministic strategy for the target action network; Update the parameters of the online action evaluation network using the optimizer of the critic network. The optimizer is based on rewards and and the motion value evaluation function output by the online motion evaluation network The parameters of the online action evaluation network are updated by minimizing the loss function. : L( )= = （12） In the formula, For future rewards and, This is the state variable of the current sample. The actual acceleration of the current sample. The reward for the current sample. This represents the state of the current sample at the next time step. Here, N is the discount factor, and N is the sample size. For the action value evaluation function, A deterministic strategy for the target action network; Both the target action network and the target action evaluation network update their parameters using a soft update method. and Updated periodically: （13） In the formula, For the target action network parameters, The network parameters are used to evaluate the target action. For online action network parameters, For online motion evaluation network parameters, This refers to the following distance; The lower-level controller includes a drive / brake switching module and a desired throttle / brake pressure calculation module. The drive / brake switching module converts the desired acceleration value output by the upper-level controller into an acceleration / braking signal and sends the acceleration or braking signal to the desired throttle / brake pressure calculation module. The desired throttle / brake pressure calculation module calculates the throttle opening or desired braking pressure and sends it to the actuator. The actuator includes a power unit control unit, an electronic throttle, a brake control unit, and a brake cylinder. The power unit control unit controls the electronic throttle according to the calculated throttle opening, and the brake control unit controls the brake cylinder according to the desired brake pressure.

2. The control method for a personalized adaptive cruise system based on deep reinforcement learning according to claim 1, characterized in that, Includes the following steps: Step 1: Start the system. The millimeter-wave radar will detect whether there is a vehicle in front within the detection range. If there is no vehicle in front, maintain the set speed. If there is a vehicle in front, proceed to Step 2. Step 2: Acquire signal parameters. Millimeter-wave radar and vehicle speed sensor respectively detect and obtain the relative distance d and relative speed with the vehicle in front. and vehicle speed Signal; Step 3: Input signal parameters. Input the signal parameters obtained in Step 2 into the signal parameter recording module and save them. Step 4: Identify driving style. The driving style identification module determines the current driver's style based on the signal parameters in the parameter recording module. If it is determined to be aggressive, proceed to step 5. If it is not aggressive, further determine whether it is moderate. If it is moderate, proceed to step 6. If it is not moderate, determine whether it is conservative, proceed to step 7. Store the driver's identification result in the driving style identification module. Step 5: Switch to aggressive control mode. If the driving style is determined to be aggressive by the driving style identification module, the signal parameters stored in the signal parameter recording module will be input into the aggressive action input calculation module. The output is the state quantity between the vehicle and the vehicle in front. The aggressive DDPG_aggressive control module is connected to the aggressive action input calculation module. It takes the state quantity as input and outputs the desired acceleration of the aggressive driver. Step 6: Switch to the general control mode. If the driving style is determined to be general by the driving style identification module, the signal parameters stored in the signal parameter recording module will be input into the general action input calculation module. The output is the state quantity between the vehicle and the vehicle in front. The general DDPG_normal control module is connected to the general action input calculation module. It takes the state quantity as input and outputs the expected acceleration of the general driver. Step 7: Switch to conservative control mode. If the driving style is determined to be conservative by the driving style identification module, the signal parameters stored in the signal parameter recording module will be input into the conservative action input calculation module. The output is the state quantity between the vehicle and the vehicle in front. The conservative DDPG_calm control module is connected to the conservative action input calculation module. It takes the state quantity as input and outputs the desired acceleration of the conservative driver. Step 8: Adjust acceleration. The drive / brake switching module converts the desired acceleration value output by the upper controller into an acceleration / braking signal and sends the acceleration / or braking signal to the desired throttle / brake pressure calculation module. The desired throttle / brake pressure calculation module calculates the throttle opening or desired braking pressure and sends it to the actuator. The power unit control unit controls the electronic throttle according to the throttle opening, or the brake control unit controls the brake cylinder or actuator to maintain the current state according to the desired braking pressure. If the system continues to run, return to step 2 and repeat the loop to step 8. If the system is shut down, proceed to step 9. Step Nine: End. When shutting down or parking the system, the system will stop running after step eight.

3. The control method for a personalized adaptive cruise system based on deep reinforcement learning according to claim 2, characterized in that, The method for identifying driving style in step four is as follows: S1. Calculate the impact intensity and read the vehicle speed transmitted from the environmental perception layer by the signal parameter recording module. Relative distance d, relative vehicle speed The impact calculation module receives the vehicle speed from the vehicle speed sensor, along with signals indicating whether there are vehicles ahead. The signal's impact J(t) is calculated using the formula: J(t)= / (1) In the formula, Let t be the speed of the vehicle at time t; S2, Calculate the driver's driving state recognition coefficient. The driver state recognition coefficient calculation module calculates the driver state recognition coefficient based on the calculated impact intensity. And input it into the driving style recognition module. Driver driving status recognition coefficient : = = （2） In the formula, To identify the standard deviation of the impact within the period, This represents the average impact force experienced by a standard driver. Let be the impact intensity at time i, and T be the recognition period; S3. Determine the driver's style. In the driving style recognition module, based on the real-time calculated driver driving state recognition coefficient... To determine the driver's style, if > Then the driver's style is aggressive. This is the critical value for the driving state recognition coefficient of aggressive drivers; if < If so, the driver's style is conservative; if < < If the driver's style is determined to be "general", the driver's identification result will be stored in the driving style identification module.

4. The control method for a personalized adaptive cruise system based on deep reinforcement learning according to claim 2, characterized in that: In steps five, six, and seven, the state parameters between the vehicle and the vehicle in front include distance error. Bicycle speed and relative velocity ; The parameter calculation module calculates the distance based on the formula for the desired distance. = * Calculate the expected distance In the formula, Let be the speed of the vehicle at time t. For following distance, To maintain a safe following distance, the parameter calculation module combines the actual distance d and the vehicle's speed. and relative velocity The signal is used to calculate the distance error. Distance error Bicycle speed and relative velocity Signal parameters are saved in the parameter recording module.

5. The control method for a personalized adaptive cruise system based on deep reinforcement learning according to claim 1, characterized in that: The state update module updates the relative speed between the vehicle and the vehicle in front. Bicycle speed Distance error between actual spacing and expected spacing As the state input, the network parameter update module outputs the desired acceleration as the state output. Given the state quantity at time t, the state quantity at the next time is updated. This state input and the actual acceleration are defined as a sample empirical sequence. ),in This is the state variable of the current sample. The actual acceleration of the current sample. The reward for the current sample. This represents the state of the current sample at the next time step. （3） In the formula, For the update time step, The speed of the vehicle in front. For the vehicle's speed, The relative speed between the vehicle and the vehicle in front. This refers to the actual distance between your vehicle and the vehicle in front. This represents the distance error between the actual vehicle spacing and the desired vehicle spacing. The desired vehicle spacing; Initialize the state variables before training begins: ； （4） ； （5） ； （6） In the formula, It is the vehicle speed at time t=0, the start of training. It is the speed of the vehicle at time t=0 in a randomly sampled following video clip. It is the distance between the two cars at the start of training, t=0. It is the actual distance between the two vehicles at time t=0 in a randomly sampled following segment. It represents the relative speed between the vehicle and the vehicle in front at time t=0, the start of training. It is the actual relative speed between the two vehicles in a randomly sampled following segment.

6. The control method for a personalized adaptive cruise system based on deep reinforcement learning according to claim 1, characterized in that: The experience update module selects high-value sample experiences and adds them to the corresponding style experience pool, collecting sample experiences from drivers of the corresponding style. ), Define the average reward of training samples of aggressive-style drivers for: （7） In the formula, n is the number of iterations from the beginning of training to the current training. It is the reward obtained during the m-th training session; Reward for gaining current experience (r) actual When r actual ≥ When the experience is zero, it is stored in the experience pool. <r actual < When, calculate the probability p=r actual / The experience is stored in the experience pool with probability p. When r actual When the value is less than 0, the experience is discarded. For both average and conservative style drivers, the same update method as for aggressive style drivers' experience update module is applied. The average reward for average style drivers is defined as follows: The average reward for a conservative-style driver is defined as .

7. The control method for a personalized adaptive cruise system based on deep reinforcement learning according to claim 1, characterized in that: The reward function module is divided into a safety reward function, a following performance reward function, and a comfort reward function, which are linearly integrated and the weights of each performance are continuously adjusted during training. The security reward function R1 is as follows: R1 （14） Where, d act d represents the actual distance between the vehicle and the vehicle in front. safe The minimum safe distance between the vehicle and the vehicle in front is set. If the actual distance between the vehicles is less than the safe distance and the vehicle is still accelerating to approach, a penalty of -10 is imposed. The following reward function R2 is as follows: R2 = -| | （15） In the formula, This represents the actual distance between your vehicle and the vehicle in front. This represents the expected distance between the vehicle and the vehicle in front. The comfort reward function R3 is as follows: R3 = - （16） In the formula, This represents the rate of change of acceleration.

Images