Intelligent cruise control method, device, electronic device and storage medium

Abstract

The embodiment of the present invention provides an intelligent cruise control method, device, electronic device and storage medium, wherein the method comprises: determining the current state signal of an automatic control vehicle; inputting the current state signal of the automatic control vehicle into an intelligent optimization control model to realize intelligent cruise control of the automatic control vehicle; wherein the intelligent optimization control model is obtained by training the neural network parameters of a Markov decision process model based on real-time state samples collected from a vehicle queue formed by the automatic control vehicle. The present invention solves the problems of unpredictability of complex traffic environments and unreliability of networks in the current cruise control method based on network control.

Description

Technical Field

[0001] The present invention relates to the field of automatic control technology, and in particular to an intelligent cruise control method, device, electronic equipment and storage medium. Background Art

[0002] Cruise control is an advanced assisted driving method that can effectively reduce the burden on drivers and improve road traffic efficiency, driving safety and fuel economy. At present, although cruise control methods based on network control such as adaptive cruise control (ACC), cooperative adaptive cruise control (CACC) and connected cruise control (CCC) have received widespread attention and application, there are still many limitations. For example, the ACC method combines multiple sensor technologies to perceive road traffic information. Due to the poor perception sensitivity of the sensor and its susceptibility to interference from the external environment, the stability and safety of the ACC method are insufficient. On the basis of ACC, the CACC method introduces the vehicle-to-vehicle (V2V) communication technology in the Internet of Vehicles to promote the vehicles in the fleet to actively exchange their motion status information. However, the CACC method requires that each vehicle in the fleet is equipped with ACC automatic driving equipment to assist in cooperative control, and its communication topology is usually fixed. When there are manually driven vehicles in the fleet or the road conditions change, it will inevitably lead to a decrease in the performance and stability of CACC, which also limits its application in future traffic scenarios. In order to achieve more flexible vehicle platoon design, connection structure and communication topology, the CCC proposed further allows the controlled vehicle to receive status information broadcast by multiple vehicles in front without equipping all vehicles with sensors, while improving the information perception and control capabilities of each vehicle without the need to uniformly design the entire platoon. Although the CCC system does not require a designated head vehicle or a fixed communication structure, it can communicate selectively, allowing modular design and better scalability. However, under the constraints of environmental changes, controlled vehicle movement, network node transmission capacity and link quality, its topology, network communication delay and expected state will be dynamic and time-varying. The unpredictability of complex traffic environments and the unreliability of networks will bring severe challenges to cruise control methods based on networked control. Summary of the invention

[0003] The embodiments of the present invention provide an intelligent cruise control method, device, electronic device and storage medium to solve some or all of the above-mentioned problems existing in the current cruise control method based on network control.

[0004] In a first aspect, an embodiment of the present invention provides an intelligent cruise control method, comprising:

[0005] determining a current state signal of an automatically controlled vehicle;

[0006] Inputting the current state signal of the automatic control vehicle into the intelligent optimization control model to implement intelligent cruise control of the automatic control vehicle;

[0007] The intelligent optimization control model is obtained by training the neural network parameters of the Markov decision process model based on the real-time state samples collected from the vehicle queue formed by the automatic control vehicles.

[0008] Preferably, the construction process of the Markov decision process model comprises the following steps:

[0009] Acquiring queue state information of a vehicle queue formed by automatically controlling vehicles, and establishing a dynamic equation of a queue system according to the queue state information;

[0010] According to the dynamic equation of the queue system, a quadratic optimization control equation is constructed with minimization of state error and input as the objective function;

[0011] A Markov decision process model for networked control is constructed based on the dynamic equation of the queue system and the quadratic optimization control equation.

[0012] Preferably, the step of acquiring queue state information of a vehicle queue formed by automatically controlling vehicles and establishing a dynamic equation of a queue system according to the queue state information comprises the following steps:

[0013] Obtain the distance, speed and acceleration information of each vehicle in the vehicle queue through vehicle-to-vehicle communication;

[0014] Establishing a dynamic equation for each vehicle in the vehicle queue according to the vehicle distance, vehicle speed and acceleration information of each vehicle in the vehicle queue;

[0015] Obtaining the expected speed of the leading vehicle, obtaining the expected distance between each vehicle based on a preset range strategy, and establishing a state error equation for each vehicle according to the expected speed of the leading vehicle, the expected distance between each vehicle, and the current speed and distance between each vehicle;

[0016] The state error equations of the vehicles are combined with the state equations of the vehicles in the queue in continuous time, and the dynamic equations of the queue system are obtained after discretization.

[0017] Preferably, the pre-set range strategy includes:

[0018] If the current vehicle distance is less than the preset minimum vehicle distance, the expected vehicle speed is 0;

[0019] If the current vehicle distance is not less than the preset minimum vehicle distance and not greater than the preset maximum vehicle distance, the expected vehicle speed is obtained according to the preset maximum vehicle speed, the current vehicle distance, the preset minimum vehicle distance and the preset maximum vehicle distance. The calculation formula is: Where V(h) represents the expected vehicle speed, h represents the vehicle distance, and h min Indicates the preset minimum vehicle distance, h max Indicates the preset maximum vehicle distance, v max Indicates the preset maximum vehicle speed;

[0020] If the current vehicle distance is greater than the preset maximum vehicle distance, the expected vehicle speed is the preset maximum vehicle speed;

[0021] The expected vehicle distance between each vehicle is obtained according to the expected vehicle speed.

[0022] Preferably, the dynamic equation of the queue system obtained after the discretization process is as follows:

[0023] y i+1 =A0y i +B1u i +B2u i-1 ;

[0024] Among them, y i =y(iΔT) and u i =u(iΔT) respectively represent the state variables and acceleration control strategy at the current moment,

[0025]

[0026] i is the sampling interval number, ΔT is the sampling interval, τ is the network induced delay, λ j and represents the system parameters related to human driving behavior, j is the vehicle number in the queue, m is the total number of vehicles in the queue except the head vehicle, is the partial derivative of the range strategy at the desired vehicle distance.

[0027] Preferably, according to the dynamic equation of the queue system, a quadratic optimization control equation is constructed with minimization of state error and input as the objective function as follows:

[0028]

[0029] Where N is the number of sampling intervals, and C and D are coefficient matrices:

[0030]

[0031] c1 and c2 are preset coefficients.

[0032] Preferably, the intelligent optimization control model is obtained by training the neural network parameters of the Markov decision process model based on the real-time state samples collected from the vehicle platoon formed by the automatic control vehicles, and includes:

[0033] Constructing a deep deterministic policy gradient algorithm including a current actor network, a current critic network, a target actor network, and a target critic network to update the parameters of the Markov decision process model;

[0034] In each time slot, according to the input state s k , the current actor network will output the corresponding action strategy μ(s k |θ μ ), execution strategy And according to the state transfer function, the next state s is obtained k+1 , and the corresponding reward r is obtained according to the reward function k , will (s k , a k ,s k+1 , r k ) is stored as a sample in the experience playback buffer to obtain a state sample; wherein,

[0035] The current critic network updates its parameters θ by minimizing the following mean square error loss function: Q :

[0036]

[0037] Among them, M is the number of samples in the mini-batch sampling, Q(s t ,a t |θ Q ) is the current Q value, by t with a t Input into the current critic network to get, x t is the target Q value, expressed as:

[0038] x t =r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ )

[0039] In the formula, r t is the corresponding reward function value, Q′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ) is the next Q value generated by the target critic network, μ′(s t+1 |θ μ′ ) is the target actor network according to the input state s t+1 The generated next action strategy;

[0040] The current actor network updates its parameters θ through the following policy gradient function μ :

[0041]

[0042] in, is the gradient operator;

[0043] The target actor network and the target critic network update their parameters θ respectively in the following way Q' and θ μ' :

[0044] θ Q′ ←δθ Q +(1-δ)θ Q′

[0045] θ μ′ ←δθ μ +(1-δ)θ μ′

[0046] Among them, δ is a fixed constant, 0<δ<<1.

[0047] In a second aspect, an embodiment of the present invention provides an intelligent cruise control device, including a state signal unit and an intelligent control unit;

[0048] The state signal unit is used to determine the current state signal of the automatic control vehicle;

[0049] The intelligent control unit is used to input the current state signal of the automatic control vehicle into the intelligent optimization control model to implement intelligent cruise control of the automatic control vehicle;

[0050] The intelligent optimization control model is obtained by training a Markov decision process model based on real-time state samples collected from a vehicle platoon formed by the automatic control vehicles.

[0051] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the intelligent cruise control method as provided in any one of the first aspects above are implemented.

[0052] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent cruise control method as provided in any one of the first aspects above.

[0053] An intelligent cruise control method, device, electronic device and storage medium provided by an embodiment of the present invention realizes intelligent cruise control of the automatic control vehicle by inputting the current state signal of the automatic control vehicle into the intelligent optimization control model; wherein the intelligent optimization control model is obtained by training the neural network parameters of the Markov decision process model based on the real-time state samples collected from the vehicle queue formed by the automatic control vehicle. The embodiment of the present invention can continuously and intelligently learn and adjust the optimization control strategy of the networked cruise control by continuously interacting with the environment, so as to adapt to the actual complex and changeable network dynamic scenes, realize the safe and stable driving of the autonomous driving vehicle, and thus solve the problems of the unpredictability of complex traffic environments and the unreliability of networks in the current cruise control methods based on networked control. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 It is a flow chart of an intelligent cruise control method provided by the present invention;

[0056] Figure 2 It is a schematic diagram of a smart cruise control scenario based on network control provided by the present invention;

[0057] Figure 3 It is a diagram of the intelligent cruise control architecture based on network control provided by the present invention;

[0058] Figure 4 It is a structural schematic diagram of an intelligent cruise control device provided by the present invention;

[0059] Figure 5 It is a block diagram of the intelligent optimization control module provided by the present invention;

[0060] Figure 6 It is a block diagram of the system modeling module provided by the present invention;

[0061] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Combine the following Figure 1-Figure 7 The present invention describes an intelligent cruise control method, device, electronic device and storage medium.

[0064] An embodiment of the present invention provides an intelligent cruise control method. Figure 1 A flow chart of the intelligent cruise control method provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the method includes:

[0065] Step 110, determining a current state signal of the automatic control vehicle;

[0066] Specifically, in the embodiment of the present invention, the vehicle queue includes manually driven vehicles and CCC vehicles. Each vehicle in the queue is equipped with communication equipment. Through V2V communication technology, the CCC autonomous driving vehicle can receive status information from other vehicles, including headway, vehicle speed and acceleration.

[0067] Step 120, inputting the current state signal of the automatic control vehicle into the intelligent optimization control model to implement intelligent cruise control of the automatic control vehicle;

[0068] The intelligent optimization control model is obtained by training the neural network parameters of the Markov decision process model based on the real-time state samples collected from the vehicle queue formed by the automatic control vehicles.

[0069] Specifically, by analyzing vehicle dynamics and wireless network characteristics, the dynamic equations of the vehicle platoon system are constructed. The impact of the dynamic and time-varying network communication delay and the expected state is considered, and an optimization control problem is established to construct an MDP model. The DRL algorithm is used to generate samples and train the neural network through continuous interaction with the environment. Finally, an intelligent optimization control strategy for the automatically controlled vehicle is obtained, which enables the automatically controlled vehicle to track the ideal expected speed and always maintain a safe distance with the vehicle in front. At the same time, it ensures the smooth operation of the control system and the vehicle platoon under dynamic network conditions.

[0070] The method provided by the embodiment of the present invention can continuously and intelligently learn and adjust the optimization control strategy of the networked cruise control through continuous interaction with the environment, thereby adapting to the actual complex and changeable network dynamic scenarios and achieving safe and smooth driving of CCC autonomous driving vehicles.

[0071] Based on any of the above embodiments, the process of constructing the Markov decision process model includes the following steps:

[0072] Acquiring queue state information of a vehicle queue formed by automatically controlling vehicles, and establishing a dynamic equation of a queue system according to the queue state information;

[0073] It should be noted that, due to the flexible network topology between vehicles in the CCC system, each vehicle can communicate with nearby vehicles. Through wireless V2V communication, CCC vehicles can obtain real-time status information such as headway, speed and acceleration of other vehicles in the fleet, so that the entire vehicle queue can be modeled. At the same time, since CCC can provide services for heterogeneous vehicle queues, the order and number of manually driven vehicles and CCC automatically controlled vehicles in the fleet are variable, which is more in line with the requirements of vehicle queue flexibility in real traffic scenarios. Usually, automatically controlled vehicles do not need to consider the vehicle status of the vehicles behind them. In order to describe the technical solution more clearly, the embodiment of the present invention takes the rear vehicle as a CCC automatically controlled vehicle and the other vehicles as manually driven vehicles as an example. In addition, the method provided in the embodiment of the present invention is also applicable to the control of automatically controlled vehicles in more complex models. When the queue model changes, the modeling method proposed in the embodiment of the present invention can be used to construct the corresponding system dynamic equation according to the specific situation of the queue.

[0074] According to the dynamic equation of the queue system, a quadratic optimization control equation is constructed with minimization of state error and input as the objective function;

[0075] It should be noted that the goal of cruise control is to enable the vehicles in the vehicle queue to track the desired speed and maintain the desired distance, while achieving comfortable and smooth acceleration control. Therefore, with the goal of minimizing the speed and distance errors and control input, a quadratic optimization control problem can be constructed. However, on the one hand, due to the high-dimensional state space and complex physical properties, it is difficult to directly obtain an analytical solution to this optimization control problem. On the other hand, due to the influence of actual network communication delays and the dynamic time-varying characteristics of the desired state, the traditional optimization decision-making method that relies on fixed parameter models and static strategies often has high robustness and stability risks. Therefore, the embodiment of the present invention proposes an intelligent optimization control method based on DRL (Deep Reinforcement Learning) to improve the adaptability and stability of automatic control vehicles under complex dynamic conditions.

[0076] A Markov decision process model for networked control is constructed based on the dynamic equation of the queue system and the quadratic optimization control equation.

[0077] It should be noted that the reinforcement learning (RL) problem is usually described by MDP (Markov Decision Process), which generally includes state, action, state transfer function and reward function. The MDP model of the system is established according to the system model and the optimization problem. According to the MDP model, an intelligent optimization control strategy is obtained by using an algorithm based on deep reinforcement learning (DRL). Traditional artificial intelligence algorithms based on discrete actions, such as Q-learning, DQN (Deep Q-learning), actor-critic, etc., often suffer from performance degradation due to poor convergence and stability when dealing with control problems with continuous action values ​​such as cruise control. The embodiment of the present invention is based on the Deep Deterministic Policy Gradient (DDPG) algorithm in DRL. According to the defined MDP model, samples are collected and trained by continuously interacting with the environment, and the neural network parameters are continuously optimized with the goal of maximizing the reward function. Finally, the intelligent optimization control strategy output signal can be generated in real time according to the current state input of the CCC automatic control vehicle, thereby achieving safe and stable control of the CCC automatic control vehicle.

[0078] Based on any of the above embodiments, the step of obtaining queue state information of a vehicle queue formed by automatically controlling vehicles and establishing a dynamic equation of a queue system according to the queue state information comprises the following steps:

[0079] Obtain the distance, speed and acceleration information of each vehicle in the vehicle queue through vehicle-to-vehicle communication;

[0080] Establishing a dynamic equation for each vehicle in the vehicle queue according to the vehicle distance, vehicle speed and acceleration information of each vehicle in the vehicle queue;

[0081] Obtaining the expected speed of the leading vehicle, obtaining the expected distance between each vehicle based on a preset range strategy, and establishing a state error equation for each vehicle according to the expected speed of the leading vehicle, the expected distance between each vehicle, and the current speed and distance between each vehicle;

[0082] The state error equations of the vehicles are combined with the state equations of the vehicles in the queue in continuous time, and the dynamic equations of the queue system are obtained after discretization.

[0083] Specifically, a queue system model is established according to the queue, including:

[0084] Collect the distance, speed and acceleration information of each vehicle in the queue based on V2V communication;

[0085] According to the information of vehicle distance, vehicle speed and acceleration, the dynamic equation of each vehicle in the queue is established;

[0086] The expected speed of the leading vehicle is obtained, and combined with the range strategy, the expected distance between each vehicle is obtained;

[0087] According to the expected vehicle speed and expected vehicle distance as well as the current vehicle speed and vehicle distance of each vehicle, the state error equation of each vehicle is established;

[0088] By combining the state error equations of each vehicle, the queue state equation based on continuous time is obtained, and after discretization processing, the queue system model based on discrete time can be obtained.

[0089] Since wireless V2V communication is introduced to promote the sharing and communication of status information between vehicles, the vehicle dynamic equation with time delay is obtained by analyzing the influence of the delay characteristics in the wireless network on the CCC automatic control vehicle. Then, the state error equations of all manually driven vehicles and CCC automatic driving vehicles in the queue are combined to obtain the continuous time system state error equation. Then, the embodiment of the present invention discretizes the continuous time system state equation through sampling to obtain a queue system model based on discrete time.

[0090] Based on any of the above embodiments, the pre-set range strategy includes:

[0091] If the current vehicle distance is less than the preset minimum vehicle distance, the expected vehicle speed is 0;

[0092] If the current vehicle distance is not less than the preset minimum vehicle distance and not greater than the preset maximum vehicle distance, the expected vehicle speed is obtained according to the preset maximum vehicle speed, the current vehicle distance, the preset minimum vehicle distance and the preset maximum vehicle distance. The calculation formula is: Where V(h) represents the expected vehicle speed, h represents the vehicle distance, and h min Indicates the preset minimum vehicle distance, h max Indicates the preset maximum vehicle distance, v max Indicates the preset maximum vehicle speed;

[0093] If the current vehicle distance is greater than the preset maximum vehicle distance, the expected vehicle speed is the preset maximum vehicle speed;

[0094] The expected vehicle distance between each vehicle is obtained according to the expected vehicle speed.

[0095] It should be noted that dynamic analysis is performed on manually driven vehicles and CCC automatically controlled vehicles respectively, and the state information of each vehicle in the queue, such as vehicle distance, speed, and acceleration, is obtained through V2V communication. Then, the vehicle dynamic equation can be established based on the relationship between them. The speed of the head vehicle in the queue is used as the expected speed of other vehicles, and the expected vehicle distance can be obtained according to the range strategy. After obtaining the expected speed and expected vehicle distance, the state error equation of each vehicle can be obtained. Among them, the expected vehicle distance and speed meet the following range strategy:

[0096]

[0097] Wherein, V(h) represents the desired vehicle speed, h represents the current vehicle distance, and h min Indicates the preset minimum vehicle distance, h max Indicates the preset maximum vehicle distance, v max Indicates the preset maximum vehicle speed.

[0098] Based on any of the above embodiments, the dynamic equation of the queue system obtained after the discretization process is as follows:

[0099] y i+1 =A0y i +B1u i +B2u i-1 ;

[0100] Among them, y i =y(iΔT) and u i =u(iΔT) respectively represent the state variables and acceleration control strategy at the current moment,

[0101]

[0102] i is the sampling interval number, ΔT is the sampling interval, τ is the network induced delay, λ j and represents the system parameters related to human driving behavior, j is the vehicle number in the queue, m is the total number of vehicles in the queue except the head vehicle, is the partial derivative of the range strategy at the desired vehicle distance.

[0103] Based on any of the above embodiments, according to the dynamic equation of the queue system, a quadratic optimization control equation is constructed with minimizing the state error and input as the objective function as follows:

[0104]

[0105] Where N is the number of sampling intervals, and C and D are coefficient matrices:

[0106]

[0107] c1 and c2 are preset coefficients.

[0108] Specifically, Figure 2 A schematic diagram of a smart cruise control scenario based on networked control provided in an embodiment of the present invention. For ease of understanding, the vehicle queue in an embodiment of the present invention consists of m+1 vehicles, of which the last vehicle, i.e., vehicle #1, is a CCC autonomous driving vehicle, and the other vehicles are all manually driven vehicles. The front vehicle, i.e., vehicle #m+1, is the head vehicle. Each vehicle in the queue is equipped with communication equipment. Through V2V communication technology, the CCC autonomous driving vehicle can receive status information from other vehicles, including headway, speed, and acceleration. In order to clearly illustrate the technical solution of an embodiment of the present invention, the head vehicle in an embodiment of the present invention serves as the tracking target of the CCC autonomous driving vehicle and travels at a dynamically changing speed.

[0109] like Figure 2 As shown, the dynamic equations of a human manually driving a vehicle can be defined as follows:

[0110]

[0111] Among them, v j (t) represents the speed of the jth vehicle, h j (t) represents the distance between the jth vehicle and the previous vehicle, represents the differential of v(t) with respect to time t, λ j and represents the system parameters related to human driving behavior, and V(h) is the expected speed based on the vehicle distance.

[0112] The dynamic equation of the CCC autonomous driving vehicle can be defined as follows:

[0113]

[0114] Among them, u(t) represents the control strategy, that is, the acceleration of the CCC autonomous driving vehicle, and τ(t) represents the network-induced delay in the networked control process.

[0115] The goal of each car in the convoy is to achieve the desired distance h * (t) and expected vehicle speed v * (t) = V(h * (t)). The vehicle distance error can be defined based on the deviation between the actual state and the expected state: Speed ​​error According to the vehicle dynamics model, a linear first-order approximation is used The error dynamics model of the vehicle queue can be obtained as:

[0116]

[0117] Define the state vector:

[0118] The error dynamics equations of each vehicle are combined to obtain the system dynamic equation:

[0119]

[0120] In the above formula,

[0121]

[0122] By sampling the discretized system dynamic equation, the discrete time system dynamic model of the i-th sampling interval [iΔT, (i+1)ΔT) is obtained as follows:

[0123] y i+1 =A0y i +B1u i +B2u i-1

[0124] Among them, y i =y(iΔT) and u i =u(iΔT) respectively represent the state variables and acceleration control strategy at the current moment, ΔT represents the sampling interval, and other parameters are:

[0125]

[0126] The goal of cruise control is to make the vehicles follow the target distance and speed so that the entire convoy always maintains a balanced state. * ≡0. In order to achieve optimal control, the quadratic cost function is defined as:

[0127]

[0128] In the above formula, N is the number of sampling intervals, and C and D are coefficient matrices:

[0129]

[0130] Wherein, c1 and c2 are preset coefficients, which can be taken as 1 and 0.1 respectively in the embodiment of the present invention.

[0131] In summary, the cruise control system optimization problem can be constructed as:

[0132]

[0133] sty i+1 =A0y i +B1u i +B2u i-1

[0134] Based on the influence of the dynamic time-varying characteristics of the network, in order to improve the environmental adaptability and self-learning ability of the networked intelligent cruise control system, an embodiment of the present invention proposes a DRL-based intelligent optimization control method to solve the above optimization problem.

[0135] MDP is usually used to formally describe RL problems. At each time slot k, the agent observes the current state from the environment and makes a decision. After executing the action, it obtains the next state and adjusts the strategy through the feedback reward value. The embodiment of the present invention defines the state, action, state transition function and reward function in MDP based on the cruise control system model constructed in the network dynamic scenario and the optimization problem.

[0136] 1) Status

[0137] Considering that the optimization control strategy is jointly affected by the current state and the delayed control signal caused by network delay, the new state vector is defined as:

[0138]

[0139] 2) Action

[0140] For a networked cruise control system, the action can be defined as an acceleration control strategy:

[0141] a k =u k

[0142] 3) State transfer function

[0143] According to the discrete time system model of the networked cruise control system and the state vector s k , the state transfer function can be expressed as:

[0144] s k+1 =s k E+a k F

[0145] in,

[0146]

[0147] 4) Reward Function

[0148] Different from minimizing the cost function in optimization theory, the goal of the intelligent algorithm is to maximize the long-term cumulative reward value, so the reward function can be defined as:

[0149]

[0150] in,

[0151]

[0152] The long-term cumulative reward value is called the return, which is expressed as follows:

[0153]

[0154] In the above formula, 0<γ<1 is the discount factor.

[0155] Since the action values ​​of the cruise control system are continuous, the DDPG method in DRL can well solve the problem of system performance degradation caused by discrete action design. Therefore, the embodiment of the present invention proposes an intelligent optimization control method based on DDPG to obtain an intelligent control strategy, thereby improving the convergence and stability of the system.

[0156] Based on any of the above embodiments, the intelligent optimization control model is obtained by training the neural network parameters of the Markov decision process model based on the real-time state samples collected from the vehicle queue formed by the automatic control vehicle, and includes:

[0157] Constructing a deep deterministic policy gradient algorithm including a current actor network, a current critic network, a target actor network, and a target critic network to update the parameters of the Markov decision process model;

[0158] It should be noted that the intelligent cruise control architecture based on network control is Figure 3 As shown in the figure, DDPG mainly includes four deep neural networks: the current actor network μ(s|θ μ ), the target actor network μ′(s|θ μ′ ), the current critic network Q(s,a|θ Q ), target critic network Q′(s,a|θ Q′ ), where μ(·) is a deterministic action strategy, Q(·) is an action value evaluation function, and θ represents the corresponding neural network parameters. The agent learns the control strategy μ by training the actor network, and evaluates the control strategy by training the critic network to obtain the corresponding Q value.

[0159] In each time slot, according to the input state s k , the current actor network will output the corresponding action strategy μ(s|θ μ ), execution strategy And according to the state transfer function, the next state s is obtained k+1 , and the corresponding reward r is obtained according to the reward function k , will (s k , a k ,s k+1 , r k ) is stored as a sample in the experience playback buffer to obtain a state sample; wherein,

[0160] The current critic network updates its parameters θ by minimizing the following mean square error loss function: Q :

[0161]

[0162] Among them, M is the number of samples in the mini-batch sampling, Q(s t ,a t |θ Q ) is the current Q value, by t with a t Input into the current critic network to get, x t is the target Q value, expressed as:

[0163] x t =r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ )

[0164] In the formula, r t is the corresponding reward function value, Q′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ) is the next Q value generated by the target critic network, μ′(s t+1 |θ μ′ ) is the target actor network according to the input state s t+1 The generated next action strategy;

[0165] The current actor network updates its parameters θ through the following policy gradient function μ :

[0166]

[0167] in, is the gradient operator;

[0168] The target actor network and the target critic network update their parameters θ respectively in the following way Q' and θ μ' :

[0169] θ Q′ ←δθ Q +(1-δ)θ Q′

[0170] θ μ′ ←δθ μ+(1-δ)θ μ′

[0171] Among them, δ is a fixed constant, 0<δ<<1.

[0172] Specifically, the intelligent cruise control method based on networked control can be divided into two steps: sampling and training.

[0173] 1) Sampling

[0174] First, we need to collect enough samples for training. In each time slot, according to the input state s k , the current actor network will output the corresponding action strategy μ(s|θ μ ). In order to ensure effective exploration in the continuous action space, random noise η is added to obtain the exploration strategy:

[0175]

[0176] Execution strategy According to the state transfer function, the next state s can be obtained k+1 , and the corresponding reward r is obtained according to the reward function k , then (s k ,a k ,s k+1 ,r k ) is stored as a sample in the experience replay buffer. Repeat the above steps to generate enough samples.

[0177] 2) Training

[0178] The training process of the embodiment of the present invention is based on 200 time slots as one episode. In each episode, M samples (s t ,a t ,s t+1 ,r t ) is used for training to reduce the correlation of sample data and improve training efficiency.

[0179] The current critic network updates its parameters θ by minimizing the following mean square error loss function: Q :

[0180]

[0181] Among them, M is the number of samples in the mini-batch sampling, Q(s t ,a t |θ Q ) is the current Q value, by t with a t Input into the current critic network to get, x tis the target Q value, which can be expressed as:

[0182] x t =r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ )

[0183] In the above formula, r t is the corresponding reward function value, Q′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ) is the next Q value generated by the target critic network, μ′(s t+1 |θ μ′ ) is the target actor network according to the input state s t+1 The generated next action policy.

[0184] The current actor network updates its parameters θ through the following policy gradient function μ :

[0185]

[0186] Among them, M is the number of samples sampled in a small batch, ▽ is the gradient operator, and the main goal of the above formula is to increase the probability of the current actor network obtaining an action with a larger Q value.

[0187] Then, the target actor network and the target critic network update their parameters θ respectively through the following "soft update" method Q' and θ μ' :

[0188] θ Q′ ←δθ Q +(1-δ)θ Q′

[0189] θ μ′ ←δθ μ +(1-δ)θ μ′

[0190] Among them, 0<δ<<1 is a fixed constant.

[0191] Finally, after enough training episodes, the optimized current actor network parameters can be obtained. Therefore, according to the input state s obtained each time, the current actor network can generate the optimal control strategy of the networked cruise control system in real time as follows:

[0192]

[0193] An intelligent cruise control device provided by the present invention is described below, and the intelligent cruise control method described below and above can be referred to each other.

[0194] Figure 4 A schematic diagram of the structure of an intelligent cruise control device provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the device includes a state signal unit 410 and an intelligent control unit 420;

[0195] The state signal unit 410 is used to determine the current state signal of the automatic control vehicle;

[0196] The intelligent control unit 420 is used to input the current state signal of the automatic control vehicle into the intelligent optimization control model to implement intelligent cruise control of the automatic control vehicle;

[0197] The intelligent optimization control model is obtained by training a Markov decision process model based on real-time state samples collected from a vehicle platoon formed by the automatic control vehicles.

[0198] The device provided by the embodiment of the present invention can continuously and intelligently learn and adjust the optimization control strategy of the networked cruise control through continuous interaction with the environment, thereby adapting to the actual complex and changeable network dynamic scenarios and achieving safe and smooth driving of CCC autonomous driving vehicles.

[0199] Based on any of the above embodiments, the intelligent control unit includes an intelligent optimization control module;

[0200] like Figure 5 As shown, the intelligent optimization control module includes a system modeling module 510, a problem building module 520, an MDP building module 530 and a calculation processing module 540;

[0201] The system modeling module 510 is used to obtain queue state information of a vehicle queue formed by automatic control vehicles, and establish a dynamic equation of the queue system according to the queue state information;

[0202] The problem construction module 520 is used to construct a quadratic optimization control equation based on the dynamic equation of the queue system with minimization of state error and input as the objective function;

[0203] The MDP construction module 530 is used to construct a Markov decision process model of networked control according to the dynamic equation of the queue system and the quadratic optimization control equation;

[0204] The computing and processing module 540 is used to generate samples and train based on the DRL algorithm and the environment to continuously interact, so as to obtain an intelligent optimization control strategy.

[0205] Based on any of the above embodiments, Figure 6 As shown, the system modeling module includes a state acquisition module 610, a dynamic construction module 620, a state error construction module 630 and a system dynamic module 640;

[0206] The state acquisition module 610 is used to acquire the distance, speed and acceleration information of each vehicle in the vehicle queue through vehicle-to-vehicle communication;

[0207] The dynamic construction module 620 is used to establish a dynamic equation for each vehicle in the vehicle queue according to the vehicle distance, vehicle speed and acceleration information of each vehicle in the vehicle queue;

[0208] The state error building module 630 is used to obtain the expected vehicle speed through the lead vehicle, obtain the expected vehicle distance of each vehicle based on a preset range strategy, and establish the state error equation of each vehicle according to the expected vehicle speed of the lead vehicle, the expected vehicle distance of each vehicle and the current vehicle speed and vehicle distance of each vehicle;

[0209] The system dynamics module 640 is used to combine the state error equations of each vehicle and obtain the dynamic equation of the queue system after discretization based on the state equations of each vehicle in the queue in continuous time.

[0210] Based on any of the above embodiments, the pre-set range strategy includes:

[0211] If the current vehicle distance is less than the preset minimum vehicle distance, the expected vehicle speed is 0;

[0212] If the current vehicle distance is not less than the preset minimum vehicle distance and not greater than the preset maximum vehicle distance, the expected vehicle speed is obtained according to the preset maximum vehicle speed, the current vehicle distance, the preset minimum vehicle distance and the preset maximum vehicle distance. The calculation formula is: Where V(h) represents the expected vehicle speed, h represents the vehicle distance, and h min Indicates the preset minimum vehicle distance, h max Indicates the preset maximum vehicle distance, v max Indicates the preset maximum vehicle speed;

[0213] If the current vehicle distance is greater than the preset maximum vehicle distance, the expected vehicle speed is the preset maximum vehicle speed;

[0214] The expected vehicle distance between each vehicle is obtained according to the expected vehicle speed.

[0215] Based on any of the above embodiments, the dynamic equation of the queue system obtained after the discretization process is as follows:

[0216] y i+1 =A0y i +B1u i +B2ui-1 ;

[0217] Among them, y i =y(iΔT) and u i =u(iΔT) respectively represent the state variables and acceleration control strategy at the current moment,

[0218]

[0219] i is the sampling interval number, ΔT is the sampling interval, τ is the network induced delay, λ j and represents the system parameters related to human driving behavior, j is the vehicle number in the queue, m is the total number of vehicles in the queue except the head vehicle, is the partial derivative of the range strategy at the desired vehicle distance.

[0220] Based on any of the above embodiments, according to the dynamic equation of the queue system, a quadratic optimization control equation is constructed with minimizing the state error and input as the objective function as follows:

[0221]

[0222] Where N is the number of sampling intervals, and C and D are coefficient matrices:

[0223]

[0224] c1 and c2 are preset coefficients.

[0225] Based on any of the above embodiments, the intelligent optimization control model is obtained by training the neural network parameters of the Markov decision process model based on the real-time state samples collected from the vehicle queue formed by the automatic control vehicle, and includes:

[0226] Constructing a deep deterministic policy gradient algorithm including a current actor network, a current critic network, a target actor network, and a target critic network to update the parameters of the Markov decision process model;

[0227] In each time slot, according to the input state s k , the current actor network will output the corresponding action strategy μ(s|θ μ ), execution strategy And according to the state transfer function, the next state s is obtained k+1 , and the corresponding reward r is obtained according to the reward function k , will (s k , a k ,s k+1 , r k) is stored as a sample in the experience playback buffer to obtain a state sample; wherein,

[0228] The current critic network updates its parameters θ by minimizing the following mean square error loss function: Q :

[0229]

[0230] Among them, M is the number of samples in the mini-batch sampling, Q(s t ,a t |θ Q ) is the current Q value, by t with a t Input into the current critic network to get, x t is the target Q value, expressed as:

[0231] x t =r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ )

[0232] In the formula, r t is the corresponding reward function value, Q′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ) is the next Q value generated by the target critic network, μ′(s t+1 |θ μ′ ) is the target actor network according to the input state s t+1 The generated next action strategy;

[0233] The current actor network updates its parameters θ through the following policy gradient function μ :

[0234]

[0235] Where M is the number of samples in the mini-batch sampling, and ▽ is the gradient operator;

[0236] The target actor network and the target critic network update their parameters θ respectively in the following way Q' and θ μ' :

[0237] θ Q′ ←δθ Q +(1-δ)θ Q′

[0238] θμ′ ←δθ μ +(1-δ)θ μ′

[0239] Among them, δ is a fixed constant, 0<δ<<1.

[0240] In summary, the intelligent cruise control method and device provided by the embodiment of the present invention, by comprehensively analyzing the vehicle dynamics and wireless network characteristics, constructs the dynamic equation of the overall vehicle queue system, considers the influence of the dynamic time-varying network communication delay and the expected state, establishes the optimization control problem, thereby constructs the MDP model, adopts the intelligent algorithm based on DRL, generates samples and trains the neural network through continuous interaction with the environment, and continuously accumulates experience, thereby obtaining the intelligent optimization control strategy of the automatic control vehicle, which not only enables the automatic control vehicle to track the ideal expected speed and always maintain a safe distance with the vehicle in front, but also ensures that it can operate autonomously and smoothly in the actual complex network dynamic scene. That is, the embodiment of the present invention obtains the intelligent optimization control strategy of the cruise control system based on network control by modeling the vehicle queue as a whole, and combines the optimization control theory and the artificial intelligence method in the scenario of dynamic changes in network communication delay and system expected state, thereby realizing the stable control of the CCC automatic control vehicle. The advantage of the present invention is that the network control and artificial intelligence technology are applied to the vehicle automatic cruise control system, the influence of the complex dynamic environment on the control system is considered, and then the DRL-based method is designed to obtain the intelligent optimization control strategy, which promotes the environmental adaptability and self-learning ability of the cruise control system.

[0241] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the intelligent cruise control method, which includes: determining the current state signal of the automatic control vehicle; inputting the current state signal of the automatic control vehicle into the intelligent optimization control model to realize the intelligent cruise control of the automatic control vehicle; wherein the intelligent optimization control model is obtained by training the neural network parameters of the Markov decision process model based on the real-time state samples collected from the vehicle queue formed by the automatic control vehicle.

[0242] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0243] On the other hand, an embodiment of the present invention further provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the intelligent cruise control method provided by the above-mentioned methods, and the method includes: determining a current state signal of an automatically controlled vehicle; inputting the current state signal of the automatically controlled vehicle into an intelligent optimization control model to implement intelligent cruise control of the automatically controlled vehicle; wherein the intelligent optimization control model is obtained by training neural network parameters of a Markov decision process model based on real-time state samples collected from a vehicle queue formed by the automatically controlled vehicle.

[0244] On the other hand, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned intelligent cruise control methods, the method comprising: determining a current state signal of an automatically controlled vehicle; inputting the current state signal of the automatically controlled vehicle into an intelligent optimization control model to implement intelligent cruise control of the automatically controlled vehicle; wherein the intelligent optimization control model is obtained by training neural network parameters of a Markov decision process model based on real-time state samples collected from a vehicle queue formed by the automatically controlled vehicles.

[0245] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0246] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0247] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent cruise control method, characterized in that: include: determining a current state signal of an automatically controlled vehicle; Inputting the current state signal of the automatic control vehicle into the intelligent optimization control model to implement intelligent cruise control of the automatic control vehicle; Wherein, the intelligent optimization control model is obtained by training the neural network parameters of the Markov decision process model based on the real-time state samples collected from the vehicle platoon formed by the automatic control vehicle; The construction process of the Markov decision process model includes the following steps: Acquiring queue state information of a vehicle queue formed by automatically controlling vehicles, and establishing a dynamic equation of a queue system according to the queue state information; According to the dynamic equation of the queue system, a quadratic optimization control equation is constructed with minimization of state error and input as the objective function; Constructing a Markov decision process model of networked control according to the dynamic equation of the queue system and the quadratic optimization control equation; According to the dynamic equation of the queue system, the quadratic optimization control equation is constructed with minimizing the state error and input as the objective function as follows: Where N is the number of sampling intervals, y i and u i They represent the current state variables and acceleration control strategy respectively, and C and D are coefficient matrices: c1 and c2 are preset coefficients, and m is the total number of vehicles in the vehicle queue excluding the head vehicle; The intelligent optimization control model is obtained by training the neural network parameters of the Markov decision process model based on the real-time state samples of the vehicle queue formed by the automatic control vehicle, and includes: Constructing a deep deterministic policy gradient algorithm including a current actor network, a current critic network, a target actor network, and a target critic network to update the parameters of the Markov decision process model; In each time slot, according to the input state s k , the current actor network will output the corresponding action strategy μ(s k |θ μ ), execution strategy And according to the state transfer function, the next state s is obtained k+1 , and the corresponding reward r is obtained according to the reward function k , will (s k , a k ,s k+1 , r k ) is stored as a sample in the experience playback buffer to obtain a state sample; wherein, η represents random noise; The current critic network updates its parameters θ by minimizing the following mean square error loss function: Q : Among them, M is the number of samples in the mini-batch sampling, Q(s t ,a t |θ Q ) is the current Q value, by t with a t Input into the current critic network to get, x t is the target Q value, expressed as: x t =r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ) In the formula, r t is the corresponding reward function value, Q′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ) is the next Q value generated by the target critic network, μ′(s t+1 |θ μ′ ) is the target actor network according to the input state s t+1 The generated next action strategy, γ represents the discount factor; The current actor network updates its parameters θ through the following policy gradient function μ : in, is the gradient operator, Represents the μ Find the gradient, It means to find the gradient of a, and J represents the current actor network function; The target actor network and the target critic network update their parameters θ respectively in the following way Q' and θ μ' : i Q′ ←sth Q +(1-δ)θ Q′ i μ′ ←sth μ +(1-δ)θ μ′ Among them, δ is a fixed constant, 0<δ<<1.

2. The intelligent cruise control method according to claim 1, characterized in that: The step of obtaining queue state information of a vehicle queue formed by automatically controlling vehicles and establishing a dynamic equation of a queue system according to the queue state information comprises the following steps: Obtain the distance, speed and acceleration information of each vehicle in the vehicle queue through vehicle-to-vehicle communication; Establishing a dynamic equation for each vehicle in the vehicle queue according to the vehicle distance, vehicle speed and acceleration information of each vehicle in the vehicle queue; Obtaining the expected speed of the leading vehicle, obtaining the expected distance between each vehicle based on a preset range strategy, and establishing a state error equation for each vehicle according to the expected speed of the leading vehicle, the expected distance between each vehicle, and the current speed and distance between each vehicle; The state error equations of the vehicles are combined with the state equations of the vehicles in the queue in continuous time, and the dynamic equations of the queue system are obtained after discretization.

3. The intelligent cruise control method according to claim 2, characterized in that: The pre-set scope strategies include: If the current vehicle distance is less than the preset minimum vehicle distance, the expected vehicle speed is 0; If the current vehicle distance is not less than the preset minimum vehicle distance and not greater than the preset maximum vehicle distance, the expected vehicle speed is obtained according to the preset maximum vehicle speed, the current vehicle distance, the preset minimum vehicle distance and the preset maximum vehicle distance. The calculation formula is: Where V(h) represents the expected vehicle speed, h represents the vehicle distance, and h min Indicates the preset minimum vehicle distance, h max Indicates the preset maximum vehicle distance, v max Indicates the preset maximum vehicle speed; If the current vehicle distance is greater than the preset maximum vehicle distance, the expected vehicle speed is the preset maximum vehicle speed; The expected vehicle distance between each vehicle is obtained according to the expected vehicle speed.

4. The intelligent cruise control method according to claim 2, characterized in that: The dynamic equation of the queue system obtained after the discretization process is as follows: and i+1 =A0y i +B1u i +B2u i-1 ; Among them, y i =y(iΔT) and u i =u(iΔT) respectively represent the state variables and acceleration control strategy at the current moment, i is the sampling interval number, ΔT is the sampling interval, τ is the network induced delay, λ j and represents the system parameters related to human driving behavior, j is the vehicle number in the queue, m is the total number of vehicles in the queue except the head vehicle, is the partial derivative of the range strategy at the desired vehicle distance.

5. An intelligent cruise control device, characterized in that: It includes a status signal unit and an intelligent control unit; The state signal unit is used to determine the current state signal of the automatic control vehicle; The intelligent control unit is used to input the current state signal of the automatic control vehicle into the intelligent optimization control model to implement intelligent cruise control of the automatic control vehicle; Wherein, the intelligent optimization control model is obtained by training the Markov decision process model based on the real-time state samples collected from the vehicle platoon formed by the automatic control vehicle; The construction process of the Markov decision process model includes the following steps: Acquiring queue state information of a vehicle queue formed by automatically controlling vehicles, and establishing a dynamic equation of a queue system according to the queue state information; According to the dynamic equation of the queue system, a quadratic optimization control equation is constructed with minimization of state error and input as the objective function; Constructing a Markov decision process model of networked control according to the dynamic equation of the queue system and the quadratic optimization control equation; According to the dynamic equation of the queue system, the quadratic optimization control equation is constructed with minimizing the state error and input as the objective function as follows: Where N is the number of sampling intervals, y i and u i They represent the current state variables and acceleration control strategy respectively, and C and D are coefficient matrices: c1 and c2 are preset coefficients, and m is the total number of vehicles in the vehicle queue excluding the head vehicle; The intelligent optimization control model is obtained by training the neural network parameters of the Markov decision process model based on the real-time state samples of the vehicle queue formed by the automatic control vehicle, and includes: Constructing a deep deterministic policy gradient algorithm including a current actor network, a current critic network, a target actor network, and a target critic network to update the parameters of the Markov decision process model; In each time slot, according to the input state s k , the current actor network will output the corresponding action strategy μ(s k |θ μ ), execution strategy And according to the state transfer function, the next state s is obtained k+1 , and the corresponding reward r is obtained according to the reward function k , will (s k , a k ,s k+1 , r k ) is stored as a sample in the experience playback buffer to obtain a state sample; wherein, η represents random noise; The current critic network updates its parameters θ by minimizing the following mean square error loss function: Q : Among them, M is the number of samples in the mini-batch sampling, Q(s t ,a t |θ Q ) is the current Q value, by t with a t Input into the current critic network to get, x t is the target Q value, expressed as: x t =r t +γQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ) In the formula, r t is the corresponding reward function value, Q′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q′ ) is the next Q value generated by the target critic network, μ′(s t+1 |θ μ′ ) is the target actor network according to the input state s t+1 The generated next action strategy, γ represents the discount factor; The current actor network updates its parameters θ through the following policy gradient function μ : in, is the gradient operator, Represents the μ Find the gradient, It means to find the gradient of a, and J represents the current actor network function; The target actor network and the target critic network update their parameters θ respectively in the following way Q' and θ μ' : i Q′ ←sth Q +(1-δ)θ Q′ i μ′ ←sth μ +(1-δ)θ μ′ Among them, δ is a fixed constant, 0<δ<<1.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the intelligent cruise control method according to any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent cruise control method as claimed in any one of claims 1 to 4 are implemented.

Images