Intelligent network connection vehicle formation centralized control method based on deep reinforcement learning

Abstract

The invention relates to an intelligent network connection vehicle formation centralized control method based on deep reinforcement learning, and belongs to the technical field of intelligent traffic. A communication network architecture in a vehicle-road-cloud set is constructed, a vehicle-mounted computing task is unloaded to an edge cloud node, and global optimization control is realized in combination with a DDPG algorithm. The vehicle node obtains state information through a vehicle-mounted sensor and transmits the state information to the edge cloud node through the roadside node; and the edge cloud integrates multi-vehicle data collected by the RSU into formation global state information, generates a control instruction in real time by using a pre-trained DDPG intelligent agent, and broadcasts the control instruction to formation vehicles for execution through C-V2X communication. And the DDPG intelligent agent is guided to train through a reward function and is combined with a fixed time interval strategy and a second-order kinematics model to realize rapid convergence of a vehicle distance and a speed error. According to the invention, the load of vehicle-mounted hardware is reduced through a centralized computing architecture, and the real-time performance and safety of formation control are improved; complex algorithm global optimization is supported by using cloud computing power.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent transportation, and relates to a centralized control method for intelligent connected vehicle formation based on deep reinforcement learning. Background Art

[0002] In modern transportation systems, with the rapid growth of vehicle ownership and the continuous increase in traffic flow, traditional traffic management models have become difficult to cope with increasingly complex traffic problems. There is an urgent need for a more intelligent and efficient solution to optimize traffic operation. The formation control of intelligent connected vehicles (ICVs) emerged under this background and has gradually become an important research direction in the field of intelligent transportation. The formation control of intelligent connected vehicles organizes multiple vehicles in a tight formation and realizes the coordinated driving between vehicles by using advanced communication technologies, sensing technologies, and control algorithms. This technology can not only significantly improve road traffic efficiency but also effectively reduce fuel consumption, carbon emissions, and improve driving safety.

[0003] Although significant progress has been made in the technology of intelligent connected vehicle formation control, there are still many challenges for its large-scale application in the future. In the intelligent connected vehicle formation control system, the leader vehicle is responsible for setting the driving path and speed, while the follower vehicles dynamically adjust their own driving states by receiving the state information (such as position, speed, acceleration, etc.) of the leader vehicle and other neighboring vehicles in real time to maintain a stable formation and a safe distance. In currently commonly used distributed formation control algorithms, the in-vehicle domain control units (DCUs) bear the computational load required to implement formation control. However, with the increase in algorithm complexity and the diversification of application scenarios, the computational requirements of ICVs for DCUs are also gradually increasing, and the traditional distributed architecture cannot fundamentally solve the problem of limited computing power.

[0004] In recent years, the development of artificial intelligence technology has injected new impetus into the development of intelligent vehicle control technology. Deep reinforcement learning (DRL), as an artificial intelligence algorithm, uses a reward function to guide an agent to continuously interact with the environment and trains the agent to autonomously explore and learn control strategies by maximizing the obtained reward feedback. The addition of deep reinforcement learning has brought significant improvements to the intelligent connected vehicle formation control. The DRL agent autonomously learns the optimal strategy, enabling the vehicle to adapt to complex dynamic environments, optimize multi-objective performance (such as vehicle safety, fuel efficiency, comfort, etc.), and improve the robustness and communication efficiency of the formation system.

[0005] Existing intelligent connected vehicle formation control methods based on DRL mainly rely on a distributed communication network architecture. Under this architecture, ICVs obtain and send information through Vehicle-to-Vehicle (V2V) distributed communication, and rely on their own DCU computing power to process and utilize external information to generate optimal control inputs in a timely manner. However, under the distributed communication network architecture, distributed communication topologies such as predecessor-follower (PF) and Predecessor-Leader Follower (PLF) among ICVs cannot efficiently obtain formation global state information, which may cause the DRL model to ignore some individual cases during the training process and thus cannot guarantee the global optimality of the system. In addition, for DRL intelligent formation control based on the distributed communication network architecture, the computing power requirement for the DCU is relatively high. When the vehicle is in a complex environment or the number of ICVs in the formation is large, there may be security problems such as slow system response due to limited DCU computing power.

[0006] To address these problems, the present invention proposes a centralized control method for intelligent connected vehicle formation based on deep reinforcement learning. This method includes designing a centralized communication architecture for formation control, considering offloading the computational burden from in-vehicle DCUs to edge cloud devices. The Mobile Edge Computing (MEC) unit processes the driving data collected by Roadside Units (RSUs) and trains a control strategy based on DRL. During the formation control operation, the ICV shares its state information with the edge cloud device through the RSU. Then, the edge cloud device responds with specific control inputs for each ICV.

[0007] The present invention demonstrates significant technical advantages and application potential in intelligent connected vehicle formation control, provides an efficient and flexible computing architecture for intelligent connected vehicle formation control, helps improve the global optimization ability and real-time response performance of formation control, and promotes the development of intelligent connected vehicle technology to a higher level. Summary of the Invention

[0008] In view of the above, the purpose of the present invention is to provide a centralized control method for intelligent connected vehicle formation based on deep reinforcement learning. By offloading the computational burden from in-vehicle DCUs to edge cloud devices and establishing a control strategy based on Deep Deterministic Policy Gradient (DDPG) under a centralized communication architecture, the flexibility and security of intelligent connected vehicle formation control are improved, while the hardware cost and energy consumption are reduced.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] A centralized control method for intelligent connected vehicle formation based on deep reinforcement learning. The overall method includes the design of a centralized communication network architecture, the training, testing, and application of deep reinforcement learning algorithms, and can be carried out in the following five steps:

[0011] S1-1: Construct a vehicle-road-cloud centralized communication network architecture, which includes vehicle nodes, roadside nodes, and edge cloud nodes;

[0012] S1-2: Deploy and train a deep reinforcement learning agent based on the Deep Deterministic Policy Gradient (DDPG) algorithm in the edge cloud node. The DDPG agent is a type of actor-critic deep reinforcement learning network framework. It includes 1 online Actor network, 1 target Actor network, 1 online Critic network, and 1 target Critic network;

[0013] S1-3: Under the deep reinforcement learning application framework, the vehicle formation serves as the interaction environment of the reinforcement learning agent. Through C-V2X (Cellular Vehicle-to-Everything), it broadcasts its own state information centrally to the roadside communication unit. The roadside communication unit collects the state information of intelligent connected vehicle formations within its range and transmits all state information to the edge cloud node, which is further integrated into the formation global state information;

[0014] S1-4: Based on the formation global state information s t , the edge cloud node uses the DDPG agent to generate action control a t , and transmits it to each intelligent connected vehicle (ICV) in the vehicle formation through the C-V2X broadcast communication of the roadside node;

[0015] S1-5: The ICVs in the formation execute the centralized control instruction u(t) = a(t) and perform corresponding actions to achieve the cooperative control of the formation.

[0016] In view of the requirements of intelligent connected vehicles for vehicle-road-cloud centralized communication in a centralized control method for intelligent connected vehicle formation based on deep reinforcement learning, a centralized communication network architecture including vehicles, roads, and clouds is studied and designed. This architecture provides a clearer information flow direction and more reliable communication guarantee for the centralized control of intelligent connected vehicle formations from two aspects: node composition and communication process. The design of the vehicle-road-cloud centralized communication network architecture includes the following steps:

[0017] S2-1: The vehicle-road-cloud nodes in the architecture include:

[0018] (1) Vehicle node - ICV: Obtains its own status information through on-vehicle sensors and transmits the status information to the roadside node through the Onboard Unit (OBU).

[0019] (2) Roadside node - RSU: Regularly collects the status information of all ICVs within its communication range and transmits the status information to the edge cloud node.

[0020] (3) Computing unit - MEC: Receives and integrates the status information from multiple roadside nodes, generates control inputs based on the DDPG agent, and feeds back the control inputs to the roadside node.

[0021] S2-2: Design the vehicle-road-cloud communication process for intelligent connected vehicle formation control.

[0022] (1) Based on the vehicle-road-cloud centralized communication topology, the ICV within the communication coverage of the RSU transmits the vehicle status information s at time t through the direct communication interface (5 th interface of ProSe Communication, PC5) of C-V2X to the RSU; i (t) broadcasts to the RSU;

[0023] (2) The RSU transmits s i (t) to the MEC through optical fiber and further integrates it into the formation status information vector s t = [s1(t), s2(t),... s i (t)] T ;

[0024] (3) The DDPG centralized controller deployed in the MEC gives a reasonable action decision output a t according to the formation status information s input at this moment t = [a1(t), a2(t),... a n (t)] T , and broadcasts it to the ICV formation through the PC5 interface of the RSU. Among them, the i-th ICV executes the centralized control instruction u i (t) = a i (t).

[0025] In the deep reinforcement learning application framework, it mainly includes two parts: the agent and the interaction environment. Among them, the construction of the interaction environment under the deep reinforcement learning framework, that is, the modeling of intelligent connected vehicle formation. The specific steps are as follows:

[0026] S3-1: The motion model of the ICV adopts a second-order kinematic model, and the specific expression is:

[0027]

[0028] where x i (t), v i (t), u i (t) are the position, speed, and control input of the i-th vehicle, respectively.

[0029] S3-2: The spacing strategy between ICVs in the intelligent connected vehicle formation adopts the Constant Time Headway (CTH) strategy, and the specific expression is:

[0030] L i (t) = l c + l s + T c v i (t)

[0031] where L i (t) is the spacing between the front bumper of the i-th vehicle and the rear bumper of the (i - 1)-th vehicle in the vehicle queue at time t, which includes l c being the vehicle length, l s being the minimum static spacing between vehicles, T c being the fixed time constant, and v i (t) being the speed of the i-th vehicle at this moment.

[0032] S3-3: The control objectives of the intelligent connected vehicle formation based on S3-1 include: ① The spacing error of the vehicle formation gradually approaches zero; ② The speed error of the vehicle formation gradually approaches zero. That is:

[0033]

[0034] where is the spacing error between adjacent vehicles, is the speed error between adjacent vehicles.

[0035] Furthermore, the training of the agent in the deep reinforcement learning application framework is studied. In the vehicle-road-cloud centralized communication network architecture, the deep reinforcement learning algorithm mainly relies on the powerful computing power of the edge cloud computing unit. The DDPG agent deployed on the MEC needs to be trained offline before online application, and its training process includes:

[0036] S4-1: Initialize the hyperparameters φ θ and φ π of the online Actor network and the online Critic network, as well as the hyperparameters φ θ′ and φ π′ of the target Actor network and the target Critic network;

[0037] S4-2: Formation state information s from the vehicle platoon environment t , where the state information of the i-th ICV includes the position, speed, spacing error, speed error of the ICV, and the control input at the previous moment, i.e., at time t - Δt, that is:

[0038]

[0039] where Δt is the system sampling period and also the simulation step size of the system.

[0040] S4-3: Input the state information at time t into the online Actor network and introduce exploration noise ν t , to improve the system generalization ability. After passing through the online Actor network, generate the action a corresponding to time t t = π(s t ) + ν t . Further execute a in the formation environment t , to obtain the expected state information s at time t + Δt t+Δt and the execution reward value r at time t feedback from the formation environment t ;

[0041] S4-4: Take the series of data obtained in S4-3 as a set of experience data (s t , a t , s t+Δt , r t ) and store it in the experience replay buffer M. During each subsequent round of training, continuously randomly extract a small portion of data with a batch size of N from the buffer for training;

[0042] S4-5: During the training process, the online Critic network evaluates the experience data (s t , a t ) and gives the evaluated value Q at time t = θ(s t , a t ); The target Critic network then uses the experience data (s t , r t ) and the action output s of the target Actor network t+Δt , to obtain the target evaluated value ψ(r i , θ′(s t+Δt )) at time t + Δt. Finally, update the parameters of the online Actor network and the online Critic network by minimizing the loss function Loss and reducing the policy gradient , and update the parameters of the target Actor network and the target Critic network by the soft update policy. The loss function and the policy gradient function are as follows:

[0043]

[0044] S4-6: Repeat S4-2 to S4-5 until the training is completed to obtain a DDPG agent that can be used for online applications.

[0045] Regarding the offline training of the DDPG agent mentioned in Part 4, the DDPG agent for longitudinal centralized control of intelligent connected vehicle formations, the reward value r of its training episode at time t t includes: a spacing error reward term R x , which is used to guide the vehicle formation to maintain the desired spacing; a speed error reward term R v , which is used to guide the vehicle formation to maintain speed consistency; a too-close spacing penalty term P x , which is used to penalize the situation where the vehicle spacing is too close; an acceleration fluctuation penalty term P u , which is used to penalize frequent acceleration changes and improve driving comfort; a collision penalty term P done , which is used to penalize potential collision situations. In the training episode corresponding to time t, the total reward function R obtained by the agent total is the weighted sum of each reward term and penalty term, and the specific expression is:

[0046] r t = R total = ω1R x + ω2R v + ω3P x + ω4P u + ω5P done

[0047] where ω1 to ω5 are weight parameters, and the weights corresponding to the reward terms are positive numbers; the weights corresponding to the penalty terms are set as negative numbers, and are adjusted according to the importance of each reward term and penalty term, so as to positively guide the DDPG agent to make control decisions beneficial to vehicle formations.

[0048] The beneficial effects of the present invention are as follows: The present invention can not only reduce the computing burden of the in-vehicle DCU, but also establish a DDPG-based control strategy under a centralized communication architecture, effectively improving the uniformity of the fleet state. The cloud provides powerful computing capabilities, supports complex algorithms and global optimization, and improves the formation passing efficiency, safety and comfort. Through the cloud deployment of the deep reinforcement learning algorithm and the centralized communication of roadside RSUs, larger-scale formation cooperative control and more complex driving service applications can be achieved. At the same time, by using edge cloud to offload computing tasks, the in-vehicle hardware cost and energy consumption are reduced, and the hardware life is extended. In addition, the application of the vehicle-road-cloud centralized communication network architecture is also beneficial to the operation and maintenance of the central node device and the update and expansion of the algorithm.

[0049] Other advantages, objects, and features of the present invention will be set forth in part in the following description, and in part will be obvious to those skilled in the art upon examination of the following, or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to make the objects, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail preferably with reference to the accompanying drawings, wherein:

[0051] Figure 1 is a specific flowchart of the method implementation of the present invention;

[0052] Figure 2 is a centralized control framework for intelligent connected vehicle formation based on deep reinforcement learning in Embodiment 1;

[0053] Figure 3 is the data flow of each node under the vehicle-road-cloud centralized communication network architecture in Embodiment 1

[0054] Figure 4 is the vehicle formation control reward value curve of DDPG in Embodiment 2;

[0055] Figure 5 is the longitudinal position curve of a six-vehicle formation in Embodiment 3;

[0056] Figure 6 is the longitudinal position error curve of a six-vehicle formation in Embodiment 3;

[0057] Figure 7 is the longitudinal speed curve of a six-vehicle formation in Embodiment 3;

[0058] Figure 8 is the longitudinal speed error curve of a six-vehicle formation in Embodiment 3;

[0059] Figure 9 is the longitudinal acceleration curve of a six-vehicle formation in Embodiment 3. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] The following specific examples illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present invention schematically, and the following embodiments and the features in the embodiments can be combined with each other without conflict.

[0061] Among them, the attached drawings are only for illustrative purposes, showing only schematic diagrams rather than physical diagrams, and should not be construed as a limitation on the present invention; in order to better illustrate the embodiments of the present invention, some components in the attached drawings will be omitted, enlarged or reduced, which does not represent the size of the actual product; for those skilled in the art, it is understandable that some well-known structures and their descriptions in the attached drawings may be omitted.

[0062] In the attached drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "rear", etc. indicating the orientation or positional relationship, they are based on the orientation or positional relationship shown in the attached drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, the terms describing the positional relationship in the attached drawings are only for illustrative purposes and should not be construed as a limitation on the present invention. For those of ordinary skill in the art, the specific meanings of the above terms can be understood according to specific circumstances.

[0063] Embodiment 1

[0064] This embodiment provides a centralized control method for intelligent connected vehicle formation based on deep reinforcement learning. As Figure 1 shown, the specific process of this embodiment includes the following steps:

[0065] S1-1: Construct a vehicle-road-cloud centralized communication network architecture, which includes vehicle nodes, roadside nodes, and edge cloud nodes;

[0066] S1-2: Deploy and train a deep reinforcement learning agent based on the DDPG algorithm in the edge cloud node. The DDPG agent is one of the actor-critic deep reinforcement learning network frameworks. As Figure 2 shown, it includes 1 online Actor network, 1 target Actor network, 1 online Critic network, and 1 target Critic network;

[0067] S1-3: Under the deep reinforcement learning application framework, the vehicle formation serves as the interaction environment of the reinforcement learning agent. The vehicle formation broadcasts its own state information to the roadside communication unit through C-V2X in a centralized manner. The roadside communication unit collects the state information of the intelligent connected vehicle formation within its range and transmits all the state information to the edge cloud node, and further integrates it into the formation global state information;

[0068] S1-4: The edge cloud node generates an action control a t using the DDPG agent based on the formation global state information s t, and transmitted to each ICV in the vehicle formation through the C-V2X broadcast communication of the roadside node;

[0069] S1-5: The ICVs in the formation execute the centralized control instruction u(t) = a(t) and perform corresponding actions to achieve the cooperative control of the formation.

[0070] Furthermore, a centralized control method for intelligent connected vehicle formations based on deep reinforcement learning provided in this example studies and designs a centralized communication network architecture including vehicles, roads, and clouds to meet the needs of intelligent connected vehicles for centralized communication, as Figure 2 , Figure 3 shown.

[0071] This architecture provides a clearer information flow and more reliable communication guarantee for the centralized control of intelligent connected vehicle formations from two aspects: node composition and communication process. The design of the vehicle-road-cloud centralized communication network architecture includes the following steps:

[0072] S2-1: The vehicle-road-cloud nodes in the architecture, as Figure 2 shown, include:

[0073] (1) Vehicle node - ICV: Obtains its own status information through on-vehicle sensors and transmits the status information to the roadside node through the on-vehicle communication unit OBU;

[0074] (2) Roadside node - RSU: Regularly collects the status information of all ICVs within its communication range and transmits the status information to the edge cloud node;

[0075] (3) Computing unit - Multi-Access Edge Computing (MEC): Receives and integrates the status information from multiple roadside nodes, generates control inputs based on the DDPG agent, and feeds back the control inputs to the roadside node.

[0076] S2-2: As Figure 3 shown, based on the vehicle-road-cloud centralized communication network architecture, a vehicle-road-cloud communication process for intelligent connected vehicle formation control is designed.

[0077] (1) Based on the vehicle-road-cloud centralized communication topology, the centralized central node RSU obtains the vehicle status information s i (t) of the ICVs within the communication coverage through the PC5 direct connection communication interface of C-V2X, that is, the basic vehicle safety information;

[0078] (2) The RSU transmits s i (t) to the MEC through optical fiber and further integrates it into the formation status information vector s t= [s1(t), s2(t),... s i (t)] T ;

[0079] (3) The DDPG centralized controller deployed in the MEC gives a reasonable action decision output a t according to the formation state information s t = [a1(t), a2(t),... a n (t)] T at this moment, and broadcasts it to the ICVs formation through the PC5 interface of the RSU. Among them, the i-th ICV executes the centralized control instruction u i (t) = a i (t).

[0080] Further, for a centralized control method of intelligent connected vehicle formation based on deep reinforcement learning provided in this example, the construction of the interaction environment under the deep reinforcement learning framework involved, that is, the modeling of the intelligent connected vehicle formation, as Figure 2 shown, can be specifically implemented in the following steps:

[0081] S3-1: For a vehicle formation composed of n ICVs, the motion model of the ICV adopts a second-order kinematic model, and the specific expression is:

[0082]

[0083] where x i (t), v i (t), u i (t) are the position, speed and control input of the i-th vehicle respectively.

[0084] S3-2: The spacing strategy between ICVs in the intelligent connected vehicle formation adopts the CTH fixed-time interval strategy, and the specific expression is:

[0085] L i (t) = l c + l s + T c v i (t)

[0086] where l c is the vehicle length, l s is the minimum static spacing between vehicles, and T c is a fixed time constant.

[0087] S3-3: The control objectives of the intelligent connected vehicle formation based on S3-1 include: ① The spacing error of the vehicle formation gradually approaches zero; ② The speed error of the vehicle formation gradually approaches zero. That is:

[0088]

[0089] wherein is the spacing error between adjacent vehicles, is the speed error between adjacent vehicles.

[0090] Embodiment 2

[0091] Furthermore, the training of the agent in the deep reinforcement learning application framework is studied. In the vehicle-road-cloud centralized communication network architecture, the deep reinforcement learning algorithm mainly relies on the powerful computing power of the edge cloud computing unit. The DDPG agent deployed in the MEC needs to be trained offline before online application, and its training process includes the following steps:

[0092] S4-1: The network structure of the DDPG agent applied in this example is as follows:

[0093]

[0094] Among them, the Actor network includes an online Actor network and a target Actor network; the Critic network includes an online Critic network and a target Critic network.

[0095] In the early stage of training, first initialize the hyperparameters φ θ and φ π of the online Actor network and the online Critic network, θ′ and φ π′ of the target Actor network and the target Critic network.

[0096] Among them, the structural parameters of the Actor network in the DDPG agent are shown in Table 1.

[0097] Table 1 Structural parameters of the Actor network in the DDPG agent

[0098] Parameter type Input layer Fully connected layer Output layer Number of network layers 1 2 1 Number of neurons Number of following vehicles × Number of state information elements 256 Number of following vehicles Activation function 1 ReLU Tanh

[0099] The structural parameters of the Critic network in the DDPG agent are shown in Table 2.

[0100] Table 2 Structural parameters of the Critic network in the DDPG agent

[0101]

[0102] S4-2: The formation state information s t from the vehicle formation environment, where the state information of the i-th ICV includes the position, speed, spacing error, speed error of the ICV, and the control input at the previous moment, i.e., the t-Δt moment, that is:

[0103]

[0104] Where Δt is the system sampling period and also the simulation step size of the system.

[0105] S4-3: Input the state information at time t into the online Actor network and introduce exploration noise ν t , improve the system generalization ability, and through the online Actor network, generate the action a corresponding to time t t = π(s t ) + ν t . Further execute a in the formation environment t , obtain the expected state information s at time t+Δt t+Δt and the execution reward value r at time t feedback from the formation environment t ;

[0106] S4-4: Take the series of data obtained in S4-3 as a set of experience data (s t , a t , s t+Δt , r t ) and store it in the experience replay buffer M. During each subsequent round of training, a small portion of data with a batch size of N will be randomly drawn from the buffer for training;

[0107] S4-5: During the training process, the online Critic network evaluates the experience data (s t , a t ) and gives the evaluated value Q at time t = θ(s t , a t ); The target Critic network then uses the experience data (s t , r t ) and the action output s of the target Actor network t+Δt , and obtains the target evaluated value ψ(r i , θ′(s t+Δt )) at time t+Δt. Further, by minimizing the loss function Loss and reducing the policy gradient to update the parameters of the online Actor network and the online Critic network, the loss function and the policy gradient function are as follows:

[0108]

[0109] Finally, update the parameters of the target Actor network and the target Critic network through the following soft update strategy.

[0110]

[0111] where τ is the soft update factor.

[0112] S4-6: Repeat S4-2 to S4-5 until the training is completed to obtain a DDPG agent that can be used for online applications.

[0113] Regarding the offline training of the DDPG agent mentioned in Part 4, the reward value r at time t during the training process of the DDPG agent t includes: a spacing error reward term R x , which is used to guide the vehicle formation to maintain the desired spacing; a speed error reward term R v , which is used to guide the vehicle formation to maintain speed consistency; a penalty term P for too close spacing x , which is used to penalize the situation where the vehicle spacing is too close; a penalty term P for acceleration fluctuations u , which is used to penalize frequent acceleration changes and improve driving comfort; a penalty term P for collisions done , which is used to penalize potential collision situations. The specific reward value design is shown in Table 3.

[0114] Table 3 Composition of the reward function

[0115]

[0116]

[0117] In the training episode corresponding to time t, the total reward function R obtained by the agent total is the weighted sum of each reward term and penalty term, and the specific expression is:

[0118] r t = R total = ω1R x + ω2R v + ω3P x + ω4P u + ω5P done

[0119] where ω1 to ω5 are weight parameters, which are adjusted according to the importance of each reward term and penalty term.

[0120] In this embodiment, in order to verify the effectiveness of the method proposed in Embodiment 1, a simulation environment is first established. It is assumed that a fleet of 6 ICVs on a straight lane follows the control strategy. Initially, the initial positions of each ICV in the formation are set to x(0) = {137.5, 111, 84, 56.5, 28.5, 0}, and the corresponding initial speeds are set to v(0) = {15, 15, 15, 15, 15, 15}. In addition, the parameters related to the CTH spacing strategy are set to L = 5m, lc = 5m, T c= 1.1 s. When training the DDPG agent, the hyperparameters of the DDPG agent training process are shown in Table 4.

[0121] Table 4 Hyperparameters for DDPG Agent Training

[0122]

[0123]

[0124] During the simulation process, the sampling interval is set to T s = 0.1 s, and the simulation duration is T f = 200 s, and the entire training consists of 2000 episodes.

[0125] To enhance the generalization ability of the DRL model, during the training process, a sine function is introduced into the speed trajectory of the leading vehicle. Therefore, the trajectory curve of the leading vehicle for training is as follows:

[0126]

[0127] where the amplitude A = max(2 + 0.1·randn, 0.1), and the frequency ω = max(1 + 0.1·randn, 0.1).

[0128] Figure 4 Shows the rewards of the DDPG training environment, namely the rewards for each episode and the average reward curve for 2000 episodes. In the figure, before the 250th episode, the agent is in the exploration stage of controlling the longitudinal driving of the formation. When a collision occurs within the formation, it will cause the interruption of this training episode. This results in the average reward of about 200 episodes in the early stage of training being around -1000. After about 220 episodes, under the positive guidance of the reward function, the agent avoids the huge penalty value caused by vehicle collisions and controls the vehicle formation to approach the control target (the spacing error and speed error of the vehicle formation gradually tend to zero). As the training cycle increases, the reward value shows an increasing trend. Around the 700th episode, the reward value for a single episode gradually converges to a bounded interval, and the average reward also starts to converge around the 600th episode and tends to a stable value after the 700th episode. These results indicate that the proposed DDPG agent will find a stable and effective control strategy.

[0129] Example 3

[0130] Based on the powerful computing power of MEC, a DDPG agent for centralized control of intelligent connected vehicle formations is trained. To verify the effectiveness of the agent, this embodiment provides a simulation test for centralized control of vehicle formations based on deep reinforcement learning under a vehicle-road-cloud centralized communication architecture. Under the centralized communication network architecture, the DDPG agent trained in MEC is used to integrate and process the state information collected from the RSU. The control input, i.e., the action u i (t) of the agent, is published to all following vehicles via the V2I link.

[0131] During the test, the leading vehicle will run according to the following speed function:

[0132]

[0133] The remaining ICVs will start to follow the leading vehicle according to the trained DDPG-based control strategy.

[0134] Figures 5 to 9 It shows that the DDPG-based control strategy with a centralized communication architecture can control the vehicle fleet to complete the control objective safely and comfortably.

[0135] Figure 5 、 Figure 6 respectively show the position of the vehicle formation and the spacing error between adjacent vehicles, i.e., x i (t) and In Figure 5 , the following vehicles closely follow the leading vehicle during the test period and no collision occurs, which indicates that this method can provide effective formation control; in Figure 6 , the spacing error between adjacent vehicles converges to 0 within t ∈ [0s, 2s) and then remains fluctuating within a small range near 0 thereafter. During the acceleration phase of t ∈ [50s, 80s), this method can keep the maximum absolute value of the spacing error within 0.06 meters. The results show that the DDPG-based control strategy performs well in maintaining safety and consistency.

[0136] Figure 7 、 Figure 8 respectively show the curves of the speed and the speed error, i.e., v i (t) and of the change. In Figure 7 , the speed of the following vehicles remains consistent with the speed of the leading vehicle throughout the process. In Figure 8 , the proposed method can make the speed error converge to 0 quickly and smoothly within t ∈ [0s, 10s), and the speed of the ICV vehicle is in Figure 7It is compact in performance, and the maximum absolute value of the speed error shows a decreasing trend as the number of vehicles increases, especially in the acceleration phase within t ∈ [65s, 80s). This indicates that the present invention performs excellently in controlling the formation speed consistency. Therefore, the centralized controller based on the DDPG agent helps to improve the existing controller in improving traffic flow and fuel efficiency.

[0137] Figure 9 Shows the acceleration of all ICVs in the formation, that is, the change curve of u i (t). It can be seen from the figure that most of the accelerations can be maintained within a low range by this method. And the acceleration of this method converges smoothly within t ∈ [0s, 10s). The accelerations of all following vehicles based on the DDPG controller show a decreasing trend in sequence as the number of vehicles increases, and show less jitter during the control process. These results indicate that the control method based on DDPG can effectively improve the control ability and comfort.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the purpose and scope of the present technical solution, and they should all be covered within the scope of the claims of the present invention.

Claims

1. A centralized control method for intelligent connected vehicle formation based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Building a vehicle-road-cloud centralized communication network architecture, the architecture including vehicle nodes, roadside nodes and edge cloud nodes; S2: deploying and training an agent based on a deep deterministic policy gradient (DDPG) algorithm in the edge cloud node, wherein the agent includes an online Actor network, a target Actor network, an online Critic network, and a target Critic network; S3: Collecting global status information of the intelligent connected vehicle fleet through the vehicle-road-cloud centralized communication network architecture, and transmitting the global status information to the edge cloud node; S4: using the DDPG agent to generate a centralized control instruction based on the global state information, and broadcasting the control instruction to each intelligent connected vehicle in the formation through a roadside node; S5: The intelligent connected vehicles execute the centralized control instructions to achieve coordinated control of the formation.

2. The centralized control method for intelligent connected vehicle formation based on deep reinforcement learning according to claim 1 is characterized in that: In S1, the communication process of the vehicle-road-cloud centralized communication network architecture includes: (1) The vehicle node sends its status information to the roadside node through the on-board communication unit OBU; (2) The roadside node transmits the status information to the edge cloud node via optical fiber; (3) The edge cloud node integrates status information from multiple roadside nodes to generate a global status information vector of the formation.

3. The centralized control method for intelligent connected vehicle formation based on deep reinforcement learning according to claim 1 is characterized in that: In S2, the training process of the DDPG agent includes: (1) Initialize the hyperparameters of the online Actor network, the online Critic network, and the target network; (2) Obtaining status information through the vehicle formation environment and introducing exploration noise generation action instructions; (3) Based on the experience replay buffer, data is extracted to complete the offline training of the agent by minimizing the loss function of the Critic network and updating the policy gradient of the Actor network.

4. The centralized control method for intelligent connected vehicle formation based on deep reinforcement learning according to claim 3 is characterized by: During the training process, the reward function includes: a weighted sum of a spacing error reward item, a speed error reward item, a too-close spacing penalty item, an acceleration fluctuation penalty item, and a collision penalty item.

5. The centralized control method for intelligent connected vehicle formation based on deep reinforcement learning according to claim 1 is characterized in that: In S3, the global state information includes the position, speed, spacing error, speed error and control input of each intelligent connected vehicle at the previous moment.

6. The centralized control method for intelligent connected vehicle formation based on deep reinforcement learning according to claim 1, characterized in that: In S4, the centralized control instruction is broadcast to the intelligent connected vehicles in the formation through the PC5 direct communication interface of C-V2X.

7. The centralized control method for intelligent connected vehicle formation based on deep reinforcement learning according to claim 1 is characterized in that: The spacing strategy of the intelligent connected vehicle formation adopts a fixed time interval strategy CTH, and the expression of the strategy is: L i (t)=l c +l s +T c v i (t) Where L i (t) is the distance between the front bumper of the i-th vehicle and the rear bumper of the i-1-th vehicle in the vehicle queue at time t, l c is the vehicle length, l s is the minimum static distance between vehicles, T c is a fixed time constant, v i (t) is the speed of the i-th vehicle at this moment.

8. The centralized control method for intelligent connected vehicle formation based on deep reinforcement learning according to claim 1, characterized in that: For a vehicle formation consisting of n intelligent connected vehicles, the motion model of the vehicle node is a second-order kinematic model, and its expression is: where x i (t), v i (t),u i (t) are the position, velocity and control input of the i-th vehicle respectively.

9. The centralized control method for intelligent connected vehicle formation based on deep reinforcement learning according to claim 1, characterized in that: The edge cloud node is deployed in a multi-access edge computing MEC unit, and the MEC unit is connected to the roadside node via optical fiber.

10. The centralized control method for intelligent connected vehicle formation based on deep reinforcement learning according to claim 1, characterized in that: The Actor network and Critic network structures of the DDPG agent respectively include an input layer, a fully connected layer and an output layer, wherein the output layer activation function of the Actor network is Tanh, and the output layer activation function of the Critic network is ReLU.

Images