Automatic driving freight formation collaborative decision-making method, device and equipment and storage medium

Abstract

The invention provides an automatic driving freight formation collaborative decision-making method, device and equipment and a storage medium. Specifically, the invention provides a customized partially observable Markov decision-making process (POMDP) model, and the model aims at freight formation collaborative decision-making in an expressway scene, and enhances perception information of freight formation in a training stage by using a formation control method of a staggered structure. And the effect of the agent decision network in the test stage is further improved. Furthermore, a multi-agent reinforcement learning framework based on a centralized training and decentralized execution (CTDE) training normal form is constructed in combination with a QMIX algorithm, and the safety, the passing efficiency and the energy consumption target of the freight formation are optimized, so that the freight formation is promoted to generate spontaneous cooperative behaviors. Compared with a traditional freight formation control method, the provided model and framework show excellent performance in the aspect of improving freight formation collaborative decision-making.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a method, device, equipment and storage medium for collaborative decision-making of an autonomous driving freight formation. Background Art

[0002] In today's global economic system, freight transportation plays an indispensable role, among which road freight transportation has become a key logistics channel due to its high proportion. With the improvement of urbanization and the continuous expansion of the economy, the problems faced by the road freight system have become increasingly prominent, such as long driving time, large freight volume, low efficiency, etc. These problems not only increase the risk of accidents, energy consumption and emissions, but also increase transportation costs. Therefore, autonomous driving freight platoons, with their environmental perception, intelligent decision-making and collaborative control capabilities, have become a new trend in the development of road freight.

[0003] An autonomous freight platoon usually consists of two or more trucks, which are connected through V2V communication technology and use a "leader-follower" mode to travel in platoon. However, traditional freight platoon control methods show low flexibility in high-traffic scenarios such as highways, and place extremely high demands on the control strategies of vehicles and platoons, making it difficult to formulate effective control strategies. Although the latest technology has proposed a freight platoon control method based on multi-agent reinforcement learning (MARL), the decision of each agent is not only affected by its own state and observation, but also by the behavior of other agents. This highly dynamic and interdependent characteristic complicates model training and has the disadvantages of low driving safety, high traffic efficiency and overall energy consumption. Summary of the invention

[0004] The present invention provides a method, device, equipment and storage medium for collaborative decision-making of an autonomous driving freight formation, which can realize the collaborative decision-making control function of an autonomous driving freight formation in a highway scenario, and has the advantages of low driving safety, high traffic efficiency and overall energy consumption.

[0005] In a first aspect, the present invention provides a collaborative decision-making method for an autonomous driving freight convoy, comprising:

[0006] Construct a highway multi-lane scenario model based on the highway-env simulation environment, wherein the highway multi-lane scenario model is configured with highway road information and vehicle information of multiple intelligent freight vehicles, wherein the vehicle information includes speed and acceleration, and the intelligent freight vehicles have the ability to perceive the surrounding environment;

[0007] Designing a formation control method for a staggered structure, wherein the formation control method is based on a collaborative decision-making model of a freight formation, wherein the freight formation is composed of the plurality of intelligent freight vehicles;

[0008] According to the highway multi-lane scenario model and the platoon control method, a customized partially observable Markov decision process is established, wherein the elements of the Markov decision process include the local state space, the global state space, the joint action space and the team reward function of the intelligent freight vehicle;

[0009] Designing a multi-agent reinforcement learning training framework through the QMIX algorithm according to the customized partially observable Markov decision process;

[0010] Based on the multi-agent reinforcement learning training framework, the collaborative decision-making model of the freight formation is optimized and trained with respect to safety, traffic efficiency and energy consumption goals;

[0011] As an optional implementation, the multi-agent reinforcement learning training framework is based on the CTDE training paradigm.

[0012] As an optional implementation, the local state space is composed of a basic state space and a formation control state space;

[0013] The global state space is formed by stacking local state spaces.

[0014] As an optional implementation, before designing a multi-agent reinforcement learning training framework through a QMIX algorithm based on the customized partially observable Markov decision process, the method further includes:

[0015] The formation control method of the staggered structure is applied to the formation control state space in the local state space and the team reward function.

[0016] As an optional implementation, the optimizing training of the collaborative decision-making model of the freight formation includes:

[0017] Inputting the local state space of each of the intelligent freight vehicles at the current moment and the joint action space at the previous moment into the decision network of the QMIX algorithm, and inputting the global state space into the hybrid network of the QMIX algorithm;

[0018] The parameters of the hybrid network and the parameters of the decision network are updated by minimizing the predicted joint Q value and the target joint Q value in the QMIX algorithm.

[0019] As an optional implementation, the optimization training of the collaborative decision-making model of the freight formation with respect to safety, traffic efficiency and energy consumption targets includes:

[0020] The collaborative decision-making model of the freight formation is optimized and trained according to the team reward function, wherein the team reward function includes a collision penalty function, a speed reward function, a dangerous action penalty function, an energy consumption penalty function, an unnecessary lane-changing behavior penalty function and a formation control reward function.

[0021] As an optional implementation, the plurality of intelligent freight vehicles in the highway multi-lane scenario model are controlled by a preset hierarchical framework, and the hierarchical framework includes an upper decision layer and a lower control layer;

[0022] The decision layer is used to provide the intelligent freight vehicles with a formation control method of the staggered structure;

[0023] The control layer is used to control the speed and acceleration of the intelligent freight vehicle and calculate the instantaneous energy consumption of the intelligent freight vehicle.

[0024] In a second aspect, the present invention further provides an autonomous driving freight formation collaborative decision-making device, comprising:

[0025] The first module is used to construct a highway multi-lane scenario model based on the highway-env simulation environment, wherein the highway multi-lane scenario model is configured with highway road information and vehicle information of multiple intelligent freight vehicles, wherein the vehicle information includes speed and acceleration, and the intelligent freight vehicles have the ability to perceive the surrounding environment;

[0026] The second module is used to design a formation control method of a staggered structure, wherein the formation control method is based on a collaborative decision-making model of a freight formation, and the freight formation is composed of the multiple intelligent freight vehicles;

[0027] A third module is used to establish a customized partially observable Markov decision process according to the highway multi-lane scenario model and the formation control method, wherein the elements of the Markov decision process include the local state space, the global state space, the joint action space and the team reward function of the intelligent freight vehicle;

[0028] The fourth module is used to design a multi-agent reinforcement learning training framework through the QMIX algorithm according to the customized partially observable Markov decision process;

[0029] The fifth module is used to optimize the training of the collaborative decision-making model of the freight formation based on the multi-agent reinforcement learning training framework, aiming at safety, traffic efficiency and energy consumption goals.

[0030] As an optional implementation, the multi-agent reinforcement learning training framework is based on the CTDE training paradigm.

[0031] As an optional implementation, the local state space is composed of a basic state space and a formation control state space;

[0032] The global state space is formed by stacking local state spaces.

[0033] As an optional implementation, a sixth module is further included, which is used to apply the formation control method of the staggered structure to the formation control state space in the local state space and the team reward function.

[0034] As an optional implementation manner, the fourth module is specifically used to:

[0035] Inputting the local state space of each of the intelligent freight vehicles at the current moment and the joint action space at the previous moment into the decision network of the QMIX algorithm, and inputting the global state space into the hybrid network of the QMIX algorithm;

[0036] The parameters of the hybrid network and the parameters of the decision network are updated by minimizing the predicted joint Q value and the target joint Q value in the QMIX algorithm.

[0037] As an optional implementation manner, the sixth module is specifically used for:

[0038] The collaborative decision-making model of the freight formation is optimized and trained according to the team reward function, wherein the team reward function includes a collision penalty function, a speed reward function, a dangerous action penalty function, an energy consumption penalty function, an unnecessary lane-changing behavior penalty function and a formation control reward function.

[0039] As an optional implementation, the plurality of intelligent freight vehicles in the highway multi-lane scenario model are controlled by a preset hierarchical framework, and the hierarchical framework includes an upper decision layer and a lower control layer;

[0040] The decision layer is used to provide the intelligent freight vehicles with a formation control method of the staggered structure;

[0041] The control layer is used to control the speed and acceleration of the intelligent freight vehicle and calculate the instantaneous energy consumption of the intelligent freight vehicle.

[0042] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the autonomous driving freight formation collaborative decision-making method as described in any one of the first aspects above is implemented.

[0043] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed, the autonomous driving freight formation collaborative decision-making method as described in any one of the first aspects above is implemented.

[0044] As can be seen from the above, the present invention proposes a customized partially observable Markov decision process (POMDP) ​​model, which is aimed at the collaborative decision-making of freight formations in highway scenarios. The model uses a staggered structured formation control method to enhance the perception information of the freight formation in the training phase, and further improves the effect of the agent decision network in the testing phase. Furthermore, we combined the QMIX algorithm to construct a multi-agent reinforcement learning framework based on the centralized training and decentralized execution (CTDE) training paradigm to optimize the safety, traffic efficiency and energy consumption targets of the freight formation, thereby prompting the freight formation to produce spontaneous collaborative behavior. Compared with traditional freight formation control methods, the proposed model and framework of the present invention show superior performance in improving the collaborative decision-making of freight formations, which not only greatly improves driving safety and traffic efficiency, but also significantly reduces the overall energy consumption of freight formations.

[0045] The beneficial effects of the additional technical features of the present invention will be described in the specific implementation section of this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 It is a flowchart of a collaborative decision-making method for an autonomous driving freight formation provided by an embodiment of the present invention;

[0048] Figure 2 It is a structural schematic diagram of an autonomous driving freight formation collaborative decision-making device provided by an embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention;

[0050] Figure 4It is a flowchart of a specific research method provided by an embodiment of the present invention;

[0051] Figure 5 is a schematic diagram of a random dynamic traffic flow environment provided by an embodiment of the present invention;

[0052] Figure 6 is a schematic diagram of a formation control state space provided by an embodiment of the present invention;

[0053] Figure 7 is a schematic diagram of a dangerous action scenario provided by an embodiment of the present invention;

[0054] Figure 8 is a schematic diagram of the effect of a formation control method provided by an embodiment of the present invention on lane conflicts;

[0055] Fig. 9 is a schematic diagram of vehicle information of a formation control method provided by an embodiment of the present invention;

[0056] Fig.10 A schematic diagram comparing a staggered formation provided by an embodiment of the present invention with a traditional formation

[0057] Fig.11 It is a schematic diagram of average speed and energy consumption distribution of a proposed model fleet provided by an embodiment of the present invention;

[0058] Fig.12 is a schematic diagram of average speed and energy consumption distribution of a Baseline A model fleet provided by an embodiment of the present invention;

[0059] Fig.13 is a schematic diagram of average speed and energy consumption distribution of a Baseline B model fleet provided by an embodiment of the present invention;

[0060] Fig.14 is a schematic diagram of an implicit negotiation behavior test scenario provided by an embodiment of the present invention;

[0061] Fig.15 It is a schematic diagram of the comparison of CTCE, DTDE, and CTDE training algorithms provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] This paper proposes a collaborative decision-making method for freight formations by using a deep reinforcement learning algorithm for multiple intelligent freight vehicles. The specific contributions are as follows:

[0064] 1. In the highway scenario, considering the local perception of a single intelligent freight vehicle within the freight formation and the formation information perception after V2V communication sharing within the fleet, a customized partially observable Markov decision process model was established, and a staggered structured formation control method was used to enhance the perception information of the freight formation in the training phase, which was added to the reward function to further improve the effect of the intelligent agent decision network in the testing phase.

[0065] 2. Combined with the QMIX algorithm, a multi-agent reinforcement learning training framework based on the CTDE training paradigm is proposed. Under this framework, optimization is performed for safety, traffic efficiency, and energy consumption targets. Formation control and energy consumption optimization work together to encourage freight formations to produce spontaneous collaborative behavior.

[0066] 3. Through experimental verification and experimental comparison with conventional models and methods, the superior performance of the proposed model and framework in improving the collaborative decision-making of freight platoons is verified.

[0067] See also Figure 1 and Figure 4 , Figure 1 is a flowchart of a collaborative decision-making method for an autonomous driving freight platoon provided by an embodiment of the present invention. Figure 4 1 is a flowchart of a specific research method provided by an embodiment of the present invention. The autonomous driving freight formation collaborative decision-making method shown in the figure includes:

[0068] S110, construct a highway multi-lane scenario model based on the highway-env simulation environment.

[0069] This embodiment constructs a one-way four-lane highway scene based on the highway-env simulation environment, and configures the roads and vehicles in the scene in detail. Specifically, the highway multi-lane scene model is configured with highway road information and vehicle information of multiple intelligent freight vehicles, the vehicle information includes speed and acceleration, and the intelligent freight vehicles have the ability to perceive the surrounding environment.

[0070] Those skilled in the art should understand that "highway-env" is an open source autonomous driving simulation environment that can be used to simulate the environment of highway driving. This embodiment can use this environment to train an autonomous driving car.

[0071] The multiple lanes in this embodiment may be more than three lanes, and four lanes are used as an example in the specific simulation experiment hereinafter.

[0072] S120, design a formation control method for a staggered structure.

[0073] Among them, the staggered structure (also called diamond structure) formation means that the smart freight vehicles in the structure do not simply follow the previous vehicle, but are staggered between different lanes.

[0074] It should be pointed out that the formation control method is based on a collaborative decision-making model of a freight formation, and the freight formation is composed of the multiple intelligent freight vehicles.

[0075] S130: Establish a customized partially observable Markov decision process according to the highway multi-lane scenario model and the platoon control method.

[0076] Specifically, the freight platoon collaborative decision-making model established in this application simulates the perception limitations of autonomous vehicles in real-world scenarios, assuming that each intelligent freight vehicle in the fleet can only obtain its own local observation state and the formation information obtained by V2V communication within the formation. Since the perception of each agent in the simulation is also limited, the multi-agent environment can be constructed as a customized partially observable Markov decision process (POMDP, Partially Observable Markov Decision Process).

[0077] The elements of the Markov decision process in this embodiment include the local state space, the global state space, the joint action space and the team reward function of the intelligent freight vehicle.

[0078] In this embodiment, the local state space is composed of a basic state space and a formation control state space; and the global state space is formed by stacking the local state spaces.

[0079] Furthermore, the formation control method of the staggered structure is applied to the formation control state space in the local state space and the team reward function.

[0080] In POMDP, the agent cannot directly observe the true state of the system and must rely on the global state and its own local state to infer the distribution of the state, which can be expressed as a tuple:

[0081] <S,A,Z,P,O,R,γ>

[0082] Where S is the global state space, recording all observed states of the entire freight fleet. A={a1,a2,…a n} is the joint action space, which records the actions of n intelligent freight vehicles. n} is the joint local state space, which records the observed states of each of the n intelligent freight vehicles. The intelligent freight vehicle selects an action according to the strategy, and the actions of all intelligent freight vehicles form a joint action, and then the next state is generated according to the state transition function P. O is the observation probability function, and the relationship between observation and state is reflected by the observation probability. The freight formation shares a team reward function R, γ is the discount factor, and the observed states of each intelligent freight vehicle. The intelligent freight vehicle selects an action according to the strategy, and the goal of all intelligent freight vehicles is to maximize the team reward.

[0083] In this embodiment, the constructed freight formation POMDP model adopts the CTDE training paradigm, and the concept of local state space needs to be clarified. In the training phase, although the agents use the local state space to execute decisions, they indirectly use the global state space through a centralized learning mechanism during training. In the testing phase, the agents can no longer access the global state space, and they must rely entirely on their own local state space to make decisions. This local state space becomes the real state space of the agent. Therefore, in this embodiment, all local state spaces are referred to as local observation spaces, which means that the local state space is the actual environmental observation of the agent during the training and testing phases.

[0084] Among them, the basic state space of each intelligent freight vehicle in the freight formation is independent and can be divided into two state variables:

[0085]

[0086] In order to obtain the specific information of the surrounding vehicles, the vectors of the relative distances and speeds of the five surrounding vehicles are established. The surrounding vehicles considered are the five environmental vehicles in front, left front, left rear, right front and right rear of each intelligent freight vehicle, such as Figure 5 Therefore, the first state variable

[0087]

[0088] Where, d n,t represents the relative distance of n intelligent freight vehicles to the intelligent freight vehicle at the current time t; v n,t Represents the speed of n intelligent freight vehicles at the current time t.

[0089] like Figure 5 As shown, since the perception range of autonomous driving vehicles is also limited in real scenarios, it is set in the simulation environment that each intelligent freight vehicle can perceive vehicles within a relative distance of up to 150 meters.

[0090] In order to make the intelligent freight vehicle aware of the instantaneous energy consumption, the second state variable is the speed and acceleration of the intelligent freight vehicle (for specific reasons, please refer to the reward function section below):

[0091]

[0092] The vehicles in the staggered structure are staggered between different lanes. This embodiment takes three intelligent freight vehicles as an example, namely the leading vehicle, the middle vehicle and the rear vehicle. In order to ensure that the convoy can maintain its formation, the convoy formation control state space is increased according to the characteristics of the freight formation, such as Figure 6 As shown, (a) relative distance within the fleet, (b) direction coding. The formation control state space of the three intelligent freight vehicles in the freight formation is shown in the following three equations:

[0093]

[0094]

[0095] in, and They are the formation control state spaces of the leading vehicle, the middle vehicle and the rear vehicle respectively. The relative distance between car i and car j is sensed, with a maximum distance of 100 meters. i is the global direction of vehicle i, which is recorded in binary coding, such as Figure 6 (b) As shown, there is no distance limit. This embodiment assumes that an independent intra-team communication mechanism is established within the convoy. On this basis, in view of the implementation of relatively strict formation control, the leading vehicle is always at the front of the convoy and is responsible for being the first to enter complex traffic scenes. The driving direction of the leading vehicle determines the overall direction of travel of the entire convoy. However, if the leading vehicle chooses the wrong driving path, the information it provides can still be used by the two following vehicles to correct the error. The main task of the last two vehicles in the convoy is to maintain the formation structure of the convoy. To this end, the middle vehicle and the rear vehicle need to obtain the relative distance and global direction information with the front vehicle in real time, while the leading vehicle needs to master the relative distance data with the two vehicles behind to ensure the coordination and driving safety of the convoy.

[0096] This state variable will enable the two following vehicles to be aware of the specific position of the leading vehicle in the formation, and will also provide the environment state for controlling the formation shape in the reward function.

[0097] By limiting the perception range of each intelligent freight vehicle in the local observation space, including the perception of the surrounding vehicles and the perception inside the formation, the local observation space of each truck is established, which is composed of the basic state space S of the three intelligent freight vehicles. basics and their respective formation control state spaces, as shown below:

[0098]

[0099] In the training phase, the global state space assumes that information sharing is achieved within the entire formation and is composed of the local observation spaces of the three intelligent freight vehicles, as shown in the following formula:

[0100] S global =[S l ,S m ,S t ]

[0101] The construction of the action space is aimed at realizing the behavioral decision-making of autonomous vehicles. Therefore, the action space consists of six discrete actions, namely, turn left, turn right, hold, accelerate, decelerate and decelerate sharply.

[0102] S140, designing a multi-agent reinforcement learning training framework through the QMIX algorithm according to the customized partially observable Markov decision process.

[0103] Those skilled in the art should understand that the QMIX algorithm is a multi-agent reinforcement learning algorithm that aims to solve the problem of how multiple agents can achieve global optimization through learning in a cooperative environment. The algorithm decomposes the global Q function into a linear combination of multiple local Q functions, and uses a hybrid network to learn the impact of different agents on the global in the current state, thereby combining them into a global Q value. This decomposition needs to satisfy the monotonicity constraint to ensure that the optimal action selected based on the local Q function is also the optimal action of the global Q function. The QMIX algorithm uses centralized training and distributed execution to enable each agent to learn a distributed strategy based on its local observation information, so that the action with the maximum cumulative expected reward is selected through the local value function during execution.

[0104] Specifically, in the training phase, the local observation space of each intelligent freight vehicle and the sequence experience of the joint action at the last moment are input into the decision network of the QMIX algorithm. At the same time, the global state space of the fleet formed by combining the local observation space is input into the hybrid network of the QMIX algorithm; the parameters of the hybrid network and the parameters of the decision network are updated by minimizing the joint Q value and the target joint Q value in the predicted QMIX algorithm. In the testing phase, only the decision network and local observation space of each intelligent freight vehicle are used for decision making.

[0105] Among them, there may be more than three intelligent freight vehicles in this embodiment, and three vehicles are used as an example in the specific simulation experiment below.

[0106] In this embodiment, in the multi-agent scenario of freight formation, the QMIX algorithm is used to make collaborative decisions of multiple intelligent freight vehicles. In the specific implementation, the QMIX algorithm uses a hybrid network to merge the local value functions of single agents and adds global state information assistance during the training and learning process to improve the algorithm performance, as shown in the following formula:

[0107]

[0108] The QMIX algorithm uses the same update method as the DQN algorithm, as shown below:

[0109]

[0110] Where b is the batch size sampled from the cache replay pool, and y is calculated using the target network tot = r + γmax a′ Q tot (τ′,a′,s′;θ-),θ - are the parameters of the target network.

[0111] like Figure 4 As shown, this embodiment uses the CTDE (Controlled Template Dependent Elongation, centralized training + decentralized decision-making) method to train the QMIX algorithm. CTDE is a centralized training and distributed execution training paradigm, which is mainly used in multi-agent reinforcement learning. In the QMIX algorithm, during the training phase, the hybrid network is used as a central controller to obtain the global state space and team reward function, as well as the action value calculated by each agent's respective policy network to calculate a joint action value, and a similar DQN algorithm is used to minimize the TD error to update the hybrid network and each agent's respective policy network. In the decision-making stage, this hybrid network does not need to be used. Each agent only needs to know its own local observation information to make effective decisions.

[0112] S150, based on the multi-agent reinforcement learning training framework, the collaborative decision-making model of the freight formation is optimized and trained with respect to safety, traffic efficiency and energy consumption goals.

[0113] In this embodiment, the multiple intelligent freight vehicles in the highway multi-lane scenario model are controlled by a preset hierarchical framework, which includes an upper decision-making layer and a lower control layer; the decision-making layer is used to provide the intelligent freight vehicles with the formation control method of the staggered structure; the control layer is used to control the speed and acceleration of the intelligent freight vehicles, and calculate the instantaneous energy consumption of the intelligent freight vehicles.

[0114] This embodiment formulates an optimization scheme for three goals (safety, traffic efficiency, and energy consumption goals) based on the team reward function, and adds a formation control reward function. Specifically, the team reward function includes a collision penalty function, a speed reward function, a dangerous action penalty function, an energy consumption penalty function, an unnecessary lane change penalty function, and a formation control reward function.

[0115] The team reward function regards the task as a multi-objective optimization problem, mainly considering the optimization of safety, traffic efficiency, energy consumption and formation control, which is specifically defined as:

[0116]

[0117] w i are the weight coefficients of each reward function, which are -3, 0.5, -0.4, -0.2, -0.3, 0.3, and 0.4 respectively. The weights represent rewards for the speed and formation control of the freight formation, and penalties for collisions, dangerous actions, lane changes, and energy consumption, as follows:

[0118] ①Collision penalty function:

[0119]

[0120] ②Speed ​​reward function:

[0121] v θ =v×cos(θ)

[0122] r speed =v θ ∈[20,32]→[0,1]

[0123] Where θ is the heading angle of the intelligent freight vehicle, v is the true speed, and v θ The deviation speed is controlled between 20m / s and 32m / s to ensure that it does not exceed the minimum and maximum speed limits of all lanes.

[0124] ③ Dangerous action penalty function: According to the dangerous scenarios that occur during the lane change process, four dangerous actions are defined. Assume that the current autonomous driving vehicle wants to take a left turn and the target lane is the left lane. Figure 7 As shown in (a), in order to avoid following the vehicle too closely, the minimum following distance is defined as d1. Figure 7 As shown in (b), the distance to the vehicle ahead in the target lane is defined as d2. Figure 7 As shown in (c), the distance to the vehicle behind the target lane is defined as d3. Figure 7 As shown in (d), the vehicle body length is defined as length, there is a parallel vehicle in the target lane, and the longitudinal center distance between the parallel vehicle and the target lane is d4. The specific formula is as follows:

[0125]

[0126] ④ Unnecessary lane-changing behavior penalty function: In order to avoid meaningless frequent lane changes of autonomous vehicles, a weak penalty r is given lane If you change lanes when the distance to the vehicle in front is greater than 30 meters, you will be penalized.

[0127] ⑤ Energy consumption penalty function: Since the control data of speed and acceleration are relatively easy to obtain, this embodiment uses the battery energy flow model proposed in the prior art to calculate the energy consumption. By inputting the speed and acceleration information in one frame after each decision action arrives at the simulation environment, the instantaneous energy consumption of the vehicle at the current decision step is calculated. The energy consumption penalty r is defined energy ,According to the vehicle’s speed and acceleration as well as the energy consumption model, the instantaneous energy consumption is obtained in real time and properly normalized.

[0128] It should be pointed out that the above five reward functions are calculated independently for the three intelligent freight vehicles.

[0129] ⑥ Formation control reward function: Finally, in order to achieve formation control, a formation control reward function was established based on the driving characteristics of the freight convoy, including formation control of longitudinal position and lane, such as Figure 6 (a). The formation control reward function of the longitudinal position is as follows:

[0130]

[0131] Lane formation control reward function As shown below:

[0132]

[0133] Among them, i front is the lane ID of the preceding vehicle, i behind is the lane ID of the rear vehicle. This formula regards the leading vehicle as the front vehicle and the middle vehicle as the rear vehicle to control the front and rear lanes; it regards the middle vehicle as the front vehicle and the rear vehicle as the rear vehicle to control the front and rear lanes.

[0134] As can be seen from the above, the present invention proposes a customized partially observable Markov decision process (POMDP) ​​model, which is aimed at the collaborative decision-making of freight formations in highway scenarios. The model uses a staggered structured formation control method to enhance the perception information of the freight formation in the training phase, and further improves the effect of the intelligent agent decision network in the testing phase. Furthermore, we combined the QMIX algorithm to construct a multi-agent reinforcement learning framework based on the centralized training and decentralized execution (CTDE) training paradigm to optimize the safety, traffic efficiency and energy consumption targets of the freight formation, thereby prompting the freight formation to produce spontaneous collaborative behavior. Compared with traditional freight formation control methods, the proposed model and framework of the present invention show superior performance in improving the collaborative decision-making of freight formations, which not only greatly improves driving safety and traffic efficiency, but also significantly reduces the overall energy consumption of freight formations.

[0135] In order to present the technical effect, this embodiment verifies the formation control method and energy consumption optimization target of the staggered structure, experimentally evaluates the effect of these strategies on the performance improvement of multi-agent reinforcement learning collaborative decision-making, and examines their adaptability under different traffic flows. Finally, the advantages of the modeling method, multi-agent reinforcement learning algorithm and training paradigm proposed in this application in the freight formation scenario are explored.

[0136] Scenario construction: This application constructs a four-lane one-way highway scenario based on the Highway-env simulation environment, and sets lane speed limits. For vehicles such as small trucks, a single-track model is used for dynamic control, and related parameters are adjusted to be more suitable for the underlying control of small trucks. Environmental vehicles all use the IDM model and MOBIL model for upper-level decision-making, and the bottom layer uses a nonlinear single-vehicle model for kinematic control. The specific scenarios and key vehicle parameters are shown in the following table.

[0137] Parameter Type Numeric Number of lanes 4 Lane ID1 (leftmost lane) speed limit range m / s [30,32] Lane ID2 speed limit range m / s [28,30] Lane ID3 speed limit range m / s [26,28] Lane ID4 (rightmost lane) speed limit m / s [24,26] Traffic flow veh / h 1200 Speed ​​limit range for autonomous trucks m / s [20,32] Self-driving truck mass kg 2500 Center of gravity height of autonomous truck m 0.8 Wheelbase of autonomous truck m 3.0 Wheelbase of autonomous truck m 1.6

[0138] Algorithm configuration: Each intelligent freight vehicle in the QMIX algorithm is controlled by an RNN network, which contains three layers: input layer (MLP), middle layer (GRU) and output layer (MLP). The number of neurons in the middle layer GRU is 64. The hybrid network accepts the Q value and global state space calculated by the RNN network of the agent, and mainly consists of two layers of DNN networks with 64 neurons. During the training process, the cosine annealing strategy is used to update the learning rate and the exponential calculation method is used to update the exploration rate. All experiments were conducted on i5-12400F CPU and NVIDIA GeForce RTX 4060Ti computers. The specific hyperparameter settings are shown in the following table:

[0139] Settings size Batch-size 32 Initial learning rate 1e-3 Cache pool size 5e3 Target network update frequency 200 Discount Factor 0.99

[0140] The comparison model settings in the following experiments are shown in the following table:

[0141] Model Specific settings Proposed model The model proposed in this paper Baseline A model Cancel formation control Baseline B model Parallel formation structure Baseline C model Cancel energy consumption penalty

[0142] In order to test the effect of the staggered platoon layout control method, this section conducted an experiment on a 3000-meter-long highway section and fixed the starting positions of all vehicles. Figure 8 As shown in the figure. When the platoon control method is applied, if two vehicles are in the same lane, it is considered a lane conflict. In the figure, the intersection point of each track is the conflict point.

[0143] Experimental results show that the Proposed model with formation control has more lane conflict locations between vehicles than the Baseline A model without formation control, and lane changes are more frequent. This conflict may cause the formation to be disrupted, but formation control allows the team to temporarily disband when avoiding obstacles, increasing the flexibility of dynamic risk avoidance. However, frequent lane conflicts may cause the team to lose local state information of other lanes, thereby affecting the information integrity of the global state space. Since the hybrid network of the QMIX algorithm relies on global state space information, this information loss will seriously affect the training effect of the model. Fig. 9 (a) Comparison of reward curves shows that the reward curve of the Baseline A model fluctuates greatly due to the lack of formation control rewards.

[0144] When the leading vehicle brakes suddenly, if the following vehicles do not maintain the formation, the overall traffic efficiency of the convoy will be reduced due to emergency avoidance, and a chain rear-end collision may occur. Fig. 9 (b) Comparison of the average speed distribution of the fleet shows that after 1000 rounds of testing on the same road, the average speed distribution of the fleet of the Proposed model is more concentrated and basically close to the set maximum speed limit, while the speed distribution of the Baseline A model fluctuates greatly.

[0145] As shown in the table below, after 1,000 rounds of the same road test, the collision rate of the Proposed model was reduced by 17.8% compared to the Baseline A model.

[0146] method Collision round ratio (%) Proposed model 3.7 Baseline A model 21.5 Baseline B model 42.7

[0147] This experiment compares the differences between staggered formation and traditional formation (parallel structure). Fig. 9As shown in (b), the average speed of the Baseline B model fleet with the traditional formation structure fluctuates significantly, especially when the traffic volume is large, it is difficult to maintain a compact formation, resulting in frequent changes in the formation and increased driving risks. As shown in the table above, the collision rate of the Proposed model is 39% lower than that of the Baseline B model. In the multi-agent reinforcement learning environment, especially the multi-agent collaborative decision-making algorithm of this paper, vehicles driving in the same lane for a long time will lead to the loss of information about other lanes, such as Fig.10 As shown in Figure 1, the more lanes there are and the greater the traffic volume, the more obvious this phenomenon is. The staggered structure helps the fleet obtain the global state information of the entire road, which is beneficial for algorithm training. Fig. 9 As shown in (a), the reward curve of the Proposed model is more stable than that of the Baseline B model.

[0148] In order to verify the adaptability of the proposed method to diverse traffic conditions, this study compared the performance of three models, Proposed, Baseline A, and Baseline B, within 1000 rounds, with the same test road type but different traffic flows.

[0149] The results are shown in Figures 11 to 13 As the traffic volume increases, the performance of the proposed model in terms of average speed and energy consumption decreases slightly, but the overall performance remains at a high level. For example, at a traffic volume of 1600 veh / h, its minimum speed is 28.4 m / s and the median is 30.26 m / s.

[0150] In contrast, the Baseline A model exhibits high efficiency at low traffic volumes due to the lack of platooning control, but its performance significantly decreases as traffic volumes increase, especially at a traffic volume of 1,600 veh / h, where the average speed fluctuates greatly.

[0151] For the Baseline B model, it has a significant negative impact on the average speed and energy consumption under high traffic flow. Fig.13 (b) shows that at a traffic flow of 1600veh / h, the energy consumption drops to a minimum of 12.01kWh. This is mainly due to the significant increase in the collision rate, which causes vehicles to collide soon, thereby exacerbating the overall energy consumption fluctuation.

[0152] As shown in the table below, the proposed model maintains a leading position under all traffic flows, indicating that the proposed method has significant advantages over other traffic conditions in a multi-agent reinforcement learning environment.

[0153]

[0154] Collaborative decision-making experiment: When the convoy is about to enter a congested section, energy consumption optimization enables vehicles to change lanes or slow down in advance to avoid frequent acceleration and deceleration when approaching. At the same time, due to platoon control, the following vehicles will observe the direction of the leading vehicle, showing an implicit negotiation behavior. This behavior naturally occurs without explicitly setting a mechanism in this article. This section aims to explore the reasons for this negotiation behavior.

[0155] By observing the behavioral tendencies of three vehicles in the same lane-changing scenario, it is found that under the guidance of the leading vehicle, the middle vehicle in the proposed model tends to turn left because the lane where the leading vehicle is located is relatively unobstructed. Fig.14 As shown in the figure, at the 38th decision step, the middle car showed an intention to turn left and completed the left turn at the 42nd step. In the absence of collaborative decision-making, the middle car has a high probability of turning right because of the dangerous action penalty in the reward function. There are environmental cars in the left front, left rear, and front of the perception range, while the environmental car in the right front is not in the perception range and there is no environmental car in the right rear. However, in the actual 42nd step, the three environmental cars in front are driving in parallel. If they turn right, it will hinder the passage of the two cars in the rear of the truck formation.

[0156] This experiment conducted 1000 rounds of action selection tests on the Proposed model, Baseline A model, and Baseline C model in this scenario, aiming to compare the impact of energy consumption optimization objectives and formation control methods on collaborative decision-making. The results are shown in the following table:

[0157] Model Left turn rate (%) Ratio of right turns (%) Proposed model 98.2 1.8 Baseline A model 55.1 44.9 Baseline C model 83.7 16.3

[0158] The results show that the Baseline A model has a higher right turn frequency because it does not use formation control. Similarly, the Baseline C model has an increased right turn frequency because it does not include energy optimization in its objectives, which shows that energy optimization also has a small impact on collaborative decision-making. Since the reward function of formation control directly affects the lane change strategy, formation control plays a decisive role in collaborative decision-making.

[0159] Training paradigm comparison experiment:

[0160] In order to explore the advantages of POMDP modeling in multi-agent collaborative decision-making, this experiment designed three comparative experiments based on the multi-agent training paradigm:

[0161] “Centralized training + centralized decision making” (CTCE) trains the D3QN algorithm.

[0162] “Decentralized Training + Decentralized Decision” (DTDE) training D3QN algorithm

[0163] “Centralized training + decentralized decision-making” (CTDE) trains the QMIX algorithm, which is the method used in the freight formation of this application.

[0164] This experiment uses the D3QN algorithm to compare with the QMIX algorithm because the QMIX algorithm is a value-based multi-agent reinforcement learning algorithm that combines the ideas of Actor-Critic and DQN. The D3QN algorithm is a value-based single-agent reinforcement learning algorithm, which is an improved version of the DQN algorithm. The two algorithms have certain commonalities and are suitable for different scenarios.

[0165] In the CTCE method, when training the D3QN algorithm, based on the global state space of the POMDP process designed in this application, the vehicle number information of the three intelligent freight vehicles is added, so that the vehicles can identify information related to themselves in the global state space, similar to MDP process modeling, and the three trucks use the same global state space and model to output the actions of the three vehicles at the same time. In contrast, the DTDE method trains a D3QN model for each truck independently, similar to training a single-agent model, and each truck uses its own local observation space in the POMDP process as the state space, and uses its own model to output its own actions.

[0166] like Fig.15 As shown, the QMIX algorithm trained under the CTDE architecture achieved a steady increase in cumulative rewards and significantly reduced the collision rate in the later stages of training. In contrast, the D3QN algorithm trained with the CTCE architecture failed to converge and the collision rate remained high, indicating that even with very complete state information, it is still challenging for a unified decision network to output multi-vehicle decisions. On the other hand, the D3QN algorithm trained with the DTDE architecture is similar to single-agent training, so it converged in about 700 rounds. Although the collision rate has decreased, the lack of global state space assistance in the training phase of the architecture leads to a lack of coordination among the three vehicles while driving on the road, causing the collision rate to remain above 0.2.

[0167] See also Figure 2 , Figure 2 1 is a schematic diagram of the structure of an autonomous driving freight platoon collaborative decision-making device provided by an embodiment of the present invention. The autonomous driving freight platoon collaborative decision-making device as shown in the figure includes:

[0168] The first module constructs a highway multi-lane scenario model based on the highway-env simulation environment. The highway multi-lane scenario model is configured with highway road information and vehicle information of multiple intelligent freight vehicles, the vehicle information includes speed and acceleration, and the intelligent freight vehicles have the ability to perceive the surrounding environment;

[0169] The second module is used to design a formation control method of a staggered structure, wherein the formation control method is based on a collaborative decision-making model of a freight formation, and the freight formation is composed of the multiple intelligent freight vehicles;

[0170] A third module is used to establish a customized partially observable Markov decision process according to the highway multi-lane scenario model and the formation control method, wherein the elements of the Markov decision process include the local state space, the global state space, the joint action space and the team reward function of the intelligent freight vehicle;

[0171] The fourth module is used to design a multi-agent reinforcement learning training framework through the QMIX algorithm according to the customized partially observable Markov decision process;

[0172] The fifth module is used to optimize the training of the collaborative decision-making model of the freight formation based on the multi-agent reinforcement learning training framework, aiming at safety, traffic efficiency and energy consumption goals.

[0173] As an optional implementation, the multi-agent reinforcement learning training framework is based on the CTDE training paradigm.

[0174] As an optional implementation, the local state space is composed of a basic state space and a formation control state space;

[0175] The global state space is formed by stacking local state spaces.

[0176] As an optional implementation, a sixth module is further included, which is used to apply the formation control method of the staggered structure to the formation control state space in the local state space and the team reward function.

[0177] As an optional implementation manner, the fourth module is specifically used to:

[0178] Inputting the local state space of each of the intelligent freight vehicles at the current moment and the joint action space at the previous moment into the decision network of the QMIX algorithm, and inputting the global state space into the hybrid network of the QMIX algorithm;

[0179] The parameters of the hybrid network and the parameters of the decision network are updated by minimizing the predicted joint Q value and the target joint Q value in the QMIX algorithm.

[0180] As an optional implementation manner, the sixth module is specifically used for:

[0181] The formation control method of the staggered structure is re-optimized according to the team reward function, wherein the team reward function includes a collision penalty function, a speed reward function, a dangerous action penalty function, an energy consumption penalty function, an unnecessary lane changing behavior penalty function and a formation control reward function.

[0182] As an optional implementation, the plurality of intelligent freight vehicles in the highway multi-lane scenario model are controlled by a preset hierarchical framework, and the hierarchical framework includes an upper decision layer and a lower control layer;

[0183] The decision layer is used to provide the intelligent freight vehicles with a formation control method of the staggered structure;

[0184] The control layer is used to control the speed and acceleration of the intelligent freight vehicle and calculate the instantaneous energy consumption of the intelligent freight vehicle.

[0185] See also Figure 3 , Figure 3 The electronic device shown in the figure includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned Figure 1 The collaborative decision-making method for autonomous driving freight platooning described in the corresponding embodiment.

[0186] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned Figure 1 The collaborative decision-making method for autonomous driving freight platooning described in the corresponding embodiment.

[0187] Aiming at the highway freight platooning scenario, the present invention establishes a partially observable Markov decision process customized to the characteristics of the scenario, proposes a control method for staggered layout platooning, and proposes a multi-agent reinforcement learning training framework in combination with the reinforcement learning environment. Finally, the safety, traffic efficiency and energy consumption targets are optimized, which has the following advantages:

[0188] 1. In the freight formation scenario, the formation control method using an interlaced structure provides more comprehensive global state space information in POMDP process modeling and algorithm training, improves the effect of model training, and reduces the collision rate by 17.8% and 39% respectively compared with the non-formation control method and the traditional formation structure.

[0189] 2. In terms of multi-agent collaborative decision-making, the interaction between energy consumption optimization goals and formation control enables the fleet to avoid congested sections in advance, forming a negotiation behavior that can be generated without a clear mechanism. This negotiation behavior further promotes the tendency of collaborative decision-making within the formation, prompting each vehicle in the entire fleet to execute decisions that are more beneficial to the entire fleet.

[0190] 3. In the freight platooning scenario, the fully customized POMDP process modeling, the use of multi-agent reinforcement learning algorithm and the CTDE training architecture show better performance than other conventional modeling and methods, significantly improve the decision-making effect of freight platooning, and can adapt to environments with higher traffic flow.

[0191] The above disclosure is only the preferred embodiment of the present invention, and its description is relatively specific and detailed. Of course, it cannot be used to limit the scope of the rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A collaborative decision-making method for autonomous driving freight platooning, comprising: Construct a highway multi-lane scenario model based on the highway-env simulation environment, wherein the highway multi-lane scenario model is configured with highway road information and vehicle information of multiple intelligent freight vehicles, wherein the vehicle information includes speed and acceleration, and the intelligent freight vehicles have the ability to perceive the surrounding environment; Designing a formation control method for a staggered structure, wherein the formation control method is based on a collaborative decision-making model of a freight formation, wherein the freight formation is composed of the plurality of intelligent freight vehicles; According to the highway multi-lane scenario model and the platoon control method, a customized partially observable Markov decision process is established, wherein the elements of the Markov decision process include the local state space, the global state space, the joint action space and the team reward function of the intelligent freight vehicle; Designing a multi-agent reinforcement learning training framework through the QMIX algorithm according to the customized partially observable Markov decision process; Based on the multi-agent reinforcement learning training framework, the collaborative decision-making model of the freight formation is optimized and trained with respect to safety, traffic efficiency and energy consumption goals.

2. The autonomous driving freight platoon collaborative decision-making method according to claim 1, characterized in that: The multi-agent reinforcement learning training framework is based on the CTDE training paradigm.

3. The autonomous driving freight platoon collaborative decision-making method according to claim 1, characterized in that: The local state space is composed of a basic state space and a formation control state space; The global state space is formed by stacking local state spaces.

4. The autonomous driving freight platoon collaborative decision-making method according to claim 3, characterized in that: Before designing a multi-agent reinforcement learning training framework through the QMIX algorithm according to the customized partially observable Markov decision process, the method further includes: The formation control method of the staggered structure is applied to the formation control state space in the local state space and the team reward function.

5. The autonomous driving freight platoon collaborative decision-making method according to claim 3, characterized in that: The optimizing training of the collaborative decision-making model of the freight formation includes: Inputting the local state space of each of the intelligent freight vehicles at the current moment and the joint action space at the previous moment into the decision network of the QMIX algorithm, and inputting the global state space into the hybrid network of the QMIX algorithm; The parameters of the hybrid network and the parameters of the decision network are updated by minimizing the predicted joint Q value and the target joint Q value in the QMIX algorithm.

6. The autonomous driving freight platoon collaborative decision-making method according to claim 1, characterized in that: The optimization training of the collaborative decision-making model of the freight formation according to the safety, traffic efficiency and energy consumption goals includes: The collaborative decision-making model of the freight formation is optimized and trained according to the team reward function, wherein the team reward function includes a collision penalty function, a speed reward function, a dangerous action penalty function, an energy consumption penalty function, an unnecessary lane-changing behavior penalty function and a formation control reward function.

7. The method for collaborative decision-making of an autonomous driving freight convoy according to any one of claims 1 to 6, characterized in that: The plurality of intelligent freight vehicles in the highway multi-lane scenario model are all controlled by a preset hierarchical framework, and the hierarchical framework includes an upper decision layer and a lower control layer; The decision layer is used to provide the intelligent freight vehicles with a formation control method of the staggered structure; The control layer is used to control the speed and acceleration of the intelligent freight vehicle and calculate the instantaneous energy consumption of the intelligent freight vehicle.

8. An autonomous driving freight platoon collaborative decision-making device, comprising: The first module is used to construct a highway multi-lane scenario model based on the highway-env simulation environment, wherein the highway multi-lane scenario model is configured with highway road information and vehicle information of multiple intelligent freight vehicles, wherein the vehicle information includes speed and acceleration, and the intelligent freight vehicles have the ability to perceive the surrounding environment; The second module is used to design a formation control method of a staggered structure, wherein the formation control method is based on a collaborative decision-making model of a freight formation, and the freight formation is composed of the multiple intelligent freight vehicles; A third module is used to establish a customized partially observable Markov decision process according to the highway multi-lane scenario model and the formation control method, wherein the elements of the Markov decision process include the local state space, the global state space, the joint action space and the team reward function of the intelligent freight vehicle; The fourth module is used to design a multi-agent reinforcement learning training framework through the QMIX algorithm according to the customized partially observable Markov decision process; The fifth module is used to optimize the training of the collaborative decision-making model of the freight formation based on the multi-agent reinforcement learning training framework, aiming at safety, traffic efficiency and energy consumption goals.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the autonomous driving freight formation collaborative decision-making method as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the autonomous driving freight formation collaborative decision-making method as described in any one of claims 1 to 7 is implemented.

Images