Vehicle platooning method

Abstract

The present disclosure provides a vehicle formation method, which includes: obtaining vehicle information and state space information of each vehicle; inputting the vehicle information and state space information into a first network model stored in the vehicle for decision-making and obtaining a decision result; obtaining feedback of executing a corresponding behavior action according to the decision result, and training the first network model according to the decision result and the feedback of the corresponding behavior action; uploading the trained first network models of Q vehicles among P vehicles to be formed into a server, the server obtaining an aggregate model according to the first network models of the Q vehicles, and transmitting the aggregate model to the vehicle; after the training of the first network model is completed, inputting the vehicle information and state control information of the P vehicles to be formed into the trained first network model, and controlling the vehicle to execute the output result of the first network model, so as to realize the formation driving of the P vehicles.

Description

Technical Field

[0001] The present invention relates to a vehicle formation method, and belongs to the technical field of vehicle network. Background Art

[0002] The field of artificial intelligence is developing rapidly, and reinforcement learning has made significant progress and achieved great success in solving various sequential decision-making problems in machine learning. With the deepening of research, multi-agent reinforcement learning is also showing great vitality in the field of reinforcement learning.

[0003] After several years of development and innovation, multi-agent deep reinforcement learning has produced many algorithms, rules, and frameworks, and has been widely used in various real-world fields. For example, multi-agents have made amazing progress in playing real-time strategy games, robot control, card games, and autonomous driving.

[0004] In addition to the popular methods mentioned above, multi-agent deep reinforcement learning also has potential applications in other fields, including cyber-physical systems, finance, sensor / communication networks, and social sciences. The development from single to multi, from simple to complex, and from low-dimensional to high-dimensional shows that in the field of machine learning and even artificial intelligence, multi-agent deep reinforcement learning is gradually becoming a hot research and application direction, with extremely high research value and significance.

[0005] In recent years, with the increase in the number of cars, traffic efficiency and safety have become very serious issues, and the air pollution caused by cars has become more and more serious. Vehicle platooning, as a vehicle traffic management technology, has been identified as a promising framework in intelligent transportation systems. Forming vehicle platoons on automated highways can optimize traffic and save fuel, while also improving passenger comfort and safety. In order to study the problem of vehicle platooning and then build a high-performance algorithm, multi-agent path planning based on reinforcement learning is essential.

[0006] Vehicle platooning is an important means of transportation to reduce vehicle energy consumption and improve air quality. How to control sparse vehicles on the road to form a platoon is a very meaningful research point. Summary of the invention

[0007] In order to solve one of the above technical problems, the present disclosure provides a vehicle platooning method.

[0008] According to one aspect of the present disclosure, a vehicle platooning method is provided, comprising:

[0009] Acquire vehicle information and state space information of each vehicle, wherein the vehicle information includes the type, longitudinal coordinate, lateral coordinate, speed and acceleration of the vehicle; the state space information includes a central area, a forward area, a rearward area, a left area, a right area, a left front area, a left rear area, a right front area and a right rear area centered on the vehicle;

[0010] Inputting the vehicle information and the state space information into a first network model stored in the vehicle for decision making, and obtaining a decision result, wherein the decision result includes the probability of a behavior action that each vehicle may currently perform, wherein the behavior action includes speed and lane changing behavior, wherein the speed includes accelerating, driving at a constant speed, and decelerating; and the lane changing behavior includes changing lanes to the left, maintaining the current lane, and changing lanes to the right;

[0011] According to the decision result, feedback of executing the corresponding behavior action is obtained, and the first network model is trained according to the decision result and the feedback of the corresponding behavior action; wherein the feedback includes a reward R1 obtained when the formation is not completely formed, a reward R2 obtained when a collision occurs between vehicles, and a reward R3 obtained when the entire formation process ends after the vehicles form a formation:

[0012] The trained first network models of Q vehicles among the P vehicles to be formed are uploaded to the server, the server obtains an aggregate model according to the first network models of the Q vehicles, transmits the aggregate model to the vehicle, and uses the aggregate model to update the first network model stored in the vehicle;

[0013] After the training of the first network model is completed, the vehicle information and state control information of the P vehicles to be formed are input into the trained first network model, and the vehicles are controlled to execute the output results of the first network model to realize the formation driving of the P vehicles.

[0014] According to the vehicle platooning method of at least one embodiment of the present disclosure, the first network model includes a mask layer, and the mask layer is used to limit behavioral actions in the decision results.

[0015] According to the vehicle platooning method of at least one embodiment of the present disclosure, during the vehicle platooning process, the lane with the largest number of vehicles to be platooned is selected as the platooning lane, and vehicles traveling in the platooning lane are not allowed to change lanes.

[0016] According to the vehicle platooning method of at least one embodiment of the present disclosure, a vehicle is not allowed to change lanes again within a preset time after changing lanes.

[0017] According to the vehicle platooning method of at least one embodiment of the present disclosure, the reward obtained when the platoon is not completely formed is obtained by the following formula:

[0018] R1=c1*(R power - cost +R power-save )+c2*R v-eff +c3*R lane-cost

[0019] Where c1 = 0.4, c2 = 0.1, c3 = 0.3; R power-cost is the reward for energy consumption in the running state; R power-save is the reward for energy saving; R v-eff is a reward for speed efficiency; R lane-cost It's a lane change reward.

[0020] According to the vehicle platooning method of at least one embodiment of the present disclosure, the reward for energy consumption in the running state is obtained by the following formula:

[0021]

[0022] Where W v，a Represents the energy consumed by the vehicle per unit time, W v-max It represents the energy consumed per unit time when the vehicle travels at maximum speed.

[0023] According to the vehicle platooning method of at least one embodiment of the present disclosure, the speed efficiency reward can be obtained by the following formula:

[0024]

[0025] Among them, v current is the current speed of the vehicle; v min is the minimum speed limit of the vehicle; v max The maximum speed limit for the vehicle.

[0026] According to the vehicle platooning method of at least one embodiment of the present disclosure, the lane change reward is obtained by the following formula:

[0027]

[0028] According to the vehicle platooning method of at least one embodiment of the present disclosure, in the reward obtained when a collision occurs between vehicles, when a collision occurs between vehicles, R2=-10.

[0029] According to the vehicle platooning method of at least one embodiment of the present disclosure, the reward obtained by the vehicles at the end of the entire platooning process after the platoon is formed is obtained by the following formula:

[0030]

[0031] Where, C = 300; W iis the energy consumed by the i-th vehicle during the formation process; represents the profit coefficient of the preceding vehicle of the i-th vehicle to the i-th vehicle; It represents the benefit coefficient of the following vehicle of the i-th vehicle to the i-th vehicle, and P represents the number of vehicles to be formed. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0033] Figure 1 is a flowchart of a vehicle platooning method according to one embodiment of the present disclosure.

[0034] Figure 2 is a state observation area diagram of a vehicle according to one embodiment of the present disclosure.

[0035] Figure 3 It is a framework diagram of a first network model according to an embodiment of the present disclosure.

[0036] Figure 4 It is a block diagram of the hardware structure used in the vehicle platooning method according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0038] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0039] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0040] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0041] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0042] For descriptive purposes, the present disclosure may use spatially relative terms such as "under", "beneath", "under", "down", "over", "upper", "above", "higher", and "side (e.g., as in "sidewall")" to describe the relationship of one component to another (other) component as shown in the accompanying drawings. The spatially relative terms are intended to encompass different orientations of the device in use, operation, and / or manufacture in addition to the orientation depicted in the accompanying drawings. For example, if the device in the accompanying drawings is turned over, components described as "under" or "beneath" other components or features would then be positioned "over" the other components or features. Thus, the exemplary term "under" can encompass both the "over" and "under" orientations. Furthermore, the device may be otherwise oriented (e.g., rotated 90 degrees or at other orientations), and the spatially relative descriptors used herein should be interpreted accordingly.

[0043] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0044] Figure 1 is a flowchart of a vehicle platooning method according to one embodiment of the present disclosure.

[0045] like Figure 1 As shown, the present disclosure provides a vehicle formation method, which includes: S10, obtaining vehicle information and state space information of each vehicle, wherein the vehicle information includes the type, longitudinal coordinates, lateral coordinates, speed and acceleration of the vehicle; the state space information includes a central area centered on the vehicle, a forward area, a rearward area, a left area, a right area, a left front area, a left rear area, a right front area and a right rear area; S20, inputting the vehicle information and state space information into a first network model stored in the vehicle for decision-making, and obtaining a decision result, the decision result including the probability of a behavior action that each vehicle may currently perform, wherein the behavior action includes speed and lane change behavior, the speed aspect includes accelerating driving, constant speed driving and decelerating driving; the lane change behavior includes changing lanes to the left, keeping the current lane and changing lanes to the right; S30, obtaining a lane change according to the decision result The first network model is trained according to the decision result and the feedback of the corresponding behavior action; wherein the feedback includes the reward R1 obtained when the formation is not completely formed, the reward R2 obtained when the collision occurs between vehicles, and the reward R3 obtained when the vehicle forms a formation and the entire formation process ends; S40, uploading the trained first network model of Q vehicles out of the P vehicles to be formed to the server, the server obtains an aggregate model according to the first network model of the Q vehicles, and transmits the aggregate model to the vehicle, and uses the aggregate model to update the first network model stored in the vehicle; S50, after the first network model is trained, the vehicle information and state control information of the P vehicles to be formed are input into the trained first network model, and the vehicle is controlled to execute the output result of the first network model to realize the formation driving of the P vehicles.

[0046] The above steps will be described in detail below.

[0047] The vehicles in S10 are P vehicles to be formed into a formation. The vehicle formation method disclosed in the present invention can enable these P vehicles to form a formation, and the energy consumption of the formation is the lowest during driving.

[0048] In addition, the platoon vehicles need to comply with corresponding rules during driving. For example, collisions cannot occur. In one embodiment, an improved following model can be used to ensure that two vehicles in the front-to-back direction (i.e., the direction in which the vehicles are running) do not collide and that the two vehicles run at the fastest speed.

[0049] The improved car-following model in the present disclosure is expressed by the following formula:

[0050] d=y a -y b -l

[0051] D(v b )+v b τ<D(v a )+d

[0052]

[0053]

[0054] In the improved car-following model, it is assumed that there are a front car a and a rear car b, y a is the longitudinal coordinate of the front vehicle a, y b is the longitudinal coordinate of the rear vehicle; the longitudinal direction is the forward running direction of the vehicle; l is the length of the vehicle itself, and d is the distance between the front vehicle a and the rear vehicle b.

[0055] In order to prevent the front vehicle a and the rear vehicle b from colliding during driving, it is necessary to satisfy D(v b )+v b τ<D(v a )+d, that is, the distance covered by the rear vehicle b during the entire braking process must be less than the sum of the braking distance of the front vehicle a and the distance between the two vehicles, where D represents the braking distance, τ represents the driver's reaction time, and v a represents the current speed of car a; v b represents the current speed of vehicle b. t represents the time it takes for the vehicle to reduce its current speed to 0 at the maximum deceleration, and D(v) represents the braking distance. With the above formulas, we can obtain the maximum safe following speed v allowed by vehicle b. safe .

[0056] On the other hand, when the vehicle platoon is completed, the energy consumption during driving can be obtained through the vehicle energy consumption model.

[0057] Specifically, the vehicle energy consumption W can be obtained by the following formula:

[0058]

[0059] W represents the total energy consumed per unit time T, m represents the weight of the vehicle, v0 and a represent the speed and acceleration of the vehicle respectively, μ represents the rolling resistance coefficient, g represents the acceleration of gravity, ρ represents the density of air, A represents the front airflow area of ​​the vehicle, C d Represents the aerodynamic drag coefficient.

[0060] Figure 2 is a state observation area diagram of a vehicle according to one embodiment of the present disclosure.

[0061] In the present disclosure, the state observation area map is mainly the environment around the vehicle. In a specific embodiment, each vehicle in the vehicles to be formed includes vehicle information and state space information, wherein the vehicle information includes the type, longitudinal coordinates, lateral coordinates, speed and acceleration of the vehicle; the state space information includes a central area centered on the vehicle, a forward area, a rearward area, a left area, a right area, a left front area, a left rear area, a right front area and a right rear area.

[0062] In actual application scenarios, when the roads to be driven by the vehicles in the platoon are two roads in the same direction, there is only one area to the left of the vehicle or the area to the right of the vehicle. In this case, the matrix can be filled with 0. For the convenience of explanation, this disclosure takes the example of 8 areas for vehicles for explanation.

[0063] When a vehicle is driving, the vehicle directly in front and the vehicle directly behind can be divided into the forward area and the rear area within a range of 70m. In other words, the front and rear observation range of the vehicle is 70m. When the position of the vehicle in the adjacent lane is within 0.5m of the position of the main vehicle, the vehicle is considered to be in the adjacent area of ​​the main vehicle, and the way the vehicle is divided into the left front, left rear, right front and right rear areas is similar to the above division method. Therefore, from the regional perspective, the dimension of the state information is fixed, that is, the corresponding 9 areas.

[0064] The vehicle information reflects the movement information of vehicles in the area. The longitudinal coordinate of the vehicle is the distance traveled on the road; the lateral coordinate of the vehicle is the lane information of the vehicle; and there are different types of vehicles, and different types correspond to different models, which means different energy consumption caused by different weights and areas, as well as the energy saving effect of different vehicles after forming a formation. Therefore, the dimension of the state space is 9*5, and when input into the first network model (i.e., the decision model), the fixed size of the state information is 45.

[0065] Figure 3 It is a framework diagram of a first network model according to an embodiment of the present disclosure.

[0066] In the present disclosure, the behavior of the vehicle is determined by the decision model according to the input state. Specifically, the vehicle information and state space information are input into the first network model stored in the vehicle for decision making, and a decision result is obtained, which includes the probability of the action that each vehicle may currently perform, wherein the action includes speed and lane change behavior, the speed includes accelerating, driving at a constant speed, and decelerating; the lane change behavior includes changing lanes to the left, keeping the current lane, and changing lanes to the right.

[0067] Preferably, the first network model of the present disclosure includes two parts, namely, the Actor network responsible for outputting actions and the Critic network responsible for outputting action scores. The Actor network consists of 3 linear layers and 1 recurrent neural network, and the sizes of the three linear layers are 45*128, 128*64, and 64*9 respectively. The Critic network consists of three linear layers, and the sizes of its linear layers are 45*128, 128*64, and 64*1 respectively. The activation functions used in the Actor network and the Critic network are both tanh functions.

[0068] The input of the Actor network includes not only the required state information and the required hidden state of the recurrent neural network, but also a mask information.

[0069] That is, the first network model includes a mask layer, and the mask layer is used to limit the behavior actions in the decision result.

[0070] The main purpose of the MASK mask layer is to restrict the behavior output by the decision model. Specifically, from the perspective of safety, vehicles on the road should avoid continuous lane changes as much as possible. Therefore, after the action output is lane change, in the future, the entire decision model can only output the current lane in the actions output, and the lane change action cannot be output; when the vehicle is in the edge lane and there is no lane on the left or right side of the vehicle, there is naturally no possibility of changing lanes in the corresponding direction, so it is necessary to prohibit the action of changing lanes to the corresponding lane; in the process of vehicle formation, the lane with the largest number of vehicles to be formed is selected as the formation lane, and vehicles driving in the formation lane are not allowed to change lanes; that is, if vehicles are allowed to change lanes at will, after a long time, multiple vehicles may still not be able to form a formation.

[0071] In the present disclosure, at a certain time point, a lane with more than half of the vehicles that need to form a formation is fixedly selected as the final formation lane. When a vehicle changes lanes into this lane, its lane changing capability is cancelled, so that all vehicles eventually enter the same lane and form a formation.

[0072] Each action has a corresponding value in the MASK layer. When a certain action needs to be prohibited, the value of the corresponding position in the MASK layer will be 0, and then the probability value of the corresponding action in the output layer of the neural network will be set to 0. Finally, the probability of each action is recalculated through the softmax layer, which is the final output result.

[0073] S30. Obtain feedback for executing a corresponding action according to the decision result, and train the first network model according to the decision result and the feedback for the corresponding action; wherein the feedback includes a reward R1 obtained when the formation is not completely formed, a reward R2 obtained when a collision occurs between vehicles, and a reward R3 obtained when the vehicles form a formation and the entire formation process ends.

[0074] The reward function is a very important part of reinforcement learning. An excellent reward function has a positive effect on the convergence of the model.

[0075] Specifically, the reward obtained when the formation is not fully formed is obtained by the following formula:

[0076] R1=c1*(R power-cost +R power-save )+c2*R v-eff +c3*R lane-cost

[0077] Where c1 = 0.4, c2 = 0.1, c3 = 0.3; R power-cost is the reward for energy consumption in the running state; R power-save is the reward for energy saving; R v-eff is a reward for speed efficiency; R lane-cost It's a lane change reward.

[0078] Among them, the reward for energy consumption in the running state is obtained by the following formula:

[0079]

[0080] Where W v，a Represents the energy consumed by the vehicle per unit time, W v-max It represents the energy consumed per unit time when the vehicle travels at the maximum speed. In the present disclosure, the energy consumption reward represents the energy consumed per unit time during the operation of the vehicle, which can be obtained through the energy consumption model described above.

[0081] Among them, the energy saving reward can be obtained by the following formula:

[0082]

[0083] The energy saving reward means that during the process of sparse vehicles forming a platoon, some vehicles that temporarily maintain the platoon are rewarded to encourage vehicles to form a platoon faster. The parameter b refers to the benefit coefficient. Both the front and rear vehicles have a certain amount of assistance to the vehicle, so that the vehicle can reduce energy consumption.

[0084] The speed efficiency bonus can be obtained by the following formula:

[0085]

[0086] Among them, v current is the current speed of the vehicle; v min is the minimum speed limit of the vehicle; v max The maximum speed limit for the vehicle.

[0087] The energy consumption of a vehicle is related to speed, mass, and front area. The speed can be changed. The greater the speed and acceleration, the greater the corresponding traction, and the more energy the vehicle consumes. Therefore, if there is no speed efficiency reward, in order to obtain a greater reward and obtain the least energy consumption, the entire model will tend to control the vehicle speed at the lowest speed as the training process proceeds, which seriously affects the efficiency of the vehicle on the road and may cause traffic congestion. The speed efficiency reward is to adapt to the problem of increased energy consumption caused by speed increase. When the speed v current When the speed is upgraded, the speed efficiency bonus received by the vehicle will become larger, allowing the vehicle to maintain a higher speed while forming a formation.

[0088] The lane change reward is obtained by the following formula:

[0089]

[0090] In the present disclosure, a penalty value is set as a representative of the energy consumption of lane change to prevent the vehicle from changing lanes without restriction.

[0091] Although the following model is helpful, there is still a possibility of collision. For example, if a vehicle in lane 0 and a vehicle in lane 2 have the motivation to change lanes to lane 1 at the same time, and the distance between them is very close, a collision will occur after the lane change. The reward R2 is for this situation. When a collision occurs, R2 = -10.

[0092] The reward obtained by the vehicles at the end of the entire formation process after the formation is formed is obtained by the following formula:

[0093]

[0094] Where, C = 300; W i is the energy consumed by the i-th vehicle during the formation process; represents the profit coefficient of the preceding vehicle of the i-th vehicle to the i-th vehicle; It represents the benefit coefficient of the following vehicle of the i-th vehicle to the i-th vehicle, and P represents the number of vehicles to be formed.

[0095] The reward obtained by the vehicles at the end of the entire formation process after forming a formation, that is, the sum of the energy saved after all the current vehicles form a fleet to gain benefits, accounts for the proportion of the total energy consumed by each vehicle before forming a formation, and then multiplied by a reward coefficient C, the value of C is 300, which is used to evaluate the formation effect. In other words, the more energy that can be saved in the final formation, the more rewards will be obtained in the end.

[0096] Figure 4 It is a block diagram of the hardware structure used in the vehicle platooning method according to one embodiment of the present disclosure.

[0097] During the model training process, there are P vehicles to be formed in the environment. Each vehicle has its own neural network model parameters, optimizer and experience pool. After starting to train a certain batch t, ​​the server will randomly select the decision models of Q vehicles for aggregation. After the aggregation is completed, it will be sent to all vehicles to update the model running on the current vehicle, and the next round of training will be carried out based on this model, and so on.

[0098] In other words, the trained first network models of Q vehicles out of the P vehicles to be formed are uploaded to the server, the server obtains an aggregate model based on the first network models of the Q vehicles, transmits the aggregate model to the vehicle, and uses the aggregate model to update the first network model stored in the vehicle.

[0099] Through this method, the communication frequency between the vehicle and the server can be greatly reduced, so that the first network model only needs to communicate with the central server for model aggregation after being updated locally a certain number of times. This greatly reduces the communication volume, improves the problem of excessive communication volume in centralized training reinforcement learning, and improves the stability of the entire system.

[0100] After the training of the first network model is completed, the vehicle information and state control information of the P vehicles to be formed are input into the trained first network model, and the vehicles are controlled to execute the output results of the first network model to realize the formation driving of the P vehicles.

[0101] The vehicle platooning method disclosed herein will be described below with reference to specific examples.

[0102] The present invention implements the vehicle platooning method in Python using PyTorch and conducts experiments on a 64-bit Linux machine. The machine has a 4-core Intel Xeon@2.80GHz CPU and an NVIDIA GeForce RTX 1080 Ti GPU. The simulation experiment platform used is the classic vehicle simulation platform SUMO.

[0103] Table 4-1 Vehicle attribute information table

[0104]

[0105] In the experiment, three lanes were set up and six cars were randomly generated. There were three possible types of cars, small, medium, and large. As shown in Table 4-1, different types of cars have different weights and front areas. There are three types of cars evenly distributed among the six cars, that is, two cars of each type. In addition, the benefits to the front and rear cars after the formation are also different. The specific parameters are shown in Table 4-2.

[0106] Table 4-2 Vehicle formation energy saving table

[0107]

[0108] The initial speed of each vehicle is randomly selected in the interval between the minimum and maximum speeds. In the vehicle energy consumption model, the rolling resistance coefficient μ is set to 0.02 and the gravity acceleration g is set to 9.81 m / s 2 , the air density ρ is set to 1.0 kg / m 3 , aerodynamic drag coefficient C d Set to 1.2.

[0109] The time step in the simulation process is set to 0.1s. After each lane change, the vehicle must keep a straight time of at least 2s in the lane after the lane change to avoid continuous lane changes. In the first 60s, the vehicle can change lanes at will to adjust the vehicle position to form a formation shape with better energy saving effect. After 60s, the lane with the most vehicles at the current moment is selected as the formation lane. Vehicles in this lane are prohibited from changing lanes, and vehicles in other lanes are only allowed to change lanes and enter this lane. When the running time exceeds 180s, the formation is considered to have failed and the current round of formation is stopped. During the entire experiment, the maximum speed of the vehicle is set to 33m / s and the minimum speed is 15m / s.

[0110] For the training setting of the entire framework, the optimizer uses the Adam optimizer, the learning rate is set to 1e-6, the eps is set to 1e-5, the federated learning aggregation batch t is set to 10, and the number of models randomly selected during aggregation is set to 4, that is, 5 training aggregations are performed once. For the parameter settings in the PPO algorithm, the discount factor Y is set to 0.99, the GAE parameter λ is set to 0.95, the cross entropy coefficient is set to 0.01, the capacity of the entire experience pool is 2048, the batch size for each training is 64, and the number of iterations is set to 6. When the round reaches about 200, the entire training curve has stabilized and the model has basically completed convergence.

[0111] The technical effects achieved in this solution are mainly in three aspects: First, in order to accurately evaluate the energy consumption problem in the process of vehicle formation, this solution models the vehicle energy consumption model and establishes a vehicle energy consumption model that can reflect the vehicle energy consumption to a certain extent. Secondly, the reinforcement learning algorithm is introduced into the process of vehicle formation, and multiple vehicles make asynchronous decisions to improve the efficiency of vehicle formation, and save overall time through carefully designed reward functions. In addition, in order to solve the problem of frequent communication in centralized training of reinforcement learning and the increase in state space as the number of intelligent agents increases, federated learning is introduced into training, which greatly reduces the communication volume of the entire system.

[0112] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0113] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0114] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A vehicle platooning method, characterized in that: include: Acquire vehicle information and state space information of each vehicle, wherein the vehicle information includes the type, longitudinal coordinate, lateral coordinate, speed and acceleration of the vehicle; the state space information includes a central area, a forward area, a rearward area, a left area, a right area, a left front area, a left rear area, a right front area and a right rear area centered on the vehicle; Inputting the vehicle information and the state space information into a first network model stored in the vehicle for decision making, and obtaining a decision result, wherein the decision result includes the probability of a behavior action that each vehicle may currently perform, wherein the behavior action includes speed and lane changing behavior, wherein the speed includes accelerating, driving at a constant speed, and decelerating; and the lane changing behavior includes changing lanes to the left, maintaining the current lane, and changing lanes to the right; Obtaining feedback for executing corresponding behavior actions according to the decision result, and training the first network model according to the decision result and the feedback for the corresponding behavior actions; wherein the feedback includes a reward R1 obtained when the formation is not completely formed, a reward R2 obtained when a collision occurs between vehicles, and a reward R3 obtained when the entire formation process ends after the vehicles form a formation; The trained first network models of Q vehicles among the P vehicles to be formed are uploaded to the server, the server obtains an aggregate model according to the first network models of the Q vehicles, transmits the aggregate model to the vehicle, and uses the aggregate model to update the first network model stored in the vehicle; After the training of the first network model is completed, the vehicle information and state control information of the P vehicles to be formed are input into the trained first network model, and the vehicles are controlled to execute the output results of the first network model to realize the formation driving of the P vehicles.

2. The vehicle platooning method according to claim 1, characterized in that: The first network model includes a mask layer, and the mask layer is used to limit behavioral actions in the decision result.

3. The vehicle platooning method according to claim 2, characterized in that: During the process of vehicle platooning, the lane with the largest number of vehicles to be platooned is selected as the platooning lane, and vehicles traveling in the platooning lane are not allowed to change lanes.

4. The vehicle platooning method according to claim 2, characterized in that: The vehicle is not allowed to change lanes again within the preset time after changing lanes.

5. The vehicle platooning method according to claim 1, characterized in that: The rewards for not fully forming a formation are obtained through the following formula: R1=c1*(R power-cost +R power-save )+c2*R v-eff +c3*R lane-cost Where c1=0.4, c2=0.1, c3=0.3; R power-cost is the reward for energy consumption in the running state; R power-save is the reward for energy saving; R v-eff is a reward for speed efficiency; R lane-cost It's a lane change reward.

6. The vehicle platooning method according to claim 5, characterized in that: The reward for energy consumption in the running state is obtained by the following formula: Where W v,a Represents the energy consumed by the vehicle per unit time, W v-max It represents the energy consumed per unit time when the vehicle travels at maximum speed.

7. The vehicle platooning method according to claim 5, characterized in that: The speed efficiency bonus can be obtained by the following formula: Among them, v current is the current speed of the vehicle; v min is the minimum speed limit of the vehicle; v max The maximum speed limit for the vehicle.

8. The vehicle platooning method according to claim 5, characterized in that: The lane change reward is obtained by the following formula:

9. The vehicle platooning method according to claim 1, characterized in that: In the bonus obtained when a collision occurs between vehicles, when a collision occurs between vehicles, R2 = -10.

10. The vehicle platooning method according to claim 1, characterized in that: The reward obtained by the vehicles at the end of the entire formation process after the formation is formed is obtained by the following formula: Where, C = 300; W i is the energy consumed by the i-th vehicle during the formation process; represents the profit coefficient of the preceding vehicle of the i-th vehicle to the i-th vehicle; It represents the benefit coefficient of the following vehicle of the i-th vehicle to the i-th vehicle, and P represents the number of vehicles to be formed.

Images