A cooperative control method for intelligent connected vehicles in hybrid traffic scenarios based on self-organized grouping of physical information

Abstract

This invention discloses a cooperative control method for intelligent connected vehicles (ICVs) in hybrid traffic scenarios based on self-organized grouping of physical information. The method includes acquiring historical state information and actions of each vehicle in the road network, thereby obtaining observation information; sampling based on the observation information; dividing communication groups based on the sampling results; constructing an ICV cooperative model; training the ICV cooperative model based on the observation information and actions of each ICV after the communication groups are formed; acquiring the observation information of each vehicle in the road network at the current moment, grouping it, and inputting it into the trained ICV cooperative model to output the next action; the ICV then executes this action to optimize the road network's operating efficiency. This invention solves the technical problems of traditional methods' dependence on infrastructure and the limitations of existing ICV control algorithms in communication and scalability, giving it good practical application potential.

Description

Technical Field

[0001] This invention belongs to the field of vehicle cooperative control, and specifically relates to a cooperative control method for intelligent connected vehicles in hybrid traffic scenarios based on self-organized grouping of physical information. Background Technology

[0002] In modern transportation systems, highway bottlenecks are a major cause of traffic congestion, resulting in severe socioeconomic losses and environmental pollution. To alleviate bottleneck congestion, active traffic management is widely used to regulate traffic flow. These methods adjust traffic flow through roadside infrastructure to ensure that demand entering the bottleneck area does not exceed its capacity. With the development of vehicle-to-everything (V2X) and autonomous driving technologies, intelligent connected vehicles (ICVs) offer a completely new approach to solving traffic congestion. ICVs can influence the trajectories of surrounding vehicles through advanced communication and autonomous decision-making capabilities, thereby optimizing the overall traffic flow distribution and improving road network efficiency. Against this backdrop, reinforcement learning, due to its powerful ability to handle complex decision-making problems, has become an important tool in the research of control algorithms for intelligent connected vehicles.

[0003] However, existing traffic bottleneck management technologies and intelligent connected vehicle control methods still have the following problems: (1) Traditional active traffic control strategies rely heavily on fixed roadside infrastructure for traffic status detection and issuing management instructions, making it difficult to cope with non-fixed bottlenecks caused by sudden events such as traffic accidents or temporary construction zones. (2) Human driver behavior is highly random and unpredictable, and the uncertainty of compliance greatly weakens the actual effect of active control strategies. (3) Most existing intelligent connected vehicle control research scenarios are oversimplified, for example, only tested in single-lane or low-vehicle environments, and their research conclusions are difficult to generalize to real road networks with a large number of lanes and vehicles. (4) Existing multi-agent reinforcement learning algorithms face huge challenges in communication and computation when applied to large-scale intelligent connected vehicle cooperative control. Some methods assume that communication is completely reliable and ignore the bandwidth limitations and instabilities in real networks; while other graph neural network-based methods, although able to integrate global information, have extremely high requirements for real-time and stability of communication, making it difficult to extend to the real world with a large number of vehicles, and facing problems such as information transmission delay and algorithm convergence difficulties. Summary of the Invention

[0004] To address the problems existing in the background technology, this invention provides a cooperative control method for intelligent connected vehicles in hybrid traffic scenarios based on self-organized grouping of physical information. This method solves the technical problems of dependence on infrastructure in traditional methods and the limitations of existing intelligent connected vehicle control algorithms in terms of communication and scalability, thus giving this invention good potential for practical application.

[0005] The technical solution adopted in this invention includes:

[0006] S1. Obtain the historical state information of each car in the road network and the actions of each intelligent connected vehicle. Each intelligent connected vehicle obtains observation information based on the state information of all cars. Based on all observation information, perform group leader sampling and follower sampling on the intelligent connected vehicles respectively, and divide the communication groups according to the sampling results.

[0007] S2. Construct an intelligent connected vehicle collaborative model. Based on the observation information and actions of each intelligent connected vehicle after dividing the communication groups in step S1, train the intelligent connected vehicle collaborative model to obtain the trained intelligent connected vehicle collaborative model.

[0008] S3. Obtain the observation information of each car in the road network at the current time. Based on all the observation information, perform group leader sampling and follower sampling on the intelligent connected vehicles in the car. Divide the communication groups according to the sampling results. Input the observation information of each intelligent connected vehicle after dividing the communication groups into the trained intelligent connected vehicle cooperative model to output the next action set. Each intelligent connected vehicle executes the corresponding action in the action set.

[0009] The status information includes its own speed and its own position; the observation information includes its own speed, its own position, and the speed and position of other vehicles within the perception range; the action is one of the following: rapid acceleration, acceleration, constant speed, deceleration, rapid deceleration, left turn, and right turn.

[0010] Step S1 specifically involves:

[0011] S11. Obtain the historical state information of each car in the road network and the action space of intelligent connected vehicles; each intelligent connected vehicle obtains observation information based on the state information of all cars; the cars in the road network are divided into intelligent connected vehicles and human-driven vehicles.

[0012] S12. Obtain the vehicle density of the road segment to which each intelligent connected vehicle belongs. Based on the changes in vehicle density and its own speed, use the leader probability generation method to obtain the probability of each intelligent connected vehicle acting as the group leader. Then, sample several group leaders based on the probability of each intelligent connected vehicle acting as the group leader.

[0013] S13. For intelligent connected vehicles that are not selected as leaders, obtain the distance between each intelligent connected vehicle and the nearest intelligent connected vehicle that is the group leader. Based on the obtained distance, use the follower probability generation method to obtain the probability of each intelligent connected vehicle that is not selected as a leader being a follower of the nearest group leader. Then, sample several followers around each group leader based on the probability of each intelligent connected vehicle being a follower.

[0014] S14. Each group leader and all its corresponding followers form a communication group; if a connected vehicle is neither sampled as a group leader nor as a follower, then the connected vehicle itself becomes a leader and forms an independent communication group.

[0015] Each vehicle determines other vehicles within its perception range using the following formula:

[0016] μ(i,j)=1, if φ i =1 and φ j =1

[0017] μ(i,j)=1, if φ i =1 and φ j =0 and [(x j t -x i t ) 2 +(y j t -y i t ) 2 ] 0.5 ≤200m

[0018] μ(i,j)=1, if φ i =0 and φ j =0 and [(x j t -x i t ) 2 +(y j t -y i t ) 2 ] 0.5 ≤50m

[0019] μ(i,j)=0,others

[0020] Where i and j both represent indices; μ(i,j)=1 indicates that the j-th car is within the perception range of the i-th car; μ(i,j)=0 indicates that the j-th car is not within the perception range of the i-th car; φ i and φ j φ represents the type of the i-th and j-th cars respectively; i =1 and φ j =1 indicates that the i-th and j-th cars are intelligent connected vehicles; φ i =0 and φ j =0 indicates that the i-th and j-th cars are driven by humans, respectively; x i tand y i t Let x represent the lateral and longitudinal coordinates of the i-th car, respectively; i t and y j t Let x and y represent the horizontal and vertical coordinates of the j-th car, respectively.

[0021] The probability of the group leader is obtained according to the following formula for leader probability generation:

[0022] p i =sigmoid(n i t / v i t +ln(|v i t -v i t-1 |))

[0023] Where, p i Let n represent the probability that the i-th intelligent connected vehicle is the group leader; i t V represents the vehicle density of the i-th intelligent connected vehicle; i t v represents the speed of the i-th intelligent connected vehicle at time t; i t-1 Let represent the speed of the i-th intelligent connected vehicle at time t-1; ln() represents the natural logarithm to the base e.

[0024] The probability of each intelligent connected vehicle that was not selected as a leader becoming a follower is obtained according to the following follower probability generation method:

[0025] d i =sigmoid(ln(|l i -l leader |))

[0026] Where, d i This represents the probability that the i-th intelligent connected vehicle is not selected as a leader and instead becomes a follower; i This indicates the position of the i-th intelligent connected vehicle that was not selected as a leader. leader This indicates the position of the i-th intelligent connected vehicle that is the most recent intelligent connected vehicle that acts as the group leader.

[0027] The intelligent connected vehicle cooperative model includes a communication message processing module and a reinforcement learning module. A reinforcement learning experience replay pool is constructed based on the observation information and action space of the intelligent connected vehicles. The observation information of the intelligent connected vehicles after dividing the communication groups at historical moments, along with the reinforcement learning experience replay pool, is processed by the communication message processing module to obtain the latest communication message for each intelligent connected vehicle. The observation information and the latest communication message of each intelligent connected vehicle are then input into the reinforcement learning module to obtain the action result. The intelligent connected vehicle cooperative model is continuously optimized and trained based on the loss function of the communication message processing module and the loss function of the reinforcement learning module.

[0028] The communication message processing module processes the latest communication message according to the following steps:

[0029] D1. The observation information of each intelligent connected vehicle and the future information sequence in the reinforcement learning experience replay pool are input together into the Gaussian random number generator to generate compressed communication messages.

[0030] D2. In the preset communication steps, the leader of each communication group takes the average value of the compressed communication messages of all intelligent connected vehicles in the communication group to obtain the average communication message, and uses the average communication message as the latest communication message of each intelligent connected vehicle in the communication group.

[0031] The reinforcement learning module is optimized and trained according to the following steps:

[0032] Each intelligent connected vehicle's own observation information and latest communication messages are input into a deep learning-based policy network. The Q-value of each action in the discrete action space is calculated based on the reward function. The Q-values ​​of all intelligent connected vehicles are input into the QMIX network to generate a joint action value function. The parameters of the entire QMIX network are optimized and trained by minimizing the mean squared error loss function between the joint action value function and the target value.

[0033] The reward function is set according to the following formula:

[0034] r t =α v r t v +α vcav r t vcav +α s r t s

[0035] r t v =max{min{(v t all -v min ) / (vmax -v min ),1},0}

[0036] r t vcav =max{min{(v t cav -v min ) / (v max -v min ),1},0}

[0037] r t s =-1, a collision occurs; r t s =0, no collision

[0038] Where, r t Let r be the reward function value at time t; t v r t vcav and r t s These are the efficiency reward function for the road network, the efficiency reward function for intelligent connected vehicles, and the collision penalty function, respectively; v t all v is the average speed of all cars in the road network. t cav v is the average speed of all intelligent connected vehicles in the road network. min and v max These are the minimum and maximum speed limits for the road network, respectively; α v α vcav and α s All are weighting coefficients.

[0039] The beneficial effects of this invention are:

[0040] 1. This invention can utilize the physical characteristics of traffic flow to enable intelligent connected vehicles facing similar traffic conditions to spontaneously form communication groups, thereby achieving efficient and targeted collaborative communication and control.

[0041] 2. The present invention develops a communication message processing module based on the principle of "maximizing mutual information". This module extracts key content from the original perceived information and filters redundant data, thereby compressing the amount of communication data and reducing the transmission frequency, thereby alleviating network bandwidth and stability problems.

[0042] 3. The traffic network efficiency controlled by the method of this invention is significantly better than the benchmark model and other scenarios. Compared with a traffic network composed entirely of human drivers, the average speed of the network can be increased by up to 27%. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating the self-organization of physical information in the reasoning process of the method of the present invention.

[0044] Figure 2 This is a schematic diagram of the overall architecture of the method of the present invention.

[0045] Figure 3 This is a simulated road network topology diagram used in the embodiments of the present invention.

[0046] Figure 4 This is a diagram illustrating the control effect of the QMIX-A network in an embodiment of the present invention.

[0047] Figure 5 This is a diagram illustrating the control effect of the REFIL network in an embodiment of the present invention.

[0048] Figure 6 This is a diagram illustrating the control effect of the GCQ network in an embodiment of the present invention.

[0049] Figure 7 This is a diagram illustrating the control effect of the method of the present invention in an embodiment of the present invention. Detailed Implementation

[0050] The present invention will now be described in more detail with reference to the accompanying drawings and embodiments. However, the present invention is not limited thereto. For those skilled in the art, several improvements and modifications can be made without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention. Contents not described in detail in this specification are prior art known to those skilled in the art.

[0051] like Figure 2 and Figure 3 As shown in this embodiment, the intelligent connected vehicle cooperative control method for mixed traffic scenarios includes the following steps:

[0052] S1. The vehicle obtains the historical state information of each car in the road network and the action space of the intelligent connected vehicles in the vehicle through its own sensors. Each intelligent connected vehicle obtains observation information based on the state information of all cars. Based on all observation information, the intelligent connected vehicles are sampled for group leader and follower respectively. The communication groups are divided according to the sampling results.

[0053] S11. The vehicle acquires the historical state information of each car in the road network and the action space of the intelligent connected vehicles in the road network through its own sensors; each intelligent connected vehicle acquires observation information based on the state information of all cars; the cars in the road network are divided into intelligent connected vehicles and human-driven vehicles; specifically, intelligent connected vehicles are cars that are interconnected by accessing the same network; human-driven vehicles are cars that are not connected to the network and cannot rely on the network for collaborative control, but can only rely on the human to regulate the state of the car.

[0054] State information includes its own speed and position; observation information includes its own speed, its own position, and the speed and position of every other car within its perception range (excluding itself); actions include one of the following: rapid acceleration, acceleration, constant speed, deceleration, rapid deceleration, left turn, and right turn. The action space is: {rapid acceleration, acceleration, constant speed, deceleration, rapid deceleration, left turn, right turn}.

[0055] Since intelligent connected vehicles and human-driven vehicles have different perception ranges, there is a need for standards with clearly defined perception ranges.

[0056] Each vehicle determines other vehicles within its perception range using the following formula:

[0057] μ(i,j)=1, if φ i =1 and φ j =1

[0058] μ(i,j)=1, if φ i =1 and φ j =0 and [(x j t -x i t ) 2 +(y j t -y i t ) 2 ] 0.5 ≤200m

[0059] μ(i,j)=1, if φ i =0 and φ j =0 and [(x j t -x i t ) 2 +(y j t -y i t ) 2 ] 0.5 ≤50m

[0060] μ(i,j)=0,others

[0061] Where i and j both represent indices; μ(i,j)=1 indicates that the j-th car is within the perception range of the i-th car; μ(i,j)=0 indicates that the j-th car is not within the perception range of the i-th car; φ i and φ j φ represents the type of the i-th car and the j-th car, respectively; i=1 indicates that the i-th car is an intelligent connected vehicle; φ i =0 indicates that the i-th car is driven by a human; φ j =1 indicates that the j-th car is an intelligent connected vehicle; φ j =0 indicates that the j-th car is driven by a human; x i t and y i t Let x represent the lateral and longitudinal coordinates of the i-th car, respectively; j t and y jt Let represent the lateral and longitudinal coordinates of the j-th car, respectively. Here, t represents time, and represents time t.

[0062] S12. Obtain the vehicle density of the road segment to which each intelligent connected vehicle belongs by sensors deployed on the road. Based on the changes in vehicle density and its own speed, use the leader probability generation method to obtain the probability of each intelligent connected vehicle acting as the group leader. Sample several group leaders based on the probability of each intelligent connected vehicle acting as the group leader.

[0063] The probability of a group leader is obtained using the following formula for leader probability generation:

[0064] p i =sigmoid(n i t / v i t +ln(|v i t -v i t-1 |))

[0065] Where i represents the index; p i Let n represent the probability that the i-th intelligent connected vehicle is the group leader; i t V represents the vehicle density of the i-th intelligent connected vehicle at time t; i t v represents the speed of the i-th intelligent connected vehicle at time t; i t-1 Let represent the speed of the i-th intelligent connected vehicle at time t-1; ln() represents the natural logarithm to the base e; sigmoid() represents the activation function.

[0066] S13. For intelligent connected vehicles that are not selected as leaders, obtain the distance between each intelligent connected vehicle and the nearest intelligent connected vehicle that is the group leader. Based on the obtained distance, use the follower probability generation method to obtain the probability that each intelligent connected vehicle that is not selected as a leader is a follower of the nearest group leader. Then, sample several followers around each group leader based on the probability that each intelligent connected vehicle that is not selected as a leader is a follower.

[0067] The probability of each intelligent connected vehicle that was not selected as a leader becoming a follower is obtained using the following follower probability generation method:

[0068] d i =sigmoid(ln(|l i -l leader |))

[0069] Where i represents the index; d i This represents the probability that the i-th intelligent connected vehicle is not selected as a leader and instead becomes a follower; i This indicates the position of the i-th intelligent connected vehicle that was not selected as a leader. leader This indicates the position of the i-th intelligent connected vehicle that was not selected as the leader, and the position of the intelligent connected vehicle that was most recently selected as the group leader.

[0070] S14. Each group leader and all its corresponding followers form a communication group; if a connected vehicle is neither sampled as a group leader nor as a follower, then the connected vehicle itself becomes a leader and forms an independent communication group.

[0071] S2. Construct an intelligent connected vehicle cooperative model. Train the intelligent connected vehicle cooperative model based on the observation information and action space of each intelligent connected vehicle after dividing the communication groups in step S1, and obtain the trained intelligent connected vehicle cooperative model.

[0072] The intelligent connected vehicle cooperative model includes a communication message processing module and a reinforcement learning module. A reinforcement learning experience replay pool is constructed based on the observation information and action space of the intelligent connected vehicles. The communication message processing module processes the observation information of the intelligent connected vehicles after dividing the communication groups at historical moments, along with the reinforcement learning experience replay pool, to obtain the latest communication messages for each intelligent connected vehicle. Each intelligent connected vehicle's own observation information and latest communication messages are then input into the reinforcement learning module to obtain action results. The intelligent connected vehicle cooperative model is continuously optimized and trained based on the loss functions of the communication message processing module and the reinforcement learning module.

[0073] The communication message processing module processes the latest communication message according to the following steps:

[0074] D1. The observation information of each intelligent connected vehicle and the future information sequence in the reinforcement learning experience replay pool are input together into the Gaussian random number generator to generate compressed communication messages.

[0075] The future information in the reinforcement learning experience replay pool is the observation information at the current time t corresponding to the reinforcement learning experience replay pool. i t and action a i t Begin by sequentially selecting the observation information sequence for the next T-1 time steps (o i t+1 ,o i t+2 ,…,o i t+T-1 ) and the corresponding action sequence (a i t+1 ,a i t+2 ,…,a i t+T-1 The selected observation information and corresponding actions form the future information sequence τ. i t =(o i t ,a i t ,…,o i t+T-1 ,a i t+T-1 ).

[0076] The reinforcement learning experience replay pool is a buffer that stores the historical interaction experiences of intelligent connected vehicles for subsequent training and learning. During training, a certain amount of experience data is randomly sampled from the experience replay pool to update the policy and model parameters of the collaborative model.

[0077] D2. In the preset communication steps, the leader of each communication group takes the average value of the compressed communication messages of all intelligent connected vehicles in the communication group to obtain the average communication message, and broadcasts the average communication message as the latest communication message of each intelligent connected vehicle in the communication group.

[0078] The reinforcement learning module is optimized and trained according to the following steps:

[0079] Each intelligent connected vehicle's own observation information and latest communication messages are input into a deep learning-based policy network. The Q-value of each action in the discrete action space is calculated based on the reward function. The Q-values ​​of all intelligent connected vehicles are input into the QMIX network to generate a joint action value function. The parameters of the entire QMIX network are optimized and trained by minimizing the mean squared error loss function between the joint action value function and the target value.

[0080] The reward function is set according to the following formula:

[0081] r t =α v r t v +α vcav r t vcav +α s r t s

[0082] r t v =max{min{(v t all -v min ) / (v max -v min ),1},0}

[0083] r t vcav =max{min{(v t cav -v min ) / (v max -v min ),1},0}

[0084] r t s =-1, a collision occurs; r t s =0, no collision

[0085] Where, r t Let r be the reward function value at time t; t v r t vcav and r t s These are the efficiency reward function for the road network, the efficiency reward function for intelligent connected vehicles, and the collision penalty function, respectively; v t all v is the average speed of all cars in the road network. t cav v is the average speed of all intelligent connected vehicles in the road network. min and v max These are the minimum and maximum speed limits for the road network, respectively; α v α vcav and α s All are weighting coefficients.

[0086] The loss function and target value of the QMIX network are set according to the following formula:

[0087] L Q (θ)=E (ot,ζt,at)~D [(y tot -Q total (o t ,ζ t ,a t ,θ)) 2 ]

[0088] y tot =r t +γmax at+1 Q total (o t+1 ,ζ t+1 ,a t+1 ,θ ^ )

[0089] Among them, L Q (θ) represents the loss function; t ζ t and a t These represent all observation information, all compressed communication messages, and all actions at time t, respectively; D represents the experience playback buffer; E (ot,ζt,at)~D Indicates expectation; y tot Q represents the target value; total Represents the value function of joint actions; o t+1 ζ t+1 and a t+1 These represent all observation information, all compressed communication messages, and all actions at time t+1, respectively; γ represents the discount factor; r t Represents the reward function value at time t; θ and θ' ^ These represent the parameters of the main network and the target network, respectively.

[0090] The parameters in the QMIX network are iteratively optimized using the following stochastic gradient descent algorithm:

[0091] θ'←θ+η▽ θ L Q (θ)

[0092] Φ'←Φ+η(▽ Φ L P (Φ)+ ▽ Φ L CEB (Φ))

[0093] ψ'←ψ+η(▽ ψ L P (ψ)+ ▽ ψ L CEB (ψ))

[0094] Where θ, Φ, and ψ represent the parameters before the update; θ', Φ', and ψ' represent the parameters after the update; ▽ θ L Q (θ) represents the loss function L Q (θ) Gradient of parameter θ; ▽ Φ L P (Φ) represents the loss function L P (Φ) Gradient of Φ; ▽ Φ L CEB (Φ) represents the loss function L CEB (Φ) Gradient of Φ; ▽ ψ L P (ψ) represents the loss function L P (ψ) gradient with respect to ψ; ▽ ψ L CEB (ψ) represents the loss function L CEB (ψ) represents the gradient of ψ; η represents the learning rate.

[0095] Each intelligent connected vehicle generates a compressed communication message based on its observation information and the future information sequence in the reinforcement learning experience replay pool sequence, using a Gaussian random variable generator parameterized by ψ. This compressed communication message is designed to reflect the true future observation information with a small amount of information.

[0096] The Gaussian random number generator is trained by optimizing a loss function that combines the evidence lower bound (ELBO) and the conditional entropy bottleneck (CEB) to maximize the mutual information between compressed communication messages and future traffic state information.

[0097] Among them, Gaussian random number generator f ψ The loss function for () is set according to the following formula:

[0098] L=L P (Φ,ψ)+L CEB (Φ,ψ)

[0099] L P (Φ,ψ)=-E oi_t,ζi_t,ai_t [logq Φ (ζ i t |τ i t ,o i t )]-H(ζ i t |o i t )

[0100] L CEB (Φ,ψ)=E oi_t,τi_t,ai_t [log(fψ (ζ i t |o i t ) / q Φ (ζ i t |τ i t ,o i t ))-log(Kq Φ (ζ i t |τ i t ,o i t ) / Σ j=1 K q Φ (ζ j t |τ j t ,o j t ))]

[0101] Where i and j are both indices; K represents the total number of intelligent connected vehicles in the batch; L P (Φ,ψ) represents the evidence lower bound loss function, used to maximize the compression of communication messages and future traffic state information; L CEB (Φ,ψ) represents the conditional entropy bottleneck loss function, used to reduce the impact of irrelevant information in the current state on communication information; q Φ () denotes the variational estimator parameterized by Φ; H(ζ) i t |o i t ) represents observation information o i t Compressed communication message ζ i t Information entropy between; f ψ () represents a Gaussian random number generator; ζ i t and ζ j t These represent the compressed communication messages of the i-th and j-th intelligent connected vehicles, respectively; τ i t and τ j t Let o represent the future information sequences of the i-th and j-th intelligent connected vehicles, respectively; i t and o j t These represent the observation information of the i-th and j-th intelligent connected vehicles, respectively.

[0102] The loss function of the Gaussian random number generator will be optimized simultaneously with the loss function of the reinforcement learning module during the iteration.

[0103] In this embodiment, the network parameters are set as follows: the batch size during training is 32, the optimizer is RMSProp, and the learning rate is 0.0005. Regarding the network structure, the hidden layer dimension and the RNN hidden layer dimension are both 64, the mix embedding dimension is 32, and the hypernet embedding dimension is 128. Other key parameters include a buffer of size 1000, a discount factor (Gamma) of 0.99, and a target network update interval of 200 time steps.

[0104] S3. Deploy the trained intelligent connected vehicle cooperative model in the intelligent connected vehicle centralized control and processing center. Obtain the observation information of each vehicle in the road network at the current time through the vehicle's own sensors. Based on all the observation information, perform group leader sampling and follower sampling on the intelligent connected vehicles in the vehicle. Divide the communication groups according to the sampling results. Input the observation information of each intelligent connected vehicle after dividing the communication groups into the trained intelligent connected vehicle cooperative model, output the next action set and transmit it to each intelligent connected vehicle. Each intelligent connected vehicle executes the corresponding action in the action set.

[0105] like Figure 1 The diagram shows the physical information self-organization process of the reasoning process of this invention. First, traffic conditions are perceived, then groups are formed into convoys, an organizer (group leader) is determined, then actions are executed, and finally the convoys are dispersed, and traffic conditions are perceived again. This process is repeated to continuously coordinate and control all intelligent connected vehicles in the road network. Figure 1 G1, G2, and G3 in the above are all communication groups.

[0106] like Figure 4 , Figure 5 , Figure 6 and Figure 7 As shown, to demonstrate the beneficial effects of the present invention, this embodiment obtains the observation information of each intelligent connected vehicle based on the historical observation information of each vehicle in the road network and the action space of each intelligent connected vehicle. Based on the observation information and actions of each intelligent connected vehicle, it directly employs existing technologies such as QMIX-A, REFIL, and GCQ networks for cooperative control through reinforcement learning. The comparison results of directly using QMIX-A, REFIL, GCQ networks, the method of the present invention, and no control are shown in the table below:

[0107]

[0108] Experimental results show that all reinforcement learning algorithms converged under the experimental conditions. Considering factors such as vehicle operating efficiency and safety, the method of this invention performs well under different experimental penetration rates, with convergence speed and results surpassing classic multi-agent reinforcement learning algorithms such as QMIX-A, REFIL, and GCQ. The control effect of the algorithm improves accordingly with increasing penetration rate.

[0109] The method of this invention solves the technical problems of traditional methods' dependence on infrastructure and the limitations of existing intelligent connected vehicle control algorithms in terms of communication and scalability. The traffic network efficiency under the control of the method of this invention is significantly better than the benchmark model and other scenarios. Compared with a traffic network composed entirely of human drivers, the average speed of the road network can be increased by up to 27%, making this invention have better potential for practical application.

[0110] The above-described embodiments are merely preferred embodiments provided to fully illustrate the present invention, and the scope of protection of the present invention is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on the present invention are all within the scope of protection of the present invention. The scope of protection of the present invention is defined by the claims.

Claims

1. A method for intelligent connected vehicle cooperative control in a mixed traffic scenario based on self-organization grouping of physical information, characterized in that, The method comprises the following steps: S1, obtaining state information of each vehicle in the historical road network and actions of each intelligent connected vehicle in the vehicle, each intelligent connected vehicle obtaining observation information according to the state information of all vehicles, performing group leader sampling and follower sampling on the intelligent connected vehicle according to all observation information, and dividing a communication group according to a sampling result; S2, constructing an intelligent connected vehicle cooperation model, training the intelligent connected vehicle cooperation model according to observation information and actions of each intelligent connected vehicle after the communication group is divided in step S1, and obtaining a trained intelligent connected vehicle cooperation model; The intelligent connected vehicle cooperation model comprises a communication message processing module and a reinforcement learning module; a reinforcement learning experience replay pool is constructed according to observation information and an action space of the intelligent connected vehicle, the observation information of each intelligent connected vehicle and a future information sequence in the reinforcement learning experience replay pool are input into a Gaussian random number generator together to generate compressed communication messages; In a preset communication step, an average communication message is obtained by averaging the compressed communication messages of all intelligent connected vehicles in each communication group, and the average communication message is used as the latest communication message of each intelligent connected vehicle in the communication group; the observation information of each intelligent connected vehicle and the latest communication message are input into the reinforcement learning module to obtain an action result, and the intelligent connected vehicle cooperation model is continuously optimized and trained according to a loss function of the communication message processing module and a loss function of the reinforcement learning module; S3, obtaining observation information of each vehicle at a current time in the road network, performing group leader sampling and follower sampling on the intelligent connected vehicles in the vehicle according to all observation information, dividing a communication group according to a sampling result, inputting the observation information of each intelligent connected vehicle after the communication group is divided into the trained intelligent connected vehicle cooperation model to output a next action, and each intelligent connected vehicle performing a corresponding action.

2. The intelligent connected vehicle cooperation control method based on physical information self-organization grouping of a hybrid traffic scene according to claim 1, characterized in that: The state information comprises a speed and a position; the observation information comprises the speed, the position, speeds and positions of other vehicles in a perception range; and the action is one of rapid acceleration, acceleration, constant speed, deceleration, rapid deceleration, left turn and right turn.

3. The intelligent vehicle connected cooperative control method based on physical information self-organizing packet hybrid traffic scene according to claim 1, characterized in that, The step S1 is specifically: S11, obtaining state information of each vehicle in the historical road network and an action space of the intelligent connected vehicle in the vehicle; each intelligent connected vehicle obtains observation information according to the state information of all vehicles; and the vehicles in the road network are divided into intelligent connected vehicles and human-driven vehicles; S12, obtaining a vehicle density of a road section to which each intelligent connected vehicle belongs, generating a probability of each intelligent connected vehicle as a group leader by using a leader probability generation method according to a change in the vehicle density and the speed, and sampling a plurality of group leaders according to the probability of each intelligent connected vehicle as the group leader. S13, for the intelligent connected vehicles not selected as leaders, obtaining the distance between each intelligent connected vehicle and the nearest intelligent connected vehicle as a group leader, and obtaining the probability of each intelligent connected vehicle not selected as a leader as a follower of the nearest group leader according to the obtained distance using a follower probability generation method, and sampling a number of followers around each group leader according to the probability of each intelligent connected vehicle as a follower; S14, each group leader and all corresponding followers form a communication group; if an intelligent connected vehicle is neither sampled as a group leader nor as a follower, the intelligent connected vehicle itself becomes a leader and forms an independent communication group.

4. The hybrid traffic scene intelligent connected vehicle cooperative control method based on physical information self-organizing grouping according to claim 3, characterized in that: Each vehicle determines other vehicles within the perception range according to the following formula: μ(i,j) = 1, if φ i = 1 and φ j = 1 μ(i,j) = 1, if φ i = 1 and φ j = 0 and [(x j t - x i t ) 2 + (y j t - y i t ) 2 ] 0.5 ≤ 200 m μ(i,j) = 1, if φ i = 0 and φ j = 0 and [(x j t - x i t ) 2 + y j t - y i t ) 2 ] 0.5 ≤ 50 m μ(i,j)=0,others where i and j represent indices; μ(i, j) = 1 means the jth vehicle is within the perception range of the ith vehicle; μ(i, j) = 0 means the jth vehicle is not within the perception range of the ith vehicle; φ i and φ j represent the types of the ith and jth vehicles, respectively; φ i = 1 and φ j = 1 mean the ith and jth vehicles are intelligent connected vehicles, respectively; φ i = 0 and φ j = 0 mean the ith and jth vehicles are human-driven vehicles, respectively; x i t and y i t represent the lateral and longitudinal coordinates of the ith vehicle, respectively; x i t and y j t represent the lateral and longitudinal coordinates of the jth vehicle, respectively.

5. The hybrid traffic scene intelligent connected vehicle cooperative control method based on physical information self-organizing grouping according to claim 3, characterized in that: The probability of the group leader is obtained according to the following leader probability generation method formula: p i = sigmoid(n i t / v i t + ln(|v i t -v i t-1 |)) where p i represents the probability of the ith intelligent connected vehicle as a group leader; n i t represents the vehicle density of the road section to which the ith intelligent connected vehicle belongs; v i t represents the speed of the ith intelligent connected vehicle at time t; v i t-1 represents the speed of the ith intelligent connected vehicle at time t-1; ln() represents the natural logarithm with base e.

6. The hybrid traffic scene intelligent connected vehicle cooperative control method based on physical information self-organizing grouping according to claim 3, characterized in that: The probability of each intelligent connected vehicle not selected as a leader as a follower is obtained according to the following follower probability generation method: d i =sigmoid(ln(|l i -l leader |)) wherein d i represents the probability of not being selected as a follower of the i-th intelligent connected vehicle among the leaders; l i represents the position of the i-th intelligent connected vehicle among the leaders that has not been selected, l leader represents the position of the most recent intelligent connected vehicle that served as a group leader for the i-th intelligent connected vehicle.

7. The method of claim 1, wherein the physical information self-organizing packet-based hybrid traffic scenario intelligent connected vehicle cooperative control method is characterized by, The reinforcement learning module is optimized and trained according to the following steps: The observation information and the latest communication message of each intelligent connected vehicle are input into the deep learning-based policy network, the Q value of each action in the discrete action space is calculated according to the reward function, the Q values of all intelligent connected vehicles are input into the QMIX network to generate a joint action value function, and the parameters of the entire QMIX network are optimized and trained by minimizing the mean square error loss function between the joint action value function and the target value.

8. The hybrid traffic scene intelligent connected vehicle cooperative control method based on physical information self-organizing grouping according to claim 7, characterized in that: The reward function is set according to the following formula: r t =α v r t v +α vcav r t vcav +α s r t s r t v =max{min{(v t all -v min ) / (v max -v min ),1},0} r t vcav =max{min{(v t cav -v min ) / (v max -v min ),1},0} r t s = -1, collision occurs; r t s =0, no collision wherein r t is the reward function value at time t; r t v , r t vcav , and r t s are the efficiency reward function of the road network, the efficiency reward function of the intelligent connected vehicle, and the collision penalty function, respectively; v t all is the average speed of all vehicles in the road network; v t cav is the average speed of all intelligent connected vehicles in the road network; v min and v max are the minimum and maximum values of the speed limit of the road network, respectively; a v , a vcav , and a s are weight coefficients.

Images