Robust QMIX-based automatic driving multi-vehicle cooperative control method and system

Abstract

The application discloses a kind of automatic driving multi-vehicle coordination control method and system based on robust QMIX, belong to vehicle networking and automatic driving field, including: by cluster graph representation module, the local graph structure of description in cluster vehicle interaction and the global graph structure of inter-cluster interaction, to represent the interaction relationship inside and outside vehicle cluster;Through the noise perception mechanism of robust strategy learning module, dynamically evaluate the environmental disturbance intensity, adopt the weighted QMIX framework to adjust the value function weight, and combine the robust loss function to optimize strategy;Through cross-cluster reward coordination module, according to the reward distribution difference between clusters, dynamically reduce the reward value of high-reward cluster to balance global optimization;Through central dispatching system, cooperative decision-making instruction is distributed to each automatic driving vehicle for execution.The application can realize long-term stable strategy optimization in complex traffic system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of Internet of Vehicles and automatic driving, and particularly relates to an automatic driving multi-vehicle cooperative control method and system based on robust QMIX. BACKGROUND

[0002] In practical applications such as automatic driving, the uncertainty of the environment, such as perception errors, communication interruptions and state loss, greatly interferes with the strategy learning and execution stability of unmanned vehicles. As a typical unmanned vehicle reinforcement learning algorithm, QMIX (Q-Mixing networks) effectively alleviates the training difficulties caused by non-stationarity and information asymmetry by introducing an additive value function decomposition structure. This method represents the global joint Q value as a nonlinear combination of individual local Q values under the framework of centralized training and distributed execution, ensuring the attainability of global optimality in the strategy learning process. At the same time, QMIX uses a hybrid network structure to fuse the decision-making contributions of different unmanned vehicles, providing a stable and efficient optimization path for strategy cooperation in heterogeneous systems.

[0003] However, existing unmanned vehicle reinforcement learning methods have difficulty effectively handling strategy bias caused by information asymmetry and observation uncertainty in heterogeneous environments, especially in mixed traffic scenarios with frequent noise disturbances, which can lead to fluctuations in decision-making performance and a decline in system robustness. QMIX underestimates the value of some joint actions in some non-monotonic tasks, and lacks an effective adaptive update strategy when facing environments with high noise, which can easily lead to over-learning on high-noise samples, affecting the convergence performance of the model. In addition, existing methods have deficiencies in reward coordination between clusters, and some clusters will make self-interested behaviors to maximize their cumulative rewards, which will hinder other clusters from making lane-changing actions, affecting the cooperative efficiency and learning effect of the entire system. SUMMARY

[0004] To solve the above technical problems, the application provides an automatic driving multi-vehicle cooperative control method and system based on robust QMIX to solve the problems existing in the prior art.

[0005] To achieve the above purpose, in a first aspect, the application provides an automatic driving multi-vehicle cooperative control method based on robust QMIX, comprising:

[0006] The cluster graph representation module describes the local graph structure of vehicle interaction within the cluster and the global graph structure of interaction between clusters to represent the interaction relationship within and outside the vehicle cluster.

[0007] The noise perception mechanism of the robust policy learning module dynamically evaluates the environmental disturbance intensity, adopts a weighted QMIX framework to adjust the value function weight, and optimizes the policy in combination with a robust loss function;

[0008] Through the cross-cluster reward coordination module, the reward values of high-reward clusters are dynamically reduced to balance global optimization according to the differences in reward distribution between clusters.

[0009] Through the central scheduling system, the collaborative decision-making instructions are distributed to each autonomous vehicle for execution.

[0010] Preferably, the cluster graph representation module includes a node feature matrix and an adjacency matrix.

[0011] The node feature matrix is used to record the vehicle motion state and safety parameters.

[0012] The adjacency matrix is used to construct the intra-cluster and inter-cluster connection topology through the relative position relationship.

[0013] Preferably, the noise perception mechanism quantifies the influence of environmental noise on the cluster through a noise estimation network, and dynamically adjusts the weight coefficient of the hybrid network.

[0014] Preferably, the weighted QMIX framework assigns lower weights to samples severely disturbed by noise and higher weights to reliable samples.

[0015] Preferably, the robust loss function uses a piecewise function design, which uses quadratic penalty for small errors and linear penalty for large errors.

[0016] Preferably, the cross-cluster reward coordination module includes:

[0017] Calculate the algebraic sum of all cluster rewards.

[0018] Identify negative reward clusters and calculate the total amount of punishment.

[0019] Proportionally reduce the reward values of positive reward clusters.

[0020] Preferably, the local reward function is composed of six sub-reward functions; wherein the sub-reward functions include speed, task, safety, acceleration, lane changing frequency and energy consumption.

[0021] In a second aspect, the present application also provides an autonomous vehicle multi-vehicle collaborative control system based on robust QMIX, which includes:

[0022] A cluster graph representation module is used to describe the local graph structure of vehicle interaction within the cluster and the global graph structure of inter-cluster interaction, to represent the interaction relationship within and outside the vehicle cluster.

[0023] A robust policy learning module is used to dynamically evaluate the environmental disturbance intensity through a noise perception mechanism, adjust the value function weight using a weighted QMIX framework, and optimize the policy in combination with a robust loss function;

[0024] A cross-cluster reward coordination module is used to dynamically reduce the reward value of high-reward clusters to balance global optimization according to the reward distribution difference between clusters.

[0025] A central scheduling system is used to distribute collaborative decision instructions to each autonomous vehicle for execution.

[0026] In a third aspect, the present application also discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the method in the first aspect.

[0027] In a fourth aspect, the present application also discloses a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the steps of the method in the first aspect.

[0028] Compared with the prior art, the present application has the following advantages and technical effects:

[0029] The present application provides an automatic driving multi-vehicle collaborative control method based on robust QMIX, which comprises the following steps: first, a cluster graph representation module is used to describe the local graph structure of vehicle interaction within the cluster and the global graph structure of inter-cluster interaction to represent the interaction relationship within and outside the vehicle cluster; second, a noise perception mechanism of a robust policy learning module is used to dynamically evaluate the environmental disturbance intensity, a weighted QMIX framework is used to adjust the value function weight, and a robust loss function is used to optimize the policy; third, a cross-cluster reward coordination module is used to dynamically reduce the reward value of high-reward clusters to balance global optimization according to the reward distribution difference between clusters; finally, a central scheduling system is used to distribute collaborative decision instructions to each autonomous vehicle for execution.

[0030] The present application proposes a robust QMIX for solving the collaborative decision problem of unmanned vehicle systems in a hierarchical mixed traffic environment under the background of observation uncertainty and heterogeneity. By constructing a cluster graph structure and a noise perception weighting mechanism, the robustness and stability of the system under the conditions of incomplete information and sensor distortion are effectively improved. The proposed algorithm theoretically guarantees convergence and can achieve long-term stable policy optimization in a complex traffic system.

[0031] The present application introduces a cross-cluster reward coordination mechanism, which realizes local optimization within a cluster through sub-target design, and guides the system to evolve towards the global optimal direction by dynamically adjusting the reward weight between clusters. This mechanism alleviates the credit allocation problem commonly existing in unmanned vehicle systems, enabling unmanned vehicles to obtain clearer policy feedback in complex collaboration. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which are a part of this application, are included to provide a further understanding of the application, and are incorporated in and constitute a part of this application. The embodiments of this application, and their

[0033] Figure 1 A method flowchart of an embodiment of the application. DETAILED DESCRIPTION

[0034] It should be noted that the embodiments and features of the embodiments in the present application can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0035] It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0036] Embodiment one

[0037] As shown in the figure, an automatic driving multi-vehicle cooperative control method based on robust QMIX is provided in the embodiment, which comprises: Figure 1

[0038] S1, through a cluster graph representation module, describe the local graph structure of vehicle interaction within the cluster and the global graph structure of interaction between clusters, to represent the interaction relationship within and outside the vehicle cluster;

[0039] Further, the cluster graph representation module comprises a node feature matrix and an adjacency matrix;

[0040] The node feature matrix is used to record the vehicle motion state and safety parameters.

[0041] The adjacency matrix is used to construct the intra-cluster and inter-cluster connection topology through the relative position relationship.

[0042] Specifically, in the cluster graph representation module, the local and global node feature matrix and the adjacency matrix are constructed to represent the interaction relationship within and outside the cluster.

[0043] QMIX is a kind of unmanned vehicle reinforcement learning algorithm, which aims to solve the problem of "credit distribution" in the process of unmanned vehicle cooperation. It adopts the framework of centralized training with decentralized execution (CTDE), in which each unmanned vehicle has its local Q function wherein, ​for the local observation of the ego vehicle, for its action, are the corresponding network parameters.

[0044] On this basis, QMIX defines a mixing network for combining the local Q values of each ego vehicle into a joint Q value:

[0045] (1)

[0046] where, is the global state information, is the joint action of each ego vehicle, is the parameter of the mixing network, denotes the set of all parameters. In addition, to ensure that each ego vehicle can independently select the optimal action based on its own local Q value during training, QMIX adds a monotonicity constraint in the mixing network, i.e.:

[0047] (2)

[0048] where, is the local Q value of the i-th ego vehicle.

[0049] This constraint can ensure that the partial derivative of the mixing network with respect to each local Q value is always non-negative, thereby ensuring that the joint Q value is monotonically increasing with respect to each local Q value. Further, when selecting an action, if the action that maximizes each local Q value is selected, the global Q value will be optimal, thereby achieving decentralized execution.

[0050] The embodiment improves the mixing network structure of QMIX, introduces a robust loss function (such as Huber loss) and a noise perception adjustment factor, and embeds an attention gate mechanism in the network structure, further improving the expression ability and robustness of the value decomposition network. Combined with the strategy of centralized training and distributed execution, the model can ensure global consistency while supporting each vehicle to make efficient responses based on local observations.

[0051] During training, QMIX updates the parameters by minimizing the temporal difference (TD) error, and the loss function is defined as:

[0052] (3)

[0053] where, is the joint action taken by the ego vehicle And the reward obtained, is a discount factor, is a target network parameter.

[0054] The optimization objective of QMIX is:

[0055] (4)

[0056] wherein, denotes the Q function that can be expressed under the QMIX restriction, and belongs to the function space . The above formula is the QMIX operator, that is:

[0057] (5)

[0058] wherein, is a mapping factor, that is:

[0059] (6)

[0060] wherein, denotes a given joint action value function, which is not limited by the QMIX structure itself, is a Bellman optimal operator, that is:

[0061] (7)

[0062] S2, dynamically evaluate the environmental disturbance intensity through the noise perception mechanism of the robust policy learning module, adjust the value function weight by using the weighted QMIX framework, and optimize the policy in combination with the robust loss function;

[0063] Further, the noise perception mechanism quantifies the influence degree of environmental noise on the cluster through a noise estimation network, and dynamically adjusts the mixed network weight coefficient.

[0064] Further, the weighted QMIX framework gives a lower weight to samples severely disturbed by noise and a higher weight to reliable samples.

[0065] Further, the robust loss function is designed by using a piecewise function, which adopts square penalty for small errors and linear penalty for large errors.

[0066] Specifically, in the robust policy learning module, the noise perception mechanism and the weighted QMIX structure are combined to design a robust value function to improve the stability of the algorithm in the observation error and communication loss environment.

[0067] To determine the influence degree of noise on each cluster under the current state, the embodiment defines a noise estimation network . Considering the observation of CAVs on the position of their neighbor vehicles under noise interference, assuming the real distance between them is , the observed distance is , then we have:

[0068] (8)

[0069] where, is the observation range of the CAV, is the maximum deviation of the observation noise of the CAV. Therefore, according to the observation value , the real value can be solved, and and are obtained. Let , the influence of noise on the CAV can be calculated:

[0070] (9)

[0071] Noise estimation network According to the above derivation, the input of the network is the observation of each cluster on the environment, and the output value will dynamically adjust the weight coefficient in the mixed network to improve the robustness of the CAV cluster in the high-noise environment. The definition of the network is:

[0072] (10)

[0073] where, is a nonlinear activation function, is a minimum value. The network will calculate the average influence of environmental noise on each cluster, and then adjust the weight in the loss function of QMIX.

[0074] Robust weighting mechanism, in some non-monotonic tasks, QMIX will underestimate the value of some joint actions. Based on this, Weighted QMIX (WQMIX) is added to the weight operator based on QMIX, to weight the importance of each joint action in the loss function of QMIX, that is:

[0075] (11)

[0076] where the weight is defined as:

[0077] (12)

[0078] where, is a constant,​ is the true value function. is the noise estimate of the noisy observation network. This way, all the underestimated action values are given more weight, while the rest are given less weight.

[0079] To accurately obtain the weights, the function is used to fit the true value function . It is updated by the following operator:

[0080] (13)

[0081] where, The update of is done by maximizing the monotonic function to select the action.

[0082] (14)

[0083] is the WQMIX operator.

[0084] During training, and are trained by minimizing the following loss functions, respectively:

[0085] (15)

[0086] (16)

[0087] where, denotes the number of mini-batch samples drawn from the experience pool during training; the target value is:

[0088] (17)

[0089] However, in the scenario set in the present embodiment, the high heterogeneity is presented among the clusters of autonomous vehicles, and the noise has different effects on the clusters of autonomous vehicles due to the different communication capabilities. In this case, if all the tuples are equally moved into the experience pool to participate in updating the network weights, it will cause the local network and the mixed network to overfit on the data that is seriously affected by the noise. On the one hand, the disturbed TD error will make the model frequently adjust under the influence of the noise, and it is difficult to converge to a stable strategy; on the other hand, for the data that is less affected by the noise, it cannot help the learning of the optimal strategy because it is given the same weight as the rest of the data.

[0090] The Huber loss function can retain the characteristics of mean squared error (MSE) that is sensitive to small errors when the error is small, and use absolute error (Mean Absolute Error, SAE) when the error is too large to avoid the problem of pure MSE being too sensitive to data points with too much deviation, which is defined as follows:

[0091] (18)

[0092] wherein, represents the difference between the true value and the predicted value, is a hyperparameter used to control the threshold value from quadratic penalty to linear penalty.

[0093] The convergence of the robust QMIX not only determines whether it can stably find a reasonable strategy, but also directly affects its performance in complex environments. The convergence of the robust QMIX algorithm proposed in this embodiment will be proved by combining the Bellman optimal operator. The Bellman optimal operator is used to describe the update rule of the strategy in reinforcement learning, which is derived from the Bellman equation of dynamic programming. It updates by mapping the action value function of the current state to the value of the next state, so as to find the optimal strategy. In standard Q learning, the Bellman optimal operator is defined as:

[0094] (19)

[0095] This embodiment will prove the convergence of the robust QMIX through two lemmas.

[0096] (1) is a contraction mapping;

[0097] Consider any two Q functions and , after mapping, the difference is:

[0098] (20)

[0099] wherein, represents the maximum value of the function .

[0100] Consider the triangle inequality , let:

[0101] (21)

[0102] (22)

[0103] It can be obtained that:

[0104] (23)

[0105] Considering the definition in the above formula, we have:

[0106] (24)

[0107] For any two real number sets and , we have:

[0108] (25)

[0109] Let and , we have:

[0110] (26)

[0111] Considering the definition of the infinite norm:

[0112] (27)

[0113] We can get:

[0114] (28)

[0115] Combining the above formulas, we have:

[0116] (29)

[0117] Since , , we have . Also, since:

[0118] (30)

[0119] Therefore, we have:

[0120] (31)

[0121] Continuing to process the right side of the inequality, we have:

[0122] (32)

[0123] When taking the supremum over all , it is independent of and , so we can take it out, and we have:

[0124] (33)

[0125] Based on the above derivation, we can conclude that:

[0126] (34)

[0127] because , and Therefore:

[0128] (35)

[0129] therefore, satisfy - Compression mapping condition, its compression factor is:

[0130] (36)

[0131] (2) The existence of a unique fixed point;

[0132] According to Banach's fixed-point principle, any contraction mapping has a unique fixed point in a complete metric space. For any operator If a certain satisfy:

[0133] (37)

[0134] Then it is called for The fixed point.

[0135] Combining robust Bellman operators The definition of its fixed point satisfy:

[0136] (38)

[0137] The results were:

[0138] (39)

[0139] because Therefore, the necessary and sufficient condition for the above expression to always hold true is:

[0140] (40)

[0141] Right now:

[0142] (41)

[0143] therefore, It is also a standard Bellman operator. fixed point. Since It is known that there exists a unique fixed point , so:

[0144] (42)

[0145] From the above derivations, we can know that there exists a unique fixed point in complete metric space , and .

[0146] (3) Convergence of robust QMIX;

[0147] The robust QMIX algorithm proposed in this embodiment uses an empirical sample-based random update to approximate the fixed point, and the update formula is:

[0148] (43)

[0149] where represents a noise error term with zero mean and bounded error, and represents a sampling error or other random disturbance.

[0150] To ensure that the algorithm takes into account both sufficient exploration of the initial state and reduction of noise influence in the update process, the learning rate must satisfy the following conditions:

[0151] (44)

[0152] (45)

[0153] The first formula ensures that the algorithm can fully explore the environment and eliminate initial errors, and the second formula ensures that the cumulative effect of noise is limited, so it will not hinder the convergence of the algorithm.

[0154] When the above restrictions on the learning rate are met, according to the Robbins-Monro random approximation theory, the random process corresponding to the update formula follows a corresponding deterministic differential equation:

[0155] (46)

[0156] According to the conclusions (1) in this embodiment, the above formula has a unique and globally asymptotically stable fixed point , so the global Q value estimate of robust QMIX almost certainly converges to the optimal Q value :

[0157] (47)

[0158] S3, dynamically reducing the reward value of the high-reward cluster to balance the global optimization according to the inter-cluster reward distribution difference through the cross-cluster reward coordination module;

[0159] Further, the cross-cluster reward coordination module comprises:

[0160] calculating the algebraic sum of the rewards of all clusters;

[0161] identifying the negative reward cluster and calculating the total amount of punishment thereof;

[0162] proportionally reducing the reward value of the positive reward cluster.

[0163] Specifically, the cross-cluster reward coordination module guides the multi-vehicle system to achieve the global collaborative goal by dynamically adjusting the reward distribution strategy among clusters.

[0164] In reinforcement learning, the reward function provides a basis for evaluating the behavior of the unmanned vehicle. By giving positive or negative reward values for different actions, the unmanned vehicle can learn behaviors that can achieve the goal through continuous interaction with the environment. The reward function can provide immediate feedback on the decision-making result at each time step and can guide the unmanned vehicle to optimize the strategy to maximize the long-term cumulative reward, which has a profound impact on the overall performance and convergence speed of the algorithm. In order to coordinate the rewards among clusters and prevent the controlled unmanned vehicles from converging to a local optimum, the present embodiment proposes a cross-cluster reward coordination mechanism.

[0165] In the unmanned vehicle system provided in the present embodiment, the clusters are in a cooperative relationship and have consistent interests and the same goal: to improve the efficiency, safety, comfort, energy saving, etc. of the traffic flow in the traffic scene. However, in the learning process, some clusters will make self-interested behaviors to maximize their cumulative rewards, such as vehicles in a cluster driving at a high speed in the middle lane, thereby hindering the lane-changing actions of other clusters. This strategy will make them obtain higher rewards, but the sum of the rewards obtained by each cluster does not reach the optimum, thereby affecting the learning effect. Based on this, the present embodiment proposes a cross-cluster reward coordination mechanism that reduces the rewards of well-performing clusters when a certain cluster obtains too low a reward, thereby guiding the entire system towards a more optimal strategy space.

[0166] Let be the sum of the rewards obtained by all clusters, i.e.

[0167] (48)

[0168] wherein, for the cluster total reward obtained. When the elements of the set are not all positive or all negative, consider the sum of the absolute values of the negative terms in the set and the sum of the absolute values of all terms :

[0169] (49)

[0170] (50)

[0171] For positive terms in the set , the value will decrease to some extent according to and and the numerical value itself, and the extent of the decrease is:

[0172] (51)

[0173] Thus, the sum of the rewards of all clusters can be represented as:

[0174] (52)

[0175] S4, distribute the collaborative decision-making instructions to each autonomous vehicle for execution through the central scheduling system.

[0176] Further, the local reward function is composed of six sub-reward functions weighted; wherein the sub-reward functions include speed, task, safety, acceleration, lane changing frequency and energy consumption.

[0177] Specifically, the optimized strategy instructions are distributed to each autonomous vehicle by the central scheduling system, achieving safe, efficient and robust decision-making in complex traffic scenarios.

[0178] Embodiment Two

[0179] ​This embodiment uses a three-lane highway as an example. The decision-making objective of the autonomous driving system should prioritize safety, ensuring that the controlled CAV maintains a safe distance and relative speed with the vehicle in front to prevent collisions. For the entire traffic flow, a sufficiently high average speed should be maintained to keep overall efficiency at a high level. For individual vehicles, timely lane changes and acceleration / deceleration should improve their driving efficiency to efficiently reach the predetermined destination. However, to meet comfort requirements, vehicles should not make lane changes or sudden acceleration / deceleration too frequently. These requirements interact, forming a multi-objective optimization problem. Therefore, for individual vehicles within the cluster, this embodiment comprehensively considers six indicators, designing six different sub-reward functions for speed, task, safety, acceleration, lane-changing frequency, and energy consumption, and assigning different weights to each to obtain the cluster's local reward function.

[0180] First, establish the node feature matrix.

[0181] Recorded in Time Cluster Features of the internal CAV and all observed vehicle nodes. eigenvectors This includes the vehicle's own condition and safety parameters relative to surrounding vehicles, namely:

[0182] (53)

[0183] Next, construct the adjacency matrix.

[0184] Intra-cluster adjacency matrix Describes the interactions within the cluster. It can be represented as:

[0185] (54)

[0186] Inter-cluster adjacency matrix It can be represented as:

[0187] (55)

[0188] Therefore, the global adjacency matrix can be represented by all intra-cluster and inter-cluster adjacency matrices, i.e.:

[0189] (56)

[0190] Then, QMIX defined a hybrid network. This is used to combine the local Q values ​​of each autonomous vehicle into a joint Q value:

[0191] (57)

[0192] wherein, is the global state information, is the joint action of each ego vehicle, is the parameter of the mixing network, denotes the set of all parameters. In addition, to ensure that each ego vehicle can independently select the optimal action based on the local Q value during training, QMIX adds a monotonicity constraint in the mixing network, that is:

[0193] (58)

[0194] The weight is redefined in this embodiment, that is:

[0195] (59)

[0196] An adaptive update strategy is provided for an environment with high noise. When the model judges through the noise estimation network that the observation of some state is interfered by strong noise, the weight of these samples is reduced, and the influence of these samples on parameter update is reduced, so as to avoid over-learning of the model on the samples with high noise.

[0197] In this embodiment, the Huber loss function is defined as:

[0198] (60)

[0199] wherein, is the TD error, and the threshold is defined as:

[0200] (61)

[0201] wherein, represents taking the median, is a very small positive number, which is used to prevent the phenomenon of degradation in the training process.

[0202] Combining the above robust weighting mechanism, the overall Huber loss function is defined as:

[0203] (62)

[0204] wherein, is the number of ego vehicles in the environment; is the network parameter of the local network; is a regularization coefficient, which can be used to control the trade-off between the loss term and the regularization term. Therefore, the network parameters of the mixing network and the local network can be obtained by minimizing the total loss Optimization:

[0205] (63)

[0206] (64)

[0207] where, and are the learning rates of the mixed network and the local network, respectively. In addition, to make the training smooth, the target mixed network parameters are updated using a soft update policy:

[0208] (65)

[0209] where, is the soft update coefficient, which is used to ensure the gradual adjustment of the target network parameters.

[0210] Finally, in the robust QMIX algorithm proposed in this embodiment, the above-mentioned robust weighting mechanism is combined, which is redefined as the robust Bellman operator :

[0211] (66)

[0212] where, is the learning rate, is the noise-based weight, which takes a value in the range of .

[0213] (1) Speed reward function;

[0214] The speed reward function has different speed requirements for autonomous vehicles in different lanes.

[0215] When the controlled autonomous vehicle is in the outermost lane, the speed reward function has the following form:

[0216] (67)

[0217] where, represents the speed of the autonomous vehicle, with the unit of m / s.

[0218] When the controlled vehicle is in the middle lane, the speed reward function has the following form:

[0219] (68)

[0220] When the autonomous vehicle is in the innermost lane, the speed reward function is:

[0221] (69)

[0222] (2) Task reward function;

[0223] Three different task reward functions are set for the three clusters in the environment.

[0224] Thus, the speed is added to the independent variable of the task reward in this embodiment:

[0225] (77)

[0226] wherein, represent cluster 1, 2 and 3 respectively, represent the outermost, middle and innermost lanes respectively.

[0227] (3) Safety reward function;

[0228] A safety reward function is designed based on TTC (Time to Collision) to limit dangerous driving behavior. For the autonomous vehicle controlled in this embodiment, it will face the following three situations when driving:

[0229] a) No other vehicles in front of its lane: ;

[0230] b) There are vehicles in front and the speed of the controlled autonomous vehicle is lower than that of the other party: ;

[0231] c) There are vehicles in front and the speed of the controlled autonomous vehicle is higher than that of the other party: At this time, the calculation formula of TTC is as follows: ; (78)

[0232] wherein, is the distance between the two vehicles, is the relative speed of the two vehicles, which is set to 4 s, when TTC is less than 4 s, the safety reward function will punish the autonomous vehicle, and its form is:

[0233] (79)

[0234] (4) Acceleration reward function;

[0235] The acceleration reward function set in this embodiment is as follows:

[0236] (80)

[0237] wherein, is the threshold acceleration, in the scenario of this embodiment, It was set to 4.5 m / s 2 .

[0238] (5) Lane change reward function;

[0239] The lane-change reward function set in this embodiment is as follows:

[0240] (81)

[0241] in, This refers to the number of lane changes per second for autonomous vehicles. In this embodiment, the allowable frequency is set to 0.2.

[0242] (6) Energy consumption reward function;

[0243] First, based on the car's motion equation, by calculating the resistance it experiences during motion, we can obtain the power required to drive the vehicle. Therefore, the power required by the power battery pack can be obtained. Ultimately, the energy consumption of the vehicle per unit time step can be calculated. for:

[0244] (82)

[0245] The energy reward function designed for this scenario is based on The calculation is as follows:

[0246] (83)

[0247] Thus, a cluster is obtained. The sum of the rewards received by each autonomous vehicle is:

[0248] (84)

[0249] in, For the first The weights corresponding to the item reward function, For cluster The number of internal CAVs, For cluster Inner The first CAV Sub-reward function.

[0250] Table 1 Robust QMIX Pseudocode

[0251] Based on the above robust QMIX model design, it is jointly trained with a Graph Attention Network (GAT). By introducing a robust weighting mechanism and the Huber loss function, the model's policy convergence performance under conditions of observation noise and communication uncertainty is effectively improved, thereby achieving stable cooperative control of multi-vehicle systems in heterogeneous environments.

[0252] During the scenario, at each time step, the actions output by each CAV cluster under the local graph structure are aggregated to the central scheduling system, which then generates execution instructions and sends them to the vehicle controller for interaction with the environment. The experience quadruple formed during the interaction—comprising the state, action, reward, and next state—is stored in the experience replay pool for subsequent robust autonomous vehicle reinforcement learning training. This process continues until the training rounds are completed or the algorithm performance reaches a preset standard.

[0253] Example 3

[0254] This invention also provides a robust QMIX-based autonomous driving multi-vehicle cooperative control system, comprising:

[0255] The clustered graph representation module is used to describe the local graph structure of vehicle interactions within a cluster and the global graph structure of interactions between clusters, so as to characterize the interaction relationships between vehicles inside and outside the cluster.

[0256] The robust policy learning module is used to dynamically assess the intensity of environmental interference through a noise perception mechanism. It adopts a weighted QMIX framework to adjust the weights of the value function and combines a robust loss function to optimize the policy.

[0257] The cross-cluster reward coordination module is used to dynamically reduce the reward value of high-reward clusters based on the differences in reward distribution between clusters in order to balance global optimization.

[0258] The central dispatch system is used to distribute collaborative decision-making instructions to each autonomous vehicle for execution.

[0259] Example 4

[0260] This embodiment also discloses a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0261] Example 5

[0262] This embodiment also discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in Embodiment 1.

[0263] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A robust QMIX-based automatic driving multi-vehicle cooperative control method, characterized in that, The method comprises the following steps: The cluster graph representation module describes the local graph structure of vehicle interaction within the cluster and the global graph structure of interaction between clusters to represent the interaction relationship within and outside the vehicle cluster; The robust strategy learning module dynamically evaluates the environmental disturbance intensity through a noise perception mechanism, adjusts the value function weight using a weighted QMIX framework, and optimizes the strategy combined with a robust loss function; The noise perception mechanism quantifies the influence of environmental noise on the cluster through a noise estimation network, and dynamically adjusts the mixed network weight coefficient; ； wherein, is a weight coefficient in the mixed network, is a constant, is a noise estimate of the noisy observation network; The robust loss function adopts a piecewise function design, which adopts square penalty for small errors and linear penalty for large errors; The cross-cluster reward coordination module dynamically reduces the reward value of high-reward clusters to balance global optimization according to the reward distribution difference between clusters; The central scheduling system distributes the collaborative decision-making instructions to each autonomous vehicle for execution.

2. The method of claim 1, wherein The cluster graph representation module includes a node feature matrix and an adjacency matrix; The node feature matrix is used to record the vehicle motion state and safety parameters; The adjacency matrix is used to construct the intra-cluster and inter-cluster connection topology through the relative position relationship.

3. The method of claim 1, wherein The weighted QMIX framework assigns lower weights to samples severely disturbed by noise and higher weights to reliable samples.

4. The method of claim 1, wherein The cross-cluster reward coordination module includes: Calculate the algebraic sum of all cluster rewards; Identify negative reward clusters and calculate their total penalty; Proportionally reduce the reward values of positive reward clusters.

5. The method of claim 1, wherein The local reward function is composed of six sub-reward functions weighted; wherein the sub-reward functions include speed, task, safety, acceleration, lane changing frequency, and energy consumption.

6. A robust QMIX-based automatic driving multi-vehicle cooperative control system, characterized in that, The system for implementing the method of any one of claims 1-5 comprises: A cluster graph representation module for describing the local graph structure of vehicle interaction within the cluster and the global graph structure of interaction between clusters to represent the interaction relationship within and outside the vehicle cluster; A robust strategy learning module for dynamically evaluating the environmental disturbance intensity through a noise perception mechanism, adjusting the value function weight using a weighted QMIX framework, and optimizing the strategy combined with a robust loss function; A cross-cluster reward coordination module for dynamically reducing the reward value of high-reward clusters to balance global optimization according to the reward distribution difference between clusters; A central scheduling system for distributing collaborative decision-making instructions to each autonomous vehicle for execution.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-5.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1-5.

Images