Automatic driving multi-vehicle cooperative control method and system based on robust QMIX

Abstract

The invention discloses an automatic driving multi-vehicle cooperative control method and system based on robust QMIX, and belongs to the field of Internet of Vehicles and automatic driving, and the method comprises the steps: describing a local graph structure of vehicle interaction in a cluster and a global graph structure of inter-cluster interaction through a clustered graph representation module, so as to represent the interaction relation between the interior and exterior of the vehicle cluster; the environmental interference intensity is dynamically evaluated through a noise sensing mechanism of a robust strategy learning module, the weight of a value function is adjusted by adopting a weighted QMIX framework, and a strategy is optimized in combination with a robust loss function; through a cross-cluster reward coordination module, according to the reward distribution difference between clusters, the reward value of a high reward cluster is dynamically reduced to balance global optimization; and distributing the collaborative decision instruction to each automatic driving vehicle for execution through the central scheduling system. According to the invention, long-term stable strategy optimization can be realized in a complex traffic system.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of Internet of Vehicles and automatic driving, and particularly relates to an automatic driving multi-vehicle cooperative control method and system based on robust QMIX. BACKGROUND

[0002] In practical applications such as automatic driving, the uncertainty of the environment, such as perception errors, communication interruptions and state loss, greatly interferes with the strategy learning and execution stability of unmanned vehicles. As a typical unmanned vehicle reinforcement learning algorithm, QMIX (Q-Mixing networks) effectively alleviates the training difficulties caused by non-stationarity and information asymmetry by introducing an additive value function decomposition structure. This method represents the global joint Q value as a nonlinear combination of individual local Q values under the framework of centralized training and distributed execution, ensuring the attainability of global optimality in the strategy learning process. At the same time, QMIX uses a mixing network structure to fuse the decision-making contributions of different unmanned vehicles, providing a stable and efficient optimization path for strategy cooperation in heterogeneous systems.

[0003] However, existing unmanned vehicle reinforcement learning methods have difficulty effectively handling strategy bias caused by information asymmetry and observation uncertainty in heterogeneous environments, especially in mixed traffic scenarios with frequent noise disturbances, which can lead to fluctuations in decision-making performance and a decline in system robustness. QMIX underestimates the value of some joint actions in some non-monotonic tasks, and lacks an effective adaptive update strategy when facing environments with high noise, which can lead to over-learning on high-noise samples and affect the convergence performance of the model. In addition, existing methods have deficiencies in reward coordination between clusters, and some clusters may make self-interested behaviors to maximize their cumulative rewards, which can hinder other clusters from making lane-changing actions and affect the cooperative efficiency and learning effect of the entire system. SUMMARY

[0004] To solve the above technical problems, the application provides an automatic driving multi-vehicle cooperative control method and system based on robust QMIX to solve the problems existing in the prior art.

[0005] To achieve the above purpose, in a first aspect, the application provides an automatic driving multi-vehicle cooperative control method based on robust QMIX, comprising: The cluster graph representation module describes the local graph structure of vehicle interaction within the cluster and the global graph structure of interaction between clusters to represent the interaction relationship within and outside the vehicle cluster. The noise perception mechanism of the robust strategy learning module dynamically evaluates the environmental disturbance intensity, adjusts the value function weight using a weighted QMIX framework, and optimizes the strategy in combination with a robust loss function. By the cross-cluster reward coordination module, the reward values of high-reward clusters are dynamically reduced according to the differences in inter-cluster reward distribution to balance global optimization. The central scheduling system is used for distributing the collaborative decision-making instructions to the automatic driving vehicles for execution.

[0006] Preferably, the cluster graph representation module comprises a node feature matrix and an adjacency matrix. The node feature matrix is used for recording vehicle motion states and safety parameters. The adjacency matrix is used for constructing intra-cluster and inter-cluster connection topologies through relative position relationships.

[0007] Preferably, the noise perception mechanism quantifies the influence degree of environmental noise on the cluster through a noise estimation network, and dynamically adjusts the weight coefficients of the hybrid network.

[0008] Preferably, the weighted QMIX framework assigns a lower weight to samples severely disturbed by noise and a higher weight to reliable samples.

[0009] Preferably, the robust loss function adopts a piecewise function design, using square penalty for small errors and linear penalty for large errors.

[0010] Preferably, the cross-cluster reward coordination module comprises: calculating the algebraic sum of all cluster rewards; identifying negative reward clusters and calculating the total amount of punishment; proportionally reducing the reward values of positive reward clusters.

[0011] Preferably, the local reward function is composed of six sub-reward functions; wherein the sub-reward functions include speed, task, safety, acceleration, lane changing frequency and energy consumption.

[0012] In a second aspect, the present application also provides an automatic driving multi-vehicle collaborative control system based on robust QMIX, comprising: a cluster graph representation module for describing the local graph structure of vehicle interaction within the cluster and the global graph structure of inter-cluster interaction to represent the interaction relationship within and outside the vehicle cluster; a robust policy learning module for dynamically evaluating the environmental disturbance intensity through a noise perception mechanism, adjusting the value function weight using a weighted QMIX framework, and optimizing the policy in combination with a robust loss function; a cross-cluster reward coordination module for dynamically reducing the reward values of high-reward clusters according to the differences in inter-cluster reward distribution to balance global optimization; a central scheduling system for distributing the collaborative decision-making instructions to the automatic driving vehicles for execution.

[0013] In a third aspect, the application further discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the method in the first aspect.

[0014] In a fourth aspect, the application further discloses a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the method in the first aspect.

[0015] Compared with the prior art, the application has the following advantages and technical effects: The application provides an automatic driving multi-vehicle cooperative control method based on a robust QMIX, which comprises the following steps: first, a cluster graph representation module is used to describe the local graph structure of vehicle interaction in a cluster and the global graph structure of interaction between clusters, so as to represent the interaction relationship inside and outside the vehicle cluster; second, a noise perception mechanism of a robust strategy learning module is used to dynamically evaluate the environmental disturbance intensity, a weighted QMIX framework is used to adjust the weight of the value function, and a robust loss function is used to optimize the strategy; third, a cross-cluster reward coordination module is used to dynamically reduce the reward value of a high-reward cluster according to the reward distribution difference between clusters to balance the global optimization; and finally, a central scheduling system is used to distribute cooperative decision instructions to each automatic driving vehicle for execution.

[0016] The application proposes a robust QMIX for solving the cooperative decision problem of an unmanned vehicle system in a hierarchical mixed traffic environment under the background of observation uncertainty and heterogeneity. By constructing a cluster graph structure and a noise perception weighting mechanism, the robustness and stability of the system under the conditions of incomplete information and sensor distortion are effectively improved. The proposed algorithm has convergence guarantee in theory and can realize long-term stable strategy optimization in a complex traffic system.

[0017] The application introduces a cross-cluster reward coordination mechanism, which realizes local optimization within a cluster through sub-target design, and guides the system to evolve towards the global optimal direction by dynamically adjusting the reward weight between clusters. This mechanism alleviates the credit allocation problem commonly existing in the unmanned vehicle system, so that the unmanned vehicle can obtain clearer strategy feedback in complex cooperation. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of the application illustrated in the drawings, and their description, are used to explain the application and are not intended to limit the application. In the drawings: Figure 1 The method flowchart of the embodiments of the application. DETAILED DESCRIPTION

[0019] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0020] It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown.

[0021] Embodiment one As Figure 1 shown, the embodiment provides a robust QMIX-based automatic driving multi-vehicle cooperative control method, which includes: S1, through a cluster graph representation module, describe the local graph structure of vehicle interaction within the cluster and the global graph structure of interaction between clusters, to represent the interaction relationship within and outside the vehicle cluster; Further, the cluster graph representation module includes a node feature matrix and an adjacency matrix; Wherein, the node feature matrix is used to record the vehicle motion state and safety parameters; The adjacency matrix is used to construct the intra-cluster and inter-cluster connection topology through the relative position relationship.

[0022] Specifically, in the cluster graph representation module, the local and global node feature matrix and the adjacency matrix are constructed to represent the interaction relationship within and outside the cluster; QMIX is a kind of unmanned vehicle reinforcement learning algorithm, which aims to solve the problem of "credit distribution" in the process of unmanned vehicle cooperation. It adopts the framework of centralized training with decentralized execution (Centralized Training with Decentralized Execution, CTDE), in which each unmanned vehicle has its local Q function , where is the local observation of the unmanned vehicle, is its action, is the corresponding network parameter.

[0023] On this basis, QMIX defines a mixing network for combining the local Q values of each unmanned vehicle into a joint Q value: (1) Where, is the global state information, is the joint action of each unmanned vehicle, is the parameter of the mixing network, denotes the set of all parameters. In addition, to ensure that each unmanned vehicle can independently select the optimal action based on the respective local Q value during training, QMIX adds a monotonicity constraint in the mixing network, that is: (2) wherein, is the local Q value of the ith unmanned vehicle.

[0024] This constraint can ensure that the partial derivative of the mixing network with respect to each local Q value is always non-negative, thereby ensuring that the joint Q value is monotonically increasing with respect to each local Q value. Further, when selecting an action, if the action selected is the action that maximizes each local Q value , the global Q value will be optimal, thereby achieving decentralized execution.

[0025] The embodiment improves the mixing network structure of QMIX, introduces a robust loss function (such as Huber loss) and a noise perception adjustment factor, and embeds an attention gate mechanism in the network structure, further improving the expression ability and robustness of the value decomposition network. Combined with the strategy of centralized training and distributed execution, the model can ensure global consistency while supporting each vehicle to make efficient responses based on local observations.

[0026] During training, QMIX updates the parameters by minimizing the temporal difference (TD) error, and the loss function is defined as: (3) wherein, is the reward obtained by the unmanned vehicle taking the joint action , is a discount factor, is a target network parameter.

[0027] The optimization goal of QMIX is: (4) wherein, is a Q function that can be represented under the QMIX constraint, and belongs to the function space . The above formula is the QMIX operator, that is: (5) wherein, is a mapping factor, that is: (6) wherein, ​​denotes the given joint action value function, which is not restricted by the QMIX structure itself, is the Bellman optimality operator, i.e., (7) S2, dynamically evaluate the environmental disturbance intensity through the noise perception mechanism of the robust policy learning module, adjust the value function weight using the weighted QMIX framework, and optimize the policy in combination with the robust loss function; Further, the noise perception mechanism quantifies the influence degree of environmental noise on the cluster through a noise estimation network, and dynamically adjusts the weight coefficient of the mixed network.

[0028] Further, the weighted QMIX framework assigns lower weights to samples severely disturbed by noise and higher weights to reliable samples.

[0029] Further, the robust loss function adopts a piecewise function design, using quadratic penalty for small errors and linear penalty for large errors.

[0030] Specifically, in the robust policy learning module, the noise perception mechanism and the weighted QMIX structure are combined to design a robust value function to improve the stability of the algorithm in the environment with observation errors and communication loss; To determine the influence degree of noise on each cluster under the current state, this embodiment defines a noise estimation network . Considering the observation of the CAV on the position of its neighbor vehicles under noise disturbance, assuming that the real distance between them is , and the observed distance is , then: (8) wherein, is the observation range of the CAV, is the maximum deviation of the observation noise of the CAV. Therefore, according to the observation value , the real value can be solved, and and are obtained. Assuming , the influence degree of noise on the CAV can be calculated as: (9) Noise estimation network is constructed according to the above derivation. The input of this network is the observation of each cluster on the environment, and the output value will dynamically adjust the weight coefficient in the mixed network to improve the robustness of the CAV cluster in a high-noise environment. The definition of this network is: (10) where is a nonlinear activation function, is a minimum value. The network will calculate the average impact of environmental noise on each cluster, and adjust the weight in the loss function of QMIX.

[0031] Robust weighting mechanism, in some non-monotonic tasks, QMIX will underestimate the value of some joint actions. Based on this, Weighted QMIX (WQMIX) is added to the weight operator to weight the importance of each joint action in the loss function of QMIX, that is: (11) where the weight is defined as: (12) where is a constant, is the true value function. is the noise estimate of the noise observation network. This method gives greater weight to all underestimated action values, and lower weight to the remaining actions.

[0032] To accurately obtain the weight, the function is used to fit the true value function . It is updated by the following operator: (13) where is updated by maximizing the monotonic function to select the action.

[0033] (14) is the WQMIX operator.

[0034] During training, and are trained by minimizing the following loss functions, respectively: (15) (16) where is the number of mini-batch samples drawn from the experience pool during training; the target value is: (17) ​However, in the scenario set by the present embodiment, each autonomous vehicle cluster presents a high degree of heterogeneity, and the noise has different effects on each vehicle cluster due to the difference in communication capabilities. In this case, if all tuples are moved into the experience pool to participate in updating the network weights equally, it will cause the local network and the mixed network to overfit on the data that is seriously affected by the noise. On the one hand, the disturbed TD error will make the model frequently adjust under the influence of noise, and it is difficult to converge to a stable strategy; on the other hand, for data that is less affected by noise, it cannot help learn the optimal strategy because it is given the same weight as the rest of the data.

[0035] The Huber loss function can retain the characteristics of the mean squared error (MSE) that is sensitive to small errors when the error is small, and use the mean absolute error (SAE) when the error is too large to avoid the problem of pure MSE being too sensitive to data points with too much deviation. Its definition is as follows: (18) wherein, represents the difference between the true value and the predicted value, is a hyperparameter used to control the threshold for transitioning from quadratic penalty to linear penalty.

[0036] The convergence of the robust QMIX not only determines whether it can stably find a reasonable strategy, but also directly affects its performance in complex environments. The convergence of the robust QMIX algorithm proposed in the present embodiment will be proved by combining the Bellman optimal operator. The Bellman optimal operator is used to describe the update rule of the strategy in reinforcement learning, which is derived from the Bellman equation of dynamic programming. It updates by mapping the action value function of the current state to the value of the next state, so as to find the optimal strategy. In standard Q-learning, the Bellman optimal operator is defined as: (19) The convergence of the robust QMIX will be proved by two lemmas in the present embodiment.

[0037] (1) is a contraction mapping; Consider any two Q functions and , after mapping by , the difference is: (20) wherein, denotes the function ​The maximum value.

[0038] Consider the triangle inequality ,make: (twenty one) (twenty two) We can obtain: (twenty three) Considering the definition in the above formula, we get: (twenty four) For any two sets of real numbers and All of them have: (25) make as well as We can obtain: (26) Consider the definition of the infinite norm: (27) available: (28) Combining the above formulas, we can obtain: (29) because , Therefore Furthermore, because: (30) Therefore, we can conclude that: (31) Continuing to process the right side of the inequality, we get: (32) When for all When taking the supremum, and It is irrelevant and can be removed, hence: (33) Based on the above derivation, we can conclude that: (34) because , and Therefore: (35) Therefore, Satisfy - Compression mapping condition, the compression factor is: (36) (2) Existence of unique fixed point; According to Banach fixed point theorem, any compression mapping exists a unique fixed point on complete metric space. For any operator , if a satisfies: (37) is called the fixed point of .

[0039] Combined with the definition of robust Bellman operator , its fixed point satisfies: (38) After sorting, we get: (39) Because , so the sufficient and necessary condition for the formula to be always true is: (40) That is: (41) Therefore, is also the fixed point of the standard Bellman operator . Since is known to have a unique fixed point , so: (42) Based on the above derivation, we can see that there exists a unique fixed point on the complete metric space ( ), and .

[0040] (3) Convergence of robust QMIX; The robust QMIX algorithm proposed in this embodiment uses an empirical sample-based random update to approximate the fixed point, with the update formula: (43) where represents a noise error term with zero mean and bounded error, and

[0041] ​To ensure that the algorithm takes into account both sufficient exploration of the initial state and reduction of the influence of noise in the update process, the learning rate is required to satisfy the following conditions: (44) (45) The first formula ensures that the algorithm can fully explore the environment and eliminate initial errors, and the second formula ensures that the cumulative effect of noise is limited, so as to not hinder the convergence of the algorithm.

[0042] When the above restrictions on the learning rate are met, according to the Robbins-Monro random approximation theory, the update formula corresponds to a random process that follows a corresponding deterministic differential equation: (46) The trajectory of the differential equation. According to the conclusions of (1) in the present embodiment, (2) the existence of a unique fixed point, the above formula has a unique and globally asymptotically stable fixed point Therefore, the global Q value estimate of the robust QMIX almost certainly converges to the optimal Q value : (47) S3, through the cross-cluster reward coordination module, according to the reward distribution difference between clusters, dynamically reducing the reward value of the high-reward cluster to balance the global optimization; Further, the cross-cluster reward coordination module comprises: Calculate the algebraic sum of all cluster rewards; Identify the negative reward cluster and calculate the total amount of punishment; Proportionally reduce the reward value of the positive reward cluster.

[0043] Specifically, the cross-cluster reward coordination module dynamically adjusts the reward allocation strategy between clusters to guide the multi-vehicle system to achieve the global collaborative goal.

[0044] The cross-cluster reward coordination mechanism provides a basis for evaluating the pros and cons of behavior for unmanned vehicles in reinforcement learning through the reward function. By giving positive or negative reward values for different actions, unmanned vehicles can learn behaviors that can achieve their goals through continuous interaction with the environment. The reward function can provide immediate feedback on decision-making results at each time step and guide unmanned vehicles to optimize strategies to maximize long-term cumulative rewards, having a profound impact on the overall performance and convergence speed of the algorithm. In order to coordinate the rewards between clusters and prevent the controlled unmanned vehicles from converging to a local optimum, the present embodiment proposes a cross-cluster reward coordination mechanism.

[0045] In the autonomous vehicle system described in this embodiment, the various clusters cooperate, sharing common interests and the same goal: to improve traffic flow efficiency, safety, comfort, and energy conservation. However, during the learning process, some clusters may engage in self-interested behavior to maximize their cumulative rewards. For example, vehicles within a cluster might all travel at high speeds in the middle lane, hindering other clusters from changing lanes. This strategy may result in higher rewards for the clusters themselves, but the sum of rewards across all clusters may not be optimal, thus affecting the learning effect. Therefore, this embodiment proposes a cross-cluster reward coordination mechanism. When a cluster's reward is too low, the rewards of well-performing clusters are reduced, thereby guiding the entire system towards a more optimal policy space.

[0046] set up The sum of rewards obtained by all clusters, i.e.: (48) in, For clusters The total reward obtained. When the set When the elements of a set are not all positive or all negative, consider the set Sum of the absolute values ​​of the negative terms and the sum of the absolute values ​​of all terms : (49) (50) For sets The positive terms in the equation will have their values ​​determined according to... and And its own value will decrease to a certain extent, the degree of decrease being: (51) Therefore, the sum of rewards for all clusters It can be represented as: (52) S4. The collaborative decision-making instructions are distributed to each autonomous vehicle for execution through the central dispatch system.

[0047] Furthermore, the local reward function is composed of a weighted average of six sub-reward functions, wherein the sub-reward functions include speed, task, safety, acceleration, lane-changing frequency, and energy consumption.

[0048] Specifically, the central dispatch system distributes optimized strategy instructions to each autonomous vehicle, enabling safe, efficient, and robust decision-making in complex traffic scenarios.

[0049] Example Two This example takes a three-lane highway as an example. The decision goal of the autonomous driving system should also give priority to safety as the primary consideration, so that the CAVs under its control maintain a safe distance and relative speed from the preceding vehicle to prevent collisions; for the entire traffic flow, the average speed should be kept high enough to maintain a high level of overall efficiency; for individual vehicles, their driving efficiency should be improved by lane changing, acceleration and deceleration in a timely manner to efficiently reach the predetermined destination; and in order to meet the requirement of comfort, vehicles should not make lane changes, sudden acceleration and deceleration too frequently. These requirements interact with each other, forming a multi-objective optimization problem. Therefore, for individual vehicles within the cluster, this example considers six indicators, designs six different sub-reward functions for speed, task, safety, acceleration, lane changing frequency and energy consumption, and assigns different weight values to obtain the local reward function of the cluster.

[0050] First, the node feature matrix is established.

[0051] The characteristics of the CAVs and all observed vehicle nodes within the cluster at time are recorded. The characteristic vector of the vehicle node includes the state of the vehicle itself and the safety parameters relative to the surrounding vehicles, i.e. (53) Secondly, the adjacency matrix is constructed.

[0052] The intra-cluster adjacency matrix describes the interactions within the cluster, which can be represented as: (54) The inter-cluster adjacency matrix can be represented as: (55) Accordingly, the global adjacency matrix can be represented by all intra-cluster and inter-cluster adjacency matrices, i.e. (56) Then, QMIX defines a mixing network to combine the local Q values of each unmanned vehicle into a joint Q value: (57) where is the global state information, is the joint action of each unmanned vehicle, is the parameter of the mixing network, ​​denotes the set of all parameters. In addition, to ensure that each ego vehicle can independently select the optimal action based on the local Q value during training, QMIX adds a monotonicity constraint in the mixing network, i.e.: (58) The weight is redefined in this embodiment, i.e.: (59) When the model judges that the observation of some state is interfered by strong noise through the noise estimation network, the weight of these samples is reduced, and the influence of these samples on parameter update is reduced, so as to avoid the model from over-learning on the samples with high noise.

[0053] In this embodiment, the Huber loss function is defined as: (60) Wherein, is the TD error, and the threshold is defined as: (61) Wherein, represents the median, is a very small positive number, which is used to prevent the phenomenon of degradation in the training process.

[0054] Combining the above robust weighting mechanism, the overall Huber loss function is defined as: (62) Wherein, is the number of ego vehicles in the environment; is the network parameter of the local network; is a regularization coefficient, which can be used to control the trade-off between the loss term and the regularization term. Therefore, the network parameters of the mixing network and the local network can be optimized by minimizing the total loss : (63) (64) Wherein, and are the learning rates of the mixing network and the local network, respectively. In addition, in order to make the training smooth, the target mixing network parameter is updated by using a soft update strategy: (65) Wherein, is a soft update coefficient, which is used to ensure the gradual adjustment of the target network parameters.

[0055] Finally, in the robust QMIX algorithm proposed in this embodiment, the above-mentioned robust weighting mechanism is combined, and the robust Bellman operator is redefined as : (66) wherein, is a learning rate, is a noise-based weight, which takes a value in the range of .

[0056] (1) Speed reward function; The speed reward function has different speed requirements for autonomous vehicles in different lanes.

[0057] When the controlled autonomous vehicle is in the outermost lane, the speed reward function has the following form: (67) wherein, represents the speed of the autonomous vehicle, with the unit of m / s.

[0058] When the controlled vehicle is in the middle lane, the speed reward function has the following form: (68) When the autonomous vehicle is in the innermost lane, the speed reward function is: (69) (2) Task reward function; This embodiment sets three different task reward functions for three types of clusters in the environment.

[0059] Therefore, this embodiment adds speed to the independent variable of the task reward: (77) wherein, represent cluster 1, 2 and cluster 3 respectively, represent the outermost, middle and innermost lanes respectively.

[0060] (3) Safety reward function; This embodiment designs a safety reward function based on TTC (Time to Collision, collision time) to limit dangerous driving behavior. For the controlled autonomous vehicle in this embodiment, it will face the following three situations when driving: a) There is no other vehicle in front of its lane: ; b) There is a vehicle in front and the speed of the controlled autonomous vehicle is lower than the other: ; c) There is a vehicle in front and the speed of the controlled autonomous vehicle is higher than the other: At this time, the calculation formula of TTC is as follows: ; (78) wherein, is the distance between the two vehicles, is the relative speed of the two vehicles, which is set to 4 s, and when TTC is less than 4 s, the safety reward function will punish the autonomous vehicle, which is in the form of: (79) (4) Acceleration reward function; The acceleration reward function set in this embodiment is as follows: (80) wherein, is the threshold acceleration, in the scenario of this embodiment, is set to 4.5 m / s 2 .

[0061] (5) Lane changing reward function; The lane changing reward function set in this embodiment is as follows: (81) wherein, is the number of lane changes per second of the autonomous vehicle, is the allowable frequency, which is set to 0.2 in this embodiment.

[0062] (6) Energy consumption reward function; Firstly, according to the driving equation of the car, the driving resistance it receives when moving is calculated as follows: , from which the power required to drive the vehicle is , and the power required to be provided by the power battery pack is . Finally, the energy consumption of the vehicle in a unit time step is calculated as follows: (82) The energy consumption reward function designed in this scenario is calculated based on , which is: (83) Thus, the sum of the rewards obtained by each autonomous vehicle in the cluster is obtained, that is: (84) wherein, the first weight corresponding to the i-th the number of CAVs in the cluster the i-th the i-th the i-th the i-th item sub-reward function of the i-th CAV in the cluster.

[0063] Table 1 Robust QMIX Pseudocode Based on the above model design of robust QMIX, it is jointly trained with a graph attention network (GAT). By introducing a robust weighting mechanism and a Huber loss function, the model's policy convergence performance under observation noise and communication uncertainty conditions is effectively improved, thereby realizing stable cooperative control of the multi-vehicle system in a heterogeneous environment.

[0064] During the scene process, the actions output by each CAV cluster under the local graph structure are summarized to the central scheduling system at each time step, which generates and sends execution instructions to the vehicle controller and the environment interaction. The experience quadruple formed by the state, action, reward and next state in the interaction process is stored in the experience replay pool for subsequent robust unmanned vehicle reinforcement learning training. The process continues until the training round is completed or the algorithm performance reaches the preset standard.

[0065] Embodiment three The application also provides an automatic driving multi-vehicle cooperative control system based on robust QMIX, comprising: a cluster graph representation module for describing the local graph structure of vehicle interaction within the cluster and the global graph structure of inter-cluster interaction to represent the interaction relationship within and outside the vehicle cluster; a robust policy learning module for dynamically evaluating the environmental disturbance intensity through a noise perception mechanism, adjusting the value function weight using a weighted QMIX framework, and optimizing the policy in combination with a robust loss function; a cross-cluster reward coordination module for dynamically reducing the reward value of high-reward clusters to balance global optimization according to the reward distribution difference between clusters; a central scheduling system for distributing cooperative decision instructions to each autonomous vehicle for execution.

[0066] Embodiment four The embodiment also discloses a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of the method of embodiment one.

[0067] Embodiment five The embodiment further discloses a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method of the embodiment one.

[0068] The above merely illustrates the preferred embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of the changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A robust QMIX-based automatic driving multi-vehicle cooperative control method, characterized in that, The method comprises the following steps: a cluster graph representation module is used to describe the local graph structure of vehicle interaction within a cluster and the global graph structure of interaction between clusters to represent the interaction relationship within and outside the vehicle cluster; a robust strategy learning module is used to dynamically evaluate the environmental disturbance intensity through a noise perception mechanism, adjust the value function weight using a weighted QMIX framework, and optimize the strategy in combination with a robust loss function; a cross-cluster reward coordination module is used to dynamically reduce the reward value of high-reward clusters to balance global optimization according to the reward distribution difference between clusters; a central scheduling system is used to distribute collaborative decision-making instructions to each autonomous vehicle for execution.

2. The method of claim 1, wherein the cluster graph representation module comprises a node feature matrix and an adjacency matrix; the node feature matrix is used to record the vehicle motion state and safety parameters; the adjacency matrix is used to construct the connection topology within and between clusters through the relative position relationship.

3. The method of claim 1, wherein the noise perception mechanism quantifies the influence of environmental noise on the cluster through a noise estimation network and dynamically adjusts the mixed network weight coefficient.

4. The method of claim 1, wherein the weighted QMIX framework assigns lower weights to samples severely disturbed by noise and higher weights to reliable samples.

5. The method of claim 1, wherein the robust loss function uses a piecewise function design, which uses quadratic penalty for small errors and linear penalty for large errors.

6. The method of claim 1, wherein the cross-cluster reward coordination module comprises: calculating the algebraic sum of all cluster rewards; identifying negative reward clusters and calculating the total amount of punishment; proportionally reducing the reward value of positive reward clusters.

7. The method of claim 1, wherein the local reward function is composed of six sub-reward functions weighted together; the sub-reward functions include speed, task, safety, acceleration, lane changing frequency, and energy consumption.

8. A robust QMIX-based automatic driving multi-vehicle cooperative control system, characterized in that, comprise: a cluster graph representation module for describing the local graph structure of vehicle interaction within a cluster and the global graph structure of interaction between clusters to represent the interaction relationship within and outside the vehicle cluster; a robust strategy learning module for dynamically evaluating the environmental disturbance intensity through a noise perception mechanism, adjusting the value function weight using a weighted QMIX framework, and optimizing the strategy in combination with a robust loss function; a cross-cluster reward coordination module for dynamically reducing the reward value of high-reward clusters to balance global optimization according to the reward distribution difference between clusters; a central scheduling system for distributing collaborative decision-making instructions to each autonomous vehicle for execution.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the steps of the method of any one of claims 1-7.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by a processor to implement the steps of the method of any one of claims 1-7.

Images