Multi-agent cross-terminal collaborative control method and system based on deep reinforcement learning

Abstract

The present application relates to the technical field of artificial intelligence, and more particularly to a multi-agent cross-end collaborative control method and system based on deep reinforcement learning, which collects original perception data of each agent and converts them into a state vector set of unified dimensions, inputs a graph neural network to model topological relationships and generate enhanced state representations, generates candidate actions through a policy network and selects the action with the highest value to issue to the heterogeneous execution end for execution, collects environment state transition data and immediate reward signals and stores them in an experience replay buffer, samples historical interaction trajectories and introduces a contrastive learning loss function to distinguish and optimize positive collaborative behavior and negative conflict behavior, and synchronizes the updated policy network parameters to each agent execution end, which improves the efficiency and stability of multi-agent collaborative decision-making.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-agent cross-terminal collaborative control method and system based on deep reinforcement learning. Background Technology

[0002] In the field of multi-agent cooperative control, current conventional practices are mainly based on a centralized training and distributed execution framework. Specifically, during the training phase, each agent shares global information and learns cooperative policies through a centralized value network or policy network. During deployment, each agent makes independent decisions based on local observations. Common implementations include value decomposition methods and multi-agent reinforcement learning algorithms based on policy gradients (e.g., [example missing]). These methods typically assume that the interaction relationships between agents are statically fixed, and that the execution environment is isomorphic with negligible communication latency. During training, empirical data is uniformly sampled for global network updates, but the behavioral interaction patterns between agents are not explicitly distinguished.

[0003] Existing conventional approaches suffer from two main drawbacks. First, in dynamic topological environments, the interactions between agents can change drastically over time, making it difficult for the aforementioned methods to capture such non-stationary neighborhood structures. Static modeling prevents agents from allocating attention appropriately to relevant neighbors, and state representations lose crucial spatial and temporal correlation information, leading to decision biases or oscillations. Second, traditional experience replay mechanisms fail to differentiate between cooperative and conflicting behaviors. In scenarios with shared rewards or inconsistent individual rewards, positive cooperative trajectories and negative conflicting trajectories are mixed and stored in a buffer, with both contributing the same weight during parameter updates, making it difficult for policy networks to effectively reinforce beneficial cooperative patterns. This easily leads agents to get trapped in local optima, resulting in conflicting behaviors such as "resource grabbing" or "mutual interference." Especially in cross-platform heterogeneous execution environments, the differences in physical characteristics between different execution terminals amplify the negative effects of such conflicts. Summary of the Invention

[0004] This invention provides a multi-agent cross-terminal cooperative control method and system based on deep reinforcement learning, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a multi-agent cross-terminal cooperative control method based on deep reinforcement learning, comprising:

[0006] Collect raw perception data from each agent in a distributed execution environment, extract the state features of the raw perception data and convert them into a state vector of a unified dimension, forming a set of state vectors with additional timestamp identifiers and agent identifiers;

[0007] The set of state vectors is input into a graph neural network to model topological relationships. State information is propagated between agent nodes through a message passing mechanism to construct a dynamic interaction graph and generate an enhanced state representation that integrates neighborhood information.

[0008] Based on the enhanced state representation, a set of candidate actions is generated for each agent through a policy network, the action value of each candidate action is calculated, and the candidate action with the highest action value is selected as the action to be executed.

[0009] The execution actions are sent to the heterogeneous execution terminals corresponding to each intelligent agent for execution. Environmental state transition data and real-time reward signals after execution are collected and stored in the experience playback buffer.

[0010] Historical interaction trajectories are sampled from the experience replay buffer, and a contrastive learning loss function is introduced to differentiate between positive cooperative behavior and negative conflict behavior, so that the policy network strengthens cooperative mode and suppresses conflict mode during parameter update.

[0011] The updated policy network parameters are synchronized to the corresponding execution terminals of each agent.

[0012] Extract the state features of the original sensing data and convert them into state vectors of a unified dimension, forming a set of state vectors with additional timestamps and agent identifiers, including:

[0013] Multi-scale feature extraction is performed on the original sensing data, response values ​​are calculated at each scale level, and feature fusion is performed on the features at each scale. Nonlinear transformation is performed on the fusion result to generate original feature vectors representing the current state of each agent.

[0014] Obtain a preset unified dimension parameter. When the feature dimension of the original feature vector exceeds the unified dimension parameter, the original feature vector is mapped to a low-dimensional space through a dimension reduction projection matrix. When the feature dimension of the original feature vector is lower than the unified dimension parameter, zero-value elements are added to the end of the original feature vector to obtain a standard feature vector with unified dimension.

[0015] The standard feature vector is subjected to cross-agent batch normalization processing, and the corresponding timestamp is converted into periodic time code, the corresponding agent identifier is converted into embedded identity code, and concatenated to the standard feature vector to form a complete state vector carrying spatiotemporal identifier. The complete state vectors of each agent are collected to form a state vector set.

[0016] The set of state vectors is input into a graph neural network for topological relationship modeling. State information is propagated between agent nodes through a message passing mechanism, constructing a dynamic interaction graph and generating an enhanced state representation that incorporates neighborhood information, including:

[0017] Each state vector in the state vector set is used as a node feature in the graph structure. The similarity between corresponding nodes of any two agents is calculated. When the similarity exceeds the connection threshold, a connection edge is established between the corresponding node pairs. The similarity is used as the edge weight to construct the initial topology graph structure.

[0018] The neighborhood aggregation operation is performed on each node in the initial topology graph structure. The state vector of the current node and the state vectors of its neighboring nodes are weighted and combined according to the edge weights. The combined state vector is mapped to an intermediate representation vector through a learnable transformation matrix. A nonlinear activation function is applied to the intermediate representation vector and updated to the corresponding node to complete the single-layer message passing.

[0019] Iteratively execute multi-level message passing, and dynamically adjust the connection relationship and edge weight in the initial topology graph structure according to the changes in node state in each iteration to form a dynamic interaction graph;

[0020] After completing the preset number of iterations, the fusion state vector of each node in the dynamic interaction graph is read, and the fusion state vector is used as an enhanced state representation of the fusion neighborhood information.

[0021] Based on the enhanced state representation, a set of candidate actions is generated for each agent through a policy network. The action value of each candidate action is calculated, and the candidate action with the highest action value is selected as the action to be executed, including:

[0022] The enhanced state representation is input into the policy network, and multiple sets of action parameters are generated in parallel through the multi-head output layer of the policy network. Each head outputs the action mean vector and action variance vector respectively. Multiple independent Gaussian distributions are constructed based on the action mean vector and action variance vector output by each head. Corresponding candidate actions are sampled from each Gaussian distribution and summarized to form an initial candidate action set.

[0023] Perform action deduplication on the initial candidate action set, calculate the Euclidean distance between any two candidate actions, and when the distance is less than the preset deduplication threshold, calculate the sampling probability density value corresponding to the two candidate actions and retain the candidate action with the higher probability density value. Perform boundary correction on the deduplicated candidate actions, trim action components that exceed the upper bound of the action space to the upper bound value, and trim action components that are below the lower bound of the action space to the lower bound value to obtain the corrected candidate action set.

[0024] Each candidate action in the modified candidate action set is concatenated with the enhanced state representation to form multiple state-action pairing vectors. Each state-action pairing vector is input into the value network, which is mapped to a value estimate through a fully connected layer. The value estimates corresponding to each candidate action are sorted in descending order, and the candidate action with the highest value estimate is selected as the action to be executed.

[0025] Multiple sets of action parameters are generated in parallel through a multi-head output layer of the policy network. Each head outputs the action mean vector and action variance vector, including:

[0026] The enhanced state representation is input into the multi-head output layer of the policy network, which contains multiple parallel parameter generation heads. Each parameter generation head has a mean generation branch and a variance generation branch.

[0027] The mean generation branch maps the enhanced state representation to the original mean vector through a fully connected layer, applies a hyperbolic tangent activation function to the original mean vector and multiplies it by the amplitude coefficient of the action space to generate the action mean vector.

[0028] The variance generation branch performs a fully connected transformation on the enhanced state representation, uses a soft activation function to map the transformation result to the positive real number domain and applies a squaring operation, introduces a learnable lower bound parameter of variance and performs element-wise addition operation with the squaring operation result to generate an action variance vector;

[0029] Calculate the mean distance between the action mean vectors output by the parameter generator. When the mean distance is lower than the diversity threshold, add a perturbation vector that follows a zero-mean Gaussian distribution to the action mean vector. Combine the perturbation action mean vector with the corresponding action variance vector to form multiple sets of action parameters for parallel output.

[0030] Historical interaction trajectories are sampled from the experience replay buffer, and a contrastive learning loss function is introduced to discriminate between positive cooperative behavior and negative conflict behavior. This enables the policy network to strengthen cooperative patterns and suppress conflict patterns during parameter updates, including:

[0031] Historical interaction trajectory segments are extracted from the experience replay buffer by sliding sampling according to a time window, including the state observation sequence, action execution sequence and reward signal sequence of each agent;

[0032] Based on the historical interaction trajectory fragments, the joint entropy and mutual information quantization behavior coupling strength of each agent's actions are calculated. Fragments with decreasing joint entropy and increasing mutual information are identified as positive cooperative behavior patterns, while fragments with increasing joint entropy and decreasing mutual information are identified as negative conflict behavior patterns.

[0033] The enhanced state representation and action features corresponding to the positive cooperative behavior pattern are concatenated and encoded. The positive sample representation set is formed by comparing the encoder and projecting it onto the normalized cooperative representation space. The features corresponding to the negative conflict behavior pattern are projected onto the same space to form the negative sample representation set.

[0034] The cosine similarity mean between the representation corresponding to the current policy output action and the set of positive sample representations is calculated as a cooperative attraction term, and the cosine similarity mean between the representation and the set of negative sample representations is calculated as a conflict repulsion term, thus constructing a contrastive learning loss function.

[0035] The contrastive learning loss function and the temporal difference loss of the value network are fused to construct a joint training objective, and the parameters of the policy network and the value network are updated using an alternating optimization strategy.

[0036] A second aspect of this invention provides a multi-agent cross-terminal cooperative control system based on deep reinforcement learning, comprising:

[0037] The data acquisition unit is used to collect raw perception data from each agent in the distributed execution environment, extract the state features of the raw perception data and convert them into a state vector of a unified dimension, forming a set of state vectors with additional timestamp identifiers and agent identifiers.

[0038] The graph modeling unit is used to input the set of state vectors into the graph neural network to model the topological relationship, propagate state information between agent nodes through a message passing mechanism, construct a dynamic interaction graph, and generate an enhanced state representation that integrates neighborhood information.

[0039] The action selection unit is used to generate a set of candidate actions for each agent through a policy network based on the enhanced state representation, calculate the action value of each candidate action, and select the candidate action with the highest action value as the action to be executed.

[0040] The experience acquisition unit is used to send the execution action to the heterogeneous execution terminal corresponding to each intelligent agent for execution, collect the environmental state transition data and real-time reward signal after execution, and store them in the experience playback buffer.

[0041] The contrastive learning unit is used to sample historical interaction trajectories from the experience replay buffer and introduces a contrastive learning loss function to perform discriminative optimization of positive cooperative behavior and negative conflict behavior, so that the policy network strengthens cooperative mode and suppresses conflict mode during parameter update.

[0042] The parameter synchronization unit is used to synchronize the updated policy network parameters to the corresponding execution terminals of each agent.

[0043] A third aspect of the present invention provides an electronic device, comprising:

[0044] processor;

[0045] Memory used to store processor-executable instructions;

[0046] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0048] This invention, by introducing a unified-dimensional set of state vectors and dual identifiers of timestamps and agents, effectively eliminates interference caused by inconsistent data formats and timestamps between heterogeneous devices, significantly improving the accuracy and timeliness of multi-source sensing data fusion. Graph neural networks dynamically model the topological relationships between agents, fully exchanging neighborhood state information through message passing mechanisms to generate enhanced state representations containing global collaborative information, greatly improving the agent's ability to perceive complex environmental interaction patterns.

[0049] The policy network generates a set of candidate actions based on enhanced state representations and automatically selects the optimal action using an action value ranking mechanism, avoiding action selection conflicts and resource waste problems in traditional multi-agent systems. Closed-loop acquisition and storage of real-time reward signals and environmental state transition data provide a stable training sample foundation for policy iteration. An experience replay buffer stores historical interaction trajectories in an ordered manner, and combined with a contrastive learning loss function, explicitly distinguishes between positive cooperative behavior and negative conflict behavior, effectively strengthening the cooperative mode among agents and suppressing adversarial behavior. This allows the policy network to spontaneously tend towards the globally optimal cooperative policy during parameter updates.

[0050] The updated policy network parameters are synchronized to each heterogeneous execution terminal, enabling rapid deployment and adaptive adjustment of cross-platform collaborative control. This method demonstrates excellent scalability and robustness in large-scale multi-agent systems, significantly reducing communication overhead and decision latency, and is suitable for scenarios with stringent requirements for real-time performance and collaborative efficiency, such as unmanned swarm collaboration and intelligent warehouse scheduling. Through a dual mechanism of topological relationship modeling and comparative learning, the fundamental challenges of local optima and action oscillations in multi-agent collaborative control are fundamentally solved. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the multi-agent cross-end cooperative control method based on deep reinforcement learning, as described in an embodiment of the present invention.

[0052] Figure 2 This is a flowchart illustrating the method for generating enhanced state representations that incorporate neighborhood information, as described in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0055] Figure 1 This is a flowchart illustrating a multi-agent cross-terminal cooperative control method based on deep reinforcement learning, as described in an embodiment of the present invention. The multi-agent cross-terminal cooperative control method based on deep reinforcement learning provided in this embodiment includes:

[0056] Collect raw perception data from each agent in a distributed execution environment, extract the state features of the raw perception data and convert them into a state vector of a unified dimension, forming a set of state vectors with additional timestamp identifiers and agent identifiers;

[0057] The set of state vectors is input into a graph neural network to model topological relationships. State information is propagated between agent nodes through a message passing mechanism to construct a dynamic interaction graph and generate an enhanced state representation that integrates neighborhood information.

[0058] Based on the enhanced state representation, a set of candidate actions is generated for each agent through a policy network, the action value of each candidate action is calculated, and the candidate action with the highest action value is selected as the action to be executed.

[0059] The execution actions are sent to the heterogeneous execution terminals corresponding to each intelligent agent for execution. Environmental state transition data and real-time reward signals after execution are collected and stored in the experience playback buffer.

[0060] Historical interaction trajectories are sampled from the experience replay buffer, and a contrastive learning loss function is introduced to differentiate between positive cooperative behavior and negative conflict behavior, so that the policy network strengthens cooperative mode and suppresses conflict mode during parameter update.

[0061] The updated policy network parameters are synchronized to the corresponding execution terminals of each agent.

[0062] Extract the state features of the original sensing data and convert them into state vectors of a unified dimension, forming a set of state vectors with additional timestamps and agent identifiers, including:

[0063] Multi-scale feature extraction is performed on the original sensing data, response values ​​are calculated at each scale level, and feature fusion is performed on the features at each scale. Nonlinear transformation is performed on the fusion result to generate original feature vectors representing the current state of each agent.

[0064] Obtain a preset unified dimension parameter. When the feature dimension of the original feature vector exceeds the unified dimension parameter, the original feature vector is mapped to a low-dimensional space through a dimension reduction projection matrix. When the feature dimension of the original feature vector is lower than the unified dimension parameter, zero-value elements are added to the end of the original feature vector to obtain a standard feature vector with unified dimension.

[0065] The standard feature vector is subjected to cross-agent batch normalization processing, and the corresponding timestamp is converted into periodic time code, the corresponding agent identifier is converted into embedded identity code, and concatenated to the standard feature vector to form a complete state vector carrying spatiotemporal identifier. The complete state vectors of each agent are collected to form a state vector set.

[0066] When performing multi-scale feature extraction on raw sensory data, convolution or pooling operations are performed at multiple scale levels for heterogeneous data (including visual images, sensor readings, location coordinates, communication status, etc.) collected by different agents to capture sensory information of different granularities. At each scale level... Calculate the response value for this level. The shallow scale focuses on capturing local detailed features, while the deep scale focuses on extracting global semantic features. The response values ​​at each scale level are fused through feature concatenation or weighted summation to form a fused feature representation containing multi-granularity information. After fusion, a nonlinear transformation is performed on the fusion result, specifically using a multilayer perceptron structure. This transformation introduces nonlinear mapping capabilities through activation functions (such as ReLU or GELU), mapping the multi-scale fused features into original feature vectors representing the current state of each agent. ,in This serves as the agent's index number. This original feature vector integrates the agent's multi-dimensional perceptual information at the current moment, and can comprehensively describe the local environmental state in which the agent is located.

[0067] Because different agents have different sensor types, sampling frequencies, and data structures, the original feature vectors generated by each agent are different. Inconsistency in dimensionality can hinder subsequent graph neural network processing. Therefore, it is necessary to obtain a pre-defined, uniform dimensionality parameter. Perform dimension alignment on all original feature vectors. When the original feature vectors... Feature Dimensions Exceeding uniform dimension parameters At that time, through the dimension reduction projection matrix Will Mapping to a lower-dimensional space yields a dimension of The compressed feature vector, whose projection matrix is ​​learned through a training process, can retain as much key information as possible from the original features while reducing dimensionality. When the original feature vector... Feature Dimensions Below uniform dimension parameters At that time, End supplement The vector is expanded to the target dimension by adding zero-valued elements. After the above processing, the original feature vectors of all agents are transformed into standard feature vectors with uniform dimensions. This eliminates the dimensional inconsistency problem caused by sensor heterogeneity, laying the foundation for subsequent unified processing.

[0068] It is worth noting that the dimension reduction projection matrix The design needs to balance information retention and computational efficiency. In practical deployments, projection matrices can be trained separately for different types of agents to adapt to the inherent structural characteristics of different sensor data. For dimensionality expansion, while zero-padded operations are simple and efficient, it's crucial to ensure that subsequent normalization processes correctly handle these padding elements to avoid unnecessary interference with feature distribution. In some scenarios, learnable padding embedding vectors can be used instead of fixed zero-padded vectors to improve the flexibility of feature representation.

[0069] Standard feature vectors after dimensional unification Perform cross-agent batch normalization. Unlike regular batch normalization, this normalization operation uses the feature distribution of the entire agent group within the same batch as a benchmark to calculate the mean across the cross-agent dimension. and variance Furthermore, the standard feature vector of each agent is standardized, ensuring that the feature vectors of different agents fall within a similar numerical range. This operation effectively alleviates the feature scale imbalance problem caused by significant differences in the environments in which agents operate, and helps improve the stability and convergence speed of subsequent message passing in the graph neural network. The normalized feature vector is denoted as... .

[0070] Timestamp information and agent identification information are indispensable components of the state vector set. Both are converted into concatenable vector forms through specific encoding methods. For timestamps, a periodic time encoding scheme is used. Converted into a periodic encoding vector composed of sine and cosine functions. Specifically, the first... The dimensional components are respectively and ,in As a preset time scale base, This is the dimension for time encoding. Periodic time encoding enables the model to perceive the periodicity of time (such as the alternation of day and night, the rhythm of periodic tasks), while avoiding gradient problems caused by excessively large timestamp values. For agent identification, an embedded identity encoding scheme is adopted, where the unique identifier of each agent is encoded through a learnable embedding matrix. Mapped to a fixed-dimensional identity encoding vector The embedding matrix is ​​updated along with the policy network parameters during training, enabling the identity encoding to implicitly capture the differences in roles, abilities, or responsibilities among different agents.

[0071] The normalized feature vector Time-coded vector and identity encoding vector By concatenating along the feature dimensions, a complete state vector carrying spatiotemporal identifiers is obtained. This complete state vector not only contains the agent's perceptual state information but also carries temporal contextual information and the agent's individual identity information. This allows the graph neural network to distinguish the state contributions of different agents at different times during message passing, thereby achieving more accurate collaborative decision-making. The complete state vectors of all agents are collected to form a state vector set V. ,in This represents the total number of agents participating in the collaborative control. This set of state vectors is then fed into a graph neural network for topological relationship modeling, supporting the subsequent multi-agent collaborative decision-making process.

[0072] At the engineering implementation level, the multi-scale feature extraction module and the dimension alignment module can be independently configured for different types of agents to adapt to diverse data formats from heterogeneous execution ends. The statistics for cross-agent batch normalization are dynamically estimated based on mini-batch samples during the training phase, and then use the moving average statistics accumulated during training during the inference phase. This ensures that each agent can independently complete feature normalization in a distributed execution environment without real-time communication. (Dimensional parameters for time encoding and identity encoding are also mentioned.) and embedding matrix All dimensions are tuned as hyperparameters to achieve a balance between expressive power and computational overhead.

[0073] Figure 2 This is a flowchart illustrating the method for generating enhanced state representations that integrate neighborhood information, as described in an embodiment of the present invention. The method involves inputting the set of state vectors into a graph neural network for topological relationship modeling, propagating state information between agent nodes through a message passing mechanism, constructing a dynamic interaction graph, and generating enhanced state representations that integrate neighborhood information. The process includes:

[0074] Each state vector in the state vector set is used as a node feature in the graph structure. The similarity between corresponding nodes of any two agents is calculated. When the similarity exceeds the connection threshold, a connection edge is established between the corresponding node pairs. The similarity is used as the edge weight to construct the initial topology graph structure.

[0075] The neighborhood aggregation operation is performed on each node in the initial topology graph structure. The state vector of the current node and the state vectors of its neighboring nodes are weighted and combined according to the edge weights. The combined state vector is mapped to an intermediate representation vector through a learnable transformation matrix. A nonlinear activation function is applied to the intermediate representation vector and updated to the corresponding node to complete the single-layer message passing.

[0076] Iteratively execute multi-level message passing, and dynamically adjust the connection relationship and edge weight in the initial topology graph structure according to the changes in node state in each iteration to form a dynamic interaction graph;

[0077] After completing the preset number of iterations, the fusion state vector of each node in the dynamic interaction graph is read, and the fusion state vector is used as an enhanced state representation of the fusion neighborhood information.

[0078] Each state vector in the set of state vectors is used as a node feature in the graph structure, and a corresponding graph node is assigned to each agent. Let the _th_ state vector be the node feature of the graph structure. The first agent and the second The similarity between the agents is Calculated using the normalized inner product method: ,in and The first The first agent and the second The complete state vector of each agent. When Exceeding the preset connection threshold At the node With nodes Establish directed or undirected edges between them, and... Directly used as the edge weight of that edge. After traversing all agent node pairs, the initial topology graph structure is obtained. ,in For a set of nodes, This is the initial edge set. Connection threshold. The value of the threshold directly determines the sparsity of the graph: too low a threshold will introduce a large number of noisy edges, causing irrelevant information to interfere during message transmission; too high a threshold will result in an overly sparse graph structure, with some agent nodes becoming isolated and unable to participate in collaborative modeling. In actual deployment, the threshold can be adjusted based on the total number of agents. The coordination density with the task scenario is affected by the policy convergence speed on the validation set. Make adaptive adjustments.

[0079] After the initial topology graph structure is established, a neighborhood aggregation operation is performed on each node in the graph. For each node... Let its set of adjacent nodes be The aggregation process requires weighted summation of the state vectors of adjacent nodes according to their edge weights to obtain the neighborhood aggregation vector. The calculation method is as follows ,in Indicates the first Nodes during layer iteration The state vector, This corresponds to the edge weights. Then, the current node's own state vector is... Aggregate vector with neighborhood By splicing or combining, a combined vector can be formed. Through learnable transformation matrices right Perform a linear mapping to obtain the intermediate representation vector. ,in This is the bias vector for the corresponding layer. Apply a nonlinear activation function (such as ReLU or ELU) to obtain the updated node state vector. This completes a single-layer message passing process. The learnable transformation matrix... By not sharing parameters between different layers, message passing at each layer can capture the interaction relationships at different levels of abstraction.

[0080] After completing the first After layer message passing, the graph topology is dynamically adjusted based on changes in the state vectors of each node. Specifically, the topology of any node pair is recalculated. Similarity between them based on the updated state vectors And update the edge weights based on the new similarity values. If the updated similarity is lower than If the similarity between nodes that originally had no connecting edges exceeds a certain threshold, then remove the corresponding connecting edge; Then, add connecting edges. After the above dynamic adjustments, the graph structure changes from... Evolved into This forms a dynamic interactive graph. The core significance of this dynamic adjustment mechanism is that as message passing deepens, the state representation of nodes gradually incorporates information from their neighbors, and the actual semantic relevance between nodes may change. A static and fixed graph structure cannot accurately reflect this evolution, while the dynamic adjustment mechanism can ensure that message passing at each layer is based on the most accurate topological relationship at present, thereby improving the accuracy of collaborative modeling.

[0081] The message passing and dynamic topology adjustment process described above is executed iteratively, for a total of a preset number of layers. Second-rate. The value of determines the range of hops for information to propagate in the graph: when At one time, each node can only perceive the state of its directly adjacent nodes; when As the size increases, nodes can indirectly perceive the state of agents over a greater distance. However, Excessive scalar representation can lead to oversmoothing, causing the representations of different nodes to become too uniform and losing their individual distinctiveness. In practice, this is usually addressed by... The value is set between 2 and 4, and combined with a residual join mechanism to alleviate the oversmoothing problem, that is, during each layer update, the value is... and Weighted stacking preserves some information from the original node features. The introduction of residual connections also facilitates the backpropagation of gradients in multi-layer graph neural networks, accelerating the convergence speed of the policy network.

[0082] After completing all After layer iteration, read the dynamic interaction graph. The final fused state vector of each node , take it as the first An enhanced state representation that integrates neighborhood information for each agent. This is compared to the original state vector. In comparison, enhanced state representation It not only includes the agent's own perception information, timestamp information, and identity information, but also integrates the state information of neighboring agents within the global topology through multi-layer message passing, possessing stronger context awareness capabilities. In multi-agent cooperative control scenarios, this enhanced representation enables the policy network to fully consider the current state and behavioral intentions of other agents when generating candidate actions for each agent, thereby reducing the probability of action conflicts between multiple agents and improving overall cooperative efficiency. The dimensions of the enhanced state representation and To maintain consistency, by the learnable transformation matrix The output dimension is determined so that subsequent policy networks can directly receive it without additional dimension alignment operations.

[0083] Based on the enhanced state representation, a set of candidate actions is generated for each agent through a policy network. The action value of each candidate action is calculated, and the candidate action with the highest action value is selected as the action to be executed, including:

[0084] The enhanced state representation is input into the policy network, and multiple sets of action parameters are generated in parallel through the multi-head output layer of the policy network. Each head outputs the action mean vector and action variance vector respectively. Multiple independent Gaussian distributions are constructed based on the action mean vector and action variance vector output by each head. Corresponding candidate actions are sampled from each Gaussian distribution and summarized to form an initial candidate action set.

[0085] Perform action deduplication on the initial candidate action set, calculate the Euclidean distance between any two candidate actions, and when the distance is less than the preset deduplication threshold, calculate the sampling probability density value corresponding to the two candidate actions and retain the candidate action with the higher probability density value. Perform boundary correction on the deduplicated candidate actions, trim action components that exceed the upper bound of the action space to the upper bound value, and trim action components that are below the lower bound of the action space to the lower bound value to obtain the corrected candidate action set.

[0086] Each candidate action in the modified candidate action set is concatenated with the enhanced state representation to form multiple state-action pairing vectors. Each state-action pairing vector is input into the value network, which is mapped to a value estimate through a fully connected layer. The value estimates corresponding to each candidate action are sorted in descending order, and the candidate action with the highest value estimate is selected as the action to be executed.

[0087] The enhanced state representation, generated by the message passing mechanism of the graph neural network, carries the fusion information of each agent and its neighboring nodes, possessing strong global perception capabilities. This enhanced state representation is used as input to the policy network. The encoding layer of the policy network performs a nonlinear transformation on the input vector to extract high-level semantic features, which then enters the multi-head output layer. The multi-head output layer contains several parallel output heads, each independently outputting a set of action mean vectors. With action variance vector ,in This refers to the index number of the output header. Action mean vector. This describes the central location of the action distribution suggested by the head, and the action variance vector. This describes the degree of uncertainty in the distribution. Based on each output head... and Construct an independent Gaussian distribution A candidate action vector is generated by randomly sampling from the Gaussian distribution. The candidate action vectors sampled from all output heads are aggregated to form an initial candidate action set. ,in This represents the total number of output heads. The multi-head parallel output design allows the candidate action set to cover multiple potential regions of the action space, avoiding the policy from getting trapped in local optima, while providing a sufficient basis for comparison in subsequent value evaluation.

[0088] The initial candidate action set may contain highly similar duplicate actions that are semantically almost equivalent. Including all of these in the value evaluation stage would cause unnecessary computational redundancy and lead to evaluation results biased towards a specific local region. Therefore, an action deduplication operation is performed on the initial candidate action set. Specifically, for any two candidate actions in the set... and ( ), calculate the Euclidean distance between them .when Less than the preset deduplication threshold If two candidate actions are deemed too similar, the one with higher quality should be retained. The criterion is to calculate the probability density values ​​of each candidate action under its corresponding Gaussian distribution, i.e. and Candidate actions with higher probability density values ​​are retained, while those with lower probability density values ​​are discarded. Higher probability density values ​​indicate that the candidate action is closer to the action distribution center currently proposed by the policy network, possessing higher policy confidence, and therefore are preferred. The deduplication operation is performed iteratively until the Euclidean distance between any two candidate actions in the set is not less than [value missing]. This yields a set of candidate actions after deduplication.

[0089] After deduplication, a boundary correction operation is performed on each action vector in the candidate action set to ensure that all candidate actions fall within the legal action space. Each dimension of the action space... All have upper bound values. and lower bound value For candidate action vectors The Each component ,like Then cut it into ;like Then cut it into Otherwise, the original value remains unchanged. The boundary correction operation is performed independently on each dimension of the action vector, and the corrected candidate action vector... All components satisfy the action space constraints. All candidate actions that have undergone boundary correction are aggregated to form a corrected candidate action set. The candidate actions in this set have undergone deduplication to ensure diversity and boundary correction to ensure legitimacy, and can directly proceed to the value assessment stage.

[0090] The value assessment phase uses a value network to score each candidate action in the revised candidate action set. To enable the value network to simultaneously perceive the joint information of the current environmental state and the candidate actions, each candidate action... With enhanced state representation Perform vector concatenation to form state-action pairing vectors. ,in This represents a vector concatenation operation. It pairs vectors representing different states and actions. The inputs are fed into a value network, which consists of several stacked fully connected layers. The input vectors undergo a layer-by-layer nonlinear transformation, and finally, the output layer maps them into a scalar value estimate. This indicates that the candidate action will be executed in the current state. The expected cumulative return. The estimated value of all candidate actions in the revised candidate action set. ( The candidate actions (after deduplication and correction) are sorted in descending order, and the candidate action with the highest value estimate is selected as the final action to be executed. ,in .

[0091] The aforementioned action generation and selection process organically combines the diverse exploration capabilities of the policy network with the precise evaluation capabilities of the value network. Multi-head Gaussian sampling ensures the breadth of the candidate action set, deduplication and boundary correction guarantee the quality and legitimacy of the candidate actions, and the joint evaluation of the value network ensures that the final action has the optimal expected value in the current state. In heterogeneous multi-agent collaborative scenarios, the action space dimensions and constraints of different agents may differ, and the upper and lower bound parameters in the boundary correction operation... and Each agent is configured separately to adapt to the action space characteristics of different execution terminals, ensuring that the action instructions issued to each heterogeneous execution terminal conform to their physical or logical constraints.

[0092] Multiple sets of action parameters are generated in parallel through a multi-head output layer of the policy network. Each head outputs the action mean vector and action variance vector, including:

[0093] The enhanced state representation is input into the multi-head output layer of the policy network, which contains multiple parallel parameter generation heads. Each parameter generation head has a mean generation branch and a variance generation branch.

[0094] The mean generation branch maps the enhanced state representation to the original mean vector through a fully connected layer, applies a hyperbolic tangent activation function to the original mean vector and multiplies it by the amplitude coefficient of the action space to generate the action mean vector.

[0095] The variance generation branch performs a fully connected transformation on the enhanced state representation, uses a soft activation function to map the transformation result to the positive real number domain and applies a squaring operation, introduces a learnable lower bound parameter of variance and performs element-wise addition operation with the squaring operation result to generate an action variance vector;

[0096] Calculate the mean distance between the action mean vectors output by the parameter generator. When the mean distance is lower than the diversity threshold, add a perturbation vector that follows a zero-mean Gaussian distribution to the action mean vector. Combine the perturbation action mean vector with the corresponding action variance vector to form multiple sets of action parameters for parallel output.

[0097] After passing through multiple layers of message passing in the graph neural network, the augmented state representation carries rich neighborhood fusion information and is input into the multi-head output layer of the policy network. This layer contains multiple parameter generators operating in parallel. Each parameter generator is structurally independent, with its own mean generation branch and variance generation branch. The two branches share the same augmented state representation input, but each performs different mapping tasks through independent learnable parameters. This design of multiple parameter generators operating in parallel allows the policy network to explore multiple regions in the action space at the same decision time, thus providing a sufficient foundation for the diversity selection of subsequent candidate actions.

[0098] The calculation process for the mean-generating branch is as follows: The enhanced state representation vector is... The input is fed into a fully connected layer, which maps it back to the original mean vector through a linear transformation. ,in Generate the index number of the parameter head. Original mean vector. The values ​​of each component are unconstrained and will exceed the legal range of the action space. Therefore, a hyperbolic tangent activation function needs to be applied to compress them to a smaller value range. The interval, multiplied by the amplitude coefficient of the motion space. This ensures that the output value range matches the amplitude range of the actual motion space. Amplitude coefficient Pre-defined based on the action space definition of the specific task, for example, when the action space is... At that time, after the hyperbolic tangent activation and the product of the amplitude coefficient, the action mean vector Each component naturally falls within the legal range. This design avoids generating a large number of out-of-bounds actions during the mean generation stage, reducing the processing burden of subsequent boundary correction steps.

[0099] The calculation logic of the variance generation branch is independent of that of the mean generation branch. This enhances the state representation vector. The intermediate vector is obtained by feeding it into another fully connected layer and performing a linear transformation. Since the variance value must be a positive real number, directly using linear output or the ReLU activation function both suffer from numerical instability: linear output can produce negative values, and ReLU's gradient becomes zero when the input is negative, causing training to stall. Therefore, a soft-applied activation function is used. The soft-addition function's output is always positive and globally differentiable, ensuring smooth gradient propagation. Squaring the output of the soft-addition activation function further expands the dynamic range of the variance, allowing the policy network to flexibly adjust the exploration intensity. To prevent the variance from approaching zero in the early stages of training or under certain conditions, leading to sampling degradation, a learnable lower bound parameter for variance is introduced. Calculate the result of the squaring operation and... Perform element-wise addition to obtain the action variance vector. . As a network parameter participating in gradient updates, it can adaptively adjust according to task characteristics, tending to a smaller value in tasks requiring precise control and a larger value in tasks requiring extensive exploration.

[0100] After generating the mean and variance vectors for each parameter generator, it is necessary to evaluate and adjust the diversity of the multi-head output. Calculate the pairwise Euclidean distance between each pair of the action mean vectors from all parameter generators, and average all distance values ​​to obtain the mean distance between the multi-head action mean vectors. .Will With the preset diversity threshold Comparison: When When this occurs, it indicates that the parameter generation heads have formed a sufficiently dispersed coverage in the action space, requiring no additional intervention; when If this occurs, it indicates that the mean vector of each head's output is too concentrated, and the exploration advantage of the multi-head design has not been fully utilized. In this case, it is necessary to apply a perturbation to the mean vector of the action.

[0101] The perturbation mechanism is implemented as follows: For each parameter, generate the head's action mean vector. Superimpose a perturbation vector that follows a zero-mean Gaussian distribution. ,in , For disturbance intensity parameters, It is the identity matrix. The dimension of the perturbation vector is consistent with that of the action mean vector. The perturbation action mean vector is denoted as... ,Right now Disturbance intensity parameters It can be set to a fixed value, or it can be adjusted according to... and The perturbation intensity is adjusted adaptively based on the distance from the mean; the larger the distance, the stronger the perturbation, ensuring that the dispersion of the mean vectors of each head after perturbation meets the diversity requirements. It is important to note that the perturbation operation is triggered only when the distance from the mean is below a threshold, and the perturbation vector is sampled independently during each forward inference and does not participate in gradient backpropagation to avoid introducing additional training instability.

[0102] Perturbed action mean vector Generate the action variance vector of the head output with the corresponding parameters. Combine to form the first Each parameter generator outputs complete motion parameters. The motion parameter outputs from all parameter generators are aggregated to form multiple parallel sets of motion parameters, each set corresponding to a Gaussian distribution. Candidate motion vectors are then generated by sampling from these Gaussian distributions. The process then proceeds to deduplication, boundary correction, and value assessment, ultimately selecting the action to be executed. This combination of a multi-head parallel generation mechanism and a diversity control mechanism enables the policy network to systematically cover multiple potentially high-quality regions in the action space while maintaining training stability, significantly improving the decision-making quality and robustness in multi-agent collaborative control scenarios.

[0103] Historical interaction trajectories are sampled from the experience replay buffer, and a contrastive learning loss function is introduced to discriminate between positive cooperative behavior and negative conflict behavior. This enables the policy network to strengthen cooperative patterns and suppress conflict patterns during parameter updates, including:

[0104] Historical interaction trajectory segments are extracted from the experience replay buffer by sliding sampling according to a time window, including the state observation sequence, action execution sequence and reward signal sequence of each agent;

[0105] Based on the historical interaction trajectory fragments, the joint entropy and mutual information quantization behavior coupling strength of each agent's actions are calculated. Fragments with decreasing joint entropy and increasing mutual information are identified as positive cooperative behavior patterns, while fragments with increasing joint entropy and decreasing mutual information are identified as negative conflict behavior patterns.

[0106] The enhanced state representation and action features corresponding to the positive cooperative behavior pattern are concatenated and encoded. The positive sample representation set is formed by comparing the encoder and projecting it onto the normalized cooperative representation space. The features corresponding to the negative conflict behavior pattern are projected onto the same space to form the negative sample representation set.

[0107] The cosine similarity mean between the representation corresponding to the current policy output action and the set of positive sample representations is calculated as a cooperative attraction term, and the cosine similarity mean between the representation and the set of negative sample representations is calculated as a conflict repulsion term, thus constructing a contrastive learning loss function.

[0108] The contrastive learning loss function and the temporal difference loss of the value network are fused to construct a joint training objective, and the parameters of the policy network and the value network are updated using an alternating optimization strategy.

[0109] When extracting historical interaction trajectories from the experience replay buffer, a time window sliding sampling mechanism is used, with a fixed step size sliding the time window within the buffer, and the extraction length for each step being [length missing]. The sampling process involves continuous trajectory segments. Each segment contains the state observation sequence, action execution sequence, and corresponding reward signal sequence of each agent within the window. The sliding step size is set to half the length of the time window to preserve a certain overlap between adjacent sampling segments, ensuring that sparse cooperative or conflicting events are not missed due to sampling boundary issues. For trajectory segments stored earlier in the buffer, priority weights are introduced for weighted sampling. The priority is determined based on the mean absolute value of the reward signal within the segment; segments with larger absolute reward values ​​are assigned higher sampling probabilities, thus ensuring that the training process includes a sufficient number of information-intensive interaction samples.

[0110] Based on extracted historical interaction trajectory fragments, the behavioral coupling strength between each agent is quantified. For within a window... The action execution sequence of each agent is used to discretize the action vector of each agent into action category labels. The joint probability distribution of all agent actions is calculated, and then the joint entropy is obtained. The sum of the marginal entropy of the actions of each agent Mutual information content pass The calculation yielded, where It is the sum of the edge entropy of the actions of each agent. Joint entropy. When the actions of the agents within a trajectory segment are highly coordinated, the joint entropy... It tends to decrease, while mutual information A tendency to increase indicates a strong, positively cooperative relationship between the actions of the agents; conversely, when the actions of the agents interfere with each other or resource competition occurs, the joint entropy decreases. Increase mutual information A decrease indicates that the behavioral coupling exhibits negative conflict characteristics. This is based on a preset threshold for joint entropy change. With mutual information change threshold As a criterion, the reduction in joint entropy must simultaneously exceed [a certain threshold]. And the increase in mutual information exceeds The trajectory segments are marked as positive cooperative behavior patterns, while simultaneously satisfying the condition that the increase in joint entropy exceeds [a certain value]. And the reduction in mutual information exceeds The segments that are negative conflict behavior patterns are labeled as such, while the remaining segments are considered neutral samples and do not participate in the construction of the contrastive learning loss.

[0111] For the identified positive cooperative behavior patterns and negative conflict behavior patterns, the enhanced state representation vector and action feature vector at the corresponding time points are extracted respectively. These two vectors are then concatenated along the feature dimension to obtain the original joint feature vector of the behavior pattern. This joint feature vector is input into a contrastive encoder, which consists of two fully connected layers using ReLU activation. The output layer is followed by L2 normalization to project the features onto a cooperative representation space on a unit hypersphere. The projection results corresponding to the positive cooperative behavior patterns constitute the positive sample representation set. The projection results corresponding to the negative conflict behavior patterns constitute the negative sample representation set. The normalization operation ensures that the magnitude of all representation vectors is 1, making subsequent cosine similarity calculations equivalent to inner product operations, thus providing good numerical stability. The parameters of the contrast encoder are updated along with the policy network during joint training, eliminating the need for a separate learning rate scheduling strategy.

[0112] The current policy network outputs an action given an augmented state representation. After concatenating the augmented state representation corresponding to the action with the action features, it is projected onto the cooperative representation space by a contrast encoder to obtain the current policy representation vector. .calculate With positive sample representation set The cosine similarity of all positive sample representation vectors is used to obtain the co-attractive term. ;calculate With negative sample representation set The cosine similarity of all negative sample representation vectors is used to obtain the conflict rejection term. Contrastive learning loss function Defined as: ,in The weighting coefficients for the exclusion terms, This is a preset rejection interval threshold, used to control the minimum distance boundary between the current policy representation and the negative sample representation. When Below When the exclusion term does not contribute to the gradient, unnecessary optimization pressure is avoided on the strategy that is already far enough away from the negative samples.

[0113] Temporal difference loss of value network Based on the Bellman equation, the target value is calculated using state transition data sampled from the empirical replay buffer. The target value is determined by the immediate reward signal and a discount factor. The weighted next-time value estimates are superimposed, and the temporal difference error is expressed as mean square error. Joint training objective. The contrastive learning loss and the temporal difference loss are fused using weighted coefficients: ,in To compare the fusion weight coefficients of the learning loss, the influence of the comparison optimization signal on the overall parameter update direction is controlled.

[0114] An alternating optimization strategy is employed to update the parameters of the policy network and the value network separately. In each training iteration, the policy network parameters are initially fixed, and only temporal difference loss is used. Gradient updates are performed on the value network to improve the accuracy of value estimation; subsequently, the value network parameters are fixed, and a joint training objective is used. Gradient updates are performed on the policy network and the contrastive encoder. Contrastive learning loss guides the policy network to converge towards cooperative behavior patterns in the parameter space, while simultaneously distancing it from conflicting behavior patterns. The ratio of alternating optimization rounds is set to one per update of the value network. The policy network will be updated once after this. The value is dynamically adjusted according to the convergence speed of the value network, and is usually initialized to 2. This alternation mechanism effectively avoids the gradient interference problem caused by the synchronous update of the policy network and the value network, while ensuring that the contrastive learning signal can play a full role in the policy optimization stage. Ultimately, it enables the policy network to continuously strengthen the cooperative mode and effectively suppress the conflict mode in the multi-agent cross-end cooperative control task.

[0115] A second aspect of this invention provides a multi-agent cross-terminal cooperative control system based on deep reinforcement learning, comprising:

[0116] The data acquisition unit is used to collect raw perception data from each agent in the distributed execution environment, extract the state features of the raw perception data and convert them into a state vector of a unified dimension, forming a set of state vectors with additional timestamp identifiers and agent identifiers.

[0117] The graph modeling unit is used to input the set of state vectors into the graph neural network to model the topological relationship, propagate state information between agent nodes through a message passing mechanism, construct a dynamic interaction graph, and generate an enhanced state representation that integrates neighborhood information.

[0118] The action selection unit is used to generate a set of candidate actions for each agent through a policy network based on the enhanced state representation, calculate the action value of each candidate action, and select the candidate action with the highest action value as the action to be executed.

[0119] The experience acquisition unit is used to send the execution action to the heterogeneous execution terminal corresponding to each intelligent agent for execution, collect the environmental state transition data and real-time reward signal after execution, and store them in the experience playback buffer.

[0120] The contrastive learning unit is used to sample historical interaction trajectories from the experience replay buffer and introduces a contrastive learning loss function to perform discriminative optimization of positive cooperative behavior and negative conflict behavior, so that the policy network strengthens cooperative mode and suppresses conflict mode during parameter update.

[0121] The parameter synchronization unit is used to synchronize the updated policy network parameters to the corresponding execution terminals of each agent.

[0122] A third aspect of the present invention provides an electronic device, comprising:

[0123] processor;

[0124] Memory used to store processor-executable instructions;

[0125] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0126] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0127] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent cross-terminal cooperative control method based on deep reinforcement learning, characterized in that, include: Collect raw perception data from each agent in a distributed execution environment, extract the state features of the raw perception data and convert them into a state vector of a unified dimension, forming a set of state vectors with additional timestamp identifiers and agent identifiers; The set of state vectors is input into a graph neural network to model topological relationships. State information is propagated between agent nodes through a message passing mechanism to construct a dynamic interaction graph and generate an enhanced state representation that integrates neighborhood information. Based on the enhanced state representation, a set of candidate actions is generated for each agent through a policy network, the action value of each candidate action is calculated, and the candidate action with the highest action value is selected as the action to be executed. The execution actions are sent to the heterogeneous execution terminals corresponding to each intelligent agent for execution, and the environmental state transition data and real-time reward signals after execution are collected and stored in the experience playback buffer. Historical interaction trajectories are sampled from the experience replay buffer, and a contrastive learning loss function is introduced to differentiate between positive cooperative behavior and negative conflict behavior, so that the policy network strengthens cooperative mode and suppresses conflict mode during parameter update. The updated policy network parameters are synchronized to the corresponding execution terminals of each agent.

2. The method according to claim 1, characterized in that, Extract the state features of the original sensing data and convert them into state vectors of a unified dimension, forming a set of state vectors with additional timestamps and agent identifiers, including: Multi-scale feature extraction is performed on the original sensing data, response values ​​are calculated at each scale level, and feature fusion is performed on the features at each scale. Nonlinear transformation is performed on the fusion result to generate original feature vectors representing the current state of each agent. Obtain a preset unified dimension parameter. When the feature dimension of the original feature vector exceeds the unified dimension parameter, the original feature vector is mapped to a low-dimensional space through a dimension reduction projection matrix. When the feature dimension of the original feature vector is lower than the unified dimension parameter, zero-value elements are added to the end of the original feature vector to obtain a standard feature vector with unified dimension. The standard feature vector is subjected to cross-agent batch normalization processing, and the corresponding timestamp is converted into periodic time code, the corresponding agent identifier is converted into embedded identity code, and concatenated to the standard feature vector to form a complete state vector carrying spatiotemporal identifier. The complete state vectors of each agent are collected to form a state vector set.

3. The method according to claim 1, characterized in that, The set of state vectors is input into a graph neural network for topological relationship modeling. State information is propagated between agent nodes through a message passing mechanism, constructing a dynamic interaction graph and generating an enhanced state representation that incorporates neighborhood information, including: Each state vector in the state vector set is used as a node feature in the graph structure. The similarity between corresponding nodes of any two agents is calculated. When the similarity exceeds the connection threshold, a connection edge is established between the corresponding node pairs. The similarity is used as the edge weight to construct the initial topology graph structure. The neighborhood aggregation operation is performed on each node in the initial topology graph structure. The state vector of the current node and the state vectors of its neighboring nodes are weighted and combined according to the edge weights. The combined state vector is mapped to an intermediate representation vector through a learnable transformation matrix. A nonlinear activation function is applied to the intermediate representation vector and updated to the corresponding node to complete the single-layer message passing. Iteratively execute multi-level message passing, and dynamically adjust the connection relationship and edge weight in the initial topology graph structure according to the changes in node state in each iteration to form a dynamic interaction graph; After completing the preset number of iterations, the fusion state vector of each node in the dynamic interaction graph is read, and the fusion state vector is used as an enhanced state representation of the fusion neighborhood information.

4. The method according to claim 1, characterized in that, Based on the enhanced state representation, a set of candidate actions is generated for each agent through a policy network. The action value of each candidate action is calculated, and the candidate action with the highest action value is selected as the action to be executed, including: The enhanced state representation is input into the policy network, and multiple sets of action parameters are generated in parallel through the multi-head output layer of the policy network. Each head outputs the action mean vector and action variance vector respectively. Multiple independent Gaussian distributions are constructed based on the action mean vector and action variance vector output by each head. Corresponding candidate actions are sampled from each Gaussian distribution and summarized to form an initial candidate action set. Perform action deduplication on the initial candidate action set, calculate the Euclidean distance between any two candidate actions, and when the distance is less than the preset deduplication threshold, calculate the sampling probability density value corresponding to the two candidate actions and retain the candidate action with the higher probability density value. Perform boundary correction on the deduplicated candidate actions, trim action components that exceed the upper bound of the action space to the upper bound value, and trim action components that are below the lower bound of the action space to the lower bound value to obtain the corrected candidate action set. Each candidate action in the modified candidate action set is concatenated with the enhanced state representation to form multiple state-action pairing vectors. Each state-action pairing vector is input into the value network, which is mapped to a value estimate through a fully connected layer. The value estimates corresponding to each candidate action are sorted in descending order, and the candidate action with the highest value estimate is selected as the action to be executed.

5. The method according to claim 4, characterized in that, Multiple sets of action parameters are generated in parallel through a multi-head output layer of the policy network. Each head outputs the action mean vector and action variance vector, including: The enhanced state representation is input into the multi-head output layer of the policy network, which contains multiple parallel parameter generation heads. Each parameter generation head has a mean generation branch and a variance generation branch. The mean generation branch maps the enhanced state representation to the original mean vector through a fully connected layer, applies a hyperbolic tangent activation function to the original mean vector and multiplies it by the amplitude coefficient of the action space to generate the action mean vector. The variance generation branch performs a fully connected transformation on the enhanced state representation, uses a soft activation function to map the transformation result to the positive real number domain and applies a squaring operation, introduces a learnable lower bound parameter of variance and performs element-wise addition operation with the squaring operation result to generate an action variance vector; Calculate the mean distance between the action mean vectors output by the parameter generator. When the mean distance is lower than the diversity threshold, add a perturbation vector that follows a zero-mean Gaussian distribution to the action mean vector. Combine the perturbation action mean vector with the corresponding action variance vector to form multiple sets of action parameters for parallel output.

6. The method according to claim 1, characterized in that, Historical interaction trajectories are sampled from the experience replay buffer, and a contrastive learning loss function is introduced to discriminate between positive cooperative behavior and negative conflict behavior. This enables the policy network to strengthen cooperative patterns and suppress conflict patterns during parameter updates, including: Historical interaction trajectory segments are extracted from the experience replay buffer by sliding sampling according to a time window, including the state observation sequence, action execution sequence and reward signal sequence of each agent; Based on the historical interaction trajectory fragments, the joint entropy and mutual information quantization behavior coupling strength of each agent's actions are calculated. Fragments with decreasing joint entropy and increasing mutual information are identified as positive cooperative behavior patterns, while fragments with increasing joint entropy and decreasing mutual information are identified as negative conflict behavior patterns. The enhanced state representation and action features corresponding to the positive cooperative behavior pattern are concatenated and encoded. The positive sample representation set is formed by comparing the encoder and projecting it onto the normalized cooperative representation space. The features corresponding to the negative conflict behavior pattern are projected onto the same space to form the negative sample representation set. The cosine similarity mean between the representation corresponding to the current policy output action and the set of positive sample representations is calculated as a cooperative attraction term, and the cosine similarity mean between the representation and the set of negative sample representations is calculated as a conflict repulsion term, thus constructing a contrastive learning loss function. The contrastive learning loss function and the temporal difference loss of the value network are fused to construct a joint training objective, and the parameters of the policy network and the value network are updated using an alternating optimization strategy.

7. A multi-agent cross-terminal cooperative control system based on deep reinforcement learning, used to implement the method as described in any one of claims 1-6, characterized in that, include: The data acquisition unit is used to collect raw perception data from each agent in the distributed execution environment, extract the state features of the raw perception data and convert them into a state vector of a unified dimension, forming a set of state vectors with additional timestamp identifiers and agent identifiers. The graph modeling unit is used to input the set of state vectors into the graph neural network to model the topological relationship, propagate state information between agent nodes through a message passing mechanism, construct a dynamic interaction graph, and generate an enhanced state representation that integrates neighborhood information. The action selection unit is used to generate a set of candidate actions for each agent through a policy network based on the enhanced state representation, calculate the action value of each candidate action, and select the candidate action with the highest action value as the action to be executed. The experience acquisition unit is used to send the execution action to the heterogeneous execution terminal corresponding to each intelligent agent for execution, collect the environmental state transition data and real-time reward signal after execution, and store them in the experience playback buffer. The contrastive learning unit is used to sample historical interaction trajectories from the experience replay buffer and introduces a contrastive learning loss function to perform discriminative optimization of positive cooperative behavior and negative conflict behavior, so that the policy network strengthens cooperative mode and suppresses conflict mode during parameter update. The parameter synchronization unit is used to synchronize the updated policy network parameters to the corresponding execution terminals of each agent.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images