Three-pillar driven cloud collaborative multi-agent deep reinforcement learning decision system

Abstract

The present application relates to a kind of three pillar drives cloud coordination multi-agent deep reinforcement learning decision system, belong to agent decision field.The system of the present application includes: end side execution module, cloud side teacher guidance module, cloud side action cooperation module, cloud side parameter sharing module and cloud side global optimization module.The present application utilizes cloud coordination mode, establishes dynamic cooperation graph based on self-attention mechanism, utilizes federal cooperation mechanism to realize the network parameter sharing of same task area, adopts width learning optimization strategy network, establishes the three pillar architecture of dynamic cooperation, parameter sharing and global optimization to guarantee the convergence of multi-agent deep reinforcement learning;The present application can accelerate the convergence speed of multi-agent deep reinforcement learning, enhance the coordination between agent, improve the stability of system.

Description

Technical Field

[0001] This invention belongs to the field of intelligent agent decision-making, specifically relating to a three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system. Background Technology

[0002] As the number of agents increases, the state dimension of the environment increases exponentially, making algorithm convergence difficult. Furthermore, considering the interactions between multiple agents and the constant changes in the global state, a change in one agent's policy leads to adjustments in the policies of all other agents. Therefore, the entire process of multi-agent training, collaboration, and task execution becomes more complex, and must be integrated with specific application scenarios and requirements, facing numerous problems and challenges. In particular, when the number of agents is large and the number of rounds and the duration of task execution are long, the current mainstream "centralized training-distributed execution" architecture of multi-agent deep reinforcement learning techniques is prone to problems such as the "curse of dimensionality" in the state space and the "curse of history" in iterative updates, urgently requiring optimization and improvement in practical applications. Moreover, in most real-world scenarios, the agent's vision and perception are limited; an agent can hardly obtain all the state and behavioral information of other agents. At the same time, due to limitations such as size and payload, agents usually cannot carry high-performance computing and large energy storage devices. This situation increases the instability of the agent's environmental modeling and the difficulty of the multi-agent joint policy converging to the global optimum. Summary of the Invention

[0003] (a) Technical problems to be solved

[0004] The technical problem to be solved by this invention is how to provide a three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision system to solve the problem of convergence difficulty of multi-agent deep reinforcement learning algorithms caused by the non-static and partially observable characteristics of real-world environments, as well as the problem of limited resources on the multi-agent edge side.

[0005] (II) Technical Solution

[0006] To address the aforementioned technical issues, this invention proposes a three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system, which includes: an edge execution module, a cloud-based teacher guidance module, a cloud-based action collaboration module, a cloud-based parameter sharing module, and a cloud-based global optimization module.

[0007] The edge execution module includes a graph network and a student action network. The graph network obtains spatiotemporal features based on information observed by the sensors of the multi-agent platform, and the student action network realizes intelligent decision-making for the multi-agent based on the spatiotemporal features.

[0008] The cloud-based teacher guidance module includes: a teacher action network, which serves as the optimization target for the student action network, guiding the student action network to update its parameters, and simultaneously outputting agent reference actions to the cloud-based dynamic collaboration module;

[0009] The cloud-side dynamic collaboration module uses an attention mechanism to enhance the differentiated representation of each agent's attention to other agents, establishes a dynamic collaboration graph, feeds it back to the edge as a reference for graph network optimization, and outputs it to the cloud-side parameter sharing module.

[0010] The cloud-side parameter sharing module includes: a cloud-side policy network and a parameter aggregation and collaboration submodule. The cloud-side policy network is used to construct state-action value functions of states and actions based on a dynamic collaboration graph. The parameter aggregation and collaboration submodule aggregates the parameters of all cloud-side policy networks into new shared parameters, which are then used as new parameters for the target policy network.

[0011] The cloud-side global optimization module uses breadth learning to achieve global optimization of cloud-side policy network parameters and guides the teacher's action network in updating network parameters.

[0012] (III) Beneficial Effects

[0013] This invention proposes a three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system. Addressing the convergence challenge of multi-agent deep reinforcement learning algorithms due to the non-static and partially observable characteristics of real-world environments, this invention utilizes a cloud-based collaborative approach. It establishes a dynamic collaborative graph based on a self-attention mechanism, achieves network parameter sharing within the same task region through a federated collaboration mechanism, and employs a width-learning optimization strategy network. This three-pillar architecture, encompassing dynamic collaboration, parameter sharing, and global optimization, guarantees convergence in multi-agent deep reinforcement learning. Compared to traditional methods, this approach accelerates convergence, enhances coordination among agents, and improves system stability. This invention has significant reference and practical value for intelligent decision-making in resource-constrained multi-agent platforms. Attached Figure Description

[0014] Figure 1 This is the overall technical roadmap of the present invention. Detailed Implementation

[0015] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0016] This invention is a three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system. The method includes: deploying a lightweight student action network on the edge; outputting decision actions based on sensor observations from a multi-agent platform; and performing dynamic collaboration, parameter sharing, and global optimization on the cloud side, while guiding the edge-side student action network to update its network parameters. It mainly comprises five parts: an edge-side execution module, a cloud-side teacher guidance module, a cloud-side action collaboration module, a cloud-side parameter sharing module, and a cloud-side global optimization module. The overall framework of this invention is as follows: Figure 1 As shown. Among them,

[0017] The edge execution module includes a graph network and a student action network. The graph network obtains spatiotemporal features based on information observed by the sensors of the multi-agent platform, and the student action network realizes intelligent decision-making for the multi-agent based on the spatiotemporal features.

[0018] The cloud-based teacher guidance module includes: a teacher action network, which serves as the optimization target for the student action network, guiding the student action network to update its parameters, and simultaneously outputting agent reference actions to the cloud-based dynamic collaboration module;

[0019] The cloud-side dynamic collaboration module uses an attention mechanism to enhance the differentiated representation of each agent's attention to other agents, establishes a dynamic collaboration graph, feeds it back to the edge as a reference for graph network optimization, and outputs it to the cloud-side parameter sharing module.

[0020] The cloud-side parameter sharing module includes: a cloud-side policy network and a parameter aggregation and collaboration submodule. The cloud-side policy network is used to construct state-action value functions of states and actions based on a dynamic collaboration graph. The parameter aggregation and collaboration submodule aggregates the parameters of all cloud-side policy networks into new shared parameters, which are then used as new parameters for the target policy network.

[0021] The cloud-side global optimization module uses breadth learning to achieve global optimization of cloud-side policy network parameters and guides the teacher's action network in updating network parameters.

[0022] Example 1:

[0023] like Figure 1 The diagram shows the structural block of a cloud-based collaborative multi-agent deep reinforcement learning decision-making system driven by three pillars: dynamic collaboration, parameter sharing, and parameter optimization. The invented method comprises five parts: an edge execution module, a cloud-based teacher guidance module, a cloud-based action collaboration module, a cloud-based parameter sharing module, and a cloud-based global optimization module. The cloud-based action collaboration module, cloud-based parameter sharing module, and cloud-based global optimization module address the convergence challenge in multi-agent deep reinforcement learning. Based on the cloud-based teacher guidance module, the edge execution module is guided to output decision actions to complete the specified task.

[0024] The specific implementation methods of each module are described below.

[0025] (1) End-side execution module

[0026] Each edge execution module includes three sub-modules: a signal acquisition sub-module, a spatiotemporal feature extraction sub-module, and a decision action output sub-module. The spatiotemporal feature extraction sub-module includes a graph network for feature extraction, and the decision action output sub-module includes a student action network.

[0027] First, the signal acquisition submodule of the multi-agent (the agent is a device such as a drone or unmanned vehicle) acquires the state information of itself and the surrounding agents based on its own sensors, such as the position, velocity, acceleration and other information of the agent itself and the neighboring agents, and transmits it to the spatiotemporal feature extraction submodule.

[0028] Secondly, the graph network of the spatiotemporal feature extraction submodule is shared by the end side. Based on the information obtained by all agents and the dynamic collaboration graph output by the cloud-side dynamic collaboration module, the graph network extracts the spatiotemporal information of interest to each agent according to principles such as distance, forms spatiotemporal features, and transmits them to the decision action output submodule.

[0029] Finally, the decision action output submodule inputs the extracted specific spatiotemporal features into each individual student action network. Each agent corresponds to one student action network. The student action network processes the spatiotemporal features, obtains the decision action of each agent through a fully connected layer, and outputs it.

[0030] (2) Cloud-based teacher guidance module

[0031] This module includes: a teacher action network, with each student action network corresponding to a teacher action network. The cloud-side teacher guidance module performs updates to the teacher action network parameters themselves and updates to the student action network parameters on the client side.

[0032] In the first process, the teacher action network updates the parameters of all teacher action networks based on the state-action value function output by the corresponding cloud-side policy network, combined with the spatiotemporal characteristics of each agent, using the backpropagation mechanism and gradient descent algorithm.

[0033] In the second process, the output data of the fully connected layer of the student action network is augmented to the same dimension as the output data of the fully connected layer of the teacher action network using the adaptation layer. The L2 loss of the two vectors is calculated, and the corresponding student action network parameters are updated based on this loss using the backpropagation mechanism and gradient descent algorithm.

[0034] (3) Cloud-based motion collaboration module

[0035] The cloud-based action collaboration module is used to construct dynamic collaboration graphs and update the graph network. In this module, it receives agent reference actions from the teacher's action network, uses a self-attention mechanism to obtain the agent's attention weights towards other agents based on the agent's state-action pair information, constructs a dynamic collaboration graph representing important collaborative relationships, and feeds this dynamic collaboration graph back to the edge-side graph network to improve agent collaboration efficiency. The dynamic collaboration graph is essentially a vector number, which serves as input data to the graph network. Simultaneously, the cloud-based action collaboration module inputs the dynamic collaboration graph into the policy network.

[0036] The agent state-action pair information consists of the state of the environment and other agents observed by the agent's sensors, such as the speed and position of other drones, obstacle and road information in the environment, and the agent reference actions of the teacher's action network.

[0037] When constructing a dynamic collaboration graph, a sorting method is used to select the agents corresponding to the top N weight values ​​as agents with important collaborative relationships with this agent.

[0038] (4) Cloud-side parameter sharing module

[0039] The cloud-side parameter sharing module includes a cloud-side policy network and a parameter aggregation and collaboration submodule. The cloud-side policy network is used to construct state-action value functions based on a dynamic collaboration graph, while the parameter aggregation and collaboration submodule is used to aggregate the parameters of the cloud-side policy network to obtain the target policy network. Each agent corresponds to one cloud-side policy network.

[0040] In this process, the parameter aggregation and collaboration submodule uses a federated averaging method to aggregate the parameters of all cloud-side policy networks. Network parameters are aggregated into a new, shareable target policy network.

[0041] (1)

[0042] Where N represents the number of cloud-side policy networks. New shared parameters will then be... The parameters are distributed to each policy network on the cloud side, replacing the previous policy network's parameters, and are used as the new parameters for the cloud-side policy network at this moment.

[0043] (5) Cloud-side global optimization module

[0044] The cloud-side global optimization module is used for communication and optimization between the cloud-side policy network and the teacher action network. Specifically, it includes: the cloud-side policy network employs a Deep Deterministic Policy Gradient (DDPG) algorithm architecture, and the parameters of the target policy network are obtained through parameter optimization collaboration based on multiple policy networks. For each agent's data transmitted to the cloud based on its spatiotemporal features and dynamic collaboration graph, the module calculates the outputs of the corresponding cloud-side policy network and the target policy network, and then calculates the difference between the output values ​​of the cloud-side policy network and the target policy network. e(k) is used to calculate the link weights from newly added network nodes to the output nodes in the cloud-side policy network using the matrix pseudoinverse. (k+1), ultimately achieving the goal of optimizing the cloud-side policy network using breadth learning.

[0045] b e(k) (2)

[0046] The value of 'b' is obtained by calculating the pseudo-inverse of the matrix using the outputs of newly added network nodes and the outputs of old historical nodes. A width-learning optimization strategy network is employed here; when discrepancies exist, they are eliminated by adding network nodes.

[0047] After optimizing the cloud-side policy network, the obtained state-action value function is output to the corresponding teacher action network to guide the teacher action network in updating its network parameters.

[0048] This invention relates to a multi-agent collaborative decision-making method, specifically a three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system. It mainly comprises five parts: an edge execution module, a cloud-based teacher guidance module, a cloud-based dynamic collaboration module, a cloud-based parameter sharing module, and a cloud-based global optimization module. Through dynamic collaboration, parameter sharing, and global optimization, it addresses the convergence difficulties of multi-agent deep reinforcement learning algorithms caused by the non-static and partially observable characteristics of real-world environments. It utilizes cloud collaboration technology to transfer computationally intensive training tasks to the cloud and employs a complex teacher action network to guide a lightweight student action network, thus solving the problem of limited edge resources for multi-agent systems and achieving intelligent collaborative decision-making among resource-constrained multi-agent systems.

[0049] This invention addresses the convergence challenge of multi-agent deep reinforcement learning algorithms due to the non-static and partially observable characteristics of real-world environments. It utilizes a cloud-based collaborative approach, establishing a dynamic collaborative graph based on a self-attention mechanism. A federated collaboration mechanism enables network parameter sharing within the same task region. A width-learning optimization strategy network is employed to establish a three-pillar architecture that guarantees convergence in multi-agent deep reinforcement learning, encompassing dynamic collaboration, parameter sharing, and global optimization. This three-pillar-driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system is proposed. Compared to traditional methods, this approach accelerates the convergence speed of multi-agent deep reinforcement learning, enhances coordination among agents, and improves system stability. This invention has significant reference and practical value for intelligent decision-making in resource-constrained multi-agent platforms.

[0050] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A three-pillar driven cloud collaborative multi-agent deep reinforcement learning decision system, characterized in that, The system includes: a terminal execution module, a cloud-based teacher guidance module, a cloud-based action collaboration module, a cloud-based parameter sharing module, and a cloud-based global optimization module; The edge execution module includes a graph network and a student action network. The graph network obtains spatiotemporal features based on information observed by the sensors of the multi-agent platform, and the student action network realizes intelligent decision-making for the multi-agent based on the spatiotemporal features. The cloud-based teacher guidance module includes: a teacher action network, which serves as the optimization target for the student action network, guiding the student action network to update its parameters, and simultaneously outputting agent reference actions to the cloud-based dynamic collaboration module; The cloud-side dynamic collaboration module uses an attention mechanism to enhance the differentiated representation of each agent's attention to other agents, establishes a dynamic collaboration graph, feeds it back to the edge as a reference for graph network optimization, and outputs it to the cloud-side parameter sharing module. The cloud-side parameter sharing module includes: a cloud-side policy network and a parameter aggregation and collaboration submodule. The cloud-side policy network is used to construct state-action value functions based on a dynamic collaboration graph. The parameter aggregation and collaboration submodule aggregates all parameters of the cloud-side policy network into new shared parameters, which are used as new parameters of the target policy network. The cloud-side global optimization module uses breadth learning to achieve global optimization of cloud-side policy network parameters and guides the teacher action network to update network parameters. in, The edge execution module includes three sub-modules: a signal acquisition sub-module, a spatiotemporal feature extraction sub-module, and a decision action output sub-module. The spatiotemporal feature extraction sub-module includes a graph network for feature extraction, and the decision action output sub-module includes a student action network. First, the signal acquisition submodule of the multi-agent system acquires the state information of itself and the surrounding agents based on its own sensors and transmits it to the spatiotemporal feature extraction submodule. Secondly, the graph network of the spatiotemporal feature extraction submodule is shared by the end side. Based on the information obtained by all agents and the dynamic collaboration graph output by the cloud-side dynamic collaboration module, the graph network extracts the spatiotemporal information of interest to each agent, forms spatiotemporal features, and transmits them to the decision action output submodule. Finally, the decision action output submodule inputs the extracted specific spatiotemporal features into each individual student action network. Each agent corresponds to one student action network. The student action network processes the spatiotemporal features, obtains the decision action of each agent through the fully connected layer, and outputs it. Each student action network corresponds to a teacher action network. The cloud-side teacher guidance module performs updates to the teacher action network parameters and updates to the student action network parameters on the device side. The teacher action network updates the parameters of all teacher action networks based on the state-action value function output by the corresponding cloud-side policy network, combined with the spatiotemporal characteristics of each agent, using backpropagation and gradient descent algorithms. The adaptation layer is used to augment the output data of the fully connected layer of the student action network to the same dimension as the output data of the fully connected layer of the teacher action network. The L2 loss of the two vectors is calculated, and the corresponding student action network parameters are updated based on this loss using the backpropagation mechanism and gradient descent algorithm. The cloud-side action collaboration module receives agent reference actions from the teacher's action network, uses a self-attention mechanism to obtain the agent's attention weights to other agents based on the agent's state action pair information, constructs a dynamic collaboration graph based on the attention weights, represents important collaboration relationships in the collaboration graph, and feeds the dynamic collaboration graph back to the edge-side graph network to improve agent collaboration efficiency.

2. The three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system as described in claim 1, characterized in that, The intelligent agent is a drone or unmanned vehicle, and the state information of the intelligent agent includes: the position, velocity, and acceleration information of the intelligent agent itself and neighboring intelligent agents.

3. The three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system as described in claim 1, characterized in that, The agent state-action pair information consists of the state of the environment and other agents observed by the agent's sensors, including: the speed and position of other drones, obstacle and road information in the environment, road information, and agent reference actions from the teacher's action network.

4. The three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system as described in claim 1, characterized in that, When constructing a dynamic collaboration graph, a sorting method is used to select the agents corresponding to the top N weight values ​​as agents with important collaborative relationships with this agent.

5. The three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system as described in claim 1, characterized in that, In the cloud-side parameter sharing module, each agent corresponds to a cloud-side policy network; the parameter aggregation and collaboration submodule uses a federated averaging method to aggregate the parameters of all cloud-side policy networks. Network parameters aggregated into a new shared target policy network: (1) Where N is the number of cloud-side policy networks, and then new shareable parameters will be added. The parameters are distributed to each policy network on the cloud side, replacing the previous policy network's parameters, and are used as the new parameters for the cloud-side policy network at this moment.

6. The three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system as described in claim 5, characterized in that, The cloud-side global optimization module includes: the cloud-side policy network employs the Deep Deterministic Policy Gradient (DDPG) algorithm, and the parameters of the target policy network are obtained by parameter optimization collaboration based on multiple policy networks; for each agent's data transmitted to the cloud based on its spatiotemporal characteristics and dynamic collaboration graph, the module calculates the outputs of the corresponding cloud-side policy network and the target policy network, and calculates the difference between the output values ​​of the cloud-side policy network and the target policy network. e ( k Using the matrix pseudoinverse, the link weights from newly added network nodes to the output nodes in the cloud-side policy network are calculated. k+ 1 ) Ultimately, this achieves the goal of optimizing the cloud-side policy network using breadth learning; After optimizing the cloud-side policy network, the obtained state-action functions are output to the corresponding teacher action network to guide the teacher action network in updating its network parameters.

7. The three-pillar driven cloud-based collaborative multi-agent deep reinforcement learning decision-making system as described in claim 6, characterized in that, Add link weights from network nodes to output nodes k+1) is: b e ( k )(2) in, b The value is obtained by calculating the pseudo-inverse of the matrix using the outputs of the newly added network nodes and the outputs of the old historical nodes.

Images