Multi-agent reinforcement learning method and system based on hierarchical consistency learning

Abstract

A multi-agent reinforcement learning method and system based on hierarchical consistency learning combines a deep ensemble network with a variational autoencoder, and is introduced into multi-agent reinforcement learning to regulate the group behavior of a sorting robot cluster in a large-scale automatic sorting task. The present invention introduces auxiliary unsupervised learning tasks and self-supervised learning tasks to learn efficient team intention representation and individual intention representation, and imposes diversity constraints on team intentions and consistency constraints on individual intentions to ensure close collaboration within the team and diverse exploration between teams, thereby effectively improving the exploration efficiency and collaboration efficiency of a large-scale multi-agent system when completing collaborative tasks. The present invention can effectively improve the exploration efficiency and collaboration efficiency of a large-scale multi-agent system when completing collaborative tasks.

Description

Technical Field

[0001] The present invention relates to a technology in the field of warehouse automation, specifically a multi-agent reinforcement learning method and system based on hierarchical consistency learning for solving large-scale automatic sorting tasks of sorting robot clusters in the field of warehouse automation. Background Art

[0002] Most existing multi-agent reinforcement learning algorithms follow the centralized training-decentralized execution framework. In the centralized training stage, agents need to learn decentralized strategies by sharing local observations, parameters or gradients. However, existing algorithms model agents as independent individuals in the centralized training stage. Although these agents can collaborate through explicit or implicit communication, it is difficult to learn effective communication protocols through end-to-end training, which hinders agents from efficiently exploring and collaborating in large-scale multi-agent systems. A cluster of sorting robots that need to collaborate to complete sorting tasks constitutes a typical large-scale multi-agent system. The behavior of agents is determined by team goals and perception of the environment. Therefore, diverse team goals and consistent environmental perception are the key to efficient exploration and collaboration of agents. In order to design a multi-agent reinforcement learning algorithm with efficient exploration and collaboration, two issues are crucial: how to generate diversity between teams and how to work closely together within the team. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art in solving automatic sorting tasks in the field of warehouse automation, the present invention proposes a multi-agent reinforcement learning method and system based on hierarchical consistency learning, introduces auxiliary unsupervised learning tasks and self-supervised learning tasks, and learns efficient team intention representation and individual intention representation. By imposing diversity constraints on team intentions and consistency constraints on individual intentions, close collaboration within the team and diverse exploration between teams are guaranteed, thereby effectively improving the exploration efficiency and collaboration efficiency of large-scale multi-agent systems when completing collaborative tasks. The present invention can effectively improve the exploration efficiency and collaboration efficiency of large-scale multi-agent systems when completing collaborative tasks.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a multi-agent reinforcement learning method based on hierarchical consistency learning, which constructs and randomly initializes a team intention network, an individual intention network and a multi-agent decision network; uses a hierarchical clustering algorithm to cluster and group all agents to form a team; calculates unsupervised contrast loss, self-supervised regression loss, consistency loss, reconstruction loss and decision loss through the team intention network, the individual intention network and the multi-agent decision network respectively and obtains the overall loss comprehensively, optimizes the team intention network, the individual intention network and the multi-agent decision network according to the overall loss, and the sorting robot cluster will be dynamically grouped in completing large-scale sorting tasks, decomposing large-scale sorting tasks into welfare small-scale subtasks. The agents within the team will achieve closer collaboration, and the agents between teams will show the diversity of sorting strategies, so as to effectively complete each subtask, and finally achieve a high degree of task completion in large-scale automatic sorting tasks.

[0006] The team intention network includes: a state encoder, a deep ensemble network and an intention encoder, wherein: the state encoder receives the local observations and global state information of each agent as input, and generates the local state code corresponding to each agent; the deep ensemble network receives the local state codes of all agents in the team, and after linear encoding and nonlinear mapping, performs an average operation on the local state code corresponding to each agent to obtain the team code corresponding to the entire team; the intention encoder receives the team code, and after linear encoding and nonlinear mapping, outputs the team intention.

[0007] The state encoder is a residual neural network (ResNet), which is implemented using but not limited to the method in Kaiming He et al. "Deep Residual Learning for Image Recognition" CVPR (2016).

[0008] The deep set network is a fully connected neural network with a nonlinear activation layer, which is implemented using but not limited to the method in Zaheer, Manzil et al. "Deep Sets." NeurIPS (2017).

[0009] The specific network architecture of the intent encoder is also a fully connected neural network with a nonlinear activation layer.

[0010] The individual intention network includes: an individual encoder, a graph convolutional network and an individual intention variational encoder, wherein: the individual encoder receives local observations of each agent and outputs the individual code corresponding to each agent; the graph convolutional network regards all agents in the team as a fully connected undirected graph, the nodes in the graph represent each agent, and the node code is the individual code output by the individual encoder. The graph convolutional network performs a graph convolution operation on the individual code in the graph to obtain the individual perception corresponding to each agent; the individual intention variational encoder takes the team intention output by the team intention network and the features spliced ​​together with the individual perception output by the graph convolutional network as input, and outputs the individual intention corresponding to each agent.

[0011] The individual encoder is a residual neural network (ResNet), which is implemented using but not limited to the method in Kaiming He et al. "Deep Residual Learning for Image Recognition" CVPR (2016).

[0012] The graph convolutional network is a graph convolutional neural network, which is implemented using but not limited to the method in Kipf, Thomas and MaxWelling. "Semi-Supervised Classification with Graph Convolutional Networks." ICLR (2017).

[0013] The individual intention variational encoder is a variational autoencoder, which adopts but is not limited to the technology in Diederik P. Kingma and Max Welling. "Auto-Encoding Variational Bayes." ICLR (2014).

[0014] The multi-agent decision network generates decision actions for all agents in the team based on the joint observations of all agents in each team. The multi-agent decision network is a residual neural network (ResNet), which is implemented using but not limited to the method in Kaiming He et al. "Deep Residual Learning for Image Recognition" CVPR (2016).

[0015] The present invention relates to a multi-agent reinforcement learning system based on hierarchical consistency for large-scale automatic sorting tasks of sorting robot clusters in the field of warehouse automation, which implements the above method, and includes: an initialization unit, a heuristic grouping unit, a team intention unit, an individual intention unit, a multi-agent decision unit and a model optimization unit, wherein: the initialization unit constructs and randomly initializes the team intention network, the individual intention network and the multi-agent decision network; the heuristic grouping unit uses a hierarchical clustering algorithm to cluster and group all agents to form teams; the team intention unit uses the team intention network to generate a team intention for each team according to the joint observation of all agents in each team, and uses the team intention network to calculate the unsupervised contrast loss according to different team intentions, and calculates the self-supervised loss according to the joint observation of all agents in each team. Regression loss; the individual intention unit uses the individual intention network to generate individual intentions for all agents, and uses the individual intention network to calculate the consistency loss according to the hierarchical consistency constraint, and calculates the reconstruction loss according to the loss of the variational autoencoder; the multi-agent decision unit uses the multi-agent decision network to generate decision actions for all agents according to the team intention of each team and the individual intentions of all agents in the team, and interacts with the environment to obtain the reward feedback of each agent, and uses the multi-agent decision network to calculate the decision loss according to the reward obtained by each agent; the model optimization unit performs a weighted summation of the unsupervised contrast loss, self-supervised regression loss, consistency loss, reconstruction loss and decision loss to obtain the overall loss, and optimizes the team intention network, individual intention network and multi-agent decision network accordingly.

[0016] Technical Effects

[0017] The present invention combines a deep ensemble network with a variational autoencoder and introduces it into multi-agent reinforcement learning to regulate the group behavior of a cluster of sorting robots in large-scale automatic sorting tasks. That is, the deep ensemble network and the individual intention variational encoder are combined with unsupervised contrast loss and consistency loss, so that the behavior of the agents between teams shows sufficient diversity, and the behavior of the agents within the team shows sufficient collaboration, thereby ultimately improving the time efficiency and task completion of the automatic sorting task. The existing methods solve the above two problems separately, so that the automatic sorting strategy learned by the sorting robot cannot meet the requirements of diversity and collaboration at the same time, thus having a negative impact on time efficiency and task completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the method of the present invention;

[0019] Figure 2 A schematic diagram of a team intention network in an embodiment;

[0020] Figure 3A schematic diagram of an individual intention network in an embodiment;

[0021] Figure 4 A schematic diagram of the calculation logic of the unsupervised contrast loss and the self-supervised regression loss in the embodiment;

[0022] Figure 5 A schematic diagram of the calculation logic of consistency loss and reconstruction loss in the embodiment;

[0023] Figure 6 A schematic diagram of collaborative tasks in an embodiment;

[0024] Figure 7 A schematic diagram of collaborative tasks in an embodiment;

[0025] Figure 8 Schematic diagram of a multi-agent reinforcement learning system based on hierarchical consistency learning in an embodiment. DETAILED DESCRIPTION

[0026] Example 1

[0027] This embodiment relates to the specific implementation of the multi-agent reinforcement learning method based on hierarchical consistency learning in the large-scale automatic sorting scenario of sorting robot clusters in the field of warehouse automation, such as Figure 6 shown.

[0028] In the sorting task of this embodiment, 12 sorting robots are located in the central area of ​​the square map, and 32 shelves are evenly distributed in the four corner areas of the square map. The sorting robot obtains rewards by navigating to the shelf. The local observation of each sorting robot includes all the information in the square area with a radius of 7 unit lengths centered on itself, that is, the two-dimensional coordinates of the remaining sorting robots and the two-dimensional coordinates of the shelf. The decision-making actions of the sorting robot include moving up, moving down, moving left, moving right, staying still and lifting the shelf, and the moving distance only includes the area of ​​one unit length around. The sorting robot can only perform the action of lifting the shelf after reaching the location of the shelf. Sorting robots will be punished if they collide with each other or perform the operation of lifting the shelf in a position where there is no shelf. At the same time, in order to encourage the sorting robot to complete the sorting task faster, it will also be punished less for each movement action. The intelligent agent needs to learn to automatically group and reasonably decompose large-scale sorting tasks, so that each team can move to different shelf areas to sort the shelves.

[0029] like Figure 1 As shown, a multi-agent reinforcement learning method based on hierarchical consistency involved in this embodiment includes:

[0030] S110, construct and randomly initialize the team intention network, individual intention network, multi-agent decision network and joint decision evaluation function, and use the Xavier method (which adopts but is not limited to the technical implementation in Glorot, Xavier and Yoshua Bengio. "Understanding the difficulty of training deep feedforward neural networks." AISTATS (2010).) to initialize all parameters of the network / function.

[0031] As shown in Table 1, these are the network parameters of the team intention network and the individual intention network.

[0032]

[0033] like Figure 2 As shown, the team intention network includes: a state encoder, a deep set network and an intention encoder, wherein: the state encoder first concatenates the 14×14 two-dimensional local observation matrix of each sorting robot with the 14×14 two-dimensional global state matrix of the global state to obtain a 14×14×2 feature map, which is upsampled to 224×224×2, and then convolved with 64 7×7 convolution kernels with a step size of 2 to obtain a 112×112×64 feature map; after a maximum pooling of 3×3 with a step size of 2, it is convolved with 64 3×3 convolution kernels twice to obtain a 56×56×64 feature map; and then convolved with 128 3×3 convolution kernels twice, 256 3×3 convolution kernels twice and 2 After convolution with 56 3×3 convolution kernels, a 7×7×512 feature map is obtained, and finally after 7×7 average aggregation, a 512-dimensional state code is obtained; then, the deep ensemble network receives the state codes of all agents in the team as input, first passes the state code of each agent through a layer of fully connected neural network and ReLU non-linear activation layer, and obtains the 64-dimensional hidden layer features corresponding to each agent, and then averages all the hidden layer features that can be extracted in the team to obtain a 64-dimensional team code; finally, the intent encoder receives the 64-dimensional team code and the expanded 196 (14×14)-dimensional global state as input, and after passing through a layer of fully connected neural network and ReLU non-linear activation layer, the 32-dimensional team intention corresponding to each team is obtained.

[0034] like Figure 3As shown, the individual intention network includes: an individual encoder, a graph convolutional network and an individual intention variational encoder, wherein: the state encoder first upsamples the 14×14 two-dimensional local observation matrix of each sorting robot to 224×224, and then convolves it with 64 7×7 convolution kernels with a step size of 2 to obtain a 112×112×64 feature map; after a maximum pooling of 3×3 with a step size of 2, it is convolved with 64 3×3 convolution kernels twice to obtain a 56×56×64 feature map; after two convolutions with 128 3×3 convolution kernels, two convolutions with 256 3×3 convolution kernels, and two convolutions with 256 3×3 convolution kernels, a 7×7×512 feature map is obtained, and finally, after an average pooling of 7×7, a 512 feature map is obtained. dimensional individual code; then, the graph convolutional network receives the 512-dimensional individual codes of all agents in the team as input, and passes through the fully connected neural network, ReLU non-linear activation layer, and graph convolution layer in sequence to obtain a 64-dimensional individual perception code; finally, the individual intention variational encoder receives the 64-dimensional individual perception code of each agent in the team as input. On the one hand, the team intention and the individual perception code are concatenated and passed through the fully connected neural network and the ReLU non-linear activation layer to obtain a 64-dimensional individual intention; on the other hand, the individual perception code passes through the individual perception decoder and the individual decoder whose architecture is symmetrical with the individual encoder and the graph convolutional network to obtain a 224×224 reconstructed local feature, and then the reconstructed state is downsampled to obtain a 14×14 reconstructed local observation.

[0035] The multi-agent decision network first concatenates the 64-dimensional individual intention of each sorting robot in the team and the 32-dimensional team intention of the entire team to obtain a 96-dimensional feature vector; finally, after passing through a layer of fully connected neural network, a 6-dimensional decision action output of each intelligent agent is obtained.

[0036] S120, clustering and grouping all agents using a hierarchical clustering algorithm to form a team;

[0037] S130, such as Figure 4 As shown in the figure, the team intention network generates team intention for each team based on the joint observations of all agents in each team, calculates its corresponding unsupervised contrast loss, and calculates the self-supervised regression loss based on the joint observations of all agents in each team.

[0038] The team intention is to splice the local observations of each agent and the global state information through the team intention network, and then input them into the state encoder, and generate the local state code corresponding to each agent according to the above process; then, the local state code is input into the deep ensemble network, and as described in the above process, the local state code corresponding to each agent is subjected to nonlinear mapping and averaged to obtain the team code corresponding to the entire team; finally, after the team code passes through the intention encoder, the team intention c is obtained according to the above process. t .

[0039] The unsupervised contrast loss Where: m is a positive real hyperparameter, and teams k and u have the same label Teams k, v have different labels Teams u and v are randomly sampled from all weakly connected components. Specifically, for a specific team k obtained by the method through the organizational control network at time t, firstly, according to the spatial positions of all agents in the team Calculate the team's spatial center of gravity by averaging Combined with the current timestamp t, we can get the spatiotemporal feature representation of team k at time t Then, the Euclidean distance between each team is calculated based on the spatiotemporal feature representation of all teams, that is, Then, the method models all teams as a weighted undirected graph model, where nodes represent teams, all nodes are connected by edges, and the weight of the edge represents the Euclidean distance between teams. Then, the method uses the following rules to prune the edges of the weighted undirected graph: That is, there is an edge connection between two teams if and only if one team belongs to the closest team to the other team. Then, the method uses the traditional graph theory Tarjan algorithm to find all weakly connected components in the undirected graph after edge pruning, and numbers each weakly connected component as a label. Then, the method constructs a labeled training set for each team based on its team intention generated through the team intention network and the number of the weakly connected component to which the team belongs. And calculate the unsupervised contrastive loss.

[0040] The self-supervised regression loss Where: f ξ (·) is the team intent decoder, which receives the team intent As well as global state information and predict the joint observation of all agents in the corresponding team at the next moment

[0041] S140, such as Figure 5As shown, individual intentions are generated for all agents through the individual intention network, the consistency loss is calculated according to the hierarchical consistency constraint, and the reconstruction loss is calculated according to the loss of the variational autoencoder.

[0042] The individual intention, through the individual encoder g φ The local observation of each agent i in team k is encoded into individual code according to the above process The agents in the team are then modeled as a fully connected undirected graph and a graph neural network g is used ψ , as described in the above process, encode the individual codes of all agents in the team Perform graph convolution operations to obtain individual perception: Then through the individual intention variation encoder Stitching together team intentions and individual perceptions Calculate individual intentions

[0043] The sum of the consistency loss and the reconstruction loss Among them: for agent i belonging to the same team k, j represents different agents belonging to the same team, q is the reconstructed local observation obtained by the variational autoencoder, and the specific calculation process is as described above.

[0044] S150, generating decision actions for all agents according to the team intention of each team and the individual intentions of all agents in the team through the multi-agent decision network according to the above process, and interacting with the environment to obtain reward feedback for each agent. At the same time, the multi-agent decision network calculates the decision loss using the QMIX algorithm based on the reward obtained by each agent;

[0045] The rewards mentioned include external rewards and internal rewards.

[0046] External rewards described It means: Agent i, that is, the sorting robot, will get a reward of -0.01 for each move, a reward of 5 for successfully lifting the shelf, and a reward of -0.1 for lifting the shelf in a place where there is no shelf.

[0047] Internal rewards described in: and Respectively represent the team intentions of two different teams.

[0048] The decision loss in: Local decision loss The joint decision loss of all agents The local and global aspects satisfy: Where: QMIX =0.01 indicates a positive real hyperparameter, which is used to balance the two losses; γ = 0.99 indicates a discount factor between 0 and 1, which is used to balance short-term and long-term gains.

[0049] S160, weighted sum of unsupervised contrast loss, self-supervised regression loss, consistency loss, reconstruction loss and decision loss is performed to obtain the overall loss, and the team intention network, individual intention network and multi-agent decision network are optimized according to the overall loss. Specifically, all network parameters are optimized using the Adam optimizer, with a learning rate of 0.0001, a batch size of 512, 500 training rounds, a maximum number of iterations in each round of 250, and a parameter update frequency of 4 times per batch of data.

[0050] The training set of the training is generated in the following way: in each time step of each round of training, the sorting robots obtain their respective decision actions according to the team intention network, the individual intention network and the multi-agent decision network according to the above process, and execute them in the simulation environment. After the environment receives the joint decision of all sorting robots, it will update the state of each sorting robot, that is, the two-dimensional coordinate information, and the state of the shelf, and calculate the reward for each sorting robot. Therefore, each time step will generate a triple such as (joint observation, joint decision, reward). The triples generated in each time step are placed in an experience replay cache to form the data set required for training. In this embodiment, the size of the experience replay cache is 1,000,000.

[0051] After ten experiments with different random seed initializations, this embodiment can achieve a cumulative reward value of 767±74; in all simulation experiments, it can achieve 100% task completion, that is, sorting to all 32 shelves.

[0052] Example 2

[0053] Based on Example 1, this example introduces interference factors into the sorting environment, such as observation noise, decision delay, and stray target, which will affect the behavior of the sorting robot. This example generally and abstractly models these real interference factors by introducing additional interference agents. Specifically, compared with Example 1, the schematic diagram of the automatic sorting task processed by this example is as follows: Figure 8As shown, it includes: 16 interference robots, 16 faster-moving sorting robots and 32 shelves. The sorting robot can only be rewarded if it successfully reaches the shelf position and lifts the shelf, while the interference robot obtains the reward by reaching the position of the sorting robot. The actions of all intelligent agents include moving up, moving down, moving left, moving right and staying still; the sorting robot has the additional action of lifting the shelf. The movement range of the interference robot is 1 unit length around it, and the movement range of the sorting robot is 2 unit lengths around it. Sorting robots will be punished not only for colliding with each other, but also for colliding with interference robots. . Since the moving speed of the sorting robot is faster than that of the interference robot, the sorting robot needs to learn how to decompose large-scale tasks by grouping, and also learn how to avoid interference robots. In addition to the above settings, the remaining settings of this embodiment are consistent with Example 1.

[0054] In this embodiment, for the external rewards of the environment itself, all robot movements will not be subject to additional penalties. The sorting robot will be penalized -1 if it is caught up by the interfering robot, and the interfering robot will receive a reward of +1 accordingly. The sorting robot will receive a reward of +5 if it successfully lifts a shelf. Similarly, the sorting robot will receive a penalty of -0.1 if it lifts a shelf in a location where there is no shelf.

[0055] In this embodiment, the interference robot is trained using the same algorithm as the sorting robot. As the training progresses, the interference robot's ability will become stronger and stronger, which means that the interference amplitude in the real environment is getting larger and larger. The sorting robot needs to learn to adopt a better cooperation strategy to counter the increasing interference amplitude during the training process.

[0056] After ten experiments with different random seed initializations, this embodiment can achieve a cumulative reward value of 620±44. In all simulation experiments, it can eventually achieve 100% task completion, that is, all 32 shelves are sorted in the presence of interfering robots.

[0057] Example 3

[0058] like Figure 8As shown, a multi-agent reinforcement learning system based on hierarchical consistency for implementing the above method includes: an initialization unit 510, a heuristic grouping unit 520, a team intention unit 530, an individual intention unit 540, a multi-agent decision unit 550 and a model optimization unit 560, wherein: the initialization unit 510 constructs and randomly initializes the team intention network, the individual intention network and the multi-agent decision network; the heuristic grouping unit 520 uses a hierarchical clustering algorithm to cluster and group all agents to form teams; the team intention unit 530 uses the team intention network to generate a team intention for each team according to the joint observation of all agents in each team, and uses the team intention network to calculate the unsupervised contrast loss according to different team intentions, and calculates the self-supervised regression loss according to the joint observation of all agents in each team; the individual intention network is used to calculate ... The agent intention unit 540 uses the individual intention network to generate individual intentions for all agents, and uses the individual intention network to calculate the consistency loss according to the hierarchical consistency constraint, and calculates the reconstruction loss according to the loss of the variational autoencoder; the multi-agent decision unit 550 uses the multi-agent decision network to generate decision actions for all agents according to the team intention of each team and the individual intentions of all agents in the team, and interacts with the environment to obtain the reward feedback of each agent, and uses the multi-agent decision network to calculate the decision loss according to the reward obtained by each agent; the model optimization unit 560 performs weighted summation of the unsupervised contrast loss, self-supervised regression loss, consistency loss, reconstruction loss and decision loss to obtain the overall loss, and optimizes the team intention network, the individual intention network and the multi-agent decision network accordingly.

[0059] The initialization unit 510 includes: a team intention network initialization unit, an individual intention network initialization unit and a multi-agent decision network initialization unit. The three units respectively perform parameter initialization according to the specific architecture information of the team intention network, the individual intention network and the multi-agent decision network using the Xavier method.

[0060] The heuristic grouping unit clusters and groups all the sorting robots according to the two-dimensional coordinate information of all the sorting robots using a hierarchical clustering algorithm to form teams, and assigns a team ID to each sorting robot.

[0061] The team intention unit generates a team intention for each team using a team intention network based on the joint observations of all agents in each team, and then calculates an unsupervised contrast loss based on the obtained team intention, and calculates a self-supervised regression loss based on the joint observations of all agents in each team.

[0062] After specific practical experiments, in the specific environment settings of Example 1 and Example 2, all network parameters were optimized using the Adam optimizer, with a learning rate of 0.0001, a batch size of 512, a number of training rounds of 500, a maximum number of iterations in each round of 250, and a parameter update frequency of 4 times per batch of data. In the experiment of ten samplings with different random seed initializations, a cumulative reward value of 767±74 can be achieved in the setting of Example 1, and in all simulation experiments, 100% of the task completion can be achieved, that is, all 32 shelves can be sorted; in the experiment of ten samplings with different random seed initializations, a cumulative reward value of 620±44 can be achieved in the setting of Example 2, and in all simulation experiments, 100% of the task completion can be achieved in the end, that is, all 32 shelves can be sorted in the presence of interfering robots.

[0063] Compared with the prior art, the performance of the device of the present invention is superior to the existing multi-agent reinforcement learning method in terms of the scale of sorting robots and shelves that can be processed, the time efficiency of completing sorting tasks, and the degree of completion of sorting tasks.

[0064] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A multi-agent reinforcement learning method based on hierarchical consistency learning, characterized in that: By constructing and randomly initializing the team intention network, individual intention network and multi-agent decision network; using the hierarchical clustering algorithm to cluster all agents and form teams; calculating the unsupervised contrast loss, self-supervised regression loss, consistency loss, reconstruction loss and decision loss through the team intention network, individual intention network and multi-agent decision network respectively and synthesizing the overall loss, the team intention network, individual intention network and multi-agent decision network are optimized according to the overall loss. The sorting robot cluster will be dynamically grouped in completing large-scale sorting tasks, decomposing large-scale sorting tasks into small-scale sub-tasks of welfare. The agents within the team will achieve closer collaboration, and the agents between teams will show the diversity of sorting strategies, so as to effectively complete each sub-task, and finally achieve a high degree of task completion in large-scale automatic sorting tasks; The team intention network includes: a state encoder, a deep ensemble network and an intention encoder, wherein: the state encoder receives the local observation and global state information of each agent as input, and generates the local state code corresponding to each agent; the deep ensemble network receives the local state code of all agents in the team, and after linear encoding and nonlinear mapping, performs an average operation on the local state code corresponding to each agent to obtain the team code corresponding to the entire team; the intention encoder receives the team code, and after linear encoding and nonlinear mapping, outputs the team intention; The individual intention network includes: an individual encoder, a graph convolutional network and an individual intention variational encoder, wherein: the individual encoder receives local observations of each agent and outputs the individual code corresponding to each agent; the graph convolutional network regards all agents in the team as a fully connected undirected graph, the nodes in the graph represent each agent, and the node code is the individual code output by the individual encoder. The graph convolutional network performs a graph convolution operation on the individual code in the graph to obtain the individual perception corresponding to each agent; the individual intention variational encoder takes the team intention output by the team intention network and the features spliced ​​together with the individual perception output by the graph convolutional network as input, and outputs the individual intention corresponding to each agent.

2. The multi-agent reinforcement learning method based on hierarchical consistency learning according to claim 1, characterized in that: The team intention network includes: a state encoder, a deep set network and an intention encoder, wherein: the state encoder first concatenates the 14×14 two-dimensional local observation matrix of each sorting robot with the 14×14 two-dimensional global state matrix of the global state to obtain a 14×14×2 feature map, upsamples it to 224×224×2, and then convolves it with 64 7×7 convolution kernels with a step size of 2 to obtain a 112×112×64 feature map; after a maximum pooling of 3×3 with a step size of 2, it is convolved with 64 3×3 convolution kernels twice to obtain a 56×56×64 feature map; and then convolves with 128 3×3 convolution kernels twice, 256 3×3 convolution kernels twice and 256 After convolution with a 3×3 convolution kernel, a 7×7×512 feature map is obtained, and finally after 7×7 average aggregation, a 512-dimensional state code is obtained; then, the deep ensemble network receives the state codes of all agents in the team as input, first passes the state code of each agent through a layer of fully connected neural network and ReLU nonlinear activation layer, and obtains the 64-dimensional hidden layer features corresponding to each agent, and then averages all the hidden layer features that can be extracted in the team to obtain a 64-dimensional team code; finally, the intent encoder receives the 64-dimensional team code and the expanded 196 (14×14)-dimensional global state as input, and after passing through a layer of fully connected neural network and ReLU nonlinear activation layer, a 32-dimensional team intent corresponding to each team is obtained; The individual intention network includes: an individual encoder, a graph convolutional network and an individual intention variational encoder, wherein: the state encoder first upsamples the 14×14 two-dimensional local observation matrix of each sorting robot to 224×224, and then convolves it with 64 7×7 convolution kernels with a step size of 2 to obtain a 112×112×64 feature map; after a maximum pooling of 3×3 with a step size of 2, it is convolved with 64 3×3 convolution kernels twice to obtain a 56×56×64 feature map; after two convolutions with 128 3×3 convolution kernels, two convolutions with 256 3×3 convolution kernels, and two convolutions with 256 3×3 convolution kernels, a 7×7×512 feature map is obtained, and finally, after an average pooling of 7×7, a 512-dimensional individual coding; then, the graph convolutional network receives the 512-dimensional individual coding of all agents in the team as input, and passes through the fully connected neural network, ReLU nonlinear activation layer, and graph convolution layer in sequence to obtain a 64-dimensional individual perception coding; finally, the individual intention variational encoder receives the 64-dimensional individual perception coding of each agent in the team as input. On the one hand, the team intention and the individual perception coding are concatenated and passed through the fully connected neural network and the ReLU nonlinear activation layer to obtain a 64-dimensional individual intention; on the other hand, the individual perception coding passes through the individual perception decoder and the individual decoder whose architecture is symmetrical with the individual encoder and the graph convolutional network to obtain a 224×224 reconstructed local feature, and then the reconstructed state is downsampled to obtain a 14×14 reconstructed local observation.

3. The multi-agent reinforcement learning method based on hierarchical consistency learning according to claim 1 or 2, characterized in that: The individual intention, through the individual encoder g φ The local observation of each agent i in team k is encoded into individual code according to the above process Then the agents in the team are modeled as a fully connected undirected graph and a graph neural network g is used ψ , as described in the above process, the individual codes of all agents in the team Perform graph convolution operations to obtain individual perception: Then through the individual intention variation encoder Stitching together team intentions and individual perceptions Calculate individual intentions 4. The multi-agent reinforcement learning method based on hierarchical consistency learning according to claim 1 is characterized by: , the consistency loss, the sum of which and the reconstruction loss Among them: for the agent i belonging to the same team k, j represents different agents belonging to the same team, and q is the reconstructed local observation obtained by the variational autoencoder.

5. A multi-agent reinforcement learning system based on hierarchical consistency for large-scale automatic sorting tasks of sorting robot clusters in the field of warehouse automation, which implements the method described in any one of claims 1 to 4, characterized in that: include: Initialization unit, heuristic grouping unit, team intention unit, individual intention unit, multi-agent decision unit and model optimization unit, wherein: the initialization unit constructs and randomly initializes the team intention network, individual intention network and multi-agent decision network; the heuristic grouping unit uses a hierarchical clustering algorithm to cluster and group all agents to form teams; the team intention unit uses the team intention network to generate team intentions for each team based on the joint observations of all agents in each team, and uses the team intention network to calculate unsupervised contrast losses based on different team intentions, and calculates self-supervised regression losses based on the joint observations of all agents in each team; The individual intention unit uses the individual intention network to generate individual intentions for all agents, and uses the individual intention network to calculate the consistency loss according to the hierarchical consistency constraint, and calculates the reconstruction loss according to the loss of the variational autoencoder; the multi-agent decision unit uses the multi-agent decision network to generate decision actions for all agents according to the team intention of each team and the individual intentions of all agents in the team, and interacts with the environment to obtain the reward feedback of each agent, and uses the multi-agent decision network to calculate the decision loss according to the reward obtained by each agent; the model optimization unit performs a weighted summation of the unsupervised contrast loss, self-supervised regression loss, consistency loss, reconstruction loss and decision loss to obtain the overall loss, and optimizes the team intention network, individual intention network and multi-agent decision network accordingly.

6. The multi-agent reinforcement learning system according to claim 5, characterized in that: The initialization unit includes: a team intention network initialization unit, an individual intention network initialization unit and a multi-agent decision network initialization unit. The three units respectively perform parameter initialization according to the specific architecture information of the team intention network, the individual intention network and the multi-agent decision network using the Xavier method.

7. The multi-agent reinforcement learning system according to claim 5, characterized in that: The heuristic grouping unit clusters and groups all the sorting robots according to the two-dimensional coordinate information of all the sorting robots using a hierarchical clustering algorithm to form teams, and assigns a team ID to each sorting robot.

8. The multi-agent reinforcement learning system according to claim 5, characterized in that: The team intention unit generates a team intention for each team using a team intention network based on the joint observations of all agents in each team, and then calculates an unsupervised contrast loss based on the obtained team intention, and calculates a self-supervised regression loss based on the joint observations of all agents in each team.

Images