Self-organizing method and system of multi-agent system based on enhanced organizational control

Abstract

A multi-agent system self-organization method and system based on enhanced organizational control, which models the self-organization problem of sorting robot clusters in large-scale automatic sorting tasks as a multi-agent reinforcement learning problem through binary decision-making. Each agent can spontaneously decide whether to form a system with adjacent agents through a binary decision-making unit, and use traditional graph theory algorithms in the self-organization problem of agents, which can ultimately improve the time efficiency and task completion of automatic sorting tasks. The present invention can ensure close collaboration within the system and achieve the division of complex collaborative tasks, thereby effectively improving the scalability of learning algorithms when applied to large-scale multi-agent systems.

Description

Technical Field

[0001] The present invention relates to a technology in the field of warehouse automation, specifically a self-organizing method and system of a multi-agent system based on enhanced organizational control. The system is applied to solving large-scale automatic sorting tasks of sorting robot clusters in the field of warehouse automation. Background Art

[0002] Most existing multi-agent reinforcement learning algorithms follow the centralized training-decentralized execution framework. In the centralized training stage, agents need to learn decentralized strategies by sharing local observations, parameters or gradients. However, due to the existence of centralized controllers, existing algorithms cannot be extended to large-scale multi-agent systems, and a cluster of sorting robots that need to collaborate to complete large-scale sorting tasks constitutes a typical large-scale multi-agent system. In human society, when groups complete complex collaborative tasks, humans will spontaneously form many systems based on changes in the external environment and task progress, and divide and conquer complex tasks through close collaboration within the system. Therefore, in order to make the algorithm have good scalability, an urgent problem to be solved is how agents can concurrently form a system, and how the system composition can adapt to changes in the external environment. Existing methods mainly group agents through heuristic rules or end-to-end black box optimization, so it is difficult to learn a dynamic teaming strategy with good adaptability. Summary of the invention

[0003] In view of the above-mentioned deficiencies of the prior art in solving large-scale sorting tasks, the present invention proposes a self-organizing method and system for a multi-agent system based on reinforced organizational control. The method realizes the spontaneous grouping of intelligent agents, that is, the sorting robots, through adaptive organizational control based on multi-agent reinforcement learning, to ensure close collaboration within the system and achieve the division of complex collaborative tasks, thereby effectively improving the scalability of learning algorithms when applied to large-scale multi-agent systems.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a self-organizing method of a multi-agent system based on enhanced organizational control. By constructing and randomly initializing an organizational control network and a multi-agent decision network, a corresponding binary decision is generated for the local observation of each sorting robot as an intelligent agent through the organizational control network, and the system composed of the multi-agents is modeled as a directed graph, and all weakly connected components are extracted from it to group the intelligent agents; then, the multi-agent decision network generates decision actions for all intelligent agents according to the joint observation of each intelligent agent system, and interacts with the environment to obtain the respective reward feedback of each intelligent agent; then, the organizational control loss and the decision loss are calculated according to the rewards through the organizational control network and the multi-agent decision network, so as to optimize the organizational control network and the multi-agent decision network; finally, the sorting robot cluster will spontaneously, dynamically and reasonably group in completing the large-scale sorting task, decompose the large-scale sorting task into independent small-scale sub-tasks, and the intelligent agent system will generate an efficient collaboration strategy through the multi-agent reinforcement learning method, so as to effectively complete each sub-task, and finally achieve a high task completion degree in the large-scale automatic sorting task.

[0006] The organizational control network is a residual neural network (ResNet, which is implemented using but not limited to the method described in Glorot, Xavier and Yoshua Bengio. "Understanding the difficulty of training deep feedforward neural networks." AISTATS (2010).), which receives local observations of each intelligent agent, i.e., the sorting robot, as input and outputs a dynamic teaming strategy for each intelligent agent, i.e., a binary decision on whether to form a system with neighboring intelligent agents.

[0007] The multi-agent decision network is a residual neural network (ResNet, which adopts but is not limited to Kaiming He et al.

[0008] "Deep Residual Learning for Image Recognition" CVPR (2016). The network generates decision actions for all agents in the system based on the joint observations of all agents in each system.

[0009] The present invention relates to a multi-agent system self-organizing mechanism for implementing the above method, comprising: an initialization unit, a binary decision unit, a graph model construction unit, an organization control unit, a multi-agent decision unit and a model optimization unit, wherein: the initialization unit constructs and randomly initializes the organization control network and the multi-agent decision network; the binary decision unit inputs the local observation of each intelligent agent into the organization control network, and outputs the dynamic teaming strategy of each intelligent agent, that is, the binary decision of whether to form a system with adjacent intelligent agents; the graph model construction unit models the entire multi-agent system as a graph model according to the binary decision output by the organization control network; the organization control unit Traditional graph theory algorithms are used to find all weakly connected components in the graph model, and all agents are grouped according to the weakly connected components. Agents belonging to the same weakly connected component will be divided into the same system; the multi-agent decision unit uses the multi-agent decision network to generate decision actions for all agents based on the joint observations of each system, and interacts with the environment to obtain each agent's respective reward feedback. At the same time, the organizational control network and the multi-agent decision network are used to calculate the organizational control loss and decision loss based on the rewards obtained by each agent; the model optimization unit optimizes the organizational control network and the multi-agent decision network according to the organizational control loss and decision loss.

[0010] Technical Effects

[0011] The present invention models the self-organization problem of the sorting robot cluster in the large-scale automatic sorting task as a multi-agent reinforcement learning problem through binary decision making. Each agent can spontaneously decide whether to form a system with the neighboring agent through a binary decision unit. Compared with the existing multi-agent reinforcement learning technology, which mainly groups agents in a predefined way, such as selecting the nearest k agents, or selecting all agents in a circular area with a radius of r to form a system, the organization control unit uses a traditional graph theory algorithm to find all the weakly connected components in the directed graph output by the graph model construction unit, and groups all agents according to the weakly connected components. Agents belonging to the same weakly connected component will be divided into the same system. The present invention applies traditional graph theory algorithms to the self-organization problem of agents, and can ultimately improve the time efficiency and task completion of the automatic sorting task. However, due to its heuristic grouping strategy, the prior art cannot achieve the optimal time efficiency and task completion. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flow chart of the present invention;

[0013] Figure 2 This is a schematic diagram of the collaborative task;

[0014] Figure 3 This is a schematic diagram of the collaborative task;

[0015] Figure 4 Schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0016] Example 1

[0017] This embodiment relates to the specific implementation of the multi-agent system self-organization method based on enhanced organizational control in the large-scale automatic sorting scenario of sorting robot clusters in the field of warehouse automation, such as Figure 2 shown.

[0018] In the sorting task of this embodiment, 12 sorting robots are located in the central area of ​​the square map, and 32 shelves are evenly distributed in the four corner areas of the square map. The sorting robot obtains rewards by navigating to the shelf. The local observation of each sorting robot includes all the information in the square area with a radius of 7 unit lengths centered on itself, that is, the two-dimensional coordinates of the remaining sorting robots and the two-dimensional coordinates of the shelf. The decision-making actions of the sorting robot include moving up, moving down, moving left, moving right, staying still and lifting the shelf, and the moving distance only includes the area of ​​one unit length around. The sorting robot can only perform the action of lifting the shelf after reaching the location of the shelf. Sorting robots will be punished if they collide with each other or perform the operation of lifting the shelf in a position where there is no shelf. At the same time, in order to encourage the sorting robot to complete the sorting task faster, it will also be punished less for each movement action it takes. The intelligent agent needs to learn to automatically group and reasonably decompose large-scale sorting tasks, so that each system can move to different shelf areas to sort the shelves.

[0019] like Figure 1 As shown, the above sorting task is specifically implemented through the following steps:

[0020] S110, construct and randomly initialize the organizational control network, multi-agent decision network and joint decision evaluation function, and use the Xavier method (Glorot, Xavier and Yoshua Bengio. "Understanding the difficulty of training deep feedforward neural networks." AISTATS (2010).) to initialize all parameters of the network / function.

[0021] The tissue control network comprises: an upsampling layer, five convolution layers and an average pooling layer, wherein: the upsampling layer upsamples the 14×14 two-dimensional local observation matrix of each sorting robot to 224×224, and then convolves with 64 7×7 convolution kernels with a step size of 2 to obtain a 112×112×64 feature map; after a maximum pooling of 3×3 with a step size of 2, it is convolved with 64 3×3 convolution kernels twice to obtain a 56×56×64 feature map; after two convolutions with 128 3×3 convolution kernels, two convolutions with 256 3×3 convolution kernels, and two convolutions with 256 3×3 convolution kernels, a 7×7×512 feature map is obtained, and finally after an average pooling of 7×7, a 512-dimensional feature vector is obtained; finally, after a layer of fully connected neural network, a 2-dimensional binary decision output is obtained.

[0022] The multi-agent decision network includes: an upsampling layer, five convolution layers and an average pooling layer, wherein: the upsampling layer splices the 14×14 two-dimensional local observation matrix of each sorting robot in the multi-agent system together to form a 14×14×3 joint observation. If the number of system members is less than 3, it is padded with 0; then the joint observation is upsampled to 224×224×3, and then convolved with 64 7×7 convolution kernels with a step size of 2 to obtain a 112×112×64 feature map; after 3 ×3, maximum pooling with a step size of 2, and then two convolutions with 64 3×3 convolution kernels to obtain a 56×56×64 feature map; then two convolutions with 128 3×3 convolution kernels, two convolutions with 256 3×3 convolution kernels, and two convolutions with 256 3×3 convolution kernels to obtain a 7×7×512 feature map, and finally after 7×7 average pooling, a 512-dimensional feature vector is obtained; finally, after a layer of fully connected neural network, a 6-dimensional decision action output of each intelligent agent is obtained.

[0023] As shown in Table 1, this is a schematic diagram of the partial network architecture of the organization control network and the multi-agent decision-making network.

[0024]

[0025] S120, receiving local observations of each agent in the multi-agent system through the organizational control network and outputting the dynamic teaming strategy of each agent according to the above processing flow, that is, the binary decision of whether to form a system with adjacent agents.

[0026] The binary decision-making refers to: the binary decision space of each agent i Belongs to an m-dimensional binary decision vector, which represents the decision of agent i based on its local observation The output and the surrounding m nearest neighbor agents The connection action of agent i and agent j, that is, whether there should be an edge connection between them.

[0027] S130, modeling the entire multi-agent system as a directed graph according to the binary decision, converting it into an undirected graph, and using the Tarjan algorithm to find all weakly connected components in the undirected graph.

[0028] In the directed graph, nodes represent agents and edges reflect binary decision results.

[0029] The undirected graph in: i, j represent any two different agents, represents the m neighboring agents corresponding to the two agents respectively; ∨ represents the “or” operation, that is, for two agents, as long as one of them decides to connect with the other, there will be an edge connection between the two agents.

[0030] The worst time complexity of the Tarjan algorithm is That is, the sum of the number of edges and the number of nodes in an undirected graph.

[0031] S140, grouping all agents according to all weakly connected components, that is, agents belonging to the same weakly connected component will be grouped into the same system.

[0032] S150, through the joint observation of each system by the multi-agent decision network, generate decision actions for all agents according to the above processing flow, and interact with the environment to obtain reward feedback for each agent; calculate the organizational control loss and decision loss according to the reward obtained by each agent through the organizational control network and the multi-agent decision network;

[0033] The reward mentioned above includes the sum of external reward (subscript e) and internal reward (subscript u), that is,

[0034] The external reward means that the intelligent agent, that is, the sorting robot, will receive a reward of -0.01 for each movement, a reward of 5 for successfully lifting the shelf, and a reward of -0.1 for lifting the shelf in a place where there is no shelf.

[0035] Internal rewards described Where: GED(·,·) represents the graph edit distance, and represents a subgraph containing only agent i and its m nearest neighbors before and after making a binary decision.

[0036] Loss of tissue control That is, the optimization goal of the organizational control network is to maximize the expected cumulative discounted reward of the agent, where: τ u Represents the complete trajectory sequence sampled by agent i in the environment, which consists of continuous local observations, actions, and rewards. The local observation is a 14x14 two-dimensional matrix, the action is a 6-dimensional unique hot encoding, and the reward is a one-dimensional real scalar; Represents the sum of all rewards in the trajectory sequence.

[0037] The optimization goal is achieved by using the DQN algorithm to obtain the organizational control loss by minimizing the following time difference: ,in: represents the organizational control network, which outputs binary decisions based on the local observations of the agent. The network outputs real numbers

[0038] The reward after the multi-agent decision network makes a decision is calculated in the following way: In addition to the external reward of the environment itself In addition, in order to stabilize the training process, the method additionally designs Figure 1 Consistent internal rewards (indicated by the subscript l)

[0039] The decision loss is calculated by the QMIX algorithm (Tabish Rashid et al. "QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning." ICML (2018).), specifically: in: Local decision loss The joint decision loss of all agents Satisfaction between local and global in: Represents a multi-agent decision network that outputs action decisions for each agent; represents the reward function for evaluating the joint action decision within the system, which is used to assist the training of the multi-agent decision network; QMX =0.01 indicates a positive real hyperparameter, which is used to balance the two losses; γ = 0.99 indicates a discount factor between 0 and 1, which is used to balance short-term and long-term gains.

[0040] The reward function for evaluating the decision of joint actions in the system has a specific architecture in which the input of the reward function includes the joint observations of all agents in the system and the joint actions of all agents in the system. The above input information is processed separately by two branches. First, the reward function splices the 14x14 two-dimensional local observation matrix of each sorting robot in the system together to form a 14x14x3 joint observation. If the number of system members is less than 3, it is padded with 0; then the joint observation is upsampled to 224x224x3, and then convolved with 64 7x7 convolution kernels with a step size of 2 to obtain a 112x112x64 feature map; after maximum pooling of 3x3 with a step size of 2, it is convolved with 64 3x3 convolution kernels twice to obtain a 56x56x64 feature map; then it is convolved with 128 3x3 convolution kernels twice, and two convolutions are performed respectively. After 256 convolutions with 3x3 kernels and two convolutions with 256 3x3 kernels, a 7x7x512 feature map is obtained, and finally after 7x7 average aggregation, a 512-dimensional feature vector is obtained; at the same time, the reward function concatenates the decisions represented by the 6-dimensional unique hot vectors made by each sorting robot in the system according to the multi-agent decision network to form a 36-dimensional joint decision vector. After two layers of fully connected neural networks with 32 units, a 32-dimensional joint decision code is obtained, and finally a one-dimensional scalar is obtained after a layer of fully connected neural network. The size of the scalar represents the value of the joint decision.

[0041] S160, optimize the organizational control network and the multi-agent decision network according to the organizational control loss and decision loss. Specifically: use the Adam optimizer to optimize all network parameters, use a learning rate of 0.0001, a batch size of 512, a number of training rounds of 500, a maximum number of iterations in each round of 250, and a parameter update frequency of 4 times per batch of data.

[0042] The data required for algorithm training is generated in the following way. In each time step of each round of training, the sorting robots obtain their own decision actions based on the organizational control network and the multi-agent decision network, and execute them in the simulation environment. After the environment receives the joint decision of all sorting robots, it will update the state of each sorting robot, that is, the two-dimensional coordinate information, and the state of the shelf, and calculate the reward for each sorting robot. Therefore, each time step will generate a triple such as (joint observation, joint decision, reward). The algorithm puts the triples generated at each time step into an experience replay cache to form the data set required for algorithm training. In Example 1, the size of the experience replay cache is 1,000,000.

[0043] After ten experiments with different random seed initializations, the first embodiment of the present invention can achieve a cumulative reward value of 889 (±78), and the value in brackets represents the variance of the algorithm; in all simulation experiments, the algorithm can achieve 100% task completion, that is, sorting to all 32 shelves.

[0044] Example 2

[0045] Based on Example 1, this embodiment introduces interference factors into the sorting environment, such as observation noise, decision delay, and stray target, which will affect the behavior of the sorting robot. This embodiment generally and abstractly models these real interference factors by introducing additional interference agents. Specifically, compared with Example 1, the automatic sorting tasks handled by this embodiment are as follows: Figure 3 As shown, it includes: 16 interference robots, 16 faster-moving sorting robots and 32 shelves. The sorting robot can only be rewarded if it successfully reaches the shelf position and lifts the shelf, while the interference robot obtains the reward by reaching the position of the sorting robot. . The actions of all intelligent agents include moving up, moving down, moving left, moving right and staying still; the sorting robot has the additional action of lifting the shelf. The movement range of the interference robot is 1 unit length around it, and the movement range of the sorting robot is 2 unit lengths around it. Sorting robots will be punished not only for colliding with each other, but also for colliding with interference robots. . Since the moving speed of the sorting robot is greater than that of the interference robot, the sorting robot needs to learn how to decompose large-scale tasks by grouping, and also learn how to avoid interference robots. In addition to the above settings, the remaining settings of this embodiment are consistent with Example 1.

[0046] In this embodiment, for the external rewards of the environment itself, all robot movements will not be subject to additional penalties. The sorting robot will be penalized -1 if it is caught up by the interfering robot, and the interfering robot will receive a reward of +1 accordingly. The sorting robot will receive a reward of +5 if it successfully lifts a shelf. Similarly, the sorting robot will receive a penalty of -0.1 if it lifts a shelf in a location where there is no shelf.

[0047] In this embodiment, the interference robot is trained using the same algorithm as the sorting robot. As the training progresses, the interference robot's ability will become stronger and stronger, which means that the interference amplitude in the real environment is getting larger and larger. The sorting robot needs to learn to adopt a better cooperation strategy to counter the increasing interference amplitude during the training process.

[0048] After ten experiments with different random seed initializations, this embodiment 2 can achieve a cumulative reward value of 635 (±38). The values ​​in brackets represent the variance of the algorithm; in all simulation experiments, the algorithm can eventually achieve 100% task completion, that is, sorting all 32 shelves in the presence of interfering robots.

[0049] Example 3

[0050] like Figure 4 As shown, a self-organizing mechanism of a multi-agent system based on enhanced organizational control to implement the above method includes: an initialization unit 510, a binary decision unit 520, a graph model construction unit 530, an organizational control unit 540, a multi-agent decision unit 550 and a model optimization unit 560, wherein: the initialization unit 510 constructs and randomly initializes the organizational control network and the multi-agent decision network; the binary decision unit 520 inputs the local observation of each agent into the organizational control network, and outputs the dynamic teaming strategy of each agent, that is, the binary decision of whether to form a system with adjacent agents; the graph model construction unit 530 models the entire multi-agent system as a graph model according to the binary decision output by the organizational control network, in which the nodes represent the agents and the edges reflect the binary decision results; The organization control unit 540 uses traditional graph theory algorithms to find all weakly connected components in the graph model, and groups all agents according to the weakly connected components. Agents belonging to the same weakly connected component will be divided into the same system; the multi-agent decision unit 550 uses the multi-agent decision network to generate decision actions for all agents based on the joint observations of each system, and interacts with the environment to obtain reward feedback for each agent. At the same time, the organization control network and the multi-agent decision network are used to calculate the organization control loss and decision loss based on the reward obtained by each agent; the model optimization unit 560 uses the gradient back propagation algorithm to update all parameters of the organization control network, the multi-agent decision network and the joint decision evaluation function according to the organization control and decision losses obtained by the multi-agent decision unit.

[0051] The initialization unit 510 includes: an organization control network initialization unit, a multi-agent decision network initialization unit and a joint decision evaluation function initialization unit. The three units respectively perform parameter initialization according to the specific architecture information of the organization control network, the multi-agent decision network and the joint decision evaluation function using the Xavier method.

[0052] The organization control unit 540 includes: a weakly connected component search unit and a grouping unit, wherein: the weakly connected component search unit uses traditional graph theory algorithms to find all weakly connected components in the graph model, and the grouping unit groups all intelligent agents according to all the searched weakly connected components, and intelligent agents belonging to the same weakly connected component will be divided into the same system.

[0053] The multi-agent decision unit 550 includes: an agent decision unit, an environment interaction unit and a loss calculation unit, wherein: the agent decision unit uses the multi-agent decision network to generate decision actions for all agents according to the joint observation of each system output by the organization control unit; the environment interaction unit inputs the decision actions of all agents into the simulation environment to obtain the updated states of all agents and the rewards fed back by the environment; finally, the loss calculation unit calculates the organization control and decision losses according to the rewards of all agents fed back by the environment.

[0054] After specific practical experiments, under the specific environment settings of Example 1 and Example 2, the Adam optimizer was used to optimize all network parameters, the learning rate used was 0.0001, the batch size was 512, the number of training rounds was 500, the maximum number of iterations in each round was 250, and the parameter update frequency was 4 times per batch of data.

[0055] In the experiment of running ten times with different random seed initializations with the above hyperparameters, a cumulative reward value of 889 (±78) can be achieved in the setting of Example 1, and the value in brackets indicates the variance of the algorithm; in all simulation experiments, the algorithm can achieve 100% task completion, that is, sorting to all 32 shelves; after ten experiments with different random seed initializations, in the setting of Example 2, a cumulative reward value of 635±38 can be achieved. The value in brackets indicates the variance of the algorithm; in all simulation experiments, the algorithm can finally achieve 100% task completion, that is, sorting to all 32 shelves in the presence of interfering robots.

[0056] Compared with the prior art, the performance of the device of the present invention is superior to the existing multi-agent reinforcement learning method in terms of the scale of sorting robots and shelves that can be processed, the time efficiency of completing sorting tasks, and the degree of completion of sorting tasks.

[0057] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A self-organizing method for a multi-agent system based on enhanced organizational control, characterized in that: By constructing and randomly initializing the organizational control network and the multi-agent decision network, the organizational control network generates corresponding binary decisions for the local observations of each sorting robot as an intelligent agent, and based on it, the system composed of multi-agents is modeled as a directed graph, and all weakly connected components are extracted from it to group the intelligent agents; then, the multi-agent decision network generates decision actions for all intelligent agents based on the joint observations of each intelligent agent system, and interacts with the environment to obtain the reward feedback of each intelligent agent; then, the organizational control loss and decision loss are calculated based on the rewards through the organizational control network and the multi-agent decision network, so as to optimize the organizational control network and the multi-agent decision network; finally, the sorting robot cluster will spontaneously, dynamically and reasonably group in completing large-scale sorting tasks, decomposing large-scale sorting tasks into independent small-scale sub-tasks, and the intelligent agent system will generate efficient collaboration strategies through multi-agent reinforcement learning methods, so as to effectively complete each sub-task, and finally achieve a high degree of task completion in large-scale automatic sorting tasks; The binary decision-making refers to: the binary decision space of each agent i Belongs to an m-dimensional binary decision vector, which represents the decision of agent i based on its local observation The output and the surrounding m nearest neighbor agents The connection action, that is, whether there should be an edge connection between agent i and agent j; The organizational control network is a residual neural network, which receives local observations of each agent, i.e., the sorting robot, as input and outputs a dynamic teaming strategy for each agent, i.e., a binary decision of whether to form a system with neighboring agents; The multi-agent decision network is a residual neural network that generates decision actions for all agents in the system based on the joint observations of all agents in each system; Loss of tissue control in: represents the organizational control network, which outputs binary decisions based on the local observations of the agent. The network outputs real numbers It is the sum of external rewards and internal rewards. For external rewards, For internal rewards; The decision loss is calculated by the QMIX algorithm, specifically: in: Local decision loss The joint decision loss of all agents Satisfaction between local and global in: Represents a multi-agent decision network that outputs action decisions for each agent; represents the reward function for evaluating the joint action decision within the system, which is used to assist the training of the multi-agent decision network; QMIX =0.01 indicates a positive real hyperparameter, which is used to balance the two losses; γ = 0.99 indicates a discount factor between 0 and 1, which is used to balance short-term and long-term gains.

2. The multi-agent system self-organization method based on enhanced organizational control according to claim 1 is characterized in that: The tissue control network includes: an upsampling layer, five convolution layers and an average pooling layer, wherein: the upsampling layer upsamples the 14×14 two-dimensional local observation matrix of each sorting robot to 224×224, and then convolves with 64 7×7 convolution kernels with a step size of 2 to obtain a 112×112×64 feature map; after a maximum pooling of 3×3 with a step size of 2, it is convolved with 64 3×3 convolution kernels twice to obtain a 56×56×64 feature map; after two convolutions with 128 3×3 convolution kernels, two convolutions with 256 3×3 convolution kernels, and two convolutions with 256 3×3 convolution kernels, a 7×7×512 feature map is obtained, and finally after an average pooling of 7×7, a 512-dimensional feature vector is obtained; finally, after a layer of fully connected neural network, a 2-dimensional binary decision output is obtained; The multi-agent decision network includes: an upsampling layer, five convolution layers and an average pooling layer, wherein: the upsampling layer splices the 14×14 two-dimensional local observation matrix of each sorting robot in the multi-agent system together to form a 14×14×3 joint observation. If the number of system members is less than 3, it is padded with 0; then the joint observation is upsampled to 224×224×3, and then convolved with 64 7×7 convolution kernels with a step size of 2 to obtain a 112×112×64 feature map; after 3 ×3, maximum pooling with a step size of 2, and then two convolutions with 64 3×3 convolution kernels to obtain a 56×56×64 feature map; then two convolutions with 128 3×3 convolution kernels, two convolutions with 256 3×3 convolution kernels, and two convolutions with 256 3×3 convolution kernels to obtain a 7×7×512 feature map, and finally after 7×7 average pooling, a 512-dimensional feature vector is obtained; finally, after a layer of fully connected neural network, a 6-dimensional decision action output of each intelligent agent is obtained.

3. A multi-agent system self-organizing mechanism for implementing the method described in claim 1 or 2, characterized in that: include: Initialization unit, binary decision unit, graph model construction unit, organization control unit, multi-agent decision unit and model optimization unit, among which: the initialization unit constructs and randomly initializes the organization control network and the multi-agent decision network; the binary decision unit inputs the local observation of each agent into the organization control network, and outputs the dynamic teaming strategy of each agent, that is, the binary decision of whether to form a system with the neighboring agents; the graph model construction unit models the entire multi-agent system as a graph model according to the binary decision output by the organization control network; the organization control unit uses the traditional graph theory algorithm to find all the weakly connected components in the graph model, and according to the weakly connected components, All agents are grouped, and agents belonging to the same weakly connected component will be divided into the same system; the multi-agent decision unit uses the multi-agent decision network to generate decision actions for all agents based on the joint observations of each system, and interacts with the environment to obtain reward feedback for each agent. At the same time, the organizational control network and the multi-agent decision network are used to calculate the organizational control loss and decision loss based on the rewards obtained by each agent; the model optimization unit uses the gradient backpropagation algorithm to update all parameters of the organizational control network, the multi-agent decision network and the joint decision evaluation function based on the organizational control and decision losses obtained by the multi-agent decision unit.

4. The multi-agent system self-organizing mechanism according to claim 3, characterized in that: The initialization unit includes: an organization control network initialization unit, a multi-agent decision network initialization unit and a joint decision evaluation function initialization unit. The three units respectively perform parameter initialization according to the specific architecture information of the organization control network, the multi-agent decision network and the joint decision evaluation function using the Xavier method.

5. The multi-agent system self-organizing mechanism according to claim 3, characterized in that: The organization and control unit includes: a weakly connected component search unit and a grouping unit, wherein: the weakly connected component search unit uses traditional graph theory algorithms to find all weakly connected components in the graph model, and the grouping unit groups all intelligent agents according to all the searched weakly connected components, and intelligent agents belonging to the same weakly connected component will be divided into the same system.

6. The multi-agent system self-organizing mechanism according to claim 3, characterized in that: The multi-agent decision-making unit includes: an agent decision unit, an environment interaction unit and a loss calculation unit, wherein: the agent decision unit uses a multi-agent decision network to generate decision actions for all agents according to the joint observation of each system output by the organization control unit, the environment interaction unit inputs the decision actions of all agents into the simulation environment, obtains the updated states of all agents and the rewards fed back by the environment, and finally, the loss calculation unit calculates the organizational control and decision losses according to the rewards of all agents fed back by the environment.

Images