Transform-based multi-robot cooperative working method and system in automatic logistics scene

Abstract

The invention discloses a Transform-based multi-robot cooperative working method and system in an automatic logistics scene, and the method comprises the steps: obtaining one piece of incomplete observation information of a current environment through each robot, and carrying out the time sequence modeling, spatial modeling, Q value generation, motion selection and award value obtaining. In the process, incomplete observation information of each robot, global state information of the current environment and a reward value are stored in an experience playback pool B, the above steps are repeated, and finally model optimization is completed by using data stored in the B. The problem of partial observability and the problem of reputation allocation in the multi-robot cooperative working process are solved.

Description

Technical Field

[0001] The present invention belongs to the field of multi-agent reinforcement learning, and specifically relates to a Transformer-based multi-robot collaborative working method and system in an automated logistics scenario. Background Art

[0002] In multi-robot collaborative decision-making, the partial observability problem refers to the fact that robots may only be able to perceive obstacles within a certain range, shelf locations, the locations of other robots, and other information. If there is a blockage nearby or the path is occupied, they may not be able to know immediately. This causes robots to face information asymmetry when collaborating, which may lead to wrong decisions or inconsistent strategies. The key to solving this problem is how to enable each robot to obtain more complete observation information, so as to achieve better coordination and cooperation.

[0003] In multi-robot collaborative decision-making, the credit allocation problem refers to the cooperative behavior between robots (such as completing their own tasks, avoiding other robots, preventing congestion, etc.), which is often difficult to quantify directly, but has a significant impact on overall efficiency. How to reflect these contributions in rewards is a major challenge. This complexity stems from the nonlinear characteristics of collective behavior. The collective responsibilities and contributions are not equal to the simple accumulation of the responsibilities and contributions of each robot. The responsibilities and contributions of robots in the entire environment are not only affected by their own behavior, but also by the actions of other robots and environmental changes.

[0004] The sensors in the present invention are Pandar64 lidar sensor from Hesai Technology, RealSense D455 depth vision sensor from Intel, and FSG15N1A pressure sensor from Honeywell. Summary of the invention

[0005] The present invention aims to solve some observability problems and credibility distribution problems existing in the prior art in multi-robot collaborative decision-making, which cause robots to face information asymmetry when collaborating, which may lead to decision-making errors or inconsistent strategies, as well as how to evaluate and allocate the responsibilities and contributions of each robot. The responsibilities and contributions of robots in the entire environment are affected not only by their own behavior, but also by the actions of other robots and environmental changes.

[0006] In order to solve the above problems, the present invention provides a Transformer-based multi-robot collaborative working method in an automated logistics scenario, comprising the following steps:

[0007] Step S1: data acquisition;

[0008] At time t, each robot scans the entire environment through the sensors it carries and obtains incomplete observation information of the current environment, which can be expressed as: The observation information includes the following parts: self-status information: the current position, direction and speed of the robot in the warehouse, whether it is carrying goods, the type and weight of the current load, and the remaining battery power of the robot; surrounding environment information: the position of other robots, the position of shelves, the position of obstacles such as walls, the degree of crowding or blockage of nearby passages; task-related information: the type, volume and quantity of items to be transported, and the specific location of the task destination. At the same time, by splicing the observation information of all robots, a global state information of the current environment is obtained. t , store both together in an experience replay pool B;

[0009] Step S2: timing modeling;

[0010] Robot i sends its own observation information Input into the recurrent neural network (GRU network) used to extract time series information to obtain the time series observation representation of robot i The formula is as follows:

[0011]

[0012] in, represents the action taken by robot i at time t-1, represented by one-hot encoding and stored in the experience replay pool B. is the time series observation representation of robot i at time t-1, initialized to 0 vector. After this step, the time series observation representation of all robots at time t is obtained

[0013] Step S3: spatial modeling;

[0014] The temporal observation representations of all robots obtained in step S2 are simultaneously input into the Transformer Encoder to extract the spatial information between robots and obtain the final observation representations of all robots. The formula is as follows:

[0015]

[0016] Among them, W q , W k , W v They represent the parameter matrices for generating query vectors, key vectors, and value vectors in the self-attention mechanism, respectively. k Represents the dimension of the query vector;

[0017] Step S4: generating a Q value;

[0018] Q value is used in reinforcement learning to estimate the expected return value of a state-action pair under the current strategy. Specifically, Q value represents the cumulative reward that the robot expects to obtain in the future after selecting an action in the current state. The calculation of Q value is to measure the value of different actions. By calculating Q value, the robot can evaluate the potential benefits of each action based on the current observation information and select the best action. To this end;

[0019] The observation representation obtained in step S3 Input them into a Q network composed of a multi-layer perceptron MLP to obtain the Q value vector of robot i The formula is as follows:

[0020]

[0021] Among them, K represents the number of action categories that robot i can choose. Through this step, the Q value of all robots at time t is obtained

[0022] Step S5: All robots follow the ε-greedy strategy to choose their own actions at time t The formula is as follows:

[0023]

[0024] Among them, ε∈(0,1) is a hyperparameter, random is a randomly generated value between 0 and 1, and then each robot performs the corresponding action, obtains the reward r at time t, and sets a t and r are stored in the experience replay pool B;

[0025] Step S6: Repeat steps S1-S5 until the task is completed at time T;

[0026] Based on the above model building method, the training process is as follows:

[0027] W1: Data acquisition;

[0028] Sample a complete round of data from the experience replay pool B {o 1 ,o 2 ,…o T},{a 0 ,a 1 ,…a T-1},{s 1 ,s 2 ,…s T},{r 1 ,r2 ,…r T},in

[0029] W2: For each time t, execute steps S1-S5 to obtain the final observation representation of all robots And the Q value vectors of all robots in And according to the action of each robot at time t Get the Q value corresponding to each robot at time t The formula is as follows:

[0030]

[0031] W3: The global state information obtained by sampling s = {s 1 ,s 2 ,…s T}After adding it to the position code, it is input into the mask attention mechanism in the Transformer Decoder to obtain a global state representation at time t The formula is as follows:

[0032]

[0033] Among them, M represents the mask matrix, which makes the model at time t unable to see the global state information after time t. It's N*d e A vector of dimension, d e To observe the characterization The corresponding dimension will be Divide into N d e dimensional vector, and then N d e The vectors of dimension are combined into a matrix S t ;

[0034] W4: S obtained in step W3 t As the query input of the next layer of attention mechanism, and the e obtained in step W2 is t As the key and value input of the attention mechanism, we finally get the role representation of each robot l = (l1, l2, ..., l N ), the formula is as follows:

[0035]

[0036] W5:(l1,l2,…,l N ) are concatenated into a vector Will and Input them into the value mixing network together, and finally get Q tot , the formula is as follows:

[0037]

[0038] Among them, α is the parameter of the value mixing network, and α is based on produce;

[0039] W6: Calculate the loss function and update the entire network according to the following formula:

[0040] L(θ)=E τ,a [|yQ tot (τ,a;θ)|]

[0041] y=r+γagr Q max a Q tot (τ next ,a;θ - )

[0042] Where r is the sampled reward value, τ = (o 1 ,a 1 ,o 2 ,a 2 …o t ) is the historical trajectory, τ next is the historical trajectory of the next moment, θ is the parameter of the entire online network, θ - It is the parameter of the target network. The target network has the same architecture as the online network. The parameters of the online network are regularly copied to update the parameters of the target network to ensure the stability of training.

[0043] Preferably, the target network in step W6 has the same architecture as the online network.

[0044] A Transformer-based multi-robot collaborative decision-making reinforcement learning system in a complex environment, including: a data acquisition module, a time series modeling module, and a space modeling module;

[0045] Step M1: The data acquisition module is implemented as follows:

[0046] At time t, each robot scans the entire environment through the sensors it carries and obtains incomplete observation information of the current environment, which can be expressed as: The observation information includes the following parts: self-status information: the current position, direction and speed of the robot in the warehouse, whether it is carrying goods, the type and weight of the current load, and the remaining battery power of the robot; surrounding environment information: the position of other robots, the position of shelves, the position of obstacles such as walls, the degree of crowding or blockage of nearby passages; task-related information: the type, volume and quantity of items to be transported, and the specific location of the task destination. At the same time, by splicing the observation information of all robots, a global state information of the current environment is obtained. t , store both together in an experience replay pool B;

[0047] Step M2: The timing modeling module is implemented as follows:

[0048] Robot i sends its own observation information Through the modeling module, it is input into the recurrent neural network (GRU network) used to extract timing information to obtain the timing observation representation of robot i The formula is as follows:

[0049]

[0050] in, represents the action taken by robot i at time t-1, represented by one-hot encoding and stored in the experience replay pool B. is the time series observation representation of robot i at time t-1, initialized to 0 vector. After this step, the time series observation representation of all robots at time t is obtained

[0051] Step M3: The spatial modeling module is implemented as follows:

[0052] The temporal observation representations of all robots obtained in step M2 are simultaneously input into the Transformer Encoder through the modeling module to extract the spatial information between robots and obtain the final observation representations of all robots. The formula is as follows:

[0053]

[0054] Among them, W q , W k , W v They represent the parameter matrices for generating query vectors, key vectors, and value vectors in the self-attention mechanism, respectively. k Represents the dimension of the query vector;

[0055] Step M4: Generate Q value;

[0056] Q value is used in reinforcement learning to estimate the expected return value of a state-action pair under the current strategy. Specifically, Q value represents the cumulative reward that the robot expects to obtain in the future after selecting an action in the current state. The calculation of Q value is to measure the value of different actions. By calculating Q value, the robot can evaluate the potential benefits of each action based on the current observation information and select the best action. To this end;

[0057] Represent the observations obtained in step M3 Input them into a Q network composed of a multi-layer perceptron MLP to obtain the Q value vector of robot i The formula is as follows:

[0058]

[0059] Among them, K represents the number of action categories that robot i can choose. Through this step, the Q value of all robots at time t is obtained

[0060] Step M5: All robots follow the ε-greedy strategy to choose their own actions at time t The formula is as follows:

[0061]

[0062] Among them, ε∈(0,1) is a hyperparameter, random is a randomly generated value between 0 and 1, and then each robot performs the corresponding action, obtains the reward r at time t, and sets a t and r are stored in the experience replay pool B;

[0063] Step M6: Repeat steps M1-M5 until the task is completed at time T.

[0064] Beneficial effects of the present invention: The present invention combines the encoder of GRU and Transformer during the execution process, and by modeling the observation sequence of the robot, the robot can effectively combine the historical observation information with the observation information at the current moment and the observation information of other robots at the current moment. The robot obtains more comprehensive state information of the entire warehouse, reducing task delays, path conflicts, repeated path planning and unnecessary waiting caused by missing information. It is crucial for the robot to make correct decisions. In this way, the decision-making ability of the robot is significantly improved, enabling it to effectively coordinate and cooperate with other robots in complex dynamic scenes. In addition, the decoder of Transformer is introduced in the training process to guide the global state to focus on the representation of each robot based on the attention mechanism, so that more expressive credit allocation can be performed in the subsequent value mixing process. By solving the credit allocation problem, the collaborative contribution between robots can be effectively quantified. For example, when a robot actively avoids or helps clear the channel to avoid obstruction, this collaborative behavior will be identified and rewarded, thereby motivating the robot to work better together in future tasks. In this way, the robot will give priority to behaviors that are beneficial to the system as a whole, reduce task conflicts and unnecessary waiting, and improve the overall operation efficiency of the warehouse. Finally, the loss function of the entire network is calculated by the temporal difference method, and then backpropagation is performed to update the entire network. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a schematic diagram of the training process architecture of the present invention;

[0066] Figure 2 It is a schematic diagram of the training process steps of the present invention. DETAILED DESCRIPTION

[0067] Embodiment 1:

[0068] A Transformer-based multi-robot collaborative working method in an automated logistics scenario includes the following steps:

[0069] Step S1: data acquisition;

[0070] At time t, each robot scans the entire environment through the sensors it carries and obtains incomplete observation information of the current environment, which can be expressed as: The observation information includes the robot's own status information: the robot's current position, direction and speed in the warehouse, whether it is carrying goods, the type and weight of the current load, and the robot's remaining power information; the surrounding environment information: the location of other robots, the location of shelves, the location of obstacles such as walls, the degree of congestion or blockage of nearby passages; task-related information: the type, volume and quantity of items to be transported, and the specific location of the task destination; at the same time, by splicing the observation information of all robots, a global status information of the current environment is obtained. t , store both together in an experience replay pool B;

[0071] Step S2: timing modeling;

[0072] Robot i sends its own observation information Input into the recurrent neural network for extracting timing information to obtain the timing observation representation of robot i The formula is as follows:

[0073]

[0074] in, represents the action taken by robot i at time t-1, represented by one-hot encoding and stored in the experience replay pool B. is the time series observation representation of robot i at time t-1, initialized to 0 vector. After this step, the time series observation representation of all robots at time t is obtained

[0075] Step S3: spatial modeling;

[0076] The temporal observation representations of all robots obtained in step S2 are simultaneously input into the Encoder of the Transformer to extract the spatial information between the robots and obtain the final observation representations of all robots. The formula is as follows:

[0077]

[0078] Among them, W q , W k , W v They represent the parameter matrices for generating query vectors, key vectors, and value vectors in the self-attention mechanism, respectively. k Represents the dimension of the query vector;

[0079] Step S4: generating a Q value;

[0080] The observation representation obtained in step S3 Input them into a Q network composed of a multi-layer perceptron MLP to obtain the Q value vector of robot i The formula is as follows:

[0081]

[0082] Among them, K represents the number of action categories that robot i can choose. Through this step, the Q value of all robots at time t is obtained

[0083] Step S5: All robots follow the ε-greedy strategy to choose their own actions at time t The formula is as follows:

[0084]

[0085] Among them, ε∈(0,1) is a hyperparameter, random is a randomly generated value between 0 and 1, and then each robot performs the corresponding action, obtains the reward r at time t, and sets a t and r are stored in the experience replay pool B;

[0086] Step S6: Repeat steps S1-S5 until the task is completed at time T;

[0087] Based on the above method, the training process is as follows:

[0088] W1: Data acquisition;

[0089] Sample a complete round of data from the experience replay pool B {o 1 ,o 2 ,…o T},{a 0 ,a 1 ,…a T-1},{s 1 ,s 2 ,…s T},{r 1 ,r 2 ,…r T},in

[0090] W2: For each time t, execute steps S1-S5 to obtain the final observation representation of all robots And the Q value vectors of all robots in And according to the action of each robot at time t Get the Q value corresponding to each robot at time t The formula is as follows:

[0091]

[0092] W3: The global state information obtained by sampling s = {s 1 ,s 2 ,…s T}After adding it to the position code, it is input into the mask attention mechanism in the Transformer Decoder to obtain a global state representation at time t The formula is as follows:

[0093]

[0094] Among them, M represents the mask matrix, which makes the model at time t unable to see the global state information after time t. It's N*d e A vector of dimension, d e To observe the characterization The corresponding dimension will be Divide into N d e dimensional vector, and then N d e The vectors of dimension are combined into a matrix S t ;

[0095] W4: S obtained in step W3 t As the query input of the next layer of attention mechanism, and the e obtained in step W2 is t As the key and value input of the attention mechanism, we finally get the role representation of each robot l = (l1, l2, ..., l N ), the formula is as follows:

[0096]

[0097] W5:(l1,l2,…,l N ) are concatenated into a vector Will and Input them into the value mixing network together, and finally get Q tot , the formula is as follows:

[0098] Q tot =f α (q t )

[0099]

[0100] Among them, α is the parameter of the value mixing network, and α is based on produce;

[0101] W6: Calculate the loss function and update the entire network according to the following formula:

[0102] L(θ)=E τ,a [|yQ tot (τ,a;θ)|]

[0103] y=r+γagr Q max a Q tot (τ next ,a;θ - )

[0104] Where r is the sampled reward value, τ = (o 1 ,a 1 ,o 2 ,a 2 …o t ) is the historical trajectory, τ next is the historical trajectory of the next moment, θ is the parameter of the entire online network, θ - It is the parameter of the target network. The parameters of the online network are regularly copied to update the parameters of the target network to ensure the stability of training.

[0105] The target network in step W6 has the same architecture as the online network.

[0106] A Transformer-based multi-robot collaborative working system in an automated logistics scenario, characterized by comprising: a data acquisition module, a time series modeling module, and a space modeling module;

[0107] Step M1: The data acquisition module is implemented as follows:

[0108] At time t, each robot scans the entire environment through the sensors it carries and obtains incomplete observation information of the current environment, which can be expressed as: The observation information includes the following parts: self-state information: the current position, direction and speed of the robot in the warehouse, whether it is carrying goods, the type and weight of the current load, and the remaining power of the robot; surrounding environment information: the position of other robots, the position of shelves, the position of obstacles such as walls, the degree of congestion or blockage of nearby passages; task-related information: the type, volume and quantity of items to be transported, and the specific location of the task destination; at the same time, by splicing the observation information of all robots, a global state information of the current environment is obtained. t , store both together in an experience replay pool B;

[0109] Step M2: The timing modeling module is implemented as follows:

[0110] Robot i sends its own observation information Through the modeling module, it is input into the recurrent neural network (GRU network) used to extract timing information to obtain the timing observation representation of robot i The formula is as follows:

[0111]

[0112] in, represents the action taken by robot i at time t-1, represented by one-hot encoding and stored in the experience replay pool B. is the time series observation representation of robot i at time t-1, initialized to 0 vector. After this step, the time series observation representation of all robots at time t is obtained

[0113] Step M3: The spatial modeling module is implemented as follows:

[0114] The temporal observation representations of all robots obtained in step M2 are simultaneously input into the Transformer Encoder through the modeling module to extract the spatial information between the robots and obtain the final observation representations of all robots. The formula is as follows:

[0115]

[0116] Among them, W q , W k , W v They represent the parameter matrices for generating query vectors, key vectors, and value vectors in the self-attention mechanism, respectively. k Represents the dimension of the query vector;

[0117] Step M4: Generate Q value;

[0118] Represent the observations obtained in step M3 Input them into a Q network composed of a multi-layer perceptron MLP to obtain the Q value vector of robot i The formula is as follows:

[0119]

[0120] Among them, K represents the number of action categories that robot i can choose. Through this step, the Q value of all robots at time t is obtained

[0121] Step M5: All robots follow the ε-greedy strategy to choose their own actions at time t The formula is as follows:

[0122]

[0123] Among them, ε∈(0,1) is a hyperparameter, random is a randomly generated value between 0 and 1, and then each robot performs the corresponding action, obtains the reward r at time t, and sets a t and r are stored in the experience replay pool B;

[0124] Step M6: Repeat steps M1-M5 until the task is completed at time T.

[0125] The above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A Transformer-based multi-robot collaborative working method in an automated logistics scenario, characterized in that: The steps include: Step S1: data acquisition; At time t, each robot scans the entire environment through sensors and obtains incomplete observation information of the current environment, which is expressed as: The observation information includes the robot's own status information: the robot's current position, direction and speed in the warehouse, whether it is carrying goods, the type and weight of the current load, and the robot's remaining power information; the surrounding environment information: the location of other robots, the location of shelves, the location of obstacles such as walls, the degree of congestion or blockage of nearby passages; task-related information: the type, volume and quantity of items to be transported, and the specific location of the task destination; at the same time, by splicing the observation information of all robots, a global status information of the current environment is obtained. t , store both together in an experience replay pool B; Step S2: timing modeling; Robot i sends its own observation information Input into the recurrent neural network for extracting timing information to obtain the timing observation representation of robot i The formula is as follows: in, represents the action taken by robot i at time t-1, represented by one-hot encoding and stored in the experience replay pool B. is the time series observation representation of robot i at time t-1, initialized to 0 vector, and the time series observation representation of all robots at time t is obtained Step S3: spatial modeling; The temporal observation representations of all robots obtained in step S2 are simultaneously input into the Encoder of the Transformer to extract the spatial information between the robots and obtain the final observation representations of all robots. The formula is as follows: Among them, W q , W k , W v They represent the parameter matrices for generating query vectors, key vectors, and value vectors in the self-attention mechanism, respectively. k Represents the dimension of the query vector; Step S4: generating a Q value; The observation representation obtained in step S3 Input them into a Q network composed of a multi-layer perceptron MLP to obtain the Q value vector of robot i The formula is as follows: Among them, K represents the number of action categories that robot i can choose, and the Q value of all robots at time t is obtained Step S5: All robots follow the ε-greedy strategy to choose their own actions at time t The formula is as follows: Among them, ε∈(0,1) is a hyperparameter, random is a randomly generated value between 0 and 1, and the reward r at time t is obtained, and a t and r are stored in the experience replay pool B; Step S6: Repeat steps S1-S5 until the task is completed at time T; Based on the above method, the training process is as follows: W1: Data acquisition; Sample a complete round of data from the experience replay pool B {o 1 ,o 2 ,…o T },{a 0 ,a 1 ,…a T- 1},{s 1 ,s 2 ,…s T },{r 1 ,r 2 ,…r T },in W2: For each time t, execute steps S1-S5 to obtain the final observation representation of all robots And the Q value vectors of all robots in And according to the action of each robot at time t Get the Q value corresponding to each robot at time t The formula is as follows: W3: The global state information obtained by sampling s = {s 1 ,s 2 ,…s T }After adding it to the position code, it is input into the mask attention mechanism in the Transformer Decoder to obtain a global state representation at time t The formula is as follows: Among them, M represents the mask matrix, which makes the model at time t unable to see the global state information after time t. It's N*d e A vector of dimension, d e To observe the characterization The corresponding dimension will be Divide into N d e dimensional vector, and then N d e The vectors of dimension are combined into a matrix S t ; W4: S obtained in step W3 t As the query input of the next layer of attention mechanism, and the e obtained in step W2 is t As the key and value input of the attention mechanism, we finally get the role representation of each robot l = (l1, l2, ..., l N ), the formula is as follows: W5:(l1,l2,…,l N ) are concatenated into a vector Will and Input them into the value mixing network together, and finally get Q tot , the formula is as follows: Q tot =f α (q t ) Among them, α is the parameter of the value mixing network, and α is based on produce; W6: Calculate the loss function and update the entire network according to the following formula: L(θ)=E τ,a [|yQ tot (τ,a;θ)|] y=r+γagr Q max a Q tot (t next ,a;θ - ) Where r is the sampled reward value, τ = (o 1 ,a 1 ,o 2 ,a 2 …o t ) is the historical trajectory, τ next is the historical trajectory of the next moment, θ is the parameter of the entire online network, θ - It is the parameter of the target network. The parameters of the online network are regularly copied to update the parameters of the target network to ensure the stability of training.

2. The Transformer-based multi-robot collaborative working method in the automated logistics scenario according to claim 1 is characterized in that: The target network in step W6 has the same architecture as the online network.

3. A Transformer-based multi-robot collaborative working system in an automated logistics scenario, characterized in that: include: Data acquisition module, time series modeling module, and spatial modeling module; Step M1: The data acquisition module is implemented as follows: At time t, each robot scans the entire environment through the sensors it carries and obtains incomplete observation information of the current environment, which can be expressed as: The observation information includes the following parts: self-state information: the current position, direction and speed of the robot in the warehouse, whether it is carrying goods, the type and weight of the current load, and the remaining power of the robot; surrounding environment information: the position of other robots, the position of shelves, the position of obstacles such as walls, the degree of congestion or blockage of nearby passages; task-related information: the type, volume and quantity of items to be transported, and the specific location of the task destination; at the same time, by splicing the observation information of all robots, a global state information of the current environment is obtained. t , store both together in an experience replay pool B; Step M2: The timing modeling module is implemented as follows: Robot i sends its own observation information Through the modeling module, it is input into the recurrent neural network (GRU network) used to extract timing information to obtain the timing observation representation of robot i The formula is as follows: in, represents the action taken by robot i at time t-1, represented by one-hot encoding and stored in the experience replay pool B. is the time series observation representation of robot i at time t-1, initialized to 0 vector. After this step, the time series observation representation of all robots at time t is obtained Step M3: The spatial modeling module is implemented as follows: The temporal observation representations of all robots obtained in step M2 are simultaneously input into the Transformer Encoder through the modeling module to extract the spatial information between the robots and obtain the final observation representations of all robots. The formula is as follows: Among them, W q , W k , W v They represent the parameter matrices for generating query vectors, key vectors, and value vectors in the self-attention mechanism, respectively. k Represents the dimension of the query vector; Step M4: Generate Q value; Represent the observations obtained in step M3 Input them into a Q network composed of a multi-layer perceptron MLP to obtain the Q value vector of robot i The formula is as follows: Among them, K represents the number of action categories that robot i can choose. Through this step, the Q value of all robots at time t is obtained Step M5: All robots follow the ε-greedy strategy to choose their own actions at time t The formula is as follows: Among them, ε∈(0,1) is a hyperparameter, random is a randomly generated value between 0 and 1, and then each robot performs the corresponding action, obtains the reward r at time t, and sets a t and r are stored in the experience replay pool B; Step M6: Repeat steps M1-M5 until the task is completed at time T.

Images