Intelligent factory multi-device cooperative control method based on mutual information driven sparse reward

Abstract

The invention provides an intelligent factory multi-device cooperative control method based on mutual information driven sparse reward. According to the method, a loss function is constructed by maximizing mutual information between continuous observation features, and a neural network is used for outputting feature differences to generate distillation errors; meanwhile, an internal reward mechanism is constructed by calculating mutual information between the action of each device and the accumulated distillation error of other devices. And finally, fusing the distillation error and an internal reward design total reward function to realize cooperative control of a multi-equipment system. According to the method, the problem that effective strategies are difficult to learn due to the fact that traditional multi-agent reinforcement learning excessively depends on external reward guidance in a sparse reward scene is solved, the cooperative performance of a multi-device system is remarkably improved, and a key technical support is provided for application of a multi-agent reinforcement learning algorithm in the field of intelligent manufacturing.

Description

Technical Field

[0001] The present invention proposes a collaborative control method for multiple devices in an intelligent factory based on mutual information-driven sparse rewards. This method constructs a loss function by maximizing the mutual information between continuously observed features, and generates a distillation error using the difference in neural network output features. At the same time, an intrinsic reward mechanism is constructed by calculating the mutual information between the actions of each device and the cumulative distillation errors of other devices. Finally, the total reward function is designed by integrating the distillation error and the intrinsic reward to achieve collaborative control of multiple device systems. This method solves the problem that traditional multi-agent reinforcement learning is difficult to learn effective strategies in sparse reward scenarios due to excessive reliance on external reward guidance. It significantly improves the collaborative performance of multi-device systems and provides key technical support for the application of multi-agent reinforcement learning algorithms in the field of intelligent manufacturing. Background Art

[0002] The rapid development of artificial intelligence and machine learning technologies is driving the widespread application of multi-agent systems in fields such as military, aviation, and intelligent manufacturing. In intelligent manufacturing, multi-agent collaborative control has become a key technology, with applications spanning equipment collaboration, intelligent warehouse management, and workshop scheduling optimization. Traditional collaborative control methods typically rely on precise dynamic models, limiting their adaptability in complex dynamic environments. Therefore, developing model-free adaptive methods is crucial for achieving efficient collaborative control in dynamic industrial environments.

[0003] Multi-agent reinforcement learning-based approaches significantly improve the coordinated control performance of systems by promoting collaboration between devices. However, the effectiveness of such approaches relies on the real-time guidance of dense reward signals. A lack of rewards can lead to policy learning failures. However, dense rewards are difficult to design in real industrial scenarios, and communication-constrained environments can hinder signal transmission. Therefore, achieving efficient coordinated control in sparse reward scenarios has significant economic value and application prospects.

[0004] The current mainstream multi-device collaborative control method based on multi-agent reinforcement learning enhances policy diversity by injecting random perturbations into the action space to avoid being trapped in local optima. This mechanism improves the efficiency of device policy learning in dense reward scenarios, but it struggles to obtain effective learning samples in sparse reward environments, leading to policy failure. Therefore, it is crucial to design a multi-device collaborative control method that can effectively guide devices to collect high-quality experience data.

[0005] To address the sparse reward problem, existing research has constructed intrinsic reward mechanisms to drive devices to explore unknown state spaces, thereby increasing the probability of acquiring valid empirical data from the environment. However, the non-stationary nature of smart factory environments poses a core challenge: changes in the strategy of a single device can cascade through environmental dynamics, altering the decision-making processes of other devices. Therefore, intrinsic rewards that only encourage individual exploration are insufficient to ensure the coordinated control performance of multi-device systems; it is necessary to design intrinsic reward mechanisms that can model the interactions between devices to substantially improve the effectiveness of coordinated control.

[0006] This paper proposes a multi-device collaborative control method for smart factories based on mutual information-driven sparse rewards, aiming to address the challenge of efficient collaborative control of multi-device systems in sparse reward environments. First, a mutual information-based loss function is designed to model the temporal dependencies between observation features. Second, the distillation error between neural network output features is used as an intrinsic incentive to drive devices to actively explore unknown environmental states, thereby overcoming the lack of effective sample data in sparse reward scenarios. Subsequently, the mutual information principle and the distillation error are integrated to construct an intrinsic reward mechanism, which is combined with the extrinsic environmental reward to form a total reward function. This design not only drives devices to explore unknown areas, but also strengthens inter-device collaborative behavior through the collaborative properties of intrinsic rewards. Finally, a multi-agent proximal policy optimization algorithm is used to train the actor-critic network of each device to learn an efficient collaborative control policy model. This method significantly improves the ability of multi-device systems to perform complex collaborative tasks in sparse reward smart factory environments, providing innovative solutions and technical support for the application of multi-agent collaborative control in large-scale manufacturing, especially in sparse reward scenarios. Summary of the Invention

[0007] 1. A method for collaborative control of multiple devices in a smart factory based on mutual information-driven sparse rewards, characterized by constructing a sparse reward smart factory multi-device environment, designing a mutual information-based loss function, obtaining a distillation error, constructing a mutual information-driven intrinsic reward, establishing a total reward function, and using a multi-agent proximal policy optimization algorithm to train the multi-device system to achieve multi-device collaborative control. The method includes the following steps:

[0008] (1) Building a sparse reward smart factory multi-device environment

[0009] In the smart factory scenario, each device is defined as a 3-DOF robotic arm and treated as an intelligent agent. A multi-agent reinforcement learning algorithm is used to train the multi-device system. The smart factory environment contains three devices, the environment has a round length of 300 steps, and the environment area is 10×5. The environmental task objective is to enable the three devices to collaboratively grasp a circular workpiece with a radius of 1.5. The devices need to learn the correct strategy to grasp the workpiece as quickly as possible. The observation space of each device is 19 dimensions, including its own end position (3 dimensions), three-axis speed (3 dimensions), the end positions of the other two devices (6 dimensions), the workpiece position (3 dimensions), the position of the workpiece's graspable point (3 dimensions), and the end grasping state (1 dimension). The action space is 11 dimensions, which controls the three-axis movement (9 dimensions) and the end grasping and releasing operations (2 dimensions). At the end of the round, the number of devices that successfully grasped the workpiece is obtained based on the distance between the end of the device and the workpiece's graspable point and the end grasping state. If num devices successfully grasp the workpiece, each device receives an external reward from the environment. If all devices complete the grasping task ahead of time (t≤300), the round ends ahead of time, and each device receives an external reward from the environment. The external rewards obtained are used to establish the total reward function. The external rewards obtained by the device during the rest of the time are Implement sparse reward settings;

[0010] (2) Designing a loss function based on mutual information

[0011] Define any device in the environment as A i , the three devices share the training batch during training, and device A i The observation corresponding to index τ in the batch is defined as o i,τ ; For device A i , design a neural network module Contains a 3-layer fully connected layer and a linear layer, Θ i For modules The learnable parameters, parameter Θ i Included modules The weights and biases of the multi-agent reinforcement learning algorithm; during the training process of the multi-agent reinforcement learning algorithm, the algorithm accesses each index in the training batch of each device; at index τ, the module The fully connected layer part input is the observation o i,τ ∈R 19 , the output is feature h i,τ ∈R 64 , the input of the linear layer is feature h i,τ ∈R 64 , the output is feature x i,τ ∈R 32 ; Similarly, at index τ+1, module The fully connected layer part input is the observation o i,τ+1 , the output is feature h i,τ+1 , the input of the linear layer is feature h i,τ+1 , the output is feature x i，τ+1 ; Set the batch size to 3200 and use the loss function

[0012]

[0013] To maximize the feature x i，τ+1 and h i,τ The mutual information between i (·) is the score function, and its calculation formula is

[0014]

[0015] in, is a learnable weight matrix; using the loss function

[0016]

[0017] To maximize the feature h i,τ+1 and h i,τ The mutual information between is the scoring function, and its calculation formula is

[0018]

[0019] in, is a learnable weight matrix; using the loss function

[0020]

[0021] Training Module The learnable parameter Θ i , and the score function φ i (·)and The learnable weight matrix and The learning rate is 3×10 -4 ;

[0022] (3) Obtaining distillation error

[0023] Define device A i The observation at time t is o i，t , input the observation into the module Get feature x i,t ; In addition, build a neural network module Contains a 5-layer fully connected layer and a linear layer, Θ′ i For modules The learnable parameters, the learnable parameters Θ′ i Included modules The weights and biases of i，t Input Module Get the feature x′ i，t ∈R 32 ;use

[0024] e i,t =||x′ i,t -x i,t ||2 (6)

[0025] Calculate the time t for device A i The distillation error e i，t ; During the training process of the multi-agent reinforcement learning algorithm, for index τ, the observation o i,τ Input Module Get the feature x′ iτ , using the loss function

[0026]

[0027] To train the module The learnable parameter Θ′ i , the learning rate is 3×10 -4 ;

[0028] (4) Constructing intrinsic rewards driven by mutual information

[0029] Define except device A i Any device other than A j , according to formula (6), we can get device A j The distillation error e j,t ;use

[0030]

[0031] Get device A j The cumulative distillation error of 1 round; except for device A i The remaining two devices are considered as a whole and this whole is defined as device use

[0032]

[0033] computing devices Cumulative distillation error For any device A i , building a neural network module Contains a 3-layer fully connected layer and a linear layer, where ψ i For modules The learnable parameters of Weights and biases of modules The input of device A at time t i Observation o i,t , and equipment Cumulative distillation error The output is the predicted action The probability distribution of Construct device A using the mutual information between actions and cumulative distillation errors i Intrinsic reward at time t:

[0034]

[0035] in, For device A i Actor Network The output of θ i Actor Network The learnable parameters of the actor network The weights and biases of For real action i,t The conditional probability distribution of the real action a i,t From the probability distribution Sampling obtained by using loss function

[0036]

[0037] To train the module The learnable parameter ψ i , the learning rate is 3×10 -4 ;in and They are probability distributions and The probability value of each action space dimension d at index τ in the training batch;

[0038] (5) Establishing the total reward function

[0039] Using device A i Intrinsic rewards extrinsic rewards equipment Cumulative distillation error And the sum of the distillation errors of the three devices, construct device A i Total reward at time t:

[0040]

[0041] (6) Using multi-agent proximal strategy optimization algorithm to train multi-device systems

[0042] Set the maximum number of training steps for the multi-agent proximal policy optimization algorithm to 1×10 7 , the number of repeated updates of the actor-critic network is 10, and the number of parallel environments is 128;

[0043] Set the architecture of the actor network: 1 layer of multi-layer perceptron, whose input layer dimension is 19, hidden layer dimension is 64, and output layer dimension is 64; 1 layer of gated recurrent unit, whose input layer, hidden layer and output layer dimension are all 64; 1 layer of fully connected layer, whose input layer dimension is 64, output layer dimension is 11, and the activation function is the ReLU function;

[0044] The critic network architecture is set as follows: 1 multi-layer perceptron with an input layer dimension of 57, a hidden layer dimension of 64, and an output layer dimension of 64; 1 gated recurrent unit with an input layer dimension of 64, a hidden layer dimension of 64, and an output layer dimension of 64; 1 fully connected layer with an input layer dimension of 64 and an output layer dimension of 1, and the activation function is the ReLU function;

[0045] use

[0046]

[0047] For device A i Actor Network The learnable parameters θ i Training is performed with a learning rate of 7×10 -4 ;in is the ratio of the actor network output before and after the update, min(·) means taking the smaller value of the two, and clip(·) means changing δ i,τ The value of is limited to [1-0.2,1+0.2]; A i,τ is the advantage function, and its calculation formula is:

[0048]

[0049] Among them, Φ i For device A i The learnable parameters of the critic network include the weights and biases of the critic network. The input of the critic network is the concatenated vector of the three device observations, and the output is a scalar value. and The training batch indexes are τ+t″ and τ+1+t respectively ″ Time device A i The output of the critic network, r i,τ+t″ is the total reward when the training batch index is τ+t″; is the policy entropy, and its calculation formula is:

[0050]

[0051] use

[0052]

[0053] For device A i The learnable parameters Φ of the critic network i Training is performed with a learning rate of 7×10 -4 ; where max(·) is the larger of the two numbers, is the output of the critic network when the batch index is τ, is the output of the critic network before updating, is the sum of the discounted total rewards from time t to the end of the round;

[0054] When the number of training steps of the algorithm reaches 1×10 7 When , the training ends;

[0055] (7) Realize multi-device collaborative control

[0056] After completing the training of all neural network modules in the smart factory multi-device collaborative control method based on mutual information-driven sparse rewards, the round reward and control strategy model are saved, and the model is applied to the sparse reward smart factory environment to realize the collaborative control of multi-device systems.

[0057] This method first constructs a sparse reward smart factory multi-device environment. The multi-device system obtains sample data, including observations, rewards, and actions, by interacting with the environment. Secondly, this data is transmitted to the server and used to design a mutual information-based loss function, distillation error, mutual information-driven intrinsic reward, and total reward. Then, the multi-agent proximal policy optimization algorithm is used to train the actor and critic networks of each device. Finally, the trained model is used in the sparse reward smart factory environment to complete the collaborative control task of the multi-device system. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 Architecture diagram of the multi-device collaborative control method for smart factories based on mutual information-driven sparse rewards

[0059] Figure 2 Schematic diagram of environmental tasks DETAILED DESCRIPTION

[0060] A method for collaborative control of multiple devices in a smart factory based on mutual information-driven sparse rewards is characterized by constructing a sparse reward smart factory multi-device environment, designing a loss function based on mutual information, obtaining a distillation error, constructing a mutual information-driven intrinsic reward, establishing a total reward function, and using a multi-agent proximal policy optimization algorithm to train the multi-device system to achieve multi-device collaborative control. The method includes the following steps:

[0061] (1) Building a sparse reward smart factory multi-device environment

[0062] In the smart factory scenario, each device is defined as a 3-DOF robotic arm and treated as an intelligent agent. A multi-agent reinforcement learning algorithm is used to train the multi-device system. The smart factory environment contains three devices, the environment has a round length of 300 steps, and the environment area is 10×5. The environmental task objective is to enable the three devices to collaboratively grasp a circular workpiece with a radius of 1.5. The devices need to learn the correct strategy to grasp the workpiece as quickly as possible. The observation space of each device is 19 dimensions, including its own end position (3 dimensions), three-axis speed (3 dimensions), the end positions of the other two devices (6 dimensions), the workpiece position (3 dimensions), the position of the workpiece's graspable point (3 dimensions), and the end grasping state (1 dimension). The action space is 11 dimensions, which controls the three-axis movement (9 dimensions) and the end grasping and releasing operations (2 dimensions). At the end of the round, the number of devices that successfully grasped the workpiece is obtained based on the distance between the end of the device and the workpiece's graspable point and the end grasping state. If num devices successfully grasp the workpiece, each device receives an external reward from the environment. If all devices complete the grasping task ahead of time (t≤300), the round ends ahead of time, and each device receives an external reward from the environment. The external rewards obtained are used to establish the total reward function. The external rewards obtained by the device during the rest of the time are Implement sparse reward settings;

[0063] (2) Designing a loss function based on mutual information

[0064] Define any device in the environment as A i , the three devices share the training batch during training, and device A i The observation corresponding to index τ in the batch is defined as o i,τ ; For device A i , design a neural network module Contains a 3-layer fully connected layer and a linear layer, Θ i For modules The learnable parameters, parameter Θ i Included modules The weights and biases of the multi-agent reinforcement learning algorithm; during the training process of the multi-agent reinforcement learning algorithm, the algorithm accesses each index in the training batch of each device; at index τ, the module The fully connected layer part input is the observation o i,τ ∈R 19 , the output is feature h i,τ ∈R 64 , the input of the linear layer is feature h i,τ ∈R 64 , the output is feature x i,τ ∈R 32 ; Similarly, at index τ+1, module The fully connected layer part input is the observation o i,τ+1 , the output is feature h i,τ+1 , the input of the linear layer is feature h i,τ+1 , the output is feature x i，τ+1 ; Set the batch size to 3200 and use the loss function

[0065]

[0066] To maximize the feature x i,τ+1 and h i,τ The mutual information between i (·) is the score function, and its calculation formula is

[0067]

[0068] in, is a learnable weight matrix; using the loss function

[0069]

[0070] To maximize the feature h i,τ+1 and h i,τ The mutual information between is the scoring function, and its calculation formula is

[0071]

[0072] in, is a learnable weight matrix; using the loss function

[0073]

[0074] Training Module The learnable parameter Θ i , and the score function φ i (·)and The learnable weight matrix and The learning rate is 3×10 -4 ;

[0075] (3) Obtaining distillation error

[0076] Define device A i The observation at time t is o i,t , input the observation into the module Get feature x i,t ; In addition, build a neural network module Contains a 5-layer fully connected layer and a linear layer, Θ′ i For modules The learnable parameters, the learnable parameters Θ′ i Included modules The weights and biases of i,t Input Module Get the feature x′ i,t ∈R 32 ;use

[0077] e i,t =||x′ i,t -x i,t ||2 (6)

[0078] Calculate the time t for device A i The distillation error e i,t ; During the training process of the multi-agent reinforcement learning algorithm, for index τ, the observation o i,τ Input Module Get the feature x′ i,τ , using the loss function

[0079]

[0080] To train the module The learnable parameter Θ′ i , the learning rate is 3×10 -4 ;

[0081] (4) Constructing intrinsic rewards driven by mutual information

[0082] Define except device A i Any device other than A j , according to formula (6), we can get device A j The distillation error e j,t ;use

[0083]

[0084] Get device Aj The cumulative distillation error of 1 round; except for device A i The remaining two devices are considered as a whole and this whole is defined as device use

[0085]

[0086] computing devices Cumulative distillation error For any device A i , building a neural network module Contains a 3-layer fully connected layer and a linear layer, where ψ i For modules The learnable parameters of Weights and biases of modules The input of device A at time t i Observation o i，t , and equipment Cumulative distillation error The output is the predicted action The probability distribution of Construct device A using the mutual information between actions and cumulative distillation errors i Intrinsic reward at time t:

[0087]

[0088] in, For device A i Actor Network The output of θ i Actor Network The learnable parameters of the actor network The weights and biases of For real action i,t The conditional probability distribution of the real action a i,t From the probability distribution Sampling obtained by using loss function

[0089]

[0090] To train the module The learnable parameter ψ i , the learning rate is 3×10 -4 ;in and They are probability distributions and The probability value of each action space dimension d at index τ in the training batch;

[0091] (5) Establishing the total reward function

[0092] Using device A i Intrinsic rewards extrinsic rewards equipment Cumulative distillation error And the sum of the distillation errors of the three devices, construct device A i Total reward at time t:

[0093]

[0094] (6) Using multi-agent proximal strategy optimization algorithm to train multi-device systems

[0095] Set the maximum number of training steps for the multi-agent proximal policy optimization algorithm to 1×10 7 , the number of repeated updates of the actor-critic network is 10, and the number of parallel environments is 128;

[0096] Set the architecture of the actor network: 1 layer of multi-layer perceptron, whose input layer dimension is 19, hidden layer dimension is 64, and output layer dimension is 64; 1 layer of gated recurrent unit, whose input layer, hidden layer and output layer dimension are all 64; 1 layer of fully connected layer, whose input layer dimension is 64, output layer dimension is 11, and the activation function is the ReLU function;

[0097] The critic network architecture is set as follows: 1 multi-layer perceptron with an input layer dimension of 57, a hidden layer dimension of 64, and an output layer dimension of 64; 1 gated recurrent unit with an input layer dimension of 64, a hidden layer dimension of 64, and an output layer dimension of 64; 1 fully connected layer with an input layer dimension of 64 and an output layer dimension of 1, and the activation function is the ReLU function;

[0098] use

[0099]

[0100] For device A i Actor Network The learnable parameters θ i Training is performed with a learning rate of 7×10 -4 ;in is the ratio of the actor network output before and after the update, min(·) means taking the smaller value of the two, and clip(·) means changing δ i,τ The value of is limited to [1-0.2,1+0.2]; A i,τ is the advantage function, and its calculation formula is:

[0101]

[0102] Among them, Φ i For device A i The learnable parameters of the critic network include the weights and biases of the critic network. The input of the critic network is the concatenated vector of the three device observations, and the output is a scalar value. and Device A when the training batch index is τ+t″ and τ+1+t″ respectively i The output of the critic network, r i,τ+t″ is the total reward when the training batch index is τ+t″; is the policy entropy, and its calculation formula is:

[0103]

[0104] use

[0105]

[0106] For device A i The learnable parameters Φ of the critic network i Training is performed with a learning rate of 7×10 -4 ; where max(·) is the larger of the two numbers, is the output of the critic network when the batch index is τ, is the output of the critic network before updating, is the sum of the discounted total rewards from time t to the end of the round;

[0107] When the number of training steps of the algorithm reaches 1×10 7 When , the training ends;

[0108] (7) Realize multi-device collaborative control

[0109] After completing the training of all neural network modules in the smart factory multi-device collaborative control method based on mutual information-driven sparse rewards, the round reward and control strategy model are saved, and the model is applied to the sparse reward smart factory environment to realize the collaborative control of multi-device systems.

[0110] This method first constructs a sparse reward smart factory multi-device environment. The multi-device system obtains sample data, including observations, rewards, and actions, by interacting with the environment. Secondly, this data is transmitted to the server and used to design a mutual information-based loss function, distillation error, mutual information-driven intrinsic reward, and total reward. Then, the multi-agent proximal policy optimization algorithm is used to train the actor and critic networks of each device. Finally, the trained model is used in the sparse reward smart factory environment to complete the collaborative control task of the multi-device system.

Claims

1. A smart factory multi-device collaborative control method based on mutual information driven sparse rewards, characterized by: The following steps are involved: (1) Building a sparse reward smart factory multi-device environment In the smart factory scenario, each device is defined as a 3-DOF robotic arm and treated as an intelligent agent. A multi-agent reinforcement learning algorithm is used to train the multi-device system. The smart factory environment contains three devices, the environment has a round length of 300 steps, and the environment area is 10×5. The environmental task objective is to enable the three devices to collaboratively grasp a circular workpiece with a radius of 1.

5. The devices need to learn the correct strategy to grasp the workpiece as quickly as possible. The observation space of each device is 19 dimensions, including its own end position (3 dimensions), three-axis speed (3 dimensions), the end positions of the other two devices (6 dimensions), the workpiece position (3 dimensions), the position of the workpiece's graspable point (3 dimensions), and the end grasping state (1 dimension). The action space is 11 dimensions, which controls the three-axis movement (9 dimensions) and the end grasping and releasing operations (2 dimensions). At the end of the round, the number of devices that successfully grasped the workpiece is obtained based on the distance between the end of the device and the workpiece's graspable point and the end grasping state. If num devices successfully grasp the workpiece, each device receives an external reward from the environment. If all devices complete the grasping task ahead of time t, t≤300; the round ends ahead of time, and each device receives an external reward from the environment The external rewards obtained are used to establish the total reward function. The external rewards obtained by the device during the rest of the time are Implement sparse reward settings; (2) Designing a loss function based on mutual information Define any device in the environment as A i , the three devices share the training batch during training, and device A i The observation corresponding to index τ in the batch is defined as o i,τ ; For device A i , design a neural network module Contains a 3-layer fully connected layer and a linear layer, Θ i For modules The learnable parameters, parameter Θ i Included modules The weights and biases of the multi-agent reinforcement learning algorithm; during the training process of the multi-agent reinforcement learning algorithm, the algorithm accesses each index in the training batch of each device; at index τ, the module The fully connected layer part input is the observation o i,τ ∈R 19 , the output is feature h i,τ ∈R 64 , the input of the linear layer is feature h i,τ ∈R 64 , the output is feature x i,τ ∈R 32 ; Similarly, at index τ+1, module The fully connected layer part input is the observation o i,τ+1 , the output is feature h i,τ+1 , the input of the linear layer is feature h i,τ+1 , the output is feature x i,τ+1 ; Set the batch size to 3200 and use the loss function To maximize the feature x i,τ+1 and h i,τ The mutual information between i (·) is the score function, and its calculation formula is in, is a learnable weight matrix; using the loss function To maximize the feature h i,τ+1 and h i,τ The mutual information between is the scoring function, and its calculation formula is in, is a learnable weight matrix; using the loss function Training Module The learnable parameter Θ i , and the score function φ i (·)and The learnable weight matrix and The learning rate is 3×10 -4 ; (3) Obtaining distillation error Define device A i The observation at time t is o i,t , input the observation into the module Get feature x i,t ; In addition, build a neural network module Contains a 5-layer fully connected layer and a linear layer, Θ′ i For modules The learnable parameters, the learnable parameters Θ′ i Included modules The weights and biases of i,t Input Module Get the feature x′ i,t ∈R 32 ;use e i,t =||x′ i,t -x i,t ||2 (6) Calculate the time t for device A i The distillation error e i,t ; During the training process of the multi-agent reinforcement learning algorithm, for index τ, the observation o i,τ Input Module Get the feature x′ i,τ , using the loss function To train the module The learnable parameter Θ′ i , the learning rate is 3×10 -4 ; (4) Constructing intrinsic rewards driven by mutual information Define except device A i Any device other than A j , according to formula (6), we can get device A j The distillation error e j,t ;use Get device A j The cumulative distillation error of 1 round; except for device A i The remaining two devices are considered as a whole and this whole is defined as device use computing devices Cumulative distillation error For any device A i , building a neural network module Contains a 3-layer fully connected layer and a linear layer, where ψ i For modules The learnable parameters of Weights and biases of modules The input of device A at time t i Observation o i,t , and equipment Cumulative distillation error The output is the predicted action The probability distribution of Construct device A using the mutual information between actions and cumulative distillation errors i Intrinsic reward at time t: in, For device A i Actor Network The output of θ i Actor Network The learnable parameters of the actor network The weights and biases of For real action i,t The conditional probability distribution of the real action a i,t From the probability distribution Sampling obtained by using loss function To train the module The learnable parameter ψ i , the learning rate is 3×10 -4 ;in and They are probability distributions and The probability value of each action space dimension d at index τ in the training batch; (5) Establishing the total reward function Using device A i Intrinsic rewards extrinsic rewards equipment Cumulative distillation error And the sum of the distillation errors of the three devices, construct device A i Total reward at time t: (6) Using multi-agent proximal strategy optimization algorithm to train multi-device systems (7) Realize multi-device collaborative control After completing the training of all neural network modules in the smart factory multi-device collaborative control method based on mutual information-driven sparse rewards, the round reward and control strategy model are saved, and the model is applied to the sparse reward smart factory environment to realize the collaborative control of multi-device systems.

2. The method according to claim 1, characterized in that Step (6) is as follows: Set the maximum number of training steps for the multi-agent proximal policy optimization algorithm to 1×10 7 , the number of repeated updates of the actor-critic network is 10, and the number of parallel environments is 128; Set the architecture of the actor network: 1 layer of multi-layer perceptron, whose input layer dimension is 19, hidden layer dimension is 64, and output layer dimension is 64; 1 layer of gated recurrent unit, whose input layer, hidden layer and output layer dimension are all 64; 1 layer of fully connected layer, whose input layer dimension is 64, output layer dimension is 11, and the activation function is the ReLU function; The critic network architecture is set as follows: 1 multi-layer perceptron with an input layer dimension of 57, a hidden layer dimension of 64, and an output layer dimension of 64; 1 gated recurrent unit with an input layer dimension of 64, a hidden layer dimension of 64, and an output layer dimension of 64; 1 fully connected layer with an input layer dimension of 64 and an output layer dimension of 1, and the activation function is the ReLU function; use For device A i Actor Network The learnable parameters θ i Training is performed with a learning rate of 7×10 -4 ;in is the ratio of the actor network output before and after the update, min(·) means taking the smaller value of the two, and clip(·) means changing δ i,τ The value of is limited to [1-0.2,1+0.2]; A i,τ is the advantage function, and its calculation formula is: Among them, Φ i For device A i The learnable parameters of the critic network include the weights and biases of the critic network. The input of the critic network is the concatenated vector of the three device observations, and the output is a scalar value. and Device A when the training batch index is τ+t″ and τ+1+t″ respectively i The output of the critic network, r i,τ+t″ is the total reward when the training batch index is τ+t″; is the policy entropy, and its calculation formula is: use For device A i The learnable parameters Φ of the critic network i Training is performed with a learning rate of 7×10 -4 ; where max(·) is the larger of the two numbers, is the output of the critic network when the batch index is τ, is the output of the critic network before updating, is the sum of the discounted total rewards from time t to the end of the round; When the number of training steps of the algorithm reaches 1×10 7 When , the training ends; Set the architecture of the actor network: 1 layer of multi-layer perceptron, whose input layer dimension is 19, hidden layer dimension is 64, and output layer dimension is 64; 1 layer of gated recurrent unit, whose input layer, hidden layer and output layer dimension are all 64; 1 layer of fully connected layer, whose input layer dimension is 64, output layer dimension is 11, and the activation function is the ReLU function; The critic network architecture is set as follows: 1 multi-layer perceptron with an input layer dimension of 57, a hidden layer dimension of 64, and an output layer dimension of 64; 1 gated recurrent unit with an input layer dimension of 64, a hidden layer dimension of 64, and an output layer dimension of 64; 1 fully connected layer with an input layer dimension of 64 and an output layer dimension of 1, and the activation function is the ReLU function; use For device A i Actor Network The learnable parameters θ i Training is performed with a learning rate of 7×10 -4 ;in is the ratio of the actor network output before and after the update, min(·) means taking the smaller value of the two, and clip(·) means changing δ i,τ The value of is limited to [1-0.2,1+0.2]; A i,τ is the advantage function, and its calculation formula is: Among them, Φ i For device A i The learnable parameters of the critic network include the weights and biases of the critic network. The input of the critic network is the concatenated vector of the three device observations, and the output is a scalar value. and Device A when the training batch index is τ+t″ and τ+1+t″ respectively i The output of the critic network, r i,τ+t″ is the total reward when the training batch index is τ+t″; is the policy entropy, and its calculation formula is: use For device A i The learnable parameters Φ of the critic network i Training is performed with a learning rate of 7×10 -4 ; where max(·) is the larger of the two numbers, is the output of the critic network when the batch index is τ, is the output of the critic network before updating, is the sum of the discounted total rewards from time t to the end of the round; When the number of training steps of the algorithm reaches 1×10 7 When , the training ends; Set the architecture of the actor network: 1 layer of multi-layer perceptron, whose input layer dimension is 19, hidden layer dimension is 64, and output layer dimension is 64; 1 layer of gated recurrent unit, whose input layer, hidden layer and output layer dimension are all 64; 1 layer of fully connected layer, whose input layer dimension is 64, output layer dimension is 11, and the activation function is the ReLU function; The critic network architecture is set as follows: 1 multi-layer perceptron with an input layer dimension of 57, a hidden layer dimension of 64, and an output layer dimension of 64; 1 gated recurrent unit with an input layer dimension of 64, a hidden layer dimension of 64, and an output layer dimension of 64; 1 fully connected layer with an input layer dimension of 64 and an output layer dimension of 1, and the activation function is the ReLU function; use For device A i Actor Network The learnable parameters θ i Training is performed with a learning rate of 7×10 -4 ;in is the ratio of the actor network output before and after the update, min(·) means taking the smaller value of the two, and clip(·) means changing δ i,τ The value of is limited to [1-0.2,1+0.2]; A i,τ is the advantage function, and its calculation formula is: Among them, Φ i For device A i The learnable parameters of the critic network include the weights and biases of the critic network. The input of the critic network is the concatenated vector of the three device observations, and the output is a scalar value. and Device A when the training batch index is τ+t″ and τ+1+t″ respectively i The output of the critic network, r i,τ+t″ is the total reward when the training batch index is τ+t″; is the policy entropy, and its calculation formula is: use For device A i The learnable parameters Φ of the critic network i Training is performed with a learning rate of 7×10 -4 ; where max(·) is the larger of the two numbers, is the output of the critic network when the batch index is τ, is the output of the critic network before updating, is the sum of the discounted total rewards from time t to the end of the round; When the number of training steps of the algorithm reaches 1×10 7 The training ends when .

Images