Multi-device cooperative control method based on exponential graph information communication

Abstract

The invention provides a multi-device cooperative control method based on index graph information communication. According to the method, a dynamic information interaction rule between devices is defined by constructing an index graph communication topology, and an information feature coding module is designed to realize efficient extraction and processing of interaction information; meanwhile, a mutual information loss function and independent deep Q network algorithm loss are combined to construct a total loss function, and an equipment decision module is trained to generate an optimal control strategy. According to the method, the problems that in traditional multi-device cooperative control, communication topology expandability is poor, information transmission is low in efficiency, and device decision-making lacks effective interaction guidance are solved, the cooperative performance and strategy convergence speed of a multi-device system are remarkably improved, and the method is suitable for large-scale popularization and application. And a key technical support is provided for application of a multi-equipment cooperative control technology in the fields of intelligent manufacturing, automatic assembly and the like.

Description

Technical Field

[0001] This invention proposes a multi-device collaborative control method based on exponential graph information communication. This method defines information interaction rules between devices by constructing an exponential graph communication topology, designs an information feature encoding module to generate interaction information between devices, optimizes information features using a mutual information loss function, constructs a total loss function by combining an independent deep Q-network algorithm loss, and trains device decision modules to generate control strategies. Ultimately, it enables efficient collaborative control of multiple devices in complex dynamic environments, improving upon the problems of low communication topology information transmission efficiency and lack of effective interactive guidance for device decision-making in traditional multi-device collaborative control. This helps improve the collaborative control performance of multi-device systems and provides technical support for the application of multi-device collaborative control technology in fields such as intelligent manufacturing and automated assembly. Background Technology

[0002] The rapid development of artificial intelligence and automation control technologies has made multi-device collaborative control a crucial technology in fields such as intelligent manufacturing and intelligent assembly. Its applications encompass a variety of complex scenarios, including multi-device collaborative assembly, workshop scheduling, and multi-robot collaborative operations. In these scenarios, multiple devices need to coordinate their actions through real-time information exchange to complete complex tasks that a single device cannot accomplish independently. Therefore, a reasonable communication topology design and effective information encoding methods are essential factors in ensuring the performance of multi-device collaborative control.

[0003] Multi-device cooperative control methods based on multi-agent reinforcement learning achieve autonomous cooperative decision-making by training device decision-making modules, thus improving the system's adaptability to some extent. However, these methods still have some drawbacks. Firstly, existing communication-based multi-agent reinforcement learning methods typically employ a fixed, fully connected communication topology. In this topology, each device needs to communicate with all other devices at every time step, resulting in poor scalability. Furthermore, fully connected communication topologies may generate a large amount of redundant information, interfering with the policy learning process and hindering the improvement of cooperative control performance. Secondly, existing methods use relatively simple information encoding methods, often directly sending raw data without an effective information feature encoding mechanism. This makes it difficult to promote cooperative performance between devices, which in turn affects the training effect of the decision-making modules, leading to slow policy convergence and unstable cooperative performance.

[0004] To address the shortcomings of existing technologies, a multi-device collaborative control method capable of efficient information interaction and collaborative decision-making is urgently needed. This invention proposes a multi-device collaborative control method based on exponential graph information communication, aiming to improve the poor scalability of communication topologies and low information encoding efficiency in traditional methods. First, a ring-shaped communication topology based on an exponential graph is designed, dynamically adjusting device information transmission rules by combining communication cycles and exponential characteristics, which helps improve information transmission efficiency. Simultaneously, each device communicates with only one device at a single time step, rather than global communication, achieving better scalability. Second, an information feature encoding module integrating device decision-making hidden states and interaction information is constructed to achieve efficient feature extraction. Subsequently, a total loss function is constructed by combining mutual information loss functions and independent deep Q-network algorithm loss functions, balancing individual decision optimization and inter-device collaboration, which helps accelerate policy convergence. Finally, control strategies are generated by training device decision modules to achieve multi-device collaborative control, providing solutions and technical support for the application of multi-device collaborative control technology in complex scenarios. Summary of the Invention

[0005] 1. A multi-device collaborative control method based on exponential graph information communication, characterized by constructing a multi-device environment, designing an exponential graph communication topology, establishing a device decision module, training the device decision module, and realizing multi-device collaborative control, comprising the following steps:

[0006] (1) Building a multi-device environment

[0007] Construct a multi-device environment with 8 devices, and set the maximum number of time steps per round to 1. The task objective is to achieve collaborative assembly of one workpiece by eight devices. Each device has a 10-dimensional observation space, specifically including its own end-effector position (3D), end-effector velocity in 3D space (3D), workpiece assembly point position (3D), and end-effector assembly state (1D). Each device has a 5-dimensional motion space, corresponding to end-effector movement along coordinate axes (3D) and end-effector stop / start assembly operations (2D). Each device also corresponds to a dedicated assembly point on the workpiece. At each time step… Using the formula

[0008] (1)

[0009] Calculate the first Rewards for individual devices ,in For each device, For the 3-dimensional space The Euclidean distance between the end of the device and the assembly point. For the first The end state of each device is calculated using the following formula:

[0010] (2)

[0011] When all 8 devices have completed their respective assembly tasks at their respective assembleable points, each device receives an additional reward of 20, and the round ends, completing the construction of the multi-device environment.

[0012] (2) Design an exponential graph communication topology

[0013] An exponential graph is used to construct the communication topology between devices. Eight devices are arranged in a ring structure to form a directed graph communication architecture. The communication cycle is set to 3. The information transmission rules of the devices at different time steps are defined using the characteristics of the exponential graph, as follows:

[0014] If the current time step When it is an integer multiple of 3, that is Based on index 2 0 =1 sets the communication direction: In a circular directed graph structure, each device sends information to the device whose number differs from the target device number by 1; specifically, the 0th device sends information to the 1st device, the 1st device sends information to the 2nd device, and so on, the 6th device sends information to the 7th device, and the 7th device sends information to the 0th device.

[0015] If the current time step When it is a multiple of 3 plus 1, that is The communication direction is set based on the exponent 2¹=2: each device sends information to the device whose number differs from the target device number by 2; specifically, the 0th device sends information to the 2nd device, the 1st device sends information to the 3rd device, and so on, with the 6th device sending information to the 0th device and the 7th device sending information to the 1st device.

[0016] If the current time step When it is a multiple of 3 plus 2, that is The communication direction is set based on the exponent 2²=4: each device sends information to the device whose number differs from the target device number by 4; specifically, the 0th device sends information to the 4th device, the 1st device sends information to the 5th device, and so on, the 6th device sends information to the 2nd device, and the 7th device sends information to the 3rd device.

[0017] Using the above rules, complete the construction of the exponential graph communication topology;

[0018] (3) Establish equipment decision-making module

[0019] The equipment decision-making module consists of two parts: an individual Q-network module and an information feature encoding module. The specific construction process is as follows:

[0020] Construct the first Individual Q network module of each device: will the first Each device at time step Observations With time step action Feature concatenation is performed to obtain the feature vector. The concatenated feature vector is then input into a single linear layer. Feature extraction is completed in the linear layer. The input dimension is 15 and the output dimension is 64; the linear layer The feature extraction results are input into a single-layer gated recurrent unit. Output time steps The hidden state of the gated loop unit Among them, the gated loop unit The input dimension is 64 and the output dimension is 64; the hidden state will be... With time step Next Information to be sent by each device Feature concatenation is performed to obtain the feature vector. The feature vector is then input into a multilayer perceptron module containing two fully connected layers. Output the first Individual action value of a device Among them, the multilayer perceptron module The input dimension is 128, the hidden layer dimension is 64, and the output dimension is 5; based on maximizing the value of individual actions. Following the principle of greedy strategy, the first step is to select the... Each device at time step Actions to be performed This completes the construction of the individual Q-network module;

[0021] Construct the first The information feature encoding module of each device: determines the time step according to the information transmission rules set in step (2). Down to the The device number that sends information from the device (denoted as the No. 1) (a device), and the information sent by that device. ; receive the information With the Hidden state of individual Q-network devices By concatenating the vectors, we obtain the feature vectors. The feature vector is input into a single linear layer. Feature extraction is completed, including linear layers. The input dimension is 128, and the output dimension is 64; the extracted features are then input into a single-layer gated recurrent unit. In the middle, output time step Next Information to be sent by each device Among them, the gated loop unit The input dimension is 64, and the output dimension is 64; the construction of the information feature encoding module is completed;

[0022] (4) Training equipment decision module

[0023] Using formula

[0024] (3)

[0025] Get time steps The numbering is different from Any device number ,in For any integer in the interval [0, 297]; from the playback buffer Collection of the first The device and the first Each device at time step The information sent below ;

[0026] From the playback buffer Collection and Time Step Time steps differing by more than one communication cycle Next Information of each device ,in The range of values ​​is ,Will and Composition information pair ;

[0027] Each piece of information in all the above information pairs is input into a 2-layer multilayer perceptron module. Feature extraction is completed in the middle, including the multilayer perceptron module. The input dimension is 128, the hidden layer dimension is 64, and the output dimension is 80; the corresponding feature vectors are obtained, denoted as follows: and ;

[0028] Establish mutual information loss function

[0029] (4)

[0030] Train the information feature encoding module for all devices, using the pruning function. The value of the dot product of the eigenvectors is limited to ±5.

[0031] Loss function using independent deep Q-network algorithm

[0032] (5)

[0033] Train individual Q-network modules for all devices, where This indicates that the optimal value is taken for all possible joint actions in the next time step state; For the target Q-network, its network parameter update rule is: when the interval between two consecutive parameter updates reaches 200 rounds, the current individual Q-network is updated. The parameters are synchronized to the target Q network. ;

[0034] Using the total loss function

[0035] (6)

[0036] Training device decision module; setting the learning rate and training time steps to 5×10. -4 and 2×10 6 When the time step reaches 2×10 6 At this point, training ends, and the device decision network model trained at the last time step is saved;

[0037] (5) Achieve multi-device collaborative control

[0038] After training the device decision module in the multi-device collaborative control method based on exponential graph information communication, save the round reward and device decision network model to realize the collaborative control of the multi-device system. Attached Figure Description

[0039] Figure 1 Equipment decision module architecture diagram Detailed Implementation

[0040] 1. A multi-device collaborative control method based on exponential graph information communication, characterized by constructing a multi-device environment, designing an exponential graph communication topology, establishing a device decision module, training the device decision module, and realizing multi-device collaborative control, comprising the following steps:

[0041] (1) Building a multi-device environment

[0042] Construct a multi-device environment with 8 devices, and set the maximum number of time steps per round to 1. The task objective is to achieve collaborative assembly of one workpiece by eight devices. Each device has a 10-dimensional observation space, specifically including its own end-effector position (3D), end-effector velocity in 3D space (3D), workpiece assembly point position (3D), and end-effector assembly state (1D). Each device has a 5-dimensional motion space, corresponding to end-effector movement along coordinate axes (3D) and end-effector stop / start assembly operations (2D). Each device also corresponds to a dedicated assembly point on the workpiece. At each time step… Using the formula

[0043] (1)

[0044] Calculate the first Rewards for individual devices ,in For each device, For the 3-dimensional space The Euclidean distance between the end of the device and the assembly point. For the first The end state of each device is calculated using the following formula:

[0045] (2)

[0046] When all 8 devices have completed their respective assembly tasks at their respective assembleable points, each device receives an additional reward of 20, and the round ends, completing the construction of the multi-device environment.

[0047] (2) Design an exponential graph communication topology

[0048] An exponential graph is used to construct the communication topology between devices. Eight devices are arranged in a ring structure to form a directed graph communication architecture. The communication cycle is set to 3. The information transmission rules of the devices at different time steps are defined using the characteristics of the exponential graph, as follows:

[0049] If the current time step When it is an integer multiple of 3, that is Based on index 2 0 =1 sets the communication direction: In a circular directed graph structure, each device sends information to the device whose number differs from the target device number by 1; specifically, the 0th device sends information to the 1st device, the 1st device sends information to the 2nd device, and so on, the 6th device sends information to the 7th device, and the 7th device sends information to the 0th device.

[0050] If the current time step When it is a multiple of 3 plus 1, that is The communication direction is set based on the exponent 2¹=2: each device sends information to the device whose number differs from the target device number by 2; specifically, the 0th device sends information to the 2nd device, the 1st device sends information to the 3rd device, and so on, with the 6th device sending information to the 0th device and the 7th device sending information to the 1st device.

[0051] If the current time step When it is a multiple of 3 plus 2, that is The communication direction is set based on the exponent 2²=4: each device sends information to the device whose number differs from the target device number by 4; specifically, the 0th device sends information to the 4th device, the 1st device sends information to the 5th device, and so on, the 6th device sends information to the 2nd device, and the 7th device sends information to the 3rd device.

[0052] Using the above rules, complete the construction of the exponential graph communication topology;

[0053] (3) Establish equipment decision-making module

[0054] The equipment decision-making module consists of two parts: an individual Q-network module and an information feature encoding module. The specific construction process is as follows:

[0055] Construct the first Individual Q network module of each device: will the first Each device at time step Observations With time step action Feature concatenation is performed to obtain the feature vector. The concatenated feature vector is then input into a single linear layer. Feature extraction is completed in the linear layer. The input dimension is 15 and the output dimension is 64; the linear layer The feature extraction results are input into a single-layer gated recurrent unit. Output time steps The hidden state of the gated loop unit Among them, the gated loop unit The input dimension is 64 and the output dimension is 64; the hidden state will be... With time step Next Information to be sent by each device Feature concatenation is performed to obtain the feature vector. The feature vector is then input into a multilayer perceptron module containing two fully connected layers. Output the first Individual action value of a device Among them, the multilayer perceptron module The input dimension is 128, the hidden layer dimension is 64, and the output dimension is 5; based on maximizing the value of individual actions. Following the principle of greedy strategy, the first step is to select the... Each device at time step Actions to be performed This completes the construction of the individual Q-network module;

[0056] Construct the first The information feature encoding module of each device: determines the time step according to the information transmission rules set in step (2). Down to the The device number that sends information from the device (denoted as the No. 1) (a device), and the information sent by that device. ; receive the information With the Hidden state of individual Q-network devices By concatenating the vectors, we obtain the feature vectors. The feature vector is input into a single linear layer. Feature extraction is completed, including linear layers. The input dimension is 128, and the output dimension is 64; the extracted features are then input into a single-layer gated recurrent unit. In the middle, output time step Next Information to be sent by each device Among them, the gated loop unit The input dimension is 64, and the output dimension is 64; the construction of the information feature encoding module is completed;

[0057] (4) Training equipment decision module

[0058] Using formula

[0059] (3)

[0060] Get time steps The numbering is different from Any device number ,in For any integer in the interval [0, 297]; from the playback buffer Collection of the first The device and the first Each device at time step The information sent below ;

[0061] From the playback buffer Collection and Time Step Time steps differing by more than one communication cycle Next Information of each device ,in The range of values ​​is ,Will and Composition information pair ;

[0062] Each piece of information in all the above information pairs is input into a 2-layer multilayer perceptron module. Feature extraction is completed in the middle, including the multilayer perceptron module. The input dimension is 128, the hidden layer dimension is 64, and the output dimension is 80; the corresponding feature vectors are obtained, denoted as follows: and ;

[0063] Establish mutual information loss function

[0064] (4)

[0065] Train the information feature encoding module for all devices, using the pruning function. The value of the dot product of the eigenvectors is limited to ±5.

[0066] Loss function using independent deep Q-network algorithm

[0067] (5)

[0068] Train individual Q-network modules for all devices, where This indicates that the optimal value is taken for all possible joint actions in the next time step state; For the target Q-network, its network parameter update rule is: when the interval between two consecutive parameter updates reaches 200 rounds, the current individual Q-network is updated. The parameters are synchronized to the target Q network. ;

[0069] Using the total loss function

[0070] (6)

[0071] Training device decision module; setting the learning rate and training time steps to 5×10. -4 and 2×10 6 When the time step reaches 2×10 6 At this point, training ends, and the device decision network model trained at the last time step is saved;

[0072] (5) Achieve multi-device collaborative control

[0073] After training the device decision module in the multi-device collaborative control method based on exponential graph information communication, save the round reward and device decision network model to realize the collaborative control of the multi-device system.

Claims

1. A multi-device collaborative control method based on exponential graph information communication, characterized in that, The process of constructing a multi-device environment, designing an exponential graph communication topology, establishing a device decision-making module, training the device decision-making module, and realizing multi-device collaborative control includes the following steps: (1) Building a multi-device environment Construct a multi-device environment with 8 devices, and set the maximum number of time steps per round to 1. The task objective is to achieve collaborative assembly of one workpiece by eight devices. Each device has a 10-dimensional observation space, specifically including its own end-effector position (3D), end-effector velocity in 3D space (3D), workpiece assembly point position (3D), and end-effector assembly state (1D). Each device has a 5-dimensional motion space, corresponding to end-effector movement along coordinate axes (3D) and end-effector stop / start assembly operations (2D). Each device also corresponds to a dedicated assembly point on the workpiece. At each time step… Using the formula (1) Calculate the first Rewards for individual devices ,in For each device, For the 3-dimensional space The Euclidean distance between the end of the device and the assembly point. For the first The end state of each device is calculated using the following formula: (2) When all 8 devices have completed their respective assembly tasks at their respective assembleable points, each device receives an additional reward of 20, and the round ends, completing the construction of the multi-device environment. (2) Design an exponential graph communication topology An exponential graph is used to construct the communication topology between devices. Eight devices are arranged in a ring structure to form a directed graph communication architecture. The communication cycle is set to 3. The information transmission rules of the devices at different time steps are defined using the characteristics of the exponential graph, as follows: If the current time step When it is an integer multiple of 3, that is Based on index 2 0 =1 sets the communication direction: In a circular directed graph structure, each device sends information to the device whose number differs from the target device number by 1; specifically, the 0th device sends information to the 1st device, the 1st device sends information to the 2nd device, and so on, the 6th device sends information to the 7th device, and the 7th device sends information to the 0th device. If the current time step When it is a multiple of 3 plus 1, that is The communication direction is set based on the exponent 2¹=2: each device sends information to the device whose number differs from the target device number by 2; specifically, the 0th device sends information to the 2nd device, the 1st device sends information to the 3rd device, and so on, with the 6th device sending information to the 0th device and the 7th device sending information to the 1st device. If the current time step When it is a multiple of 3 plus 2, that is The communication direction is set based on the exponent 2²=4: each device sends information to the device whose number differs from the target device number by 4; specifically, the 0th device sends information to the 4th device, the 1st device sends information to the 5th device, and so on, the 6th device sends information to the 2nd device, and the 7th device sends information to the 3rd device. Using the above rules, complete the construction of the exponential graph communication topology; (3) Establish equipment decision-making module The equipment decision-making module consists of two parts: an individual Q-network module and an information feature encoding module. The specific construction process is as follows: Construct the first Individual Q network module of each device: will the first Each device at time step Observations With time step action Feature concatenation is performed to obtain the feature vector. The concatenated feature vector is then input into a single linear layer. Feature extraction is completed in the linear layer. The input dimension is 15 and the output dimension is 64; the linear layer The feature extraction results are input into a single-layer gated recurrent unit. Output time steps The hidden state of the gated loop unit Among them, the gated loop unit The input dimension is 64 and the output dimension is 64; the hidden state will be... With time step Next Information to be sent by each device Feature concatenation is performed to obtain the feature vector. The feature vector is then input into a multilayer perceptron module containing two fully connected layers. Output the first Individual action value of a device Among them, the multilayer perceptron module The input dimension is 128, the hidden layer dimension is 64, and the output dimension is 5; based on maximizing the value of individual actions. Following the principle of greedy strategy, the first... Each device at time step Actions to be performed This completes the construction of the individual Q-network module; Construct the first The information feature encoding module of each device: determines the time step according to the information transmission rules set in step (2). Down to the The device number that sends information from the device (denoted as the No. 1) (a device), and the information sent by that device. ; receive the information With the Hidden state of individual Q-network devices By concatenating the vectors, we obtain the feature vectors. The feature vector is then input into a single linear layer. Feature extraction is completed, including linear layers. The input dimension is 128, and the output dimension is 64; the extracted features are then input into a single-layer gated recurrent unit. In the middle, output time step Next Information to be sent by each device Among them, the gated loop unit The input dimension is 64, and the output dimension is 64; the construction of the information feature encoding module is completed; (4) Training equipment decision module Using formula (3) Get time steps The numbering is different from Any device number ,in For any integer in the interval [0, 297]; from the playback buffer Collection of the first The device and the first Each device at time step The information sent below ; From the playback buffer Collection and Time Step Time steps differing by more than one communication cycle Next Information of each device ,in The range of values ​​is ,Will and Composition information pair ; Each piece of information from all the above information pairs is input into a 2-layer multilayer perceptron module. Feature extraction is completed in the middle, including the multilayer perceptron module. The input dimension is 128, the hidden layer dimension is 64, and the output dimension is 80; the corresponding feature vectors are obtained, denoted as follows: and ; Establish mutual information loss function (4) Train the information feature encoding module for all devices, using the pruning function. The value of the dot product of the eigenvectors is limited to ±5. Loss function using independent deep Q-network algorithm (5) Train individual Q-network modules for all devices, where This indicates that the optimal value is taken for all possible joint actions in the next time step state; For the target Q-network, its network parameter update rule is: when the interval between two consecutive parameter updates reaches 200 rounds, the current individual Q-network... The parameters are synchronized to the target Q network. ; Using the total loss function (6) Training device decision module; setting the learning rate and training time steps to 5×10. -4 and 2×10 6 When the time step reaches 2×10 6 At this point, training ends, and the device decision network model trained at the last time step is saved; (5) Achieve multi-device collaborative control After training the device decision module in the multi-device collaborative control method based on exponential graph information communication, save the round reward and device decision network model to realize the collaborative control of the multi-device system.

Images