Modular robot self-reconfiguration method based on swarm agent deep reinforcement learning

Abstract

The application discloses a modular robot self-reconfiguration method based on swarm agent deep reinforcement learning, first, the configuration space and the motion space of the modular robot are modeled, the position of each module can be represented by coordinates (x, y). The upper, lower, left and right directions of each module can be connected with any other module, and all modules of the self-reconfiguration robot must be connected in this way to form a whole. Then, the motion space of a single module is modeled, so that it can get the support of other modules and avoid collision or other conflicts with other modules when moving. Finally, from the perspective of swarm agent deep reinforcement learning, each isomorphic module is regarded as a single agent with learning and perception ability, and a modular robot self-reconfiguration algorithm based on QMIX is proposed. The application can realize parallel movement of agents, to a certain extent, solve the coordination and cooperation problem among multiple agents, and thus realize the transformation from the initial configuration to the target configuration.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology, specifically relating to a modular robot self-reconfiguration method. Background Technology

[0002] Deep reinforcement learning is an artificial intelligence method that combines the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning. Through agents continuously taking actions and receiving feedback in their environment, deep neural networks select actions based on the current state, thereby obtaining a strategy to achieve the optimal solution to the problem. Deep reinforcement learning can be further divided into single-agent and multi-agent deep reinforcement learning based on the number of agents. Many researchers have proposed numerous effective algorithms in this field. For example, Deep Q-Network (DQN), combining convolutional neural networks from deep learning with Q-value-based decision-making in reinforcement learning, has achieved performance levels similar to top human players in games like Atari. Value decomposition-based cooperative multi-agent learning (VDN) decomposes the overall team Q-function into the sum of individual agent Q-functions, allowing individual agents to maximize team rewards through local greedy algorithms, thus improving the situation of spurious rewards and lazy agents in multi-agent reinforcement learning. The QMIX multi-agent reinforcement learning algorithm, based on value decomposition, uses a hybrid network to fuse the Q-functions of individual agents to obtain the Q-function of the joint action of the multi-agents, achieving better results than VDN.

[0003] Driven by the Fourth Industrial Revolution, characterized by intelligent manufacturing and the Internet Plus, robotics-related technologies and industries have received widespread attention. In large-scale industrial production, the high efficiency of dedicated robots has been fully demonstrated; however, their long design cycles and high manufacturing costs have made them a pressing problem for the industry. The modular concept, which has emerged in recent years, aims to decompose robots into a certain number of standard modules with independent functions, offering an effective way to shorten robot design cycles and reduce manufacturing and maintenance costs. Self-reconfigurable modular robots are a type of modular robot that can adaptively adjust their configuration according to changes in the environment or different tasks. As the number of modules increases, the configuration space grows exponentially. Finding a fast and energy-efficient optimization path between the initial and target configurations is currently a hot and challenging issue in self-reconfigurable modular robot research. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention provides a modular robot self-reconfiguration method based on swarm intelligence deep reinforcement learning. First, the modular robot's configuration space and motion space are modeled. Modules exist in the Cartesian plane, and the position of each module can be represented by coordinates (x, y). Each module can be connected to any other module in the four directions (up, down, left, and right), and all modules of the self-reconfigurable robot must be connected in this way to form a whole. Then, the motion space of a single module is modeled to ensure that its movement is supported by other modules and avoids collisions or other conflicts. Finally, from the perspective of swarm intelligence deep reinforcement learning, each isomorphic module is regarded as a single agent with learning and perception capabilities, and a modular robot self-reconfiguration algorithm based on QMIX is proposed. This invention enables parallel movement of agents, solves the coordination and cooperation problem between multiple agents to a certain extent, and thus realizes the transformation from the initial configuration to the target configuration.

[0005] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0006] Step 1: Construct a modular robot configuration space;

[0007] The robot's modules exist in the Cartesian plane, and the position of each module is represented by coordinates (x, y);

[0008] Each module can connect to any other module in the four directions of up, down, left, and right. All modules of the robot are connected in this way to form a whole. The different absolute positions and connection relationships of the robot's modules form different configurations.

[0009] When distinguishing different configurations, the absolute coordinates of the modules are used as the criterion, which requires not only that the shapes of the components of the configurations be the same, but also that the positions of the robots in the two-dimensional plane be the same; no distinction is made between the components of each configuration, that is, each module is isomorphic.

[0010] Step 2: Construct a modular robot motion space;

[0011] Each module is circular in shape and moves by rotating around its edge. Each module can move in eight directions or remain stationary, forming a motion space of size 9. That is, if the module is at (x,y), its position after moving will be (x,y-1), (x,y+1), (x-1,y), (x+1,y), (x-1,y-1), (x-1,y+1), (x+1,y-1), (x+1,y-1), (x+1,y+1), or (x,y).

[0012] Step 3: Reward function based on module overlap;

[0013] A reward function is defined:

[0014]

[0015] Among them O T+1 O represents the degree of overlap between the new configuration derived from the current configuration at time step T and the target configuration after one step expansion. T N represents the degree of overlap between the current configuration and the target configuration at time step T. agent O0 indicates the number of agents, and O0 indicates the degree of overlap between the initial configuration and the target configuration.

[0016] Step 4: Implementation of modular robot self-reconstruction based on deep reinforcement learning of swarm intelligence;

[0017] Step 4-1: Initialization;

[0018] The agent's observations are obtained based on the interaction between the agent and the environment. i Get agent behavior avail_act i Based on intelligent agent observations i The QMIX network yields a value function q_value, which is used to select an action with a probability of 1-ε. i =argmax i q_value, otherwise in the optional action available_act i Randomly select actions in the action i , will the action i Add a t middle;

[0019] Using action vector a t The agent's state environment is updated in parallel, and the reward r is obtained according to the reward function. t At the same time, it enters the next state s t+1 The state transition pairs of all agents are stored in the experience pool; for each agent, legal actions are selected and added to avail_act. i The list is checked for overlap, and avail_act is modified accordingly. i , will avail_act i Add avail_cations t+1 ;

[0020] Step 4-2: Determine whether the configuration matches the target configuration, and update the network parameters accordingly;

[0021] Step 4-3: The QMIX loss function is used for backpropagation to correct network parameters, expressed as:

[0022]

[0023] Where h is the number of samples taken from the experience replay pool; Q represents the reward associated with the target configuration. tot (s,a,τ;θ) represents the configuration reward obtained by the configuration reconstruction method based on QMIX;

[0024] The update process is completed using the following formula:

[0025]

[0026] in Here, r represents the reward function, and γ represents the discount factor. This represents the reward received by the target network;

[0027] Step 5: After obtaining the target network, perform configuration transformation according to the target configuration.

[0028] Preferably, the initialization includes an experience replay pool D with a capacity of M, a maximum number of steps per round T, a number of agents N, a current number of steps taken (step = 0), and a maximum number of training steps (max_step). The QMIX network parameters θ are initialized.

[0029] The beneficial effects of this invention are as follows:

[0030] This invention enables parallel movement of intelligent agents, which to some extent solves the problem of coordination and cooperation among multiple intelligent agents, thereby realizing the transformation from the initial configuration to the target configuration. Attached Figure Description

[0031] Figure 1 This is a conceptual diagram of the method of the present invention.

[0032] Figure 2 This refers to the action space and sequence number of the module in the embodiment of the present invention. Detailed Implementation

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] This invention proposes a modular robot self-reconfiguration method based on deep reinforcement learning. The method utilizes the following principle: during the self-reconfiguration process, each module can be considered a single agent with certain perception and learning capabilities. Rapid transformation from the initial configuration to the target configuration is achieved through swarm intelligence deep reinforcement learning training. By designing a novel reward function and restricting the agent's action space, the agent continuously takes actions in the environment and receives feedback. The deep neural network selects actions based on the current state, thereby obtaining a strategy to achieve the optimal solution to the problem. Deep reinforcement learning is an artificial intelligence method that combines the perception capabilities of deep learning with the decision-making capabilities of reinforcement learning. Since the modular robot can be considered a swarm intelligence, a combination of swarm intelligence reinforcement learning algorithms and deep learning algorithms can be used to automate its configuration transformation.

[0035] A modular robot self-reconstruction method based on deep reinforcement learning includes the following steps:

[0036] Step 1: Constructing the Modular Robot Configuration Space

[0037] The modules exist in the Cartesian plane, and the position of each module can be represented by coordinates (x, y).

[0038] Each module can connect to any other module in all four directions (up, down, left, and right). All modules of the self-reconfigurable robot must be connected in this way to form a whole. The different absolute positions and connection relationships of the modules of the self-reconfigurable robot result in different configurations.

[0039] When distinguishing between different configurations, the absolute coordinates of the modules are used as the criterion. This means that not only must the shapes of the components of the configurations be the same, but the positions of the robots in the two-dimensional plane must also be the same. The constituent modules of each configuration are not differentiated; that is, each module is isomorphic, and swapping the positions of any two modules will not result in a new configuration.

[0040] Step 2: Constructing a modular robot motion space

[0041] The algorithm focuses on single-type modules, each circular in shape, that move by rotating around the edge. Ideally, each module can move in eight directions or remain stationary, creating an action space of size 9. That is, if a module is initially at (x, y), its position after moving could be (x, y-1), (x, y+1), (x-1, y), (x+1, y), (x-1, y-1), (x-1, y+1), (x+1, y-1), (x+1, y+1), (x+1, y+1), or (x, y).

[0042] Step 3: Reward function based on module overlap

[0043] It is inappropriate to only award rewards when a module reaches the target configuration, because the number of configurations is vast, and in many cases the agent cannot reach the target configuration and therefore cannot receive the corresponding reward. The algorithm defines a new reward function:

[0044]

[0045] Among them O T+1 O represents the degree of overlap between the new configuration derived from the current configuration at time step T and the target configuration after one step expansion. T represents the degree of overlap between the current configuration and the target configuration at time step T, while N agent O0 represents the number of agents, and O0 represents the degree of overlap between the initial configuration and the target configuration.

[0046] This formula reflects that the reward for each step of the swarm agent is related to the overlap between the new and old configurations, which encourages multi-agents to learn strategies that maximize overlap. It also considers time delay, meaning that each step of the swarm agent receives a very small negative reward, which to some extent motivates the module group to quickly transform into the target configuration.

[0047] Step 4: Implementation of Modular Robot Self-Reconfiguration Based on Swarm Intelligence Deep Reinforcement Learning

[0048] First, initialize the experience replay pool D with a capacity of M, the maximum number of steps per round T, the number of agents N, the current number of steps taken step = 0, and the maximum number of training steps max_step. Initialize the QMIX network parameters θ.

[0049] The agent's observations are obtained based on the interaction between the agent and the environment. i Get agent behavior avail_act i Based on intelligent agent observations i The QMIX network yields a value function q_value, which is used to select an action with a probability of 1-ε. i =argmax i q_value, otherwise in the optional action available_act i Randomly select actions in the action i , will the action i Add a t middle.

[0050] Using action vector a t The agent's state environment is updated in parallel, and the reward r is obtained according to the reward function. t At the same time, it enters the next state s t+1And store all agent state transition pairs in the experience pool. For each agent, filter for valid actions and add them to avail_act. i The list is checked for overlap, and avail_act is modified accordingly. i . avail_act i Add avail_actions t+1 .

[0051] Determine whether the configuration matches the target configuration, and update the network parameters accordingly.

[0052] The QMIX loss function, used for backpropagation to correct network parameters, can be expressed as:

[0053]

[0054] Where h is the number of samples taken from the experience replay pool. The update process utilizes traditional deep neural network concepts and can be accomplished using the following formula:

[0055]

[0056] in It is the target network.

[0057] Once the target network is obtained, configuration transformation can be performed based on the target configuration. Specific implementation examples:

[0059] Step 1: Constructing the Modular Robot Configuration Space

[0060] The modules exist in the Cartesian plane, and the position of each module can be represented by coordinates (x, y).

[0061] Each module can connect to any other module in all four directions (up, down, left, and right). All modules of the self-reconfigurable robot must be connected in this way to form a whole. The different absolute positions and connection relationships of the modules of the self-reconfigurable robot result in different configurations.

[0062] When distinguishing between different configurations, the absolute coordinates of the modules are used as the criterion. This means that not only must the shapes of the components of the configurations be the same, but the positions of the robots in the two-dimensional plane must also be the same. The constituent modules of each configuration are not differentiated; that is, each module is isomorphic, and swapping the positions of any two modules will not result in a new configuration.

[0063] Step 2: Constructing a modular robot motion space

[0064] The algorithm focuses on single-type modules, each circular in shape, that move by rotating around the edge. Ideally, each module can move in eight directions or remain stationary, creating a motion space of size 9. That is, if a module is initially at (x, y), its post-movement position could be (x, y-1), (x, y+1), (x-1, y), (x+1, y), (x-1, y-1), (x-1, y+1), (x+1, y-1), (x+1, y+1), or (x, y). Figure 2 As shown. Moving a module requires meeting three basic requirements:

[0065] 1. The movement of a module must be supported by other modules to ensure that it can rotate around a fulcrum from one side of the starting position to the target position and stop to form a new connection.

[0066] 2. During the movement of the module, sufficient geometric space must be reserved so that the module will not collide with other modules when it is flipped.

[0067] 3. The target position of the module's movement must not coincide with the position of other modules, and the movement of the module must not disrupt the overall connectivity of the modular robot. To meet the above requirements, when the module moves in any of the eight directions, it must be ensured that there is a module providing support on one side, while leaving space for flipping on the other side.

[0068] Step 3: Reward function based on module overlap

[0069] It is inappropriate to only award rewards when a module reaches the target configuration, because the number of configurations is vast, and in many cases the agent cannot reach the target configuration and therefore cannot receive the corresponding reward. The algorithm defines a new reward function:

[0070]

[0071] Among them O T+1 O represents the degree of overlap between the new configuration derived from the current configuration at time step T and the target configuration after one step expansion. T represents the degree of overlap between the current configuration and the target configuration at time step T, while N agent O0 represents the number of agents, and O0 represents the degree of overlap between the initial configuration and the target configuration.

[0072] This formula reflects that the reward for each step of the swarm agent is related to the overlap between the new and old configurations, which encourages multi-agents to learn strategies that maximize overlap. It also considers time delay, meaning that each step of the swarm agent receives a very small negative reward, which to some extent motivates the module group to quickly transform into the target configuration.

[0073] Step 4: Implementation of Modular Robot Self-Reconfiguration Based on Swarm Intelligence Deep Reinforcement Learning

[0074] First, initialize the experience replay pool D with a capacity of M, the maximum number of steps per round T, the number of agents N, the current number of steps taken (step = 0), and the maximum number of training steps (max_step). Initialize the QMIX network parameters θ, and the network structure is as follows: Figure 1 As shown.

[0075] The agent's observations are obtained based on the interaction between the agent and the environment. i Get agent behavior avail_act i Based on intelligent agent observations i The QMIX network yields a value function q_value, which is used to select an action with a probability of 1-ε. i =argmax i q_value, otherwise in the optional action available_act i Randomly select actions in the action i , will the action i Add a t middle.

[0076] Using action vector a t The agent's state environment is updated in parallel, and the reward r is obtained according to the reward function. t At the same time, it enters the next state s t+1 And store all agent state transition pairs in the experience pool. For each agent, filter for valid actions and add them to avail_act. i The list is checked for overlap, and avail_act is modified accordingly. i . avail_act i Add avail_actions t+1 .

[0077] Determine whether the configuration matches the target configuration, and update the network parameters accordingly.

[0078] The QMIX loss function, used for backpropagation to correct network parameters, can be expressed as:

[0079]

[0080] Where h is the number of samples taken from the experience replay pool. The update process utilizes traditional deep neural network concepts and can be accomplished using the following formula:

[0081]

[0082] in It is the target network.

[0083] Once the target network is obtained, configuration transformation can be performed based on the target configuration.

[0084] Table 1 shows the success rate under different time step limits.

[0085] Table 1 Comparison of success rates under different time and step limits

[0086]

[0087] The results show that the self-reconstruction power of the QMIX reinforcement learning algorithm cannot reach 100% as the training time step limit increases, but it still maintains a high success rate. This is because the random action space during reinforcement learning training is very large, making it difficult for training to converge to a model with very high accuracy. Regarding the average number of self-reconstruction steps, the QMIX self-reconstruction algorithm performs well under strict time step limits, which is inseparable from the parallel movement of modules. In most configurations, the QMIX-based self-reconstruction algorithm can achieve a number of steps that is close to that of traditional algorithms, indicating that the agent trained using the QMIX-based self-reconstruction algorithm can effectively complete the self-reconstruction process.

Claims

1. A modular robot self-reconfiguration method based on swarm agent deep reinforcement learning, characterized in that, Comprising the following steps: Step 1: Constructing the modular robot configuration space; The modules of the robot exist in a Cartesian plane, and the position of each module is represented by coordinates (x, y); Each module can be connected to any other module in four directions, i.e., up, down, left, and right, and all modules of the robot are connected in this way to form a whole; Different configurations are formed by different absolute positions and connection relationships of the modules of the robot; When distinguishing different configurations, the absolute coordinates of the modules are used as the criterion, i.e., not only the shape of the configuration is required to be the same, but also the position of the robot in the two-dimensional plane is required to be the same; For the modules constituting each configuration, no distinction is made, i.e., each module is isomorphic; Step 2: Constructing the action space of the modular robot; The shape of each module is circular, and it moves in a rotating manner around the edge; each module can move or remain stationary in eight directions around it, forming an action space of size 9, i.e., if the position of the module is (x, y), after moving, the position of the module is (x, y-1), (x, y+1), (x-1, y), (x+1, y), (x-1, y-1), (x-1, y+1), (x+1, y-1), (x+1, y+1), or (x, y); where O T+1 represents the overlap between the new configuration extended from the current configuration at the Tthtime step and the target configuration, O T represents the overlap between the current configuration at the Tthtime step and the target configuration, N agent represents the number of agents, and O0represents the overlap between the initial configuration and the target configuration. Step 3: Reward function based on module coincidence degree; A reward function is defined: The agent's observations are obtained based on the interaction between the agent and the environment. i Get agent behavior available_act i Based on intelligent agent observations i The QMIX network yields a value function q_value, which is used to select an action with a probability of 1-ε. i =argmax i q_value, otherwise in the optional action available_act i Randomly select actions in the action i , will the action i Add a t middle; Utilizing action vector a t Parallelizing the update of the agent state environment, and obtaining the reward r according to the reward function t , and entering the next state s simultaneously t+1 , and storing the state transition pair of all agents into the experience pool; screening the legal actions for each agent, and adding avail_act i into the avail_act list, performing coincidence judgment, and modifying avail_act i , and adding avail_act i into avail_actions t+1 ; Step 4: Self-reconfiguration of the modular robot based on swarm agent deep reinforcement learning; Step 4-1: Initialization; where h is the number of samples from the experience replay pool; Qtargetrepresents the reward represented by the target configuration tot (s, a, τ; θ) represents the configuration reward obtained based on the QMIX-based configuration reconstruction method; Step 4-2: Determine whether the configuration coincides with the target configuration, and update the network parameters accordingly; wherein is the target network, r denotes the reward function, γ denotes the discount factor, denotes the reward obtained by the target network; Step 4-3: The loss function of QMIX is used for backpropagation to correct the network parameters, which is represented as:

2. The modular robot self-reconfiguration method based on swarm agent deep reinforcement learning according to claim 1, characterized in that, The update process is completed using the following formula: Step 5: After obtaining the target network, the configuration transformation is realized according to the target configuration. The initialization includes an experience replay pool D with a capacity of M, a maximum number of steps T per round, a number of agents N, a current number of steps step = 0, and a maximum training step max_step; Initialize the QMIX network parameters θ.

Images