A self-reconfiguration method for free-form modular robots

Abstract

This invention discloses a self-reconfiguration method for free-form modular robots. First, an abstract model of the basic elements of a class of free-form modular robots is constructed, where each robot module can be connected to each other from any direction via active and passive connectors. Then, based on a hardware prototype, a reasonable action space and observation space are designed for this type of resource-constrained modular robot. The action space includes the drive parameters of two planar continuous motion motors and the motion parameters of one connector, while the observation space includes the motion information and local connectivity relationships of the robot itself and its neighboring modules. The reconfiguration problem of the free-form modular robot is modeled as a distributed partially observable Markov decision process. Based on a proximal policy optimization algorithm, an efficient and conflict-free multi-agent reinforcement learning method for reconfiguration in the continuous action space can be achieved. Finally, the timely termination of the reconfiguration process is achieved under the conditions of local observability and limited communication.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, specifically relating to a modular robot self-reconfiguration method. Background Technology

[0002] Modular robotic systems can change between different configurations, enabling them to adapt to complex and dynamic real-world environments. Compared to traditional fixed-structure robots, modular robots offer potential advantages in terms of low cost, robustness, and versatility. Therefore, this concept has garnered widespread attention in many fields, including robotics and artificial intelligence, and has shown initial application prospects in areas such as industrial manufacturing, smart homes, and aerospace.

[0003] Free-form modular robot systems, as a novel type of modular robot, are a research hotspot in the modular robotics community. In this system, the modular robot can move independently and possesses a relatively free connection mechanism, forming an amorphous configuration. The free-form connection mechanism is the most important characteristic of free-form modular robot systems; it is achieved using permanent magnets, viscous materials, and loosely coupled mechanical structures. Free-form modular robots can be reconfigured more freely in a continuous configuration space, improving self-reconfiguration efficiency and reducing connection errors. However, free-form modular robot systems have a larger configuration space, and the movement between modules is more prone to collisions. How to accurately and effectively change the configuration—the self-reconfiguration problem—remains challenging.

[0004] Researchers have explored reconfiguration motion planning for modular robots using search-based and control-based methods. Due to the complexity of the reconfiguration problem and the limited computational resources of modular robots, search-based methods require a long time to find the reconfiguration path. More research attempts to design reconfiguration control rules that do not require global information and are better suited to the distributed characteristics of modular robots. Biomimetic or manually designed rules, gradient calculations based on various virtual forces, and the use of scaffolding all attempt to solve the reconfiguration problem for various types of modular robots. In these methods, modules automatically execute parallel distributed reconfiguration motions with higher runtime efficiency. Conflicts occur during distributed motion due to kinematic constraints between modules. The probability of conflict increases with the number of modules in reconfiguration. Conflicts will lead to reconfiguration failure or reduced reconfiguration efficiency. In existing work, conflict handling means restricting the movement of certain modules. For example, collisions are avoided by providing a gap between robots in transport with an empty unit; or by manually setting detection points to avoid blockage when creating cavity shapes. In the continuous reconfiguration of free-form modular robots, the conflict problem is more severe, and conflict avoidance methods designed for discrete motion spaces are not suitable. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention provides a self-reconfiguration method for free-form modular robots. First, an abstract model of the basic elements of a class of free-form modular robots is constructed. Each robot module contains necessary sensing, computing, and driving devices, and they can be connected to each other from any direction via active and passive connectors. Then, based on a hardware prototype, a reasonable action space and observation space are designed for this type of resource-constrained modular robot. The action space includes the driving parameters of two planar continuous motion motors and the motion parameters of one connector. The observation space includes the motion information and local connectivity relationships of the robot itself and its neighboring modules. The reconfiguration problem of free-form modular robots is modeled as a distributed, partially observable Markov decision process. Based on a proximal policy optimization algorithm, a two-stage collaborative optimization mechanism inspired by altruistic scales is introduced, proposing a multi-agent reinforcement learning method that can achieve efficient and conflict-free reconfiguration in a continuous action space. Finally, a distributed termination criterion is designed to achieve timely termination of the reconfiguration process under conditions of local observability and limited communication.

[0006] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0007] Step 1: Construct a free-form modular robot abstract model;

[0008] Each module of the free-form modular robot is considered as a standard sphere, including active connectors, passive connectors, and sensing devices. The modules are driven by motion motors; the spherical shell of the module is used as a passive connector; the motion motor of the module has continuous motion capability in two-dimensional space, and the module can move on the surface of the connected modules; the modules can communicate with each other.

[0009] Consider a modular robot system {M} i}, consisting of n modules, given a pair of initial configurations x from configuration space X. start and target configuration x target The goal of the self-reconstruction method is to find an action sequence {u} within the allowed action space U. i，t This allows the reconstructed action sequence to complete x in the shortest possible time T. start Convert to x target Any given configuration x consists of the location information Pi of each module and the topological connection information δ of all modules. i，j The sole decision; during the reconstruction process, the action space U contains the actions that enable the motor torque and speed vectors of each module to satisfy the corresponding motion constraints under the current conditions;

[0010] Step 2: Observation and motion space setup;

[0011] Each module's observations include proprioception and environmental perception; proprioception includes the position vector P. i Velocity vector v i Altruism factor α i and active-passive connection relationship δ i，I\{i} Each module can also sense the position P of other modules j within a radius d. j and speed v j ;

[0012] The connection or disconnection of the active connector is incorporated into the module's action space, which is represented by a tuple <σ1, σ2, δ. link > indicates the linking action δ for each module. link It is stated that only modules in contact with each other can perform the action of connecting or disconnecting; the torques of the two motion motors are σ1 and σ2, which are used to drive the module to move in any direction on the plane; the maximum moving speed of the module is limited to v. max ;

[0013] Step 3: Design the reward function;

[0014] Design a reward function shared by all modules to train modules to learn the reconstruction strategy:

[0015]

[0016] In this equation, the first term on the right-hand side is the topological reward, the second term is the geometric reward, and the third term is the time reward, determined by the hyperparameter c. p c q and c t balance;

[0017] For each module M i The connection relationship with other n-1 modules is represented by δ. i，j Let j belong to I{i}; the topological reward design is as follows:

[0018]

[0019] In the equation, the first and last two terms of the numerator on the right are M at time steps t-1 and t, respectively. i The distance between the local topology and the target topology;

[0020] Incorporating further geometric information, the geometric reward design is as follows:

[0021]

[0022] Wherein, the first term on the right side of the formula is the displacement vector of the module; For P i，t-1 Starting from point P i，tP i，G The angle between the two vectors at the endpoints; the geometric reward measures the effective distance the module moves to the target position in the correct direction at each time step;

[0023] Step 4: Altruistic proximate strategy optimization;

[0024] Introducing an altruism mechanism into multi-agent reinforcement learning to achieve collaboration between modules;

[0025] The mean field reward for each module is defined based on the mean field concept:

[0026]

[0027] Where, N i，t ={j:||P it -P jt ||≤d} is a dynamically changing set containing M i The index of other modules within a radius of d; the mean field reward represents the value in module M. i The average reward of the surrounding dynamically changing neighboring modules;

[0028] Introducing an altruism scale to form altruistic rewards:

[0029]

[0030] Where, α i It is an altruistic factor defined on (-1, 1) to measure each module's attitude toward the gains of others; the vector α is α i The set represents the distribution of altruistic tendencies within the entire population; the altruism factor α i It is each module M i A personalized attribute;

[0031] The first stage of the altruistic proximal policy optimization method trains the proximal policy optimization PPO through centralized training and distributed execution CTDE to maximize the altruistic reward R. AS :

[0032]

[0033] The altruistic advantage function is:

[0034] There exists an optimal α. i The distribution α* optimizes the performance of the model trained in the first stage of the method.

[0035] Based on meta-reinforcement learning, appropriate personalized α is optimized for each module. iThe second-stage optimization objective of the altruistic proximal strategy optimization method is:

[0036]

[0037] By using the chain rule and Taylor series derivation, the gradient of the above optimization objective is rewritten with personalized altruistic factors, so that the personalized altruistic factors of all modules can be optimized during training:

[0038]

[0039] The altruistic proximal policy optimization method performs gradient descent on the event scale according to the above formula to optimize the altruistic factor of each module, and performs gradient descent on the time step scale to maximize the altruistic reward RAS.

[0040] Step 5: Distributed Asynchronous Termination Criteria;

[0041] All modules operate under the same distributed asynchronous termination criterion; each module determines whether its local connectivity matches the target configuration based on its local observations, and when a match is found and the in-place signal Ω is received from all sub-modules, the in-place signal Ω is sent to the parent module; in this way, the in-place signal is progressively and distributedly transmitted from the leaf nodes to the root node; then, the root node passes the termination signal Ω2 to the child nodes and transmits it level by level; each module that receives the Ω2 signal will immediately terminate the reconstruction process.

[0042] The beneficial effects of this invention are as follows:

[0043] 1. This invention designs a reconstruction algorithm based on multi-agent reinforcement learning for free-form modular robot systems. Free-form modular robot systems possess a larger configuration space and are more prone to conflicts during continuous reconstruction. Existing search-based and control-based methods are not suitable for the reconstruction problem of this type of modular robot. Compared with existing methods used in the reconstruction problems of other types of modular robot systems, this invention achieves automatic design of the reconstruction model based on neural network training, avoiding the manual design of reconstruction motion rules and conflict avoidance mechanisms, thus improving the ease of application for free-form modular robots.

[0044] 2. The core of this invention is a multi-agent reinforcement learning algorithm called the Altruistic Proximal Policy Optimization Algorithm. Based on the proximal policy optimization algorithm and the centralized training distributed execution framework, the Altruistic Proximal Policy Optimization Algorithm introduces a two-stage collaborative optimization mechanism inspired by altruism scales. Through parameter sharing and meta-reinforcement learning, it trains the distribution of personalized altruistic factors in modules, enabling modules to explore and optimize their own reconstruction motion strategies. The reconstruction problem of free-form modular robots is essentially a sequential social dilemma problem in a continuous action space. Thanks to the personalized altruistic factors and two-stage collaborative optimization mechanism of this invention, robot modules can learn to adopt appropriate cooperative behaviors to avoid conflicts during the reconstruction process and improve the system's reconstruction performance. Compared with existing advanced baseline algorithms based on the proximal policy optimization algorithm, this invention reduces the average conflict rate during the reconstruction process (by at least 7.1%), thereby increasing the reconstruction success rate (by an average increase of 6.8%) and reducing the average reconstruction time (by an average reduction of 1.3 seconds).

[0045] 3. This invention includes an asynchronous distributed termination criterion for free-form modular robots. This termination criterion is designed so that each module determines whether its local connectivity matches the target configuration based on local observations, transmitting in-place signals from bottom to top and termination signals from top to bottom, thus avoiding reliance on real-time updates of global configuration information. Free-form modular robots have limited computational resources for individual modules, making it difficult for the system to synchronize global configuration information without delay. A termination criterion that does not rely on global configuration information is beneficial for the application of learning-based algorithms in various scientific problems involving free-form modular robots. Attached Figure Description

[0046] Figure 1 The following is a conceptual diagram of a free-form modular robot abstract model in an embodiment of the present invention: (a) a configuration consisting of four modules and an explanation of the important components of the modules; (b) a configuration consisting of the same modules that is geometrically heterogeneous to the configuration in (a); and (c) a configuration consisting of the same modules that is topologically heterogeneous to the configuration in (a).

[0047] Figure 2 This is a framework diagram of the altruistic proximal strategy optimization algorithm in an embodiment of the present invention. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0049] This invention proposes a self-reconfiguration method for free-form modular robots, utilizing the following principle: The reconfiguration problem of free-form modular robots is essentially a sequential social dilemma problem in a continuous action space. For each module, adopting a selfish strategy can only improve short-term benefits, while adopting appropriate cooperative behavior can improve overall long-term benefits. Based on this principle, and using the sociological principle of altruistic scales, a personalized altruistic factor is introduced for each module. Training based on proximal policy optimization allows modules to explore and optimize their own reconfiguration motion strategies, which can improve the local reconfiguration performance of each module. Training based on meta-reinforcement learning optimizes the altruistic factor of each module, avoiding reconfiguration failures and performance losses caused by reconfiguration motion conflicts between modules, and can synergistically improve the reconfiguration performance of the entire robot system.

[0050] A self-reconfiguration method for a free-form modular robot includes the following steps:

[0051] Step 1: Construct a free-form modular robot abstract model

[0052] The main components of a free-form modular robot are modeled as follows: Each module is considered a standard sphere, including an active connector, a passive connector, and necessary sensing devices, driven by motion motors. The entire spherical shell can be used as a passive connector, and the active connector within the module determines how it connects with the passive connector of the module it is contacting. The module's motion motors have continuous motion capabilities in two-dimensional space, and the module can move on the surface of the connected modules. Modules can communicate with each other within a certain range. Theoretically, this abstract model can represent all connections and motion modes of any free-form modular robot.

[0053] Consider a modular robot system {M} i}, consisting of n modules, given a pair of initial configurations x from configuration space X. start and target configuration x target The goal of the reconstruction algorithm is to find an action sequence {u} within the allowed action space U. i，t This allows the reconstructed action sequence to complete x in the shortest possible time T. start Convert to x target Any given configuration x consists of the location information Pi of each module and the topological connection information δ of all modules. i，j The sole decision. During the reconstruction process, the action space U contains the actions that ensure the motor torque and velocity vectors of each module satisfy the corresponding motion constraints under the current conditions.

[0054] Step 2: Observation and Action Space Setup

[0055] The method aims to achieve autonomous module reconstruction through multi-agent reinforcement learning. This requires appropriately setting the perceptual and motion parameters of each module to avoid redundant information and improve training speed. Each module's observations include ontology perception and environmental perception. Ontology perception includes the position vector P. i Velocity vector v i Altruism factor α i and active-passive connection relationship δ i，I\{i} Each module can also sense the position P of other modules j within a radius d. j and speed v j Although all modules share the same policy network parameters when executed in a step-by-step manner, each module acts differently based on its unique observations.

[0056] The method directly incorporates the connection / disconnection of the active connector into the module's action space, which can be represented by a tuple <σ1, σ2, δ. link The > symbol represents the linking action of each module. link It is stated that only modules in contact with each other can perform the action of connecting or disconnecting. Two motion motors with torques σ1 and σ2 drive the modules to move in any direction on the plane. Furthermore, the maximum moving speed v of the modules is artificially limited. max .

[0057] Step 3: Design the reward function

[0058] To address the aforementioned reconstruction problem, we designed a reward function shared by all modules to train the modules to learn the reconstruction strategy:

[0059]

[0060] The first term is the topological reward, the second is the geometric reward, and the third is a small time reward, determined by the hyperparameter c. p c q and c t Balance. This encourages modules to rapidly approach the target configuration in both geometric and topological connectivity directions.

[0061] Topology rewards are designed based on local topology information of the configuration. For each module M i Its connection relationship with other n-1 modules can be expressed by δ. i，j Let j belong to I{i}. To incentivize modules to take actions that bring their local topological connections closer to their local topological relationships in the target configuration, the topological reward is designed as follows:

[0062]

[0063] The two terms before and after the molecular term are time step t-1 and time t, respectively.i The distance between the local topology and the target topology. This difference form avoids the sparsity of the reward. The denominator is only related to the module M in the initial configuration and the target configuration. i The reward is normalized as a normalization factor related to the topology of the local connections, thus avoiding repeated tuning of the hyperparameters.

[0064] Utilizing connectivity can only promote topological similarity of configurations. To ensure the overall configuration maintains shape consistency with the target configuration, further geometric information needs to be introduced. The geometric reward design is as follows:

[0065]

[0066] The first term is the displacement vector of the module. For P i,t-1 Starting from point P i,t P i,G (M in the target configuration) i The position vector is the angle between the two vectors at the endpoints. Therefore, the geometric reward measures the effective distance the module moves to the target position in the correct direction at each time step.

[0067] Step 4: Altruistic Proximity Strategy Optimization

[0068] This method addresses the reconfiguration problem of free-form modular robots from the perspective of hybrid motivation, enabling each module to learn coordination and conflict avoidance. Inspired by altruism scales, the method introduces altruistic mechanisms into multi-agent reinforcement learning to achieve cooperation between modules, avoid conflict, and thus successfully and effectively complete the reconfiguration.

[0069] First, the method considers how to measure the gains of others. The neighbors of a module and the relationships between them change dynamically during the reconstruction process. The mean-field reward for each module is defined based on the mean-field concept:

[0070]

[0071] Where, N i，t ={j:||P it -P jt ||≤d} is a dynamically changing set containing M i The indices of other modules within a radius of d. Thus, the mean field reward represents the reward in module M. i The average reward of neighboring modules that are dynamically changing. Each module M i Only the impact on this single reward needs to be considered, rather than the impact of the action on each specific module individually. This reduces interaction costs and ensures the scalability of the method.

[0072] Then, the sociological mechanism of an altruism scale is introduced to measure the social tendency of each module toward the benefit of others, and to form altruistic rewards:

[0073]

[0074] Where, α i It is an altruistic factor defined on (-1,1), which measures each module's attitude toward the gains of others. Vector α is α i The set represents the distribution of altruistic tendency within the entire population. The altruism factor α... i It is each module M i One of its personalized attributes, heterogeneity and diversity, is that it can improve overall performance.

[0075] The first stage of the altruistic proximal policy optimization method trains the PPO (proximal policy optimization) through CTDE (centralized training and distributed execution) to maximize the altruistic reward R. AS :

[0076]

[0077] The altruistic advantage function is:

[0078] There exists an optimal α. i The distribution α* optimizes the model performance during the first stage of training. However, manually designing each αi is impractical. Therefore, based on meta-reinforcement learning, the method includes another layer of training to optimize appropriate personalized α for each module. i The second-stage optimization objective of the altruistic proximal strategy optimization method is:

[0079]

[0080] By using the chain rule and Taylor series derivation, the gradient of the above optimization objective is rewritten with personalized altruistic factors, so that the personalized altruistic factors of all modules can be optimized during training:

[0081]

[0082] Thus, the altruistic proximal policy optimization method performs gradient descent on the event scale according to the above formula to optimize the altruistic factor of each module, and performs gradient descent on the time step scale to maximize the altruistic reward RAS.

[0083] Step 5: Distributed Asynchronous Termination Criteria

[0084] Reconfiguration requires explicit termination criteria so that modular robotic systems can continue performing tasks using the target configuration. In practical deployments of modular robots, directly specifying the termination time step is not suitable. Furthermore, due to the limited communication and observation capabilities of the modules, it is difficult to obtain real-time global information about the configuration.

[0085] All modules operate under the same distributed asynchronous termination criterion. Each module determines whether its local connectivity matches the target configuration based on its local observations. When a match is found and the module receives the in-place signal Ω from all its sub-modules (connected by the current module), it sends the in-place signal Ω to its parent module (connected to the current module). In this way, the in-place signals are progressively distributed from the leaf nodes to the root node (modules without a parent node). The root node then passes the termination signal Ω2 to its child nodes, and this transmission continues level by level. Each module receiving the Ω2 signal immediately terminates the reconfiguration process. This asynchronous termination criterion avoids reliance on real-time updates to global configuration information. Specific implementation examples:

[0087] Step 1: Construct a free-form modular robot abstract model

[0088] like Figure 1 As shown in (a), the main components of the free-form modular robot are modeled as follows: Each module is considered a standard sphere, including an active connector, a passive connector, and necessary sensing devices, driven by motion motors. The entire spherical shell can be used as a passive connector, and the active connector in the module determines how it connects with the passive connector of the contacting module. The module's motion motors have continuous motion capability in two-dimensional space, and the module can move on the surface of the connected modules. The modules can communicate with each other within a certain range. Figure 1 As shown in (b) and (c), modules can form different configurations that are topologically or geometrically heterogeneous. Theoretically, this abstract model can represent all the connections and motion modes of any free-form modular robot.

[0089] Consider a free-form modular robot system {M} consisting of n modules. i Given a pair of initial configurations x from configuration space X. start and target configuration x target The goal of the reconstruction algorithm is to find an action sequence {u} within the allowed action space U. i，t This allows the reconstructed action sequence to complete x in the shortest possible time T. start Convert to x targetTo address the aforementioned reconstruction problem, we employ a multi-agent reinforcement learning approach. This approach considers two general design principles of modular robots: (1) each module has limited sensing capabilities and can only communicate with a limited range of neighboring modules; (2) modules can move in parallel and in a distributed manner to quickly complete reconstruction. To conform to these design principles, we treat the modular robot reconstruction problem as a distributed partially observable Markov decision process (Dec-POMDP). Each module {M i} cannot directly access environment state s t However, it can be observed through its observation model Z. i Obtain local observations i，t At time step t, each module performs observations based on its observations. i，t and strategy π i Select action u i，t To form a joint action u t The environment depends on probability P(s). t+1 |s t u t Transition to state s t+1 All modules share the same objective: to maximize the cumulative reward from similar discounts, i.e., the value function V. i (s t This method utilizes a neural network to approximate the value function V. i，t To handle the high-dimensional configuration space and motion space of free-form modular robots.

[0090] Step 2: Observation and Action Space Setup

[0091] The method aims to achieve autonomous module reconstruction through multi-agent reinforcement learning. This requires appropriately setting the perceptual and motion parameters of each module to avoid redundant information and improve training speed. Each module's observations include ontology perception and environmental perception. Ontology perception includes the position vector P. i Velocity vector v i Altruism factor α i and active-passive connection relationship δ i，I\{i} Each module can also sense the position P of other modules j within a radius d. j and speed v j Although all modules share the same policy network parameters when executed in a step-by-step manner, each module acts differently based on its unique observations.

[0092] This method directly incorporates the connection / disconnection of the active connector into the module's action space, which can be represented by a tuple <σ1, σ2, δ. link The > symbol represents the linking action of each module. linkThis means that only modules in contact with each other can perform the connection or disconnection action. The torques of the two motion motors are σ1 and σ2, driving the modules to move in any direction on the plane. Furthermore, the maximum moving speed v of the modules is artificially limited. max .

[0093] Step 3: Design the reward function

[0094] To address the aforementioned reconstruction problem, we designed a reward function shared by all modules to train the modules to learn the reconstruction strategy:

[0095]

[0096] The first term is the topological reward, the second is the geometric reward, and the third is a small time reward, determined by the hyperparameter c. p c q and c t Balance. This encourages modules to rapidly approach the target configuration in both geometric and topological connectivity directions.

[0097] Topology rewards are designed based on local topology information of the configuration. For each module M i Its connection relationship with other n-1 modules can be expressed by δ. i，j Let j belong to I{i}. To incentivize modules to take actions that bring their local topological connections closer to their local topological relationships in the target configuration, the topological reward is designed as follows:

[0098]

[0099] The two terms before and after the molecular term are time step t-1 and time t, respectively. i The distance between the local topology and the target topology. This difference form avoids the sparsity of the reward. The denominator is only related to the module M in the initial configuration and the target configuration. i The reward is normalized as a normalization factor related to the topology of the local connections, thus avoiding repeated tuning of the hyperparameters.

[0100] Utilizing connectivity can only promote topological similarity of configurations. To ensure the overall configuration maintains shape consistency with the target configuration, further geometric information needs to be introduced. The geometric reward design is as follows:

[0101]

[0102] The first term is the displacement vector of the module. For P i,t-1 Starting from point P i,t P i,G (M in the target configuration) iThe position vector is the angle between the two vectors at the endpoints. Therefore, the geometric reward measures the effective distance the module moves to the target position in the correct direction at each time step.

[0103] Step 4: Altruistic Proximity Strategy Optimization

[0104] This method addresses the reconfiguration problem of free-form modular robots from the perspective of hybrid motivation, enabling each module to learn coordination and conflict avoidance. Inspired by altruism scales, the method introduces altruistic mechanisms into multi-agent reinforcement learning to achieve cooperation between modules, avoid conflict, and thus successfully and effectively complete the reconfiguration.

[0105] First, the method considers how to measure the gains of others. The neighbors of a module and the relationships between them change dynamically during the reconstruction process. The mean-field reward for each module is defined based on the mean-field concept:

[0106]

[0107] Where, N i，t ={j:||P it -P jt ||≤d} is a dynamically changing set containing M i The indices of other modules within a radius of d. Thus, the mean field reward represents the reward in module M. i The average reward of neighboring modules that are dynamically changing. Each module M i Only the impact on this single reward needs to be considered, rather than the impact of the action on each specific module individually. This reduces interaction costs and ensures the scalability of the method.

[0108] Then, we introduce the sociological mechanism of the altruism scale to measure the social tendency of each module toward the benefit of others, and to form altruistic rewards:

[0109]

[0110] Where, α i It is an altruistic factor defined on (-1,1), which measures each module's attitude toward the gains of others. Vector α is α i The set represents the distribution of altruistic tendency within the entire population. The altruism factor α... i It is each module M i One of its personalized attributes, heterogeneity and diversity, is that it can improve overall performance.

[0111] The first stage of the altruistic proximal policy optimization method trains the PPO (proximal policy optimization) through CTDE (centralized training and distributed execution) to maximize the altruistic reward R.AS :

[0112]

[0113] The altruistic advantage function is:

[0114] There exists an optimal α. i The distribution α* optimizes the model performance during the first stage of training. However, manually designing each αi is impractical. Therefore, based on meta-reinforcement learning, the method includes another layer of training to optimize appropriate personalized α for each module. i The second-stage optimization objective of the altruistic proximal strategy optimization method is:

[0115]

[0116] By using the chain rule and Taylor series derivation, the gradient of the above optimization objective is rewritten with personalized altruistic factors, so that the personalized altruistic factors of all modules can be optimized during training:

[0117]

[0118] Thus, as Figure 2 As shown, during training, the altruistic proximal policy optimization method performs gradient descent on the event scale according to the above formula to optimize the personalized altruism factor of each module, and performs gradient descent on the time step scale, while simultaneously increasing the altruistic reward R. AS maximize.

[0119] Step 5: Distributed Asynchronous Termination Criteria

[0120] Reconfiguration requires explicit termination criteria so that modular robotic systems can continue performing tasks using the target configuration. In practical deployments of modular robots, directly specifying the termination time step is not suitable. Furthermore, due to the limited communication and observation capabilities of the modules, it is difficult to obtain real-time global information about the configuration.

[0121] All modules operate under the same distributed asynchronous termination criterion. Each module determines whether its local connectivity matches the target configuration based on its local observations. When a match is found and the module receives the in-place signal Ω from all its sub-modules (connected by the current module), it sends the in-place signal Ω to its parent module (connected to the current module). In this way, the in-place signals are progressively distributed from the leaf nodes to the root node (modules without a parent node). The root node then passes the termination signal Ω2 to its child nodes, and this transmission continues level by level. Each module receiving the Ω2 signal immediately terminates the reconfiguration process. This asynchronous termination criterion avoids reliance on real-time updates to global configuration information.

Claims

1. A free-form modular robotic self-reconfiguration method, characterized by, Comprising the following steps: Step 1: Constructing the abstract model of free-form modular robot; Each module of the free-form modular robot is regarded as a standard sphere, including active connectors, passive connectors and sensing devices, and the module is driven by a motion motor; the spherical shell of the module is used as a passive connector; the motion motor of the module has continuous motion capability in two-dimensional space, and the module can move on the surface of the connected module; the modules can communicate with each other; Consider a modular robot system {M} i }, consisting of n modules, given a pair of initial configurations x from configuration space X. start and target configuration x target The goal of the self-reconstruction method is to find an action sequence {u} within the allowed action space U. i，t This allows the reconstructed action sequence to complete x in the shortest possible time T. start Convert to x target Any given configuration x consists of the location information Pi of each module and the topological connection information δ of all modules. i，j The sole determining factor is that, during the reconstruction process, the action space U contains the actions that ensure the motor torque and speed vectors of each module satisfy the corresponding motion constraints under the current conditions. Step 2: Observation and action space setting; The observation of each module includes body perception and environment perception; the body perception includes position vector P i , velocity vector v i , altruism factor α i and active-passive connection relationship δ i，I\{i} ; each module can also perceive the position P j and velocity v j of other modules j within the radius d; The connection or disconnection of the active connector is incorporated into the action space of the module, which is represented by a tuple <σ1, σ2, δ link > ; the linking action of each module is represented by δ link , and it is assumed that only the modules in contact with each other can perform the connection or disconnection action; the torques of the two movement motors are σ1 and σ2, which are used to drive the module to move in any direction on the plane; the maximum movement speed of the module is limited to v max ; Step 3: Reward function design; A reward function shared by all modules is designed to train the modules to learn the reconstruction strategy: wherein the first term on the right is the topological reward, the second term is the geometric reward, and the third term is the temporal reward, by hyperparameters c p , c q , and c t balance; For each module M i The connection relationship with other n1 modules is denoted by δ i，j , where j belongs to I\{i}; the topology reward is designed as follows: wherein the right hand side of the equation is the position of M at time step t i the distance between the local topology of M and the target topology; Further geometric information is introduced, and the geometric reward is designed as follows: Wherein, the first term on the right side of the formula is the displacement vector of the module; For P i,t-1 Starting from point P i,t P i,G The angle between the two vectors at the endpoints; the geometric reward measures the effective distance the module moves to the target position in the correct direction at each time step; Step 4: Altruistic proximal policy optimization; Introduce altruistic mechanism in multi-agent reinforcement learning to realize the cooperation between modules; Based on the idea of mean field, the mean field reward of each module is defined: where N i，t = {j: ||P it - P jt ||≤ d} is a dynamically changing set containing indices of other modules within a range of radius d from module M i ; the mean field reward represents the average reward of neighboring modules dynamically changing around module M i ; Introduce altruistic scale to form altruistic reward: Where, α i It is an altruistic factor defined on (-1,1) to measure each module's attitude toward the gains of others; the vector α is α i The set represents the distribution of altruistic tendencies within the entire population; the altruism factor α i It is each module M i A personalized attribute; The first phase of the altruism proximal policy optimization method trains a proximal policy optimization PPO to maximize an altruism reward R in a way that concentrates training and disperses execution of the CTDE AS : where the altruistic advantage function is: There is an optimal a i Distribution a * that makes the model trained in the first phase of the method perform best; Based on meta-reinforcement learning, optimize appropriate individualized a for each module i The optimization objective of the second stage of the altruistic proximal policy optimization method is to Through the chain rule and Taylor series, the gradient of the above optimization objective is rewritten with personalized altruistic factors, so that the personalized altruistic factors of all modules can be optimized in training: The altruism proximal policy optimization method performs gradient descent on the event scale according to the above formula to optimize the altruism factor of each module, and performs gradient descent on the time step scale while maximizing the altruism reward R AS ; Step 5: Distributed asynchronous termination criterion; All modules run the same distributed asynchronous termination criterion; each module judges whether its local connection relationship is consistent with the target configuration according to its local observation, and when it is matched and consistent and receives the just-in-place signal Ω of all sub-modules, it sends the just-in-place signal Ω to the parent module; in this way, the just-in-place signal will be distributed from the leaf node to the root node; then, the root node transmits the termination signal Ω2 to the child node and transmits it level by level; each module receiving the Ω2 signal will immediately terminate the reconstruction process.

Images