Spatial multi-arm robot path planning method and system based on reinforcement learning

Abstract

The invention provides a space multi-arm robot path planning method and system based on reinforcement learning, and belongs to the field of space robot path planning. A simulation model of a space multi-arm robot and a path planning task environment is constructed; performing off-line task training on the space multi-arm robot by adopting a soft actor commentator algorithm of a self-adaptive entropy adjustment mechanism to obtain an optimal control strategy; the soft actor commentator algorithm comprises the steps that an objective function accumulates rewards by maximizing expectation, an entropy regular term is introduced to maximize strategy entropy, a weight coefficient of the strategy entropy is dynamically adjusted through a self-adaptive mechanism, and network parameters are softly updated through a Polyak average method; and in the simulation model, when the optimal control strategy converges, outputting the trained optimal control strategy. According to the method provided by the invention, the system can autonomously learn a positioning task capable of inhibiting the disturbance of the floating base and accurately finishing path planning through offline task training without an accurate model.

Description

Technical Field

[0001] This invention belongs to the field of path planning for space robots, specifically relating to a path planning method and system for space multi-armed robots based on reinforcement learning. Background Technology

[0002] A space robot is a key piece of equipment used to replace or assist astronauts in operations in the space environment. It typically consists of a base and one or more robotic arms mounted on that base, and can switch between three modes: free flight, free float, and base-controlled operation. For energy conservation, space robots generally operate in free float mode, where the base's posture is uncontrolled. However, due to the inherent dynamic coupling characteristics of space robots, in free float mode, the movement of the robotic arms can interfere with the base's posture, and changes in the base's posture can affect the stability and operational accuracy of the robotic arms.

[0003] To reduce the disturbance to the base caused by the movement of the robotic arm while ensuring the accuracy of the end effector positioning, researchers have conducted studies. For example, a counterweight compensation strategy is adopted, in which one arm performs the task as the task arm, and the other arm acts as a counterweight arm to counteract the disturbance to the base; a trajectory planning is performed by establishing an equivalent kinematic model of the center of mass to simultaneously stabilize the base attitude and the center of mass position; based on optimization theory, the trajectory planning problem is transformed into a constrained optimization problem, and an intelligent optimization algorithm is used to solve it.

[0004] However, the above methods are mostly for path planning of single-arm space robots and all rely on accurate mathematical models. But in general, it takes a long time to build an accurate mathematical model and has poor versatility. Once the robot model is changed, it is necessary to rebuild an accurate mathematical model. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a path planning method for spatial multi-arm robots based on reinforcement learning. This method enables the system to autonomously learn a localization task that can both suppress disturbances of the floating base and accurately complete path planning through offline task training without the need for an accurate model.

[0006] In a first aspect, the present invention provides a path planning method for a spatial multi-arm robot based on reinforcement learning, comprising:

[0007] Step S1: Construct a simulation model of the spatial multi-arm robot and the path planning task environment;

[0008] Step S2: In the simulation model, define the reinforcement learning problem and design the constraints. The reinforcement learning problem includes: state space, action space and reward function.

[0009] Step S3: In the simulation model, initialize the pose and target position of the spatial multi-arm robot;

[0010] Step S4: The soft actor critic algorithm with adaptive entropy adjustment mechanism is used to train the space multi-arm robot for offline tasks to obtain the optimal control strategy; the soft actor critic algorithm with adaptive entropy adjustment mechanism includes: the objective function maximizes the expected cumulative reward, and an entropy regularization term is introduced to maximize the policy entropy. The weight coefficients of the policy entropy are dynamically adjusted through an adaptive mechanism. The network parameters of the dual Critic network in the soft actor critic algorithm with adaptive entropy adjustment mechanism are softly updated using the Polyak averaging method.

[0011] Step S5: In the simulation model, continuously monitor the convergence of the optimal control strategy during the offline task training process. If the optimal control strategy does not converge to the preset effect, then fine-tune the hyperparameters in the soft actor critic algorithm of the adaptive entropy adjustment mechanism and repeat step S4. If the optimal control strategy converges to the preset effect, proceed to step S6.

[0012] Step S6: Output the optimal control strategy trained, and control the movement of the spatial multi-arm robot according to the optimal control strategy.

[0013] The constraints include: limitations on the range of motion of each joint angle of the multi-arm robot;

[0014] The state space includes: the position of the floating base of the multi-arm robot, the attitude of the floating base of the multi-arm robot, the angular velocity of the floating base of the multi-arm robot, the joint angles of the left and right robotic arms, the joint angular velocities of the left and right robotic arms, the positions of the end effectors of the left and right robotic arms, and the target position.

[0015] The motion space includes: the joint angle increments of each joint of the left and right robotic arms;

[0016] The reward function is the sum of the end-task reward, the floating base disturbance penalty, the motion smoothing penalty, and the success reward.

[0017] The end-task reward is constructed using a combination of quadratic and logarithmic errors, and the calculation formula is as follows:

[0018] ;

[0019] in, , This represents the Euclidean distance between the positions of the end effectors of the left and right robotic arms and the target position. The positions of the end effectors of the left and right robotic arms. For the target location, It can be either the left or right robotic arm. For the quadratic term error, For logarithmic error, As a reward for the final task, The weight of the quadratic term error. As the weight of the logarithmic error, This is the logarithmic protection constant.

[0020] The floating base disturbance penalty penalizes changes in the attitude and angular velocity of the floating base of the multi-arm robot, and the calculation formula is as follows:

[0021] ;

[0022] in, Penalty for disturbance of floating base The orientation of the floating base of the multi-armed robot. This is the initial base orientation. Let be the angular velocity of the floating base of the multi-armed robot. As the weight of the attitude change, This represents the weight of the change in angular velocity.

[0023] The motion smoothing penalty penalizes the changes in the joint angles of the left and right robotic arms at adjacent time points, and the calculation formula is as follows:

[0024] ;

[0025] in, Penalty for smooth movement The weight of motion smoothness penalty The joint angle of the left robotic arm. This is the joint angle of the right robotic arm.

[0026] The success reward is given when the ends of the left and right robotic arms reach the target position simultaneously and the error is less than a preset error threshold. The calculation formula is as follows:

[0027] ;

[0028] in, As a reward for success, To ensure the left robotic arm's end effector reaches the target position, For the right robotic arm end effector to reach the target position, This is the preset error threshold.

[0029] The objective function maximizes the expected cumulative reward by introducing an entropy regularization term to maximize the policy entropy, as shown in the following formula:

[0030] ;

[0031] in, Indicates the state Lower policy entropy, The weighting coefficients are the policy entropy. This means finding the target strategy that maximizes the sum of expected cumulative return and strategy entropy. , The strategy to be optimized At time step state, Indicates at time step The action, Indicates from time step Start, State Next action The expected value of the accumulated discount rewards. For strategy Steady-state distribution of induced state-action pairs.

[0032] The weight coefficients of the strategy entropy are dynamically adjusted through an adaptive mechanism, and the calculation formula is:

[0033] ;

[0034] in, The weighting coefficients are the adjusted policy entropy. The weighting coefficients are the policy entropy. Indicates the state Lower policy entropy, Let the target entropy be half the dimension of the action space. This is the learning rate.

[0035] The network parameters of the dual-critic network in the soft actor / critic algorithm of the adaptive entropy adjustment mechanism are softly updated using the Polyak averaging method, including:

[0036] The soft actor critic algorithm with adaptive entropy adjustment mechanism uses two independently initialized and identical Critic networks. It selects the minimum value of the target network parameters in the two Critic networks and uses the Polyak averaging method to softly update the minimum value of the target network parameters.

[0037] Secondly, the present invention also provides a path planning system for a spatial multi-arm robot based on reinforcement learning, comprising:

[0038] The simulation model building module is used to build simulation models of the spatial multi-arm robot and the path planning task environment;

[0039] The parameter condition definition module is used to define the reinforcement learning problem and design constraints in the simulation model. The reinforcement learning problem includes: state space, action space and reward function.

[0040] The model initialization module is used to initialize the pose and target position of the spatial multi-arm robot in the simulation model.

[0041] The parameter optimization module is used to train the space multi-arm robot for offline tasks using a soft actor critic algorithm with an adaptive entropy adjustment mechanism to obtain the optimal control strategy. The soft actor critic algorithm with an adaptive entropy adjustment mechanism includes: maximizing the expected cumulative reward by maximizing the policy entropy by introducing an entropy regularization term; dynamically adjusting the weight coefficients of the policy entropy through an adaptive mechanism; and using the Polyak averaging method to softly update the network parameters of the dual Critic network in the soft actor critic algorithm with an adaptive entropy adjustment mechanism.

[0042] The offline simulation module is used to continuously monitor the convergence of the optimal control strategy during the offline task training process in the simulation model. If the optimal control strategy does not converge to the preset effect, the hyperparameters in the soft actor critic algorithm of the adaptive entropy adjustment mechanism are tuned and the step parameter optimization module is repeated. If the optimal control strategy converges to the preset effect, the module is switched to the execution strategy output module.

[0043] The strategy output module is used to output the optimal control strategy trained, and to control the movement of the spatial multi-arm robot according to the optimal control strategy.

[0044] Beneficial effects:

[0045] This application proposes a path planning method and system for spatial multi-arm robots based on reinforcement learning. The method employs reinforcement learning, avoiding reliance on precise dynamic modeling. By designing a composite reward function, this invention effectively achieves end-effector localization while reducing disturbance to the base. Furthermore, it utilizes a soft actor-critic algorithm with an adaptive entropy adjustment mechanism. During training, the algorithm autonomously adjusts the entropy weight coefficients to effectively and dynamically balance the relationship between exploration and utilization. The trained strategy can be directly used for online control, demonstrating good engineering practicality. Attached Figure Description

[0046] Figure 1 Flowchart of a path planning method for a spatial multi-arm robot based on reinforcement learning according to an embodiment of the present invention;

[0047] Figure 2 A schematic flowchart of a path planning method for a spatial dual-arm robot based on reinforcement learning according to an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the visualization model of an embodiment of the present invention;

[0049] Figure 4A schematic diagram of the reward value change curve in an embodiment of the present invention;

[0050] Figure 5 A schematic diagram illustrating the task execution process of a space dual-arm robot according to an embodiment of the present invention; wherein, Figure 5 (a) is the initial position of the ends of the arms relative to the target; 5(b) is the position of the ends of the arms approaching the target; 5(c) is the position of the ends of the arms nearing the target; 5(d) is the position of the ends of the arms reaching the target.

[0051] Figure 6 A schematic diagram of a path planning system for a spatial multi-arm robot based on reinforcement learning according to an embodiment of the present invention. Detailed Implementation

[0052] The specific implementation methods of this application will be further described in detail below with reference to the accompanying drawings and embodiments.

[0053] Example 1:

[0054] This embodiment proposes a path planning method for spatial multi-arm robots based on reinforcement learning, such as... Figure 1 , Figure 2 As shown, it includes:

[0055] Step S1: Construct a simulation model of the spatial multi-arm robot and the path planning task environment;

[0056] In this embodiment, a simulation model of a spatial dual-arm robot and path planning task environment is built in the MuJoCo simulator. Specifically, the spatial dual-arm robot model consists of two 6-DOF robotic arms and a cube-shaped floating base. The arms and the floating base have a dynamic coupling relationship, and the simulation environment is set to weightlessness.

[0057] In the specific implementation, the space dual-arm robot and its mission environment were modeled in the MuJoCo simulator. The two arms were selected as the 6-DOF UR5 manipulators, with parameter settings identical to those of a real UR5 manipulator. The base was cubic in shape and rigidly connected to the two arms. The entire system operated in a weightless environment. Detailed model parameters and the overall visualization model in MuJoCo are shown in Tables 1 and 2, respectively. Figure 3 As shown.

[0058] Table 1. Parameters of the space dual-arm robot model;

[0059] ;

[0060] Step S2: In the simulation model, define the reinforcement learning problem and design the constraints. The reinforcement learning problem includes: state space, action space and reward function.

[0061] The state space includes: the position of the floating base of the multi-arm robot, the attitude of the floating base of the multi-arm robot, the angular velocity of the floating base of the multi-arm robot, the joint angles of the left and right robotic arms, the joint angular velocities of the left and right robotic arms, the positions of the end effectors of the left and right robotic arms, and the target position.

[0062] The motion space includes: the joint angle increments of each joint of the left and right robotic arms;

[0063] The reward function is the sum of the end-task reward, the floating base disturbance penalty, the motion smoothing penalty, and the success reward.

[0064] In this embodiment, the reinforcement learning problem framework is a paradigm for describing and modeling how an agent learns to make optimal decisions through interaction with its environment. Its core elements include the definitions of state space, action space, and reward function. The state space is defined as follows:

[0065] ;

[0066] Where S is the state space. For the location of the floating base, The attitude of the floating base, Angular velocity of the floating base, Used to characterize the disturbance effect of the floating base caused by the motion of the multi-arm. The joint angles of the left and right robotic arms. The angular velocities of the left and right robotic arm joints. The positions of the end effectors of the left and right robotic arms. This refers to the location of the target positioning point. In the... At each decision moment, the system state is denoted as... .

[0067] The action space is defined as follows:

[0068] In a free-floating state, the space robot base does not apply active control; the system's motion is achieved solely through the joints of two 6-DOF robotic arms. Its motion space is defined as follows:

[0069] ;

[0070] in, This represents the increment of the joint angles of the left robotic arm. For the increments of the joint angles of the right robotic arm, the agent in the first... The action output at each time step is denoted as ;

[0071] The end-point task reward, designed to encourage the multi-arm end effector to continuously approach the target position to meet high-precision positioning requirements, employs a reward function that combines quadratic error and logarithmic error, calculated as follows:

[0072] ;

[0073] in, , This represents the Euclidean distance between the positions of the end effectors of the left and right robotic arms and the target position. The positions of the end effectors of the left and right robotic arms. For the target location, It can be either the left or right robotic arm. This is a quadratic error term, used to provide a stable gradient in regions with large errors, guiding the terminal to quickly approach the target. To reduce the logarithmic error, by... To increase the reward value and achieve higher sensitivity when the error is small. As a reward for the final task, The weight of the quadratic term error. As the weight of the logarithmic error, As a logarithmic protection constant, this embodiment .

[0074] The floating base disturbance penalty is implemented to suppress the interference of multi-arm movements on the floating base. Penalties are imposed on changes in the base's attitude and angular velocity to guide the agent to reduce disturbances to the base during multi-arm cooperative movements. The penalties are applied to changes in the floating base's attitude and angular velocity, calculated as follows:

[0075] ;

[0076] in, Penalty for disturbance of floating base The orientation of the floating base of the multi-armed robot. This is the initial base orientation. Let be the angular velocity of the floating base of the multi-armed robot. As the weight of the attitude change, This represents the weight of the change in angular velocity.

[0077] The motion smoothing penalty is implemented to ensure a relatively smooth robotic arm trajectory. It penalizes rapid changes in joint angles at adjacent time points, preventing drastic fluctuations in joint angles and improving the continuity and stability of the control trajectory. The penalty applies to changes in the joint angles of the left and right robotic arms at adjacent time points, calculated as follows:

[0078] ;

[0079] in, Penalty for smooth movement Weighting of the action smoothing penalty.

[0080] The success reward is given when the ends of the left and right robotic arms reach the target position simultaneously and the error is less than a preset threshold. To accelerate strategy convergence and improve task completion efficiency, a success reward is given when the ends of the left and right robotic arms reach the target position simultaneously and the error is lower than a preset error threshold. The calculation formula is as follows:

[0081] ;

[0082] in, As a reward for success, To ensure the left robotic arm's end effector reaches the target position, For the right robotic arm end effector to reach the target position, This is the preset error threshold.

[0083] The constraints include: the range of motion of each joint angle of the multi-arm robot, calculated as follows:

[0084] ;

[0085] in, This represents the upper limit of the parameters for the angles of each joint of the left and right robotic arms. This represents the upper limit of the angular velocity parameters for each joint of the left and right robotic arms. This indicates the angles of each joint on the left and right robotic arms. This represents the angular velocity of each joint of the left and right robotic arms, where i=1 represents the left robotic arm and i=2 represents the right robotic arm.

[0086] Step S3: In the simulation model, initialize the pose and target position of the spatial multi-arm robot;

[0087] In this embodiment, the center position of the base is The offset of the two robotic arms relative to the base is The target location is .

[0088] Step S4: The soft actor critic algorithm with adaptive entropy adjustment mechanism is used to train the space multi-arm robot for offline tasks to obtain the optimal control strategy; the soft actor critic algorithm with adaptive entropy adjustment mechanism includes: the objective function maximizes the expected cumulative reward, and an entropy regularization term is introduced to maximize the policy entropy. The weight coefficients of the policy entropy are dynamically adjusted through an adaptive mechanism. The network parameters of the dual Critic network in the soft actor critic algorithm with adaptive entropy adjustment mechanism are softly updated using the Polyak averaging method.

[0089] Step S5: In the simulation model, continuously monitor the convergence of the optimal control strategy during the offline task training process. If the optimal control strategy does not converge to the preset effect, then fine-tune the hyperparameters in the soft actor critic algorithm of the adaptive entropy adjustment mechanism and repeat step S4. If the optimal control strategy converges to the preset effect, proceed to step S6.

[0090] Step S6: Output the optimal control strategy trained, and control the movement of the spatial multi-arm robot according to the optimal control strategy.

[0091] In this embodiment, a soft actor-critic algorithm with an adaptive entropy adjustment mechanism is used to train the robot system. This algorithm maximizes the expected cumulative reward while introducing an entropy regularization term to maximize the policy entropy, thereby obtaining the optimal policy for controlling the cooperative motion of the multi-armed robot. Specifically, the algorithm trains the policy by optimizing the following objective function:

[0092] ;

[0093] in, Indicates the state The policy entropy below, The weighting coefficients are the policy entropy. This means finding the target strategy that maximizes the sum of expected cumulative return and strategy entropy. , The strategy to be optimized Indicates at time step state, Indicates at time step The action, Indicates from time step Start, State Next action The expected value of the accumulated discount rewards. For strategy Steady-state distribution of induced state-action pairs.

[0094] The weight coefficients of the strategy entropy are dynamically adjusted through an adaptive mechanism, and the calculation formula is:

[0095] ;

[0096] in, The weighting coefficients are the adjusted policy entropy. Let the target entropy be half the dimension of the action space. This is the learning rate.

[0097] In this embodiment, the target entropy Set the learning rate to half the action space dimension. Set it to 0.001.

[0098] The network parameters of the dual-critic network in the soft actor / critic algorithm of the adaptive entropy adjustment mechanism are softly updated using the Polyak averaging method, including:

[0099] The soft actor critic algorithm with adaptive entropy adjustment mechanism uses two independently initialized and identical Critic networks. It selects the minimum value of the target network parameters in the two Critic networks and uses the Polyak averaging method to softly update the minimum value of the target network parameters.

[0100] In this embodiment, the soft actor-critic algorithm with adaptive entropy adjustment mechanism belongs to the deep reinforcement learning structure of the Actor-Critic architecture. The Actor network generates actions and interacts with the environment, while the Critic network evaluates the value of the Actor's output actions to guide policy optimization and updates. Based on the maximum entropy Bellman equation, the policy entropy term can be considered as part of the value function, and the parameters of the Critic network are iteratively updated using a temporal difference method. Its loss function is defined as follows:

[0101] ;

[0102] Let be the loss function of the Critic network. These are the parameters of the Critic network, where, For the current Critic network in state and actions The estimated Q value, For the target Q value part, Indicates at time step From the state Next action The instant rewards received This is a discount factor used to weigh the importance of future rewards. For the Q-value estimation of the next state-action pair, For entropy weighting coefficients, For policy function In the next state Select action The probability of.

[0103] The described soft actor-critic algorithm employs a dual-critic network structure and a target network mechanism. Specifically, it uses two independently initialized critic networks with identical structures. and When calculating the target Q-value, the minimum value of the two target output networks is selected, i.e.:

[0104] ;

[0105] in, For the first Each Critic network estimates the Q-value for the next state. For the Actor network in state The output is a deterministic action.

[0106] Target network parameters The Polyak averaging method is used for soft updates, and its update formula is as follows:

[0107] ;

[0108] in, These are the target network parameters after the soft update. This represents the minimum value of the target network parameters in the dual-Critic network structure. The soft update coefficient is set to 0.005 in this embodiment.

[0109] The update objective of the policy network is to minimize the KL divergence between the policy distribution and the optimal distribution induced by the Q-value. Its loss function can be expressed as:

[0110] ;

[0111] in, Let Actor be the objective function of the Actor network. These are the parameters of the Actor network. This indicates the buffer from the experience replay. Sampling status and based on the current strategy Generated actions Find the expected value of the joint distribution formed. For policy function In the current state Select action The probability of.

[0112] During training, state transition samples are generated through the interaction between the agent and the environment. The samples are stored in the experience replay buffer. During the parameter update phase, a batch of samples is randomly sampled from the experience replay buffer for updating the Actor and Critic network parameters.

[0113] Continuously monitor the policy convergence during training until the policy achieves the expected result. If the policy does not converge to the expected result, fine-tune the hyperparameters. The specific hyperparameters used are shown in Table 2.

[0114] Table 2. Hyperparameters of the soft actor-critic algorithm for adaptive entropy adjustment mechanism;

[0115] ;

[0116] Finally, the optimal training policy is output to control the robot to move according to that policy.

[0117] The reward value change curve of the training process of the soft actor-critic algorithm for the spatial dual-arm robot embodiment obtained using the method of this embodiment is shown below. Figure 4 As shown, the process of a space dual-arm robot performing a task is as follows: Figure 5 As shown, Figure 5 (a) shows the initial position of the arms' ends relative to the target; (b) shows the position of the arms' ends approaching the target; (c) shows the position of the arms' ends near the target; and (d) shows the position of the arms' ends reaching the target. It can be seen that the method provided in this embodiment can complete path planning for the arms under conditions of small perturbations in the base posture.

[0118] Example 2:

[0119] This embodiment also provides a path planning system for a spatial multi-arm robot based on reinforcement learning, such as... Figure 6 As shown, it includes: a simulation model construction module, a parameter condition definition module, a model initialization module, a parameter optimization module, an offline simulation module, and a strategy output module. The simulation model construction module is connected to the parameter condition definition module, the parameter condition definition module is connected to the model initialization module, the model initialization module is connected to the parameter optimization module, the parameter optimization module is connected to the offline simulation module, and the offline simulation module is connected to both the strategy output module and the parameter optimization module.

[0120] The simulation model building module is used to build simulation models of the spatial multi-arm robot and the path planning task environment;

[0121] The parameter condition definition module is used to define the reinforcement learning problem and design constraints in the simulation model. The reinforcement learning problem includes: state space, action space and reward function.

[0122] The model initialization module is used to initialize the pose and target position of the spatial multi-arm robot in the simulation model.

[0123] The parameter optimization module is used to train the space multi-arm robot for offline tasks using a soft actor critic algorithm with an adaptive entropy adjustment mechanism to obtain the optimal control strategy. The soft actor critic algorithm with an adaptive entropy adjustment mechanism includes: maximizing the expected cumulative reward by maximizing the policy entropy by introducing an entropy regularization term; dynamically adjusting the weight coefficients of the policy entropy through an adaptive mechanism; and using the Polyak averaging method to softly update the network parameters of the dual Critic network in the soft actor critic algorithm with an adaptive entropy adjustment mechanism.

[0124] The offline simulation module is used to continuously monitor the convergence of the optimal control strategy during the offline task training process in the simulation model. If the optimal control strategy does not converge to the preset effect, the hyperparameters in the soft actor critic algorithm of the adaptive entropy adjustment mechanism are tuned and the step parameter optimization module is repeated. If the optimal control strategy converges to the preset effect, the module is switched to the execution strategy output module.

[0125] The strategy output module is used to output the optimal control strategy trained, and to control the movement of the spatial multi-arm robot according to the optimal control strategy.

[0126] Example 3:

[0127] This embodiment proposes an electronic device, including: one or more processors, and a memory, wherein the memory is used to store instructions, and when the instructions are executed by the one or more processors, the one or more processors execute the aforementioned reinforcement learning-based spatial multi-arm robot path planning method.

[0128] The electronic device can be a mobile phone, computer, or tablet computer, etc., and includes a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements a reinforcement learning-based path planning method for a spatial multi-arm robot as described in the embodiments. It is understood that the electronic device may also include input / output (I / O) interfaces and communication components.

[0129] The processor is used to execute all or part of the steps in the reinforcement learning-based spatial multi-arm robot path planning method described in the above embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0130] The processor can be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute the reinforcement learning-based spatial multi-arm robot path planning method described in the above embodiments.

[0131] Example 4:

[0132] This embodiment proposes a computer-readable storage medium that stores executable instructions. When these instructions are executed, if they are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0133] The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the reinforcement learning-based spatial multi-arm robot path planning method described in the various embodiments of this application.

[0134] The aforementioned storage media include: flash memory, hard disk, multimedia card, card-type memory (e.g., SD (Secure Digital Memory Card) or DX (Memory Data Register, MDR) memory), random access memory (RAM), static random-access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, disk, optical disk, server, APP (Application) application store, and other media capable of storing program verification codes. These media store computer programs, which, when executed by a processor, can implement the various steps of the reinforcement learning-based spatial multi-arm robot path planning method described above.

[0135] Example 5:

[0136] This embodiment proposes a computer program product, including a computer program or instructions, which, when executed by a processor, implements the aforementioned reinforcement learning-based path planning method for a spatial multi-arm robot.

[0137] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a computer program product.

[0138] The various embodiments in this application are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0139] The scope of protection of this application is not limited to the embodiments described above. Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from the scope and spirit of this disclosure. If such modifications and variations fall within the scope of equivalent technology of this disclosure, then the intent of this disclosure also includes such modifications and variations.

Claims

1. A path planning method for a spatial multi-arm robot based on reinforcement learning, characterized in that, include: Step S1: Construct a simulation model of the spatial multi-arm robot and the path planning task environment; Step S2: In the simulation model, define the reinforcement learning problem and design constraints. The reinforcement learning problem includes: state space, action space and reward function. Step S3: In the simulation model, initialize the pose and target position of the spatial multi-arm robot; Step S4: The soft actor critic algorithm with adaptive entropy adjustment mechanism is used to train the space multi-arm robot for offline tasks to obtain the optimal control strategy; the soft actor critic algorithm with adaptive entropy adjustment mechanism includes: the objective function maximizes the expected cumulative reward, and an entropy regularization term is introduced to maximize the policy entropy. The weight coefficients of the policy entropy are dynamically adjusted through an adaptive mechanism. The network parameters of the dual Critic network in the soft actor critic algorithm with adaptive entropy adjustment mechanism are softly updated using the Polyak averaging method. Step S5: In the simulation model, continuously monitor the convergence of the optimal control strategy during the offline task training process. If the optimal control strategy does not converge to the preset effect, then fine-tune the hyperparameters in the soft actor critic algorithm of the adaptive entropy adjustment mechanism and repeat step S4. If the optimal control strategy converges to the preset effect, proceed to step S6. Step S6: Output the optimal control strategy trained, and control the movement of the spatial multi-arm robot according to the optimal control strategy.

2. The path planning method for a spatial multi-arm robot based on reinforcement learning according to claim 1, characterized in that, The constraints include: limitations on the range of motion of each joint angle of the multi-arm robot; The state space includes: the position of the floating base of the multi-arm robot, the attitude of the floating base of the multi-arm robot, the angular velocity of the floating base of the multi-arm robot, the joint angles of the left and right robotic arms, the joint angular velocities of the left and right robotic arms, the positions of the end effectors of the left and right robotic arms, and the target position. The motion space includes: the joint angle increments of each joint of the left and right robotic arms; The reward function is the sum of the end-task reward, the floating base disturbance penalty, the motion smoothing penalty, and the success reward.

3. The path planning method for a spatial multi-arm robot based on reinforcement learning according to claim 2, characterized in that, The end-task reward is constructed using a combination of quadratic and logarithmic errors, and the calculation formula is as follows: ； in, , This represents the Euclidean distance between the positions of the end effectors of the left and right robotic arms and the target position. The positions of the end effectors of the left and right robotic arms. For the target location, It can be either the left or right robotic arm. For the quadratic term error, For logarithmic error, As a reward for the final task, The weight of the quadratic term error. As the weight of the logarithmic error, This is the logarithmic protection constant.

4. The path planning method for a spatial multi-arm robot based on reinforcement learning according to claim 2, characterized in that, The floating base disturbance penalty penalizes changes in the attitude and angular velocity of the floating base of the multi-arm robot, and the calculation formula is as follows: ； in, Penalty for disturbance of floating base The orientation of the floating base of the multi-armed robot. This is the initial base orientation. Let be the angular velocity of the floating base of the multi-armed robot. As the weight of the attitude change, This represents the weight of the change in angular velocity.

5. A path planning method for a spatial multi-arm robot based on reinforcement learning according to claim 2, characterized in that, The motion smoothing penalty penalizes the changes in the joint angles of the left and right robotic arms at adjacent time points, and the calculation formula is as follows: ； in, Penalty for smooth movement The weight of motion smoothness penalty This refers to the joint angle of the left robotic arm. This is the joint angle of the right robotic arm.

6. The path planning method for a spatial multi-arm robot based on reinforcement learning according to claim 2, characterized in that, The success reward is given when the ends of the left and right robotic arms reach the target position simultaneously and the error is less than a preset error threshold. The calculation formula is as follows: ； in, As a reward for success, To ensure the left robotic arm's end effector reaches the target position, For the right robotic arm end effector to reach the target position, This is a preset error threshold.

7. The path planning method for a spatial multi-arm robot based on reinforcement learning according to claim 1, characterized in that, The objective function maximizes the expected cumulative reward by introducing an entropy regularization term to maximize the policy entropy, as shown in the following formula: ； in, Indicates the state Lower policy entropy, The weighting coefficients are the policy entropy. This means finding the target strategy that maximizes the sum of expected cumulative return and strategy entropy. , The strategy to be optimized At time step state, Indicates at time step The action, Indicates from time step Start, State Next action The expected value of the accumulated discount rewards. For strategy Steady-state distribution of induced state-action pairs.

8. The path planning method for a spatial multi-arm robot based on reinforcement learning according to claim 1, characterized in that, The weight coefficients of the strategy entropy are dynamically adjusted through an adaptive mechanism, and the calculation formula is: ； in, The weighting coefficients are the adjusted policy entropy. The weighting coefficients are the policy entropy. Indicates the state Lower policy entropy, Let the target entropy be half the dimension of the action space. This is the learning rate.

9. A path planning method for a spatial multi-arm robot based on reinforcement learning according to claim 1, characterized in that, The network parameters of the dual-critic network in the soft actor / critic algorithm of the adaptive entropy adjustment mechanism are softly updated using the Polyak averaging method, including: The soft actor critic algorithm with adaptive entropy adjustment mechanism uses two independently initialized and identical Critic networks. It selects the minimum value of the target network parameters in the two Critic networks and uses the Polyak averaging method to softly update the minimum value of the target network parameters.

10. A path planning system for a spatial multi-arm robot based on reinforcement learning, characterized in that, include: The simulation model building module is used to build simulation models of the spatial multi-arm robot and the path planning task environment; The parameter condition definition module is used to define the reinforcement learning problem and design constraints in the simulation model. The reinforcement learning problem includes: state space, action space and reward function. The model initialization module is used to initialize the pose and target position of the spatial multi-arm robot in the simulation model. The parameter optimization module is used to train the space multi-arm robot offline using the soft actor critic algorithm with an adaptive entropy adjustment mechanism to obtain the optimal control strategy. The soft actor critic algorithm with an adaptive entropy adjustment mechanism includes: the objective function maximizes the expected cumulative reward, and an entropy regularization term is introduced to maximize the policy entropy. The weight coefficients of the policy entropy are dynamically adjusted through an adaptive mechanism. The network parameters of the dual Critic network in the soft actor critic algorithm with adaptive entropy adjustment mechanism are softly updated using the Polyak averaging method. The offline simulation module is used to continuously monitor the convergence of the optimal control strategy during the offline task training process in the simulation model. If the optimal control strategy does not converge to the preset effect, the hyperparameters in the soft actor critic algorithm of the adaptive entropy adjustment mechanism are tuned and the step parameter optimization module is repeated. If the optimal control strategy converges to the preset effect, the module is switched to the execution strategy output module. The strategy output module is used to output the optimal control strategy trained, and to control the movement of the spatial multi-arm robot according to the optimal control strategy.

Images