Space floating mechanical arm motion planning method and system based on zero reaction space and reinforcement learning

Abstract

The invention discloses a space floating mechanical arm motion planning method and system based on zero reaction space and reinforcement learning, and relates to the field of mechanical arm motion planning. The invention aims to solve the problem of base attitude disturbance caused by movement of an existing space floating mechanical arm. According to the technical scheme, the motion space of the space mechanical arm is designed to be the zero-reaction space, it is ensured that the motion of the mechanical arm does not generate unexpected disturbance on the base, and therefore the stability of the system and the task execution precision are improved. In addition, a reinforcement learning algorithm is combined, and the motion space is set as a zero-reaction motion space. The efficient search and optimization capability of reinforcement learning enables the mechanical arm to find an optimal motion strategy in a complex task space, and meanwhile, disturbance to the base is effectively avoided. The strategy can be continuously adjusted through multiple rounds of training and reinforcement learning so as to adapt to different task requirements, and it is guaranteed that when the mechanical arm executes space grabbing, maintenance and other tasks, high-precision operation can be kept, and it can be guaranteed that the base is stable. Experimental results show that by means of the method, deviation of the attitude of the base can be remarkably reduced, the stability and efficiency of task execution are improved, and a feasible solution is provided for on-orbit service tasks of the space mechanical arm in the future.

Description

Technical Field

[0001] The present invention relates to the field of robot arm motion planning, and in particular to a space floating robot arm motion planning method and system based on zero reaction space and reinforcement learning. Background Art

[0002] With the rapid development of space technology, on-orbit service missions have become an important research direction in the aerospace field. OOS missions aim to complete a series of complex on-orbit operations through actuators such as space manipulators, such as satellite maintenance, orbital refueling, parts replacement, and space debris cleaning. The implementation of these tasks can not only effectively extend the service life of satellites, but also improve the reliability and economic benefits of satellite operations. As the core execution equipment in on-orbit service missions, the motion planning of space manipulators plays a vital role in the success of missions. However, in a complex space environment, the motion planning of space manipulators faces many challenges. One of the most significant problems is the dynamic disturbance caused by the operation of the manipulator to the base. In a microgravity environment, the system as a whole is constrained by the law of conservation of momentum. The operation of the manipulator will react to the base, causing the attitude change and position drift of the base. This disturbance will not only affect the stability of the entire system, but may also reduce the operational accuracy of the mission and even lead to mission failure. Summary of the invention

[0003] To this end, the present invention proposes a space floating robot arm motion planning method based on zero reaction space and reinforcement learning, so as to overcome the problem of base attitude disturbance caused by the existing space floating robot arm motion.

[0004] The present invention proposes a space floating manipulator motion planning method based on zero reaction space and reinforcement learning, the method comprising the following steps:

[0005] Step 1: Establish a space floating robotic arm model: The space floating robotic arm model is a single-arm model and its base is in a free floating state in space;

[0006] Step 2: Establish a Markov decision process model for motion planning of a space floating robot arm, provide a mathematical description for the interaction between the space floating robot arm and the environment, and determine the five basic elements of the Markov decision process: observation space, action space, state transfer function, reward function, and discount factor:

[0007] Step 3: Establish a motion planning method for a space floating robot based on the SAC algorithm, and determine the inputs of the policy network and value network of the SAC algorithm neural network architecture, where the policy network generates the robot action distribution and the value network evaluates the value of the state action;

[0008] Step 4: Establish a reinforcement learning model for the end of the robot arm approaching the target position. This step selects appropriate observation space, action space, state transfer function and designs reward function for the task scenario of the end of the robot arm approaching the target position.

[0009] Step 5: Based on the Mujoco simulation engine, in the process of the space floating robot arm interacting with the environment, neural network training is performed based on data to obtain a trained neural network for the space floating robot arm motion planning for the task of the robot arm end approaching the target position.

[0010] Further, in step 1, the process of establishing the space floating robotic arm model is as follows:

[0011] First, the space floating robotic arm is modeled (such as Figure 1 As shown in FIG. 1 , the space floating manipulator (hereinafter referred to as the system) includes a floating base and an n-DOF manipulator. Assume that link 0 (number) is the main part of the base, link i (i=1,...,n) is the i-th link of the manipulator, and joint j is the connection link (i-1) and link i; Assume that:

[0012] (1) All components of the space manipulator are rigid;

[0013] (2) The system remains stationary in the initial state, that is, the initial linear momentum and angular momentum of the space manipulator are zero;

[0014] (3) There is no external force acting on the system;

[0015] (4) Measure the position and attitude angle of the robot and the object in the inertial coordinate system;

[0016] All vectors and inertia tensors are defined based on the inertial reference frame, and the symbol definitions are shown in the following table:

[0017] Table 2 Symbol Definition

[0018]

[0019]

[0020] The motion equation of the space floating robotic arm is expressed as:

[0021]

[0022] in

[0023]

[0024] are the Jacobian matrices of the base motion related part and the robot arm related part respectively;

[0025] Based on the assumption that the total momentum of the system is zero, the linear momentum and angular momentum P and L of the system are expressed as follows:

[0026]

[0027] in

[0028]

[0029] The derivation process of zero reaction motion space is as follows:

[0030] The system motion equation of the free-floating space manipulator is expressed as follows:

[0031]

[0032] Linear and angular velocities of the base selected by generalized coordinates And the angular velocity of the robot joint The robotic arm system is an n-degree-of-freedom robotic arm installed on a floating base;

[0033] The external force and torque on the system are zero, that is, Therefore, the motion of the robot arm is only affected by the joint torque τ, and the total momentum of the system is conserved.

[0034]

[0035] Where: represents linear momentum, represents angular momentum;

[0036] Since there is no second-order integral in the angular momentum equation, we can obtain the base angular velocity ω b and robot arm movement speed The first-order nonholonomic constraint equation represented by is as follows:

[0037]

[0038] in, is the corrected inertia matrix, is the angular momentum due to the operation of the manipulator, represents the initial angular momentum;

[0039] Due to the demand for on-orbit services in space, it is hoped that the system will be kept stable when the robotic arm is performing its mission. The zero-disturbance solution is derived from the angular momentum equation:

[0040] The initial angular momentum is The angular momentum equation when there is no disturbance b = 0 is as follows:

[0041]

[0042] The following null space solution is obtained:

[0043]

[0044] The joint motion that satisfies this equation does not disturb the posture of the base, where For any vector, the inertia matrix The null space is called the zero reaction motion space.

[0045] Furthermore, the specific process of step 2 includes:

[0046] To solve the problem of robot motion planning using reinforcement learning, the following elements need to be built into the MDP model:

[0047] (1) Observation space: In the robot motion planning task, the observation space mainly includes the robot's own information, joint angles, end position and base posture, as well as obstacle information and target information;

[0048] (2) Action space: In the robot arm motion planning task, the action space dimension matches the robot arm’s degrees of freedom; in the task setting, the action space is set to the robot arm’s zero reaction motion space and joint angular velocity space depending on whether zero reaction planning is used;

[0049] (3) State transfer function: The state transfer function describes the probability that the system will transfer to the next state after the robot arm performs a certain action in the current state. The hyperbolic tangent function is used at the end of the policy network for normalization, limiting the output value of the network to a specific range, making the output more stable and controllable.

[0050] (4) Reward function: The reward function of the robot arm motion planning is designed differently according to different tasks. In the end approach task, the reward function is mainly designed based on the position deviation between the end effector and the target point; while in the obstacle avoidance task, it is designed based on the shortest distance between the robot arm and the obstacle.

[0051] (5) Discount factor: The discount factor is used to discount the sum of future rewards and weigh short-term and long-term rewards. In robotic arm motion planning, a larger discount factor encourages consideration of long-term goals and path planning, while a smaller discount factor makes the agent pay more attention to immediate rewards. According to the specific task requirements, choosing an appropriate discount factor can find a balance between fast response and global optimality.

[0052] Furthermore, in step three, the process of establishing a space floating manipulator motion planning method based on the SAC algorithm is as follows:

[0053] The core of SAC is composed of three neural networks: Value Network, Soft Q Network and Policy Network. The training goal of the spatial floating robot motion planning based on the SAC algorithm is to obtain a motion planning policy network, which selects appropriate vectors in the action space to map to the robot action by analyzing the observation space information.

[0054] During the training process, at the beginning of each new round, the parameters will be initialized first, including the initial state of the robot arm, the target position of the end of the robot arm, the reinforcement learning parameters, and the nearby obstacle satellite model; the experience pool will record the steps of the interaction between the agent and the environment during the training process; when the agent and the environment interact for a preset number of times, a set of data will be screened from the experience pool, and the parameters of the value network and the policy network will be updated through the gradient calculation formula in the algorithm.

[0055] Furthermore, in step 4, the specific process of establishing a reinforcement learning model for the end of the robotic arm approaching the target position includes:

[0056] The mission task is set as close contact between the serving star and the served star, and the end of the single robotic arm on the serving star reaches the designated position on the served star;

[0057] Robotic arm decision model (such as Figure 2 The observation information of the robot is mainly divided into two parts: self-information and target information; self-information includes joint angles, end effector positions, and base posture information; target information includes the set approach target point information and the posture information of the satellite where the target is located;

[0058] When the action space is set to zero reaction motion space, the action is the zero reaction motion space vector; when the action space is set to joint space, the action is the joint angular velocity vector;

[0059] The position deviation between the end of the robot and the target point is represented by the symbol d p It is represented as follows:

[0060] d p =||p end -p goal ||2

[0061] The reward function design includes sparse rewards and dense rewards, as shown in the following formula:

[0062]

[0063] The dense reward function f(d p ) is set as follows:

[0064]

[0065] Furthermore, in step 5, based on the Mujoco simulation engine, neural network training is performed based on data during the interaction between the space floating manipulator and the environment. The specific process includes:

[0066] First, the Mujoco simulation engine is used to build a dynamic simulation environment for the robotic arm, including initializing the joint state of the floating robotic arm and the posture of the floating base, and setting the zero reaction force constraint condition; loading the model in the environment, setting the target position and task constraints, and providing real interactive scene support for subsequent training;

[0067] In the simulation environment, the policy network is used to control the interaction between the robot and the environment, and collect state, action, reward and environmental feedback data. The collected data includes the position, speed, control input and target distance of the robot, which are used to train the reinforcement learning model.

[0068] The neural network is designed based on the SAC (Soft Actor-Critic) algorithm, which includes a policy network (Actor) and a value network (Critic); the policy network generates the probability distribution of the robot's actions, and the value network guides the policy optimization by evaluating the value function (Q value) of the state-action pair; the input data includes the robot's state information (such as position, speed, etc.), target position, and environmental characteristics, and the output is the robot's next action control signal; by maximizing the policy entropy value to balance exploration and utilization, and using the experience replay mechanism to randomly sample from historical interaction data, optimize network parameters, improve data utilization efficiency and model generalization ability, and thus achieve a more stable reinforcement learning process; the zero reaction constraint is designed in the training objective through the action space vector to ensure that the disturbance of the robot's movement to the base is minimized; combined with the optimization algorithm to improve the stability and exploratory nature of the strategy, the training obtains a motion planning strategy that meets the zero reaction condition;

[0069] Finally, the performance of the neural network is evaluated through verification tasks. The whole process aims to achieve motion planning of a floating robotic arm in space, combining reinforcement learning algorithms and neural network models to effectively cope with complex task requirements in space environments.

[0070] A space floating robot arm motion planning system based on zero reaction space and reinforcement learning, the system has a program module corresponding to the steps of the above technical solution, and executes the steps in the space floating robot arm motion planning method based on zero reaction space and reinforcement learning during operation.

[0071] A computer-readable storage medium stores a computer program, wherein the computer program is configured to implement the steps of the spatial floating manipulator motion planning method based on zero reaction space and reinforcement learning when called by a processor.

[0072] The beneficial technical effects of the present invention are:

[0073] The space floating manipulator motion planning method based on zero reaction space and reinforcement learning proposed in the present invention introduces a reinforcement learning algorithm. This method can efficiently search and optimize strategies in high-dimensional and complex action spaces, thereby achieving more intelligent and robust motion planning. This capability enables the algorithm to adaptively adjust and generate optimized planning paths when faced with dynamic environments and complex task requirements. By setting the action space of the reinforcement learning algorithm to the zero reaction motion space of the space manipulator, the algorithm not only has the above-mentioned advantages of the reinforcement learning algorithm, but can also significantly suppress the base attitude disturbance, thereby improving the success rate and efficiency of the space floating manipulator in performing on-orbit tasks.

[0074] The present invention solves the problem of base attitude disturbance caused by the movement of the existing space floating manipulator, and realizes the motion planning of the space manipulator in the on-orbit service mission to efficiently and accurately perform complex operations while minimizing the disturbance to the base. Due to the principle of conservation of momentum in the space environment, the movement of the manipulator will generate a reaction force, causing the base attitude to change, thereby affecting the stability and operation accuracy of the entire system. A motion planning method based on zero reaction motion space proposed by the present invention designs the motion space of the space manipulator as a zero reaction space to ensure that the movement of the manipulator will not cause undesirable disturbances to the base, thereby improving the stability of the system and the task execution accuracy. In addition, the present invention combines a reinforcement learning algorithm and sets its action space as a zero reaction motion space. The efficient search and optimization capabilities of reinforcement learning enable the manipulator to find the optimal motion strategy in a complex task space while effectively avoiding disturbances to the base. Through multiple rounds of training, reinforcement learning can continuously adjust strategies to adapt to different task requirements, ensuring that the manipulator can maintain high-precision operation and ensure the stability of the base when performing tasks such as space grabbing and maintenance. Experimental results show that the method of the present invention can significantly reduce the deviation of the base attitude, improve the stability and efficiency of task execution, and provide a practical solution for future on-orbit servicing missions of space robotic arms. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification, and are used to further illustrate the preferred embodiments of the present invention and explain the principles and advantages of the present invention.

[0076] Figure 1 It is a schematic diagram of a model of a free-base space floating mechanical arm of the present invention;

[0077] Figure 2 It is a schematic diagram of the motion decision layer of the reinforcement learning model of the end of the robot arm approaching the target position of the present invention;

[0078] Figure 3 It is a schematic diagram of reward changes during the neural network training process of the present invention;

[0079] Figure 4 It is a schematic diagram of the change of success rate during the neural network training process of the present invention;

[0080] Figure 5 is an example diagram of a process in which the end of the robotic arm approaches a specific position on a service satellite in an embodiment of the present invention;

[0081] Figure 6 is a curve diagram of the change of the base speed modulus length in an embodiment of the present invention;

[0082] Figure 7 is a curve diagram of the change of the base angular velocity modulus length in an embodiment of the present invention;

[0083] Figure 8 3 is a curve diagram of the change of the base angular momentum modulus in an embodiment of the present invention.

[0084] Fig. 9 A schematic diagram of the Markov decision process is presented in . DETAILED DESCRIPTION

[0085] In order to make those skilled in the art better understand the present invention, Figure 1-8 The exemplary implementations or embodiments of the present invention are described. Obviously, the described implementations or embodiments are only part of the implementations or embodiments of the present invention, not all of them. Based on the implementations or embodiments of the present invention, all other implementations or embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0086] The embodiment of the present invention proposes a space floating manipulator motion planning method based on zero reaction space and reinforcement learning, which can effectively reduce the attitude disturbance of the base when the manipulator is operating, and improve the efficiency and success rate of on-orbit operation. The method includes the following steps:

[0087] Step 1: Establish a space floating robotic arm model: The space floating robotic arm model is a single-arm model and its base is in a free floating state in space;

[0088] Step 2: Establish a Markov decision process model for motion planning of a space floating robot arm, provide a mathematical description for the interaction between the space floating robot arm and the environment, and determine the five basic elements of the Markov decision process: observation space, action space, state transfer function, reward function, and discount factor:

[0089] Markov Decision Process (MDP) is a core and widely used theoretical framework in the field of reinforcement learning. This step provides a mathematical description for the interaction between the space floating robotic arm and the environment.

[0090] Step 3: Establish a motion planning method for a space floating robot based on the SAC algorithm, and determine the inputs of the policy network and value network of the SAC algorithm neural network architecture, where the policy network generates the robot action distribution and the value network evaluates the value of the state action;

[0091] The SAC algorithm is an advanced method for solving reinforcement learning problems in continuous action spaces. The goal of robot motion planning is to train a neural network that can convert observation information into robot motion through the interaction between the agent and the environment, namely the motion planner. In the SAC algorithm, the motion planner is the trained policy network.

[0092] Step 4: Establish a reinforcement learning model for the end of the robot arm approaching the target position. This step selects appropriate observation space, action space, state transfer function and designs reward function for the task scenario of the end of the robot arm approaching the target position.

[0093] Step 5: Based on the Mujoco simulation engine, in the process of the space floating robot arm interacting with the environment, neural network training is performed based on data to obtain a trained neural network for the space floating robot arm motion planning for the task of the robot arm end approaching the target position.

[0094] The specific implementation process of the above steps is as follows:

[0095] Step 1: Modeling of floating base space robotic arm;

[0096] First, we model the space floating robotic arm, such as Figure 1 As shown. The system includes a floating base and an n-DOF manipulator. We assume that link 0 (number) is the main part of the base, link i (i = 1, ..., n) is the i-th link of the manipulator, and joint j is the connection link (i-1) and link i. Assumptions:

[0097] (1) All components of the space robot are rigid.

[0098] (2) The system remains stationary in the initial state, that is, the initial linear momentum and angular momentum of the space robot are zero.

[0099] (3) There is no external force acting on the system.

[0100] (4) The position and attitude angle of the robot and objects can be measured in the inertial coordinate system.

[0101] All vectors and inertia tensors are defined based on the inertial reference frame, and the symbol definitions are shown in Table 1:

[0102] Table 1 Symbol Definition

[0103]

[0104]

[0105] The equation of motion is expressed as:

[0106]

[0107] in

[0108]

[0109] They are the Jacobian matrices of the base motion related part and the robot arm related part respectively.

[0110] Based on the assumption that the total momentum of the system is 0, the linear momentum and angular momentum P and L of the system are expressed as follows:

[0111]

[0112] in

[0113]

[0114] The derivation process of zero reaction motion space is as follows:

[0115] The system motion equation of the free-floating space manipulator is expressed as follows:

[0116]

[0117] Linear and angular velocities of the base selected by generalized coordinates And the angular velocity of the robot joint The robotic arm system considered in this paper is an n-degree-of-freedom robotic arm installed on a floating base.

[0118] The external force and torque on the system are zero, that is, Therefore, the motion of the robot arm is only affected by the joint torque τ, and the total momentum of the system is conserved.

[0119]

[0120] Since there is no second-order integral in the angular momentum equation, we obtain the base angular velocity ω b and robot arm movement speed The first-order nonholonomic constraint equation represented by is as follows:

[0121]

[0122] in, is the corrected inertia matrix. is the angular momentum due to the operation of the manipulator, represents the initial angular momentum.

[0123] Due to the needs of on-orbit space services, we hope to keep the system stable when the robotic arm performs missions. By deriving the angular momentum equation, we obtained a zero-disturbance solution.

[0124] The initial angular momentum is The angular momentum equation when there is no disturbance b = 0 is as follows:

[0125]

[0126] The following null space solution is obtained:

[0127]

[0128] The joint motion that satisfies this equation does not disturb the posture of the base. For any vector, the inertia matrix The null space is called the zero reaction motion space.

[0129] Step 2: Markov decision process model for motion planning of space floating robotic arm;

[0130] To solve the robotic arm motion planning problem using reinforcement learning, the following elements are needed to establish the MDP model.

[0131] (1) Observation space: In the robot arm motion planning task, the observation space mainly includes the robot arm’s own information, such as joint angles, end position and base posture, as well as possible obstacle information and target information.

[0132] (2) Action space: In the robot arm motion planning task, the action space dimension matches the robot arm’s degrees of freedom. In the task setting of this paper, the action space is set to the robot arm’s zero reaction motion space and joint angular velocity space according to whether zero reaction planning is adopted.

[0133] (3) State transfer function: The state transfer function describes the probability that the system will transfer to the next state after the robot arm performs a certain action in the current state. The hyperbolic tangent function is used at the end of the policy network for normalization, limiting the output value of the network to a specific range, making the output more stable and controllable.

[0134] (4) Reward function: The reward function of the robot motion planning is designed differently according to the different tasks in this paper. In the end approach task, the reward function is mainly designed based on the position deviation between the end effector and the target point. In the obstacle avoidance task, it is designed based on the shortest distance between the robot and the obstacle.

[0135] (5) Discount factor: The discount factor is used to discount the sum of future rewards and weigh short-term and long-term rewards. In robot motion planning, a larger discount factor encourages consideration of long-term goals and path planning, while a smaller discount factor makes the agent pay more attention to immediate rewards. According to the specific task requirements, choosing an appropriate discount factor can find a balance between fast response and global optimality.

[0136] As a core and widely used theoretical framework in the field of reinforcement learning, Markov Decision Process (MDP) provides a mathematical description for the interaction between the agent and the environment. In this model, the environment is characterized by a state transition probability matrix, and the agent makes decisions at each discrete time point, resulting in changes in the state of the environment, and receives corresponding rewards for the currently selected action. Fig. 9 The Markov decision process is presented in , with t = 1, 2, 3, ... marking different time steps. In a particular round of interaction, the agent is in state s t Under the learned strategy π(s t ,a t ) Take action t , thereby harvesting instant rewards r t+1 .

[0137] In the robot motion planning task, the observation space includes the state information of the robot itself, such as joint angles, end effector positions, and base postures, and may also include obstacle information and target position information in the environment. The dimension of the action space matches the degree of freedom of the robot, and is set according to whether zero reaction planning is adopted, which is divided into zero reaction motion space and joint angular velocity space. The state transition function describes the probability that the system will transfer to the next state after performing a certain action in the current state. In the policy network, the end is normalized by the hyperbolic tangent function to ensure that the output value is within a specific range, which enhances the stability and controllability of the output. The reward function is designed according to different task requirements. For example, in the task of the end effector approaching the target point, the reward function is mainly calculated based on the position information deviation between the end effector and the target point.

[0138] In step 3, the process of establishing the motion planning method of the space floating manipulator based on the SAC algorithm is as follows:

[0139] The Soft Actor-Critic (SAC) algorithm is an efficient reinforcement learning method designed specifically for problems in continuous action spaces. It uses a maximum entropy reinforcement learning framework that aims to simultaneously optimize the cumulative reward of the policy and the entropy of the policy, thereby achieving more efficient exploration and a more stable learning process. The core of SAC consists of three neural networks: the Value Network, the Soft Q Network, and the Policy Network.

[0140] The goal of the spatial floating manipulator motion planning training based on the SAC algorithm is to obtain a motion planning strategy network, which selects appropriate vectors in the action space to map to the manipulator action by parsing the observation space information. The detailed pseudo code of the SAC algorithm used in the manipulator motion planning training can be found in Table 2.

[0141] Table 2 SAC motion planning algorithm pseudo code

[0142]

[0143]

[0144] At the beginning of each new round, the parameters are initialized first, including the initial state of the robot arm, the target position of the end of the robot arm, the reinforcement learning parameters, and the nearby obstacle satellite model. The experience pool will record the steps of the agent's interaction with the environment during training. When the agent and the environment interact for a preset number of times, a set of data will be selected from the experience pool, and the parameters of the value network and the policy network will be updated through the gradient calculation formula in the algorithm.

[0145] Step 4: Simulation task setting and reinforcement learning model establishment

[0146] The mission is set as a close contact between the service satellite and the served satellite, and the end of the single robotic arm on the service satellite reaches the designated position on the served satellite.

[0147] The robot arm decision model is as follows Figure 2 As shown in the figure, the observation information is mainly divided into two parts: self information and target information. Self information includes joint angles, end effector positions, and base pose information. Target information includes the set approach target point information and the pose information of the satellite where the target is located.

[0148] When the action space is set to zero reaction motion space, the action is the zero reaction motion space vector. When the action space is set to joint space, the action is the joint angular velocity vector.

[0149] The position deviation between the end of the robot and the target point is represented by the symbol d p It is represented as follows:

[0150] d p =||p end -p goal ||2

[0151] The reward function design includes sparse rewards and dense rewards, as shown in the following formula:

[0152]

[0153] The dense reward function f(d p ) is set as follows:

[0154]

[0155] In step 4, when the end of the manipulator approaches the target position, the reinforcement learning model is used. In this task, the service star and the served star come into close contact, and the end of the single manipulator on the service star needs to accurately reach the designated position of the served star. The decision layer of the manipulator is mainly based on two types of observation information: one is its own information, including joint angles, end effector positions, and base seat postures; the other is target information, including the position of the target point and the attitude of the target satellite. When the action space is set to zero reaction motion space, the action is represented by a vector in this space; in the joint space setting, the action is represented by a joint angular velocity vector.

[0156] Step 5: Based on the Mujoco simulation engine, neural network training is performed based on data during the interaction between the floating robotic arm and the environment.

[0157] First, the Mujoco simulation engine is used to build a dynamic simulation environment for the robotic arm, including initializing the joint state of the floating robotic arm, the posture of the floating base, etc., and setting the zero reaction force constraint condition. The model is loaded into the environment, the target position and task constraints are set, and real interactive scene support is provided for subsequent training.

[0158] In the simulation environment, the policy network is used to control the robot to interact with the environment and collect state, action, reward and environmental feedback data. The collected data includes the robot's position, speed, control input, target distance change, etc. These data are used to train the reinforcement learning model.

[0159] The neural network is designed based on the SAC (Soft Actor-Critic) algorithm, which includes a policy network (Actor) and a value network (Critic). The policy network generates the probability distribution of the robot's actions, and the value network guides the policy optimization by evaluating the value function (Q value) of the state-action pair. The input data includes the state information of the robot (such as position, speed, etc.), the target position, and the environmental characteristics, and the output is the control signal for the next action of the robot. By maximizing the policy entropy value to balance exploration and utilization, and using the experience replay mechanism to randomly sample from historical interaction data, optimize network parameters, improve data utilization efficiency and model generalization ability, and achieve a more stable reinforcement learning process. The zero reaction constraint is designed in the training objective through the action space vector to ensure that the disturbance of the robot's movement to the base is minimized. Combined with the optimization algorithm to improve the stability and exploration of the strategy, the training obtains a motion planning strategy that meets the zero reaction condition.

[0160] Finally, the performance of the neural network is evaluated through verification tasks. The whole process aims to achieve motion planning of a floating robotic arm in space, combining reinforcement learning algorithms and neural network models to effectively cope with complex task requirements in space environments.

[0161] The technical effect of the present invention is verified through experiments.

[0162] The parameters used in this paper's training are shown in Table 3:

[0163] Table 3. Reinforcement learning training parameter settings

[0164]

[0165] The simulation process of the end of the robot arm approaching the target position is as follows Figure 5 As shown, the red dot at the end of the robotic arm represents the position of the end effector of the robotic arm, and the red dot at the lower left of the satellite represents the target position.

[0166] like Figure 3The figure shows the reward value change curve during the approach task training process. The shaded part indicates the data fluctuation range, and the upper and lower limits are the mean plus or minus the standard deviation. The curve is the sliding average of the reward value. Figure 3 It can be seen that the rewards of both unrestricted and zero-reaction motion space planning have reached convergence, but the rewards fluctuate greatly when there is no restriction. Figure 4 This is the success rate change curve. The trend is roughly the same as the reward value trend and eventually converges to 1, verifying the effectiveness of the algorithm.

[0167] Figure 6-8 The following are the curves of base velocity, angular velocity and angular momentum modulus. At the beginning of the task, the robot arm needs to move quickly to make the end effector close to the target point, which causes the base posture to fluctuate. During most of the task time, the base velocity, angular velocity and modulus of angular momentum are better than those without restrictions when using zero reaction motion space planning, which verifies that setting the action space to zero reaction motion space can significantly suppress the disturbance caused to the base by the robot arm during operation.

[0168] Simulation results show that the algorithm can safely approach the target point on the surrounding satellites from the starting point and suppress the base disturbance.

[0169] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments are contemplated within the scope of the invention thus described. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. A space floating manipulator motion planning method based on zero reaction space and reinforcement learning, characterized in that: The following steps are involved: Step 1: Establish a space floating robotic arm model: The space floating robotic arm model is a single-arm model and its base is in a free floating state in space; Step 2: Establish a Markov decision process model for motion planning of a space floating robot arm, provide a mathematical description for the interaction between the space floating robot arm and the environment, and determine the five basic elements of the Markov decision process: observation space, action space, state transfer function, reward function, and discount factor: Step 3: Establish a motion planning method for a space floating robot based on the SAC algorithm, and determine the inputs of the policy network and value network of the SAC algorithm neural network architecture, where the policy network generates the robot action distribution and the value network evaluates the value of the state action; Step 4: Establish a reinforcement learning model for the end of the robot arm approaching the target position. This step selects appropriate observation space, action space, state transfer function and designs reward function for the task scenario of the end of the robot arm approaching the target position. Step 5: Based on the Mujoco simulation engine, in the process of the space floating robot arm interacting with the environment, neural network training is performed based on data to obtain a trained neural network for the space floating robot arm motion planning for the task of the robot arm end approaching the target position.

2. According to claim 1, a method for motion planning of a space floating manipulator based on zero reaction space and reinforcement learning is characterized in that: In step 1, the process of establishing the space floating robotic arm model is as follows: First, a space floating manipulator is modeled. The space floating manipulator includes a floating base and an n-DOF manipulator. Assume that link 0 is the main part of the base, link i (i=1,...,n) is the i-th link of the manipulator, and joint j is the connection link (i-1) and link i. Assumptions: (1) All components of the space manipulator are rigid; (2) The system remains stationary in the initial state, that is, the initial linear momentum and angular momentum of the space manipulator are zero; (3) There is no external force acting on the system; (4) Measure the position and attitude angle of the robot and the object in the inertial coordinate system; All vectors and inertia tensors are defined based on the inertial reference frame, and the symbol definitions are shown in the following table: Table 2 Symbol Definition The motion equation of the space floating robotic arm is expressed as: in are the Jacobian matrices of the base motion related part and the robot arm related part respectively; Based on the assumption that the total momentum of the system is zero, the linear momentum and angular momentum P and L of the system are expressed as follows: in The derivation process of zero reaction motion space is as follows: The system motion equation of the free-floating space manipulator is expressed as follows: Linear and angular velocities of the base selected by generalized coordinates And the angular velocity of the robot joint The robotic arm system is an n-degree-of-freedom robotic arm installed on a floating base; The external force and torque on the system are zero, that is, Therefore, the motion of the robot arm is only affected by the joint torque τ, and the total momentum of the system is conserved. Where: represents linear momentum, represents angular momentum; Since there is no second-order integral in the angular momentum equation, we can obtain the base angular velocity ω b and robot arm movement speed The first-order nonholonomic constraint equation represented by is as follows: in, is the corrected inertia matrix, is the angular momentum due to the operation of the manipulator, represents the initial angular momentum; Due to the demand for on-orbit services in space, it is hoped that the system will be kept stable when the robotic arm is performing its mission. The zero-disturbance solution is derived from the angular momentum equation: The initial angular momentum is The angular momentum equation when there is no disturbance b = 0 is as follows: The following null space solution is obtained: The joint motion that satisfies this equation does not disturb the posture of the base, where For any vector, the inertia matrix The null space is called the zero reaction motion space.

3. A space floating manipulator motion planning method based on zero reaction space and reinforcement learning according to claim 1 or 2, characterized in that: The specific process of step 2 includes: To solve the problem of robot motion planning using reinforcement learning, the following elements need to be built into the MDP model: (1) Observation space: In the robot motion planning task, the observation space mainly includes the robot's own information, joint angles, end position and base posture, as well as obstacle information and target information; (2) Action space: In the robot arm motion planning task, the action space dimension matches the robot arm’s degrees of freedom; in the task setting, the action space is set to the robot arm’s zero reaction motion space and joint angular velocity space depending on whether zero reaction planning is used; (3) State transfer function: The state transfer function describes the probability that the system will transfer to the next state after the robot arm performs a certain action in the current state. The hyperbolic tangent function is used at the end of the policy network for normalization, limiting the output value of the network to a specific range, making the output more stable and controllable. (4) Reward function: The reward function of the robot arm motion planning is designed differently according to different tasks. In the end approach task, the reward function is mainly designed based on the position deviation between the end effector and the target point; while in the obstacle avoidance task, it is designed based on the shortest distance between the robot arm and the obstacle. (5) Discount factor: The discount factor is used to discount the sum of future rewards and weigh short-term and long-term rewards. In robotic arm motion planning, a larger discount factor encourages consideration of long-term goals and path planning, while a smaller discount factor makes the agent pay more attention to immediate rewards. According to the specific task requirements, choosing an appropriate discount factor can find a balance between fast response and global optimality.

4. According to claim 3, a method for motion planning of a space floating manipulator based on zero reaction space and reinforcement learning is characterized in that: In step 3, the process of establishing the motion planning method of the space floating manipulator based on the SAC algorithm is as follows: The core of SAC consists of three neural networks: Value Network, Soft Q Network and Policy Network. The training goal of the spatial floating robot motion planning based on the SAC algorithm is to obtain a motion planning policy network, which selects appropriate vectors in the action space to map to the robot action by analyzing the observation space information. During the training process, at the beginning of each new round, the parameters will be initialized first, including the initial state of the robot arm, the target position of the end of the robot arm, the reinforcement learning parameters, and the nearby obstacle satellite model; the experience pool will record the steps of the interaction between the agent and the environment during the training process; when the agent and the environment interact for a preset number of times, a set of data will be screened from the experience pool, and the parameters of the value network and the policy network will be updated through the gradient calculation formula in the algorithm.

5. The method for motion planning of a space floating manipulator based on zero reaction space and reinforcement learning according to claim 4 is characterized in that: Step 4: The specific process of establishing a reinforcement learning model for the end of the robotic arm approaching the target position includes: The mission task is set as close contact between the serving star and the served star, and the end of the single robotic arm on the serving star reaches the designated position on the served star; The observation information of the robot arm decision model (as shown in Figure 2) is mainly divided into two parts: self-information and target information; self-information includes joint angles, end effector positions, and base posture information; target information includes the set approach target point information and the posture information of the satellite where the target is located; When the action space is set to zero reaction motion space, the action is the zero reaction motion space vector; when the action space is set to joint space, the action is the joint angular velocity vector; The position deviation between the end of the robot and the target point is represented by the symbol d p It is represented as follows: d p =||p end -p goal ||2 The reward function design includes sparse rewards and dense rewards, as shown in the following formula: The dense reward function f(d p ) is set as follows:

6. The method for motion planning of a space floating manipulator based on zero reaction space and reinforcement learning according to claim 5, characterized in that: In step 5, based on the Mujoco simulation engine, neural network training is performed based on data during the interaction between the floating robot and the environment. The specific process includes: First, the Mujoco simulation engine is used to build a dynamic simulation environment for the robotic arm, including initializing the joint state of the floating robotic arm and the posture of the floating base, and setting the zero reaction force constraint condition; loading the model in the environment, setting the target position and task constraints, and providing real interactive scene support for subsequent training; In the simulation environment, the policy network is used to control the interaction between the robot and the environment, and collect state, action, reward and environmental feedback data. The collected data includes the position, speed, control input and target distance of the robot, which are used to train the reinforcement learning model. The neural network is designed based on the SAC (Soft Actor-Critic) algorithm, which includes a policy network (Actor) and a value network (Critic); the policy network generates the probability distribution of the robot's actions, and the value network guides the policy optimization by evaluating the value function (Q value) of the state-action pair; the input data includes the robot's state information (such as position, speed, etc.), target position, and environmental characteristics, and the output is the robot's next action control signal; by maximizing the policy entropy value to balance exploration and utilization, and using the experience replay mechanism to randomly sample from historical interaction data, optimize network parameters, improve data utilization efficiency and model generalization ability, and thus achieve a more stable reinforcement learning process; the zero reaction constraint is designed in the training objective through the action space vector to ensure that the disturbance of the robot's movement to the base is minimized; combined with the optimization algorithm to improve the stability and exploratory nature of the strategy, the training obtains a motion planning strategy that meets the zero reaction condition; Finally, the performance of the neural network is evaluated through verification tasks. The whole process aims to achieve motion planning of a floating robotic arm in space, combining reinforcement learning algorithms and neural network models to effectively cope with complex task requirements in space environments.

7. A space floating manipulator motion planning system based on zero reaction space and reinforcement learning, characterized by: The system has a program module corresponding to the steps of any one of claims 1 to 6 above, and executes the steps in the space floating robot arm motion planning method based on zero reaction space and reinforcement learning when running.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the space floating manipulator motion planning method based on zero reaction space and reinforcement learning as described in any one of claims 1 to 6 when called by a processor.

Images