Remote operation space manipulator trajectory planning method based on deep reinforcement learning

Abstract

The invention provides a remote operation space manipulator trajectory planning method based on deep reinforcement learning (DRL). The trajectory planning problem caused by communication delay is effectively solved. The method comprises the steps that a kinematic model and a reinforcement learning framework of the rigid mechanical arm are established; constructing a teleoperation framework comprising a master end, a data chain and a slave end; a delay processing module (DIP) is adopted at a main end to process communication delay, so that the real-time performance of states and rewards is ensured; a DRL decision module is utilized to update the intelligent agent, and learning is carried out based on an experience playback pool and an action playback pool; the mechanical arm is guided to complete the task through interaction of the intelligent agent and the environment. According to the method, the DRL is integrated into the remote control framework, so that the decision-making ability of the intelligent agent is enhanced, the performance is excellent especially in an inherent delay environment, good robustness is shown under different noise and dynamic parameter conditions, and additional parameter adjustment is not needed.

Description

Technical Field

[0001] The present invention relates to space robot teleoperation technology, and in particular to a teleoperation space robot arm trajectory planning method based on deep reinforcement learning. Background Art

[0002] With the help of teleoperation technology, space robots have greatly improved the operational capabilities of astronauts and played an increasingly important role in orbital service missions, including capturing, refueling, repairing satellites, removing orbital debris, and assembling and maintaining large space infrastructure. Currently, the commonly used teleoperation control methods include remote programming control, bilateral control, and virtual predictive control. Remote programming control operates in a supervisory mode, where the space robot receives operation instructions from the master end and interacts with the environment at the slave end to form a closed-loop system. However, this method relies on the intelligence level of the space robot. Both bilateral control and virtual predictive control belong to the category of direct control. Bilateral control directly receives force feedback information from the remote environment and is suitable for situations with small delays. With the help of appropriate control algorithms, such as passive control, robust control, and impact control, the force and position information between the master end operator and the slave end robot is kept consistent. In contrast, virtual predictive control establishes a virtual model similar to the slave end environment at the master end. When making decisions, the master end simultaneously refers to the feedback information from the slave virtual model and the slave end, thereby alleviating the impact of large delays on system stability and operating characteristics. However, model-based methods require high accuracy of the model, so deep domain expertise is required to design complex controllers. Moreover, the space robot is a complex dynamic system. The nonlinearity caused by the dynamic coupling between the base and the manipulator, friction, and joint flexibility, etc., poses a huge challenge to modeling. In addition, even if the original system is modeled based on a physical model, many key dynamic parameters are difficult to obtain or get their accurate values. These factors will have a significant impact on the model and thus affect the performance of the controller.

[0003] Data-driven model-free deep reinforcement learning (DRL) has shown significant promise in many fields, including games, industrial control, and large language models. This approach has also been widely used by scholars in the field of space robotics, focusing mainly on trajectory planning of robotic arms. Although previous studies have demonstrated the potential of DRL in space robotics, these studies generally assume that remote robots are highly intelligent and can autonomously implement and complete designated tasks. From the current state of the art, such an assumption is unrealistic. In practice, most of the intelligence is embodied in the master end, which poses a huge challenge to DRL-based control due to the objective problems of communication delay and limited bandwidth between the master and slave ends.

[0004] Existing methods for solving the delay problem in reinforcement learning can be divided into three categories: state enhancement method, model prediction method and other methods. The state enhancement method mainly transforms the original delayed Markov decision process into a new delay-free Markov decision process based on the information state composed of the latest observed delay state and action sequence. From a theoretical perspective, when the delay increases, the state space of the information state will grow exponentially. Therefore, the agent needs an exponentially growing number of samples to update the network parameters to optimize the policy network. This not only puts higher requirements on computing resources, but also slows down the convergence of the policy. In extreme cases, the policy may even diverge. From a practical point of view, when designing a policy network based on the state enhancement method, due to the characteristics of deep neural networks, its input can only be a certain value. When there is a random delay in the environment, the information state corresponding to the delay cannot adapt to the network with a certain input size. Usually, a policy network with a maximum delay corresponding to the information state is designed to solve the random delay problem, but this design introduces redundant information to the information state that does not reach the maximum delay, thereby interfering with the decision.

[0005] The model prediction method generally includes two steps: predicting the unknown state caused by the delay, and then making the final decision based on the predicted state and the standard reinforcement learning algorithm. It is crucial to accurately simulate the dynamics of the environment. In the early stage, it was implemented by methods such as deterministic mapping and random forest. Later, recurrent neural networks, feedforward models and particle ensemble methods were used to learn transformations. To predict the unknown state caused by the delay, a forward dynamics model must be constructed first. When the data distribution is stable, the data-driven method requires a large amount of data to more realistically characterize the environment; when the data distribution is not stable, the forward dynamics model obtained based on limited data can only reflect the situation of part of the environment. In addition, assuming that there is a more accurate forward dynamics model, the delay state predicted by the latest observed delay state and action sequence has a positive correlation with the size of the delay, and the error between it and the true state increases significantly with the increase of the delay. These will have a great impact on the final decision.

[0006] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0007] The main purpose of the present invention is to solve the problems existing in the above-mentioned background technology and to provide a trajectory planning method for a teleoperated space robot based on deep reinforcement learning.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A trajectory planning method for a teleoperated space manipulator based on deep reinforcement learning (DRL), the method comprising the following steps:

[0010] S1: Establish a rigid body kinematic model of a floating base space robot with a rigid manipulator, including defining the joint angle vector and velocity vector of the manipulator, and the Jacobian matrix of the base and the manipulator, and define a reinforcement learning six-element tuple, including state, action, state transfer function, reward function, initial state distribution and discount factor, to provide kinematic parameters and a reinforcement learning framework for trajectory planning of the manipulator, and the reinforcement learning framework is used for the decision-making process of the interaction between the intelligent agent and the environment;

[0011] S2: Build the overall framework of teleoperation, including the master end, data link and slave end. The data link transmits and processes data, the slave end executes the interactive commands between the space manipulator and the environment and returns status information and rewards, and the master end performs real-time control and decision-making on the manipulator operation;

[0012] S3: According to the state information transmitted by the data link, the delay processing DIP module of the master end processes the current state of the master end to obtain the current state and the previous order reward without the influence of the communication delay between the master and the slave end;

[0013] S4: Based on the state after removing the delay effect, the DRL decision module of the master side continuously updates the agent according to the existing experience replay pool, action replay pool and current delay, and obtains a new state through the interaction between the agent and the environment to guide the robot arm to plan the trajectory;

[0014] S5: The delay processing module is used to remove the delay effect of the slave data through the data link, and the data without the effect is used to update the intelligent agent and interact with the environment, gradually guiding the slave robot arm to complete the planning task, wherein the intelligent agent generates corresponding actions according to the state and strategy network after removing the delay effect, and the slave space robot arm interacts with the environment according to the action, generates the next state and reward, and then feeds back to the master end, and the master end updates the intelligent agent according to the new state and reward.

[0015] In some embodiments, further, the present invention processes the current state of the master end through one or more of a mapping method, a prediction method and a state enhancement method; wherein the mapping method adopts a memoryless strategy, ignores delays, and makes decisions based on the most recently observed state as the true state of the environment; the prediction method uses historical trajectory data to train a forward model; the state enhancement method transforms the delayed Markov decision process into a delay-free Markov decision process by constructing an information state consisting of delayed state information and a historical action sequence.

[0016] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the trajectory planning method of a teleoperated space robot arm based on deep reinforcement learning is implemented.

[0017] The present invention has the following beneficial effects:

[0018] The method of the present invention integrates deep reinforcement learning into the traditional remote control framework, solving the complex problem of trajectory planning of remote-controlled space manipulators. In the present invention, the decision-making ability of the intelligent agent is enhanced by utilizing the delayed state information and historical actions through the delay processing module, ensuring its recovery ability in an inherent delay environment. The delay information processing module generates a state that is helpful for the current decision by considering the current delay state and historical action results at the same time. After the delay information processing is completed, the decision-making module generates corresponding actions based on the new state obtained. Furthermore, the embodiment of the present invention designs three innovative methods of mapping, prediction and state enhancement to construct a delay processing module.

[0019] The present invention effectively solves the trajectory planning problem of the space robot manipulator by applying DRL in the teleoperation scenario. Compared with the prior art, the main advantages of the present invention are as follows:

[0020] 1) This paper integrates deep reinforcement learning into the traditional remote control framework for the first time to complete the corresponding trajectory planning task;

[0021] 2) The present invention enhances decision-making capabilities by utilizing delayed state information and historical action buffering;

[0022] 3) The present invention enhances the decision-making ability of the agent while ensuring the adaptability of the algorithm in an environment characterized by inherent delays;

[0023] 4) The method proposed in the present invention is effective for various scenarios including whether the base is floating and whether the target is rotating;

[0024] 5) The method proposed in the present invention is highly robust under different noise or dynamic parameter conditions and does not require parameter adjustment.

[0025] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is an overall flow chart of the trajectory planning method of a teleoperated space robot arm based on deep reinforcement learning in an embodiment of the present invention.

[0027] Figure 2 1 is a schematic diagram of a space robot in an embodiment of the present invention.

[0028] Figure 3It is an overall flow chart of DRL-based space teleoperation in an embodiment of the present invention.

[0029] Figure 4 It is a schematic diagram of a delay processing module mapping method in an embodiment of the present invention.

[0030] Figure 5 Schematic diagram of a delay processing module prediction method in an embodiment of the present invention.

[0031] Figure 6 It is a structural diagram of a spatial seven-degree-of-freedom redundant robotic arm in an embodiment of the present invention.

[0032] Figure 7 It is a comparison chart of the returns of different algorithms in various scenarios in the embodiments of the present invention.

[0033] Figure 8 It is a comparison chart of the success rates of different algorithms in various scenarios in the embodiments of the present invention.

[0034] Fig. 9 4 is a time and memory consumption comparison chart of different algorithms in the embodiments of the present invention.

[0035] Fig.10 It is a reward evaluation graph of the training model under various action noises without fine-tuning according to an embodiment of the present invention.

[0036] Fig.11 It is a reward evaluation graph of the training model under various observation noises without fine-tuning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0038] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0039] The present invention proposes a trajectory planning method for a teleoperated space manipulator based on deep reinforcement learning (DRL), which effectively solves the trajectory planning problem caused by communication delay. The method includes: establishing a kinematic model and a reinforcement learning framework for a rigid manipulator; constructing a teleoperation framework including a master end, a data link, and a slave end; using a delay processing module (DIP) on the master end to handle communication delays to ensure the real-time nature of states and rewards; using a DRL decision module to update the agent, and learning based on an experience replay pool and an action replay pool; guiding the manipulator to complete tasks through interaction between the agent and the environment. The present invention enhances the decision-making ability of the agent by integrating DRL into the remote control framework, especially in an inherent delay environment, and exhibits good robustness under different noise and dynamic parameter conditions without the need for additional parameter adjustment.

[0040] See also Figure 1 , an embodiment of the present invention provides a trajectory planning method for a teleoperated space manipulator based on deep reinforcement learning, comprising the following steps:

[0041] Step S1: Establish a rigid body kinematic model of a floating base space robot with a rigid manipulator, including defining the joint angle vector and velocity vector of the manipulator, and the Jacobian matrix of the base and the manipulator, and at the same time define a reinforcement learning six-element tuple, including state, action, state transfer function, reward function, initial state distribution and discount factor, to provide kinematic parameters and a reinforcement learning framework for trajectory planning of the manipulator, and the reinforcement learning framework is used for the decision-making process of the interaction between the intelligent agent and the environment.

[0042] In some embodiments, establishing a rigid body kinematic model of a floating base space robot with a rigid robotic arm specifically includes: defining the joint angle vector of the robotic arm in the space robot system, representing the position state of the robotic arm at any time; defining a velocity vector corresponding to the joint angle vector, representing the dynamic motion state of the robotic arm; constructing the Jacobian matrix of the base and the robotic arm, which is used to describe the mathematical relationship between the motion of the robotic arm and the motion of the base; determining the conservation relationship between the linear momentum and angular momentum of the space robot in the absence of external force, expressed by the inertia matrix and the coupled inertia matrix; based on the conservation law and the Jacobian matrix, determining the generalized Jacobian matrix, which associates the kinematic and dynamic parameters of the robotic arm for precise calculation of subsequent trajectory planning.

[0043] Defining the six-element tuple of reinforcement learning specifically includes: designing the state space, including the joint angle vector, joint velocity vector, end effector position, end effector velocity, target position and distance between the end effector and the target of the robot arm, so as to capture the dynamic and task-related information of the robot arm; designing the action space, represented as the torque vector applied to the joint of the robot arm, each component of the torque vector is limited to a predetermined safety range to ensure the safety and effectiveness of the robot arm's action; designing the state transfer function, which predicts the next state of the robot arm in the environment based on the current state and the executed action, so as to enable the agent to learn the impact of its behavior on the environment; designing the reward function, which evaluates the value of the action taken according to the distance between the end effector of the robot arm and the target, the velocity of the robot arm and the base, and the completion of the task, so as to motivate the agent to take action to maximize the cumulative reward; introducing a discount factor to adjust and balance the impact of immediate rewards and future rewards on the current decision to balance short-term and long-term benefits; implementing the experience replay mechanism to store the sequence of states, actions, rewards and new states during the interaction between the agent and the environment, so that the agent can learn and improve its strategy from historical data.

[0044] The reward function includes: a mechanism that rewards the end effector for reducing the distance between the end effector and the target point, so as to encourage the agent to perform an action to move the end effector toward the target point; when the end effector approaches the target point, it is rewarded for its behavior of hovering near the target to ensure continuous progress toward the target; the agent is rewarded for smoothing the movement of the robotic arm and reducing the speed fluctuation of the base and the end effector to optimize the smoothness of the trajectory; a terminal reward mechanism is activated when the distance between the end effector and the target point is less than or equal to a preset threshold, and its value is proportional to the number of steps remaining when the task is completed, so as to motivate the agent to complete the task efficiently; through the comprehensive evaluation of the reward function, the agent learns to take appropriate actions at different operation stages, including approaching the target, fine-tuning and completing the task, so as to optimize the entire teleoperation process.

[0045] Step S2: Based on the kinematic model and reinforcement learning framework established in step S1, the overall framework of teleoperation is constructed, including the master end, data link and slave end, wherein the data link transmits and processes data, the slave end executes the interactive commands between the space manipulator and the environment and returns status information and rewards, and the master end performs real-time control and decision-making on the operation of the manipulator.

[0046] In some embodiments, constructing an overall framework for teleoperation specifically includes: identifying time delay problems in the teleoperation environment, including action delay, observation delay, and reward delay, and their impact on the decision-making process of the intelligent agent; using loop time delay (RTD) as a benchmark for measuring communication delay to ensure that the time difference between the observed state of the master end and the actual state of the slave end is quantified and compensated; determining the value of RTD under ideal conditions based on the transmission distance to accurately compensate for delays at the master end; processing random delays caused by environmental factors such as network congestion, and providing adaptive support for the decision-making of the intelligent agent by simulating the statistical distribution of random delays; comprehensively considering constant and random delays, and formulating strategies to ensure that the intelligent agent can make timely and accurate decisions based on the observations of the master end at each time step.

[0047] Step S3: Using the status information of the data link transmission in the teleoperation framework in step S2, the delay processing (DIP) module of the master end processes the current state of the master end to obtain the current state and the previous reward without the influence of the communication delay between the master and the slave ends.

[0048] In some embodiments, the current state of the master end is processed by one or more of a mapping method, a prediction method, and a state enhancement method. The mapping method adopts a memoryless strategy, ignores delays, and makes decisions based on the most recently observed state as the true state of the environment. The prediction method uses historical trajectory data to train a forward model. The state enhancement method transforms a delayed Markov decision process into a non-delayed Markov decision process by constructing an information state consisting of delayed state information and historical action sequences.

[0049] The mapping method specifically includes: using a memoryless strategy to process the observed state of the master end, ignoring the delay effect caused by communication delay; using the latest observed state as the true state of the environment at the current moment for the decision-making process of the intelligent agent; when the state observed by the master end is inconsistent with the expected state, identifying and confirming the existence of an unobserved state; when an unobserved state is identified, replacing the current state with the state at the previous moment, and adjusting the immediate reward to reflect the replacement.

[0050] The prediction method specifically includes: using a large amount of historical trajectory data obtained by interacting with the environment, including tuples of state, action, reward and next state, to train a forward model through supervised learning to simulate system dynamics; using the forward model as a nonlinear function, inputting the current state and action, and outputting the predicted reward and next state to model the future behavior of the system; using the last observed state and historical action sequence, iteratively calculating the predicted current state and reward of the previous state through the forward model; and performing the decision-making process of the intelligent agent according to the predicted current state to generate corresponding actions and guide the robot arm to perform the next operation;

[0051] The state enhancement method specifically includes: constructing an information state that combines delayed state information and a historical action sequence to simulate a decision-making environment without delay; when defining a new state, for an observed delayed state, combining it with a series of historical actions to form an information state representing the current environment state; adjusting the reward function, when a delayed state is observed, setting the immediate reward to zero to reflect the missing information due to the delay; and correcting the initial state distribution to include a random action sequence taken when no remote state update is received, so that the agent can make decisions when the information is incomplete.

[0052] Step S4: Based on the state after removing the delay effect in step S3, the DRL decision module of the master end continuously updates the agent according to the existing experience replay pool, action replay pool and current delay, and obtains a new state through the interaction between the agent and the environment to guide the robot arm to plan the trajectory;

[0053] In some embodiments, the DRL decision module specifically includes: designing the objective function of the master end to maximize the cumulative reward, which is based on the delayed state after DIP processing and the distribution of previously sampled states and actions, that is, the data in the replay buffer; through the objective function, introducing policy entropy to encourage intelligent agent exploration, and controlling the randomness of the optimal strategy to avoid local optimal solutions; using the optimization method based on the SAC algorithm, respectively approximating the state-action value function, the state value function and the policy function through a neural network to achieve continuous updating and optimization of the strategy; updating the parameters of the value function by minimizing the soft Bellman residual to ensure the effectiveness of the learning process and the accuracy of the strategy; updating the parameters of the soft value function by minimizing the squared residual to improve the accuracy of the state value estimation; updating the parameters of the policy network by minimizing the KL divergence, optimizing the representation of the strategy and enhancing the robustness of the strategy; constructing the policy network using the reparameterization method, allowing the policy network to generate actions in the presence of noise, and improving the adaptability of the strategy; the management strategy of the replay buffer only stores real state data, and non-real data should not be stored in the experience replay pool.

[0054] Step S5: Using the DRL decision module to update the intelligent agent, remove the delay effect of the slave data through the data link through the delay processing module, use the removed data to update the intelligent agent and interact with the environment, and gradually guide the slave manipulator to complete the planning task. Among them, the intelligent agent generates corresponding actions based on the state and strategy network that removes the delay effect, and the slave space manipulator interacts with the environment based on the action to generate the next state and reward, and then feeds back to the master, and the master updates the intelligent agent based on the new state and reward.

[0055] In a preferred embodiment, the action instructions generated by the agent are executed in the environment through the remote environment interaction module, and the environmental feedback caused by the action is collected, including new state information and rewards, and then the feedback is passed to the DRL decision module on the host side.

[0056] The embodiment of the present invention applies DRL to solve the trajectory planning problem of a space robot manipulator in a teleoperation scenario. The control process includes a delay information processing module, a DRL decision module, and a remote environment interaction module. The purpose of the delay information processing module is to generate a state that is helpful for the current decision by considering the current delay state and the historical action results at the same time. To achieve this goal, three methods of mapping, prediction, and state enhancement are designed. After the delay information processing is completed, the decision module generates corresponding actions according to the new state obtained. The Soft Actor-Critic (SAC) algorithm is selected as the decision algorithm, and its robustness of generating strategies through cumulative rewards and maximum entropy optimization is utilized. The remote environment interaction module runs in the MuJoCo simulation environment, including four single-arm scenarios: fixed base and target, fixed base and rotating target, free floating base and fixed target, and free floating base and rotating target. After receiving the torque information from the master, the space manipulator of the slave interacts with the environment, generates the next state and reward, and then feeds back to the master.

[0057] The following further describes an algorithm implementation example and experimental verification of a specific embodiment of the present invention.

[0058] A method for trajectory planning of a teleoperated space manipulator based on deep reinforcement learning comprises the following steps:

[0059] S1: Establish a rigid body kinematic model of a floating base space robot equipped with a rigid manipulator; define the specific form of the reinforcement learning 6-element tuple (state, action, state transfer function, reward function, initial state distribution, discount factor);

[0060] S2: Establish the overall framework of teleoperation, including the master end, data link and slave end. The data link facilitates the transmission and processing of data. The slave end mainly executes the commands for the space manipulator to interact with the environment, and then returns the status information and rewards generated by the environment. The master end includes the Delay Information Processing (DIP) module and the DRL decision module.

[0061] S3: Define the delay processing module, which can use three methods: mapping, prediction and state enhancement. This module processes the current state of the master end to obtain the current state without the delay effect and the previous order reward;

[0062] S4: Define the DRL decision module, continuously update the agent based on the existing experience replay pool, action replay pool and current delay, and obtain a new state through the interaction between the agent and the environment;

[0063] S5: The delay processing module removes the delay effect of the slave data through the data link, and uses the removed data to update the intelligent agent and interact with the environment, gradually guiding the slave to complete the planning task.

[0064] The processing methods of the delay processing module in S3 are divided into the following three types:

[0065] S3.1: The mapping method adopts a memoryless strategy, ignores delays, and takes the most recently observed state as the true state of the environment for decision making;

[0066] S3.2: The prediction method uses historical trajectory data to train a forward model;

[0067] S3.3: The state enhancement method transforms the delayed Markov decision process into a delay-free Markov decision process by constructing an information state consisting of delayed state information and historical action sequences.

[0068] Kinematic Modeling of Space Robot

[0069] The space robot system usually consists of a spacecraft (base) and a multi-degree of freedom (DOF) manipulator. The joint angle vector of the manipulator can be expressed as q = [θ1, θ2, …, θ n ] T , the velocity vector is like Figure 2 In addition, (v i ,ω i ) T (i=b, e) represents the speed and angular velocity of the base and the robot arm respectively. The following formula can be obtained:

[0070]

[0071] Among them, J b and J m Represent the Jacobian matrices of the base and the manipulator respectively. In the absence of external forces and torques, the linear momentum P and angular momentum L of the space robot are conserved, as shown below:

[0072]

[0073] Among them, H b and H bm are the inertia matrix and the coupling inertia matrix respectively. Further, assuming that the initial values ​​of the linear velocity and angular velocity are 0, the base velocity can be expressed as follows:

[0074]

[0075] Among them J bm represents the base Jacobian matrix. Substituting formula (3) into formula (1) will yield the following expression:

[0076]

[0077] Among them, J g represents the generalized Jacobian matrix, which is related to both kinematic and dynamic parameters.

[0078] Reinforcement Learning

[0079] Reinforcement learning is a sequential decision process based on the Markov decision process (MDP) theoretical framework. Typically, an MDP is defined by a 6-tuple Each of these items represents the state space, action space, state transition function, reward function, and discount factor for future rewards. The agent obtains the current state s t After that, based on the current strategy π, action a can be selected t ~π(·|s t , a t ) to interact with the environment. The environment then returns a new state s t+1 and reward r t =R(s t , a t ). The goal of reinforcement learning is to maximize the cumulative reward to obtain the optimal strategy.

[0080] Since the information contained in the state is of vital value to the agent's decision-making, it is necessary to consider as many relevant features that affect the decision-making as possible when designing the state. For the trajectory planning task of the space robot, especially when the space robot is in the floating base mode, there are non-holonomic constraints between the base and the manipulator, which will affect the planning of the manipulator. Therefore, for the fixed base mode, the state can be set as follows:

[0081]

[0082] Among them, each item represents the joint angle vector, joint position vector, robot end position, robot end velocity, target position, and the distance from the end to the target. For free floating mode, the state can be set as follows:

[0083]

[0084] Among them, each item represents the joint angle vector, the joint position vector, the end position of the robot arm, the end velocity of the robot arm, the base center of mass position, the base center of mass velocity, the target position, and the distance from the end to the target.

[0085] The motion is designed as a set of torques applied to the robot joints, constrained within a certain range: And a ti ∈[-max(torque i ),max(torque i )], i = 1, ..., n, where torque i represents the torque acting on the i-th joint.

[0086] The reward function is defined as follows:

[0087]

[0088] Among them, the round end condition is the distance d t is less than the threshold d threshold. Each term in formula (7) has a different purpose. The first term encourages the end effector of the robot to get as close to the target point as possible. The second term deals with the scenario where the end effector lingers near the target point and does not move further when the distance approaches the threshold, ensuring that it continues to move towards the target. The purpose of the third and fourth terms is to mitigate excessive velocity fluctuations of the robot base, end effector, and joints during trajectory planning, and to promote a smooth transition of these variables from a reward perspective. Finally, the fifth term serves as a terminal reward and is only activated when the distance is less than or equal to the threshold. The size of this reward is proportional to the remaining steps at the end of the episode. Therefore, as the agent completes the task faster, the reward value associated with this term will increase proportionally.

[0089] Time delay problem of remote operation

[0090] like Figure 3 As shown, in a space teleoperation environment, the time delay introduced by communication destroys the Markov property of the original decision-making process and seriously affects the performance of the reinforcement learning algorithm. Specifically, there will be a delay before the instruction issued by the operator at the master end reaches the slave end, that is, the action delay. At the same time, when the slave space robot interacts with the environment, the feedback from the environment (including new states and rewards) is not transmitted to the master end in time, resulting in observation delay and reward delay. Since the effects of observation delay and action delay on the decision-making process of the intelligent agent are equivalent, the example of the present invention only focuses on the observation delay and ensures that its value is equal to the value of the loop time delay (Round Time Delay, RTD). Here, the example of the present invention defines the state of the slave end at time t as s t , the master end observes the state x t .

[0091] Ideally, RTD remains constant because it is completely determined by the transmission distance. On the slave side, the state of the robot changes over time, and on the master side, due to the existence of RTD, the operator's observation lags behind the slave side by RTD time steps. This is explained by taking the RTD value as 1. After the entire system is started, the slave side sends the first state, but the master side observation is empty. At subsequent moments, the master side observation lags behind the slave side observation by one time step, expressed as: t =s t-1 δ(t≥1)+Φδ(t<1), where Φ is zero and δ is the Dirac function. In addition, during the transmission process, environmental factors such as network congestion often cause the delay to be random. Based on RTD, we assume that the random delay follows a uniform distribution with parameter ξ to simulate the random factor, expressed as d random ~Uniform(ξ). Therefore, the total communication delay is d total (t) = d RTD +d random Regardless of constant or random delay, the focus of the present invention is to make decisions at each time step based on the master-side observations.

[0092] DIP module processing method

[0093] The DIP module of the embodiment of the present invention is constructed in three ways: mapping, prediction, and state enhancement. The master-side delay value is defined as follows:

[0094]

[0095] 1) Mapping

[0096] The mapping adopts a memoryless strategy that takes into account latency and considers the most recently observed state as the true state of the environment for making decisions. t When it is different from Φ, the master observes the current state; but if x t Same as Φ, indicating that there is an unobserved state on the master side, then it is replaced by that state And will immediately reward Set to 0. Figure 4 For a simple example.

[0097] 2) Prediction

[0098] Prediction involves training a forward model f using historical trajectory data. Therefore, this method exploits a large number of tuples (s t , a t , r t ,s t+1 ) data, and train the nonlinear function f through supervised learning. Figure 5As shown, the input is s t and a t , the output is r t and t+1 , that is, s t+1 , r t =f(s t , a t ) to effectively model the system dynamics. After training the forward model, this method uses the last observed state and historical action sequences Iterative Calculation and According to the predicted Make the final decision.

[0099] 3) Status enhancement

[0100] State augmentation constructs an information state consisting of delayed state information and historical action sequences to transform the delayed MDP into a delay-free MDP, which is defined as follows:

[0101] For status When t≥d RTD , the new state can be defined as follows:

[0102]

[0103] For Action This method has not been adjusted; for the reward function The new reward can be expressed as:

[0104]

[0105] For the initial state distribution ρ, the modified form is as follows:

[0106]

[0107] where ρ(s0) is the original initial state distribution, refers to the random action taken when no remote state is received. t =Φ, in From the actions taken in the past; when x t ≠Φ, r t-1 = r(x t-1 , a t-1 ), where a t-1 Actions from past moments.

[0108] Decision module processing method

[0109] The objective function of the master is designed as follows:

[0110]

[0111] where x t Represents the delayed state after DIP processing, and D represents the distribution of previously sampled states and actions, that is, the replay buffer. Formula (12) obtains the optimal strategy by maximizing the objective function. However, end-to-end trajectory planning for space robots poses significant challenges due to its complex nature and extended state space dimensions. In addition, the optimization of formula (12) will produce a deterministic strategy, which often causes local optimal phenomena, making it difficult to find strategy parameters that achieve high cumulative rewards. To solve this problem, formula (12) is modified as follows:

[0112]

[0113] in, represents the entropy of the strategy, and α determines the relative importance of the entropy term relative to the reward term. Formula (13) not only encourages the agent to explore more effectively, but also controls the randomness of the optimal strategy.

[0114] In the present invention, formula (13) is optimized based on the SAC algorithm. Similar to the traditional SAC algorithm, the three neural networks Q θ (x t , a t ), V ψ (x t ) and π φ (a t |x t ) aims to approximate the state-action value function, the state value function and the policy function respectively. The parameter θ of the value function is updated by minimizing the soft Bellman residual:

[0115]

[0116] in, Represents and V ψ The corresponding target value network is updated by the exponential moving average of the value network weights. The parameter ψ of the soft value function is updated by minimizing the squared residual:

[0117]

[0118] Update the parameters φ of the policy network by minimizing the KL divergence:

[0119]

[0120] Among them, Z θ (x t) is the partition function and can be ignored. This method uses the reparameterization technique to minimize formula (16) and construct a policy network:

[0121] a t =f φ (∈ t ;x t ), (17)

[0122] Among them, ∈ t is the noise vector. Therefore, formula (16) can be rewritten as:

[0123]

[0124] The basic structure of the replay buffer is different from that of the traditional SAC. In the traditional method, the quadruple generated at each step will be stored in the experience replay pool. However, in the teleoperation environment, the master may not get the real state. In order to ensure the reliability of the experience replay pool data, non-real data should not be stored in the experience replay pool.

[0125] Validity Verification

[0126] based on Figure 6 The seven-DOF redundant manipulator model shown in the figure, the present invention implements a single-arm space robot model in the MuJoCo environment and applies it to four environments: fixed base fixed target, fixed base rotating target, floating base fixed target and floating base rotating target (referred to as SAFBFT, SAFBRT, SAUBFT and SAUBRT respectively). In the simulation environment, the simulation time step T m = 0.01s, an action is performed every 4 time steps, and the maximum number of simulation steps per round is T = 250, so the maximum simulation duration is 10s. The embodiment of the present invention adopts teleoperation to perform random grasping tasks, aiming to position the end effector of the space manipulator within 5mm of the target point. Therefore, considering that the diameter of the end effector is 0.02m, let d threshold =0.025m.

[0127] In order to enhance the generalization ability of the model, the embodiment of the present invention performs uniform sampling within a cubic workspace of the robot arm and uses it as the target position during training. At the same time, noise is introduced into the initial joint angles and angular velocities of the robot arm:

[0128]

[0129] The present invention conducts experiments without delay in four environments. RTD = 0, and a fixed delay d RTD =1, 2, 4, 6, 8 and their corresponding random delay ξ=2 are used to test the effect of the algorithm.

[0130] In terms of algorithm performance, Figure 7 , Figure 8 and Fig. 9 As shown in the figure, from the perspectives of reward, success rate, and time and memory consumption, the experiment obtained the following conclusions:

[0131] 1) In all four environments, the performance of the three delay handling methods deteriorates as the delay increases, regardless of whether the delay is constant or random. Due to the compound effect of the unpredictability of information delay and information loss, the performance of the three delay handling methods deteriorates with increasing delay. RTD Under these conditions, constant delays work better than random delays;

[0132] 2) Even if the task can be successfully completed, the time required will increase as the delay increases, resulting in diminishing returns, such as Figure 7 As shown;

[0133] 3) The three algorithms of the DIP module are analyzed as follows: state enhancement always shows the most stable performance, indicating the feasibility and effectiveness of transforming the delayed Markov decision process into a non-delayed Markov decision process; mapping has a good effect in slowly changing small delay scenarios such as fixed base and floating base with small delay, but the effectiveness of mapping will decrease in the case of drastic state changes or high delay; prediction requires accurate forward dynamics, and its cumulative error in iterative prediction will also have a large impact, so it has the worst effect;

[0134] 4) State enhancement methods more effectively balance efficiency and performance, such as Fig. 9 shown.

[0135] In terms of robustness analysis, according to Fig.10 , Fig.11 The following conclusions can be drawn from the analysis of the experimental data shown:

[0136] 1) In an ideal environment, noise interference significantly affects the decision-making of trained agents. The greater the possibility of noise, the greater the impact on decision-making.

[0137] 2) Compared with state noise, the destructive impact of action noise is relatively small; although action noise may cause the executed actions to deviate from the accurate decisions made based on the accurate state, resulting in partial deviation from the ideal environmental strategy, the impact is relatively limited;

[0138] 3) State noise can cause significant deviations between the observed state and the precise state, so the strategy derived from the state observed in the new environment is not optimal compared to the strategy in the ideal environment. If the original strategy continues to be used, the performance will further deteriorate.

[0139] 4) For low-probability noise scenarios or fixed-base scenarios, the original strategy remains effective only when the new environment is very similar to the ideal environment.

[0140] Compared with the prior art, the main advantages of the embodiments of the present invention are embodied in the following aspects:

[0141] 1) The present invention integrates deep reinforcement learning into the traditional remote control framework to complete the corresponding trajectory planning task;

[0142] 2) The present invention utilizes delay state information and historical action buffering to enhance decision-making capabilities; further, three construction methods of delay processing modules are proposed: mapping, prediction, and state enhancement;

[0143] 3) The present invention enhances the decision-making ability of the agent while ensuring the adaptability of the algorithm in an environment characterized by inherent delays;

[0144] 4) The method proposed in the present invention is effective for various scenarios including whether the base is floating and whether the target is rotating;

[0145] 5) The method proposed in the present invention is highly robust under different noise or dynamic parameter conditions and does not require parameter adjustment.

[0146] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0147] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0148] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0149] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory, a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0150] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0151] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0152] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0153] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0154] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0155] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0156] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0157] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0158] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A trajectory planning method for a teleoperated space manipulator based on deep reinforcement learning, characterized in that: The method comprises the following steps: S1: Establish a rigid body kinematic model of a floating base space robot with a rigid manipulator, including defining the joint angle vector and velocity vector of the manipulator, as well as the Jacobian matrix of the base and the manipulator, defining a reinforcement learning six-element tuple, including state, action, state transfer function, reward function, initial state distribution and discount factor, providing kinematic parameters and a reinforcement learning framework for trajectory planning of the manipulator, and the reinforcement learning framework is used for the decision-making process of the interaction between the intelligent agent and the environment; S2: Build the overall framework of teleoperation, including the master end, data link and slave end, where the data link transmits and processes data, the slave end executes the interactive commands between the space manipulator and the environment and returns status information and rewards, and the master end performs real-time control and decision-making on the manipulator operation; S3: According to the state information transmitted by the data link, the delay processing DIP module of the master end processes the current state of the master end to obtain the current state and the previous order reward without the influence of the communication delay between the master and the slave end; S4: Based on the state after removing the delay effect, the deep reinforcement learning DRL decision module of the master side continuously updates the agent according to the existing experience replay pool, action replay pool and current delay, and obtains a new state through the interaction between the agent and the environment to guide the robot arm to plan the trajectory; S5: The delay processing module is used to remove the delay effect of the slave data through the data link, and the data without the effect is used to update the intelligent agent and interact with the environment, gradually guiding the slave robot arm to complete the planning task, wherein the intelligent agent generates corresponding actions according to the state and strategy network after removing the delay effect, and the slave space robot arm interacts with the environment according to the action, generates the next state and reward, and then feeds back to the master end, and the master end updates the intelligent agent according to the new state and reward.

2. The trajectory planning method for a teleoperated space manipulator based on deep reinforcement learning according to claim 1, characterized in that: In step S1, establishing a rigid body kinematics model of a floating base space robot with a rigid manipulator specifically includes: Define the joint angle vector of the manipulator in the space robot system, which represents the position state of the manipulator at any time; Define the velocity vector corresponding to the joint angle vector to represent the dynamic motion state of the robot; Construct the Jacobian matrix of the base and the robot to describe the mathematical relationship between the motion of the robot and the motion of the base; Determine the conservation relationship between the linear momentum and angular momentum of the space robot in the absence of external forces, and express it through the inertia matrix and the coupled inertia matrix; Based on the conservation laws and Jacobian matrix, the generalized Jacobian matrix is ​​determined, which relates the kinematic and dynamic parameters of the robot arm for accurate calculation of subsequent trajectory planning.

3. The trajectory planning method for a teleoperated space manipulator based on deep reinforcement learning according to claim 1 or 2, characterized in that: In step S1, the definition of the reinforcement learning six-element tuple specifically includes: Design the state space, including the manipulator's joint angle vectors, joint velocity vectors, end effector position, end effector velocity, target position, and distance between the end effector and the target, to capture the manipulator's dynamic and task-related information; Designing the action space, represented as a torque vector applied to the joints of the robot arm, wherein each component of the torque vector is limited to a predetermined safety range to ensure the safety and effectiveness of the robot arm action; Design a state transfer function that predicts the next state of the robot in the environment based on the current state and the action performed, so that the agent can learn the impact of its behavior on the environment; Design a reward function that evaluates the value of actions taken based on the distance between the end effector and the target, the speed of the arm and the base, and the task completion to motivate the agent to take actions to maximize the cumulative reward; Introduce a discount factor to adjust and balance the impact of immediate rewards and future rewards on current decisions to balance short-term and long-term benefits; Implement an experience replay mechanism to store the sequence of states, actions, rewards, and new states during the agent’s interaction with the environment so that the agent can learn and improve its strategy from historical data.

4. The trajectory planning method for a teleoperated space manipulator based on deep reinforcement learning as claimed in claim 3, characterized in that: The reward function includes: a mechanism that rewards the end effector for reducing the distance between the end effector and the target point, so as to encourage the agent to perform an action to move the end effector toward the target point; when the end effector approaches the target point, it is rewarded for its behavior of hovering near the target to ensure continuous progress toward the target; the agent is rewarded for smoothing the movement of the robotic arm and reducing the speed fluctuation of the base and the end effector to optimize the smoothness of the trajectory; a terminal reward mechanism is activated when the distance between the end effector and the target point is less than or equal to a preset threshold, and its value is proportional to the number of steps remaining when the task is completed, so as to motivate the agent to complete the task efficiently; through the comprehensive evaluation of the reward function, the agent learns to take appropriate actions at different operation stages, including approaching the target, fine-tuning and completing the task, so as to optimize the entire teleoperation process.

5. The trajectory planning method for a teleoperated space manipulator based on deep reinforcement learning according to any one of claims 1 to 4, characterized in that: The overall framework of teleoperation constructed in step S2 specifically includes: Identify time delay issues in teleoperation environments, including action delay, observation delay, and reward delay, and their impact on the agent’s decision-making process; The loop time delay RTD is used as a benchmark to measure the communication delay, ensuring that the time difference between the master-side observed state and the slave-side actual state is quantified and compensated; Determine the ideal RTD value based on the transmission distance so as to perform delay compensation at the master end; Handle random delays caused by environmental factors and provide adaptive support for the decision-making of intelligent agents by simulating the statistical distribution of random delays; Taking into account both constant and random delays, a strategy is developed to enable the agent to make decisions at each time step based on the observations from the master.

6. The method for trajectory planning of a teleoperated space manipulator based on deep reinforcement learning according to any one of claims 1 to 5, characterized in that: In step S3, the current state of the master end is processed by one or more of a mapping method, a prediction method and a state enhancement method; The mapping method adopts a memoryless strategy, ignores delays, and takes the most recently observed state as the true state of the environment for decision making; The prediction method uses historical trajectory data to train a forward model; The state enhancement method transforms the delayed Markov decision process into a non-delayed Markov decision process by constructing an information state consisting of delayed state information and historical action sequences.

7. The trajectory planning method for a teleoperated space manipulator based on deep reinforcement learning according to claim 6, characterized in that: In step S3, the mapping method specifically includes: using a memoryless strategy to process the observed state of the master end, ignoring the delay effect caused by communication delay; using the latest observed state as the real state of the environment at the current moment for the decision-making process of the intelligent agent; when the state observed by the master end is inconsistent with the expected state, identifying and confirming the existence of an unobserved state; when the unobserved state is identified, replacing the current state with the state at the previous moment, and adjusting the immediate reward to reflect the replacement; The prediction method specifically includes: using a large amount of historical trajectory data obtained by interacting with the environment, including tuples of state, action, reward and next state, to train a forward model through supervised learning to simulate system dynamics; using the forward model as a nonlinear function, inputting the current state and action, and outputting the predicted reward and next state to model the future behavior of the system; using the last observed state and historical action sequence, iteratively calculating the predicted current state and reward of the previous state through the forward model; and performing the decision-making process of the intelligent agent according to the predicted current state to generate corresponding actions and guide the robot arm to perform the next operation; The state enhancement method specifically includes: constructing an information state that combines delayed state information and a historical action sequence to simulate a decision-making environment without delay; when defining a new state, for an observed delayed state, combining it with a series of historical actions to form an information state representing the current environment state; adjusting the reward function, when a delayed state is observed, setting the immediate reward to zero to reflect the missing information due to the delay; and correcting the initial state distribution to include a random action sequence taken when no remote state update is received, so that the agent can make decisions when the information is incomplete.

8. The method for trajectory planning of a teleoperated space manipulator based on deep reinforcement learning according to any one of claims 1 to 7, characterized in that: In step S4, the DRL decision module specifically includes: Design the objective function of the master to maximize the cumulative reward, which is based on the delayed state after DIP processing and the distribution of previously sampled states and actions, that is, the data in the replay buffer; Through the objective function, policy entropy is introduced to encourage agent exploration and control the randomness of the optimal strategy to avoid local optimal solutions. Using the optimization method based on the SAC algorithm, the state-action value function, state value function and policy function are approximated by neural networks respectively; Update the parameters of the value function by minimizing the soft Bellman residual; Update the parameters of the soft value function by minimizing the squared residual; Update the parameters of the policy network by minimizing the KL divergence; The policy network is constructed using a reparameterization method, which allows the policy network to generate actions in the presence of noise; The management strategy of the replay buffer only stores real state data, and non-real data should not be stored in the experience replay pool.

9. The trajectory planning method for a teleoperated space manipulator based on deep reinforcement learning according to any one of claims 1 to 8, characterized in that: Through the remote environment interaction module, the action instructions generated by the agent are executed in the environment, and the environmental feedback caused by the action is collected, including new state information and rewards, and then the feedback is passed to the DRL decision module on the host side.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the trajectory planning method of a teleoperated space robot arm based on deep reinforcement learning as described in any one of claims 1 to 9.

Images