Rrt guided space manipulator reinforcement learning path optimization control method

Abstract

The application provides an RRT-guided space manipulator reinforcement learning path optimization control method and belongs to the technical field of manipulator control. The application generates a collision-free initial path in a joint space by using an RRT algorithm, models a space manipulator-base system as a Markov decision process on the basis, constructs a local reinforcement learning training task through adjacent path points, executes and constructs an experience replay pool along the path, and reoptimizes a strategy from a starting point to a target point on a global scale by using an actor-critic architecture. The application is used for realizing integrated intelligent optimization of path planning and motion control.

Description

Technical Field

[0001] This invention belongs to the field of robotic arm control technology, and particularly relates to an RRT-guided reinforcement learning path optimization control method for spatial robotic arms. Background Technology

[0002] Space robotic arms, as key actuators in on-orbit servicing and maintenance tasks, have been gradually applied to various mission scenarios. Space robotic arms often need to perform delicate operations in complex, unstructured, or even confined space environments, such as grasping, disassembling, or repairing target components. These operations not only require the end effector's path to have strict obstacle avoidance safety, but also require minimal base attitude disturbance, smooth joint movements, and continuous speed changes during the overall motion, thus balancing mission safety, control performance, and system lifespan.

[0003] Existing path planning methods for space robotic arms are mainly based on sampling or graph search algorithms, such as Rapid Random Tree (RRT) and Improved Rapid Random Tree (RRT). ), Probability Roadmarks (PRM) and A Such methods search for collision-free paths from the initial configuration to the target configuration by randomly sampling in the configuration space or constructing a landmark map. Among them, RRT... The algorithm introduces path reconnection and cost re-evaluation mechanisms based on RRT, exhibiting asymptotic optimality and finding wide application in path planning under complex environments. However, existing RRT-based path planning methods still have the following problems and shortcomings in applications with space robotic arms:

[0004] (1) It is difficult to fully characterize the nonholonomic and dynamic constraints of the space manipulator. The space manipulator is generally installed on a free-floating or controlled base. The system as a whole is affected by multiple factors such as momentum conservation, attitude control actuators (such as flywheels and control moment gyroscopes), and joint drives, and exhibits obvious nonholonomic constraint characteristics. Existing planning methods such as RRT usually perform path search at the geometric level or on a simplified kinematic model, which only ensures the feasibility of the path within the geometric space of the obstacle. It is difficult to incorporate dynamic factors such as nonholonomic constraints, base attitude disturbances, actuator saturation, and joint velocity / acceleration constraints into the planning process in a timely and accurate manner. As a result, the obtained path often requires additional trajectory correction in the actual execution stage, and may not even be directly used for control tracking.

[0005] (2) The planning result is geometrically feasible but not control-friendly, with significant changes in attitude and velocity. The path generated by RRT is essentially composed of a large number of discrete nodes and polyline segments. Its design goal is mainly to shorten the geometric path cost, rather than to optimize the attitude stability or velocity smoothness in the control process. In a narrow environment, although the path achieves obstacle avoidance in a geometric sense, there may be large attitude jumps, sudden changes in joint angles, and velocity changes between adjacent path points, which can easily cause significant disturbances in the base attitude and structural vibrations, which are not conducive to the stable operation of the equipment. At the same time, overly abrupt movements will also increase drive energy consumption and mechanical shock, reducing system reliability. It can be seen that simply emphasizing geometrically optimal path planning is difficult to meet the comprehensive requirements of "executability" and "control friendliness" for space robotic arms.

[0006] (3) The path planning and motion control processes are separated, lacking a unified optimization mechanism. In existing technologies, path planning and control execution often adopt a serial architecture: first, algorithms such as RRT are used to search for collision-free paths in the configuration space, and then trajectory planning and control algorithms (such as polynomial interpolation, time parameterization, traditional feedback control, or model predictive control) track the path. This phased design of "planning first, then controlling" means that the control layer can only make local adjustments near the given path, lacking the ability to simultaneously balance multiple objectives such as obstacle avoidance safety, base attitude stability, speed and attitude smoothness, and energy consumption from the perspective of the overall task, thus limiting the overall performance improvement space.

[0007] (4) The foundation for reinforcement learning applications is weak, and there is a lack of effective integration with traditional planning results. In recent years, reinforcement learning has shown its potential to automatically learn control strategies in high-dimensional state spaces and complex constraint environments in the field of robot decision-making and control. Theoretically, reinforcement learning can directly optimize strategies for task objectives by interacting with the environment, and is expected to play a role in the joint planning and control of space robotic arms. However, there are still many difficulties in directly applying reinforcement learning to the global path planning and motion control of space robotic arms: on the one hand, the trial and error cost in the real space environment is extremely high, making it difficult to explore strategies through a large number of online interactions; on the other hand, if the global strategy is learned from scratch without using any prior planning results, the learning difficulty is high, the convergence process is slow, and the safety is difficult to guarantee. In existing related work, there is still a lack of a technical solution that fully utilizes the initial path generated by classic planning algorithms such as RRT to effectively "guide" and "constrain" reinforcement learning, thereby achieving strategy optimization under the premise of ensuring safety and feasibility.

[0008] (5) Joint path-motion optimization methods for narrow and highly constrained environments are still imperfect. In typical narrow environments, space robotic arms are often subject to multiple constraints simultaneously when performing tasks, such as obstacle geometry constraints, base posture constraints, end-effector pointing constraints, and joint velocity / acceleration, with extremely small available safety margins. In such scenarios, relying solely on geometric obstacle avoidance is insufficient to guarantee the safety and stability of the actual execution process; while smoothing the trajectory from a local control perspective is difficult to overcome the limitations of the predetermined path shape. Existing methods generally lack a systematic solution that can uniformly model and optimize the control actions of the entire path and its adjacent nodes based on traditional planning results, and continuously improve the quality of the path and actions through a learning mechanism.

[0009] In summary, existing spatial robotic arm path planning and control technologies suffer from the following common problems in complex, unstructured, and narrow environments: difficulty in fully integrating incompleteness and dynamic constraints during the path planning stage; a significant disconnect between the geometric feasibility of the planned path and the smoothness and stability of control execution; a lack of effective methods to guide reinforcement learning for strategy optimization using traditional planning results; and the absence of an integrated intelligent control framework from "initial path planning" to "action-level control optimization". Summary of the Invention

[0010] In view of the above-mentioned technical status, this invention proposes an RRT-guided reinforcement learning path optimization control method for spatial robotic arms.

[0011] The technical problems solved by this invention specifically involve the following aspects.

[0012] (1) To address the technical problem of difficulty in integrating non-holonomic and dynamic constraints in the path planning stage, existing path planning methods for space robotic arms based on RRT mostly perform search and optimization at the geometric or simplified kinematic level. They fail to systematically introduce dynamic constraints such as the momentum coupling relationship of the space robot-base system, the constraint characteristics of the base attitude control actuator, the maximum speed / acceleration of joint drives, and torque saturation into the planning process, which may result in the planning results being infeasible or difficult to execute at the dynamic level. This invention, while retaining the geometric search capability of RRT, models the decision-making process of the space robotic arm as a Markov Decision Process (MDP) and introduces a control optimization mechanism based on reinforcement learning. This allows the planned path to be actually executed under the condition of considering non-holonomic and dynamic constraints, thereby improving the dynamic feasibility and controllability of the path.

[0013] (2) To address the technical problem of drastic attitude and velocity changes during the execution of geometrically feasible paths, existing RRT-generated paths consist of discrete configuration nodes and polyline segments. The optimization objectives are mainly focused on geometric indicators such as path length, without specifically optimizing control-related indicators such as joint angle jumps, joint angular velocity changes, and base attitude disturbances. This leads to problems such as sudden joint motion changes, large changes in base attitude, and structural vibrations during actual tracking. Traditional trajectory interpolation and feedback control are unable to eliminate these adverse characteristics from the source. This invention constructs the motion process between adjacent nodes on the path as a local reinforcement learning task, explicitly introducing constraints on joint angular velocity, joint angle changes, and base attitude changes into the reward function. It learns a local control strategy that achieves slow motion and smooth attitude changes, thereby significantly reducing attitude jumps and velocity fluctuations during path execution while ensuring obstacle avoidance safety.

[0014] (3) To address the technical problem of the separation between path planning and motion control, which prevents unified optimization, existing technologies generally adopt a serial structure of "RRT geometric planning + trajectory tracking control". Path planning and motion control are independent of each other, and the control layer can only make local corrections near the given path. It is impossible to simultaneously optimize performance indicators such as path shape, attitude stability, velocity distribution, and energy consumption from a global perspective, thus limiting the overall system performance. This invention constructs a joint optimization framework with the RRT initial path as a priori and reinforcement learning strategy as the core. By combining local policy learning, path traversal execution, and experience playback, it achieves integrated optimization from path planning to motion control, enabling the path and control parameters to be improved collaboratively within the same framework.

[0015] (4) To address the technical problems of high learning difficulty, high sample requirements, and insufficient safety in the training process of reinforcement learning end-to-end control, existing schemes that directly use reinforcement learning to achieve end-to-end control of spatial robotic arms usually do not utilize traditional planning results as priors. They conduct unguided exploration in a high-dimensional, strongly constrained state-action space, resulting in slow policy convergence speed, large training sample requirements, and high-risk behaviors such as collisions in complex environments such as slits in the early stages of training, which is difficult to meet the requirements of engineering practice for training efficiency and safety. This invention uses the collision-free initial path generated in the joint space by RRT as a safe trajectory skeleton, decomposing the originally global high-dimensional exploration into a local policy learning task along the known safe path, thereby reducing the search space dimension and learning difficulty; in addition, by executing the policy along the safe path and collecting a quadruple of (state, action, reward, next state), an experience replay pool containing safety priors and task priors is constructed, so that reinforcement learning training is mainly based on offline experience data, reducing the dependence on high-risk online trial and error processes and improving the controllability and safety of the training stage.

[0016] (5) Addressing the technical problem of lacking a joint path-action global optimization mechanism for narrow and multi-constraint environments, in typical narrow and multi-constraint environments, the space robot is simultaneously subjected to multiple constraints such as obstacle geometry constraints, base posture constraints, end-effector pointing constraints, and joint motion constraints. Existing methods either only plan paths for geometric obstacle avoidance or only perform local control smoothing around a predetermined path, without establishing a technical approach to iteratively improve and globally optimize the entire path and its corresponding action sequence using a learning mechanism. Based on the RRT initial path and local smoothing strategy, this invention uniformly stores the quadruples generated along the path and reconstructed into an experience replay pool, and uses an actor-critic architecture reinforcement learning algorithm to retrain the overall strategy from the starting point to the target point offline, thereby achieving global re-optimization of the entire path and action control, and improving the execution safety, posture stability, and control performance of the space robot in narrow and multi-constraint environments.

[0017] (6) To address the technical problem of the lack of intelligent control enhancement mechanisms that can be directly integrated with existing RRT planning modules, some existing reinforcement learning control schemes often require a complete replacement of the existing control architecture, leaving the task entirely to the end-to-end decision-making of the learning strategy. This makes it difficult to achieve a smooth connection with the deployed RRT path planning module and traditional control module without significantly altering the existing engineering system, thus affecting engineering application and promotion. This invention, without changing the basic structure and interface of the existing RRT path planning module, superimposes an intelligent control enhancement layer based on MDP and actor-critic structure on its output path. It progressively improves the performance of the existing system through "initial planning + learning optimization," enabling the invention to be easily embedded into the existing space robotic arm control process, achieving engineering integrability and scalability.

[0018] This invention proposes an RRT-guided reinforcement learning path optimization control method for a space robotic arm, comprising:

[0019] S1. Perform MDP modeling on the space robot. MDP stands for Markov Decision Process.

[0020] The control process of a space robot performing tasks in orbit is modeled as an MDP, represented by a six-tuple, including the state space, action space, reward function, state transition function, initial state distribution, and discount factor; the space robot consists of a robotic arm with n degrees of freedom mounted on a base and its base;

[0021] S2. Perform initial path planning in the joint space using the RRT algorithm;

[0022] Given a target point and the initial configuration of the robotic arm in joint space, the target configuration in joint space is solved based on the target point. The RRT algorithm is used to perform path planning for the path between the initial configuration and the target configuration (initial point, target point), and the success of the planning is determined.

[0023] If the planning is successful, the planned path will be used as the initial path in the collision-free joint space; if the planning fails, an intermediate configuration will be selected between the initial configuration and the target configuration. For the two path segments (initial point, intermediate point) and (intermediate point, target point), the RRT algorithm will be used to perform path planning and further determine whether the planning is successful.

[0024] If planning still fails, an intermediate configuration is selected between the two failed configurations, and the RRT algorithm is used to perform path planning. The above process is repeated until multiple planned paths between the initial configuration and the target configuration are obtained and spliced ​​together to form an initial path in the collision-free joint space.

[0025] S3. Train local reinforcement learning strategies for adjacent path points on the initial path;

[0026] The initial path is divided into multiple groups of local motion subtasks and start and end condition constraints are constructed. An environment for training local reinforcement learning policies is configured, and local reinforcement learning is used to train the local control policies of the local motion subtasks.

[0027] S4. Plan and execute the trained policy along the initial path and build an experience replay pool;

[0028] The trained policy is used to traverse the initial path in the collision-free joint space generated by the RRT algorithm and collect running data to build an experience replay pool and determine sample data for global policy optimization. The sample data is represented by a quadruple, including: state, action, reward, and next state.

[0029] S5. Perform global policy optimization using sample data from the experience replay pool;

[0030] Based on sample data in the experience replay pool, a reinforcement learning algorithm based on the actor-critic architecture is used to globally optimize the trained policy used from the initial configuration to the target configuration, and the globally optimized policy is used to execute the path control of the robotic arm.

[0031] In S1, the MDP is represented by a six-tuple, including the state space, action space, reward function, state transition function, initial state distribution, and discount factor; where:

[0032] The state in the state space is defined as a vector consisting of the following components: the joint angle vector consisting of the angles of each joint of the robotic arm, the joint angular velocity vector consisting of the angular velocities of each joint of the robotic arm, the position of the end effector of the robotic arm in the task space, the linear velocity and / or angular velocity of the end effector of the robotic arm, the position of the base center of mass in the selected reference coordinate system, the velocity of the base center of mass, the position of the task target, and the distance or relative position information between the end effector of the robotic arm and the task target.

[0033] The motion space consists of the control torques applied to each joint of the robotic arm. At each discrete decision moment, the motion is defined as a vector of joint control torques, with each component corresponding to the driving torque or equivalent control input of each joint.

[0034] The reward function describes the quality of a decision given a state and action. Its determining factors include: the distance or relative error between the robotic arm and the task target, the magnitude of the joint angular velocity, the degree of deviation of the base posture and its angular velocity, the change in joint angle between adjacent time steps, and the result of whether a collision occurs with an obstacle.

[0035] The state transition function represents the probability distribution of transitioning to the current state given the state and action of the previous time step;

[0036] The initial state distribution describes the range of state values ​​and probability distribution characteristics of a space robot at the start of a task. Its determining factors include: the initial configuration of the robotic arm joints, the initial posture and velocity of the base, and the initial position of the target. Each task execution or each training round samples from the initial state that meets the task requirements.

[0037] The discount factor describes the importance of future rewards to the current decision, and its value ranges from [0,1].

[0038] In S2, the target configuration in joint space is solved; specifically, this includes:

[0039] The target pose information of the end effector of the robotic arm is obtained. Based on the kinematic model of the robotic arm, the inverse kinematics solution method is used to calculate multiple sets of joint angle solutions corresponding to the target pose of the end effector, which are used as candidate joint angle solutions and candidate configurations are formed.

[0040] Based on the obstacle distribution and joint angle constraints in the task environment, a feasibility test is performed on each candidate configuration. Specifically, if a candidate configuration collides geometrically with an obstacle or does not meet the joint angle constraints, the candidate configuration is determined to be an infeasible solution and is eliminated.

[0041] From the candidate configurations that have passed the feasibility test, the target configuration is determined by random selection in order to further perform initial path planning in the joint space.

[0042] In S2, the RRT algorithm is used to perform path planning; specifically, it includes:

[0043] After obtaining the initial and target configurations, the joint space is constructed as the search space for the RRT algorithm. The search space is bounded by the range of values ​​for each joint angle of the robotic arm, and combined with the constraint of mapping obstacles to feasible configurations in the joint space, a configuration space with obstacle regions and feasible regions is formed. With the initial configuration as the root node of the tree and the target configuration as the planning endpoint, the RRT algorithm is executed to perform path search. The search process is as follows:

[0044] Candidate joint configuration points are generated by random sampling within the joint space;

[0045] Select the tree node that is closest to the candidate joint configuration point generated by sampling from the existing tree nodes, and expand along the tree node to generate a new node;

[0046] Perform collision detection and constraint checks on the newly generated nodes;

[0047] The new node is added to the tree structure, and a reconnection operation is performed in its neighborhood. The connection relationship is optimized based on the cost function to determine the cost-optimal path.

[0048] When the tree structure expands in the configuration space to connect the initial configuration and the target configuration and satisfies the preset cost threshold condition or iteration number condition, a collision-free joint space discrete path from the initial configuration to the target configuration is obtained by backtracking from the tree. The collision-free joint space discrete path is composed of several ordered joint configuration points.

[0049] In S2, an intermediate configuration is selected between the initial configuration and the target configuration; specifically, this includes:

[0050] Between the initial configuration and the target configuration, one or more intermediate configurations are selected as transition nodes according to the task space geometry. The intermediate configurations correspond to the intermediate postures or the intermediate points of posture transitions after the robotic arm bypasses the obstacles.

[0051] The (initial point, target point) path planning task is divided into two sub-planning tasks, including: the (initial point, intermediate point) sub-planning task from the initial configuration to the intermediate configuration and the (intermediate point, target point) sub-planning task from the intermediate configuration to the target configuration.

[0052] For each sub-planning task, the RRT algorithm is called in the joint space to perform path search to obtain a planned path. The planned paths obtained from each sub-planning task are then concatenated in sequence to obtain a global joint space path from the initial configuration to the target configuration, which serves as the initial path for the joint space (initial point, target point).

[0053] In S3, the initial path is divided into multiple groups of local motion subtasks and start-end condition constraints are constructed; specifically including:

[0054] The initial path is represented as a set of ordered joint configuration points. For any two adjacent configuration points, the former configuration point is taken as the starting configuration of the local subtask, and the latter configuration point is taken as the corresponding ending configuration.

[0055] By traversing the ordered joint configuration point sequence, a set of starting points and a corresponding set of ending points are obtained. Each starting point configuration corresponds to a unique ending point configuration. The starting point-ending point configuration pairs together constitute the initial condition set and termination condition set of the environment for training local reinforcement learning strategies, which are used to define multiple sets of local motion subtasks.

[0056] In S3, local reinforcement learning is used to train the local control policy for the local motion sub-task; specifically, this includes:

[0057] Randomly select a starting configuration from the set of starting points to construct the initial state, and take the corresponding ending configuration as the task objective. When the state reaches the ending configuration, the current local motion subtask is determined to be completed.

[0058] The corresponding action is output based on the current state. This action refers to the control torque applied to each joint of the robotic arm. The next state is obtained by updating through dynamic simulation. A reward is generated based on the current state, the corresponding action, and the environment. The state-action-reward-next state is recorded as a sample, and the policy network and value network are updated accordingly.

[0059] Repeat the training of local control strategies for local motion subtasks multiple times until the training cutoff condition is met.

[0060] In S4, sample data is represented by quadruples, including: state, action, reward, and next state; where:

[0061] The states in the quadruple include: the joint angles and angular velocities of the robotic arm, the position and velocity of the robotic arm's end effector, the position and velocity of the base's center of mass, the target position, and the relative distance between the robotic arm's end effector and the target.

[0062] In a quadruple, the motion refers to the control torque applied to each joint of the robotic arm;

[0063] The factors that determine the reward in the quadruple set include: change in target distance, joint angular velocity, deviation of base posture and magnitude of angular velocity, change in joint angle between adjacent moments, and obstacle collision.

[0064] In S4, the quadruplet sample data is reconstructed with a reward based on the global evaluation after the traversal process is completed, and the reconstructed quadruplet sample data is stored in the experience replay pool. The global evaluation includes: the error between the final configuration of the robotic arm end effector and the target configuration, and the smoothness of joint motion and base posture changes.

[0065] In S5, global policy optimization uses batch sampling and gradient optimization, utilizing quadruples of sample data and new quadruples of sample data continuously generated during the global policy optimization process to perform parameter updates.

[0066] In summary, this invention proposes an RRT-guided reinforcement learning path optimization control method for a spatial robotic arm. This method first generates a collision-free initial path in joint space using RRT. Based on this, the spatial robotic arm-base system is modeled as a Markov Decision Process (MDP). A local reinforcement learning training task is constructed using adjacent path points, and then executed along the path, building an experience replay pool. Finally, an actor-critic architecture is used to further optimize the strategy from the starting point to the target point on a global scale, achieving integrated intelligent optimization of path planning and motion control.

[0067] Compared with the prior art, the present invention has the following beneficial effects.

[0068] (1) Improve dynamic feasibility and execution controllability: By modeling the space robotic arm-base system as an MDP and using joint torque as the action output, dynamic quantities such as joint speed and base state are introduced to participate in decision-making, making the control strategy generated on the basis of RRT geometric path more in line with nonholonomic constraints and dynamic constraints, which is convenient for smooth execution in actual systems.

[0069] (2) Significantly improves the smoothness and attitude stability of path execution: Construct local reinforcement learning tasks using adjacent path points, and focus on constraining joint angular velocity, joint angle change and base attitude disturbance in the reward function, so that the same geometrically collision-free path exhibits smoother speed and gentler attitude change during execution, reducing mechanical shock and base vibration.

[0070] (3) Improve learning efficiency and safety while ensuring obstacle avoidance: Based on the initial path and local smoothing strategy of RRT, execute along the path, collect and store high-quality quadruplets to build an experience replay pool, and use these samples with safety and smoothing priors for subsequent online learning to update, reduce the collision risk caused by random trial and error, and improve the convergence speed of the strategy and the stability of the training process.

[0071] (4) Achieve integrated global optimization of path planning and control, and facilitate engineering integration: Through global re-optimization based on experience playback by the actor-critic, the originally separate "path planning" and "trajectory control" are unified into the same learning framework, making the overall path execution effect better than the original RRT scheme. At the same time, this method uses the existing RRT planning module as the front end, and only adds the learning control link to its output path, which is convenient for direct integration and gradual upgrading with the existing space robotic arm control system. Attached Figure Description

[0072] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0073] Figure 1 This is a flowchart illustrating the reinforcement learning path optimization control method for a space robotic arm guided by RRT according to an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] This invention proposes an RRT-guided reinforcement learning path optimization control method for a spatial robotic arm, such as... Figure 1 As shown, the method includes:

[0076] S1. Perform MDP modeling on the space robot. MDP stands for Markov Decision Process.

[0077] The control process of a space robot performing tasks in orbit is modeled as an MDP, represented by a six-tuple, including the state space, action space, reward function, state transition function, initial state distribution, and discount factor; the space robot consists of a robotic arm with n degrees of freedom mounted on a base and its base;

[0078] S2. Perform initial path planning in the joint space using the RRT algorithm;

[0079] Given a target point and the initial configuration of the robotic arm in joint space, the target configuration in joint space is solved based on the target point. The RRT algorithm is used to perform path planning for the path between the initial configuration and the target configuration (initial point, target point), and the success of the planning is determined.

[0080] If the planning is successful, the planned path will be used as the initial path in the collision-free joint space; if the planning fails, an intermediate configuration will be selected between the initial configuration and the target configuration. For the two path segments (initial point, intermediate point) and (intermediate point, target point), the RRT algorithm will be used to perform path planning and further determine whether the planning is successful.

[0081] If planning still fails, an intermediate configuration is selected between the two failed configurations, and the RRT algorithm is used to perform path planning. The above process is repeated until multiple planned paths between the initial configuration and the target configuration are obtained and spliced ​​together to form an initial path in the collision-free joint space.

[0082] S3. Train local reinforcement learning strategies for adjacent path points on the initial path;

[0083] The initial path is divided into multiple groups of local motion subtasks and start and end condition constraints are constructed. An environment for training local reinforcement learning policies is configured, and local reinforcement learning is used to train the local control policies of the local motion subtasks.

[0084] S4. Plan and execute the trained policy along the initial path and build an experience replay pool;

[0085] The trained policy is used to traverse the initial path in the collision-free joint space generated by the RRT algorithm and collect running data to build an experience replay pool and determine sample data for global policy optimization. The sample data is represented by a quadruple, including: state, action, reward, and next state.

[0086] S5. Perform global policy optimization using sample data from the experience replay pool;

[0087] Based on sample data in the experience replay pool, a reinforcement learning algorithm based on the actor-critic architecture is used to globally optimize the trained policy used from the initial configuration to the target configuration, and the globally optimized policy is used to execute the path control of the robotic arm.

[0088] In S1, the MDP is represented by a six-tuple, including the state space, action space, reward function, state transition function, initial state distribution, and discount factor; where:

[0089] The state in the state space is defined as a vector consisting of the following components: the joint angle vector consisting of the angles of each joint of the robotic arm, the joint angular velocity vector consisting of the angular velocities of each joint of the robotic arm, the position of the end effector of the robotic arm in the task space, the linear velocity and / or angular velocity of the end effector of the robotic arm, the position of the base center of mass in the selected reference coordinate system, the velocity of the base center of mass, the position of the task target, and the distance or relative position information between the end effector of the robotic arm and the task target.

[0090] The motion space consists of the control torques applied to each joint of the robotic arm. At each discrete decision moment, the motion is defined as a vector of joint control torques, with each component corresponding to the driving torque or equivalent control input of each joint.

[0091] The reward function describes the quality of a decision given a state and action. Its determining factors include: the distance or relative error between the robotic arm and the task target, the magnitude of the joint angular velocity, the degree of deviation of the base posture and its angular velocity, the change in joint angle between adjacent time steps, and the result of whether a collision occurs with an obstacle.

[0092] The state transition function represents the probability distribution of transitioning to the current state given the state and action of the previous time step;

[0093] The initial state distribution describes the range of state values ​​and probability distribution characteristics of a space robot at the start of a task. Its determining factors include: the initial configuration of the robotic arm joints, the initial posture and velocity of the base, and the initial position of the target. Each task execution or each training round samples from the initial state that meets the task requirements.

[0094] The discount factor describes the importance of future rewards to the current decision, and its value ranges from [0,1].

[0095] In S2, the target configuration in joint space is solved; specifically, this includes:

[0096] The target pose information of the end effector of the robotic arm is obtained. Based on the kinematic model of the robotic arm, the inverse kinematics solution method is used to calculate multiple sets of joint angle solutions corresponding to the target pose of the end effector, which are used as candidate joint angle solutions and candidate configurations are formed.

[0097] Based on the obstacle distribution and joint angle constraints in the task environment, a feasibility test is performed on each candidate configuration. Specifically, if a candidate configuration collides geometrically with an obstacle or does not meet the joint angle constraints, the candidate configuration is determined to be an infeasible solution and is eliminated.

[0098] From the candidate configurations that have passed the feasibility test, the target configuration is determined by random selection in order to further perform initial path planning in the joint space.

[0099] In S2, the RRT algorithm is used to perform path planning; specifically, it includes:

[0100] After obtaining the initial and target configurations, the joint space is constructed as the search space for the RRT algorithm. The search space is bounded by the range of values ​​for each joint angle of the robotic arm, and combined with the constraint of mapping obstacles to feasible configurations in the joint space, a configuration space with obstacle regions and feasible regions is formed. With the initial configuration as the root node of the tree and the target configuration as the planning endpoint, the RRT algorithm is executed to perform path search. The search process is as follows:

[0101] Candidate joint configuration points are generated by random sampling within the joint space;

[0102] Select the tree node that is closest to the candidate joint configuration point generated by sampling from the existing tree nodes, and expand along the tree node to generate a new node;

[0103] Perform collision detection and constraint checks on the newly generated nodes;

[0104] The new node is added to the tree structure, and a reconnection operation is performed in its neighborhood. The connection relationship is optimized based on the cost function to determine the cost-optimal path.

[0105] When the tree structure expands in the configuration space to connect the initial configuration and the target configuration and satisfies the preset cost threshold condition or iteration number condition, a collision-free joint space discrete path from the initial configuration to the target configuration is obtained by backtracking from the tree. The collision-free joint space discrete path is composed of several ordered joint configuration points.

[0106] In S2, an intermediate configuration is selected between the initial configuration and the target configuration; specifically, this includes:

[0107] Between the initial configuration and the target configuration, one or more intermediate configurations are selected as transition nodes according to the task space geometry. The intermediate configurations correspond to the intermediate postures or the intermediate points of posture transitions after the robotic arm bypasses the obstacles.

[0108] The (initial point, target point) path planning task is divided into two sub-planning tasks, including: the (initial point, intermediate point) sub-planning task from the initial configuration to the intermediate configuration and the (intermediate point, target point) sub-planning task from the intermediate configuration to the target configuration.

[0109] For each sub-planning task, the RRT algorithm is called in the joint space to perform path search to obtain a planned path. The planned paths obtained from each sub-planning task are then concatenated in sequence to obtain a global joint space path from the initial configuration to the target configuration, which serves as the initial path for the joint space (initial point, target point).

[0110] In S3, the initial path is divided into multiple groups of local motion subtasks and start-end condition constraints are constructed; specifically including:

[0111] The initial path is represented as a set of ordered joint configuration points. For any two adjacent configuration points, the former configuration point is taken as the starting configuration of the local subtask, and the latter configuration point is taken as the corresponding ending configuration.

[0112] By traversing the ordered joint configuration point sequence, a set of starting points and a corresponding set of ending points are obtained. Each starting point configuration corresponds to a unique ending point configuration. The starting point-ending point configuration pairs together constitute the initial condition set and termination condition set of the environment for training local reinforcement learning strategies, which are used to define multiple sets of local motion subtasks.

[0113] In S3, local reinforcement learning is used to train the local control policy for the local motion sub-task; specifically, this includes:

[0114] Randomly select a starting configuration from the set of starting points to construct the initial state, and take the corresponding ending configuration as the task objective. When the state reaches the ending configuration, the current local motion subtask is determined to be completed.

[0115] The corresponding action is output based on the current state. This action refers to the control torque applied to each joint of the robotic arm. The next state is obtained by updating through dynamic simulation. A reward is generated based on the current state, the corresponding action, and the environment. The state-action-reward-next state is recorded as a sample, and the policy network and value network are updated accordingly.

[0116] Repeat the training of local control strategies for local motion subtasks multiple times until the training cutoff condition is met.

[0117] In S4, sample data is represented by quadruples, including: state, action, reward, and next state; where:

[0118] The states in the quadruple include: the joint angles and angular velocities of the robotic arm, the position and velocity of the robotic arm's end effector, the position and velocity of the base's center of mass, the target position, and the relative distance between the robotic arm's end effector and the target.

[0119] In a quadruple, the motion refers to the control torque applied to each joint of the robotic arm;

[0120] The factors that determine the reward in the quadruple set include: change in target distance, joint angular velocity, deviation of base posture and magnitude of angular velocity, change in joint angle between adjacent moments, and obstacle collision.

[0121] In S4, the quadruplet sample data is reconstructed with a reward based on the global evaluation after the traversal process is completed, and the reconstructed quadruplet sample data is stored in the experience replay pool. The global evaluation includes: the error between the final configuration of the robotic arm end effector and the target configuration, and the smoothness of joint motion and base posture changes.

[0122] In S5, global policy optimization uses batch sampling and gradient optimization, utilizing quadruples of sample data and new quadruples of sample data continuously generated during the global policy optimization process to perform parameter updates.

[0123] In summary, this invention proposes an RRT-guided reinforcement learning path optimization control method for a spatial robotic arm. This method first generates a collision-free initial path in joint space using RRT. Based on this, the spatial robotic arm-base system is modeled as an MDP. Local reinforcement learning training tasks are constructed using adjacent path points, and then executed along the path to build an experience replay pool. Finally, an actor-critic architecture is used to further optimize the strategy from the starting point to the target point at a global scale, achieving integrated intelligent optimization of path planning and motion control.

[0124] In a specific embodiment of the present invention, a reinforcement learning path optimization control method for a space robotic arm guided by Reinforcement Learning (RRT) is provided, comprising the following steps:

[0125] S1: Space Robot MDP Modeling. The decision-making process of the space robotic arm is modeled as an MDP, defining the state, joint torque actions, and a reward function that includes target distance, velocity, and collision penalty;

[0126] S2: RRT joint space initial path planning. The target joint configuration is obtained through inverse kinematics, and a collision-free initial path is planned in joint space using RRT (introducing intermediate points in segments if necessary);

[0127] S3: Training of local reinforcement learning strategies for adjacent path points. A set of local start and end conditions is constructed using two adjacent points on the path to train a local control strategy under constraints of slow motion and small attitude changes.

[0128] S4: Execute along the RRT path and build the experience replay pool. Utilize the pre-trained policy to execute along the initial path, record a four-element array of (state, action, reward, next state), and reconstruct the reward based on the final goal before storing it in the experience replay pool;

[0129] S5: Global policy re-optimization based on actor-critic. An actor-critic reinforcement learning algorithm is used on the experience replay pool to retrain the global path and action control policy from the starting point to the target point.

[0130] In this embodiment, regarding the MDP modeling of the space robot, the control process of the space robot performing tasks in orbit is modeled as an MDP, which can be formally represented as a six-tuple, including the state space, action space, reward function, state transition function, initial state distribution, and discount factor. This modeling approach integrates the sequential decision-making process of the space robot into a unified reinforcement learning framework, facilitating subsequent policy learning and optimization.

[0131] Regarding the structure and mission characteristics of the space robot, the space robot includes a device mounted on a base that has... A robotic arm with multiple degrees of freedom and its base are coupled together through joints and dynamic constraints to form an integrated system. When this system performs tasks such as target acquisition, grasping, assembly, or maintenance in orbit, its control process exhibits a sequential decision-making process that evolves over time. Current control decisions will affect future states and rewards, making it suitable for modeling using Multidimensional Design Principles (MDP).

[0132] The state space is used to describe the kinematic and dynamic states of a space robot system at any discrete moment and its relative relationship with the target. Specifically, the state is defined as a vector consisting of the following components: (1) the joint angle vector consisting of the angles of each joint of the robot arm; (2) the joint angular velocity vector consisting of the angular velocities of each joint of the robot arm; (3) the position of the end effector of the robot arm in the task space (optionally combined with attitude parameters); (4) the linear velocity and / or angular velocity of the end effector of the robot arm; (5) the position of the center of mass of the base in the selected reference coordinate system; (6) the velocity of the center of mass of the base (including linear velocity and optional angular velocity); (7) the position of the target (e.g., the position of the target point or the center of mass of the target object); (8) the distance or relative position information between the end effector of the robot arm and the target. This multidimensional state vector can simultaneously reflect the joint state of the robot arm itself, the motion state of the end effector, the motion state of the base, and the relative relationship with the target, thus providing sufficient state information for decision-making.

[0133] The motion space consists of the control torques applied to the joints of the robotic arm. Specifically, at each discrete decision moment, the motion is designed as a vector of joint torques, with each component corresponding to the driving torque or equivalent control input of each joint. This motion definition method can be directly used as input to the dynamic equations of the space robot, ensuring the executability of the control strategy at the dynamic level.

[0134] The reward function is used to characterize the quality of decision-making under a given state and action. The reward function is designed based on the degree of achievement of the task objective, the smoothness of motion, and safety. The reward is related to the following factors: (1) the distance or relative error between the target and the target, which is used to encourage the end effector to gradually approach the target position; (2) the magnitude of the joint angular velocity, which is used to suppress high-speed joint movement and reduce impact and energy consumption; (3) the degree of deviation of the base posture and the magnitude of its angular velocity, which is used to constrain the base posture disturbance and improve system stability; (4) the change in joint angle between adjacent moments, which is used to suppress abrupt changes in joint posture and promote smooth overall motion; (5) the result of whether a collision occurs with an obstacle, which applies a significant negative reward to the collision behavior to ensure obstacle avoidance safety. By combining the above items in the form of a weighted or functional combination to form an instantaneous reward signal, the learned strategy is both conducive to the achievement of the task objective and takes into account posture stability and safety.

[0135] A state transition function represents the probability distribution of a system transitioning to its current state given the previous state and actions. The state transition function approximates the dynamic equations of a space robot system, describing the mapping from "previous state - current action" to "current state" based on the dynamic model of the robotic arm and base, joint actuation characteristics, and external environmental constraints.

[0136] The initial state distribution describes the range and probabilistic characteristics of a space robot's state at the start of a task. This distribution can be determined by the specific task scenario, such as specifying the initial configuration of the robotic arm joints, the initial attitude and velocity of the base, and the initial position of the target, ensuring that each task execution or training round samples from an initial state that meets the task requirements. By setting the initial state distribution, typical initial conditions can be covered, improving the adaptability of the learned strategy to various initial conditions.

[0137] The discount factor reflects the importance of future rewards to the current decision, and its value typically ranges from [0,1). A larger discount factor makes the policy focus more on long-term cumulative rewards, encouraging the agent to optimize task completion quality and motion smoothness over a longer time scale; a smaller discount factor makes the policy more focused on current or near-term rewards. A suitable discount factor should be selected based on the duration of the space robot mission and the degree of focus on long-term performance (e.g., final arrival accuracy, overall attitude stability). Value, to balance immediate gains and long-term gains.

[0138] Through the above MDP modeling, this invention links the state description, control input, task objective and dynamic evolution process of a space robot under a unified mathematical framework, providing a clear formal basis for subsequent reinforcement learning-based policy optimization, path improvement and action control.

[0139] In this embodiment, regarding the initial path planning in the joint space, to improve the feasibility and safety of path planning for the space robot in complex environments, given the initial joint configuration and target point information of the space robotic arm, the following steps should be taken to complete the path planning under the fixed base: First, the target configuration in the joint space should be solved based on the target point; second, based on the obtained initial and target configurations, the RRT algorithm should be used to complete the collision-free joint space path planning; furthermore, considering the situation where the RRT algorithm is ineffective due to the complex, unstructured, and narrow characteristics of the environment, a suitable intermediate configuration point should be manually found between the initial and target configurations, and the RRT algorithm should be used to plan the path for the two paths (initial point, intermediate point) and (intermediate point, target point) respectively until a feasible path is obtained.

[0140] Solve the joint space of the target configuration. In the task scenario, the posture of the space robot's base is approximately fixed during the planning stage, or the influence of its posture changes on the end effector accuracy is negligible. Therefore, the path planning problem can be simplified to a robot arm motion planning problem under fixed base conditions.

[0141] Under these conditions, the target pose information of the robotic arm's end effector in the task space is first given according to the task requirements, such as the position and (optionally) orientation of the target point. Then, based on the kinematic model of the robotic arm, the inverse kinematics solution method is used to solve one or more sets of joint angle solutions corresponding to the target pose of the end effector. Given that spatial robotic arms often exhibit multiple solution redundancy, i.e., the same end effector pose corresponds to multiple joint angle configurations, this invention performs a feasibility test on each candidate joint angle solution in combination with the obstacle distribution and joint angle constraints in the task environment: if the robotic arm configuration corresponding to a certain joint angle solution has a geometric collision with environmental obstacles or does not meet the joint physical constraints, then the configuration is determined as an infeasible solution and eliminated; from the remaining set of feasible configuration solutions, the target joint configuration can be determined by random selection or selection based on a simple heuristic (such as small joint deviation from the initial configuration), which serves as the target state for path planning.

[0142] Perform RRT path planning in the joint space under a fixed base. After obtaining the initial joint configuration of the space robot (e.g., the configuration to be executed before the robotic arm performs the task) and the target joint configuration mentioned above, the joint angle space is constructed as the search space for the RRT algorithm. This search space is bounded by the range of values ​​for each joint angle of the robotic arm, and combined with feasible configuration constraints mapped from the task space obstacles to the joint space, forming a high-dimensional configuration space with obstacle regions and feasible regions. In this configuration space, the initial joint configuration is used as the root node of the tree, and the target joint configuration is used as the planning endpoint, and the RRT algorithm is executed to search for the path.

[0143] Path search specifically includes:

[0144] (1) Randomly sample candidate joint configuration points within the joint space;

[0145] (2) Select the node that is closest to the sampling point in a certain metric from the existing tree nodes, and try to expand along this direction to generate new nodes;

[0146] (3) Perform collision detection and constraint checks on newly generated nodes to ensure that their corresponding robotic arm configurations do not collide with obstacles and meet joint angle limits;

[0147] (4) Under the premise of feasibility, the new node is connected to the tree structure and reconnection is performed in its neighborhood. The connection relationship is optimized based on the cost function (such as path length or joint angle change) to gradually approach the path with better cost.

[0148] When the tree structure is expanded in the configuration space to connect the initial configuration and the target configuration, and the preset cost threshold or iteration number condition is met, a collision-free joint space discrete path from the starting configuration to the target configuration can be obtained by backtracking from the tree. This path consists of several ordered joint configuration nodes.

[0149] Regarding the intermediate node segmentation planning strategy, in actual task environments, due to factors such as complex obstacle distribution, strict joint constraints, and the dynamic coupling effect of the base-manipulator, it may be difficult to find a feasible path within a limited time or a limited number of samplings using the aforementioned direct "start-end" joint space RRT planning. To address this situation, this invention further introduces an intermediate node segmentation planning strategy to improve the success rate and feasibility of joint space path planning.

[0150] Specifically, it includes:

[0151] (1) Between the initial joint configuration and the target joint configuration, one or more intermediate joint configurations are selected as transition nodes based on the task space geometry, empirical rules, or heuristic methods. This intermediate configuration may correspond to the "intermediate posture" of the robotic arm after it avoids the obstacle or an intermediate point where the posture transition is relatively natural;

[0152] (2) Divide the original overall “start-to-target” path planning task into several sub-planning tasks, for example: Sub-task 1: from the starting joint configuration to the intermediate joint configuration (“start-to-intermediate point”); Sub-task 2: from the intermediate joint configuration to the target joint configuration (“intermediate point-to-target point”); If necessary, multiple intermediate configurations can be further introduced to decompose the entire path into multiple sub-paths.

[0153] (3) For each sub-planning task, the RRT algorithm is called in the joint space to perform path search. That is, in the local configuration space with the starting configuration of the sub-task as the root and the ending configuration as the target, the above random sampling, node expansion, collision detection and path reconnection process is repeated to obtain the collision-free discrete path of each segment.

[0154] Once all sub-planning tasks have successfully generated feasible paths, the sub-paths such as "start point-intermediate point" and "intermediate point-target point" are sequentially concatenated to obtain a global joint space path from the initial configuration to the target configuration. This path ensures that collision-free constraints are satisfied within each segment. Furthermore, the introduction of intermediate configurations helps to avoid complex obstacle areas, improving the success rate and feasibility of the overall path planning.

[0155] Regarding path output and subsequent use, through the above steps, this invention obtains one or more initial joint space paths composed of discrete joint configuration nodes under the assumption of a fixed base. This path guarantees geometrically no collision with obstacles, and each node satisfies joint constraints.

[0156] This invention uses the initial path in joint space as the basis for subsequent reinforcement learning training and policy optimization: on the one hand, this path provides a starting and ending configuration pair for local reinforcement learning tasks; on the other hand, as a safety trajectory skeleton, it can be used to generate empirical data with safety priors, supporting subsequent retraining and optimization of global policies. Thus, this step effectively combines RRT joint space initial path planning with the spatial robot task context, providing reliable initial conditions for subsequent learning-based path optimization and motion control.

[0157] In this embodiment, local reinforcement learning policy training is performed at adjacent path points. After obtaining the initial discrete path in the joint space, the path is refined into a series of local motion sub-tasks, and reinforcement learning is used to train the local control policy. Unlike conventional methods, the initial state of each round of local training is randomly sampled from the set of starting points, and each starting point corresponds to a unique endpoint configuration, thereby improving the generalization ability of the policy while ensuring the clarity of the task structure.

[0158] Regarding the partitioning of local motion subtasks and the construction of the start and end condition sets, the discrete joint space path obtained based on the RRT algorithm can be represented as a sequence of ordered joint configuration nodes. For any two adjacent discrete configuration points in the path, the former configuration is denoted as the starting configuration of the local subtask, and the latter configuration is denoted as the corresponding ending configuration. By traversing the entire path, a set of local starting points and a corresponding set of ending points can be obtained, that is, each starting configuration corresponds to a unique ending configuration. This pair of "start-end" configurations together constitutes the initial and ending condition sets of the local reinforcement learning training environment, which are used to define multiple sets of local motion subtasks.

[0159] Regarding the construction of the local reinforcement learning training environment, this invention does not fix a specific local path segment for sequential training during local training. Instead, it constructs the initial conditions for each round of training by randomly sampling from the starting point set: at the beginning of each round of local training, a starting point configuration is randomly selected from the starting point set as the initial configuration for that round of training; the corresponding endpoint configuration is found in the configuration pair set from this starting point configuration, and this endpoint configuration is used as the target state or termination condition for the current local task; this "starting point-endpoint" configuration pair constitutes the start and end constraints for the current round of local reinforcement learning training. In specific implementation, the state can be sampled with small-range perturbations near the starting point configuration to simulate small deviations in actual execution, but the starting point index for each round of training always comes from random sampling from the starting point set, and the endpoint is the endpoint configuration uniquely corresponding to that starting point, thereby ensuring a clear task structure and training coverage of all segments of the path. In each training session, the space robot still uses the MDP state and motion description defined in step one: the state includes joint angles, joint angular velocities, end-effector pose and velocity, base center of mass position and velocity, target position, and distance between the end-effector and the target, etc., and the motion is the torque vector of each joint of the space robot arm. The state transition is automatically achieved by the space robot after it has completed the motion in the simulation environment (or safety verification environment).

[0160] Regarding the adjustment of the reward function for slow, smooth, and obstacle-avoidance tasks, within each local "start-end" task, this invention guides the strategy to meet the requirements of slow motion, minimal posture changes, and reduced obstacle collisions while completing local transitions by making targeted weight adjustments to the reward function. Based on the global MDP reward structure, the specific adjustments include: increasing the weight of penalty terms related to joint angular velocity to suppress high-speed joint movements; increasing the weight of penalty terms related to base posture deviation and angular velocity magnitude to constrain base posture perturbations; increasing the weight of penalty terms related to joint angle changes at adjacent time points to suppress abrupt joint posture changes and promote smooth local transitions; and providing significant negative rewards for states approaching or colliding with obstacles to strengthen obstacle avoidance constraints. Through these reward designs, this invention can guide the agent to learn a control strategy that balances stability and safety within the local task "from the starting point to its corresponding end point" in each round of local training with randomly selected starting point configurations.

[0161] Regarding the reinforcement learning training mechanism for local control strategies, the execution process of each local task during training is as follows: A starting configuration is randomly selected from the starting point set to construct an initial state; the corresponding ending configuration is used as the task objective or termination condition. When the state approaches or reaches the ending configuration (or meets a set error threshold), the task is determined to be completed or terminated. During the execution of the local task, the agent outputs joint torque actions based on the current state, updates the next state through dynamic simulation, and generates an immediate reward based on the current state, action, and environmental conditions. The "state-action-reward-next state" sample is recorded, and the policy network and value network are updated according to the selected reinforcement learning algorithm (such as a continuous control algorithm based on an actor-critic structure). By repeating multiple rounds of local task training (with a pre-set maximum number of training iterations), and ensuring that the starting point of each task is randomly selected from the starting point set and the corresponding ending point is uniquely determined by the starting point, it is possible to guarantee that the policy receives sufficient training on different local path segments, thereby improving the generalization ability and stability of the policy across the entire path.

[0162] Regarding the integration of training results from multiple local subtasks and the formation of the policy network, as training progresses, the policy network is continuously updated and optimized on multiple randomly selected start-end local tasks, gradually forming a unified control policy that can adapt to each local segment of the entire RRT initial path.

[0163] This invention employs a local training mechanism of "random sampling of the starting point set + one-to-one correspondence with the endpoint constraint," which ensures that: firstly, different path segments are fully covered during training, preventing the policy from overfitting to only a few local regions; secondly, the policy continuously encounters various local start-end configurations under the same network parameters, facilitating the extraction of control laws with stronger generalization and robustness. The resulting policy network can then serve as the initial policy foundation for subsequent steps such as execution along the initial path, construction of the experience replay pool, and global re-optimization based on the actor-critic structure, achieving a natural transition from local stationary control to integrated global path and control optimization.

[0164] In this embodiment, a path retracing along the RRT is performed, and an experience replay pool is constructed. After completing the training of the aforementioned local reinforcement learning policy at adjacent path points and obtaining the corresponding control policy network, the complete joint space path generated by the previous RRT is retraced. System operation data is collected in a simulation environment (or a safety verification environment), and an experience replay pool is constructed to provide high-quality initial four-tuple samples for subsequent online reinforcement learning training. The purpose of this step is to pre-store a batch of experience data with good security and smoothness for the subsequent online learning stage without relying on a large amount of random trial and error, thereby improving the efficiency and stability of online learning. The specific process is as follows:

[0165] Regarding execution along the RRT path based on a trained strategy, firstly, the space robot system is initialized to the same initial joint configuration and base state as before. A policy network trained through local reinforcement learning is then loaded into a simulation environment or a verification environment with safety measures. Subsequently, using the discrete joint spatial path generated by RRT as a reference, each adjacent configuration interval is traversed sequentially according to the path node order, starting from the initial configuration. On each path segment, the current policy network outputs joint torque actions based on the system state, driving the space robot arm and base to evolve according to the dynamic equations, thus completing the execution of the entire initial RRT path at the dynamic level. Through this process, control can be achieved using a strategy with a certain degree of smoothing capability and obstacle avoidance awareness, based on a "geometrically safe path," thereby reducing the occurrence of obviously unreasonable or high-risk trajectories and resulting in relatively high-quality collected experience samples.

[0166] Regarding the acquisition and recording of quadruple data, during the execution along the RRT path, this invention records the system interaction process at each discrete decision moment, generating quadruple samples for reinforcement learning. Specifically, this includes: obtaining the current state through simulation or dynamic updates, which includes the joint angles and angular velocities of the robotic arm, the end effector position and velocity, the base center of mass position and velocity, the target position, and the relative distance between the end effector and the target; inputting the current state into the policy network to obtain the current action, i.e., the torque vector applied to each joint; calculating the reward according to the previously defined instantaneous reward function, which is related to factors such as the change in the current target distance, joint angular velocities, base posture deviation and angular velocity magnitude, changes in joint angles at adjacent moments, and obstacle collisions; and obtaining the next state after performing the action in the current state. Thus, a quadruple (state, action, reward, next state) is constructed at each decision moment. The complete path execution process from the initial configuration to the target configuration will form one or more quadruple sequences arranged in chronological order.

[0167] Regarding the optional reconstruction of the reward signal based on the global results, in order to better reflect the overall task effect in subsequent online learning, this invention can appropriately reconstruct the reward signal in the above four-tuple when necessary to introduce global task target information. Specifically, after completing a full path execution from the starting point to the target point, a global evaluation can be performed on the execution trajectory, such as: the error between the end point and the target configuration at the final moment; the smoothness of joint movements and base posture changes throughout the process (such as peak velocity and attitude disturbance amplitude); whether obstacle collisions or dangerous approach situations occurred during the execution. Based on this, the reward sequence on this trajectory can be uniformly adjusted according to the overall execution quality (for example, by weighting and correcting the original immediate reward, or by superimposing a component related to the global completion quality), thereby obtaining a corrected reward that takes into account both local dynamic performance and global task effect.

[0168] Regarding the construction and use of the experience replay pool, after completing the above data collection and reward reconstruction, this invention stores all quadruples into the experience replay pool. This experience replay pool has the following characteristics and functions: the stored samples originate from the execution process along the initial RRT path under the control of the trained policy, thus providing a certain level of quality assurance in terms of obstacle avoidance, safety, and motion smoothness; the replay pool contains fragments of multiple complete task trajectories, covering different regions and state distributions along the path from the starting point to the target point; these samples will serve as the initial experience set for subsequent online reinforcement learning stages, used for experience sampling before or in the early stages of online interaction, thereby reducing a large number of invalid or high-risk attempts caused by completely random exploration, improving the stability of the policy in the early stages of online updates, and accelerating the convergence speed of value estimation and policy updates during online training.

[0169] In this embodiment, a global policy re-optimization based on the actor-critic architecture is performed. Once the experience replay pool contains high-quality quadruple samples collected from execution along the initial RRT path, a reinforcement learning algorithm based on the actor-critic architecture is used to retrain and globally optimize the overall policy from the starting point to the target point. This step aims to leverage prior experience and online interactive data to uniformly improve the smoothness, safety, and target achievement quality of path execution from a holistic task perspective.

[0170] Regarding the construction of an actor-critic reinforcement learning structure, a neural network is used to construct a policy network (actor) and a value evaluation network (critic). The policy network takes the current system state as input and outputs continuous control actions, such as torque vectors applied to each joint, representing the current control policy. The value evaluation network provides a value estimate for a given state or state-action pair, evaluating the expected reward of the current policy in that state and providing a reference signal for policy improvement. The two work together: the critic provides performance evaluation and directional corrections to the actor, and the actor updates its output policy based on the critic's feedback, thereby achieving iterative policy improvement.

[0171] Regarding sample sampling and network updates based on the experience replay pool, during retraining, the algorithm continuously and randomly samples from the experience replay pool, including quadruples of state, action, immediate reward, and subsequent state. These samples are used as training data to update the critic network, enabling it to more accurately estimate the value of various states (or state-action pairs) under the current policy. The critic's value assessment is used to construct an improvement direction for action selection, thereby updating the parameters of the actor network, making the actor more inclined to choose actions with higher long-term rewards and smoother safety in a given state. Training employs a general process of mini-batch sampling and gradient optimization, repeatedly updating parameters on experience samples and newly generated online samples. As the actor-critic algorithm is continuously trained on experience replay pool samples and online interaction samples, the policy network continuously improves at the global scale, forming a complete solution with greater smoothness, stability, and practical feasibility compared to the original RRT.

[0172] The policy network interacts with the environment, taking actions based on the current state returned by the environment. When an action is executed, the environment returns the state and reward for the next time step. During this process, the value estimation network evaluates the quality of the action. The experience replay pool used for training is not initially empty; in the early stages of training, it stores path data generated by the global reward planning (RRT) that conforms to the global reward plan. Data generated during the interaction between the policy network and the environment is continuously added to the experience replay pool as new four-tuples. Simultaneously, this data is collected in mini-batches and updated using the steepest descent method to update both the policy network and the value estimation network, thereby helping them perform the task better and ultimately achieving path planning results superior to RRT.

[0173] In summary, the main technical improvements of this invention include: (1) RRT-guided initial collision-free path generation mechanism in joint space: an initial path that satisfies obstacle constraints is generated in the joint space of the space manipulator using RRT (including intermediate configuration segmentation planning) and used as a safe prior trajectory skeleton for reinforcement learning; (2) MDP modeling and local subtask construction mechanism for space manipulator: the space manipulator-base system is modeled as an MDP and adjacent configuration pairs of RRT paths are divided into local start-end subtasks to form a set of local reinforcement learning training environments; (3) Local reinforcement learning control strategy design based on "slow motion + stable posture + reduced collision": by designing the weights of specific terms in the reward function (joint angular velocity, joint angle change, base posture disturbance, obstacle approach / collision), a smooth and safe control strategy for local path segments is learned; (4) Mechanism for executing along the RRT path and constructing a high-quality experience replay pool. The locally trained policy is executed along the RRT path, and the (state, action, reward, next state) quadruple is collected and (optionally reconstructed) stored to provide high-quality initial experience for subsequent online learning; (5) A global path and control integrated re-optimization mechanism based on the actor-critic architecture. Based on the experience replay pool, the actor-critic reinforcement learning algorithm is used to re-optimize the complete policy from the starting point to the target point from a global perspective, so as to realize the integrated intelligent optimization of path planning and action control.

[0174] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims.

Claims

1. A reinforcement learning path optimization control method for a space robotic arm guided by Reinforcement Learning (RRT), characterized in that, The method includes: S1. Perform MDP modeling on the space robot. MDP stands for Markov Decision Process. The control process of a space robot performing tasks in orbit is modeled as an MDP, represented by a six-tuple, including the state space, action space, reward function, state transition function, initial state distribution, and discount factor; the space robot consists of a robotic arm with n degrees of freedom mounted on a base and its base; S2. Perform initial path planning in the joint space using the RRT algorithm; Given a target point and the initial configuration of the robotic arm in joint space, the target configuration in joint space is solved based on the target point. The RRT algorithm is used to perform path planning for the path between the initial configuration and the target configuration (initial point, target point), and the success of the planning is determined. If the planning is successful, the planned path will be used as the initial path in the collision-free joint space; if the planning fails, an intermediate configuration will be selected between the initial configuration and the target configuration. For the two path segments (initial point, intermediate point) and (intermediate point, target point), the RRT algorithm will be used to perform path planning and further determine whether the planning is successful. If planning still fails, an intermediate configuration is selected between the two failed configurations, and the RRT algorithm is used to perform path planning. The above process is repeated until multiple planned paths between the initial configuration and the target configuration are obtained and spliced ​​together to form an initial path in the collision-free joint space. S3. Train local reinforcement learning strategies for adjacent path points on the initial path; The initial path is divided into multiple groups of local motion subtasks and start and end condition constraints are constructed. An environment for training local reinforcement learning policies is configured, and local reinforcement learning is used to train the local control policies of the local motion subtasks. In each training session, the space robot uses the MDP state and motion description in S1. The motion is the torque vector of each joint of the space robot arm. The state transition is automatically achieved by the space robot after it has completed the motion in the simulation environment. S4. Plan and execute the trained policy along the initial path and build an experience replay pool; The trained policy is used to traverse the initial path in the collision-free joint space generated by the RRT algorithm and collect running data to build an experience replay pool and determine sample data for global policy optimization. The sample data is represented by a quadruple, including: state, action, reward, and next state. S5. Perform global policy optimization using sample data from the experience replay pool; Based on sample data in the experience replay pool, a reinforcement learning algorithm based on the actor-critic architecture is used to globally optimize the trained policy used from the initial configuration to the target configuration, and the globally optimized policy is used to execute the path control of the robotic arm.

2. The method for optimizing the path of a space robotic arm guided by reinforcement learning according to claim 1, characterized in that, In S1, the MDP is represented by a six-tuple, including the state space, action space, reward function, state transition function, initial state distribution, and discount factor; where: The state in the state space is defined as a vector consisting of the following components: the joint angle vector consisting of the angles of each joint of the robotic arm, the joint angular velocity vector consisting of the angular velocities of each joint of the robotic arm, the position of the end effector of the robotic arm in the task space, the linear velocity and / or angular velocity of the end effector of the robotic arm, the position of the base center of mass in the selected reference coordinate system, the velocity of the base center of mass, the position of the task target, and the distance or relative position information between the end effector of the robotic arm and the task target. The motion space consists of the control torques applied to each joint of the robotic arm. At each discrete decision moment, the motion is defined as a vector of joint control torques, with each component corresponding to the driving torque or equivalent control input of each joint. The reward function describes the quality of a decision given a state and action. Its determining factors include: the distance or relative error between the robotic arm and the task target, the magnitude of the joint angular velocity, the degree of deviation of the base posture and its angular velocity, the change in joint angle between adjacent time steps, and the result of whether a collision occurs with an obstacle. The state transition function represents the probability distribution of transitioning to the current state given the state and action of the previous time step; The initial state distribution describes the range of state values ​​and probability distribution characteristics of a space robot at the start of a task. Its determining factors include: the initial configuration of the robotic arm joints, the initial posture and velocity of the base, and the initial position of the target. Each task execution or each training round samples from the initial state that meets the task requirements. The discount factor describes the importance of future rewards to the current decision, and its value ranges from [0,1].

3. The method for optimizing the path of a space robotic arm guided by reinforcement learning according to claim 2, characterized in that, In S2, the target configuration in joint space is solved; specifically, this includes: The target pose information of the end effector of the robotic arm is obtained. Based on the kinematic model of the robotic arm, the inverse kinematics solution method is used to calculate multiple sets of joint angle solutions corresponding to the target pose of the end effector, which are used as candidate joint angle solutions and candidate configurations are formed. Based on the obstacle distribution and joint angle constraints in the task environment, a feasibility test is performed on each candidate configuration. Specifically, if a candidate configuration collides geometrically with an obstacle or does not meet the joint angle constraints, the candidate configuration is determined to be an infeasible solution and is eliminated. From the candidate configurations that have passed the feasibility test, the target configuration is determined by random selection in order to further perform initial path planning in the joint space.

4. The method for optimizing the path of a space robotic arm guided by reinforcement learning according to claim 3, characterized in that, In S2, the RRT algorithm is used to perform path planning; specifically, it includes: After obtaining the initial and target configurations, the joint space is constructed as the search space for the RRT algorithm. The search space is bounded by the range of values ​​for each joint angle of the robotic arm, and combined with the constraint of mapping obstacles to feasible configurations in the joint space, a configuration space with obstacle regions and feasible regions is formed. With the initial configuration as the root node of the tree and the target configuration as the planning endpoint, the RRT algorithm is executed to perform path search. The search process is as follows: Candidate joint configuration points are generated by random sampling within the joint space; Select the tree node that is closest to the candidate joint configuration point generated by sampling from the existing tree nodes, and expand along the tree node to generate a new node; Perform collision detection and constraint checks on the newly generated nodes; The new node is added to the tree structure, and a reconnection operation is performed in its neighborhood. The connection relationship is optimized based on the cost function to determine the cost-optimal path. When the tree structure expands in the configuration space to connect the initial configuration and the target configuration and satisfies the preset cost threshold condition or iteration number condition, a collision-free joint space discrete path from the initial configuration to the target configuration is obtained by backtracking from the tree. The collision-free joint space discrete path is composed of several ordered joint configuration points.

5. The method for optimizing the path of a space robotic arm guided by reinforcement learning according to claim 4, characterized in that, In S2, an intermediate configuration is selected between the initial configuration and the target configuration; specifically, this includes: Between the initial configuration and the target configuration, one or more intermediate configurations are selected as transition nodes according to the task space geometry. The intermediate configurations correspond to the intermediate postures or the intermediate points of posture transitions after the robotic arm bypasses the obstacles. The (initial point, target point) path planning task is divided into two sub-planning tasks, including: the (initial point, intermediate point) sub-planning task from the initial configuration to the intermediate configuration and the (intermediate point, target point) sub-planning task from the intermediate configuration to the target configuration. For each sub-planning task, the RRT algorithm is called in the joint space to perform path search to obtain a planned path. The planned paths obtained from each sub-planning task are then concatenated in sequence to obtain a global joint space path from the initial configuration to the target configuration, which serves as the initial path for the joint space (initial point, target point).

6. The method for optimizing the path of a space robotic arm guided by reinforcement learning according to claim 5, characterized in that, In S3, the initial path is divided into multiple groups of local motion subtasks and start-end condition constraints are constructed; specifically including: The initial path is represented as a set of ordered joint configuration points. For any two adjacent configuration points, the former configuration point is taken as the starting configuration of the local subtask, and the latter configuration point is taken as the corresponding ending configuration. By traversing the ordered joint configuration point sequence, a set of starting points and a corresponding set of ending points are obtained. Each starting point configuration corresponds to a unique ending point configuration. The starting point-ending point configuration pairs together constitute the initial condition set and termination condition set of the environment for training local reinforcement learning strategies, which are used to define multiple sets of local motion subtasks.

7. The method for optimizing the path of a space robotic arm guided by reinforcement learning according to claim 6, characterized in that, In S3, local reinforcement learning is used to train the local control policy for the local motion sub-task; specifically, this includes: Randomly select a starting configuration from the set of starting points to construct the initial state, and take the corresponding ending configuration as the task objective. When the state reaches the ending configuration, the current local motion subtask is determined to be completed. The corresponding action is output based on the current state. This action refers to the control torque applied to each joint of the robotic arm. The next state is obtained by updating through dynamic simulation. A reward is generated based on the current state, the corresponding action, and the environment. The state-action-reward-next state is recorded as a sample, and the policy network and value network are updated accordingly. Repeat the training of local control strategies for local motion subtasks multiple times until the training cutoff condition is met.

8. The method for optimizing the path of a space robotic arm guided by reinforcement learning according to claim 7, characterized in that, In S4, sample data is represented by quadruples, including: state, action, reward, and next state; where: The states in the quadruple include: the joint angles and angular velocities of the robotic arm, the position and velocity of the robotic arm's end effector, the position and velocity of the base's center of mass, the target position, and the relative distance between the robotic arm's end effector and the target. In a quadruple, the motion refers to the control torque applied to each joint of the robotic arm; The factors that determine the reward in the quadruple set include: change in target distance, joint angular velocity, deviation of base posture and magnitude of angular velocity, change in joint angle between adjacent moments, and obstacle collision.

9. The method for optimizing the path of a space robotic arm guided by reinforcement learning according to claim 8, characterized in that, In S4, the quadruplet sample data is reconstructed with a reward based on the global evaluation after the traversal process is completed, and the reconstructed quadruplet sample data is stored in the experience replay pool. The overall evaluation includes: the error between the final configuration of the robotic arm end effector and the target configuration, and the smoothness of joint motion and base posture changes.

10. The method for optimizing the path of a space robotic arm guided by reinforcement learning according to claim 9, characterized in that, In S5, global policy optimization uses batch sampling and gradient optimization, utilizing quadruples of sample data and new quadruples of sample data continuously generated during the global policy optimization process to perform parameter updates.

Images