A trajectory planning method and system for a free-floating space manipulator
By combining PPO reinforcement learning and potential field guidance, expert guidance signals are generated and risk gating fusion is performed, which solves the problems of high collision cost and scarcity of feasible trajectories in trajectory planning of free-floating space robotic arms, and realizes safe and efficient trajectory planning and end-effector pose control.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUN YAT SEN UNIVERSITY SHENZHEN
- Filing Date
- 2026-06-04
- Publication Date
- 2026-07-14
AI Technical Summary
Existing deep reinforcement learning-based robotic arm trajectory planning methods suffer from high collision costs and a scarcity of feasible trajectories in narrow-gap grasping tasks, resulting in a small number of effective samples, slow convergence, or even difficulty in convergence in the early stages of training, making it difficult to achieve safe and efficient trajectory planning.
A PPO reinforcement learning strategy is adopted, combined with stochastic expansion and potential field guidance methods, to generate expert guidance signals. Expert guidance actions and candidate actions are fused through a risk gating mechanism to construct a multi-objective composite reward function, optimize the policy network parameters, and output joint velocity control commands.
It reduces blind random exploration in the early stages of training, increases the proportion of effective samples, reduces the risk of collisions in complex gap environments, and enhances the safety and autonomous optimization capabilities of trajectory planning.
Smart Images
Figure CN122378732A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robotic arm control, and more particularly to a trajectory planning method and system for a free-floating robotic arm. Background Technology
[0002] With the development of on-orbit servicing and space debris removal technologies, capturing out-of-control non-cooperative targets (such as malfunctioning satellites and defunct rocket upper stages) using space robotic arms has become a critical task for ensuring the safety of space assets. In actual on-orbit capture missions, the solar panels of non-cooperative targets are usually in a deployed or semi-deployed state, and the exterior of the target often has a complex array of protrusions such as antennas, payloads, and thermal control components. The end effector of the robotic arm must precisely navigate through these obstacles composed of fragile attachments to safely reach the effective grasping surface, such as the satellite's docking ring or nozzle interface.
[0003] Unlike obstacle avoidance in ordinary open spaces, this type of task for a free-floating space robotic arm has the following characteristics: First, the collision cost is extremely high, and contact with fragile attachments may cause sudden changes in the target's attitude or even secondary debris; second, the feasible channel geometric margin is extremely small, and the success rate of the trajectory is low; third, the servicing spacecraft base is in a free-floating state, and the movement of the robotic arm will cause a reaction from the base through the conservation of momentum and angular momentum, thus making the originally narrow gap constraint exhibit dynamic time-varying characteristics relative to the robotic arm base; fourth, in addition to positional accuracy, the end effector must also meet attitude alignment requirements, making trajectory planning significantly more difficult than conventional obstacle avoidance tasks for fixed-base robotic arms.
[0004] Existing technologies mainly rely on deep reinforcement learning methods. However, pure deep reinforcement learning faces significant cold start difficulties in scenarios with high collision costs and scarce successful samples, and often struggles to obtain effective gradient guidance in the early stages of training. Summary of the Invention
[0005] In view of this, in order to address the technical problems of existing deep reinforcement learning-based robotic arm trajectory planning methods in grasping tasks with narrow gaps, which result in high collision costs and scarce feasible trajectories, leading to a small number of effective samples, slow convergence, or even failure to converge in the early stages of training, this invention proposes, in a first aspect, a trajectory planning method for robotic arms in free-floating spaces, the method comprising the following steps: The system collects task and environmental parameters to construct the current system state and obstacle constraint information. It then performs pose transformation and state reconstruction on the system state to form a state representation usable for reinforcement learning. Combining stochastic expansion and potential field guidance methods, it generates expert guidance signals using the reinforcement learning state and obstacle constraints. Based on the reinforcement learning state, it outputs candidate actions using a PPO reinforcement learning strategy. The expert-guided actions and candidate actions are fused through a risk gating mechanism to obtain the final execution action. The corresponding reward value is calculated based on the multi-objective composite reward function and the final execution action. The reward, task parameters, and environmental parameters are used as sampling samples to update the policy network parameters until the policy network training is complete. The trained policy network is deployed on the space robotic arm controller. After inputting the real-time observation state, it outputs joint speed control commands to complete the on-orbit grasping trajectory planning task in complex structural gaps.
[0006] In addition to the above method, the present invention also proposes a trajectory planning system for a free-floating space robotic arm, which includes an initialization unit, a conversion unit, an expert guidance unit, a PPO reinforcement learning unit, an action fusion unit, a reward calculation unit, a model training unit, and an inference unit.
[0007] Based on the above scheme, this invention provides a trajectory planning method and system for a free-floating space robotic arm. By setting up an APF-Bi-RRT* online expert guidance module, a collision-free landmark sequence in the joint space is generated during training, and further expert-guided actions that can directly participate in shared control are output. This reduces blind random exploration in the early stages of training, increases the proportion of effective samples, and reduces the collision risk in complex gap environments. Through a risk-gated shared control mechanism, the fusion weight between expert-guided actions and reinforcement learning strategy actions is simultaneously affected by collision risk, channel margin, base disturbance, and training progress. In high-risk, narrow-channel, or base-disturbance scenarios, expert intervention is enhanced, while in low-risk scenarios, expert intervention is reduced, thus balancing safety and policy autonomous optimization capabilities in the trajectory planning process. Attached Figure Description
[0008] Figure 1 This is a flowchart of the trajectory planning method for a free-floating space robotic arm according to the present invention; Figure 2 This is a schematic diagram of the robotic arm MDH according to a specific embodiment of the present invention. Detailed Implementation
[0009] In addition to the deep reinforcement learning methods mentioned in the background, existing technologies also include artificial potential field methods and RRT-type sampling planning methods. Artificial potential field methods have real-time performance, but they are prone to getting trapped in local minima in complex non-convex obstacle environments; sampling planning methods are suitable for high-dimensional space search, but when there are narrow passages, high-precision end-effector attitude constraints, and online replanning, a trade-off between efficiency, trajectory quality, and real-time performance is likely to occur. In addition, existing "planning + learning" hybrid schemes often have two shortcomings: one is that they only write heuristic information into the state or reward without providing expert actions that can be directly used for shared control; the other is that although external experts are introduced, for on-policy algorithms such as PPO, they do not perform consistent recording and probability recalculation of the actual execution actions after fusion, which can easily lead to inconsistencies between the sample distribution on which the policy update is based and the actual execution behavior, resulting in training oscillations or performance degradation.
[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0011] It should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this application can be combined with each other.
[0012] It should be understood that the terms "system," "apparatus," "unit," and / or "module" used in this application are a method of distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.
[0013] As indicated in this application, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements. An element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.
[0014] In the description of the embodiments of this application, "a plurality of" refers to two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.
[0015] Furthermore, flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Additionally, other operations can be added to these processes, or one or more steps can be removed from them.
[0016] Reference Figure 1 The diagram below illustrates an optional example of the trajectory planning method for a free-floating space robotic arm proposed in this invention. This method can be applied to computer devices, and the imaging method proposed in this embodiment may include, but is not limited to, the following steps: Step S1: Obtain task parameters and environmental parameters, and generate the current system status and obstacle constraint information; Step S2: Based on the current system state, perform pose transformation and state construction to generate reinforcement learning states; Step S3: Introduce random extension and potential field guidance, and generate expert-guided output based on reinforcement learning state and obstacle constraint information; Step S4: Generate candidate actions based on the reinforcement learning state and the PPO reinforcement learning policy; Step S5: Perform risk gating fusion based on expert-guided input and the candidate actions to obtain the final action to be executed; Step S6: Calculate the reward based on the multi-objective composite reward function and the final execution action; Step S7: Use reward, task parameters, and environment parameters as sampling data, and update the policy network parameters; Step S8: Based on the trained policy network, input the real-time observation status and output joint speed control commands.
[0017] In some feasible embodiments, step S1 specifically includes: Inputs: Free-floating space robotic arm model parameters, including link parameters, joint range and joint velocity constraints; non-cooperative target complex structure gap geometry parameters, including channel size, channel entrance and target capture surface; target end pose; set of environmental obstacles and set of self-collision constraints.
[0018] deal with: 1. Initialize the training round, setting the initial joint configuration of the robotic arm and the initial state of the base; 2. At the beginning of each training round, perturbations may be optionally applied to the target pose, obstacle geometry, or initial state to simulate modeling errors and perception uncertainties in the on-orbit environment; 3. Set termination conditions, which include: the end effector reaches the target allowable error manifold and maintains a preset number of control cycles, reaches the maximum number of steps, an environmental collision occurs, a self-collision occurs, or the boundary is crossed; 4. Parametric geometric representation of complex structural gaps is used to repeatedly construct channel environments with different degrees of contraction, and to calculate channel margin in subsequent risk assessments.
[0019] Output: Current system state and obstacle constraint information (including environmental obstacles and self-collision constraints).
[0020] In some feasible embodiments, step S2 specifically includes: Input: Current end effector pose (position, attitude), target pose, joint angles With joint angular velocity Base state characterization quantity.
[0021] deal with: 1. Convert the end-effector pose and target pose from Euler angles or quaternions to continuous pose representation: 2. CPR is preferably a 9-dimensional vector. ,in For position vectors, The first two columns of the rotation matrix; and through The complete rotation matrix can be reversibly recovered. 3. Calculate the pose error, including position error. Attitude error and CPR error vector ; 4. Constructing the input state vector for reinforcement learning This includes: end-stage CPR, target CPR, CPR error, , Joint angles and joint angular velocities, etc.
[0022] 5. Perform runtime mean and variance normalization on the state features to improve the numerical stability of the training process.
[0023] Output: Reinforcement learning state .
[0024] The reinforcement learning states are used for training the policy network during the training phase and for providing input to the trained policy network during the model application phase. In the model application phase, the state construction method is consistent with that used in the training phase; wherein, the normalization process preferably employs statistical parameters determined during the training phase.
[0025] In this embodiment, by introducing continuous pose representation (CPR) to construct state variables and error variables, the singularity problem of traditional Euler angle representation and the discontinuity problem of quaternion representation are alleviated. Combined with the worst-case axis-focused pose reward design, the pose convergence capability and end-effector pose control accuracy in complex gap grasping tasks are improved.
[0026] In some feasible embodiments, step S3 specifically includes: This step is used to generate expert guidance output online during the training process based on the current reinforcement learning state and obstacle constraint information. The expert guidance output includes a joint space collision-free landmark sequence and expert guidance actions that can directly participate in shared control.
[0027] Input: Reinforcement learning state Obstacle sets and self-collision constraints Process: Define expert strategies Implement the following mapping: in: It is a sequence of collision-free path points in joint space. To meet the joint velocity constraints, the guiding joint velocity movement is aligned with the direction of the landmark path.
[0028] The expert-guided module includes the following sub-steps: Joint space path generation: Two random trees are grown in joint space using BI-RRT*, starting from the initial configuration and the target position respectively; The bidirectional fast expansion random tree optimization algorithm ( The algorithm generates candidate expansion directions through random sampling and corrects the expansion directions by combining artificial potential fields. After collision verification, a new node is generated. When two random trees meet the connection conditions, a feasible path is obtained, and the path cost is reduced and the path smoothness is improved through reconnection operations.
[0029] An artificial potential field bias is introduced in the extension direction: the potential field force is formed by the attraction / repulsion force in the end space, and is mapped to the joint space increment direction through the Jacobian pseudo-inverse; The random expansion direction and the potential field guidance direction are fused according to weights to obtain the tree growth direction, new nodes are generated and collision checks are performed. When two trees can be connected, a feasible path is obtained, and the path cost can be reduced and the smoothness improved through the reconnection operation of RRT*.
[0030] Landmark tracking and guidance action generation: Extracting joint space landmark sequences from the path ; Using a road sign tracking controller to generate guidance actions For example, based on the proportional control of the current joint configuration pointing to the next landmark, and by applying a velocity cap, the joint space velocity vector is output. ; Output: Signpost sequence Guiding actions .
[0031] Preferably, in order to suppress the local minima of the artificial potential field, a directional consistency constraint can be set; when the angle between the force of the artificial potential field and the target pointing direction is greater than a preset threshold, the repulsive term is attenuated or suppressed to improve the convergence of local guidance.
[0032] In some feasible embodiments, it also includes Degradation mechanism when planning is unavailable: When the planner fails to generate a valid path or connection fails within a limited time, the expert strategy switches to local APF guidance: the combined APF force is normalized to the desired end velocity, and then mapped to a joint velocity reference via a Jacobian pseudo-inverse, serving as... Output.
[0033] In some feasible embodiments, step S4 specifically includes: This step employs the Proximal Policy Optimization (PPO) algorithm as the reinforcement learning backbone to generate policy actions in the continuous motion space of a free-floating robotic arm performing tasks within the complex gaps of a free-floating space structure. The trajectory planning of the free-floating robotic arm is modeled as a finite-time continuous state-continuous action Markov decision process. Among them, the state Constructed from step (2) (including CPR representation of the end effector and target, error amount, joint angles and angular velocities, etc.); motion This is a joint space velocity command vector (dimension equal to the robot arm's joint degrees of freedom), used to drive the simulation / physical system during the sampling period. From Transferred to In PPO, an Actor-Critic architecture is used: the policy network (Actor) uses parameters... Represents a random policy It outputs action distribution parameters based on the current state. The value network (Critic) uses parameters... Estimating the state value function At each control moment , the current state The input policy network yields action distribution parameters. The policy follows a Gaussian distribution. in It is the mean vector. This is the diagonal covariance matrix or the variance parameter output by the network. The policy action is then obtained by sampling based on the above distribution: To meet the joint velocity constraints of the space robotic arm, the strategic motion is limited before execution, employing... Projecting the compressed image onto the allowable range using a linear scaling method: in This represents the maximum speed limit for each joint. The reinforcement learning candidate actions output from this step are used for subsequent fusion with expert actions.
[0034] During training, the policy network parameters are updated using the PPO cutoff objective function, and the value network parameters are updated using mean squared error loss, in order to limit the policy update magnitude and improve training stability.
[0035] Output: Reinforcement learning candidate actions .
[0036] In some feasible embodiments, step S5 specifically includes: This step aims to transform the expert-guided action generated in step S3. With the reinforcement learning policy action generated in step S4 Dynamic fusion is performed to achieve a smooth transition from "expert demonstration" to "autonomous decision-making" and to ensure the consistency of the training data distribution of the On-policy algorithm.
[0037] Risk-gated action fusion: Based on the current collision risk, channel margin, base perturbation level, and training progress, adaptive fusion weights are determined between expert-guided actions and reinforcement learning strategy actions. ,in The value range is 0 to 1. The collision risk can be characterized by the minimum distance between the robotic arm and environmental obstacles or the robotic arm itself; the channel margin can be characterized by the remaining space between the end effector or key link and the clearance boundary; the base disturbance degree can be characterized by base attitude offset, angular velocity change, or a combination thereof. Preferably, The following control law applies: when the collision risk increases, the channel margin decreases, or the base disturbance increases, Increase; as training progresses, The overall trend is one of decline. Final execution action. The calculation is as follows: Actual actions after fusion After undergoing joint velocity constraint processing again, it is sent to the environment for execution.
[0038] On-policy data consistency correction: Since PPO is an on-policy reinforcement learning algorithm, the sampled data used for policy updates must be consistent with the actual execution behavior. Therefore, when recording trajectory data, the original policy network output actions are not directly stored. Instead, it records the actual actions performed on the environment after risk gating and fusion. .
[0039] At the same time, the actual execution action is recalculated based on the current policy network. Log probability under the current strategy The logarithmic probability, along with the state, reward, and state transition results, are stored in the trajectory cache for subsequent PPO updates.
[0040] To address the issue that expert actions directly mixed into sampled data in existing "planning + learning" hybrid schemes may disrupt the consistency between execution behavior and update basis in the on-policy algorithm, this invention proposes an on-policy data consistency correction method based on recalculation of actual execution actions. Specifically, during training, the fused actual execution actions are recorded, and the log probability of the actual execution action under the current policy is recalculated. This ensures that the sampled data in the trajectory buffer is consistent with the policy update basis, reduces policy update bias caused by expert intervention, and improves training stability and convergence reliability.
[0041] Through the above processing, even if external expert intervention is introduced during the training process, the policy update remains consistent with the actual sampling trajectory, thereby avoiding policy update deviations caused by inconsistencies between the executed actions and the sampling actions.
[0042] In some feasible embodiments, step S6 specifically includes: This step constructs a dense reward function for the constrained gap-interleaving task, used to evaluate the quality of the state transitions generated by the actions performed in step (5). Reward Function It consists of the following five weighted components: 1) Dual-channel position error reward A dual-sensitivity mechanism is constructed using the Tanh function. When the distance to the target is relatively far ( When approaching the target ( ), the saturation property of Tanh is used to limit the penalty magnitude and prevent gradient explosion; when approaching the target ( When Tanh is approximately linear, it provides a more sensitive gradient signal to improve position convergence accuracy during the approach phase. 2) Worst-case axis attitude-guided reward To address the issue of asynchronous convergence across the three axes, a "worst-case concern" mechanism is proposed. Instead of simply summing the errors across all three axes, the penalty is applied only to the axis with the largest Euler angle error at the current moment. This forces the agent to prioritize correcting the orientation axis with the largest deviation, solving the problem of "easy-to-converge axes masking difficult-to-converge axes". 3) Smoothness regularization reward Penalize abrupt changes in joint velocity by calculating the difference in joint velocity between adjacent time steps. Norm, suppressing high-frequency jitter and reducing disturbance to the free-floating base. 4) Safety collision penalty Set a safe distance threshold. The minimum distance between the robotic arm and its environment or itself. At this time, a large fixed negative reward is given, forcing the strategy to prioritize safety over task execution. 5) Sparse success reward When the terminal state enters the target-allowed error manifold When both position error and attitude error are less than the threshold, a positive reward is given.
[0043] In some feasible embodiments, step S7 specifically includes: This step uses the collected interaction data to update the network parameters using the PPO algorithm and outputs the trained trajectory planning model.
[0044] Batch data collection and advantage estimation: In each training iteration, multiple interaction trajectories are collected in the simulation environment using the current policy network and risk gating fusion mechanism until a preset batch size is reached. For the collected samples, the advantage function and reward objective are calculated by combining the reward value obtained in step S6 and the value network estimation results. Preferably, a generalized advantage estimation method is used to balance bias and variance, thereby improving training efficiency.
[0045] Policy Network and Value Network Updates: Update network parameters using batch sampling data: Update the policy network parameters to maximize the PPO shearing objective function and limit the difference between the old and new policies; update the value network parameters to minimize the error between state value prediction and reward objective; and update the overall decay term of the guiding weights according to the training progress and risk gating rules.
[0046] Model convergence and output: Repeat the batch data acquisition and model update steps until the maximum number of training rounds is reached or the policy success rate stabilizes and converges. After training, output the parameters of the trained policy network. In practical applications, the policy network is deployed on the space robotic arm controller. After inputting the real-time observation status, it outputs joint speed control commands to complete the on-orbit grasping trajectory planning task in the gaps of complex structures.
[0047] Based on the overall process described above, this invention also provides relevant experimental examples: This embodiment is performed on a computer simulation platform.
[0048] Hardware environment: Intel Core i7-14700F CPU @ 5.40GHz, 32GB RAM, NVIDIA GeForce RTX 4060Ti GPU.
[0049] Software environment: Windows 11 operating system, Python 3.10 programming language, PyTorch 1.13.1 deep learning framework, and PyBullet physics simulation engine for building a space microgravity dynamics environment.
[0050] Step 1: Construct a dynamic simulation model of a free-floating space robotic arm. Build a system consisting of a base and a 7-DOF robotic arm in PyBullet. System parameter settings: Base mass... ,size Inertia matrix The robotic arm configuration adopts MDH parameters, as shown in Table 1. A schematic diagram of the robotic arm's MDH configuration is shown below. Figure 2 As shown, the number of joints The joint angle range is limited to... (Except for joints 4, 5, and 6) Joint angular velocity limit .
[0051] Table 1. Robotic Arm MDH Parameters Microgravity dynamics simulation: turning off gravitational acceleration ( Since the base is freely floating, the dynamic coupling calculation between the base and the robotic arm is initiated. That is, when the robotic arm moves, the base generates a reaction motion based on the conservation of momentum and angular momentum. The system state is determined by the base state. and the joint status of the robotic arm Joint description.
[0052] Construct a model of a non-cooperative target (faulty satellite) and its external protrusions in a simulation environment.
[0053] Target satellite parameters: The satellite body is set as a polyhedral structure, with solar panels deployed on both sides (size set to...). Furthermore, a cylindrical antenna (0.1m radius, 0.8m length) and other geometric shapes were placed near the capture surface as vulnerable accessory obstacles.
[0054] Obstacle avoidance constraints: The solar panel, antenna, and satellite body are designed as rigid collision objects, with the highest collision penalty coefficient. The robotic arm's links are encased in a capsule for collision detection. The deployed solar panel and antenna form a funnel-shaped virtual approach gap in front of the target grasping surface (such as the docking ring interface), with an entrance width of approximately 1.2m and a minimum internal width of 0.4m.
[0055] Mission Objective: The robotic arm's end effector, starting from a safe, folded configuration around the gap, moves through the narrow gap formed by the solar panel and antenna to the target acquisition point at the docking ring of the target satellite. To simulate the perception and measurement errors of non-cooperative targets in orbit, an error is applied to the target position at the beginning of each training round. Random perturbations.
[0056] State space construction and continuous pose transitions: In each control cycle ( The system state is obtained and preprocessed: Continuous Pose Representation (CPR) Transformation: The poses of the end effector and target (originally position + quaternion / Euler angles) are converted into 9-dimensional CPR vectors. For the pose transformation matrix... Extract location and the first two columns of the rotation matrix (Column vector). Construct the CPR vector: This representation ensures the continuity of the rotation space and eliminates gimbal lock and discontinuous transitions in Euler angles. State vector combination: The state vector input to the neural network. The dimension is 43, including: end effector CPR (9-dimensional); target point CPR (9-dimensional); CPR error vector (9-dimensional); joint angle. (7-dimensional) and joint angular velocity (7-dimensional); scalar error term: Euclidean distance and attitude error angle Normalization: Normalize the mean and variance of the input state vector to make it conform to a standard normal distribution.
[0057] An APF-Bi-RRT* expert-guided module is constructed to generate reference actions online during training. Artificial potential field (APF) definition: Gravitational potential field: The guiding force directs the terminal towards the target. Repulsive potential field: When the robotic arm link is less than a safe distance from the obstacle Effective immediately. Directional consistency constraint: If the angle between the resultant force of the APF and the target direction is obtuse, the repulsive force is temporarily suppressed to prevent getting trapped in a local minimum. Path search: in joint space Simultaneously grow the starting tree and the target tree. Sampling strategy: using... The probability is used to directly sample the target point. APF bias extension: when generating new nodes, the probability is... The probability is determined by using the Jacobian pseudo-inverse direction mapped from the APF torque to the joint space as the expansion guide, with a step size... When planning is successful, a series of joint space landmarks are generated. Guided motion generation: based on the current joint angle. Calculate the path to the next landmark. The expected velocity. If planning fails (timeout), the joint velocity obtained from the APF resultant force mapping is directly output as the degraded guided motion. Output expert motion. .
[0058] To fully utilize expert knowledge in the early stages of training, gradually enhance strategic autonomy in the later stages, and ensure safety in high-risk scenarios, this embodiment includes expert-guided actions. With reinforcement learning candidate actions Implement risk gating integration.
[0059] Specifically, the fusion weights are determined based on the current collision risk, channel margin, base disturbance level, and training progress. The calculation model is as follows: in, To increase the number of training steps Linearly decaying basic guiding weights; Collision risk surge This represents the minimum distance between the robotic arm and the obstacle. This is a safety threshold. When the distance is less than the safety threshold, this factor rapidly increases the proportion of expert intervention. The base disturbance term can be expressed as the norm of the base attitude angular velocity. When the base disturbance is too large, expert intervention is enhanced to suppress violent movement. This is the corresponding sensitivity adjustment coefficient. The function ensures that the weights are always in Between. Preferably, when the collision risk increases, the channel margin decreases, or the base disturbance increases, the fusion weights. Increase; as training progresses, The overall level gradually decreases. Based on the fusion weights Calculate the actual actions performed : Actual actions after fusion After undergoing joint velocity constraint processing again, it is sent to the simulation environment for execution.
[0060] Since PPO is an on-policy reinforcement learning algorithm, in order to keep the policy updates consistent with the actual execution behavior, the original policy actions are not directly stored when recording trajectory data. Instead, it records the actual actions performed. Simultaneously, the actual actions to be executed are recalculated based on the current policy network. The logarithmic probability under the current policy is calculated and stored together with the state, reward, and state transition results in the trajectory cache for subsequent policy updates.
[0061] The compound reward function is calculated after each step of execution, and the reward is calculated accordingly. Location Rewards: Tanh is used to limit penalties at long distances and approximate linearity at close distances. Attitude reward (worst-case axis concern): calculates the three-axis Euler angle error. Penalty for taking the maximum value: Smoothness and obstacle avoidance: A penalty of 0.25 is applied for sudden changes in joint velocity; if a collision occurs, [the penalty is...]. The punishment. Success reward: When and Success is determined after 5 time steps are completed, and a reward is given. Reward and end the round.
[0062] In each training iteration, multiple interaction trajectories are collected in the simulation environment using the current policy network and risk-gated shared control mechanism until a preset batch size is reached. For the collected trajectory samples, the advantage function and reward objective are calculated by combining the reward value and the value network estimation results. Preferably, a generalized advantage estimation method is used to balance bias and variance, improving training efficiency. Then, the policy network and value network are updated using the batch samples: the policy network parameters are updated to maximize the PPO shearing objective function; the value network parameters are updated to minimize the error between the state value prediction and the reward objective; and the fusion weights are updated according to the training progress and risk gating rules. The overall attenuation term.
[0063] Repeat the above training iteration process until the maximum number of training rounds is reached or the policy performance stabilizes and converges. After training, output the parameters of the trained policy network. Deploy the policy network in the space robotic arm controller, and after inputting the real-time observation status, output joint speed control commands to complete the on-orbit grasping trajectory planning task in complex structural gaps.
[0064] After training using the above-described methods, a trajectory planning model suitable for grasping gaps in complex structures by a free-floating robotic arm can be obtained. Based on the online expert guidance mechanism and risk gating fusion mechanism described in this invention, blind random exploration can be reduced during training, the guidance and intervention capabilities in high-risk scenarios can be enhanced, and control commands that meet joint velocity constraints can be output after training. This facilitates safe trajectory planning and end-effector pose control in grasping gaps in complex structures.
[0065] A trajectory planning system for a free-floating space robotic arm includes: An initialization unit is used to execute step S1; A conversion unit is used to execute step S2; An expert-guided unit is used to execute step S3; PPO reinforcement learning unit, used to perform step S4; An action fusion unit is used to execute step S5; A reward calculation unit is used to execute step S6; The model training unit is used to perform step S7; The reasoning unit is used to execute step S8.
[0066] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0067] A trajectory planning device for a free-floating space robotic arm: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a trajectory planning method for a free-floating space robotic arm as described above.
[0068] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0069] A storage medium storing processor-executable instructions, which, when executed by a processor, are used to implement a trajectory planning method for a free-floating space robotic arm as described above.
[0070] The content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0071] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A trajectory planning method for a free-floating space robotic arm, characterized in that, Includes the following steps: Obtain task parameters and environmental parameters, and generate the current system status and obstacle constraint information; Based on the current system state, pose transformation and state construction are performed to generate reinforcement learning states; By introducing stochastic expansion and potential field guidance, an expert-guided output is generated based on the reinforcement learning state and the obstacle constraint information; Based on the reinforcement learning state and the PPO reinforcement learning policy, candidate actions are generated; Risk gating fusion is performed based on the expert-guided input and the candidate actions to obtain the final action to be executed. The reward is calculated based on the multi-objective composite reward function and the final execution action. Using the reward, the task parameters, and the environment parameters as sampling data, and updating the policy network parameters, a trained policy network is obtained. Based on the trained policy network, the system takes the real-time observed state as input and outputs joint speed control commands.
2. The trajectory planning method for a free-floating space robotic arm according to claim 1, characterized in that, The step of performing pose transformation and state construction based on the current system state to generate reinforcement learning states specifically includes: Convert the end-effector pose and the target pose into continuous pose representations respectively; Calculate the position error, attitude error, and continuous pose representation error vector; The mean-variance normalization process is applied to the continuous pose representation of the end pose, the continuous pose representation of the target pose, the position error, the attitude error, and the continuous pose representation error vector, and then the reinforcement learning state is constructed.
3. The trajectory planning method for a free-floating space robotic arm according to claim 2, characterized in that, The step of introducing random expansion and potential field guidance, and generating expert-guided output based on the reinforcement learning state and the obstacle constraint information, specifically includes: In the joint space, a bidirectional fast expanding random tree optimization algorithm is used to grow two random trees, starting from the current joint configuration and the target joint configuration corresponding to the target pose, respectively. The potential field force is constructed from the forces acting in the end space, and mapped to the incremental direction of the joint space via the Jacobian pseudo-inverse; The tree growth direction is obtained by merging the random expansion direction and the potential field guidance direction according to weights, generating new nodes and performing collision checks to obtain feasible paths. Extract the joint space landmark sequence from the feasible path; The joint space landmark sequence and constraint restrictions are generated by a landmark tracking controller to generate guiding actions. The joint space landmark sequence and the guiding action are used as expert guidance output.
4. The trajectory planning method for a free-floating space robotic arm according to claim 3, characterized in that, Also includes: If a valid path is not generated or the connection fails within the specified time, the system will switch to local APF booting. The APF resultant force is normalized to the desired end velocity, and then mapped to the joint velocity reference through the Jacobian pseudo-inverse as a guide motion.
5. The trajectory planning method for a free-floating space robotic arm according to claim 4, characterized in that, The step of performing risk-gated fusion based on the expert-guided input and the candidate actions to obtain the final action is expressed by the following formula: in, Indicates the final action to be performed. Indicates candidate actions, This indicates a guiding action performed by an expert to guide the input. Indicates the fusion weight. The base bootstrap weights represent the base bootstrap weights that decay linearly with the number of training steps. Indicates the number of training steps. This indicates a surge in collision risk. This represents the base disturbance term. The function weights always remain in between.
6. The trajectory planning method for a free-floating space robotic arm according to claim 5, characterized in that, The step of performing risk gating fusion based on the expert-guided input and the candidate actions to obtain the final action to be executed further includes: Recalculate the logarithmic probability of the final action under the current policy based on the current policy network; The final execution action, the logarithmic probability, the state, the reward, and the state transition result are stored in the trajectory cache.
7. The trajectory planning method for a free-floating space robotic arm according to claim 6, characterized in that, The multi-objective reward function includes position error reward, worst-case axis attitude guidance reward, smoothness regularization reward, safe collision penalty, and sparse success reward.
8. A trajectory planning system for a free-floating space robotic arm, characterized in that, include: The initialization unit is used to acquire task parameters and environmental parameters, and generate the current system state and obstacle constraint information. The transformation unit performs pose transformation and state construction based on the current system state to generate reinforcement learning states; The expert guidance unit introduces random expansion and potential field guidance, and generates expert guidance output based on the reinforcement learning state and the obstacle constraint information; The PPO reinforcement learning unit generates candidate actions based on the reinforcement learning state and the PPO reinforcement learning policy. The action fusion unit performs risk-gated fusion based on the expert-guided input and the candidate actions to obtain the final action to be executed. The reward calculation unit calculates the reward based on the multi-objective composite reward function and the final execution action; The model training unit uses the reward, the task parameters, and the environment parameters as sampling data to update the policy network parameters, thereby obtaining the trained policy network. The inference unit, based on the trained policy network, takes into account the real-time observed state and outputs joint speed control commands.
9. A trajectory planning device for a free-floating space robotic arm, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the trajectory planning method for a free-floating space robotic arm as described in any one of claims 1-7.