Fixed base mechanical arm point-to-point trajectory optimization method based on near-end strategy optimization algorithm

Abstract

The invention discloses a fixed base mechanical arm point-to-point trajectory optimization method based on a near-end strategy optimization algorithm, solves the problem that a high-dimensional continuous action space exists when deep reinforcement learning is directly used for a fixed base mechanical arm tail end position to reach a task, and belongs to the field of robot operation control and mechanical arm autonomous motion optimization. The method comprises the steps that a mechanical arm kinematics model is established, and constraint conditions are defined; modeling a tail end point-to-point arrival task as a sequential decision process, constructing a state vector as the input of a strategy network, and taking a tail end displacement increment direction vector as the action output of the strategy network; mapping the tail end displacement increment direction vector into a joint increment through differential inverse kinematics, and performing constraint processing according to a constraint condition to generate an executable joint control instruction; and training the strategy network by adopting a near-end strategy optimization algorithm, obtaining a single-step reward through a multi-target reward function, and evaluating the performance output by the action of the strategy network according to the single-step reward until convergence.

Description

Technical Field

[0001] This invention relates to a point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm, belonging to the field of robot operation control and autonomous motion optimization of robotic arms. Background Technology

[0002] In extraterrestrial robotic operations, fixed-base robotic arms are widely used for positioning, contact, and manipulation of end effectors. These tasks often require the end effector to quickly and safely move and reach a designated target point within a limited workspace, while satisfying constraints such as joint angle range, joint velocity / increment limits, and singular configuration avoidance. Compared to idealized environments, extraterrestrial scenarios present uncertainties in environmental parameters and operational conditions, leading to greater uncertainty in the target's relative position, contact timing, and constraint margins. This places higher demands on the end effector's online adaptability and robustness during the reaching process.

[0003] Existing end-point arrival trajectory planning methods (such as linear interpolation, cubic spline interpolation, and time-optimal trajectory planning) mostly adopt an offline trajectory generation paradigm, typically relying on relatively accurate and stable models. In the aforementioned situations, traditional methods lack the ability to select actions and adjust step sizes based on real-time feedback at the decision-making level. This often necessitates frequent replanning or the introduction of conservative trajectory parameters, leading to decreased arrival efficiency, increased end-point position error, or uneven control commands. Furthermore, it may result in decreased operability and increased action pruning when approaching singular configurations.

[0004] Reinforcement learning offers a policy-centric online decision-making approach for such problems: by learning the mapping from state to action through interactive experience, the robotic arm can output continuous control commands in real time based on information such as the relative position error between the end effector and the target point, joint states, and constraint margins, thereby maintaining stable arrival performance under uncertain and constrained conditions. However, directly applying deep reinforcement learning to the end effector position arrival task of a fixed-base robotic arm still faces challenges such as a high-dimensional continuous motion space, sparse and delayed rewards, and strong physical constraints. Especially under the conditions of high safety and controllability requirements in off-ground operations, a technical solution that combines reinforcement learning policy optimization with kinematic constraint processing is needed to ensure that the end effector position commands output by the policy can be stably executed under joint constraints and singularity constraints, and to achieve high-precision, smooth, and efficient arrival of the end effector at the target point. Summary of the Invention

[0005] To address the problems of high-dimensional continuous motion space, sparse and delayed rewards, and strong physical constraints when directly applying deep reinforcement learning to the end-effector position arrival task of a fixed-base robotic arm, this invention provides a point-to-point trajectory optimization method for a fixed-base robotic arm based on a proximal policy optimization algorithm.

[0006] The present invention provides a point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm, comprising:

[0007] Establish a kinematic model of the robotic arm and define constraints, including joint limits, joint increment limits, and singularity risk thresholds;

[0008] The task of reaching the end point point is modeled as a sequential decision-making process, and a state vector is constructed that includes at least the end point position error, joint state, target point position and the action of the previous moment.

[0009] The state vector is used as the input to the policy network, and the end displacement increment direction vector is used as the action output of the policy network.

[0010] The end displacement increment direction vector is mapped to a joint increment through differential inverse kinematics, and constrained according to the constraint conditions to generate executable joint control commands.

[0011] The policy network is trained using a proximal policy optimization algorithm. The single-step reward is obtained through a multi-objective reward function. The performance of the policy network's action output is evaluated based on the single-step reward. The policy is iteratively optimized by combining a course learning mechanism until convergence.

[0012] Preferably, the method for mapping the minute displacement increment in the end-effector Cartesian space to a joint increment via differential inverse kinematics includes:

[0013]

[0014] in, This represents the expected terminal displacement increment at the current moment. This is the direction vector of the current end displacement increment. For end-incremental scale, , The target point location, This is the current end position. This is a step size scaling function that adapts to distance; the lower limit of the step size is reduced as the target approaches to decrease overshoot and oscillation.

[0015] The differential inverse kinematics are solved using the damped least squares method, mapping the end-effector displacement increment to the joint increment. :

[0016]

[0017] in, Let Jacobian matrix be the value at the current end position. This is a damping factor that increases when the minimum singular value of the Jacobian is below the singularity risk threshold. This is to suppress the divergence of joint increments.

[0018] Preferably, the multi-objective reward function includes:

[0019] Convergence and achievement rewards are used to encourage the endpoint to move closer to the target point;

[0020] Efficiency and energy consumption penalty terms are used to reduce the number of steps and suppress the range of motion of joints;

[0021] Controllability and smoothness penalty terms are used to constrain the amplitude and rate of change of a single step movement;

[0022] Safety and enforceability penalties are used to constrain joint limit risks, joint increment limits, and limit events.

[0023] As a preferred option, convergence and achievement rewards include progress items. Absolute distance term Rewards with terminals :

[0024]

[0025] in, This represents the decrease in distance error between two adjacent steps;

[0026]

[0027] in, Location scale factor;

[0028]

[0029] in, This represents the maximum allowable error threshold between the current position of the endpoint and the position of the target point;

[0030] Efficiency and energy consumption penalties include time. and energy consumption items ;

[0031]

[0032]

[0033] Controllability and smoothness penalty terms include motion amplitude terms. and smoothing terms ;

[0034]

[0035]

[0036] Controllability and smoothness penalties include limiting risk items. Penalties for exceeding limits and motion distortion item ;

[0037]

[0038] It is a continuous function that is inversely proportional to the limit margin, or a piecewise quadratic continuous function;

[0039]

[0040] in, This indicates the amount of change in joint velocity that triggers the clipping constraint. This indicates the penalty amount when a joint position or joint movement state violates the mechanical structure's limitations.

[0041]

[0042] in, This represents the end displacement performed after dynamics, constraints, and trimming.

[0043] Assign a weight to each term in the multi-objective reward function, and then sum them up as the single-step reward.

[0044] Preferably, a random sampling strategy within the workspace is adopted to obtain the locations of candidate target points. Perform several iterations of inverse differential solution. If the error threshold is reached within a given number of iterations, the candidate target point position is included in the training sample as the target point position.

[0045] As a preferred method, the constraint processing method based on the constraint conditions includes:

[0046] Joint increment Amplitude limiting is applied to ensure that the joint increment does not exceed the upper limit of the single-step joint increment.

[0047] Limit and truncate the updated joint angles to ensure that the updated joint angles are within the allowable joint angle range;

[0048] When a singular risk is detected, the expected end displacement increment at the current moment is scaled, and the damping factor is increased simultaneously.

[0049] As a preferred method, the method of training the policy network using a proximal policy optimization algorithm includes:

[0050] S1. Initialize the policy network and value network, and set the PPO hyperparameters;

[0051] S2. Use the current policy to interact with the environment for sampling, collect trajectory sequences, and cache the collected trajectory sequences and the log probabilities of actions under the old policy; the trajectory sequence includes the state vector at the current moment. Next time step state vector The direction vector of the current end displacement increment The single-step reward at the current moment and termination conditions ;

[0052] S3. Calculate the advantage estimate and return target based on the collected trajectory sequence and the current predicted value of the value network;

[0053] S4. Based on the ratio of the old and new strategies and the calculated advantage estimate, update the policy network parameters using the PPO pruning objective function;

[0054] S5. Using the calculated return target as a supervision signal, update the value network parameters by minimizing the error between the value network output and the return target;

[0055] S6. Repeat S2 to S5 until the policy converges.

[0056] As a preferred option, the objective function for PPO pruning is:

[0057]

[0058] in, To obtain the advantage value, For the parameters of the policy network, Expressing expectations Represents the clipping function The ratio of new to old strategies. To truncate the threshold, Advantage estimation; ratio of new to old strategies , This represents the log probability of the action under the new strategy. This represents the log-probability of actions under the old strategy.

[0059] Preferably, the course learning mechanism includes at least one of the following methods for increasing difficulty:

[0060] Gradually increase the distance between the initial endpoint and the target point;

[0061] Gradually tighten the safety margin of joint limits;

[0062] External disturbances or noise are introduced during the movement.

[0063] Preferably, after the current strategy and environment interaction sampling, step S2 further includes determining whether a termination condition is met, wherein the termination condition includes at least one of the following:

[0064] When the end position error If the task is deemed successful, the current round ends, and a one-time positive reward is added to the single-step reward.

[0065] When the cumulative number of interaction sampling steps in the current round reaches the preset maximum number of steps, or the cumulative execution time exceeds the preset time limit, it is judged as a timeout and the current round is terminated;

[0066] When a violation of system security constraints is detected, the current round is immediately terminated, and a penalty is added to the single-step reward. The security constraints include at least one of the following:

[0067] The joint angle exceeds the mechanical limit range;

[0068] The joint increment exceeds the set anomaly risk threshold;

[0069] The robotic arm enters a singular configuration, and the minimum singular value of the Jacobian matrix is ​​below the singularity risk threshold.

[0070] The beneficial effects of this invention are that, while reducing reliance on precise external environment models, it can autonomously generate smooth and efficient trajectories that satisfy joint constraints and singularity constraints. Compared with traditional deterministic programming, it has better robustness and adaptability in constrained or disturbed scenarios, reducing oscillations and overshoot, shortening arrival time, and improving mission success rate and safety. Attached Figure Description

[0071] Figure 1 This is a schematic diagram illustrating the principle of the point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm as described in this invention.

[0072] Figure 2 This is a schematic diagram of the PPO algorithm flow. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0075] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but this is not intended to limit the scope of the invention.

[0076] This application models the end-effector motion process as a sequential decision problem of "state-action-reward," learning a stable motion decision strategy through simulation interaction. Compared with traditional trajectory generation methods that rely on fixed planning templates or are highly sensitive to external environmental parameters / perturbation models, this application achieves stable iteration of the strategy through the PPO pruning and updating mechanism. Combined with curriculum learning and multi-objective weighted evaluation mechanisms, the strategy can adaptively generate smooth, efficient, and executable end-effector motion sequences under different task difficulties and constraint strengths. Simultaneously, under kinematic constraints such as joint limitations, velocity / incremental constraints, and singular configuration avoidance, the end-effector position increment output by the strategy is constrained, thereby achieving point target arrival and process optimization. The point-to-point trajectory optimization method for a fixed-base robotic arm based on a proximal strategy optimization algorithm in this embodiment includes:

[0077] Step 1: Establish the kinematic model of the robotic arm and define the constraints; specifically including:

[0078] Step 11: Establish the forward kinematics model of the six-DOF fixed-base robotic arm. Use the DH parameter method to establish coordinate systems for each joint and link, obtaining the homogeneous transformation matrix from the base coordinate system to the end effector coordinate system. The resulting end position , Joint angle, This refers to the end position.

[0079] Step 12: Derive the Jacobian matrix for the end position. It satisfies a linear approximation relationship:

[0080]

[0081] in, Indicates the change in the position of the endpoint. This indicates the change in joint angle.

[0082] Step 13: Define constraints as the basis for subsequent execution mapping and security mechanisms, including:

[0083] Joint limitation:

[0084] Joint increment limit:

[0085] Singularity risk threshold Jacobi minimum singular value ;

[0086] Step 2: Model the end-point point-to-point arrival task as a sequential decision-making process, and construct a state vector that includes at least the end-point position error, joint state, target point position, and the action of the previous moment.

[0087] Each round starts from the initial joint angle Departure, with the goal of reaching a given target point. .

[0088] The state vector includes the end position error. Current end position Target point location Joint angle Joint velocity / angular velocity and the direction vector of the end displacement increment at the previous moment. ;

[0089] In a preferred embodiment, the target point location is generated using a random sampling strategy within the workspace, which can also enhance accessibility filtering.

[0090] For candidate target points If the error threshold is reached within a given number of iterations after several iterations of inverse differential solution, the target is determined to be achievable and included in the training samples, thereby reducing training noise caused by unachievable targets.

[0091] Step 3: Using the state vector as the input to the policy network and the end-effector displacement increment direction vector as the action output of the policy network, train the policy network using the proximal policy optimization algorithm, including:

[0092] Step 31: This application adopts a continuous action space, where the action is defined as a small displacement increment direction vector in the final Cartesian space:

[0093]

[0094] Map the strategy output action to the desired end displacement increment:

[0095]

[0096] in, , For end-incremental scale, This is a step size scaling function that adapts to distance; the lower limit of the step size is reduced as the target approaches to decrease overshoot and oscillation.

[0097] Step 32: Using damped least squares (DLS), the direction vector of the end displacement increment is mapped to the joint increment through differential inverse kinematics:

[0098]

[0099] Damping factor It can adaptively adjust according to singularity risk: when the Jacobian minimum singularity value is reached. Below the singularity risk threshold Time increases This is to suppress the divergence of joint increments.

[0100] Step 33: Perform constraint processing based on the constraints to generate executable joint control commands;

[0101] Changes in joint angle Limiting and the updated Cut off the joints within the allowable range to ensure that they do not exceed the limits.

[0102] Exotic avoidance protection: When detected Below the singularity risk threshold At that time, singularity avoidance processing is initiated: for the desired end Scaling is applied, and the damping coefficient can be increased simultaneously. By reducing the command amplitude and improving numerical stability, abnormal amplification of joint increments and frequent triggering of limit switches are avoided when approaching singular configurations, thus ensuring the operability of the action and the smoothness of the motion process.

[0103] Step 34: Train the policy network using the proximal policy optimization algorithm, obtain the single-step reward through the multi-objective reward function, evaluate the performance of the policy network's action output based on the single-step reward, and iteratively optimize the policy in conjunction with the course learning mechanism until convergence.

[0104] First, design the reward function, using a weighted summation of components to obtain the single-step reward:

[0105]

[0106] Specifically, it includes:

[0107] (1) Convergence and achievement reward terms: In order to ensure that the terminal motion moves in the direction of the target at every step and avoids detouring and repeatedly crossing the boundary when approaching the target, convergence and achievement reward terms are set, including:

[0108] The progress term measures the rate of decrease in distance error between two adjacent steps. Used to encourage net proximity:

[0109]

[0110] The absolute distance term is used to impose a persistent constraint on the current error magnitude, ensuring that the policy always maintains a convergent trend toward the objective.

[0111]

[0112] in, This is the location scale factor, used to normalize the distance magnitude in different scenarios. It can be taken as the upper bound of the workspace radius or the upper bound of the target sampling radius.

[0113] Terminal achievement rewards are used to reinforce the key achievement of reaching the target; a one-time reward is given upon arrival.

[0114]

[0115] This represents the maximum allowable error threshold between the current position of the end effector and the target position, used to determine whether the target has been reached; when the end effector pose error is less than this threshold, it is considered that the target has been reached at the current moment.

[0116] When these three items are used together, the progress item provides local feedback on "whether each step is getting better," while the absolute distance item provides a global constraint on "how far away from the goal," thereby reducing the possibility of detours or lingering near the goal when relying solely on the progress item. The terminal achievement reward makes the strategy more inclined to quickly cross the target threshold to complete the task, rather than lingering near the threshold and repeatedly probing.

[0117] (2) Efficiency and energy consumption penalty items

[0118] To prevent the strategy from dragging out time through meaningless long rounds and to constrain the intensity of joint movements, two types of penalties are set: time and energy consumption. The time penalty applies a fixed penalty to each step, encouraging completion of the task in fewer steps.

[0119]

[0120] The energy consumption / motion intensity term is used to constrain the joint update amplitude. Considering that joint torque and power information may not be directly available during training, the squared joint increment is used as a calculable approximation; the more drastic the joint change, the greater the penalty.

[0121]

[0122] (3) Controllability and smoothness penalty terms

[0123] To improve the controllability of the end-effector motion and reduce jitter, constraints are further imposed on the motion amplitude and rate of change. The motion amplitude term is used to suppress the strategy from outputting commands close to saturation over a long period, avoiding overshoot or frequent triggering of the limiter due to excessively large expected end-effector displacement.

[0124]

[0125] The smoothing term is used to suppress abrupt changes between adjacent actions, making the end-effector motion more continuous and reducing the policy's back-and-forth probing when approaching the target or constraint boundaries.

[0126]

[0127] (4) Safety and enforceability penalty items

[0128] To ensure that the trajectory can still be executed stably under constraints such as joint limits and near-singularity, and to reduce the situation where "instructions are given but are difficult to implement in practice", the following safety and executability penalty terms are introduced.

[0129] The risk constraint term is used to restrict the joint angle from moving away from the boundary. As the joint angle gradually approaches the upper and lower limits, the penalty gradually increases, thereby guiding the strategy to maintain a certain risk margin.

[0130]

[0131] in A continuous function that is inversely proportional to the limit margin or is piecewise quadratic can be used, which facilitates stable training.

[0132] The limit penalty is used to constrain the feasibility of execution. When a joint increment needs to be limited, or when a joint angle update hits a limit and is truncated, the instruction in this step is considered unexecutable and a penalty is imposed.

[0133]

[0134] This parameter indicates the amount of change in joint speed that triggers the clipping constraint. If the current joint speed or speed increment exceeds the preset allowable range, this parameter takes a positive value to penalize excessive joint speed changes. This parameter represents the penalty indication when a joint position or joint motion state encounters mechanical structural limitations. It takes a positive value when any joint approaches or reaches its physical limit position, velocity, or acceleration constraint, and is used to reduce the probability of the strategy producing extreme movements.

[0135] The motion distortion term is used to compare the consistency between the expected end-effector displacement and the actual end-effector displacement. When the actual end-effector motion fails to follow the expected command due to singularities, limitations, or amplitude constraints, this term's penalty increases, thereby prompting the strategy to output a more achievable end-effector increment.

[0136]

[0137] This represents the actual end-effector displacement after dynamics, constraints, and pruning. It calculates the deviation between the expected end-effector displacement generated by the strategy and the actual displacement executed by the system under dynamics and constraints. By penalizing this deviation, the strategy is guided to learn motion decisions consistent with the actual execution capability, avoiding the generation of unrealizable or unstable control commands.

[0138] Weights of each item The setup follows the principle of prioritizing achievement and convergence, with efficiency and safety as secondary considerations. Convergence and achievement terms are the primary considerations, ensuring the task consistently reaches its objective. Efficiency and energy consumption terms are used to avoid round delays and suppress excessive joint changes. Safety and executability terms guide the strategy away from risky, limiting / triggering events, and improve the consistency between expected instructions and actual movements. To facilitate numerical comparability of different dimensional terms, distance-related terms are normalized using a scale factor, while other terms are weighted after task scale normalization, ensuring that the contributions of each term are at similar levels, thereby improving the stability and generalization of strategy training.

[0139] Then, the training and policy update process based on PPO includes:

[0140] Step 341: Initialize the policy network (Actor) and value network (Critic), and set the PPO hyperparameters, including learning rate, pruning threshold, discount factor, GAE parameters, batch size and number of update rounds, and adaptively adjust them according to performance metrics during training;

[0141] Step 342, Sampling Phase: Use the current strategy to interact with the environment to sample and collect trajectory sequences. It also caches the collected trajectory sequences and the log probabilities of actions under the old strategy for subsequent calculation of the ratio between the new and old strategies;

[0142] The trajectory sequence includes the state vector at the current moment. Next time step state vector The direction vector of the current end displacement increment The single-step reward at the current moment and termination conditions ;

[0143] Step 343: Calculate the advantage estimate and return target based on the collected trajectory sequence and the current predicted value of the value network; specifically, the generalized advantage estimate (GAE) can be used to calculate the advantage estimate for each time step. and construct return targets .

[0144] Step 344: Based on the ratio of the old and new policies and the calculated advantage estimate, update the policy network parameters using the PPO pruning objective function:

[0145] Ratio between old and new strategies:

[0146]

[0147] in, This represents the log probability of the action under the new strategy. This represents the log-probability of actions under the old strategy.

[0148] The Actor is updated using the PPO pruning objective function:

[0149]

[0150] in, To obtain the advantage value, For the parameters of the policy network, Expressing expectations Represents the clipping function The ratio of new to old strategies. The truncation threshold is used to control the update magnitude of the strategy and ensure training stability.

[0151] Step 345, Value Network Update: Based on calculated return objectives As a monitoring signal, the goal is to minimize the output and return of the value network. The error between them is used to update the value network parameters;

[0152] Step 346, Course Learning and Weight Adjustment: During the training process, the reward weights and environment difficulty are adjusted every few rounds based on the results. Specifically, the course learning mechanism includes at least one of the following difficulty increase methods:

[0153] Gradually increase the distance between the initial endpoint and the target point;

[0154] Gradually tighten the safety margin of joint limits;

[0155] External disturbances or noise are introduced during the movement.

[0156] This allows the strategy to steadily improve under increasingly complex conditions.

[0157] Step 347: Repeat steps 342 to 346 until convergence.

[0158] After sampling the interaction between the current strategy and the environment, this application determines whether a termination condition is met, and the termination condition includes at least one of the following:

[0159] (1) Successful termination:

[0160] When the end position error satisfies If the task is completed and the round ends, a reward will be given to the terminal. This is used to reinforce the key event of reaching the target, making the strategy more inclined to complete the point-to-point arrival within a limited number of steps.

[0161] (2) Timeout cutoff

[0162] When the number of interaction steps reaches the maximum number of steps. (or the cumulative execution time has reached the upper limit) The round is terminated when the truncation condition is met. This truncation condition is used to limit the length of a single round, avoid invalid exploration leading to excessively long samples, thereby stabilizing the training process and controlling computational costs.

[0163] (3) Safe termination: If an unacceptable constraint violation occurs (such as joint over-limit that cannot be recovered, or a strange hard threshold triggering), terminate early and apply a failure penalty.

[0164] This application focuses on near-end strategy optimization, integrating the strategy optimization mechanism of reinforcement learning with the kinematic constraints of the robotic arm. Under the premise that the base posture remains unchanged, it outputs the end-effector position increment or equivalent end-effector velocity command through online decision-making. Under physical constraints such as joint limits, velocity / acceleration constraints and singularity avoidance, it achieves adaptive optimization and precise arrival control of the end-effector position movement process, which is applicable to task scenarios such as end-effector position navigation and motion control of off-ground operation equipment.

[0165] In summary, this application achieves an adaptive, efficient, and interpretable trajectory optimization control strategy through kinematic modeling, state and motion design, reasonable reward and termination strategies, and PPO-based closed-loop training. This strategy can autonomously learn and adjust its operational scheme while considering constraints, thus meeting the stringent accuracy and safety requirements of the track-guided robotic arm.

[0166] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm, characterized in that, include: Establish a kinematic model of the robotic arm and define constraints, including joint limits, joint increment limits, and singularity risk thresholds; The task of reaching the end point point is modeled as a sequential decision-making process, and a state vector is constructed that includes at least the end point position error, joint state, target point position and the action of the previous moment. The state vector is used as the input to the policy network, and the end displacement increment direction vector is used as the action output of the policy network. The end displacement increment direction vector is mapped to a joint increment through differential inverse kinematics, and constrained according to the constraint conditions to generate executable joint control commands. The policy network is trained using a proximal policy optimization algorithm. The single-step reward is obtained through a multi-objective reward function. The performance of the policy network's action output is evaluated based on the single-step reward. The policy is iteratively optimized by combining a course learning mechanism until convergence.

2. The point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm according to claim 1, characterized in that, The method of mapping the small displacement increment of the end Cartesian space to the joint increment through differential inverse kinematics includes: in, This represents the expected terminal displacement increment at the current moment. This is the direction vector of the current end displacement increment. For end-incremental scale, , The target point location, This is the current end position. This is a step size scaling function that adapts to distance; the lower limit of the step size is reduced as the target approaches to decrease overshoot and oscillation. The differential inverse kinematics are solved using the damped least squares method, mapping the end-effector displacement increment to the joint increment. : in, Let Jacobian matrix be the value at the current end position. This is a damping factor that increases when the minimum singular value of the Jacobian is below the singularity risk threshold. This is to suppress the divergence of joint increments.

3. The point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm according to claim 2, characterized in that, The multi-objective reward function includes: Convergence and achievement rewards are used to encourage the endpoint to move closer to the target point; Efficiency and energy consumption penalty terms are used to reduce the number of steps and suppress the range of motion of joints; Controllability and smoothness penalty terms are used to constrain the amplitude and rate of change of a single step movement; Safety and enforceability penalties are used to constrain joint limit risks, joint increment limits, and limit events.

4. The point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm according to claim 3, characterized in that, Convergence and achievement rewards include progress items. Absolute distance term Rewards with terminals : in, This represents the decrease in distance error between two adjacent steps; in, Location scale factor; in, This represents the maximum allowable error threshold between the current position of the endpoint and the position of the target point; Efficiency and energy consumption penalties include time. and energy consumption items ; Controllability and smoothness penalty terms include motion amplitude terms. and smoothing terms ; Controllability and smoothness penalties include limiting risk items. Penalties for exceeding limits and motion distortion item ; It is a continuous function that is inversely proportional to the limit margin, or a piecewise quadratic continuous function; in, This indicates the amount of change in joint velocity that triggers the clipping constraint. This indicates the penalty amount when a joint position or joint movement state violates the mechanical structure's limitations. in, This represents the end displacement performed after dynamics, constraints, and trimming. Assign a weight to each term in the multi-objective reward function, and then sum them up as the single-step reward.

5. The point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm according to claim 1, characterized in that, A random sampling strategy within the workspace is used to obtain the locations of candidate target points. Perform several iterations of inverse differential solution. If the error threshold is reached within a given number of iterations, the candidate target point position is included in the training sample as the target point position.

6. The point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm according to claim 2, characterized in that, Methods for handling constraints based on constraints include: Joint increment Amplitude limiting is applied to ensure that the joint increment does not exceed the upper limit of the single-step joint increment. Limit and truncate the updated joint angles to ensure that the updated joint angles are within the allowable joint angle range; When a singular risk is detected, the expected end displacement increment at the current moment is scaled, and the damping factor is increased simultaneously.

7. The point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm according to claim 1, characterized in that, Methods for training policy networks using proximal policy optimization algorithms include: S1. Initialize the policy network and value network, and set the PPO hyperparameters; S2. Use the current policy to interact with the environment for sampling, collect trajectory sequences, and cache the collected trajectory sequences and the log probabilities of actions under the old policy; the trajectory sequence includes the state vector at the current moment. Next time step state vector The direction vector of the current end displacement increment The single-step reward at the current moment and termination conditions ; S3. Calculate the advantage estimate and return target based on the collected trajectory sequence and the current predicted value of the value network; S4. Based on the ratio of the old and new strategies and the calculated advantage estimate, update the policy network parameters using the PPO pruning objective function; S5. Using the calculated return target as a supervision signal, update the value network parameters by minimizing the error between the value network output and the return target; S6. Repeat S2 to S5 until the policy converges.

8. The point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm according to claim 7, characterized in that, The objective function for PPO pruning is: in, To obtain the advantage value, For the parameters of the policy network, Expressing expectations Represents the clipping function The ratio of new to old strategies. To truncate the threshold, Advantage estimation; ratio of new to old strategies , This represents the log probability of the action under the new strategy. This represents the log-probability of actions under the old strategy.

9. The point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm according to claim 7, characterized in that, The course learning mechanism includes at least one of the following methods to increase difficulty: Gradually increase the distance between the initial endpoint and the target point; Gradually tighten the safety margin of joint limits; External disturbances or noise are introduced during the movement.

10. The point-to-point trajectory optimization method for a fixed-base robotic arm based on a near-end strategy optimization algorithm according to claim 7, characterized in that, After sampling the interaction between the current strategy and the environment, step S2 further includes determining whether a termination condition is met, wherein the termination condition includes at least one of the following: When the end position error If the task is deemed successful, the current round ends, and a one-time positive reward is added to the single-step reward. When the cumulative number of interaction sampling steps in the current round reaches the preset maximum number of steps, or the cumulative execution time exceeds the preset time limit, it is judged as a timeout and the current round is terminated; When a violation of system security constraints is detected, the current round is immediately terminated, and a penalty is added to the single-step reward. The security constraints include at least one of the following: The joint angle exceeds the mechanical limit range; The joint increment exceeds the set anomaly risk threshold; The robotic arm enters a singular configuration, and the minimum singular value of the Jacobian matrix is ​​below the singularity risk threshold.

Images