Trajectory redirection method, apparatus, device, storage medium, and program product

Abstract

Embodiments of the present application disclose a trajectory reorientation method, device, equipment, storage medium and program product, which are used to reduce physical unfeasible problems such as end inaccessibility, jaw penetration, contact point offset, unstable contact and release drag. The method comprises: obtaining a training sample, extracting an operation intention based on human-hand-object interaction data, and generating an initial trajectory and an initial contact state including each operation stage based on the operation intention, object geometric information and robot end parameters; generating a plurality of candidate trajectories including each operation stage in the vicinity of the initial trajectory; verifying inverse kinematics accessibility, collision, contact stability and object motion consistency of the plurality of candidate trajectories; determining a teacher trajectory and a target contact state; constructing a residual label based on a target trajectory difference between the teacher trajectory and the initial trajectory and a target contact state difference between the target contact state and the initial contact state; and training a stage adaptive residual model based on the residual label.

Description

Technical Field

[0001] This application relates to the field of trajectory redirection, and more specifically, to trajectory redirection methods, apparatus, devices, storage media, and program products. Background Technology

[0002] With the development of human hand-object interaction (HOI) data acquisition technology and its widespread application in the field of robot operation, more and more research is dedicated to transferring human operation skills to robot end effectors. Therefore, it is necessary to convert human hand movements into robot-executable end trajectories.

[0003] Existing trajectory retargeting methods mainly include: directly generating robot trajectories based on human hand joint, fingertip position, or wrist pose mapping; using imitation learning or reinforcement learning to train end-to-end control strategies; and superimposing residual actions after generating the initial trajectory for correction.

[0004] However, due to the inherent differences between human hands and robot end effectors in terms of the number of contact surfaces, gripper opening and closing range, tool center point, reachable space and contact stability, directly tracking the trajectory of human hands can easily lead to physical infeasibility problems such as end effector unreachability, gripper clipping, contact point offset, unstable contact and release dragging. Summary of the Invention

[0005] This application provides a trajectory redirection method, trajectory redirection device, trajectory redirection equipment, computer-readable storage medium, and computer program product containing instructions, for reducing physical infeasibility problems such as end-effector unreachability, gripper penetration, contact point offset, unstable contact, and release dragging.

[0006] In a first aspect, embodiments of this application provide a trajectory redirection method, including:

[0007] Acquire training samples, which include human hand-object interaction data, object geometric information, and robot end-effector parameters;

[0008] Based on the human hand-object interaction data, the operation intention is extracted, and based on the operation intention, the object's geometric information, and the robot's end effector parameters, an initial trajectory and initial contact state including each operation stage are generated.

[0009] Multiple candidate trajectories, including each operation stage, are generated by perturbing near the initial trajectory. The inverse kinematic reachability, collision, contact stability, and object motion consistency of the multiple candidate trajectories are verified by a physical simulator to determine the teacher trajectory and target contact state.

[0010] Based on the difference between the target trajectory of the teacher's trajectory and the initial trajectory, and the difference between the target contact state and the initial contact state, residual labels are constructed, and a stage adaptive residual model is trained based on the residual labels to obtain the trained stage adaptive residual model.

[0011] Secondly, embodiments of this application provide a trajectory redirection device, comprising:

[0012] An acquisition unit is used to acquire training samples, which include human hand-object interaction data, object geometric information, and robot end-effector parameters.

[0013] The generation unit is used to extract the operation intention based on the human hand-object interaction data, and generate the initial trajectory and initial contact state for each operation stage based on the operation intention, the object geometric information and the robot end-effector parameters.

[0014] The determination unit is used to generate multiple candidate trajectories including each operation stage by perturbing near the initial trajectory. The inverse kinematic reachability, collision, contact stability and object motion consistency of the multiple candidate trajectories are verified by a physical simulator to determine the teacher trajectory and target contact state.

[0015] The training unit is used to construct residual labels based on the difference between the target trajectory of the teacher trajectory and the initial trajectory, and the difference between the target contact state and the initial contact state, and to train a stage adaptive residual model based on the residual labels to obtain a trained stage adaptive residual model.

[0016] Thirdly, embodiments of this application provide a trajectory redirection device, comprising:

[0017] Central processing unit, memory, input / output interfaces, wired or wireless network interfaces, and power supply;

[0018] The memory is either a short-term storage memory or a persistent storage memory;

[0019] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the aforementioned trajectory redirection method.

[0020] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the aforementioned trajectory redirection method.

[0021] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute the aforementioned trajectory redirection method.

[0022] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: This application does not directly generate end-to-end control strategies from human hand-object interaction data, but first uses explicit rules to generate interpretable initial trajectories, and then uses a physical simulator to batch verify the inverse kinematics reachability, collision, contact stability and object motion consistency of candidate trajectories to screen teacher trajectories. Then, residual labels are constructed based on the differences between teacher trajectories and initial trajectories and the differences between target contact states and initial contact states, so that the model only learns high-level trajectory correction and contact relationship correction that have been verified by simulation. In this way, the semantic understanding, obstacle avoidance, contact stability and motion consistency originally coupled in the end-to-end control strategy are decomposed into engineering modules that can be independently diagnosed and iterated layer by layer, thereby significantly reducing physical infeasibility problems such as end-point unreachability, gripper clipping, contact point offset, unstable contact and release dragging.

[0023] Accordingly, the trajectory redirection device, trajectory redirection equipment, computer-readable storage medium, and computer program product containing instructions provided in this application also have the aforementioned technical effects. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a trajectory redirection method disclosed in an embodiment of this application;

[0025] Figure 2 This is a flowchart illustrating an initial trajectory and contact state generation method disclosed in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of a stage-adaptive residual model structure disclosed in an embodiment of this application;

[0027] Figure 4 This is a flowchart illustrating another trajectory redirection method disclosed in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the structure of a trajectory redirection device disclosed in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the structure of a trajectory redirection device disclosed in an embodiment of this application. Detailed Implementation

[0030] This application provides a trajectory redirection method, trajectory redirection device, trajectory redirection equipment, computer-readable storage medium, and computer program product containing instructions, for reducing physical infeasibility problems such as end-effector unreachability, gripper penetration, contact point offset, unstable contact, and release dragging.

[0031] Please see Figure 1 , Figure 1This is a flowchart illustrating a trajectory redirection method disclosed in an embodiment of this application. The method includes:

[0032] 101. Obtain training samples, which include human hand-object interaction data, object geometric information, and robot end-effector parameters.

[0033] In one alternative implementation, human hand-object interaction data refers to data recording the sequence of hand postures, the object's trajectory, and the hand-object contact relationship during the interaction process. Object geometric information refers to data describing the three-dimensional shape and structure of the manipulated object. Robot end effector parameters refer to parameters describing the structural characteristics and motion constraints of the robot's end effector.

[0034] 102. Extract the operation intention based on human hand-object interaction data, and generate the initial trajectory and initial contact state for each operation stage based on the operation intention, object geometric information and robot end parameters.

[0035] In one optional implementation, the operational intent refers to high-level semantic information parsed frame-by-frame from human-object interaction data to guide the robot's end effector actions, including but not limited to target contact point, contact normal, local object geometry, equivalent opening / closing width, and stage markers. Each operational stage refers to a continuous temporal sequence during the interaction between the robot's end effector and the object, including but not limited to the approach stage, contact stage, operation stage, and release stage. The initial trajectory is a temporal sequence covering the robot tool center point pose and gripper opening / closing amount for each operational stage. The initial contact state is a temporal sequence corresponding frame-by-frame to the initial trajectory, recording the target contact point, contact normal, stage contact offset, and stage markers.

[0036] 103. Multiple candidate trajectories, including each operation stage, are generated by perturbing near the initial trajectory. The inverse kinematic reachability, collision, contact stability, and object motion consistency of the multiple candidate trajectories are verified by a physical simulator to determine the teacher trajectory and target contact state.

[0037] In one optional implementation, the teacher trajectory and target contact state refer to the optimal candidate trajectory and its corresponding contact state verified and screened by the physical simulator. The teacher trajectory is a temporal sequence covering the robot tool center point pose and gripper opening / closing amount for each operation stage. The target contact state is a temporal sequence corresponding frame-by-frame to the teacher trajectory, recording the target contact point, contact normal, stage contact offset, and stage marker. Specifically, perturbations can be applied to the tool center point position, pose, and gripper opening / closing amount for each operation stage in the initial trajectory to generate multiple candidate trajectories. These candidate trajectories are then input into the physical simulator, where inverse kinematics checks, collision detection, contact mechanics deduction, and object motion consistency evaluation are performed on each trajectory. Multi-objective scoring is performed based on reachability, contact consistency, object motion consistency, and smoothness. The candidate trajectory with the best score is determined as the teacher trajectory, and the contact state generated by this teacher trajectory during simulation execution is recorded as the target contact state.

[0038] 104. Based on the difference between the target trajectory and the teacher trajectory and the initial trajectory, as well as the difference between the target contact state and the initial contact state, construct residual labels, and train the stage adaptive residual model based on the residual labels to obtain the trained stage adaptive residual model.

[0039] In one optional implementation, the target trajectory difference between the teacher trajectory and the initial trajectory refers to the frame-by-frame difference between the teacher trajectory and the initial trajectory in terms of tool center point pose and gripper opening / closing amount. The target contact state difference between the target contact state and the initial contact state refers to the frame-by-frame difference between the target contact state and the initial contact state in terms of target contact point, contact normal, stage contact offset, and stage boundary. The residual label refers to the data label used for supervising the training phase of the adaptive residual model, which consists of the target trajectory difference, the target contact state difference, and the quality score.

[0040] In this way, this application does not directly generate end-to-end control strategies from human-object interaction data. Instead, it first generates interpretable initial trajectories using explicit rules, and then uses a physical simulator to batch verify the inverse kinematic reachability, collision, contact stability, and object motion consistency of candidate trajectories to filter teacher trajectories. Residual labels are then constructed based on the differences between the teacher trajectories and the initial trajectories, as well as the differences between the target contact state and the initial contact state. This allows the model to learn only the high-level trajectory corrections and contact relationship corrections verified by simulation. This decomposes the semantic understanding, obstacle avoidance, contact stability, and motion consistency originally coupled in the end-to-end control strategy into independently diagnosable, iteratively layered processing modules, significantly reducing physically infeasible problems such as end-effector unreachability, gripper clipping, contact point offset, unstable contact, and release dragging. Secondly, by writing the simulated contact state into the residual labels, the model learns trajectory corrections and contact relationship corrections simultaneously, further ensuring physical contact stability. Furthermore, using teacher searches near the initial trajectory to form supervision signals reduces the reliance on artificial robot demonstration trajectories.

[0041] In one optional implementation, the operational intent includes the target contact point, contact normal, local object geometry, equivalent opening / closing width, and stage markers. Extracting the operational intent based on human hand-object interaction data includes: analyzing the interaction information between the human hand and the object frame by frame based on the human hand-object interaction data; extracting the target contact point from the actual contact area or high contact probability area between the human hand and the object; extracting the contact normal from the object surface geometry at the target contact point; extracting the local object geometry from the object mesh or point cloud data; calculating the equivalent opening / closing width based on the human hand contact span or the local thickness of the object; and determining the stage markers based on the hand-object distance, contact probability, and changes in object velocity. When multiple contact areas exist, the multiple contact areas are filtered or merged based on the object's motion direction, contact probability, and local thickness, and the center of the contact area is taken as the target contact point.

[0042] Specifically, the interaction information between human hand and object refers to the information recorded frame-by-frame in human hand-object interaction data, reflecting the spatial positional relationship, contact state, and relative motion process between the human hand and the object. Operational intent includes the target contact point, contact normal, local object geometry, equivalent opening / closing width, and stage markers. The target contact point is the spatial location of the actual contact area or high-probability contact area between the hand and the object, or it can be the closest area from the hand geometry to the object's surface; the contact normal is the outward normal direction of the object's surface at the target contact point. Local object geometry refers to the surface shape and size characteristics of the area adjacent to the target contact point, represented by an object mesh or point cloud. The equivalent opening / closing width is the opening / closing amount of the robot gripper, calculated based on the human hand contact span or the local thickness of the object. Stage markers are identifiers indicating whether the current operation is in the approach, contact, operation, or release stage; they are directly read from the human hand-object interaction data or automatically determined based on the hand-object distance, contact probability, and changes in object velocity. During extraction, when multiple contact areas exist within the same frame, matching areas are selected based on the object's motion direction, contact probability, and local thickness. Adjacent contact points are merged into contact patches, and the center of each patch is taken as the target contact point. The contact normal and local object geometry are read from the object's surface geometry, and the equivalent opening / closing width is calculated based on the local thickness or the span of human hand contact. The extracted target contact point and contact normal are first recorded in the object's local coordinate system, and then transformed to the world coordinate system as the object's pose changes, thereby generating the robot's action center and tool center point poses.

[0043] In this way, by extracting the target contact point from the actual contact area or high-probability area, and combining the object's local geometry and dynamic stage markings, the human operation intention can be accurately mapped into the robot's executable parameters, avoiding gripping offset or unstable gripping caused by improper selection of the contact area.

[0044] In one optional implementation, based on the operation intention, object geometry information, and robot end-effector parameters, an initial trajectory and initial contact state for each operation stage are generated. This includes: determining the tool center point pose for each operation stage based on the target contact point, contact normal, stage markers, and object geometry information, combined with the tool center point calibration offset in the robot end-effector parameters; generating the gripper opening / closing amount based on the equivalent opening / closing width and the gripper opening / closing range constraint in the robot end-effector parameters; combining the tool center point pose and gripper opening / closing amount for each operation stage frame by frame, and achieving a smooth transition between operation stages through interpolation to generate the initial trajectory; generating the contact offset corresponding to each operation stage based on the stage markers; combining the target contact point, contact normal, contact offset, and stage markers frame by frame, and achieving a smooth transition between operation stages through interpolation to generate the initial contact state.

[0045] Specifically, the tool center point calibration offset refers to the fixed geometric transformation parameters between the robot's end effector flange coordinate system and the actual center of action of the gripper. The tool center point pose for each operation stage refers to the position and orientation of the end effector at each stage calculated based on the target contact point, contact normal, and stage markers, combined with the tool center point calibration offset. The equivalent opening / closing width refers to the theoretical opening amount of the gripper, converted from the human hand contact span or the local thickness of the object. The gripper opening / closing range constraint refers to the maximum and minimum opening / closing width allowed by the mechanical structure. The tool center point pose for each operation stage is combined frame-by-frame with the constraint-trimmed gripper opening / closing amount, and interpolation is used to achieve a smooth transition between operation stages, generating the initial trajectory. The contact offset corresponding to each operation stage is generated based on the stage markers. The target contact point, contact normal, contact offset, and stage markers are combined frame-by-frame, and interpolation is used to achieve a smooth transition between operation stages, generating the initial contact state.

[0046] More specifically, the generation mechanism of the initial trajectory and initial contact state is as follows: The system takes the sequence of operational intentions and robot end-effector parameters as inputs. The end-effector parameters include the calibration offset from the tool center point to the action center, the gripper opening and closing range, the default approach axis, and the default closed axis. First, the robot action center is calculated from the target contact point, the contact normal, and the stage contact offset. Then, the spatial position of the tool center point is obtained by combining the calibration offset. At the same time, the gripper opening and closing amount is generated based on the equivalent opening and closing width and under the constraint of the gripper opening and closing range. The two together constitute the initial result, which is expressed as the following formula:

[0047] Formula 1

[0048] Wherein, τ0 represents the initial tool center point position trajectory and gripper opening / closing trajectory, and c0 represents the initial contact state; c0 consists of the target contact point, contact normal, stage contact offset, and stage marker. The stage contact offset is the signed distance of the robot's action center relative to the target contact point along the object's normal. A positive value in the approach stage indicates the robot is located outside the object; a value close to zero in the contact stage indicates it is in contact with the target area; a small negative value is maintained in the operation stage to preserve contact; and a larger positive value in the release stage indicates it is moving outward along the normal to withdraw. The tool center point attitude is determined by the target contact normal and the default approach axis. The roll angle is selected as the solution with the minimum distance from the attitude angle of the previous frame; the first frame roll angle is determined by the default closed axis and the local orientation of the object. The geometric behavior of each stage is clearly distinguished: in the approach stage, the trajectory is located outside the target contact point and approaches the object along the normal; in the contact stage, the robot reaches the target contact area; in the operation stage, the robot maintains contact with the object reference motion; and in the release stage, the robot moves outward along the normal and opens the gripper. Transition frames between stages are obtained by interpolating the contact offset of adjacent stages with the tool center point attitude to ensure a continuous and smooth trajectory.

[0049] For more details, please refer to Figure 2 , Figure 2This is a flowchart illustrating a method for generating an initial trajectory and contact state according to an embodiment of this application. Figure 2 This paper demonstrates the generation process of the initial trajectory and initial contact state. The process takes robot end-effector parameters (including tool center point calibration offset, gripper opening / closing range, and default approach axis) and operational intent (including target contact point, contact normal, local object geometry, equivalent opening / closing width, and stage markers) as input. It generates two types of outputs through basic redirection rules: first, the initial trajectory, which is the spatial position and attitude sequence of the tool center point and the gripper opening / closing sequence; second, the initial contact state, which is the sequence of target contact point, contact normal, stage contact offset, and stage markers corresponding to each frame of the trajectory. The initial trajectory is obtained by jointly mapping the robot end-effector parameters and operational intent, while the initial contact state is directly derived from the contact elements in the operational intent. These two correspond to each other and together form the benchmark for subsequent candidate trajectory generation and simulation screening.

[0050] In this way, high-level operational intentions can be transformed into end-effector motion references that the robot can directly execute, and the continuity of the trajectory and physical executability can be guaranteed through smooth transitions between stages.

[0051] In one optional implementation, multiple candidate trajectories, including each operation stage, are generated by perturbing the area near the initial trajectory. The inverse kinematics reachability, collision, contact stability, and object motion consistency of the multiple candidate trajectories are verified by a physics simulator to determine the teacher trajectory and the target contact state. This includes: perturbing the tool center point posture and gripper opening and closing amount in each operation stage of the initial trajectory to generate multiple candidate trajectories; inputting the multiple candidate trajectories into the physics simulator; performing inverse kinematics checks, collision detection, and physical deduction on each candidate trajectory; verifying the inverse kinematics reachability, collision, contact stability, and object motion consistency of the multiple candidate trajectories; performing multi-target scoring based on the above verification results; determining the candidate trajectory with the best score as the teacher trajectory; and recording the contact state generated by the teacher trajectory during the physical deduction process as the target contact state.

[0052] Specifically, multiple candidate trajectories representing each operational stage are generated near the initial trajectory. These are obtained by applying perturbations to the tool's center point position, attitude, and gripper opening / closing amount. Translational perturbations include inward / outward movement along the contact normal, local translation along the contact tangent, and forward / backward movement along the approach direction. Attitude perturbations include small-angle rotations around the approach and closing axes. Opening / closing perturbations include increasing or decreasing the gripper opening / closing amount. Candidate trajectories are generated according to time windows, with adjacent frames sharing smoothing constraints to avoid abrupt trajectory changes caused by single-frame corrections. Candidates failing inverse kinematics or collision detection are not included in the physics simulation. The candidate trajectories are input into the physics simulator, where inverse kinematics checks, collision detection, and physics simulations are performed one by one. A multi-objective score is then calculated based on the simulation screening score, using the following formula:

[0053] Formula 2

[0054] Where Mexec represents execution feasibility, Mcontact represents the consistency between the simulated contact result and the target contact area and local geometry, Mobj represents the consistency of object motion, Msmooth represents trajectory smoothness, and Mres represents the correction magnitude. By maximizing this score, the candidate trajectory with the best score is determined as the teacher trajectory, and the contact state generated by this teacher trajectory during the physical deduction process is recorded as the target contact state. Please refer to Formula 3 for details.

[0055] Formula 3

[0056] Where i is the candidate number, H is the hand-object interaction sequence, and R is the robot model and simulation environment parameters.

[0057] In this way, by systematically perturbing each stage of the initial trajectory to generate candidates and verifying them through physical simulation, the optimal execution scheme that satisfies both the robot's body constraints and ensures contact stability can be automatically searched, avoiding the possibility of clipping, slippage, or joint unreachability that may result from directly executing the initial trajectory.

[0058] In one optional implementation, residual labels are constructed based on the differences between the teacher's trajectory and the initial trajectory, as well as the differences between the target contact state and the initial contact state. This includes: determining the trajectory residual between the teacher's trajectory and the initial trajectory, where the trajectory residual includes the tool center point pose residual and the gripper opening / closing amount residual; determining the contact state residual between the target contact state and the initial contact state, where the contact state residual includes the contact normal residual, the stage contact offset residual, and the stage boundary residual; generating a quality score label based on the verification results of the teacher's trajectory using a physical simulator; and constructing residual labels based on the trajectory residual, the contact state residual, and the quality score label.

[0059] Specifically, using the teacher's trajectory and contact state verified through physical simulation as the optimal benchmark, the frame-by-frame differences between it and the initial trajectory and initial contact state are calculated. The difference between the tool center point pose and the gripper opening / closing amount constitutes the trajectory residual, while the differences in contact normal, stage contact offset, and stage boundary constitute the contact state residual. Simultaneously, a quality score is generated based on the comprehensive verification results of the teacher's trajectory by the simulator. The aforementioned trajectory residuals, contact state residuals, and quality scores are combined into residual labels, which are used as supervisory signals to train the residual model.

[0060] More specifically, residual labels are constructed based on the differences between the teacher's trajectory and the initial trajectory, as well as the differences between the target contact state and the initial contact state. The residual labels are as follows: Formula 4:

[0061] Formula 4

[0062] The trajectory residual is This includes the tool center point pose residual and the gripper opening / closing amount residual, with the contact state residual being... This includes contact normal residuals, stage contact offset residuals, and stage boundary residuals. Simultaneously, a quality score is generated based on the verification results of the teacher's trajectory using a physical simulator. The above components are combined into residual labels.

[0063] Understandably, if none of the candidates meet the minimum execution constraints, the failure type is recorded, including unreachable, collision, clipping, non-contact, slip, object movement deviation, and release drag. This failure sample can be used for quality score training or to expand the candidate search range in subsequent rounds.

[0064] In this way, residual labels essentially transform the optimal correction scheme verified by physical simulation into structured supervision data, enabling the model to learn trajectory correction and contact relationship correction simultaneously, thereby significantly improving the physical feasibility and contact stability of the output trajectory.

[0065] In one optional implementation, the stage adaptive residual model includes a temporal encoder, a stage gating network, and prediction branches for each operation stage. The stage adaptive residual model is trained based on residual labels to obtain the trained stage adaptive residual model. This includes: using the operation intention, initial trajectory, initial contact state, and stage marker within the time window as inputs to the stage adaptive residual model; extracting continuous frame temporal features through the temporal encoder; generating dynamic weights for each operation stage prediction branch based on the continuous frame temporal features and stage markers through the stage gating network; outputting the trajectory residual, contact state residual, and quality score components for each operation stage based on the continuous frame temporal features; and weighting and fusing the outputs of each operation stage prediction branch based on the dynamic weights to output the predicted trajectory residual, predicted contact state residual, and predicted quality score; and constructing a supervision signal based on the differences between the predicted trajectory residual, predicted contact state residual, and predicted quality score and the corresponding items in the residual labels to constrain the training of the stage adaptive residual model.

[0066] Specifically, continuous frame temporal features refer to the feature representation extracted by the temporal encoder from multiple consecutive frames of input within a time window, containing the sequence and dependencies between frames. Dynamic weights refer to the weight coefficients assigned by the stage gating network to each operation stage prediction branch based on the stage label of the current frame, determining its output contribution. These weights are learned by the model and dynamically adjusted as the stage changes. A time window refers to a continuous frame sequence centered on the current frame and extending forward and backward by several frames, used to provide the model with temporal context information before and after the current operation stage. The approach stage prediction branch focuses on correcting the tool center point approach pose and target contact point; the contact stage prediction branch focuses on correcting the gripper opening / closing amount, contact normal, and stage contact offset; the operation stage prediction branch focuses on correcting the tool center point operation trajectory and contact state stability; and the release stage prediction branch focuses on correcting the gripper opening / closing amount, tool center point withdrawal trajectory, and release timing.

[0067] More specifically, the stage-adaptive residual model takes the sequence of operational intentions, initial trajectory, initial contact state, local geometric features of the object, and stage markers within a time window as input. The time window covers several frames before and after the current frame, enabling the model to obtain the continuous relationship between approach, contact, operation, and release. The local geometric features of the object include the scale of the contact area, local thickness, and local orientation changes. The model comprises three core components: a temporal encoder, a stage gating, and stage prediction branches. The temporal encoder extracts temporal features from consecutive frames. The stage gating input is the hand-object interaction stage marker or a stage marker derived from the hand-object interaction. The model does not use its own predicted stage results to select branches; it only changes the output weights of each stage prediction branch without altering the input data stream. Each stage prediction branch shares the same temporal encoding result. The model prediction formula is as follows: Formula 5:

[0068] Formula 5

[0069] Where ht represents the temporal coding feature. Here, Pm represents the stage label, Pm represents the stage prediction branch, and gm represents the stage gating weight. The final execution reference trajectory for frame t is determined by the initial trajectory τ0,t and the trajectory residual. The contact state residuals are obtained by superposition. and quality rating It participates in training, screening, and review, but is not used as an independent control command. The prediction branches in each stage have different focuses: the approach stage focuses on approach distance and attitude correction; the contact stage focuses on opening / closing amount and contact direction; the manipulation stage focuses on maintaining contact and trajectory stability; and the release stage focuses on opening / closing changes, outward displacement distance, and release boundaries. During training, trajectory residuals, contact state residuals, and quality scores together form the supervision signal. Trajectory residuals constrain the final reference trajectory, contact state residuals constrain and correct the contact relationships corresponding to the trajectory, and the quality score distinguishes between executable samples and low-quality samples.

[0070] For more details, please refer to Figure 3 , Figure 3 This is a schematic diagram of a staged adaptive residual model structure disclosed in an embodiment of this application. Figure 3 It can be seen that the stage adaptive residual model takes the operation intention, initial trajectory, initial contact state and stage mark as input, extracts the temporal features of continuous frames by the time encoder, and generates dynamic weights for each stage prediction branch according to the stage mark by the stage gating. The stage prediction branch outputs trajectory residual, contact state residual and quality score respectively based on the temporal features. After weighted fusion, the trajectory residual is superimposed on the initial trajectory to generate the corrected robot end reference trajectory. The contact state residual and quality score participate in training supervision and simulation verification, and are not directly issued as control commands.

[0071] In this way, by using stage-gated dynamic weighting, the model can activate corresponding branches at different operation stages, achieving precise stage-by-stage correction, avoiding confusion of the main contradictions at different stages by a unified model, and improving the accuracy and physical feasibility of trajectory redirection.

[0072] In one optional implementation, after obtaining the trained stage adaptive residual model, the method further includes: acquiring data to be processed, which includes target human hand-object interaction data, target object geometric information, and target robot end-effector parameters; acquiring target operation intention, target trajectory, target contact state, and target stage marker based on the target human hand-object interaction data, target object geometric information, and target robot end-effector parameters; inputting the target operation intention, target trajectory, target contact state, and target stage marker into the trained stage adaptive residual model to obtain the predicted trajectory, predicted contact state, and predicted quality score corresponding to the data to be processed output by the stage adaptive residual model.

[0073] Specifically, in the application inference stage, the target's operational intent is extracted from the new target human-object interaction data, and the target's initial trajectory and initial contact state are generated. These are then input into the trained stage adaptive residual model along with the target stage label. The model directly outputs the predicted trajectory residual, the predicted contact state residual, and the predicted quality score. The predicted trajectory residual is then superimposed on the target's initial trajectory to obtain the corrected predicted trajectory.

[0074] More specifically, in the application inference stage, the target's operational intent is extracted based on the target human-object interaction data, the target object's geometric information, and the target robot's end-effector parameters. This generates the target's initial trajectory and initial contact state, which, along with the target's stage markers, are input into the trained stage adaptive residual model. The model directly outputs the predicted trajectory residual, the predicted contact state residual, and the predicted quality score. The predicted trajectory residual is superimposed on the target's initial trajectory to obtain the corrected predicted trajectory. This trajectory can be input into low-level controllers such as inverse kinematics solvers, motion planners, or model predictive controllers for execution. The predicted contact state residual and the predicted quality score do not directly drive robot actions; they are only used to determine whether the current correction result requires simulation verification.

[0075] In this way, the candidate trajectory perturbation generation and simulation screening do not need to be re-executed during the application phase, which significantly reduces inference latency and ensures the physical feasibility of the output trajectory.

[0076] In one optional implementation, after obtaining the predicted trajectory, predicted contact state, and predicted quality score corresponding to the data to be processed output by the stage adaptive residual model, the method further includes: performing simulation verification based on the predicted quality score corresponding to the data to be processed; if the verification fails, taking the predicted trajectory corresponding to the data to be processed as a new initial trajectory, and re-executing candidate trajectory generation, simulation screening, and residual label construction to supplement the training data.

[0077] Specifically, simulation verification refers to inputting the predicted trajectory into a physical simulator after generation to re-verify inverse kinematics reachability, collision and contact stability, and consistency of object motion. Samples that fail verification are returned to the teacher's trajectory generation process, where candidate trajectory perturbation generation, simulation screening, and residual label construction are re-executed to supplement training data. Samples that pass verification, along with simulation records, are added to the sample library. This closed-loop feedback consists of model output, simulation verification, teacher confirmation, and sample updates.

[0078] More specifically, after the predicted trajectory is output, it is simulated and verified based on the prediction quality score. If the quality score is lower than a preset threshold, or if unreachable, collision, clipping, non-contact, slippage, object movement deviation, or release drag occurs during the verification, the verification is deemed a failure. The predicted trajectory is then used as a new initial trajectory to re-execute candidate trajectory perturbation generation, physical simulation screening, and residual label construction. After confirming the new teacher trajectory and teacher contact status, it is added to the training set. Samples that pass the verification are saved as verification samples, along with the predicted trajectory, simulation contact record, and quality score. The closed-loop update does not change the output source in the application stage; the model always outputs the predicted trajectory. The simulation verification only determines whether the sample is returned to the teacher search process and whether the simulation record is included in the training set or the verification sample library.

[0079] For more details, please refer to Figure 4 , Figure 4 This is a flowchart illustrating another trajectory redirection method disclosed in an embodiment of this application. Figure 4 It can be seen that the human-object interaction data sequence and object geometry information can be input into the initial redirection module to generate an interpretable initial trajectory and initial contact state. Subsequently, multiple candidate trajectories are generated by perturbation near the initial trajectory. After inverse kinematics checking, collision detection, and physical deduction by the physics simulator, the optimal teacher trajectory is selected based on multi-objective scoring, and its contact state is recorded. Then, the differences between the teacher trajectory and the initial trajectory, as well as the differences between the teacher contact state and the initial contact state, are calculated to construct residual labels for the adaptive residual model during the training phase. In the application phase, the model directly outputs the predicted trajectory residual for new inputs and superimposes it onto the corrected predicted trajectory generated from the initial trajectory. This is then verified through application simulation. If the verification fails, the predicted trajectory is written back as the new initial trajectory to the candidate trajectory generation and simulation screening process to re-determine the teacher trajectory and supplement training data, forming a closed-loop feedback loop of "model output, simulation verification, teacher confirmation, and sample update."

[0080] In this way, the failure boundary of the model can be automatically identified and difficult samples can be re-introduced without manual intervention during the application phase, so that the training set can continuously cover more complex contact situations and achieve zero-downtime self-evolution of the system.

[0081] For further details, please refer to Figure 5 One embodiment of the trajectory redirection device in this application includes:

[0082] An acquisition unit is used to acquire training samples, which include human hand-object interaction data, object geometric information, and robot end-effector parameters.

[0083] The generation unit is used to extract the operation intention based on the human hand-object interaction data, and generate the initial trajectory and initial contact state for each operation stage based on the operation intention, the object geometric information and the robot end-effector parameters.

[0084] The determination unit is used to generate multiple candidate trajectories including each operation stage by perturbing near the initial trajectory. The inverse kinematic reachability, collision, contact stability and object motion consistency of the multiple candidate trajectories are verified by a physical simulator to determine the teacher trajectory and target contact state.

[0085] The training unit is used to construct residual labels based on the difference between the target trajectory of the teacher trajectory and the initial trajectory, and the difference between the target contact state and the initial contact state, and to train a stage adaptive residual model based on the residual labels to obtain a trained stage adaptive residual model.

[0086] In one alternative implementation, the generating unit may be used for:

[0087] Based on the human hand-object interaction data, the interaction information between the human hand and the object is analyzed frame by frame. The target contact point is extracted from the actual contact area or high contact probability area between the human hand and the object. The contact normal is extracted from the object surface geometry at the target contact point. The local object geometry is extracted from the object mesh or point cloud data. The equivalent opening and closing width is calculated based on the human hand contact span or the local thickness of the object. The stage marker is determined according to the hand-object distance, contact probability and object speed change.

[0088] When multiple contact areas exist, the multiple contact areas are screened or merged based on the object's movement direction, contact probability, and local thickness, and the center of the contact area is taken as the target contact point.

[0089] The operational intent includes the target contact point, contact normal, local object geometry, equivalent opening / closing width, and stage markers.

[0090] In one alternative implementation, the generating unit may be used for:

[0091] Based on the target contact point, the contact normal, the stage marker, and the object geometry information, combined with the tool center point calibration offset in the robot end effector parameters, the tool center point pose for each operation stage is determined.

[0092] The gripper opening / closing amount is generated based on the equivalent opening / closing width and the gripper opening / closing range constraint in the robot end effector parameters.

[0093] The initial trajectory is generated by combining the tool center point pose of each operation stage with the opening and closing amount of the gripper frame by frame, and by interpolation to achieve a smooth transition between operation stages.

[0094] Based on the stage marker, the contact offset corresponding to each operation stage is generated. The target contact point, the contact normal, the contact offset, and the stage marker are combined frame by frame. Interpolation is used to achieve a smooth transition between operation stages to generate the initial contact state.

[0095] In one alternative implementation, the determining unit may be used for:

[0096] The tool center point attitude and gripper opening / closing amount in each operation stage of the initial trajectory are perturbed to generate the multiple candidate trajectories.

[0097] The multiple candidate trajectories are input into the physics simulator, and inverse kinematics checks, collision detection, and physics deduction are performed on each one to verify the inverse kinematics reachability, collision and contact stability, and object motion consistency of the multiple candidate trajectories. Multi-objective scoring is then performed based on the above verification results.

[0098] The candidate trajectory with the best score is determined as the teacher trajectory, and the contact state generated by the teacher trajectory during the physical deduction process is recorded as the target contact state.

[0099] In one alternative implementation, the training unit may be used for:

[0100] Determine the trajectory residual between the teacher trajectory and the initial trajectory, wherein the trajectory residual includes the tool center point pose residual and the gripper opening / closing amount residual;

[0101] Determine the contact state residual between the target contact state and the initial contact state, wherein the contact state residual includes the contact normal residual, the stage contact offset residual, and the stage boundary residual;

[0102] Based on the verification results of the teacher's trajectory by the physical simulator, a quality score label is generated;

[0103] The residual label is constructed based on the trajectory residual, the contact state residual, and the quality score label.

[0104] In one alternative implementation, the training unit may be used for:

[0105] The operational intent, initial trajectory, initial contact state, and stage marker within the time window are used as inputs to the stage adaptive residual model, and the temporal features of consecutive frames are extracted by the temporal encoder.

[0106] The stage-gated network generates dynamic weights for the prediction branches of each operation stage based on the temporal features of the continuous frames and the stage markers.

[0107] Each operation stage prediction branch outputs the trajectory residual, contact state residual, and quality score component of each operation stage based on the temporal features of the continuous frames. The outputs of each operation stage prediction branch are weighted and fused based on the dynamic weights to output the predicted trajectory residual, predicted contact state residual, and predicted quality score.

[0108] Based on the predicted trajectory residual, the predicted contact state residual, and the difference between the predicted quality score and the corresponding item in the residual label, a supervision signal is constructed to constrain the training of the stage adaptive residual model;

[0109] The stage-adaptive residual model includes a time encoder, a stage-gated network, and prediction branches for each operation stage.

[0110] In one alternative implementation, the acquisition unit may also be used for:

[0111] Acquire data to be processed, which includes target human hand-object interaction data, target object geometric information, and target robot end effector parameters;

[0112] Based on the target human hand-object interaction data, the target object's geometric information, and the target robot's end-effector parameters, the target's operational intent, target trajectory, target contact state, and target stage markers are obtained.

[0113] The target operation intention, the target trajectory, the target contact state, and the target stage marker are input into the trained stage adaptive residual model to obtain the predicted trajectory, predicted contact state, and predicted quality score corresponding to the data to be processed output by the stage adaptive residual model.

[0114] In one optional implementation, the trajectory redirection device further includes a simulation verification unit and an execution unit;

[0115] The simulation verification unit is used to perform simulation verification on the predicted trajectory corresponding to the data to be processed based on the prediction quality score corresponding to the data to be processed.

[0116] The execution unit is used to, if the verification fails, take the predicted trajectory corresponding to the data to be processed as the new initial trajectory, and re-execute the candidate trajectory generation, simulation screening and residual label construction to supplement the training data.

[0117] For further details, please refer to Figure 6 One embodiment of the trajectory redirection device in this application includes:

[0118] Central processing unit 601, memory 605, input / output interface 604, wired or wireless network interface 603, and power supply 602;

[0119] Memory 605 is either a short-term storage memory or a persistent storage memory;

[0120] The central processing unit 601 is configured to communicate with the memory 605 and execute instructions stored in the memory 605 to perform the aforementioned operations. Figure 1 The method in the illustrated embodiment.

[0121] Furthermore, embodiments of this application also provide a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to perform the aforementioned... Figure 1 The method in the illustrated embodiment.

[0122] Furthermore, embodiments of this application also provide a computer program product containing instructions, which, when run on a computer, causes the computer to perform the aforementioned... Figure 1 The method in the illustrated embodiment.

[0123] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0126] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0127] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A trajectory redirection method, characterized in that, include: Acquire training samples, which include human hand-object interaction data, object geometric information, and robot end-effector parameters; Based on the human hand-object interaction data, the operation intention is extracted, and based on the operation intention, the object's geometric information, and the robot's end effector parameters, an initial trajectory and initial contact state including each operation stage are generated. Multiple candidate trajectories, including each operation stage, are generated by perturbing near the initial trajectory. The inverse kinematic reachability, collision, contact stability, and object motion consistency of the multiple candidate trajectories are verified by a physical simulator to determine the teacher trajectory and target contact state. Based on the difference between the target trajectory of the teacher's trajectory and the initial trajectory, and the difference between the target contact state and the initial contact state, residual labels are constructed, and a stage adaptive residual model is trained based on the residual labels to obtain the trained stage adaptive residual model. The operational intent includes the target contact point, contact normal, local object geometry, equivalent opening / closing width, and stage markers. Extracting the operational intent based on the human hand-object interaction data includes: Based on the human hand-object interaction data, the interaction information between the human hand and the object is analyzed frame by frame. The target contact point is extracted from the actual contact area or high contact probability area between the human hand and the object. The contact normal is extracted from the object surface geometry at the target contact point. The local object geometry is extracted from the object mesh or point cloud data. The equivalent opening and closing width is calculated based on the human hand contact span or the local thickness of the object. The stage marker is determined according to the hand-object distance, contact probability and object speed change. When multiple contact areas exist, the multiple contact areas are screened or merged based on the object's movement direction, contact probability, and local thickness, and the center of the contact area is taken as the target contact point.

2. The method according to claim 1, characterized in that, The process of generating initial trajectories and initial contact states for each operational stage based on the operational intent, the object's geometric information, and the robot's end-effector parameters includes: Based on the target contact point, the contact normal, the stage marker, and the object geometry information, combined with the tool center point calibration offset in the robot end effector parameters, the tool center point pose for each operation stage is determined. The gripper opening / closing amount is generated based on the equivalent opening / closing width and the gripper opening / closing range constraint in the robot end effector parameters. The initial trajectory is generated by combining the tool center point pose of each operation stage with the opening and closing amount of the gripper frame by frame, and by interpolation to achieve a smooth transition between operation stages. Based on the stage marker, the contact offset corresponding to each operation stage is generated. The target contact point, the contact normal, the contact offset, and the stage marker are combined frame by frame. Interpolation is used to achieve a smooth transition between operation stages to generate the initial contact state.

3. The method according to claim 1, characterized in that, The process involves perturbing the initial trajectory to generate multiple candidate trajectories, each representing a different operational stage. A physical simulator is used to verify the inverse kinematic reachability, collision and contact stability, and object motion consistency of these candidate trajectories, ultimately determining the teacher's trajectory and the target contact state, including: The tool center point attitude and gripper opening / closing amount in each operation stage of the initial trajectory are perturbed to generate the multiple candidate trajectories. The multiple candidate trajectories are input into the physics simulator, and inverse kinematics checks, collision detection, and physics deduction are performed on each one to verify the inverse kinematics reachability, collision and contact stability, and object motion consistency of the multiple candidate trajectories. Multi-objective scoring is then performed based on the above verification results. The candidate trajectory with the best score is determined as the teacher trajectory, and the contact state generated by the teacher trajectory during the physical deduction process is recorded as the target contact state.

4. The method according to claim 1, characterized in that, The process of constructing residual labels based on the differences between the teacher's trajectory and the initial trajectory, and the differences between the target contact state and the initial contact state, includes: Determine the trajectory residual between the teacher trajectory and the initial trajectory, wherein the trajectory residual includes the tool center point pose residual and the gripper opening / closing amount residual; Determine the contact state residual between the target contact state and the initial contact state, wherein the contact state residual includes the contact normal residual, the stage contact offset residual, and the stage boundary residual; Based on the verification results of the teacher's trajectory by the physical simulator, a quality score label is generated; The residual label is constructed based on the trajectory residual, the contact state residual, and the quality score label.

5. The method according to claim 1, characterized in that, The stage-adaptive residual model includes a temporal encoder, a stage-gated network, and prediction branches for each operation stage. The stage-adaptive residual model is trained based on the residual labels to obtain the trained stage-adaptive residual model, including: The operational intent, initial trajectory, initial contact state, and stage marker within the time window are used as inputs to the stage adaptive residual model, and the temporal features of consecutive frames are extracted by the temporal encoder. The stage-gated network generates dynamic weights for the prediction branches of each operation stage based on the temporal features of the continuous frames and the stage markers. Each operation stage prediction branch outputs the trajectory residual, contact state residual, and quality score component of each operation stage based on the temporal features of the continuous frames. The outputs of each operation stage prediction branch are weighted and fused based on the dynamic weights to output the predicted trajectory residual, predicted contact state residual, and predicted quality score. Based on the predicted trajectory residual, the predicted contact state residual, and the difference between the predicted quality score and the corresponding item in the residual label, a supervision signal is constructed to constrain the training of the stage adaptive residual model.

6. The method according to claim 1, characterized in that, After obtaining the trained stage adaptive residual model, the method further includes: Acquire data to be processed, which includes target human hand-object interaction data, target object geometric information, and target robot end effector parameters; Based on the target human hand-object interaction data, the target object's geometric information, and the target robot's end-effector parameters, the target's operational intent, target trajectory, target contact state, and target stage markers are obtained. The target operation intention, the target trajectory, the target contact state, and the target stage marker are input into the trained stage adaptive residual model to obtain the predicted trajectory, predicted contact state, and predicted quality score corresponding to the data to be processed output by the stage adaptive residual model.

7. The method according to claim 6, characterized in that, After obtaining the predicted trajectory, predicted contact state, and predicted quality score corresponding to the data to be processed from the stage adaptive residual model output by the aforementioned method, the method further includes: The predicted trajectory corresponding to the data to be processed is simulated and verified based on the predicted quality score corresponding to the data to be processed. If the verification fails, the predicted trajectory corresponding to the data to be processed will be used as the new initial trajectory, and the candidate trajectory generation, simulation screening and residual label construction will be re-executed to supplement the training data.

8. A trajectory redirection device, characterized in that, include: An acquisition unit is used to acquire training samples, which include human hand-object interaction data, object geometric information, and robot end-effector parameters. The generation unit is used to extract the operation intention based on the human hand-object interaction data, and generate the initial trajectory and initial contact state for each operation stage based on the operation intention, the object geometric information and the robot end-effector parameters. The determination unit is used to generate multiple candidate trajectories including each operation stage by perturbing near the initial trajectory. The inverse kinematic reachability, collision, contact stability and object motion consistency of the multiple candidate trajectories are verified by a physical simulator to determine the teacher trajectory and target contact state. The training unit is used to construct residual labels based on the difference between the target trajectory of the teacher trajectory and the initial trajectory and the difference between the target contact state and the initial contact state, and to train the stage adaptive residual model based on the residual labels to obtain the trained stage adaptive residual model. The generation unit is specifically used to analyze the interaction information between the human hand and the object frame by frame based on the human hand-object interaction data, extract the target contact point from the actual contact area or high contact probability area between the human hand and the object, extract the contact normal from the object surface geometry at the target contact point, extract the local object geometry from the object mesh or point cloud data, calculate the equivalent opening and closing width based on the human hand contact span or the local thickness of the object, and determine the stage marker according to the hand-object distance, contact probability and object speed change. When there are multiple contact areas, the multiple contact areas are filtered or merged according to the object movement direction, contact probability and local thickness, and the center of the contact area is taken as the target contact point. The operation intent includes the target contact point, contact normal, local object geometry, equivalent opening and closing width and stage marker.

9. A trajectory redirection device, characterized in that, include: Central processing unit and memory; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

11. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 7.

Images