A method and system for on-site transfer of gesture and pose alternating teaching sequences
By using offline teaching storage of interleaved reference sequences and on-site point cloud registration, selectively transferring arm pose steps, the problem of transferring gesture-arm pose interleaved teaching sequences when the tool pose changes is solved. This achieves complete preservation of gesture steps and synchronization of action timing, improving the success rate and safety of robot grasping.
Patent Information
- Application Number
- CN202611267498.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-20
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies cannot effectively solve the migration problem of gesture-arm pose interleaved teaching sequences when the tool pose changes, resulting in the destruction of gesture semantics or asynchronous actions, and have high deployment costs and insufficient security.
A method for on-site transfer of gesture pose interleaved teaching sequences is designed. The interleaved reference sequence is stored offline, the tool pose is obtained by on-site point cloud registration, the arm pose steps are selectively transferred, the original gesture step instructions are preserved, and an exception handling mechanism is set up to ensure the integrity of the action sequence.
It achieves complete preservation of gesture steps, adaptation of arm pose steps, and unchanged action timing in scenarios with tool pose offset, thereby improving the success rate of grasping, reducing deployment costs, and enhancing safety and flexible operation capabilities.
Smart Images

Figure CN123103591A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot teaching and skill transfer technology, and in particular to a method and system for on-site transfer of gesture and pose alternating teaching sequences for dexterous hand-robotic arm collaborative operations. Background Technology
[0002] Robot teaching is a core method for the deployment of collaborative robots, and it includes various implementation forms such as online teaching, drag-and-drop teaching, and gesture teaching. When the pose of the work object changes, the already taught action sequence needs to be transferred and adapted, also known as skill transfer or trajectory relocalization, which is a key technology for flexible robot operations. For a robotic arm system equipped with a dexterous hand, the teaching sequence consists of two completely different types of data: one is gesture data, namely the joint angle vectors of each finger of the dexterous hand, representing grasping gestures such as fingers opening, half-clenching, and fully closing, which belongs to the internal joint state of the robot body and is independent of the spatial position of the external workpiece; the other is arm pose data, namely the homogeneous pose of the robotic arm end effector in the world / base coordinate system, representing the position and orientation of the end effector in three-dimensional space. This type of data needs to be synchronously corrected according to the workpiece placement offset. When the task involves alternating between gestures and arm movements, an interleaved teaching sequence of gesture steps and arm pose steps is formed.
[0003] Taking the task of grasping a wine dispenser by a catering robot as an example, the complete teaching process is as follows: 1) Dexterous hand opens its hand gesture; 2) Robotic arm moves to approach the wine dispenser; 3) Dexterous hand closes its hand gesture; 4) Robotic arm lifts the wine dispenser off the table. These four steps alternate between hand gesture commands and arm movement commands, forming an interleaved teaching sequence. After the wine dispenser is shifted or rotated on-site, all approach and lifting spatial waypoints of the robotic arm need to shift synchronously with the wine dispenser; however, the finger joint angles of the dexterous hand opening and closing do not need any modification. If the entire trajectory is directly subjected to an overall affine transformation, a spatial rotation and translation matrix will be applied to the dexterous hand joint vectors, causing the finger opening and closing movements to become disordered; if only a single grasping point is migrated, and the approach and lifting intermediate waypoints maintain their original coordinates, the robotic arm will reach the old spatial position, but the wine dispenser has already shifted, and the gripper cannot align with the tool, resulting in grasping failure.
[0004] With the large-scale deployment of collaborative robots and dexterous hands in scenarios such as catering services, intelligent manufacturing, and flexible assembly, robots completing complex interactive operations through human teaching has become the mainstream deployment method. For tasks such as grasping irregularly shaped tools and manipulating irregular implements, the robot's operation process is completed by alternating hand gestures and robotic arm pose movements, forming a teaching sequence that interweaves gesture steps and arm pose steps. For example, in the task of grasping a wine gun, it is necessary to alternately execute the dexterous hand's opening and closing gesture commands and the robotic arm's multi-path spatial point movements to complete the tool clamping and positioning. In actual working environments, the position and posture of tools and workpieces may shift slightly each time they are placed. It is necessary for the robot to transfer the interleaved teaching sequence obtained from offline teaching to reproduce the task under the new object pose on site, ensuring that the operation semantics and action timing remain intact.
[0005] Existing related patent technologies can be divided into four categories: deep learning-based humanoid robot skill transfer technology, 3D point cloud registration algorithm technology, teaching trajectory generation technology based on human gesture mapping, and online teaching technology combining gestures and voice. The technical implementation and limitations of existing patent solutions will be analyzed one by one below.
[0006] The first category involves no-code teaching and skill transfer for humanoid robots based on multimodal feature learning, represented by patent CN122518330A, "A No-Code Teaching and Skill Transfer Method and System for Humanoid Robots." The patent's technical solution involves: collecting multimodal teaching data during the teaching process; extracting and fusing the multimodal features of the teaching data to generate fused features; dividing the teaching process into operation sub-stages using a pre-trained model based on the fused features, and adding corresponding operation semantic labels to each sub-stage; and constructing a skill learning model framework to complete skill learning and cross-scene transfer. This solution belongs to a high-level skill transfer framework, focusing on task semantic segmentation and skill learning, dividing the complete task into multiple sub-operations, and relying on the model to achieve skill reuse. The limitations of this patent are: it addresses the overall skill transfer of humanoid robots, does not define a data storage structure for the alternating sequences of gesture steps and arm pose steps, and does not propose a selective transfer rule of "transforming and transferring arm pose steps while replaying gesture steps as is." This scheme cannot distinguish which items in the teaching sequence need spatial transformation and which items must keep their original values unchanged. Directly using this scheme to process interleaved teaching sequences will either completely transform all items, destroying the semantics of the gesture joints, or leave them completely unchanged, failing to adapt to object offsets. It cannot solve the technical problem targeted by this invention.
[0007] The second category consists of patents related to 3D point cloud matching and registration algorithms, represented by patent CN122530630A, "A 3D Point Cloud Matching Method Based on Dynamic Multi-Scale and Dual Attention." The patent's technical solution involves: constructing multi-scale local point cloud blocks with different spatial receptive fields based on the original local point cloud blocks; extracting features from the multi-scale local point cloud blocks using a weight-sharing encoding network; and achieving adaptive weighted fusion of multi-scale features through a scale-gating module to obtain a fused multi-scale feature map; completing point cloud matching based on a spatial and channel dual attention mechanism, and outputting a point cloud registration transformation matrix. This patent provides a high-precision deep learning point cloud registration algorithm that can be used as a tool for obtaining the pose of objects in the field. However, the protected object of this patent is the point cloud matching network itself; its patent documents do not involve the teaching sequence data structure, the selective transfer logic of interleaved sequences, or the gesture-arm pose interleaved playback control process. This patent can only output the object pose transformation matrix and does not solve the problem of how to process the interleaved teaching sequence after obtaining the transformation matrix; it is merely an external registration component that the field registration module M2 of this invention can select (see...). Figure 2 It cannot independently complete the on-site transfer of gesture-arm posture alternating teaching sequence.
[0008] The third category involves teaching trajectory generation based on human body posture and gesture mapping, represented by patent CN116749168A, "A Method for Obtaining Rehabilitation Trajectory Based on Body Posture and Gesture Teaching." This patent's technical solution involves using a visual sensor to collect human body posture and gesture data, accurately measuring the movements of the human hand and arm, and mapping the human coordinate system posture to the robot coordinate system to control the robot's movement using gestures and arm movements, thus generating the rehabilitation robot's motion trajectory. This patent achieves the mapping teaching from human movement to robot trajectory, belonging to the teaching generation stage. The limitations of this patent are: it is used for initial generation of teaching trajectories and does not address already taught and stored interleaved sequences; when the position of the tool or object changes, the patent lacks a complete process for saving the reference object's position, on-site registration, and relative homogeneous transformation migration, and cannot perform on-site adaptation migration of historically taught interleaved sequences. If the workpiece position changes, this patent solution requires the operator to re-teach the human body gestures, and historical teaching data cannot be reused.
[0009] The fourth category is robot online teaching technology combining gestures and voice, represented by patent CN106095109A, "Method for Online Robot Teaching Based on Gestures and Voice." This patent's technical solution includes S1 a coarse adjustment process based on gestures; S2 a fine adjustment process based on voice; and S3 combining gestures and voice for robot teaching. The operator completes online robot teaching by coarse adjustment using gestures and fine adjustment using voice, resulting in more natural human-computer interaction. This patent focuses on the human-computer interaction methods during the teaching process, solving the problem of "how to complete the teaching," rather than "how to transfer and reuse the teaching sequence" after teaching is complete. This patent lacks the complete transfer logic for storing reference object poses, on-site visual registration, and selectively distinguishing between gesture keys and arm pose keys. If the workpiece placement shifts, this patent cannot adapt to the old teaching sequence.
[0010] Existing technologies suffer from two significant inherent drawbacks. First, overall affine transfer of the trajectory cannot preserve the temporal semantics of gestures. If an overall affine transformation is performed on the entire interleaved teaching sequence, multiplying the gesture joint vectors and the end-effector pose by the object transformation matrix, the end-effector pose, being a homogeneous pose in 3D space, is suitable for spatial affine transformation. However, gesture data consists of dexterous hand finger joint angle vectors, which are internal joint spatial parameters of the robot and do not change with the external workpiece spatial position. Applying spatial rotation and translation transformations to the gesture joint vectors will output physically meaningless finger angles, directly disrupting the temporal semantics of the grasping gesture, causing grasping opening and closing actions to become disordered, and leading to collisions. Second, transferring only a single-point grasping pose will result in desynchronization of multi-step interleaved sequences.
[0011] Some existing solutions only modify the pose of the single point of the gripping contact point, while replaying all other arm waypoints and gesture commands in the teaching sequence as is. For multi-step interleaved sequences, the robotic arm approach, close approach, pre-grip, grip, and lift involve multiple arm pose waypoints. Only the gripping point is modified, while the other waypoints retain their original coordinates. When the workpiece shifts, the spatial position of the intermediate waypoints no longer matches the new position of the workpiece. The timing of the gesture opening and closing is misaligned with the spatial movement of the robotic arm, the action timing-space relationship is decoupled, the entire interleaved sequence of actions is out of sync, and the task reproduction fails.
[0012] Some solutions are also tightly bound to specific deep learning models or specific registration algorithms, with the transfer logic and perception algorithm deeply coupled. Once the depth camera or registration algorithm is changed, the entire transfer process needs to be modified on a large scale, resulting in poor replaceability and high deployment costs. Many existing solutions lack complete exception handling branches, and there is no safe rollback mechanism for working conditions such as registration failure and robotic arm overtravel, resulting in insufficient safety in industrial sites.
[0013] In summary, existing technologies urgently need a complete transfer scheme specifically for gesture-arm pose interleaved teaching sequences that can simultaneously preserve the interleaved key-value sequence structure, distinguish between gesture steps and arm pose steps, transfer only arm pose steps after object offset, completely retain the original instructions for gesture steps, interleave playback according to the original time sequence, and have comprehensive anomaly safety handling. Summary of the Invention
[0014] To address the shortcomings of existing robot teaching trajectory transfer technologies, such as the inability to distinguish between gesture steps and arm pose steps within an interleaved teaching sequence, the disruption of gesture semantics during overall transfer, and the asynchrony of sequence actions caused by single-point transfer, a method and system for on-site transfer of gesture pose interleaved teaching sequences is provided. After completing multi-step gesture-arm pose interleaved teaching of irregular tools, when the on-site tool object placement and pose change, only the arm pose steps can be selectively transferred, while the gesture steps retain the original instructions, completely preserving the original action sequence, safely and stably reproducing the entire interleaved teaching sequence, and completing the tool clamping and ready operation.
[0015] Specifically, according to a first aspect of the present invention, a method for on-site transfer of a gesture pose alternation teaching sequence is provided, comprising the following steps: S1. Teaching and storage steps: Offline teaching to obtain an ordered interleaved reference sequence. The interleaved reference sequence includes alternating gesture keys and arm pose keys; the key name with a gesture class prefix is the gesture key, and the value corresponding to the gesture key... For finger joint vectors; items with the arm pose class prefix in their key names are arm pose keys, and the values corresponding to the arm pose keys are... The homogeneous pose of the robotic arm's end effector is obtained; simultaneously, the homogeneous pose of the reference object in the robot's base coordinate system is saved. ; Furthermore, the gesture class prefix is configured as follows: The arm pose class prefix is configured as follows: Gesture keys and arm pose keys are identified and distinguished by string prefix matching. The interleaved reference sequence is a key-value array that strictly preserves the order of teaching. Each array element is either a gesture key or an arm pose key, and the two alternate.
[0016] S2. On-site perception and registration steps: Collect the point cloud of the Region of Interest (ROI) where the tool is located at the work site. Perform registration operations between the on-site point cloud and the pre-stored object template point cloud to obtain the homogeneous pose of the current object in the robot's base coordinate system. If point cloud registration fails, an error message will be output, the migration process will be terminated directly, and the process will revert to the original teaching playback mode. In this step, the point cloud registration algorithm serves as an external, replaceable interface module, supporting two switchable working modes: visual registration mode and direct playback mode. Visual registration mode executes the entire process of point cloud acquisition, cropping, and registration; direct playback mode directly reuses the homogeneous pose of the reference object. The point cloud acquisition and registration process is not executed.
[0017] During the registration process, registration validity criteria are set: a threshold for the number of valid matching points and an iteration convergence threshold. If the validity criteria are not met, the point cloud registration is deemed to have failed. If point cloud registration fails, an error message is output, the migration process is directly terminated, and the process reverts to the original teaching playback mode. The error signal generated by the registration failure is sent to the error alarm module.
[0018] S3, Item-by-item selective migration step: Traverse each key-value item in the sequence according to the original order of the interleaved reference sequence; If the currently traversed item is a gesture key, then output the current gesture. It directly reuses the finger joint vectors of the original reference gesture without performing coordinate transformation; If the current traversal item is an arm pose key, then perform a homogeneous transformation operation: First, calculate the tool pose of the reference arm relative to the coordinate system of the reference object:
[0019] In the formula, The reference arm tool pose is the end-effector homogeneous pose corresponding to the arm pose key saved during the teaching phase. After recalculating the new arm end-effector pose in the base coordinate system, the following steps are performed:
[0020] Will This is converted into pose motion commands that the robot controller can recognize, such as 6D pose vectors, quaternions, or position format commands. All logic for this step is implemented by the selective migration module.
[0021] S4. Interleaved Playback Execution Steps: Execute sequentially according to the selected migration sequence; for gesture keys, call the dexterous hand joint interface to send finger joint vectors; for arm pose keys, call the robotic arm motion interface to send the migrated end-effector pose command, and wait for the robotic arm to move into position; perform real-time overtravel verification during the movement; if any step exceeds the travel limit or fails, immediately stop execution, control the robot to return to a safe position, and output a fault message; furthermore, the fault message includes a voice alarm report. This step corresponds to the interleaved playback module, and the motion command is sent to the robot hardware via OUT1 robot execution output.
[0022] S5. Readiness determination step: After all sequence items have been executed without any abnormalities, the robot enters the tool holding ready state, waiting for subsequent upper-level business task calls.
[0023] Furthermore, in step S2, the point cloud registration algorithm serves as an external replaceable interface module, supporting both visual registration mode and direct playback mode. In playback mode, the homogeneous pose of the reference object is directly reused, and the point cloud acquisition and registration process is not executed.
[0024] Furthermore, the overtravel verification in step S4 includes verification of the robotic arm joint angle exceeding the limit, verification of the dexterous hand joint angle exceeding the limit, and verification of the workspace boundary of the robotic arm end effector; failure of any one of these verifications is considered a motion failure.
[0025] Furthermore, in the event of registration failure or motion failure, fault prompts will be output via voice broadcast.
[0026] According to a second aspect of the present invention, a field transfer system for a gesture pose interleaved teaching sequence is provided, which is used to implement the above-mentioned gesture pose interleaved teaching sequence field transfer method. The system includes: a teaching storage module M1, a field registration module M2, a selective transfer module M3, an interleaved playback module M4, and an abnormal alarm module M5; the external input is IN1 depth camera input; the external output is divided into OUT1 robot execution output and OUT2 alarm output.
[0027] The teaching storage module M1 is used to acquire and persistently store ordered interleaved reference sequences during offline teaching. Simultaneously, it stores the homogeneous pose of the reference object in the robot's base coordinate system. The interleaved reference sequence contains alternating gesture keys and arm pose keys; the gesture keys correspond to finger joint vectors, and the arm pose keys correspond to homogeneous poses of the robotic arm's end effector. The teach storage module maintains key name label rules internally, distinguishing between gesture keys and arm pose keys by key name string prefixes. M1 outputs the interleaved reference sequence and reference object poses to the selective transfer module M3.
[0028] The on-site registration module M2 receives point cloud data output from the depth camera IN1, performs point cloud cropping of the ROI (Region of Interest) in the on-site work area, calls an external registration algorithm interface to register the on-site point cloud with the object template point cloud, and solves for the homogeneous pose of the current object. When registration fails, an abnormal signal is output. The on-site registration module has a configuration mode switching unit, supporting visual registration mode and direct playback mode; in playback mode, it directly reads the homogeneous pose output of the reference object inside the teaching storage module, skipping the point cloud acquisition and registration calculation process. The registration algorithm interface is a standardized external interface, which can replace any registration implementation such as ICP, NDT, and deep learning point cloud registration without changing the internal migration logic of the system. M2 outputs the current homogeneous pose of the object to the selective migration module M3, and the abnormal signal is output to the abnormal alarm module M5.
[0029] The selective migration module M3 is signal-connected to the teaching storage module M1 and the on-site registration module M2. The selective migration module internally includes a key name matching unit to identify key name prefixes and distinguish between gesture keys and arm pose keys. It is used to traverse the interleaved reference sequence in its original order. For gesture keys, it directly copies the original finger joint vectors without transformation. For arm pose keys, it performs homogeneous matrix operations to obtain the new arm end-effector pose command after migration. The completed command sequence after migration is output and sent to the interleaved playback module M4. All logic for traversing sequence keys, original gesture branches, and relative object transformation migration branches is encapsulated within this module.
[0030] The interleaved playback module M4 is connected to the output of the selective migration module M3; it is used to execute the instruction sequence after migration in sequence; the gesture key issues dexterous hand joint commands; the arm pose key issues robotic arm movement commands and waits for feedback on the movement reaching its destination; it performs overtravel verification in real time during execution; in case of an anomaly, it issues a return to safety position command and outputs an anomaly signal to the anomaly alarm module M5; when the sequence is complete and there are no anomalies, it outputs a tool-holding ready status signal. This module corresponds to the "execution" step. The robot movement commands are output to the robotic arm and dexterous hand hardware via the OUT1 robot execution output.
[0031] The abnormal alarm module M5 receives abnormal signals output by the field registration module M2 and the interleaving playback module M4, calls the voice driver unit to execute voice alarm prompts, outputs fault text logs, and outputs alarm signals to the outside via OUT2 alarm output.
[0032] Furthermore, the system hardware consists of a seven-DOF collaborative robotic arm, a five-finger dexterous hand, an RGB-D depth camera, a robot controller, and a voice playback unit; the software is deployed inside the robot controller, and the modules use inter-process communication to transmit data.
[0033] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements all the steps of the aforementioned on-site transfer method for the gesture pose interleaving teaching sequence. The storage medium may be a non-volatile memory, such as a USB flash drive, a solid-state drive, a server disk, etc.
[0034] Compared with the prior art, the present invention has at least the following beneficial effects: A differentiated selective transfer mechanism is designed for gesture-arm pose interleaved teaching sequences. The gesture step fully preserves the original finger joint vectors and does not participate in spatial affine transformation; the arm pose step completes all waypoint transfers based on the relative homogeneous transformation of the object coordinate system. The mechanism fully preserves the action timing and gesture grasping semantics of the original interleaved sequence, fundamentally solving the technical pain points of overall trajectory transfer destroying gesture commands and single-point transfer causing misalignment and asynchrony of multi-step sequence actions. This significantly improves the reproducibility success rate of irregular tool grasping tasks in workpiece offset scenarios. The entire set of branch judgment and selective processing logic has clear boundaries, making it easy to implement in engineering code.
[0035] As a standardized external interface, the point cloud registration logic of this invention focuses on the interleaved sequence storage structure, selective migration transformation rules, and interleaved playback control flow. The registration algorithm can be freely replaced with different schemes such as ICP, NDT, and deep learning registration according to hardware conditions. The perception algorithm and migration logic are decoupled, allowing for flexible engineering deployment, adaptability to various depth camera hardware, and strong portability. Registration is separated into an independent on-site registration module M2, decoupled from the selective migration module M3, intuitively reflecting the interface-based design concept.
[0036] Automatic revert to the original playback mode upon registration failure; automatic return to a safe position in case of motion overtravel or motion malfunction, coupled with dual anomaly output including voice alarms and log recording; ensuring the safety of the robot, surrounding equipment, and operators, meeting on-site operational safety requirements. A direct playback mode is also included, allowing operation without visual sensing hardware, adapting to low-cost and simple deployment scenarios.
[0037] Gestures and arm poses are distinguished by key name prefixes, resulting in a clear structure that facilitates parsing, storage, and serialization, as well as convenient importing and exporting of teaching data. Teaching data is independently managed by the teaching storage module M1.
[0038] When the tool placement experiences slight translational or rotational shifts, the entire multi-step staggered teaching sequence can be directly replicated on-site without requiring manual re-teaching by the operator. This reduces on-site debugging workload, shortens robot project deployment cycles, and enhances the robot's flexible operation capabilities. This invention can be widely applied to dexterous hand-robotic arm collaborative operation scenarios such as wine gun gripping, irregularly shaped tool handling, and flexible component assembly. Attached Figure Description
[0039] Figure 1 This is a schematic diagram illustrating the overall process of the on-site transfer method for the gesture pose alternation teaching sequence according to a specific embodiment of the present invention.
[0040] Figure 2This is a block diagram illustrating the module structure of a field transfer system for a gesture pose alternation teaching sequence according to a specific embodiment of the present invention. Detailed Implementation
[0041] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0042] This embodiment takes wine gun grasping as a typical application scenario. The robot is equipped with a seven-degree-of-freedom collaborative robotic arm and a five-finger dexterous hand to complete the on-site transfer and reproduction of the wine gun grasping teaching sequence. Of course, the present invention is not limited to wine gun grasping, but can also be applied to dexterous hand-robotic arm alternating teaching operation scenarios for various irregular workpieces and tools.
[0043] This invention addresses the collaborative operation scenario of a dexterous hand and a robotic arm, proposing a selective transfer architecture for gesture-arm pose alternating teaching sequences; the overall execution logic of this invention is as follows: Figure 1 The complete representation of the method flowchart shows the interaction relationship between the system hardware and software modules. Figure 2 The system module diagram represents the process. It fully saves the interleaved reference sequence and the pose of the reference object; it obtains the current pose of the object through on-site visual registration; when traversing the sequence, the gesture steps are replayed as is without coordinate transformation, only the arm pose steps undergo a relative homogeneous transformation based on the object coordinate system, and then the entire sequence is replayed interleaved according to the original time sequence. This invention can safely reproduce the complete interleaved operation process without re-teaching when the tool experiences a small pose shift, and can be widely used in technical scenarios such as food service robots grasping wine dispensers, flexible assembly of irregularly shaped parts, and operation of special equipment.
[0044] Figure 1 The loop branch execution logic of the method of the present invention is shown, which includes loop back edge (returning to the traversal sequence key before the sequence ends), distinguishing two parallel processing branches for gesture keys and arm pose keys, and entering the execution step after the two branches merge; Figure 2 It shows the data flow between modules, the one-way / two-way signal connection between modules, and clarifies the external input and output interfaces.
[0045] Figure 1 (In the method flowchart): Load the reference sequence and reference object pose; corresponding to method step S1, teach and store step output data; On-site sensing and registration; corresponding method steps S2: On-site sensing and registration steps; Traverse the sequence keys; loop through each key-value item in the interleaved reference sequence to complete key type identification; Gesture key branch: Original gesture; Recognized as a gesture key, directly reuses the original finger joint vector without coordinate transformation; Arm pose key branch: relative object transformation and transfer; if identified as an arm pose key, perform relative homogeneous transformation of the object and calculate the pose of the arm end effector after transfer. Execution; issue and execute the processed gesture commands and arm posture commands, and simultaneously perform overtravel verification; Sequence end check; use a diamond-shaped check node, if not finished, return to marker 3 and continue traversing the next sequence key; if finished, move to the next step. The robot is now ready to hold its weapon; once the entire sequence has been executed, the robot enters the tool-holding ready state.
[0046] Figure 2 middle: M1 - Teaching Storage Module; Completes persistent storage of teaching interleaved sequences and reference object poses; M2 - Field Registration Module: Receives depth camera input, performs point cloud processing, registration, and solves for the current object's pose. M3 Selective Migration Module; Implements selective processing of sequence traversal and gesture / arm pose branches; M4-Interleaved Playback Module; Implements command issuance, motion execution, arrival waiting, and overtravel verification; M5 - Anomaly Alarm Module: Receives various anomaly signals, executes voice alarms, and outputs logs; IN1 - Depth camera input, providing the raw point cloud data source for the field registration module M2; OUT1 - Robot execution output, issuing motion and gesture commands to the robotic arm and dexterous hand; OUT2 - Alarm output, voice broadcast, and fault log output interface.
[0047] The present invention will now be described in further detail with reference to specific embodiments.
[0048] Example 1: On-site transfer method for gesture and pose alternating teaching sequence The method flow in this embodiment is as follows: Figure 1 As shown, it includes steps S1-S5 and also includes multiple exception handling branches.
[0049] S1. Teaching and storage steps: The operator performs offline teaching by dragging the robot, executing the following actions sequentially: the dexterous hand opens; the robotic arm moves towards the wine dispenser; the dexterous hand closes to grip; the robotic arm lifts the wine dispenser. The system records an ordered, staggered reference sequence. The gesture keys and arm pose keys are arranged alternately within this sequence. Figure 1 The "Load Reference Sequence and Reference Object Pose" section persistently stores this set of teaching data. Figure 2 Teach storage module M1.
[0050] Key name with The first item is the gesture keys, storing the vectors of each finger joint in a dexterous hand. Example: For the corresponding open gesture, store the angle vectors of each joint of the 5 fingers; For the corresponding closed grasping gesture, store the finger joint vectors in the closed state.
[0051] Key name with The first item is the arm pose key, storing a 4×4 homogeneous pose matrix for the robotic arm's end effector, corresponding to the robotic arm's waypoints. Example: Approaching the location of the alcohol gun; Raise the wine gun navigation point.
[0052] Example of the interleaved reference sequence array obtained in this embodiment:
[0053] Simultaneously, the homogeneous pose of the reference wine gun object in the robot's base coordinate system is preserved. A 4×4 homogeneous matrix and an interleaved reference sequence are bound and stored in the teaching storage module and persistently saved to the local disk.
[0054] S2. On-site sensing and registration steps: The placement and orientation of the wine gun at the work site shifted. An RGB-D depth camera captured a complete point cloud of the work surface; the region of interest (ROI) was cropped from the original point cloud, and irrelevant point clouds from the desktop background and surrounding objects were filtered out according to a preset bounding box, retaining only the point cloud of the wine gun tool itself. This step corresponds to... Figure 1 "On-site perception and registration," the hardware data input is... Figure 2 The IN1 depth camera input is processed by the M2 field registration module.
[0055] The external standardized registration interface is called to perform registration calculations between the point cloud cropped on-site and the pre-saved point cloud of the wine gun template, and the homogeneous pose of the current wine gun object in the base coordinate system is output. .
[0056] The on-site registration module has two switchable modes: visual registration mode and direct playback mode.
[0057] Visual registration mode: Performs the complete process of point cloud acquisition - ROI cropping - point cloud registration, and outputs... ; Direct playback mode: Skips visual perception and registration calculations, and directly reads the data. This serves as the current object pose, used in situations where the workpiece has not shifted, does not require migration, and the teaching task can be directly replayed.
[0058] Set the registration validity criteria: if the number of interior points after registration is greater than a preset threshold and the iterative convergence residual is less than the threshold, the registration is considered successful if both conditions are met simultaneously; otherwise, the registration is considered to have failed.
[0059] When registration fails: the system outputs an abnormal signal, and the M5 abnormal alarm module announces "Object registration failed, switch to original teaching playback mode"; the selective migration process is aborted, the S3 step is not executed, and the original sequence is directly played back as is.
[0060] S3, Item-by-Item Selective Migration Steps: The system reads the complete interleaved reference sequence and traverses each key-value item in the array strictly according to the original teaching order. Figure 1 The logic for "traversing sequence key" nodes is as follows: Figure 2 Selective migration module M3 is executed.
[0061] When iterating through gesture keys, for example ,Enter Figure 1 The left branch, "original gesture," is directly assigned a value. The original finger joint vectors are completely copied without any spatial coordinate transformation. Gestures are the internal finger joint states of a dexterous hand and are not affected by changes in the spatial position of external tools. Preserving them as is ensures that the grasping semantics of opening and closing remain completely unchanged.
[0062] When iterating to the arm pose key, for example ,Enter Figure 1 The right-hand branch, "Relative Object Transformation and Transfer," performs homogeneous matrix operations: .
[0063] The homogeneous matrix of the arm end-effector reference pose stored for the teaching phase; The inverse matrix represents the homogeneous pose of the reference object; the relative pose of the arm's end-effector with respect to the wine gun's coordinate system is calculated. This relative pose represents the spatial relationship between the robotic arm's end effector and the wine gun tool, and this relative relationship remains unchanged when the tool is translated or rotated.
[0064] Then, by performing a matrix right multiplication on the current object pose obtained through on-site registration, the pose of the new arm's end effector in the base coordinate system after migration is obtained:
[0065] homogeneous matrix The pose commands are converted into position vectors and quaternions that the robot controller can recognize and then sent to the motion control unit.
[0066] The logical-physical meaning of this transformation is as follows: the relative spatial posture of each waypoint of the robotic arm with respect to the wine gun remains unchanged; the wine gun as a whole undergoes translation, rotation and offset, and all waypoints of the robotic arm offset synchronously with the wine gun, while the opening and closing gestures of the dexterous hand remain completely unchanged, and the temporal and relative spatial coupling relationship between the gestures and the waypoints is completely preserved.
[0067] S4. Steps for executing interlaced playback: The gesture processing branch and the arm pose transfer branch merge and enter... Figure 1 The "execution" step is... Figure 2 The interleaved playback module M4 processes the data. It executes each instruction sequentially according to the order in which the migration is completed.
[0068] When a gesture key is encountered, the dexterous hand driver interface is invoked, and the raw finger joint vectors are sent down. The dexterous hand completes the opening or closing action and waits for feedback on the dexterous hand joint positioning.
[0069] Upon encountering the arm pose key, the robot sends a migration-advanced end-effector pose command to the robot arm controller. The robot arm then executes the movement and waits for the robot arm to complete its movement. The movement command is sent to the hardware via the OUT1 robot execution output.
[0070] Overtravel checks are performed in real time during each step of the movement, with the three checks executed in parallel: 1) Are the angles of each joint of the robotic arm within the allowable range of motion? 2) Are the angles of each joint of the fingers in a dexterous hand within a safe range? 3) Whether the end effector of the robotic arm falls within the boundary of the robot's workspace.
[0071] If any verification fails or the movement timeout occurs without receiving a positioning signal, the movement is deemed to have failed. The system immediately stops the entire sequence execution and controls the robotic arm and dexterous hand to return to the preset safe standby position. The abnormal alarm module M5 announces "Motion overtravel, execution interrupted" in voice, and records the fault log. The alarm is output by OUT2.
[0072] S5. Readiness determination step: Enter after execution Figure 1 The diamond-shaped "Sequence End" checkpoint is used. If the sequence has not ended, the checkpoint outputs "No," and the process jumps back to the "Traverse Sequence Key" to loop through and read the next sequence item. If all sequence items have been executed, the checkpoint outputs "Yes," and the process moves to "Ready to Hold the Gun." The robot's dexterous hand reliably holds the wine gun, and the robot enters the "Ready to Hold the Gun" state, outputting a ready signal to the upper-level business system, awaiting subsequent upper-level business task instructions such as pouring and transferring wine.
[0073] Example 2: On-site transfer system for gesture and pose alternating teaching sequence This embodiment discloses a field transfer system for gesture pose alternating teaching sequences, used to implement the method described in Embodiment 1. The system block diagram is as follows: Figure 2 As shown. The system hardware consists of a service robot equipped with a seven-DOF collaborative arm, a five-finger dexterous hand, an RGB-D depth camera, and a voice playback unit; the software consists of control software deployed within the robot controller. The system modules include a teaching storage module M1, a field registration module M2, a selective migration module M3, an interleaved playback module M4, and an anomaly alarm module M5.
[0074] Teaching storage module M1: Persistently stores teaching data, storing interleaved reference sequences and homogeneous poses of reference objects; internally maintains key name label rules, distinguishing gesture keys from arm pose keys through string prefixes; supports serialized export and import of sequence data. Outputs teaching sequences and reference object poses to selective transfer module M3.
[0075] The on-site registration module M2 receives point cloud data output from the depth camera IN1 and performs ROI clipping; it calls the external point cloud registration algorithm interface to output the homogeneous pose of the current object; it has a built-in mode switching unit that supports visual registration mode / direct playback mode; and it outputs an exception signal to M5 if registration fails. The registration algorithm interface is a standardized interface that can replace ICP, NDT, and deep learning registration without changing the internal transfer logic of this system. It outputs the current object pose to the selective transfer module M3.
[0076] Selective migration module M3: Receives the interleaved reference sequence output by the teach storage module and the current object pose output by the field registration module; its internal key name matching unit identifies key prefixes to distinguish between gesture keys and arm pose keys; fully implements... Figure 1 The entire logic of traversing sequence keys, original gestures, and relative object transformation migration is processed; gesture keys directly copy the original joint vectors; arm pose keys perform homogeneous matrix transformation operations, and output the complete instruction sequence after migration to the interleaved playback module M4.
[0077] Interleaved playback module M4: Receives the sequence instructions output by the selective migration module; calls the dexterous hand joint interface and the robotic arm motion interface (OUT1 robot execution output); executes motion positioning and waits for completion, performing real-time overtravel verification; in case of an anomaly, it sends back a safety bit instruction and outputs an anomaly signal to M5; upon completion of all executions, it outputs a tool-hold-ready signal. Figure 1 The flowchart shows the "Execution" steps.
[0078] The abnormal alarm module M5 receives abnormal signals from the field registration module and the interleaving playback module, calls the voice driver unit, outputs the voice alarm OUT2, and records the text fault log.
[0079] The various modules exchange data through inter-process communication within the robot.
[0080] Example 3: Example of a computer-readable storage medium This embodiment discloses a computer-readable storage medium that stores a computer program. When the program is executed by a processor, it implements all steps of the on-site transfer method for the gesture pose interleaving teaching sequence in Embodiment 1. The storage medium can be non-volatile storage hardware such as a USB flash drive, solid-state drive, or server memory.
Claims
1. A method for on-site transfer of gesture and pose alternating teaching sequences, characterized in that, Includes the following steps: S1, teaching storage step: offline teaching obtains an ordered interleaved reference sequence ; the interleaved reference sequence comprises alternately arranged gesture keys and arm pose keys; The item with the key name carrying the gesture class prefix is a gesture key, and the value corresponding to the gesture key is the finger joint vector; the item with the key name carrying the arm pose class prefix is an arm pose key, and the value corresponding to the arm pose key is the homogeneous pose of the end of the robot arm; the reference object homogeneous pose of the reference object in the robot base coordinate system is also saved S2, on-site sensing and registration step: collect the point cloud of the region of interest ROI where the tool is located in the work site, perform registration operation on the on-site point cloud and the pre-stored object template point cloud, and solve the current object homogeneous pose of the current object in the robot base coordinate system If the point cloud registration fails, output an abnormal prompt, directly abort the migration process, and fall back to the original teaching playback mode. S3, Item-by-item selective migration step: Traverse each key-value item in the sequence according to the original order of the interleaved reference sequence; If the current traversal item is a gesture key, output the current gesture , directly reuse the finger joint vector of the original reference gesture without coordinate transformation; If the current traversal item is an arm pose key, then perform a homogeneous transformation operation: First, the reference arm tool pose is calculated relative to the object coordinate system of the reference object: , In the formula, is the reference arm tool pose, i.e. the end-effector homogeneous pose corresponding to the arm pose key saved in the teaching phase; Recalculate the new arm end pose in the base coordinate frame after migration: , Converting into a pose motion instruction recognizable by the robot controller; S4. Interleaved playback execution steps: Execute sequentially according to the sequence order after selective migration; for gesture keys, call the dexterous hand joint interface to send finger joint vectors; for arm pose keys, call the robotic arm motion interface to send the migrated end-effector pose command, and wait for the robotic arm to move into position; perform real-time overtravel verification during the movement; if any step exceeds the travel limit or the movement fails, stop execution immediately, control the robot to return to the safe position, and output a fault prompt. S5. Readiness determination step: After all sequence items have been executed without any abnormalities, the robot enters the tool holding ready state, waiting for subsequent upper-level business task calls.
2. The on-site transfer method for gesture pose alternating teaching sequences according to claim 1, characterized in that, The gesture class prefix is configured as The arm pose class prefix is configured as ; Gesture keys and arm pose keys are distinguished by key name prefixes.
3. The method of live migration of gesture pose interleaved teaching sequence of claim 1 or 2, wherein, In step S2, the point cloud registration algorithm serves as an external replaceable interface module, supporting both visual registration mode and direct playback mode.
4. The on-site transfer method for the gesture pose alternating teaching sequence according to claim 1, characterized in that, Step S4 includes overtravel verification, which includes verification of the robotic arm joint angle exceeding the limit, verification of the dexterous hand joint angle exceeding the limit, and verification of the workspace boundary of the robotic arm end effector. Failure of any one of these verifications is considered a motion failure.
5. The on-site transfer method for gesture pose alternating teaching sequences according to any one of claims 1-4, characterized in that, When registration or motion fails, a fault message is output via voice broadcast.
6. A field transfer system for a gesture and pose alternating teaching sequence, characterized in that, The on-site transfer method for implementing the gesture pose interleaved teaching sequence as described in any one of claims 1-5 includes a teaching storage module, an on-site registration module, a selective transfer module, and an interleaved playback module; The teaching storage module is used to acquire and store ordered interleaved reference sequences during offline teaching. Simultaneously, it stores the homogeneous pose of the reference object in the robot's base coordinate system. ; The interleaved reference sequence contains alternating gesture keys and arm pose keys; the gesture keys correspond to finger joint vectors, and the arm pose keys correspond to homogeneous poses of the robotic arm's end effector. The on-site registration module is used to acquire the point cloud of the ROI in the on-site work area, register the on-site point cloud with the point cloud of the object template, and solve for the homogeneous pose of the current object. Output an abnormal signal when registration fails; The selective migration module is signal-connected to the teaching storage module and the on-site registration module; it is used to traverse the interleaved reference sequence in the original order; and to directly copy the original finger joint vectors for the gesture keys. The new end-effector pose command after migration is obtained by solving the arm pose key through homogeneous transformation. The interleaved playback module is connected to the output of the selective migration module; it is used to execute the sequence after migration in order; the gesture key issues dexterous hand joint commands; the arm pose key issues robotic arm movement commands; overtravel verification is performed during execution; and when an abnormality occurs, the robot is controlled to return to the safe position and a prompt is output. Once the sequence has been fully executed, the output tool will be ready.
7. The on-site transfer system for gesture pose alternating teaching sequences according to claim 6, characterized in that, It also includes an anomaly alarm module; the anomaly alarm module receives anomaly signals output by the field registration module and the interleaving playback module, and performs voice alarm prompts.
8. The on-site transfer system for gesture pose alternating teaching sequences according to claim 6, characterized in that, The on-site registration module is configured to switch working modes, including visual registration mode and direct playback mode; in playback mode, it directly reads the homogeneous pose output of the reference object inside the teaching storage module, skipping the point cloud acquisition and registration calculation.
9. The on-site transfer system for gesture pose alternating teaching sequences according to claim 6, characterized in that, The selective migration module is internally configured with a key name matching unit that distinguishes between gesture keys and arm pose keys by recognizing key name prefixes; gesture key prefixes Arm pose key prefix .
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements all the steps of the on-site transfer method for the gesture pose interleaving teaching sequence as described in any one of claims 1-5.
Citation Information
Patent Citations
Online robot teaching method based on gestures and voice
CN106095109A
Code-free teaching and skill migration method and system for humanoid robot
CN122518330A
Three-dimensional point cloud matching method based on dynamic multiple scales and double attention
CN122530630A
