Teleoperation method, system and equipment based on smart type library and storage medium

Abstract

The invention discloses a smart type library-based teleoperation method, system and device, and a storage medium. The method comprises the following steps of: constructing a smart operation type library; analyzing the task instruction through a preset multi-modal large language model MLLM to obtain a corresponding operation step, and matching a dexterous operation type corresponding to the current step from a dexterous operation type library; intuitive accurate control between the hand and the manipulator is realized through an interpolation mapping strategy. The interpolation mapping strategy specifically comprises the following steps: capturing a fingertip position of a human hand in real time; calculating the normalized projection proportion of the current posture of the human hand relative to the stretching state and the contraction state; and according to the projection proportion, linear interpolation is carried out on the extension and contraction joint angles of the manipulator, and a manipulator joint angle instruction is generated. According to the method, the adaptability and stability of teleoperation are provided by establishing a systematized smart type library and combining a type automatic retrieval module assisted by a multi-modal large language model and an interpolation mapping control strategy.

Description

Technical Field

[0001] The present invention belongs to the technical field of teleoperation systems, and in particular relates to a teleoperation method, system, device and storage medium based on a smart type library. Background Art

[0002] With the rapid development of artificial intelligence and robotics, dexterous manipulators with high degrees of freedom are increasingly being used in various precision manipulation tasks, becoming an important means of achieving complex human-robot collaboration and remote operations. In practical applications, teleoperation, as a key technology for human-robot interaction and data collection, is widely used in scenarios such as remote control, robot training, and demonstration data collection.

[0003] Most current mainstream dexterous teleoperation methods are based on a "posture redirection" strategy, which involves capturing the posture of the human hand and mapping it to the dexterous manipulator, while maintaining the consistency of motion between the two. However, existing methods suffer from two key technical bottlenecks:

[0004] First, due to the limitations of the human hand's anatomy and range of motion, traditional redirection methods artificially restrict the manipulator's mobility, preventing it from fully leveraging its superior flexibility and structural advantages over the human hand. This constraint makes some complex operations (such as wide-open grasping and asymmetric gripping) difficult to perform, severely impacting the adaptability and success rate of teleoperation systems.

[0005] Secondly, due to the structural differences between human and robotic hands, the redirection process often results in unstable grasping, self-collision, or insufficient contact surface during spatial mapping, which impacts grasping stability and task completion. Furthermore, the high-dimensional parameter space of hand posture complicates the mapping solution and increases the difficulty of control.

[0006] In summary, traditional teleoperation systems have significant deficiencies in operational dexterity, versatility, and stability, making them difficult to meet the operational requirements of manipulators in high-precision, multi-tasking scenarios. Therefore, a new teleoperation system is urgently needed that breaks through the constraints of traditional human-hand mapping. While retaining the advantages of intuitive human-machine interaction, it can fully unleash the structural potential of the dexterous manipulator itself, achieving more efficient and robust remote operation and data acquisition capabilities. Summary of the Invention

[0007] The present invention aims to overcome the problems of unstable control caused by reliance on human hand gesture redirection and the failure to fully utilize the advantages of the manipulator structure in existing teleoperation technologies. A teleoperation method, system, device and storage medium based on a smart type library are provided. By establishing a systematic smart type library, combined with a type automatic retrieval module assisted by a multimodal large language model (MLLM) and an interpolation mapping control strategy, high adaptability and control stability of the teleoperation system are achieved.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a remote operation method based on a smart type library, comprising the following steps:

[0010] Constructing a dexterous operation type library, the dexterous operation type library is used to store multiple predefined dexterous operation types; each dexterous operation type includes: the joint angle configuration of the manipulator in the extended state and the retracted state; the category and geometric feature description of the applicable object; the contact position and direction information between the manipulator and the object; and the semantic label of the operation intention;

[0011] Receive task instructions and environmental perception data, parse the task instructions through the preset multimodal large language model MLLM to obtain the corresponding operation steps, and match the dexterous operation type corresponding to the current step from the dexterous operation type library;

[0012] Intuitive and precise control between the human hand and the robotic arm is achieved through an interpolation mapping strategy; the interpolation mapping strategy specifically includes: capturing the position of the human fingertips in real time; calculating the normalized projection ratio of the human hand's current posture relative to the extended state and the contracted state; and linearly interpolating the extended and contracted joint angles of the robotic arm based on the projection ratio to generate the robotic arm joint angle instructions.

[0013] As a preferred technical solution, the dexterous operation types adopt a hierarchical classification structure, including:

[0014] One-hand operation type is subdivided into grasping type and non-grasping type; the grasping type is further divided into general grasping type and manipulator-specific grasping type;

[0015] The two-handed collaboration type is subdivided into symmetrical collaboration type and asymmetrical collaboration type.

[0016] As a preferred technical solution, the smart operation type corresponding to the current step is matched from the smart operation type library, specifically:

[0017] Convert all operation types and their attributes in the type library into natural language prompts and input them into MLLM;

[0018] Based on the task instructions input by voice or text, MLLM outputs the task decomposition steps and the adaptive dexterous operation type for each step.

[0019] As a preferred technical solution, the generating of the manipulator joint angle instruction is specifically as follows:

[0020]

[0021] θ current=pratio·(θ contract -θ stretch )+θ stretch ;

[0022] Among them, p current is the current fingertip position of the human hand, p stretch 、p contract are the predefined extended and contracted fingertip positions, θ current ,θ contract is the corresponding joint angle.

[0023] As an optimal technical solution, it also includes the state fine-tuning operation type posture, specifically:

[0024] receiving adjustment instructions input by an operator;

[0025] The joint angles of the extension or contraction posture of the current dexterous operation type are corrected in real time based on forward and inverse kinematics.

[0026] As a preferred technical solution, the step of creating a new smart operation type is also included, specifically:

[0027] Generate motion trajectories by physically guiding the manipulator;

[0028] Based on the virtual mass-damping-stiffness model Record joint motion data, M is virtual mass, B is virtual damping, K is virtual stiffness, external force F ext Estimated by motor current and position deviation;

[0029] Trajectories that meet the stability threshold are stored as a new dexterous maneuver type.

[0030] As an optimal technical solution, the preset multimodal large language model MLLM uses GPT-4.

[0031] In a second aspect, the present invention provides a teleoperation system based on a smart type library, which is applied to the teleoperation method based on the smart type library, and includes an operation type library construction module, an operation type retrieval module, and an interpolation mapping control module;

[0032] The operation type library construction module is used to construct a dexterous operation type library, which is used to store a plurality of predefined dexterous operation types; each dexterous operation type includes: the joint angle configuration of the manipulator in the extended state and the retracted state; the category and geometric feature description of the applicable object; the contact position and direction information between the manipulator and the object; and the semantic label of the operation intention;

[0033] The operation type retrieval module is used to receive task instructions and environmental perception data, parse the task instructions through a preset multimodal large language model (MLLM) to obtain corresponding operation steps, and match the dexterous operation type corresponding to the current step from the dexterous operation type library;

[0034] The interpolation mapping control module is used to achieve intuitive and precise control between the human hand and the robot arm through an interpolation mapping strategy; the interpolation mapping strategy specifically includes: capturing the position of the human fingertips in real time; calculating the normalized projection ratio of the current posture of the human hand relative to the extended state and the contracted state; and linearly interpolating the extended and contracted joint angles of the robot arm according to the projection ratio to generate the robot arm joint angle instructions.

[0035] In a third aspect, the present invention provides a manipulator operating device, comprising:

[0036] a motion capture unit configured to collect real-time motion data of the operator's hand, including knuckle angles acquired by the motion capture gloves and wrist position and posture acquired by the VR controller;

[0037] an environmental perception unit, including an RGB-D camera, configured to collect point clouds and object geometry information of the operating scene;

[0038] a processing unit, communicatively connected to the motion capture unit and the environment perception unit, wherein the processor is capable of executing the teleoperation method based on the dexterity type library;

[0039] The execution unit includes a multi-joint dexterous manipulator and a manipulator arm that drives the manipulator to move, and receives the joint angle instruction to execute the operation task.

[0040] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the remote operation method based on the smart type library.

[0041] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0042] The teleoperation system based on the smart type library proposed in this paper significantly improves the adaptability, stability, and task completion capability of the manipulator during teleoperation by constructing a type-guided operation mode, integrating a multimodal large language model to assist decision-making, and implementing an interpolation mapping human-machine control strategy. The specific technical effects are as follows:

[0043] 1. Overcome the problem of poor adaptability of action strategies in traditional redirection methods

[0044] This paper introduces dexterous manipulation types (Dexterous Manipulation Types) as an intermediate representation, decoupling human hand control from robotic arm motion. This allows the robotic arm to no longer be constrained to mimicking the specific gestures of the human hand, but instead select the optimal execution method based on its own structural characteristics. This strategy effectively circumvents the problem of motion mismatch caused by differences in hand shape, fundamentally resolving the core bottleneck of "human hand replication does not equal task completion," significantly improving the execution stability and success rate of complex tasks.

[0045] 2. Avoids the self-collision and unstable grasping problems caused by traditional remapping methods

[0046] Through predefined and optimized action types, each action type is annotated with legal "stretch" and "contract" boundary poses in the type library. Interpolation mapping is performed within this range based on the current hand gesture, ensuring the physical feasibility of the generated action. Furthermore, the preset contact point and force direction information of the type significantly improves the contact surface fit and grasping stability, effectively avoiding problems such as slippage and self-collision caused by mapping errors.

[0047] 3. Stimulate and give full play to the advantages of the manipulator structure to support more complex operation tasks

[0048] The system's manipulation library includes a large number of "robot-specific manipulation types," some of which are beyond the reach of the human hand. Combined with an interpolation mapping mechanism and a dynamic adjustment module, the robot arm can leverage its wider range of joint motion and high-degree-of-freedom control capabilities to accomplish tasks such as wide-angle grasping, manipulation of unusually shaped objects, and dual-object grasping, thus expanding the mission adaptability of teleoperation systems.

[0049] 4. Improved operational efficiency and demonstration data quality, and enhanced subsequent imitation learning effects

[0050] Experimental data shows that the proposed system achieves significantly faster completion times than traditional redirection systems for multiple typical teleoperation tasks, shortens average demonstration duration, and significantly improves success rates. Furthermore, the high-quality operation trajectories provide a more representative data foundation for subsequent imitation learning. The resulting trained policy performs better in automated tasks, with greater generalization and robustness.

[0051] 5. Enhanced the intuition and flexibility of human-computer interaction, adapting to multi-task, multi-object complex scenarios

[0052] This invention supports multimodal task command input based on speech and images, and automatically infers task steps and operation types through a large multimodal language model (such as GPT-4o), giving the system efficient and flexible adaptability. This mechanism is particularly suitable for complex scenarios involving long time sequences, multiple stages, and multiple objects, greatly reducing the operational threshold and improving the feasibility of practical deployment.

[0053] In summary, the present invention systematically solves many technical difficulties of traditional manipulator teleoperation systems in terms of task adaptability, operational stability, structural utilization, etc. through a complete set of type-guided teleoperation strategies with clear structure and efficient collaboration. It has broad engineering application prospects and research and promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0055] Figure 1 Schematic diagram of the flow of a remote operation method based on a smart type library according to an embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the structure of the smart operation type library according to an embodiment of the present invention;

[0057] Figure 3 This is a flow chart of a teleoperation retrieval and interpolation mapping control strategy based on a smart type library according to an embodiment of the present invention;

[0058] Figure 4 This is a schematic structural diagram of a manipulator operating device according to an embodiment of the present invention;

[0059] Figure 5 Schematic diagram of the structure of a teleoperation system based on a smart type library according to an embodiment of the present invention;

[0060] Figure 6 Schematic diagram of a manipulator operating device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0062] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0063] This embodiment is based on the teleoperation method of the dexterous type library, which introduces "dexterous manipulation types" as the intermediate expression between human hand control and manipulator action. Through the type-driven teleoperation process, it breaks through the structural difference adaptation difficulties and control performance bottlenecks faced by traditional methods based on human hand posture redirection. Figure 1 As shown, the operation method specifically includes the following steps:

[0064] S1. Construct a dexterous operation type library, which is used to store multiple predefined dexterous operation types; each dexterous operation type includes: the joint angle configuration of the manipulator in the extended state and the contracted state; the category and geometric feature description of the applicable object; the contact position and direction information between the manipulator and the object; and the semantic label of the operation intention.

[0065] This embodiment first designs and constructs a structured dexterous operation type library based on the structural characteristics of the manipulator and the typical task requirements in the teleoperation scenario. The library adopts a hierarchical classification method, including single-handed operation type and two-handed collaboration type, such as Figure 2 As shown. The one-hand type is divided into grasping type and non-grasping type. The grasping type is further divided into robot-specific grasping type and general grasping type; the two-hand collaborative type is further classified according to the symmetry or asymmetry of the functions of the two manipulators. The type library is organized using a hierarchical classification method and is divided into one-hand type and two-hand collaborative type. The one-hand type is further divided into grasping type (general type and robot-specific type) and non-grasping type, and the two-hand type is divided into symmetric collaborative type and asymmetric collaborative type. Each type records its characteristic posture (extension and contraction), operable objects, contact parts, geometric features, grasping direction and intention, and is annotated in a unified data format to provide semantic support for subsequent automatic retrieval and control execution.

[0066] Furthermore, each operation type in the smart operation type library contains clear and detailed annotation information, including:

[0067] (1) The specific joint configuration of the manipulator in the two postures of stretch and contract;

[0068] (2) Applicable object categories and geometric feature descriptions;

[0069] (3) the specific contact position and direction of the corresponding object;

[0070] (4) Clear operational intention (such as grabbing, lifting or using, etc.).

[0071] The above detailed annotations form clear specifications for the selection and execution of dexterous manipulator actions, providing precise guidance for the teleoperation process.

[0072] It is understandable that in the present invention, the dexterous operation type library is manually designed and annotated by R&D personnel based on typical tasks and manipulator structural characteristics. However, in order to improve construction efficiency and expansion capabilities, the generation process of this type library can be replaced by a data-driven method based on the Grasp Synthesis Model. Specifically, by training a deep neural network model, combined with the input object point cloud information, task description and manipulator structural parameters, a set of candidate operation postures can be automatically predicted and generated. The generated operation types can be subjected to physical feasibility detection and task adaptability verification, and automatically screened to form a reasonable structure and task matching operation type library. This method can not only quickly adapt to manipulators of different structures, but also dynamically generate new types according to task requirements, further improving the versatility and adaptability of the system.

[0073] S2, realize automatic search of types through the preset multimodal large language model MLLM;

[0074] This embodiment designs an automatic type retrieval module based on MLLM to realize the intelligent selection of the most appropriate operation type in the teleoperation task, such as Figure 3 As shown on the left. The specific search process is as follows:

[0075] S21. Convert all types and their attributes in the type library into natural language prompts and input them into MLLM (such as GPT-4o);

[0076] S22. During the actual operation, the operator inputs the natural language description and instructions of the task through voice or text;

[0077] S23,MLLM module first decomposes the overall task into clear logical steps, clarifying the specific operation object and operation method of each step;

[0078] S24. Next, the MLLM module infers the specific operational requirements of each step and retrieves the optimal dexterity type corresponding to each step;

[0079] S25. Finally determine the optimal dexterity type that each manipulator should adopt in each task step.

[0080] S3, achieve intuitive and precise control between human hand and robot arm through interpolation mapping strategy, such as Figure 3 As shown on the right, the specific control process includes:

[0081] S31. Define the extension and contraction postures of the manipulator and the human hand corresponding to the task type;

[0082] S32. Capture the current posture of the human hand in real time and calculate the normalized projection ratio (pratio) of the current fingertip position relative to the stretching and contracting postures. The specific calculation formula is:

[0083]

[0084] S33. Use the pratio value to perform linear interpolation on the joint angles of the manipulator in the extended and contracted states to obtain real-time and accurate manipulator joint angle instructions:

[0085] θ current =pratio·(θ contract -θ stretch )+θ stretch

[0086] To enhance the adaptability of teleoperation systems, this invention allows operators to fine-tune the position and rotation of predefined initial pose types in real time using keyboard or voice commands. This is achieved by first using forward kinematics to obtain the initial state of the manipulator. Position or rotation offsets are then applied based on the operator's adjustment commands, and the precise adjusted manipulator pose is calculated using inverse kinematics.

[0087] The interpolation mapping strategy adopted by the present invention calculates the target joint angle of the manipulator based on the linear position change of the human hand between the extended and contracted states. In order to improve the flexibility and nonlinear modeling ability of posture mapping, the linear strategy can be replaced by a nonlinear mapping model based on a neural network. Specifically, by collecting a large amount of posture pair data of human hands and manipulators under various types of operations, a multi-layer perceptron (MLP) or graph neural network (GNN) model is trained to learn the complex mapping relationship from the spatial position of the human fingertips to the joint angle of the manipulator. This alternative strategy can improve the modeling ability of multi-finger coordinated movements and boundary state postures, enhance the smoothness of control and the accuracy of operation, and has significant advantages in dealing with non-standard postures or nonlinear joint responses.

[0088] In addition, the present invention also provides an intuitive and convenient teaching mode, which allows users to create new dexterity types according to special needs without professional skills. The teaching mode is implemented using a flexible control strategy, and the specific control formula is:

[0089]

[0090] Where M is the virtual mass, B is the virtual damping, K is the virtual stiffness, and the external force Fext is estimated by the motor current and position deviation.

[0091] In summary, the present invention forms a structurally adaptive and highly stable teleoperation system through clear dexterous type library construction, intelligent type retrieval and selection, precise interpolation mapping control strategy, and flexible posture adjustment and teaching mechanism, which significantly improves the adaptability, stability and success rate of teleoperation in complex operation tasks, and gives full play to the structural advantages of the dexterous manipulator.

[0092] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0093] Based on the same principles as the smart type library-based teleoperation method described in the aforementioned embodiment, the present invention also provides a smart type library-based teleoperation system, which can be used to execute the smart type library-based teleoperation method described above. For ease of illustration, the schematic diagram of the smart type library-based teleoperation system embodiment only illustrates portions relevant to the present invention. Those skilled in the art will appreciate that the illustrated structure does not limit the device, and the device may include more or fewer components than shown, or combine certain components, or have a different component arrangement.

[0094] See also Figure 5 In another embodiment of the present application, a teleoperation system 100 based on a smart type library is provided, the system comprising an operation type library construction module 101, an operation type retrieval module 102, and an interpolation mapping control module 103;

[0095] The operation type library construction module 101 is used to construct a dexterous operation type library, which is used to store multiple predefined dexterous operation types; each dexterous operation type includes: the joint angle configuration of the manipulator in the extended state and the retracted state; the category and geometric feature description of the applicable object; the contact position and direction information between the manipulator and the object; and the semantic label of the operation intention;

[0096] The operation type retrieval module 102 is used to receive task instructions and environmental perception data, parse the task instructions using a preset multimodal large language model (MLLM) to obtain corresponding operation steps, and match the dexterous operation type corresponding to the current step from the dexterous operation type library;

[0097] The interpolation mapping control module 103 is used to achieve intuitive and precise control between the human hand and the robotic arm through an interpolation mapping strategy; the interpolation mapping strategy is specifically as follows: capturing the position of the human fingertips in real time; calculating the normalized projection ratio of the current posture of the human hand relative to the extended state and the contracted state; and linearly interpolating the extended and contracted joint angles of the robotic arm according to the projection ratio to generate the robotic arm joint angle instructions.

[0098] It should be noted that the remote operation system based on the smart type library of the present invention corresponds one-to-one to the remote operation method based on the smart type library of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the remote operation method based on the smart type library are all applicable to the embodiment of the remote operation based on the smart type library. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0099] In addition, in the implementation of the teleoperation system based on the smart type library in the above embodiment, the logical division of each program module is only an example. In actual application, the above functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the teleoperation system based on the smart type library is divided into different program modules to complete all or part of the functions described above.

[0100] See also Figure 6 In one embodiment, a manipulator operating device is provided, comprising:

[0101] a motion capture unit configured to collect real-time motion data of the operator's hand, including knuckle angles acquired by the motion capture gloves and wrist position and posture acquired by the VR controller;

[0102] an environmental perception unit, including an RGB-D camera, configured to collect point clouds and object geometry information of the operating scene;

[0103] a processing unit, communicatively connected to the motion capture unit and the environment perception unit, wherein the processor is capable of executing the teleoperation method based on the dexterity type library;

[0104] The execution unit includes a multi-joint dexterous manipulator and a manipulator arm that drives the manipulator to move, and receives the joint angle instruction to execute the operation task.

[0105] It is understandable that the specific hardware implementation of the teleoperation system includes human hand motion capture equipment (Rokoko gloves and Meta Quest 3VR controller), robotic arm (Kinova Gen3 robotic arm) and dexterous robotic hand (LEAP Hand), as well as Realsense L515 RGB-D camera for scene perception, such as Figure 4shown.

[0106] The manipulator is controlled using a proportional-derivative (PD) control strategy for joint positions, with the target position determined in real time by an interpolation mapping strategy. The robotic arm is controlled through the high-frequency Cartesian space velocity control interface provided by the Kinova manipulator, using a smooth uniform acceleration and deceleration trajectory strategy, and applying Kalman filtering to ensure the continuity and smoothness of motion.

[0107] Furthermore, the current system uses wearable motion capture devices (such as Rokoko gloves) and VR controllers to obtain the position and posture parameters of each finger joint of the human hand in real time for subsequent posture mapping and type matching. As an alternative, a hand key point detection method based on computer vision can be used to achieve contactless acquisition. In specific implementation, RGB or RGB-D cameras are used to capture hand images, and deep learning models such as MediaPipe Hands and OpenPose are used to perform spatial reconstruction and posture estimation of hand key points. By identifying the position change of the current gesture relative to the predefined extension and contraction states, the pratio value required for control can be calculated, and the manipulator can be driven to achieve precise control. This method can achieve high-precision human hand motion capture without relying on dedicated hardware, and is suitable for scenarios such as lightweight deployment and remote control.

[0108] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0109] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0110] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A remote operation method based on a smart type library, characterized in that: The steps include: Constructing a dexterous operation type library, the dexterous operation type library is used to store multiple predefined dexterous operation types; each dexterous operation type includes: the joint angle configuration of the manipulator in the extended state and the retracted state; the category and geometric feature description of the applicable object; the contact position and direction information between the manipulator and the object; and the semantic label of the operation intention; Receive task instructions and environmental perception data, parse the task instructions through the preset multimodal large language model MLLM to obtain the corresponding operation steps, and match the dexterous operation type corresponding to the current step from the dexterous operation type library; Intuitive and precise control between the human hand and the robotic arm is achieved through an interpolation mapping strategy; the interpolation mapping strategy specifically includes: capturing the position of the human fingertips in real time; calculating the normalized projection ratio of the human hand's current posture relative to the extended state and the contracted state; and linearly interpolating the extended and contracted joint angles of the robotic arm based on the projection ratio to generate the robotic arm joint angle instructions.

2. The teleoperation method based on the smart type library according to claim 1, characterized in that: The smart operation types adopt a hierarchical classification structure, including: One-hand operation type is subdivided into grasping type and non-grasping type; the grasping type is further divided into general grasping type and manipulator-specific grasping type; The two-handed collaboration type is subdivided into symmetrical collaboration type and asymmetrical collaboration type.

3. The teleoperation method based on the smart type library according to claim 1, characterized in that: The smart operation type corresponding to the current step is matched from the smart operation type library, specifically: Convert all operation types and their attributes in the type library into natural language prompts and input them into MLLM; Based on the task instructions input by voice or text, MLLM outputs the task decomposition steps and the adaptive dexterous operation type for each step.

4. The remote operation method based on the smart type library according to claim 1, characterized in that: The generating of the manipulator joint angle instruction is specifically as follows: i current =pratio·(θ contract -θ stretch )+θ stretch ; Among them, p current is the current fingertip position of the human hand, p stretch 、p contract are the predefined extended and contracted fingertip positions, θ current ,θ contract is the corresponding joint angle.

5. The remote operation method based on the smart type library according to claim 1, characterized in that: It also includes the state fine-tuning operation type posture, specifically receiving adjustment instructions input by an operator; The joint angles of the extension or contraction posture of the current dexterous operation type are corrected in real time based on forward and inverse kinematics.

6. The teleoperation method based on the smart type library according to claim 1, characterized in that: It also includes steps for creating new smart operation types, specifically: Generate motion trajectories by physically guiding the manipulator; Based on the virtual mass-damping-stiffness model Record joint motion data, M is virtual mass, B is virtual damping, K is virtual stiffness, external force F ext Estimated by motor current and position deviation; Trajectories that meet the stability threshold are stored as a new dexterous maneuver type.

7. The teleoperation method based on the smart type library according to claim 1, characterized in that: The preset multimodal large language model MLLM uses GPT-4.

8. A teleoperation system based on a smart type library, characterized in that: A remote operation method based on a smart type library applied to any one of claims 1 to 7, comprising an operation type library construction module, an operation type retrieval module, and an interpolation mapping control module; The operation type library construction module is used to construct a dexterous operation type library, which is used to store a plurality of predefined dexterous operation types; each dexterous operation type includes: the joint angle configuration of the manipulator in the extended state and the retracted state; the category and geometric feature description of the applicable object; the contact position and direction information between the manipulator and the object; and the semantic label of the operation intention; The operation type retrieval module is used to receive task instructions and environmental perception data, parse the task instructions through a preset multimodal large language model (MLLM) to obtain corresponding operation steps, and match the dexterous operation type corresponding to the current step from the dexterous operation type library; The interpolation mapping control module is used to achieve intuitive and precise control between the human hand and the robot arm through an interpolation mapping strategy; the interpolation mapping strategy specifically includes: capturing the position of the human fingertips in real time; calculating the normalized projection ratio of the current posture of the human hand relative to the extended state and the contracted state; and linearly interpolating the extended and contracted joint angles of the robot arm according to the projection ratio to generate the robot arm joint angle instructions.

9. A manipulator operating device, characterized in that: include: a motion capture unit configured to collect real-time motion data of the operator's hand, including knuckle angles acquired by the motion capture gloves and wrist position and posture acquired by the VR controller; an environmental perception unit, including an RGB-D camera, configured to collect point clouds and object geometry information of the operating scene; a processing unit, communicatively connected to the motion capture unit and the environment perception unit, wherein the processor is capable of executing the remote operation method based on the dexterity type library according to any one of claims 1 to 7; The execution unit includes a multi-joint dexterous manipulator and a manipulator arm that drives the manipulator to move, and receives the joint angle instruction to execute the operation task.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the remote operation method based on the smart type library according to any one of claims 1 to 7 is implemented.

Images