Remote operation control method and device, storage medium, program product and remote operation system

Abstract

The embodiment of the invention provides a remote operation control method and device, a storage medium, a program product and a remote operation system. The method comprises the following steps: under the condition that a remote operation system completes a registration mode, acquiring a motion angle of a hand collected by a first sensing device and a relative position relationship between a second sensing device and a head display device; acquiring a first mapping action of a dexterous hand of the robot based on the residual information and the motion angle determined by the teleoperation system in the registration mode, and acquiring a second mapping action of a mechanical arm of the robot based on the relative position relation; and the dexterous hand is controlled to execute the first mapping action, and the mechanical arm is controlled to execute the second mapping action, so that teleoperation control of the robot is achieved. The method is used for achieving the effect that more accurate teleoperation action mapping is achieved through refined action mapping of the dexterous hand of the robot and accurate mapping and track derivation of the tail end position of the mechanical arm of the robot.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of motion capture, and in particular to a teleoperation control method and device, a storage medium, a program product and a teleoperation system. BACKGROUND

[0002] With the development of robot technology becoming more and more mature, in order to make the action of the robot more bionic, motion capture technology is widely used in the fields of robot teleoperation, virtual reality interaction, etc.

[0003] The traditional motion capture scheme often relies on a large number of optical positioning cameras, inertial sensor arrays and other hardware devices, and the system deployment is complex, the cost is high and is limited by space; if a VR head-mounted device is combined with a simple tracker to realize motion capture, although the hardware configuration is simplified to some extent, the fine action mapping of the hand will be distorted, and it is difficult to meet the fine operation control requirements of the robot. SUMMARY

[0004] The embodiments of the present application provide a teleoperation control method, device, storage medium, program product and teleoperation system, so as to achieve fine action mapping of the dexterous hand of the robot, accurate mapping of the end position of the mechanical arm of the robot and trajectory derivation, and realize more accurate teleoperation action mapping effect.

[0005] In a first aspect, the embodiments of the present application provide a teleoperation control method applied to a teleoperation system, the teleoperation system comprising a robot and a first sensing device, a second sensing device and a head-mounted device worn by a teleoperator, wherein the first sensing device is worn on the hand of the teleoperator, and the second sensing device is worn at the wrist of the teleoperator.

[0006] The method comprises:

[0007] In the case that the teleoperation system completes the registration mode, the motion angle of the hand collected by the first sensing device and the relative position relationship between the second sensing device and the head-mounted device are acquired;

[0008] Based on the residual information determined by the teleoperation system in the registration mode and the motion angle, a first mapping action of the dexterous hand of the robot is acquired, and based on the relative position relationship, a second mapping action of the mechanical arm of the robot is acquired;

[0009] The dexterous hand is controlled to execute the first mapping action, and the mechanical arm is controlled to execute the second mapping action, so as to realize the teleoperation control of the robot.

[0010] In a possible implementation, the step of determining the residual information comprises:

[0011] In a case that the teleoperation system enters a registration mode, actual posture information collected by the first sensing device and standard posture information corresponding to the plurality of registration actions of the robot are determined based on the plurality of registration actions performed by the teleoperator, to determine residual information between the first sensing device and the robot.

[0012] In a possible implementation, the actual posture information includes first rotation parameters of a plurality of joints of a hand of the teleoperator, and the standard posture information corresponding to the plurality of registration actions of the robot includes second rotation parameters of a plurality of execution components of the dexterous hand.

[0013] The actual posture information collected by the first sensing device and the standard posture information corresponding to the plurality of registration actions of the robot are determined based on the plurality of registration actions performed by the teleoperator, to determine residual information between the first sensing device and the robot, including:

[0014] A residual function is constructed based on the first rotation parameters and the second rotation parameters corresponding to each registration action.

[0015] A least square method is used to fit the residual function, to obtain a target angle adjustment coefficient and a target scale adjustment coefficient as the residual information.

[0016] In a possible implementation, the motion angle includes target rotation parameters of a plurality of joints of a hand of the teleoperator, and the first mapping action includes first mapping rotation parameters of a plurality of execution components of the dexterous hand corresponding to the plurality of target rotation parameters.

[0017] The first mapping action of the dexterous hand of the robot is obtained based on the residual information determined by the teleoperation system in the registration mode and the motion angle, including:

[0018] The target rotation parameters are rotationally corrected by using the target angle adjustment coefficient.

[0019] The rotationally corrected target rotation parameters are three-dimensionally corrected, and the three-dimensionally corrected target rotation parameters are scaled by using the target scale adjustment coefficient, to obtain the first mapping rotation parameters corresponding to the target rotation parameters.

[0020] In a possible implementation, the relative position relationship includes position information of the second sensing device relative to the head-mounted device.

[0021] The method further includes:

[0022] The first coordinate system corresponding to the head-mounted device and the world coordinate system are coordinate-registered, to obtain a first transformation matrix.

[0023] The second coordinate system corresponding to the robot and the world coordinate system are registered to obtain the second transformation matrix;

[0024] The step of obtaining the second mapped action of the robot's robotic arm based on the relative positional relationship includes:

[0025] Based on the location information, the first transformation matrix, and the second transformation matrix, the second mapped action of the robot's robotic arm is obtained.

[0026] In one possible implementation, the second mapping action includes second mapping rotation parameters corresponding to each joint on the robot's robotic arm;

[0027] The step of obtaining the second mapped action of the robot's robotic arm based on the position information, the first transformation matrix, and the second transformation matrix includes:

[0028] The position information is transformed using the first transformation matrix;

[0029] Using the second transformation matrix, the transformed position information is transformed a second time to obtain the position information of the robot's end effector in the second coordinate system corresponding to the robot;

[0030] Based on the position information of the robot's end effector, the Jacobi pseudo-inverse method is used to determine the second mapping rotation parameters corresponding to each joint on the robot's end effector.

[0031] Secondly, this application provides a remote control device for use in a remote control system. The remote control system includes a robot and a first sensing device, a second sensing device, and a head-mounted display device worn by the remote operator. The first sensing device is worn on the remote operator's hand, and the second sensing device is worn on the remote operator's wrist.

[0032] The device includes:

[0033] The first acquisition module is used to acquire the hand movement angle collected by the first sensing device and the relative positional relationship between the second sensing device and the head-mounted display device when the remote control system completes the registration mode.

[0034] The second acquisition module is used to acquire the first mapped action of the robot's dexterous hand based on the residual information determined by the remote control system in the registration mode and the motion angle, and to acquire the second mapped action of the robot's robotic arm based on the relative position relationship.

[0035] The control module is used to control the dexterous hand to perform the first mapped action and control the robotic arm to perform the second mapped action, so as to realize the remote operation control of the robot.

[0036] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0037] The memory stores computer-executed instructions;

[0038] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0040] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0041] The remote control method, device, storage medium, program product, and remote control system provided in this application embodiment employ a remote control system including a first sensing device, a second sensing device, and a head-mounted display device. This can reduce the number of sensors used to a certain extent. Furthermore, by using the head-mounted display device, the robot's motion scene can be presented to the remote operator in real time. During operation, the remote operator does not need to frequently switch their gaze to understand the robot's remote operation response. The head-mounted display device can also serve as a reference coordinate to accurately calculate the spatial position of the remote operator's arm end. This avoids situations where the relative position of the arm and head does not actually change significantly due to the remote operator's instability or other reasons, but the robot's robotic arm executes incorrect mapping actions because it senses a large movement of the remote operator's arm. This ensures the accuracy and rationality of the robot's motion mapping during remote operation.

[0042] Based on the aforementioned teleoperation system, by collecting the motion angles of each joint of the operator's hand through the first sensor and then using residual information to correct positional deviations, it can be ensured that the execution joints of the robot's dexterous hand are precisely aligned with the operator's actual hand movements. Furthermore, by using the reference spatial position of the head-mounted display and the relative positional relationship between the second sensor and the head-mounted display, a targeted spatial coordinate system is established for the operator's arm movements. This allows for precise locking of the real-time position of the operator's wrist, ensuring that the second mapped action can reproduce the operator's true arm movement trajectory. Through the refined motion mapping of the robot's dexterous hand, the precise mapping of the robot's end effector position, and trajectory derivation, more accurate teleoperation motion mapping can be achieved. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] Figure 1 A schematic diagram of the structure of a remote control system provided in this application;

[0045] Figure 2 A flowchart illustrating a remote control method provided in this application;

[0046] Figure 3 This application provides a schematic diagram of the structure of a remote control device;

[0047] Figure 4 A schematic diagram of the structure of the electronic device provided in this application.

[0048] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0049] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0050] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0051] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0052] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0053] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0054] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0055] like Figure 1 As shown, this application embodiment provides a remote control system 100, which includes a first sensing device 101, a second sensing device 102, and a head-mounted display device 103 worn by a remote operator; and a robot (not shown in the figure).

[0056] The first sensing device 101 refers to a motion sensing device worn on the hand of the remote operator, which can collect motion data such as the rotation angle and swing amplitude of the palm and finger joints in real time. The first sensing device 101 can be, for example, an inertial measurement unit, a flexible strain sensor, etc.

[0057] In one embodiment, the first sensing device 101 employs a motion capture glove capable of capturing hand movements.

[0058] Motion capture gloves are high-precision motion sensing devices worn on the hands of remote operators, specifically designed to capture real-time motion data such as the bending angle, extension range, and rotation posture of the finger joints and wrist.

[0059] The motion capture glove collects motion data such as the rotation angle, spatial posture, and bending angle of the operator's hand joints by embedding various sensing devices (such as inertial measurement units, flexible strain sensors, and fiber optic sensors) at key locations in each joint of the fingers.

[0060] The second sensing device 102 refers to a position sensing device or attitude sensing device worn at the end of the operator's arm, i.e., at the wrist, for collecting the position information of the operator's wrist.

[0061] The head-mounted display device 103 refers to the device worn on the head of the remote operator. The head-mounted display device can not only locate the spatial position of the remote operator's head, but also has a display function to present the remote operator with information such as the real-time status of the robot.

[0062] By configuring the second sensor 102 with the head-mounted display 103, precise positioning of the end of the operator's arm can be achieved.

[0063] The robot is the execution end device of the remote control system 100. For example, the robot may have hardware structures corresponding to the upper limbs of the remote operator, such as dexterous hands and robotic arms. In this way, it can perform actions corresponding to the upper limbs of the remote operator by realizing the corresponding changes of the joints of the remote operator's hand and arm. The joint degrees of freedom of the robot are usually matched with those of the human upper limb to ensure the flexibility and accuracy of the operation.

[0064] It should be noted that the remote control system 100 also includes a controller 104, the hardware structure of which can be worn on the chest of the remote operator.

[0065] The controller 104 can communicate with the first sensor 101, the second sensor 102, the head-mounted display 103, and the robot respectively. It can determine the operator's hand movement commands based on the first sensor 101, and determine the operator's arm movement commands based on the second sensor 102 and the head-mounted display 103. It then sends the hand movement commands and arm movement commands to the robot to control the robot to perform movements corresponding to the operator's upper limbs, thereby realizing remote operation control of the robot.

[0066] In one embodiment, a remote control method is provided. This embodiment uses the application of this remote control method to the controller 104 in the aforementioned remote control system 100 as an example for illustration. Figure 2 As shown, the remote control method includes:

[0067] Step 202: With the remote control system in registration mode, acquire the hand movement angle collected by the first sensor and the relative positional relationship between the second sensor and the head-mounted display.

[0068] The registration mode refers to the mode used to calibrate the spatial positional relationship between the operator's upper limbs (including hands and arms) and the corresponding actuators of the robot (including joints corresponding to the operator's hands and arms) after the remote control system is started. The registration mode is used to establish a precise mapping relationship between the operator's motion angles and distances and the robot's motion angles and distances.

[0069] The motion angle refers to the rotation angle data of each joint of the teleoperator's hand collected by the first sensing device. The motion angle is used to reflect the specific movement posture of the teleoperator's hand.

[0070] The relative positional relationship refers to the spatial positional association data between the second sensing device and the head-mounted display device, which may include, for example, the distance, orientation, and relative motion trajectory between the second sensing device and the head-mounted display device. In this embodiment, the spatial position of the head-mounted display device can be used as a fixed coordinate reference to further calculate the spatial position and posture of the teleoperator's arm.

[0071] In this embodiment, after the operator puts on the hardware devices corresponding to the first sensing device, the second sensing device, the head-mounted display device, and the controller, the operator can trigger the remote control system to enter the registration mode, for example, by long-pressing the controller.

[0072] In registration mode, the teleoperator can perform several reference actions as prompted by the head-mounted display. At this time, the controller will collect the reference position data of the teleoperator's limbs through the first and second sensors. Then, it will compare and match these reference position data with the corresponding pre-stored standard position data of the robot to determine the residual information between the teleoperator's limbs and the robot's execution parts.

[0073] When the registration mode is completed, the controller can continuously collect rotation angle data of each joint of the teleoperator's hand through the first sensor, as well as spatial position data of the second sensor relative to the head display device, in order to realize real-time monitoring of the teleoperator's upper limb movements.

[0074] Step 204: Based on the residual information and motion angle determined by the remote control system in the registration mode, obtain the first mapped action of the robot's dexterous hand, and based on the relative position relationship, obtain the second mapped action of the robot's robotic arm.

[0075] Residual information refers to the deviation data obtained after matching and calculating the reference positions of the teleoperator's upper limbs and the various actuators of the robot's dexterous hand in the registration mode. The residual data is used to compensate for the differences between the teleoperator's motion angles and distances and the robot's motion angles and distances, so as to achieve error compensation during subsequent motion mapping and improve the accuracy of teleoperation control.

[0076] The first mapped action refers to the action command that is mapped to the execution component of the robot's dexterous hand after error compensation of the specific action posture of the remote operator's hand based on residual information. The first mapped action is precisely matched with the specific action posture of the remote operator's hand to ensure that the robot's dexterous hand can reproduce the remote operator's hand action.

[0077] The second mapping action refers to the action command of the robot's mechanical arm execution component, which is mapped to the spatial motion state of the end of the remote operator's arm, i.e. the wrist, after calculating the spatial motion state of the remote operator's arm based on the relative positional relationship between the second sensing device and the head-mounted display device, so as to realize the synchronous movement of the robot's mechanical arm and the remote operator's arm.

[0078] In this embodiment, after the controller obtains the rotation angle data of each joint of the teleoperator's hand and the spatial position data of the second sensing device relative to the head-mounted display device, it can call the residual information determined in the registration mode to perform error compensation calculation on the motion angle data of each joint of the teleoperator's hand in order to correct the deviation of the teleoperator's motion angle.

[0079] Subsequently, the controller can convert the compensated motion angle data into control instructions for each joint of the robot's dexterous hand, namely the first mapped action. The control instructions specify the rotation angle and motion sequence of each joint of the robot's dexterous hand, ensuring that the robot's dexterous hand can move in complete synchronization with the operator's hand movements.

[0080] Furthermore, the controller can use the spatial position data of the second sensing device relative to the head-mounted display device, combined with the coordinate reference of the head-mounted display device, to calculate the position change of the end of the teleoperator's arm, i.e., the wrist, through spatial geometric calculations. For example, by using the Jacobi pseudo-inverse method, the position change of the teleoperator's wrist can be converted into control commands for the corresponding joints of the robot, i.e., the second mapping action. These control commands specify the motion parameters of each joint of the robot's robotic arm, ensuring that the robot's robotic arm can accurately reproduce the movement of the teleoperator's arm.

[0081] Step 206: Control the dexterous hand to perform the first mapped action and control the robotic arm to perform the second mapped action to achieve remote operation control of the robot.

[0082] The aforementioned remote control method employs a remote control system comprising a first sensing device, a second sensing device, and a head-mounted display device. This reduces the number of sensors used to some extent. Furthermore, by using the head-mounted display device, the robot's motion scene can be presented to the operator in real time. During operation, the operator does not need to frequently switch their gaze to understand the robot's remote operation response. The head-mounted display device can also serve as a reference coordinate to accurately calculate the spatial position of the operator's arm end. This avoids situations where the operator's body tilts due to instability, resulting in the robot's robotic arm executing incorrect mapping actions even though the relative position of the arm and head has not actually changed significantly. This ensures the accuracy and rationality of the robot's motion mapping during remote operation.

[0083] Based on the aforementioned teleoperation system, by collecting the motion angles of each joint of the operator's hand through the first sensor and then using residual information to correct positional deviations, it can be ensured that the execution joints of the robot's dexterous hand are precisely aligned with the operator's actual hand movements. Furthermore, by using the reference spatial position of the head-mounted display and the relative positional relationship between the second sensor and the head-mounted display, a targeted spatial coordinate system is established for the operator's arm movements. This allows for precise locking of the real-time position of the operator's wrist, ensuring that the second mapped action can reproduce the operator's true arm movement trajectory. Through the refined motion mapping of the robot's dexterous hand, the precise mapping of the robot's end effector position, and trajectory derivation, more accurate teleoperation motion mapping can be achieved.

[0084] In some optional embodiments, the step of determining residual information includes:

[0085] When the remote control system enters the registration mode, the residual information between the first sensor and the robot is determined based on the actual posture information collected by the first sensor and the standard posture information of the robot corresponding to the multiple registration actions when the remote operator performs multiple registration actions.

[0086] Registration actions refer to a series of standard posture movements performed by the teleoperator using their hands, which are wearing the first sensor, in registration mode. Registration actions can include various reference movements such as palm down + fingers straight, palm down + fingers straight + thumb out, palm down + fingers straight + thumb down, and making a fist.

[0087] Actual posture information refers to the real posture data of each joint of the hand collected in real time by the first sensing device when the teleoperator performs each registration action. For example, it may include joint motion angle, spatial position coordinates, posture vector, etc. Actual posture information is used to reflect the actual movement state of the teleoperator's hand.

[0088] Standard posture information refers to the standard posture data of each joint of the robot's dexterous hand that is pre-stored in the telecontrol system and corresponds to each registered action. Standard posture information includes, for example, the structural parameters of each joint of the robot's dexterous hand, such as degrees of freedom and range of motion.

[0089] As an example, in this embodiment, after the operator puts on the hardware devices corresponding to the first sensing device, the second sensing device, the head-mounted display device, and the controller, the operator can trigger the remote control system to enter the registration mode, for example, by long-pressing the hardware device corresponding to the controller or by clicking a specific button on a specific interface displayed on the terminal connected to the controller.

[0090] In registration mode, the controller can generate prompts and send them to the head-mounted display (HMD) to control the HMD to display a specific animation segment. This animation segment is used to show the operator the hand posture for the first registration action, prompting the operator to perform the registration action with a hand wearing a motion capture glove. After the controller detects that the operator has performed the first registration action through the first sensor, it can then send a second prompt to the HMD to control the HMD to display the next animation segment, which is used to show the operator the hand posture for the second registration action, and so on, until the operator has performed all the registration actions.

[0091] It should be noted that when the head-mounted display receives a prompt command, it can also play a specific prompt voice through the voice module integrated on the head-mounted display to prompt the operator to make the registration action with the hand wearing motion capture gloves. The prompt voice may be something like "Please fully extend your fingers and hold for three seconds".

[0092] Furthermore, the teleoperator can sequentially perform multiple registration actions according to the prompts from the head-mounted display. During each registration action, the sensing unit built into the first sensing device can collect the actual posture information of each joint of the teleoperator's hand, such as the bending angle of each finger joint, the rotation angle of the wrist, and the overall spatial position coordinates of the hand, and send them to the controller.

[0093] The controller can perform preliminary preprocessing on the actual attitude information collected by the first sensing device, such as filtering environmental noise, removing abnormal data points, and smoothing data, to obtain preprocessed actual attitude information and avoid interference data from affecting subsequent deviation calculations.

[0094] Subsequently, the controller can acquire the robot's standard posture information corresponding to each registration action, and for each registration action, it will compare and calculate the preprocessed actual posture information with the invoked standard posture information dimension by dimension to obtain the single action deviation data corresponding to that registration action. It should be noted that the actuators of the robot's dexterous hand are set one-to-one with the finger joints of the remote operator's hand to ensure that the robot's dexterous hand can accurately reproduce the remote operator's hand movements.

[0095] Single-motion deviation data can include joint angle deviation and spatial motion deviation. Specifically, the joint angle deviation can be obtained by calculating the difference between the joint angle in the actual posture information and the joint angle of the actuator corresponding to the current finger joint in the standard posture information for each finger joint of the teleoperator. The spatial motion difference can be obtained by calculating the scaling ratio between the spatial motion of each finger joint in the world coordinate system in the actual posture information and the spatial motion of the actuator corresponding to the current finger joint in the standard posture information.

[0096] The joint angle deviation may include the first angle deviation in the vertical plane. The second angular deviation from the horizontal plane .

[0097] The spatial motion deviation between the operator's finger joints and the actuators of the robot's dexterous hand can also include a first motion deviation a in the vertical plane and a second motion deviation b in the horizontal plane.

[0098] In one embodiment, actual posture information may include first rotational parameters of multiple joints of the teleoperator's hand, and standard posture information may include second rotational parameters of multiple actuators of the dexterous hand.

[0099] The first rotational parameter may include the joint angles of the finger joints in the vertical plane and the joint angles in the horizontal plane. Similarly, the second rotational parameter may include the joint angles of the dexterous hand's actuators in the vertical plane and the joint angles in the horizontal plane.

[0100] As an example, after the teleoperator performs N registration actions, the controller can collect N sets of joint angles of any one of the teleoperator's finger joints in the vertical plane. Joint angle with the horizontal plane Correspondingly, the joint angle of the dexterous hand's actuator in the vertical plane corresponding to that finger joint can be obtained. Joint angle with the horizontal plane .

[0101] Furthermore, the controller can be based on the aforementioned N sets of first rotation parameters. and N sets of second rotation parameters Further determine the first angular deviation between the operator's finger joints and the actuators of the robot's dexterous hand. Second angle deviation The first motion deviation a and the second motion deviation b, wherein the first angle deviation Second angle deviation The first motion deviation a and the second motion deviation b can be understood as the residual information between the first sensing device and the robot under a single registration action.

[0102] The aforementioned teleoperation method can parameterize the joint rotation axis, spatial posture deviation, and joint range of motion scaling deviation between the teleoperator's finger joints captured by the motion capture glove and the dexterous hand's actuators based on the residual calculation of multiple sets of registered actions. Compared with single-point calibration or empirical compensation methods, the multi-posture residual accumulation method can cover the entire joint motion range, effectively avoid calibration blind spots under local postures, and ensure the universality and accuracy of the motion mapping model under all working conditions. This eliminates the difference between the human hand's tactile sensation and the dexterous hand's motion feedback, enabling the dexterous hand to naturally replicate the human hand's fine movements such as grasping and rotating.

[0103] In some optional embodiments, the step of determining the residual information between the first sensing device and the robot based on the actual posture information collected by the first sensing device and the standard posture information of the robot corresponding to the multiple registration actions when the remote operator performs multiple registration actions includes:

[0104] Based on the first and second rotation parameters corresponding to each registration action, a residual function is constructed.

[0105] The least squares method is used to fit the residual function to obtain the target angle adjustment coefficient and the target proportion adjustment coefficient as residual information.

[0106] As an example, the steps for constructing the residual function conform to the following formula:

[0107]

[0108] in, This represents the rotation matrix along the rotation axis in the vertical plane. Represents the rotation matrix along the rotation axis in the horizontal plane; , It is a mapping function between angles and vectors, used to convert the above rotation parameters into unit vectors in three-dimensional space.

[0109] This refers to the amplitude correction of the joint angles of the dexterous hand's actuators by measuring the first motion deviation 'a' in the vertical plane and the second motion deviation 'b' in the horizontal plane between the telemanipulator's finger joints and the robot's dexterous hand. This is used to compensate for the difference in the range of motion between the telemanipulator's hand and the dexterous hand, so that the corrected angles of the dexterous hand better match the motion amplitude characteristics of the human hand joints.

[0110] Furthermore, the controller can establish a least-squares optimization problem to solve for the first angular deviation contained in the residual function. Second angle deviation Given the first motion deviation 'a' and the second motion deviation 'b', the least squares optimization problem described above conforms to the following formula:

[0111]

[0112] The above least squares optimization problem satisfies the following constraints:

[0113]

[0114] Finally, after solving the residual function, the controller can obtain the target angle adjustment coefficient: the first angle deviation. Second angle deviation And the target ratio adjustment coefficient: first angle deviation Second angle deviation As the primary sensing device, it provides universal residual information for the robot during all registration actions.

[0115] The aforementioned remote control method can accurately correlate the first rotational parameter of the operator's hand joint with the second rotational parameter of the dexterous hand's actuator, constructing a residual function with the rotation axis deviation angle and the motion range proportionality coefficient as core variables. This achieves a parameterized and quantitative characterization of the posture deviation between the human hand and the dexterous hand. Compared with traditional qualitative calibration or single-point error compensation methods, this residual function relies on the full-dimensional posture data of multiple sets of registered actions, which can comprehensively cover the joint motion range, avoid calibration blind spots under local postures, and provide accurate and calculable mathematical model support for subsequent error compensation.

[0116] In some alternative embodiments, the motion angle includes target rotation parameters of multiple joints of the telemanipulator's hand; the first mapped action includes first mapped rotation parameters of multiple actuators of the dexterous hand corresponding to the multiple target rotation parameters;

[0117] Step 204 includes:

[0118] The target rotation parameters are corrected by using a target angle adjustment coefficient.

[0119] The target rotation parameters after rotation correction are subjected to three-dimensional correction, and the target rotation parameters after three-dimensional correction are scaled using the target scaling factor to obtain the first mapped rotation parameters corresponding to the target rotation parameters.

[0120] The target angle adjustment coefficient includes the first angle deviation. Second angle deviation .

[0121] Target rotation parameters may also include, for example, the joint angles of multiple joints of the teleoperator's hand in the vertical plane. Joint angle with the horizontal plane .

[0122] In this embodiment, the controller can use a first angular deviation. The joint angles of multiple joints in the operator's hand in the vertical plane Angle correction is performed using the second angle deviation. The joint angles of multiple joints in the operator's hand in the horizontal plane Angle corrections are performed to obtain the unit vectors of multiple joints of the teleoperator's hand in three-dimensional space. This allows for the rotational correction of the target rotational parameters.

[0123] in, , .

[0124] Furthermore, the unit vectors of multiple joints in the teleoperator's hand in three-dimensional space... By performing inverse kinematics, the rotational parameters of each actuator of the dexterous hand corresponding to multiple joints of the teleoperator's hand can be obtained, thereby enabling three-dimensional correction of the target rotational parameters after rotation correction. Here, the unit vectors of the multiple joints of the teleoperator's hand in three-dimensional space are used. To illustrate, among them, These represent the spatial positions of the unit vectors of multiple joints of the teleoperator's hand in three-dimensional space within the spatial coordinate system.

[0125]

[0126]

[0127] Furthermore, by using a target scaling factor, the target rotation parameters after three-dimensional correction are scaled proportionally to obtain the first mapped rotation parameters corresponding to the target rotation parameters, which conform to the following formula:

[0128]

[0129]

[0130] , This can be understood as the first mapping rotation parameter.

[0131] Finally, the controller can control the corresponding actuators on the dexterous hand for each joint of the teleoperator's hand, and execute the first mapped rotation parameters corresponding to that joint to realize the teleoperation control of the dexterous hand.

[0132] The aforementioned remote control method can first utilize the first angular deviation in the target angle adjustment coefficient. Second angle deviation The vertical plane angle of the human hand joint Angle with the horizontal plane Spatial posture correction directly compensates for spatial deviation errors between the human hand and the dexterous hand on the vertical and horizontal rotation axes. Subsequently, the original angle parameters can be transformed into three-dimensional unit vectors that fit the dexterous hand's motion coordinate system. Then, by using the inverse vector formula, we obtain... , This achieves a precise conversion from three-dimensional vector space to the angle space of a dexterous hand, solving the problem of the correspondence between angle parameters and spatial posture. Finally, the coefficients are adjusted in conjunction with the target proportion. and By scaling the scale, the difference in range of motion between the human hand and the dexterous hand is further compensated. The final output first mapping rotation parameter can accurately replicate the movement intention of the human hand, avoiding problems such as motion misalignment and grasping inaccuracy caused by axis deviation and range mismatch.

[0133] In some alternative embodiments, the relative positional relationship includes positional information of the second sensing device relative to the head-mounted display device;

[0134] Remote control methods also include:

[0135] The first coordinate system corresponding to the head-mounted display device and the world coordinate system are registered to obtain the first transformation matrix;

[0136] The robot's second coordinate system and the world coordinate system are registered to obtain the second transformation matrix;

[0137] Step 204 includes:

[0138] Based on position information, the first transformation matrix, and the second transformation matrix, the second mapped motion of the robot's robotic arm joints is obtained.

[0139] The first coordinate system refers to a coordinate system established with the head-mounted display device as the origin. The coordinate axes of the first coordinate system are bound to the physical posture of the head-mounted display device. For example, the Z-axis of the first coordinate system is along the viewing direction of the head-mounted display device, and the X-axis is along the horizontal direction.

[0140] The world coordinate system refers to the global coordinate system in the space where the remote sensing system is located. Its origin and coordinate axis directions are fixed, and it is used to realize the spatial position transformation and unification between different coordinate systems.

[0141] The second coordinate system refers to a coordinate system established with the robot as the origin, and the coordinate axes of the second coordinate system are bound to the robot's installation position.

[0142] Coordinate registration refers to the process of determining the spatial transformation relationship between two coordinate systems.

[0143] The first transformation matrix and the second transformation matrix, for example, each contain two parts: a rotation matrix and a translation matrix.

[0144] The process of determining the first transformation matrix may include, for example, selecting at least three non-collinear spatially calibrated control points, collecting the local coordinates of each control point in the first coordinate system and the global coordinates in the world coordinate system; and then, based on optimization algorithms such as the least squares method, fitting and solving the spatial transformation matrix that can realize the transformation from local coordinates to global coordinates to obtain the first transformation matrix.

[0145] The process of determining the second transformation matrix may include, for example, selecting at least three non-collinear spatially calibrated control points, collecting the local coordinates of each control point in the second coordinate system and the global coordinates in the world coordinate system; and then, based on optimization algorithms such as the least squares method, fitting and solving the spatial transformation matrix that can realize the transformation from local coordinates to global coordinates to obtain the second transformation matrix.

[0146] Specifically, the second mapping action includes the second mapping rotation parameters corresponding to each joint on the robot's robotic arm;

[0147] The steps for obtaining the second mapped action of the robot's robotic arm based on position information, a first transformation matrix, and a second transformation matrix include:

[0148] The first transformation matrix is ​​used to transform the position information;

[0149] The second transformation matrix is ​​used to perform a second transformation on the transformed position information to obtain the position information of the robot's end effector in the second coordinate system corresponding to the robot.

[0150] Based on the position information of the robot's end effector, the Jacobi pseudo-inverse method is used to determine the second mapping rotation parameters corresponding to each joint on the robot's end effector.

[0151] In this embodiment, the controller may first extract the position information of the second sensing device relative to the head-mounted display device, and then use the first transformation matrix to convert the local coordinates into global coordinates in the world coordinate system; then use the inverse matrix of the second transformation matrix to convert the global coordinates in the world coordinate system into local coordinates in the robot's second coordinate system, thereby determining the position of the operator's wrist relative to the head and the corresponding position information of the robotic arm end.

[0152] Furthermore, the controller can calculate the angle parameters that each actuator on the robotic arm needs to rotate based on the position information of the robotic arm end effector and the Jacobi pseudo-inverse method. The set of these angle parameters is the second mapped action of the robotic arm, which drives the robotic arm to move to the spatial position corresponding to the second sensing device.

[0153] The aforementioned remote control method first uses a first transformation matrix to convert the local position of the second sensing device relative to the head-mounted display device into a global position in the world coordinate system. Then, a second transformation matrix is ​​used to complete a secondary transformation, mapping the global position to the robot's second coordinate system, directly outputting the target position that the robotic arm's end effector needs to reach. This process uses the world coordinate system as a unified reference, achieving registration between the coordinate system of the head-mounted display device and the coordinate system of the robot, ensuring a complete match between the moving target position of the robotic arm's end effector and the spatial position associated with the head-mounted display device.

[0154] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0155] Based on the same inventive concept, this application also provides a state determination apparatus for implementing the state determination method described above. The solution provided by this state determination apparatus is similar to the solution described in the state determination method above. Therefore, the specific limitations in one or more apparatus embodiments provided below can be found in the limitations of the state determination method described above, and will not be repeated here.

[0156] In one embodiment, such as Figure 3As shown, a remote control device 300 is provided for use in a remote control system. The remote control system includes a robot and a first sensing device, a second sensing device, and a head-mounted display device worn by the remote operator. The first sensing device is worn on the remote operator's hand, and the second sensing device is worn on the remote operator's wrist.

[0157] The remote control device 300 includes:

[0158] The first acquisition module 302 is used to acquire the hand movement angle collected by the first sensor and the relative positional relationship between the second sensor and the head-mounted display device when the remote control system completes the registration mode.

[0159] The second acquisition module 304 is used to acquire the first mapped action of the robot's dexterous hand based on the residual information and motion angle determined by the remote control system in the registration mode, and to acquire the second mapped action of the robot's robotic arm based on the relative position relationship.

[0160] The control module 306 is used to control the dexterous hand to perform the first mapped action and control the robotic arm to perform the second mapped action, so as to realize the remote operation control of the robot.

[0161] In some optional embodiments, the second acquisition module 304 is further configured to:

[0162] When the remote control system enters the registration mode, the residual information between the first sensor and the robot is determined based on the actual posture information collected by the first sensor and the standard posture information of the robot corresponding to the multiple registration actions when the remote operator performs multiple registration actions.

[0163] In some optional embodiments, the actual posture information includes first rotational parameters of multiple joints of the teleoperator's hand, and the standard posture information of the robot corresponding to multiple registration actions includes second rotational parameters of multiple actuators of the dexterous hand.

[0164] The second acquisition module 304 is also configured as follows:

[0165] Based on the first and second rotation parameters corresponding to each registration action, a residual function is constructed.

[0166] The least squares method is used to fit the residual function to obtain the target angle adjustment coefficient and the target proportion adjustment coefficient as residual information.

[0167] In some optional embodiments, the motion angle includes target rotational parameters of multiple joints of the teleoperator's hand; the first mapped action includes first mapped rotational parameters of multiple actuators of the dexterous hand corresponding to the multiple target rotational parameters;

[0168] The second acquisition module 304 is also configured as follows:

[0169] The target rotation parameters are corrected by using a target angle adjustment coefficient.

[0170] The target rotation parameters after rotation correction are subjected to three-dimensional correction, and the target rotation parameters after three-dimensional correction are scaled using the target scaling factor to obtain the first mapped rotation parameters corresponding to the target rotation parameters.

[0171] In some optional embodiments, the relative positional relationship includes positional information of the second sensing device relative to the head-mounted display device;

[0172] The second acquisition module 304 is also configured as follows:

[0173] The first coordinate system corresponding to the head-mounted display device and the world coordinate system are registered to obtain the first transformation matrix;

[0174] The robot's second coordinate system and the world coordinate system are registered to obtain the second transformation matrix;

[0175] Based on the position information, the first transformation matrix, and the second transformation matrix, the second mapped action of the robot's robotic arm is obtained.

[0176] In some optional embodiments, the second mapping action includes second mapping rotation parameters corresponding to each joint on the robot's robotic arm;

[0177] The second acquisition module 304 is also configured as follows:

[0178] The first transformation matrix is ​​used to transform the position information;

[0179] The second transformation matrix is ​​used to perform a second transformation on the transformed position information to obtain the position information of the robot's end effector in the second coordinate system corresponding to the robot.

[0180] Based on the position information of the robot's end effector, the Jacobi pseudo-inverse method is used to determine the second mapping rotation parameters corresponding to each joint on the robot's end effector.

[0181] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0182] Figure 4 A schematic diagram of the structure of the electronic device provided in this application. Figure 4As shown, the electronic device 400 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 400 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0183] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0184] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0185] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0186] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0187] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0188] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the following method:

[0189] When the remote control system completes the registration mode, the movement angle of the hand collected by the first sensor and the relative positional relationship between the second sensor and the head-mounted display are obtained.

[0190] Based on the residual information and motion angle determined by the remote control system in the registration mode, the first mapped action of the robot's dexterous hand is obtained, and based on the relative position relationship, the second mapped action of the robot's robotic arm is obtained.

[0191] The robot is remotely controlled by controlling the dexterous hand to perform the first mapped action and controlling the robotic arm to perform the second mapped action.

[0192] In some alternative embodiments, the computer program, when executed by the processor, is also used to implement the following methods:

[0193] When the remote control system enters the registration mode, the residual information between the first sensor and the robot is determined based on the actual posture information collected by the first sensor and the standard posture information of the robot corresponding to the multiple registration actions when the remote operator performs multiple registration actions.

[0194] In some optional embodiments, the actual posture information includes first rotational parameters of multiple joints of the teleoperator's hand, and the standard posture information of the robot corresponding to multiple registration actions includes second rotational parameters of multiple actuators of the dexterous hand.

[0195] When a computer program is executed by a processor, it is also used to implement the following methods:

[0196] Based on the first and second rotation parameters corresponding to each registration action, a residual function is constructed.

[0197] The least squares method is used to fit the residual function to obtain the target angle adjustment coefficient and the target proportion adjustment coefficient as residual information.

[0198] In some optional embodiments, the motion angle includes target rotational parameters of multiple joints of the teleoperator's hand; the first mapped action includes first mapped rotational parameters of multiple actuators of the dexterous hand corresponding to the multiple target rotational parameters;

[0199] When a computer program is executed by a processor, it is also used to implement the following methods:

[0200] The target rotation parameters are corrected by using a target angle adjustment coefficient.

[0201] The target rotation parameters after rotation correction are subjected to three-dimensional correction, and the target rotation parameters after three-dimensional correction are scaled using the target scaling factor to obtain the first mapped rotation parameters corresponding to the target rotation parameters.

[0202] In some optional embodiments, the relative positional relationship includes positional information of the second sensing device relative to the head-mounted display device;

[0203] When a computer program is executed by a processor, it is also used to implement the following methods:

[0204] The first coordinate system corresponding to the head-mounted display device and the world coordinate system are registered to obtain the first transformation matrix;

[0205] The robot's second coordinate system and the world coordinate system are registered to obtain the second transformation matrix;

[0206] Furthermore, based on the position information, the first transformation matrix, and the second transformation matrix, the second mapped action of the robot's robotic arm is obtained.

[0207] In some optional embodiments, the second mapping action includes second mapping rotation parameters corresponding to each joint on the robot's robotic arm;

[0208] When a computer program is executed by a processor, it is also used to implement the following methods:

[0209] The first transformation matrix is ​​used to transform the position information;

[0210] The second transformation matrix is ​​used to perform a second transformation on the transformed position information to obtain the position information of the robot's end effector in the second coordinate system corresponding to the robot.

[0211] Based on the position information of the robot's end effector, the Jacobi pseudo-inverse method is used to determine the second mapping rotation parameters corresponding to each joint on the robot's end effector.

[0212] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the following method:

[0213] When the remote control system completes the registration mode, the movement angle of the hand collected by the first sensor and the relative positional relationship between the second sensor and the head-mounted display are obtained.

[0214] Based on the residual information and motion angle determined by the remote control system in the registration mode, the first mapped action of the robot's dexterous hand is obtained, and based on the relative position relationship, the second mapped action of the robot's robotic arm is obtained.

[0215] The robot is remotely controlled by controlling the dexterous hand to perform the first mapped action and controlling the robotic arm to perform the second mapped action.

[0216] In some optional embodiments, the processor, when executing computer execution instructions, is also configured to implement the following methods:

[0217] When the remote control system enters the registration mode, the residual information between the first sensor and the robot is determined based on the actual posture information collected by the first sensor and the standard posture information of the robot corresponding to the multiple registration actions when the remote operator performs multiple registration actions.

[0218] In some optional embodiments, the actual posture information includes first rotational parameters of multiple joints of the teleoperator's hand, and the standard posture information of the robot corresponding to multiple registration actions includes second rotational parameters of multiple actuators of the dexterous hand.

[0219] When executing computer-executable instructions, the processor is also used to implement the following methods:

[0220] Based on the first and second rotation parameters corresponding to each registration action, a residual function is constructed.

[0221] The least squares method is used to fit the residual function to obtain the target angle adjustment coefficient and the target proportion adjustment coefficient as residual information.

[0222] In some optional embodiments, the motion angle includes target rotational parameters of multiple joints of the teleoperator's hand; the first mapped action includes first mapped rotational parameters of multiple actuators of the dexterous hand corresponding to the multiple target rotational parameters;

[0223] When executing computer-executable instructions, the processor is also used to implement the following methods:

[0224] The target rotation parameters are corrected by using a target angle adjustment coefficient.

[0225] The target rotation parameters after rotation correction are subjected to three-dimensional correction, and the target rotation parameters after three-dimensional correction are scaled using the target scaling factor to obtain the first mapped rotation parameters corresponding to the target rotation parameters.

[0226] In some optional embodiments, the relative positional relationship includes positional information of the second sensing device relative to the head-mounted display device;

[0227] When executing computer-executable instructions, the processor is also used to implement the following methods:

[0228] The first coordinate system corresponding to the head-mounted display device and the world coordinate system are registered to obtain the first transformation matrix;

[0229] The robot's second coordinate system and the world coordinate system are registered to obtain the second transformation matrix;

[0230] Furthermore, based on the position information, the first transformation matrix, and the second transformation matrix, the second mapped action of the robot's robotic arm is obtained.

[0231] In some optional embodiments, the second mapping action includes second mapping rotation parameters corresponding to each joint on the robot's robotic arm;

[0232] When executing computer-executable instructions, the processor is also used to implement the following methods:

[0233] The first transformation matrix is ​​used to transform the position information;

[0234] The second transformation matrix is ​​used to perform a second transformation on the transformed position information to obtain the position information of the robot's end effector in the second coordinate system corresponding to the robot.

[0235] Based on the position information of the robot's end effector, the Jacobi pseudo-inverse method is used to determine the second mapping rotation parameters corresponding to each joint on the robot's end effector.

[0236] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0237] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0238] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0239] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0240] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0241] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0242] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0243] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A remote control method, characterized in that, The system is applied to a remote control system, which includes a robot and a first sensing device, a second sensing device, and a head-mounted display device worn by the remote operator. The first sensing device is worn on the remote operator's hand, and the second sensing device is worn on the remote operator's wrist. The method includes: When the remote control system completes the registration mode, the movement angle of the hand collected by the first sensor and the relative positional relationship between the second sensor and the head-mounted display are obtained. Based on the residual information determined by the remote control system in the registration mode and the motion angle, the first mapped action of the robot's dexterous hand is obtained, and based on the relative positional relationship, the second mapped action of the robot's robotic arm is obtained. The robot is remotely controlled by controlling the dexterous hand to perform the first mapped action and controlling the robotic arm to perform the second mapped action.

2. The method according to claim 1, characterized in that, The steps for determining the residual information include: When the remote control system enters the registration mode, based on the actual posture information collected by the first sensing device and the standard posture information of the robot corresponding to the multiple registration actions when the remote operator performs multiple registration actions, the residual information between the first sensing device and the robot is determined.

3. The method according to claim 2, characterized in that, The actual posture information includes the first rotation parameters of multiple joints of the teleoperator's hand, and the standard posture information of the robot corresponding to multiple registration actions includes the second rotation parameters of multiple actuators of the dexterous hand. When the remote operator performs multiple registration actions, the actual posture information collected by the first sensing device and the standard posture information of the robot corresponding to the multiple registration actions are used to determine the residual information between the first sensing device and the robot, including: Based on the first rotation parameters and the second rotation parameters corresponding to each of the registration actions, a residual function is constructed; The least squares method is used to fit the residual function to obtain the target angle adjustment coefficient and the target proportion adjustment coefficient as the residual information.

4. The method according to claim 3, characterized in that, The motion angle includes target rotation parameters of multiple joints of the teleoperator's hand; the first mapped action includes first mapped rotation parameters of multiple actuators of the dexterous hand corresponding to the multiple target rotation parameters; The step of obtaining the first mapped action of the robot's dexterous hand based on the residual information determined by the telecontrol system in the registration mode and the motion angle includes: The target rotation parameters are corrected by using the target angle adjustment coefficient. The target rotation parameters after rotation correction are subjected to three-dimensional correction, and the target scaling factor is used to scale the target rotation parameters after three-dimensional correction to obtain the first mapped rotation parameters corresponding to the target rotation parameters.

5. The method according to claim 1, characterized in that, The relative positional relationship includes the positional information of the second sensing device relative to the head-mounted display device; The method further includes: The first coordinate system and the world coordinate system corresponding to the head-mounted display device are registered to obtain the first transformation matrix; The second coordinate system corresponding to the robot and the world coordinate system are registered to obtain the second transformation matrix; The step of obtaining the second mapped action of the robot's robotic arm based on the relative positional relationship includes: Based on the location information, the first transformation matrix, and the second transformation matrix, the second mapped action of the robot's robotic arm is obtained.

6. The method according to claim 5, characterized in that, The second mapping action includes the second mapping rotation parameters corresponding to each joint on the robot's robotic arm; The step of obtaining the second mapped action of the robot's robotic arm based on the position information, the first transformation matrix, and the second transformation matrix includes: The position information is transformed using the first transformation matrix; Using the second transformation matrix, the transformed position information is transformed a second time to obtain the position information of the robot's end effector in the second coordinate system corresponding to the robot; Based on the position information of the robot's end effector, the Jacobi pseudo-inverse method is used to determine the second mapping rotation parameters corresponding to each joint on the robot's end effector.

7. A remote control device, characterized in that, The system is applied to a remote control system, which includes a robot and a first sensing device, a second sensing device, and a head-mounted display device worn by the remote operator. The first sensing device is worn on the remote operator's hand, and the second sensing device is worn on the remote operator's wrist. The device includes: The first acquisition module is used to acquire the hand movement angle collected by the first sensing device and the relative positional relationship between the second sensing device and the head-mounted display device when the remote control system completes the registration mode. The second acquisition module is used to acquire the first mapped action of the robot's dexterous hand based on the residual information determined by the remote control system in the registration mode and the motion angle, and to acquire the second mapped action of the robot's robotic arm based on the relative position relationship. The control module is used to control the dexterous hand to perform the first mapped action and control the robotic arm to perform the second mapped action, so as to realize the remote operation control of the robot.

8. A remote control system, characterized in that, The remote control system includes a first sensing device, a second sensing device, a head-mounted display, and a robot worn by the remote operator; The remote control system is used to perform the steps of the method as described in any one of claims 1 to 6, controlling the robot's hand to perform a first mapped action based on the motion angle of the first sensing device, and controlling the robot's arm joints to perform a second mapped action based on the relative positional relationship between the second sensing device and the head-mounted display device, so as to realize the remote operation control of the robot.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

Images