A dexterous hand remote control method, system, device and storage medium

Abstract

The application discloses a dexterous hand remote control method, system, device and storage medium. Its features include: the back-end processing platform receives the remote site image collected by the front-end control platform through the depth camera, and renders and displays the remote immersive picture to the operation user in real time based on the remote site image; the back-end processing platform collects the joint pose data of the user hand state in real time, generates the finger bending degree data based on the joint pose data, and sends the finger bending degree data to the front-end control platform; the front-end control platform receives the finger bending degree data, and generates the joint target angle instruction based on the finger bending degree data; the front-end control platform synchronizes the joint target angle instruction to the target mechanical hand in real time, so as to control the target mechanical hand to operate the user hand state of the user synchronously. The application can effectively improve the immersive visual feedback of the remote site, automatically synchronize the states of the hand and the mechanical hand, control by using the most natural gesture, and provide non-invasive and natural interactive experience.

Description

Technical Field

[0001] This invention relates to the field of dexterous hand control technology, and more particularly to a dexterous hand remote control method, system, device and storage medium. Background Technology

[0002] In hazardous environments, telemedicine, precision assembly, and special bomb disposal scenarios, the use of end effectors such as dexterous hands for remote control operations has extremely high application value. Utilizing the user's natural hand gestures to drive a remote robotic arm in real time to perform precise operations has become an important research direction in human-computer interaction and robot control.

[0003] Currently, traditional remote control solutions typically rely on the user observing a two-dimensional video feed on a flat-panel display (such as a monitor) and operating specialized input devices such as complex mechanical handles or data gloves to control the dexterous hand. In this solution, the user wears a data glove integrated with a bending sensor and an inertial measurement unit. The glove transmits the bending angles of each finger to the control host via wired or wireless means. The user views the two-dimensional video images transmitted from the end effector camera on a flat-panel monitor in front of them. The control host linearly scales the received angle data and maps it to the target position of the dexterous hand's servo motor or motor, driving the dexterous hand to move. In practical applications, existing technologies cannot provide depth information, resulting in a lack of presence, low operational efficiency, and difficulty for users to accurately judge object distances and the spatial posture of dexterous hands, leading to clumsy operation, low success rate, and long processing time. The equipment is complex and the interaction is unnatural: dedicated data gloves are usually cumbersome to wear, expensive, and require complex calibration and mapping with different models of dexterous hands; mechanical handles and other control methods are disconnected from natural human gestures, have high learning costs, and usually require fixed workstations and large-volume computing devices, resulting in poor portability and difficulty in rapid deployment in temporary or field scenarios. Summary of the Invention

[0004] This invention provides a method, system, device, and storage medium for remote control of a dexterous hand, in order to solve the technical problem in the prior art of how to achieve stable, natural, and low-latency remote control of a dexterous hand without any burden on the user.

[0005] According to one aspect of the present invention, a method for remotely controlling a dexterous hand is provided, comprising: The back-end processing platform receives remote field images captured by the front-end control platform through a depth camera, and renders and displays the immersive remote scene to the user in real time based on the remote field images. The backend processing platform collects joint pose data of the user's hand state in real time, generates finger bending data based on the joint pose data, and sends the finger bending data to the frontend control platform. The front-end control platform receives the finger bending data and generates a joint target angle command based on the finger bending data; The front-end control platform synchronizes the joint target angle command to the target manipulator in real time, so as to control the target manipulator to synchronize with the user's hand state.

[0006] According to another aspect of the present invention, a dexterous hand remote control system is provided, comprising: The real-time rendering module is used by the back-end processing platform to receive remote field images captured by the front-end control platform through a depth camera, and to render and display the remote immersive screen to the user in real time based on the remote field images. An angle processing module is used by the back-end processing platform to collect joint pose data of the user's hand state in real time, generate finger bending data based on the joint pose data, and send the finger bending data to the front-end control platform. The instruction analysis module is used by the front-end control platform to receive the finger bending data and generate a joint target angle instruction based on the finger bending data; The operation synchronization module is used by the front-end control platform to synchronize the joint target angle command to the target manipulator in real time, so as to control the target manipulator to synchronize the user's hand state with the operation user.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the dexterous hand remote control method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the dexterous hand remote control method according to any embodiment of the present invention.

[0009] The technical solution of this invention provides users with immersive visual feedback with natural depth perception by rendering and displaying a remote immersive scene in real time. This allows users to obtain a spatial positional relationship consistent with the remote scene, greatly enhancing the sense of presence and thus significantly improving the accuracy and efficiency of complex operations. The backend processing platform collects the joint pose data of the user's hand state, realizing wearable natural gesture interaction. The frontend control platform performs smooth and precise joint control of the fingers based on the curvature data, synchronizing with the remote dexterous hand with low latency. This enables the robotic hand to accurately and smoothly follow the user's natural gestures, solving the technical problem in the prior art of how to achieve stable, natural, and low-latency remote control of a dexterous hand without any burden on the user. This invention can effectively improve the immersive visual feedback at the remote scene, automatically synchronize the state of the hand and the robotic hand, use the most natural gestures for control, and provide a non-intrusive and natural interactive experience.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a method for remotely controlling a dexterous hand is provided in an embodiment of the present invention; Figure 2 A flowchart of another dexterous hand remote control method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a dexterous hand remote control system provided in an embodiment of the present invention; Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Figure 1 This invention provides a flowchart of a remote control method for a dexterous hand, applicable to situations where users can remotely operate a dexterous hand through an immersive remote screen without wearing an operating device. This method can be executed by a remote control system for the dexterous hand, which can be implemented in hardware and / or software and can be configured in any electronic device. Figure 1 As shown, the method includes: S110: The back-end processing platform receives remote field images captured by the front-end control platform through a depth camera, and renders and displays a remote immersive screen to the user in real time based on the remote field images.

[0016] The backend processing platform can be the device used by the user. It should be noted that the user uses the backend processing platform to remotely view the immersive video, and the immersive video is rendered and generated by the backend processing platform itself. Optionally, the backend processing platform sends the remote field images to the server, waits to receive the remote immersive screen rendered by the server, and then displays the remote immersive screen to the user.

[0017] Optionally, the backend processing platform also has an image acquisition function, which can acquire images of the user's surrounding environment and the user's hand.

[0018] The target robotic arm can be deployed remotely and is a multi-degree-of-freedom, multi-joint humanoid robotic arm used to perform manipulation tasks such as grasping, manipulating, and placing. It should be noted that the target robotic arm can also be referred to as a dexterous hand in this invention. For example, the dexterous hand can be selected as the RH56 dexterous hand, which is a humanoid five-fingered dexterous hand with six or more degrees of freedom.

[0019] The front-end control platform can be a device that connects to the target robot remotely. It connects to both the depth camera and the robot via serial ports. The platform can run Python programs to access the depth camera's SDK (Software Development Kit) and acquire each frame of the remote image, then sends these images to the back-end processing platform. For example, the front-end processor can be an industrial PC or a high-performance PC, connected to both the depth camera and the robot via serial ports (USB / RS232).

[0020] The remote field images can be color depth image data captured by a depth camera. It should be noted that the remote field images are acquired in real time by the depth camera controlled by the front-end control platform, and the remote field images include the remote field environment, the object being operated on, and the dexterous hand; and the remote field images are RGB-D (color + depth) image data.

[0021] The remote immersive view can be a first-person or third-person immersive view of the remote location. For example, taking AR smart glasses as an example, the AR rendering unit of the AR smart glasses is integrated into the Unity application. After receiving the remote location image, it uses each frame of the image as a dynamic material and renders it in real time on a virtual plane within the field of view of the glasses, providing the user with a remote immersive view from a first-person or third-person perspective.

[0022] Specifically, the backend processing platform receives remote field images captured by the frontend control platform through a depth camera, and renders and displays the remote immersive screen to the user in real time based on the remote field images.

[0023] S120. The back-end processing platform collects joint pose data of the user's hand state in real time, generates finger bending data based on the joint pose data, and sends the finger bending data to the front-end control platform.

[0024] The user's hand state can be the position of the entire palm and individual finger joints in three-dimensional space. Joint pose data can be the real-time geometric shape information of the user's hand state in three-dimensional space, including the three-dimensional coordinates of at least one hand joint skeletal node. For example, the joint pose data is captured by the 21-node hand skeletal tracking UXR 3.0 SDK in the backend processing platform, collecting the three-dimensional coordinates and pose information of the hand joint skeletal nodes from 13 key node pairs out of 21 skeletal nodes.

[0025] The finger flexion data can be the angle data of each degree of freedom of the user's hand corresponding to the dexterity hand. It should be noted that the finger flexion data is calculated from the joint pose data of the user's hand. The finger flexion data consists of angle data of multiple degrees of freedom of the user's hand corresponding to the dexterity hand. This can be understood as the user's hand having the same degrees of freedom as the dexterity hand. Calculating the finger flexion data of the user's hand allows us to obtain the corresponding degrees of freedom of the dexterity hand at a remote location. For example, when the dexterity hand has 6 degrees of freedom, the finger flexion data includes 6 angle data points, corresponding to 5 finger degrees of freedom and the thumb rotation degree of freedom, respectively.

[0026] Specifically, the backend processing platform collects joint pose data of the user's hand in real time, generates finger bending data based on the joint pose data, and sends the finger bending data to the frontend control platform.

[0027] S130, The front-end control platform receives the finger bending angle data and generates a joint target angle command based on the finger bending angle data.

[0028] The joint target angle command can be a UDP command containing joint angle values ​​corresponding to each degree of freedom of the dexterous hand. The joint target angle command can be used to control each degree of freedom of the dexterous hand to adjust to the corresponding joint angle value so as to completely synchronize the user's hand state.

[0029] Optionally, after receiving the finger bending data, the front-end control platform maps the finger bending data to the joint target angle command of the dexterous hand.

[0030] Specifically, the front-end control platform receives finger bending data and generates joint target angle commands based on the finger bending data.

[0031] S140: The front-end control platform synchronizes the joint target angle command to the target robot in real time, so as to control the target robot to operate the user's hand state synchronously.

[0032] Optionally, the front-end control platform can synchronize the joint target angle command to the target robot in real time. The target robot can then parse the joint target angle command to obtain the joint angle value of each degree of freedom. The target robot can then synchronize the joint angle value of each degree of freedom and remotely control the user's hand status.

[0033] Specifically, the front-end control platform synchronizes the joint target angle command to the target robotic arm in real time, so as to control the target robotic arm to operate the user's hand state synchronously.

[0034] The technical solution of this invention provides users with immersive visual feedback with natural depth perception by rendering and displaying a remote immersive scene in real time. This allows users to obtain a spatial positional relationship consistent with the remote scene, greatly enhancing the sense of presence and thus significantly improving the accuracy and efficiency of complex operations. The backend processing platform collects the joint pose data of the user's hand state, realizing wearable natural gesture interaction. The frontend control platform performs smooth and precise joint control of the fingers based on the curvature data, synchronizing with the remote dexterous hand with low latency. This enables the robotic hand to accurately and smoothly follow the user's natural gestures, solving the technical problem in the prior art of how to achieve stable, natural, and low-latency remote control of a dexterous hand without any burden on the user. This invention can effectively improve the immersive visual feedback at the remote scene, automatically synchronize the state of the hand and the robotic hand, use the most natural gestures for control, and provide a non-intrusive and natural interactive experience.

[0035] Optionally, in this invention, the channel for finger bending data between the front-end control platform and the back-end processing platform uses port 5006, and the data format is 24 bytes = 6 × 4 bytes little-endian signed int, in the following order: little finger, ring finger, middle finger, index finger, thumb bending, and thumb rotation.

[0036] Optionally, data interaction between the front-end control platform and the back-end processing platform is based on the UDP protocol (User Datagram Protocol). For example, the video stream channel between the front-end control platform and the back-end processing platform uses port 5005, with a 4-byte int (frame length header) + N×65507-byte JPEG packets. For instance, the AR smart glasses are worn directly by the user, and an Android application developed based on the Unity engine runs on the AR glasses. The communication and decoding unit of the AR glasses receives the video stream sent by the front-end control platform via UDP and decodes the video frames.

[0037] Optionally, in this invention, when the dexterous hand contacts any object, the backend processing platform also displays the force situation during the contact process between the dexterous hand and the object in real time, specifically: The front-end control platform collects real-time force sensing data of the target manipulator and uploads the real-time force sensing data to the back-end processing platform; The backend processing platform identifies each preset force display node of the target robot in the remote field image; The backend processing platform displays the real-time force sensing data at each of the preset force display nodes in the remote immersive screen.

[0038] The real-time force sensing data can be the real-time force data at the contact point when the target manipulator comes into contact with an object. It should be noted that at least one force sensor is installed in the target manipulator. The front-end control platform collects data corresponding to one degree of freedom for each force sensor. After detecting the force on the target manipulator, the force sensor senses the real-time force magnitude and obtains the real-time force sensing data.

[0039] Optionally, the preset force display nodes can be set at the skeletal node positions corresponding to each degree of freedom of the target robot. It should be noted that setting preset force display nodes for each degree of freedom of the target robot enables a visual display of the force situation of each degree of freedom.

[0040] Specifically, the front-end control platform collects real-time force sensing data of the target robot and uploads it to the back-end processing platform; the back-end processing platform identifies each preset force display node of the target robot in the remote field image; and the back-end processing platform displays the real-time force sensing data at each preset force display node in the remote immersive screen.

[0041] Furthermore, in this invention, the backend processing platform displays the real-time force sensing data at each of the preset force display nodes in the remote immersive image, including: The backend processing platform determines the real-time force display attributes and real-time force values ​​corresponding to each preset force display node based on the real-time force sensing data. The backend processing platform matches the corresponding force visualization pattern based on the real-time force display attribute, and displays the force visualization pattern and real-time force value at each preset force display node in the remote immersive screen.

[0042] The force visualization pattern can be a dynamic display image that changes dynamically based on real-time force sensing data. Real-time force display attributes can be the display attributes of the force visualization pattern within a remote immersive screen. Real-time force values ​​can describe the force applied to the dexterous hand; these values ​​serve as labels during the force visualization pattern display process and change in real-time according to the real-time force display attributes. It should be noted that real-time force display attributes can include the size, color, and brightness of the force visualization pattern. For example, the force visualization image can be a visualized ball, whose color can be adjusted in real-time according to the force values, such as a blue-to-red gradient or rainbow colors; its luminous intensity and size can also be adjusted, and the real-time force values ​​can be displayed near the force visualization image.

[0043] Optionally, during the force data display process, the front-end control platform collects real-time force sensing data corresponding to each degree of freedom of the target manipulator. This real-time force sensing data is then sent to the back-end processing platform via UDP port 5008. The back-end processing platform starts an independent background thread to listen to UDP port 5008, reading the real-time force sensing data in real time, and thus determining the real-time force display attributes and real-time force values ​​for each preset force display node. The real-time force sensing data is in 24-byte = 6 × 4-byte big-endian signed int format.

[0044] Specifically, the backend processing platform determines the real-time force display attributes and real-time force values ​​corresponding to each preset force display node based on real-time force sensing data; the backend processing platform matches the corresponding force visualization pattern based on the real-time force display attributes; and the backend processing platform displays the force visualization pattern and real-time force values ​​at each preset force display node in the remote immersive screen.

[0045] Figure 2 This is a flowchart illustrating another remote control method for a dexterous hand provided by an embodiment of the present invention. The relationship between this embodiment and the above embodiments is that this specifically describes the process by which the backend processing platform analyzes the finger flexion data of the user's hand in real time. Figure 2 As shown, the method includes: S210: The back-end processing platform receives remote field images captured by the front-end control platform through a depth camera, and renders and displays the immersive remote scene to the operator in real time based on the remote field images.

[0046] The S220 backend processing platform collects the three-dimensional coordinates of at least one hand joint skeletal node in real time, representing the user's hand status.

[0047] The joint pose data includes the three-dimensional coordinates of at least one hand joint bone node. It should be noted that the three-dimensional coordinates of the hand joint bone node can be the three-dimensional coordinates of each joint bone node of the user's hand in the real time coordinate system.

[0048] Optionally, in this invention, the backend processing platform collects the user's hand status in real time, and collects the three-dimensional coordinates of at least one hand joint bone node of the user's hand status in each frame.

[0049] Optionally, the three-dimensional coordinates of the hand joint skeletal nodes include the three-dimensional coordinates of the fingertip joints, the three-dimensional coordinates of the joint midpoints, the three-dimensional coordinates of the wrist, or the three-dimensional coordinates of the palm base. For example, the three-dimensional coordinates of the hand joint skeletal nodes include at least one of the following: the tip of the little finger, the midpoint of the little finger, the base of the little finger, the tip of the thumb, the midpoint of the thumb, the base of the thumb, the tip of the middle finger, the midpoint of the middle finger, the base of the middle finger, the tip of the ring finger, the midpoint of the ring finger, the base of the ring finger, the tip of the index finger, the midpoint of the index finger, the base of the index finger, the wrist joint, and the palm base. The three-dimensional coordinates of the fingertip joints can be the three-dimensional coordinates corresponding to the tips of the little finger, thumb, middle finger, ring finger, and index finger; the three-dimensional coordinates of the joint midpoints can be the three-dimensional coordinates corresponding to the midpoints of the little finger, the base of the little finger, the midpoints of the thumb, the base of the thumb, the midpoints of the middle finger, the base of the middle finger, the midpoints of the ring finger, the base of the ring finger, the midpoints of the index finger, and the base of the index finger.

[0050] Specifically, the backend processing platform collects the three-dimensional coordinates of at least one hand joint skeletal node in real time, representing the user's hand status.

[0051] For each degree of freedom hand joint of the target robotic hand, the S230 and backend processing platform match at least three three-dimensional coordinates of the corresponding hand joint bone nodes, and calculate the finger bending degree of the degree of freedom hand joint based on the three-dimensional coordinates of all the hand joint bone nodes.

[0052] Optionally, the degrees of freedom of the hand joints can be the various degrees of freedom of the target manipulator. Each degree of freedom hand joint corresponds to a finger of the user's hand, and the three-dimensional coordinates of the hand joint bone nodes matched to each degree of freedom hand joint of the target manipulator are the three-dimensional coordinates of the hand joint bone nodes corresponding to the user's hand state. For example, if the degree of freedom of the hand joint is the index finger joint, then the three-dimensional coordinates of the optional hand joint bone nodes corresponding to the index finger joint degree of freedom are the three-dimensional coordinates of the hand joint bone nodes corresponding to the fingertip, the middle node, the base of the index finger, the wrist joint, and the palm base.

[0053] Among them, finger flexion can be the flexion angle of each finger in the user's hand position, and the flexion angle that the hand joints of the dexterity hand need to reproduce. It should be noted that when calculating the finger flexion of each hand joint of the degree of freedom, at least the three-dimensional coordinates of the wrist or the three-dimensional coordinates of the palm base are required to ensure the accuracy of the calculation.

[0054] Optionally, the backend processing platform synchronizes the user's hand state for each frame, and calculates the degree of freedom of each hand joint corresponding to the target robotic hand frame by frame to obtain the degree of curvature of each finger corresponding to the user's hand state in each frame.

[0055] Specifically, for each degree of freedom hand joint of the target robotic hand, the backend processing platform matches at least three three-dimensional coordinates of the corresponding hand joint bone nodes, and calculates the finger bending degree for the degree of freedom hand joint based on the three-dimensional coordinates of all the hand joint bone nodes.

[0056] Optionally, in another optional embodiment of the present invention, the step of calculating the finger flexure degree of the hand joint based on the three-dimensional coordinates of all the hand joint skeletal nodes includes: If the degree of freedom hand joint is a degree of freedom finger joint, then the fingertip joint vector and the joint wrist vector are calculated based on the three-dimensional coordinates of the fingertip joint, the three-dimensional coordinates of the joint midpoint, and the three-dimensional coordinates of the wrist, respectively, and the finger flexion is determined based on the fingertip joint vector and the joint wrist vector; if the degree of freedom hand joint is a thumb rotation joint, then the thumb rotation vector and the joint palm root vector are calculated based on the three-dimensional coordinates of the thumb fingertip joint, the three-dimensional coordinates of the index finger joint midpoint, and the three-dimensional coordinates of the palm root, respectively, and the finger flexion is determined based on the thumb rotation vector and the joint palm root vector.

[0057] Optionally, the dexterous hand of the present invention has two different types of hand joints with degrees of freedom. For each type, a finger bending degree calculation method is set. If the hand joint with degrees of freedom is a finger joint, the calculation is performed using the three-dimensional coordinates of the fingertip joint, the three-dimensional coordinates of the joint midpoint, and the three-dimensional coordinates of the wrist. If the hand joint with degrees of freedom is a thumb rotation joint, the calculation is performed using the three-dimensional coordinates of the fingertip joint, the three-dimensional coordinates of the index finger joint midpoint, and the three-dimensional coordinates of the palm root.

[0058] Optionally, the degrees of freedom hand joints can be understood as the five fingers of the user's hand corresponding to the dexterity hand, used to map the degree of bending of each finger; while the thumb rotation joint is the degree of opening and closing of the thumb relative to the metacarpal bone of the index finger.

[0059] Optionally, in the invention, the fingertip joint vector can be a vector calculated from the three-dimensional coordinates of the fingertip joint and the three-dimensional coordinates of the joint midpoint; the fingertip joint vector is used to represent the angular direction from the fingertip to the finger joint; the joint wrist vector can be a vector calculated from the three-dimensional coordinates of the joint midpoint and the three-dimensional coordinates of the wrist; the fingertip joint vector is used to represent the angular direction from the finger joint to the wrist; after calculating the fingertip joint vector and the joint wrist vector, the angle between the fingertip joint vector and the joint wrist vector is calculated as the finger curvature.

[0060] Optionally, the thumb rotation vector can be a vector calculated from the three-dimensional coordinates of the fingertip joint and the three-dimensional coordinates of the midpoint of the index finger joint. The thumb rotation vector is used to represent the angular direction vector of the thumb relative to the index finger. The palmar joint vector can be a vector calculated from the three-dimensional coordinates of the midpoint of the joint and the three-dimensional coordinates of the palmar joint. The palmar joint vector is used to represent the angular direction vector from the midpoint of the index finger joint to the palmar joint. After calculating the thumb rotation vector and the palmar joint vector, the angle between the thumb rotation vector and the palmar joint vector is calculated as the finger flexion degree corresponding to the thumb rotation degree of freedom of the dexterous hand.

[0061] Specifically, if the degree of freedom of the hand joint is the degree of freedom of the finger joint, then the fingertip joint vector and the joint wrist vector are calculated based on the three-dimensional coordinates of the fingertip joint, the three-dimensional coordinates of the joint midpoint, and the three-dimensional coordinates of the wrist. The finger flexion is determined based on the fingertip joint vector and the joint wrist vector. If the degree of freedom of the hand joint is the thumb rotation joint, then the thumb rotation vector and the joint palm root vector are calculated based on the three-dimensional coordinates of the thumb fingertip joint, the three-dimensional coordinates of the index finger joint midpoint, and the three-dimensional coordinates of the palm root. The finger flexion is determined based on the thumb rotation vector and the joint palm root vector.

[0062] The S240 backend processing platform performs a first-order exponential low-pass filter on all finger curvatures to obtain finger curvature data.

[0063] Optionally, the first-order exponential low-pass filter can be used to filter noise in the finger curvature of each frame to remove high-frequency noise present in the finger curvature and eliminate natural high-frequency hand jitter. It should be noted that the first-order exponential low-pass filter can be a first-order infinite impulse response low-pass filter, with the filtering result of the previous frame, i.e., the finger curvature data, set to a weight of 0.7, and all finger curvatures in the current frame set to a weight of 0.3, with a smoothing factor of 0.3.

[0064] Optionally, after obtaining all finger flexion degrees in each frame, the back-end processing platform uses a first-order infinite impulse response low-pass filter to filter all finger flexion degrees in each frame before sending them to the front-end control platform, thus obtaining finger flexion degree data.

[0065] Specifically, the backend processing platform performs a first-order exponential low-pass filter on all finger curvatures to obtain finger curvature data.

[0066] The S250 front-end control platform receives finger bending data and generates joint target angle commands based on the finger bending data.

[0067] Optionally, in this invention, the step of generating a joint target angle command based on the finger curvature data includes: the front-end control platform acquiring force sensing data and angle sensing data of the target manipulator; the front-end control platform mapping the finger curvature data using a preset power function mapping algorithm under the constraint of a preset mapping range to determine the target position of the dexterous hand joint; and the front-end control platform generating a command based on the force sensing data, the angle sensing data, and the target position of the dexterous hand joint to determine the joint target angle command.

[0068] Among them, the force sensing data can be the force dataset of each degree of freedom of the target robot, which records the force situation of each degree of freedom. It should be noted that the front-end control platform collects the force sensing data of the target robot through the force sensors set on each degree of freedom of the target robot.

[0069] Among them, the angle sensing data can be the angle dataset of each degree of freedom of the target robot. The angle sensing data records the real-time angle value of each degree of freedom. It should be noted that the front-end control platform collects the angle sensing data of the target robot through the angle sensors set on each degree of freedom of the target robot.

[0070] Optionally, the preset mapping range can be the range of natural opening and gripping angles pre-calibrated for each degree of freedom of the target robot. The preset mapping range can be used to control the range of commands for moving the target robot, providing fine control over the target robot.

[0071] The preset power function mapping algorithm can be used to map finger bending data to the corresponding movement positions for each degree of freedom. For example, the preset power function mapping algorithm can be an S-shaped power function mapping algorithm.

[0072] Among them, the target position of the dexterous hand joint can be the position reached by each joint of the target robot to reproduce the user's hand state. It should be noted that the user's hand state is reproduced by controlling each degree of freedom of the target robot to adjust to the target position of the dexterous hand joint.

[0073] Optionally, in this invention, when using a preset power function mapping algorithm, the finger bending data is first restricted to a preset mapping range, and the finger bending corresponding to each degree of freedom is normalized to 0 to 1 to obtain a normalized angle ratio. The preset power function mapping algorithm is then used to map the normalized angle ratio of each degree of freedom to obtain the target position of the dexterous hand joint, and the target position of the dexterous hand joint is restricted to the range of fully clenched and fully open. For example, taking an asymmetric power function mapping algorithm, where r represents the normalized angle ratio, the mapping process of the target position of the dexterous hand joint is: theta = theta_min + (theta_max - theta_min) * r^k. When k>1, the output changes smoothly near the open position (r≈0), effectively suppressing joint position shaking caused by the micro-tremors of the user's hand in the open state; the output changes faster near the clenched position (r≈1), achieving a rapid response. That is, the gain increases with the degree of finger bending, achieving shaking suppression in the fully open state. When k < 1, the grip becomes more sensitive and stable, enabling precise control. This effectively suppresses joint vibrations caused by slight hand tremors near extreme positions. Specifically, theta_min = 0 represents the fully closed grip position, and +theta_max = 1000 represents the fully open grip position.

[0074] Optionally, after obtaining the target position of the dexterous hand joint, the front-end control platform combines real-time force sensing data and angle sensing data to generate joint target angle commands to control each degree of freedom of the target manipulator to the target position of the dexterous hand joint.

[0075] Specifically, the front-end control platform collects force and angle sensing data of the target manipulator; under the constraint of a preset mapping range, the front-end control platform uses a preset power function mapping algorithm to map the finger bending data to determine the target position of the dexterous hand joint; the front-end control platform generates instructions based on the force sensing data, angle sensing data, and the target position of the dexterous hand joint to determine the joint target angle instruction.

[0076] Optionally, in another optional embodiment of the present invention, the front-end control platform generates instructions based on the force sensing data, the angle sensing data, and the target position of the dexterous hand joint, and determines the joint target angle instruction, including: The front-end control platform verifies the force sensing data and the angle sensing data to determine reasonable force data and reasonable angle data; the front-end control platform performs sliding window mean filtering on the reasonable force data and the reasonable angle data to determine smooth force data and smooth angle data; the front-end control platform generates instructions based on the smooth force data, the smooth angle data, and the target position of the dexterous hand joint to determine the joint target angle instruction.

[0077] The reasonable force data can be a frame of force sensing data that does not contain abnormal force data; the reasonable angle data can be a frame of angle data that does not contain abnormal angle data. It should be noted that if any of the force sensing data or the reasonable angle data contains abnormal data, the entire frame of force sensing data and reasonable angle data will be discarded.

[0078] Optionally, during data verification, reasonable force and angle ranges are set for the force and angle sensing data, respectively. The force sensing data for each degree of freedom is compared with the reasonable force range, and the angle sensing data is compared with the reasonable angle range. If any abnormal data is found in either the force sensing data or the reasonable angle data, the entire frame containing both force sensing data and reasonable angle data is discarded. This prevents abnormal commands from driving the dexterous hand to dangerous positions.

[0079] Optionally, after data verification, a sliding window mean filter can be applied to the reasonable force data and reasonable angle data. The smoothed force data can be the smoothed force data of the target robot arm across multiple frames, and the smoothed angle data can be the smoothed angle data of the target robot arm across multiple frames. The smoothed force data and smoothed angle data can be used to reflect the real changes in force and angle of the target robot arm, reducing the impact of instantaneous disturbances on control decisions.

[0080] Optionally, sliding window mean filtering can be a filtering method that uses a sliding window to smooth reasonable force and angle data. It should be noted that before sliding window mean filtering, the width of the sliding window must first be set, for example, to 5 frames; and a force cache list and an angle cache list must be defined to store the reasonable force and angle data. During sliding window mean filtering, the reasonable force and angle data of the latest frame are appended to the end of the force cache list and the angle cache list, respectively. The sliding window controls the amount of data in the force cache list and the angle cache list to be 5 frames. The data within the sliding window is arithmetically averaged and used as the smoothed output value for the current moment, resulting in smoothed force and angle data.

[0081] Optionally, after obtaining the smoothed force data and smoothed angle data from the front-end control platform, the front-end control platform generates control instructions for the target position, smoothed force data and smoothed angle data of the dexterous hand joint in each frame, obtains the joint target angle instruction for each frame, and synchronizes the joint target angle instruction for each frame to the target manipulator in real time.

[0082] The S260 and front-end control platform synchronize the joint target angle command to the target robot in real time, so as to control the target robot to operate the user's hand state synchronously.

[0083] The technical solution of this invention provides users with immersive visual feedback with natural depth perception by rendering and displaying a remote immersive scene in real time. This allows users to obtain a spatial positional relationship consistent with the remote scene, greatly enhancing the sense of presence and thus significantly improving the accuracy and efficiency of complex operations. The backend processing platform collects the joint pose data of the user's hand state, realizing wearable natural gesture interaction. The frontend control platform performs smooth and precise joint control of the fingers based on the curvature data, synchronizing with the remote dexterous hand with low latency. This enables the robotic hand to accurately and smoothly follow the user's natural gestures, solving the technical problem in the prior art of how to achieve stable, natural, and low-latency remote control of a dexterous hand without any burden on the user. This invention can effectively improve the immersive visual feedback at the remote scene, automatically synchronize the state of the hand and the robotic hand, use the most natural gestures for control, and provide a non-intrusive and natural interactive experience.

[0084] Optionally, in this invention, the backend processing platform uses a depth camera mounted on the platform to capture the user's hand, collecting the three-dimensional coordinates of each hand joint skeletal node corresponding to the user's hand state. Simultaneously, it also captures the user's wrist posture and transmits this data through the wrist posture channel between the backend processing platform and the frontend control platform, sending it to the frontend control platform via port 5007. During the control command generation process, the frontend control platform also combines the user's wrist posture with the target position of the dexterous hand joint, smoothed force data, and smoothed angle data of each frame, along with the wrist posture of each frame, to generate control commands, obtaining the joint target angle command for each frame. The wrist posture can be the user's wrist pose, which can be used to control the wrist joints of the robotic arm.

[0085] For example, in this invention, after the front-end control platform receives the execution command from the AR glasses, the program automatically scans the serial port and connects to the dexterous hand at a baud rate of 115200, sending a reset command to return each joint to its original position. The joint repositioning can be achieved by setting the target position of each axis to 1000. The Unity application is launched on the Rokid AR glasses, entering the scene and displaying a remote immersive view. The Rokid AR glasses automatically open UDP listening port 5005, waiting for the user to click the UI button to input the front-end control platform's IP address and port (default 5006). The user enters the front-end control platform's IP address in the Rokid AR glasses' UI and clicks the "Start Broadcast" button, starting to broadcast finger flexion data at a frequency of 30fps, while simultaneously starting wrist posture broadcasting (port 5007). After receiving the first data packet of finger flexion data, the front-end control platform begins processing the command queue and driving the dexterous hand. Simultaneously, the force feedback thread sends real-time force sensing data to the AR glasses' port 5008. The AR glasses display the connection duration in the remote immersive view through a connection status display script and automatically counts the connection duration. The connection duration is determined through an event subscription mechanism. It records the in-game time of the last received frame and automatically switches to a no-connection or timeout state when no data is received for more than 5 seconds, and updates the text display of the connection duration.

[0086] The specific process of the front-end control platform and AR glasses is as follows: The user views the immersive image displayed by the AR glasses. After observing the target object, the user extends their right hand and naturally bends their fingers to simulate a grasping motion. The AR glasses calculate the finger bending angle data of 6 channels at 30fps, smooth it through low-pass filtering, and then package it into a 24-byte UDP data packet for transmission. The front-end control platform parses the UDP data, calculates the target position of the dexterous hand joint using an asymmetric power function mapping algorithm, and asynchronously sends serial port commands via a command queue. The dexterous hand joint driver moves to the target position of the dexterous hand joint. The front-end force feedback thread of the front-end control platform reads the force data from register 0x062E at 20Hz and sends it to the AR glasses via UDP (port 5008). The force data is visualized in real time at the fingertips using colors ranging from blue to red and changes in sphere size, allowing the user to judge the gripping force. The immersive image can be a 640×480 JPEG with a frame rate of 30fps.

[0087] Optionally, the front-end control platform of this invention adopts a multi-threaded architecture, which is responsible for receiving and parsing dexterous hand commands, executing commands and retrying protection, and sending force feedback and monitoring faults of the dexterous hand. The first thread is used to bind to UDP port 5006 and continuously receive data packets sent by AR glasses. Each data packet is 24 bytes long and is parsed into 6 integer angle values, corresponding to the angles of the 6 joints of the dexterous hand. The angles are converted into the target positions of the dexterous hand joints using an asymmetric power function mapping algorithm. The 6 mapped position values ​​are packaged into a joint target angle command and placed in the command queue. When the queue is full, the oldest command is automatically discarded, and the 5 newest commands are always retained to ensure that the remote control delay is minimized and the real-time performance is maintained.

[0088] The second thread is used to retrieve joint target angle instructions to be executed from the instruction queue. If the queue is empty, it will block and wait. When an instruction is available, it will be retrieved immediately. The second thread also supports global pause checks. After the dexterous hand is automatically reset, the second thread will pause sending new instructions to prevent interference with the reset process. The second thread also supports sending write register instructions via serial port (RS485, 115200bps), supports up to 3 automatic retries, and resends if the reply verification fails. The third thread reads real-time force sensing data from the six joints of the dexterous hand at a frequency of 20 Hz, packages the real-time force sensing data in big-endian order, and sends it to port 5008 of the AR glasses via UDP to achieve visualization of the real-time force sensing data. The third thread is also used to monitor the error register, automatically resetting it after three consecutive serious errors to ensure device and operational safety.

[0089] Figure 3 This is a schematic diagram of a remote control system for a dexterous hand provided in an embodiment of the present invention. Figure 3 As shown, the device includes: a real-time rendering module 310, an angle processing module 320, a command analysis module 330, and an operation synchronization module 340; wherein, The real-time rendering module 310 is used by the back-end processing platform to receive remote field images collected by the front-end control platform through a depth camera, and to render and display the remote immersive screen to the operating user in real time based on the remote field images. Angle processing module 320 is used by the back-end processing platform to collect joint pose data of the user's hand state in real time, generate finger bending data based on the joint pose data, and send the finger bending data to the front-end control platform. The instruction analysis module 330 is used by the front-end control platform to receive the finger bending data and generate a joint target angle instruction based on the finger bending data; The operation synchronization module 340 is used by the front-end control platform to synchronize the joint target angle command to the target manipulator in real time, so as to control the target manipulator to synchronize the user's hand state with the operation user.

[0090] The technical solution of this invention provides users with immersive visual feedback with natural depth perception by rendering and displaying a remote immersive scene in real time. This allows users to obtain a spatial positional relationship consistent with the remote scene, greatly enhancing the sense of presence and thus significantly improving the accuracy and efficiency of complex operations. The backend processing platform collects the joint pose data of the user's hand state, realizing wearable natural gesture interaction. The frontend control platform performs smooth and precise joint control of the fingers based on the curvature data, synchronizing with the remote dexterous hand with low latency. This enables the robotic hand to accurately and smoothly follow the user's natural gestures, solving the technical problem in the prior art of how to achieve stable, natural, and low-latency remote control of a dexterous hand without any burden on the user. This invention can effectively improve the immersive visual feedback at the remote scene, automatically synchronize the state of the hand and the robotic hand, use the most natural gestures for control, and provide a non-intrusive and natural interactive experience.

[0091] Optionally, the angle processing module 320 is specifically used for: The joint pose data includes the three-dimensional coordinates of at least one hand joint skeletal node; The backend processing platform collects the three-dimensional coordinates of at least one of the hand joint skeletal nodes in real time, based on the user's hand status. For each degree of freedom hand joint corresponding to the target robotic hand, the backend processing platform matches at least three three-dimensional coordinates of the corresponding hand joint bone nodes for the degree of freedom hand joint, and calculates the finger bending degree for the degree of freedom hand joint based on all the three-dimensional coordinates of the hand joint bone nodes. The backend processing platform performs a first-order exponential low-pass filter on all the finger curvatures to obtain the finger curvature data.

[0092] Optionally, the angle processing module 320 is further used for: The three-dimensional coordinates of the hand joint bone nodes include the three-dimensional coordinates of the fingertip joints, the three-dimensional coordinates of the joint midpoints, the three-dimensional coordinates of the wrist, or the three-dimensional coordinates of the palm base. If the degree of freedom hand joint is a degree of freedom finger joint, then the fingertip joint vector and the joint wrist vector are calculated based on the three-dimensional coordinates of the fingertip joint, the three-dimensional coordinates of the joint midpoint, and the three-dimensional coordinates of the wrist, respectively, and the finger bending degree is determined based on the fingertip joint vector and the joint wrist vector. If the hand joint of the degree of freedom is the thumb rotation joint, then the thumb rotation vector and the palm root vector are calculated based on the three-dimensional coordinates of the thumb tip joint, the three-dimensional coordinates of the index finger joint midpoint and the three-dimensional coordinates of the palm root, respectively, and the finger flexion is determined based on the thumb rotation vector and the palm root vector.

[0093] Optionally, the instruction analysis module 330 is specifically used for: The front-end control platform collects force sensing data and angle sensing data of the target manipulator; Under the constraint of a preset mapping range, the front-end control platform uses a preset power function mapping algorithm to map the finger bending data and determine the target position of the dexterous hand joint. The front-end control platform generates instructions based on the force sensing data, the angle sensing data, and the target position of the dexterous hand joint, and determines the target angle instruction of the joint.

[0094] Optionally, the instruction analysis module 330 is further configured to: The front-end control platform performs data verification on the force sensing data and the angle sensing data to determine reasonable force data and reasonable angle data. The front-end control platform performs sliding window mean filtering on the reasonable force data and the reasonable angle data to determine smooth force data and smooth angle data. The front-end control platform generates instructions based on the smoothed force data, the smoothed angle data, and the target position of the dexterous hand joint, and determines the target angle instruction of the joint.

[0095] Optionally, the device further includes a force display module; the force display module is specifically used for: The front-end control platform collects real-time force sensing data of the target manipulator and uploads the real-time force sensing data to the back-end processing platform; The backend processing platform identifies each preset force display node of the target robot in the remote field image; The backend processing platform displays the real-time force sensing data at each of the preset force display nodes in the remote immersive screen.

[0096] The force display module is also specifically used for: The backend processing platform determines the real-time force display attributes and real-time force values ​​corresponding to each preset force display node based on the real-time force sensing data. The backend processing platform matches the corresponding force visualization pattern based on the real-time force display attribute, and displays the force visualization pattern and real-time force value at each preset force display node in the remote immersive screen.

[0097] The dexterous hand remote control system provided in the embodiments of the present invention can execute the dexterous hand remote control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0098] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their patterns are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0099] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0100] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0101] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as dexterous hand remote control methods.

[0102] In some embodiments, the dexterous hand remote control method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the dexterous hand remote control method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the dexterous hand remote control method by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the patterns / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0108] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS servers, such as high management difficulty and weak business scalability.

[0109] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0110] This embodiment provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the program implements the steps of the dexterous hand remote control method provided in any embodiment of the present invention. The method includes: The back-end processing platform receives remote field images captured by the front-end control platform through a depth camera, and renders and displays the immersive remote scene to the user in real time based on the remote field images. The backend processing platform collects joint pose data of the user's hand state in real time, generates finger bending data based on the joint pose data, and sends the finger bending data to the frontend control platform. The front-end control platform receives the finger bending data and generates a joint target angle command based on the finger bending data; The front-end control platform synchronizes the joint target angle command to the target manipulator in real time, so as to control the target manipulator to synchronize with the user's hand state.

[0111] The computer storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0112] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0113] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0114] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computing device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0116] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0117] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for remote control of a dexterous hand, characterized in that, include: The back-end processing platform receives remote field images captured by the front-end control platform through a depth camera, and renders and displays the immersive remote scene to the user in real time based on the remote field images. The backend processing platform collects joint pose data of the user's hand state in real time, generates finger bending data based on the joint pose data, and sends the finger bending data to the frontend control platform. The front-end control platform receives the finger bending data and generates a joint target angle command based on the finger bending data; The front-end control platform synchronizes the joint target angle command to the target manipulator in real time, so as to control the target manipulator to synchronize with the user's hand state.

2. The method according to claim 1, characterized in that, The joint pose data includes the three-dimensional coordinates of at least one hand joint skeletal node; the back-end processing platform collects the joint pose data of the user's hand state in real time, generates finger flexion data based on the joint pose data, and sends the finger flexion data to the front-end control platform, including: The backend processing platform collects the three-dimensional coordinates of at least one of the hand joint skeletal nodes in real time, based on the user's hand status. For each degree of freedom hand joint corresponding to the target robotic hand, the backend processing platform matches at least three three-dimensional coordinates of the corresponding hand joint bone nodes for the degree of freedom hand joint, and calculates the finger bending degree for the degree of freedom hand joint based on all the three-dimensional coordinates of the hand joint bone nodes. The backend processing platform performs a first-order exponential low-pass filter on all the finger curvatures to obtain the finger curvature data.

3. The method according to claim 2, characterized in that, The three-dimensional coordinates of the hand joint skeletal nodes include the three-dimensional coordinates of the fingertip joints, the three-dimensional coordinates of the joint midpoints, the three-dimensional coordinates of the wrist, or the three-dimensional coordinates of the palm base; the calculation of finger flexion based on all the three-dimensional coordinates of the hand joint skeletal nodes for the degree of freedom of the hand joint includes: If the degree of freedom hand joint is a degree of freedom finger joint, then the fingertip joint vector and the joint wrist vector are calculated based on the three-dimensional coordinates of the fingertip joint, the three-dimensional coordinates of the joint midpoint, and the three-dimensional coordinates of the wrist, respectively, and the finger bending degree is determined based on the fingertip joint vector and the joint wrist vector. If the hand joint of the degree of freedom is the thumb rotation joint, then the thumb rotation vector and the palm root vector are calculated based on the three-dimensional coordinates of the thumb tip joint, the three-dimensional coordinates of the index finger joint midpoint, and the three-dimensional coordinates of the palm root, respectively, and the finger flexion is determined based on the thumb rotation vector and the palm root vector.

4. The method according to claim 1, characterized in that, The step of generating the joint target angle command based on the finger curvature data includes: The front-end control platform collects force sensing data and angle sensing data of the target manipulator; Under the constraint of a preset mapping range, the front-end control platform uses a preset power function mapping algorithm to map the finger bending data and determine the target position of the dexterous hand joint. The front-end control platform generates instructions based on the force sensing data, the angle sensing data, and the target position of the dexterous hand joint, and determines the target angle instruction of the joint.

5. The method according to claim 4, characterized in that, The front-end control platform generates commands based on the force sensing data, the angle sensing data, and the target position of the dexterous hand joint, and determines the target joint angle command, including: The front-end control platform performs data verification on the force sensing data and the angle sensing data to determine reasonable force data and reasonable angle data. The front-end control platform performs sliding window mean filtering on the reasonable force data and the reasonable angle data to determine smooth force data and smooth angle data. The front-end control platform generates instructions based on the smoothed force data, the smoothed angle data, and the target position of the dexterous hand joint, and determines the target angle instruction of the joint.

6. The method according to claim 1, characterized in that, Also includes: The front-end control platform collects real-time force sensing data of the target manipulator and uploads the real-time force sensing data to the back-end processing platform; The backend processing platform identifies each preset force display node of the target robot in the remote field image; The backend processing platform displays the real-time force sensing data at each of the preset force display nodes in the remote immersive screen.

7. The method according to claim 6, characterized in that, The backend processing platform displays the real-time force sensing data at each of the preset force display nodes in the remote immersive image, including: The backend processing platform determines the real-time force display attributes and real-time force values ​​corresponding to each preset force display node based on the real-time force sensing data. The backend processing platform matches the corresponding force visualization pattern based on the real-time force display attribute, and displays the force visualization pattern and real-time force value at each preset force display node in the remote immersive screen.

8. A remote control system for a dexterous hand, characterized in that, include: The real-time rendering module is used by the back-end processing platform to receive remote field images captured by the front-end control platform through a depth camera, and to render and display the remote immersive screen to the user in real time based on the remote field images. An angle processing module is used by the back-end processing platform to collect joint pose data of the user's hand state in real time, generate finger bending data based on the joint pose data, and send the finger bending data to the front-end control platform. The instruction analysis module is used by the front-end control platform to receive the finger bending data and generate a joint target angle instruction based on the finger bending data; The operation synchronization module is used by the front-end control platform to synchronize the joint target angle command to the target manipulator in real time, so as to control the target manipulator to synchronize the user's hand state with the operation user.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the dexterous hand remote control method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the dexterous hand remote control method according to any one of claims 1-7.

Images