Real-time gesture mirroring method, device and equipment applied to dexterous hand

Abstract

The embodiment of the application provides a real-time gesture mirroring method applied to a dexterous hand, which can be applied to the technical field of dexterous hands and the technical field of remote operation. The real-time gesture mirroring method applied to the dexterous hand comprises the following steps: generating normalized vector data of a corresponding dimension based on target three-dimensional key point data corresponding to target hand identification data output by a hand key point detection model; performing weighted moving average smoothing processing on the normalized vector data according to a pre-generated joint differentiation index to generate corresponding smooth vector data; and generating a hypertext transfer protocol request issued to a preset control host computer according to the smooth vector data and a target object received through a preset device discovery interface, so as to realize real-time gesture mirroring applied to the dexterous hand. The embodiment of the application also provides a real-time gesture mirroring device applied to the dexterous hand, a device, a storage medium and a program product.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to the cross-disciplinary field of computer vision technology, human-computer interaction (HCI) technology and robot teleoperation technology, and more specifically to a method, device, equipment, medium and product for real-time gesture mirroring applied to dexterous hands. Background Technology

[0002] As bionic dexterous hands are gradually implemented in industrial flexible assembly, home service robots, special operations, and embodied intelligent data collection, a key issue has become how to enable operators to conveniently, naturally, and with low barriers to entry drive the dexterous hand. The driving requirements necessitate that the system be able to reproduce the operator's hand posture (including thumb-to-palm) in real time without relying on wearable hardware, and then send this information to the dexterous hand controller with minimal latency. Simultaneously, the system must be able to reuse the same visual algorithm across different models and variants, maintain robustness during startup under engineering realities such as when the dexterous hand is not powered on, has a delayed startup, or is hot-swapped in the field, and correctly select the target hand to avoid miscontrol when both of the operator's hands are simultaneously visible in the camera's field of view. Summary of the Invention

[0003] In view of at least one of the above problems, embodiments of the present invention aim to provide a method, apparatus, and device for real-time gesture mirroring applied to dexterous hands.

[0004] One aspect of the present invention provides a real-time gesture mirroring method for dexterous hands, comprising: generating normalized vector data of the corresponding dimension based on the target three-dimensional keypoint data corresponding to the target hand identification data output by a hand keypoint detection model; performing weighted moving average smoothing on the normalized vector data according to a pre-generated joint differentiation index to generate corresponding smoothed vector data; generating a hypertext transfer protocol request to be sent to a preset control host computer based on the smoothed vector data and the target object received through a preset device discovery interface, so as to realize real-time gesture mirroring for dexterous hands; wherein the original geometric angle vector data corresponding to the normalized vector data includes vector data that sequentially correspond to the preset joints of the dexterous hand, namely, thumb flexion angle, thumb deflection geometric measurement, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, little finger flexion angle, index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle.

[0005] According to an embodiment of the present invention, before performing weighted moving average smoothing on the normalized vector data based on the pre-generated joint differentiation index to generate corresponding smoothed vector data, the method further includes: generating an independent exponential moving average coefficient as a joint differentiation index for each joint of the normalized vector data.

[0006] According to an embodiment of the present invention, in generating corresponding smoothed vector data by performing weighted moving average smoothing on normalized vector data based on a pre-generated joint differentiation index, the method includes: updating the current frame smoothing value based on the normalized value of the current frame of the joint corresponding to the normalized vector data, the smoothing value of the previous frame, and the corresponding joint differentiation index; wherein the current frame smoothing value is used to constitute the smoothed vector data.

[0007] According to an embodiment of the present invention, before generating a Hypertext Transfer Protocol request to be sent to a preset control host computer based on smoothed vector data and target objects received through a preset device discovery interface, the method further includes: initiating a polling request to the target device interface of the preset control host computer at a preset polling interval based on a preset Hypertext Transfer Protocol, and obtaining corresponding target device list data; performing field filtering and matching on each device object in the target device array of the target device list data based on pre-configured target field matching conditions, and reading the encapsulation field data of the corresponding device; performing field encapsulation on the encapsulation field data to generate a target object to be sent to the preset device discovery interface.

[0008] According to an embodiment of the present invention, in generating a hypertext transfer protocol request to be sent to a preset control host computer based on smoothed vector data and a target object received through a preset device discovery interface, the process includes: mapping each logical value of the smoothed vector data component by component to an integer logical value corresponding to a preset program interface of the preset control host computer; and assembling the integer logical values ​​into a first dictionary data and a second dictionary data according to semantic groups; wherein the first dictionary data includes integer logical values ​​related to the thumb flexion angle, thumb deflection geometry, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, and little finger flexion angle; and the second dictionary data includes integer logical values ​​related to the index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle.

[0009] According to an embodiment of the present invention, in generating a hypertext transfer protocol request to be sent to a preset control host computer based on smoothed vector data and a target object received through a preset device discovery interface, the process further includes: generating a hypertext transfer protocol request based on a preset control host computer that performs throttling at the minimum sending interval, using a preset object identifier format to combine first dictionary data, second dictionary data, and the target object.

[0010] Another aspect of the present invention provides a real-time gesture mirroring device for a dexterous hand, comprising a vector data generation module, a differential smoothing module, and a request generation module. The vector data generation module generates normalized vector data of the corresponding dimension based on the target hand identifier data corresponding to the target three-dimensional keypoint data output by a hand keypoint detection model. The differential smoothing module performs weighted moving average smoothing on the normalized vector data according to a pre-generated joint differential index to generate corresponding smoothed vector data. The request generation module generates a hypertext transfer protocol request to a preset control host computer based on the smoothed vector data and the target object received through a preset device discovery interface, thereby realizing real-time gesture mirroring for a dexterous hand. The original geometric angle vector data corresponding to the normalized vector data includes vector data sequentially corresponding to the preset joints of the dexterous hand, including thumb flexion angle, thumb deflection geometric measurement, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, little finger flexion angle, index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle.

[0011] Another aspect of the present invention provides an electronic device including one or more processors and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above-described real-time gesture mirroring method applied to a dexterous hand.

[0012] Another aspect of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described real-time gesture mirroring method applied to a dexterous hand.

[0013] Another aspect of the present invention provides a computer program product including a computer program that, when executed by a processor, implements the above-described real-time gesture mirroring method applied to a dexterous hand.

[0014] The real-time gesture mirroring method for dexterous hands provided in this invention can at least partially solve at least one of the above-mentioned defects in dexterous hand teleoperation schemes in related technologies, such as high hardware threshold, non-standard semantic reconstruction of key points to joint space, inability of filtering coefficients to balance response and smoothness, tight coupling between vision and control, strict constraints on startup sequence, and confusion of control objects with both hands. Therefore, it can achieve at least one of the following technical effects:

[0015] (1) A filtering method is proposed that configures the EMA smoothing coefficient according to the joint differentiation. Compared with the existing scheme of using a single coefficient filter for all joints as described in Method 3 (corresponding to defect 3), by configuring a smaller update weight for the distal joints such as little finger flexion and little finger abduction, and configuring a larger update weight for the active joints such as thumb and index finger, the "active joint responsiveness" and "distal joint steady-state smoothness" are achieved simultaneously within a single EMA structure.

[0016] (2) A three-stage decoupled dexterity hand remote operating system architecture of "visual perception + REST API + execution control" is proposed. Compared with the tightly coupled monolithic scheme described in Method 4 (corresponding to defect 4), the dependence of the camera process on the specific model, variant, CAN protocol field and CAN identification scheme of the dexterity hand is unified and converged into the dependence on the single HTTP POST interface exposed by the Dashboard. The same set of visual algorithms can be adapted to multiple dexterity hand models and variants. The visual algorithm and control stack can evolve independently. Abnormalities of the camera process do not affect the control link.

[0017] (3) An automatic device discovery and hot-plug startup method based on polling of the Dashboard device discovery interface and matching of model / variant field filtering is proposed. Compared with the "enumerate devices once at startup or exit" scheme described in Method 5 (corresponding to the startup order constraint part of defect 5), the startup order constraint is eliminated, and the dexterity hand is supported for delayed power-on and field hot-plugging. The polling and field filtering combined design makes it possible to add a new dexterity hand variant without modifying the visual algorithm code, only needing to register a new variant on the Dashboard.

[0018] It should be understood that the above general description and the following specific embodiments are merely exemplary and illustrative, and do not limit the scope of the invention. Attached Figure Description

[0019] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0020] Figure 1 The illustration schematically depicts an application scenario of a data processing method for real-time gesture mirroring of a dexterous hand and a real-time gesture mirroring method, apparatus, device, medium, and program product for a dexterous hand according to embodiments of the present invention.

[0021] Figure 2A A flowchart illustrating a data processing method for real-time gesture mirroring of a dexterous hand according to an embodiment of the present invention is shown.

[0022] Figure 2BA flowchart illustrating a real-time gesture mirroring method for dexterous hands according to an embodiment of the present invention is shown schematically.

[0023] Figure 3 The diagram illustrates the overall system architecture of a real-time gesture mirroring method for dexterous hands based on monocular vision, which is a data processing method for real-time gesture mirroring of dexterous hands according to an embodiment of the present invention.

[0024] Figure 4 The illustration shows a topological diagram of 21 hand key points output by a hand key point detection model (MediaPipe Hands) of the data processing method for real-time gesture mirroring of a dexterous hand and the real-time gesture mirroring method for a dexterous hand according to an embodiment of the present invention.

[0025] Figure 5A The diagram illustrates a 10-DOF geometric solution principle (corresponding to the thumb deflection normal vector measurement) of a data processing method for real-time gesture mirroring of a dexterous hand and a real-time gesture mirroring method for a dexterous hand according to an embodiment of the present invention.

[0026] Figure 5B The diagram illustrates a 10-DOF geometric solution principle (corresponding to the three-point angle primitive) of a data processing method for real-time gesture mirroring of a dexterous hand and a real-time gesture mirroring method for a dexterous hand according to an embodiment of the present invention.

[0027] Figure 6 The diagram illustrates the configuration of joint-specific EMA coefficients for the data processing method and the real-time gesture mirroring method for dexterous hands according to embodiments of the present invention (f=0.2 for the little finger joint and f=0.4 for the others).

[0028] Figure 7 The illustration schematically shows an automatic device discovery state diagram based on device discovery interface polling and model / variant field filtering for a data processing method for real-time gesture mirroring of a dexterous hand and a real-time gesture mirroring method for a dexterous hand according to an embodiment of the present invention.

[0029] Figure 8 The diagram illustrates an end-to-end data flow timing diagram of a data processing method for real-time gesture mirroring of a dexterous hand and a real-time gesture mirroring method for a dexterous hand according to an embodiment of the present invention (corresponding to the gesture mirroring process of "camera acquisition → MediaPipe → geometric calculation and smoothing → HTTP POST → CAN → dexterous hand").

[0030] Figure 9AThis schematically illustrates a structural block diagram of a data processing apparatus for real-time gesture mirroring of a dexterous hand according to an embodiment of the present invention;

[0031] Figure 9B A schematic diagram illustrates a structural block diagram of a real-time gesture mirroring device for a dexterous hand according to an embodiment of the present invention; and

[0032] Figure 10 The diagram illustrates an electronic device suitable for implementing a data processing method for real-time gesture mirroring of a dexterous hand and a real-time gesture mirroring method for a dexterous hand, according to an embodiment of the present invention.

[0033] In this invention, all accompanying drawings do not rely on color information; different modules are distinguished by line type (solid line, dashed line, dotted line, double line) and fill pattern (no fill, diagonal fill, intersecting diagonal fill, dot matrix fill) to aid understanding by those skilled in the art, rather than to demonstrate differences in technical solutions; arrow endpoints all fall on the frame edge anchor points (top, bottom, left, right) and do not cross other frames; if polar coordinates or circular coordinates are involved, the increasing direction and wrapping point values ​​are explicitly marked; the accompanying drawings of this invention are all rectangular coordinates or topological diagrams and do not involve polar coordinates.

[0034] The accompanying drawings mentioned above are part of the specification of embodiments of the present invention, illustrating exemplary embodiments of the invention. The drawings, together with the description in the specification, serve to illustrate the principles of the embodiments of the present invention. It should be understood that the above general description with reference to the drawings and the following detailed description are merely exemplary and illustrative, and do not limit the scope of the invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the spirit of the contents disclosed in the present invention will be clearly explained below with reference to the accompanying drawings and detailed description. After understanding the embodiments of the present invention, any person skilled in the art can make changes and modifications based on the technology taught in the present invention without departing from the spirit and scope of the present invention.

[0036] The illustrative embodiments and descriptions of the present invention are used to explain the invention, but are not intended to limit the invention. Furthermore, elements / components using the same or similar reference numerals in the drawings and embodiments are used to represent the same or similar parts.

[0037] The terms "first," "second," etc., used in this invention do not specifically refer to any order or sequence, nor are they intended to limit the invention; they are merely used to distinguish elements or operations described using the same technical terms.

[0038] The directional terms used in this invention, such as up, down, left, right, front, or back, are merely for reference to the accompanying drawings. Therefore, the directional terms used are for illustrative purposes and not for limiting the scope of this invention.

[0039] The terms “comprising,” “including,” “having,” “containing,” etc., used in this invention are all open-ended terms, meaning that they include but are not limited to.

[0040] The term "and / or" as used in this invention includes any or all combinations of the things mentioned.

[0041] In this invention, "multiple" includes "two" and "more than two"; in this invention, "multiple groups" includes "two groups" and "more than two groups".

[0042] The terms "approximately," "about," etc., used in this invention are intended to modify any quantity or error that may vary slightly, but these slight variations or errors do not change the essence of the quantity or error. Generally, the range of slight variations or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0043] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0044] When expressions such as "at least one of A, B, and C" are used, they should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). When expressions such as "at least one of A, B, or C" are used, they should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). A person skilled in the art should also understand that any conjunction and / or phrase that substantially arbitrarily indicates two or more optional items, whether in the specification, claims, or drawings, should be understood to indicate the possibility of including one of these items, either of these items, or both items. For example, the phrase “A or B” should be understood as including the possibility of “A” or “B”, or “A and B”.

[0045] In the field of dexterous hand drive technology, the following five traditional implementation methods can be mainly included:

[0046] Traditional Method 1: Dexterous Hand Telemanipulation Based on Wearable Hardware Sensors. This method is characterized by the operator wearing a data glove, an inertial measurement unit (IMU) array, an electromagnetic tracker, or optical motion capture markers. The hardware directly outputs the attitude or position information of each finger joint, which is then mapped to the dexterous hand joint space using a manufacturer-provided calibration process. Typical examples of this method include telemanipulation schemes based on commercial data gloves (such as CyberGlove) and hand attitude reconstruction schemes based on multi-IMU finger ring arrays.

[0047] Traditional Method Two: A scheme based on MediaPipe's 21 3D hand key points combined with heuristic mapping. This method is characterized by: capturing RGB video frames from a camera and inputting them into the MediaPipe Hands model to obtain 21 3D hand key points (including relative depth z); then converting heuristic quantities such as fingertip-to-palm distance ratio, axis alignment angle, and 2D projection angle directly or through simple linear mapping into dexterous hand joint actuation quantities; finally, issuing joint drive commands via the vendor's Software Development Kit (SDK) or directly through the Controller Area Network (CAN) bus interface. A typical example of this method is the "fingertip distance ratio + 2D projection angle" mapping approach used in several open-source MediaPipe-driven robotic arm repositories and related papers.

[0048] Traditional Method 3: A scheme using a uniform coefficient filter for all keypoints. This method is characterized by applying a single-coefficient exponential moving average (EMA), a uniform parameter Kalman filter, or a uniform window moving average filter to all 21 keypoints output by the MediaPipe (or all downstream joint actuators). It does not configure independent filtering parameters for each joint based on the differences in detection stability among different keypoints. This method is commonly seen in the implementation of directly applying a single-parameter One-Euro Filter or EMA to all joint actuators.

[0049] Traditional Approach 4: A tightly coupled monolithic solution that places camera acquisition, model inference, joint calculation, CAN communication, and device management in the same process. This approach is characterized by: the vision algorithm directly calling the dexterous hand vendor's SDK or directly reading and writing to the CAN bus; the same process handling frame acquisition, model inference, joint calculation, protocol encapsulation, CAN message transmission and reception, and device lifecycle management; vision algorithm upgrades require simultaneous recompilation and deployment of the underlying control stack; camera crashes or model anomalies will affect the control chain via the shared process address space.

[0050] Traditional Method 5: A scheme that enumerates all devices at startup and exits otherwise, and selects the first hand as the control source according to the detection order. The characteristics of this method are: when the process starts, it enumerates all dexterous hand devices under the CAN bus or host computer management service at startup, and exits if no target dexterous hand is matched; when both hands of the operator appear in the camera's field of view at the same time, the first hand (multi_hand_landmarks[0]) is selected as the control source according to the output order of the MediaPipe detection results, and the left and right hand classification fields of the hands output by MediaPipe are not read. The above method is commonly used in implementations that only aim to "work on demand".

[0051] The traditional driving methods for the above five dexterous hands have at least the following drawbacks:

[0052] Defect 1: The wearable hardware sensor-based solution described in Method 1 suffers from drawbacks such as high hardware costs, cumbersome wearing process, the need for calibration, poor hygiene, and reliance on dedicated space. The wearable device is significantly more expensive than a monocular RGB camera, and requires recalibration for each operator before each use; hygiene and disinfection must be considered in shared scenarios; and the electromagnetic / optical motion capture solution also requires the placement of transmitters and multi-view cameras in a dedicated space. These hardware barriers fundamentally limit the scalability of the solution for home services, on-site debugging, and embodied intelligent data collection.

[0053] Defect 2: The scheme described in Method 2, based on 21 3D keypoints of MediaPipe and heuristic mapping, suffers from irregular semantic reconstruction of the 21 3D keypoints to the 10 degrees of freedom of the dexterous hand and difficulty in calculating the thumb-opposition degree of freedom. The heuristic mapping often uses fingertip distance and axis alignment angle as surrogate quantities, failing to fully utilize the geometric information carried by the relative depth z in the 3D keypoints; in particular, the thumb yaw degree of freedom is almost unobservable under two-dimensional projection, and the scheme usually ignores or roughly estimates this degree of freedom, thus limiting the dexterous hand's grasping configuration (typically manifested as the inability to complete the "oppositional grasping" posture with the thumb and little finger facing each other).

[0054] Defect 3: The scheme described in Method 3, which uses a uniform coefficient filter for all key points, suffers from the inability to simultaneously address the issues of slow active joint response and persistent end-joint jitter. The MediaPipe's end-joints (especially the little finger PIP and little finger MCP) exhibit significantly higher noise levels than other fingers in self-occlusion, rapid movement, or low-light conditions. When Method 3 uses a single coefficient filter, increasing the update weight results in significant residual end-joint jitter, while decreasing the update weight leads to noticeable sluggishness in the active joint (thumb, index finger) response. These two problems cannot be simultaneously resolved with a single coefficient. This residual jitter further causes repeated micro-movements in the dexterous hand motor, exacerbating mechanical wear.

[0055] Defect 4: The tightly coupled monolithic solution described in Method 4 suffers from drawbacks such as the vision module's dependence on a specific dexterity hand SDK, difficulty in reusing across different models, coupled upgrades of the vision algorithm and control stack, and the impact of camera or model malfunctions on control link security. In Method 4, vision algorithm upgrades (such as MediaPipe version updates or feature model replacements) require simultaneous recompilation of the underlying control stack; the same vision algorithm needs to have its device management and distribution code rewritten when adapted to different dexterity hand models (such as different joint counts or different CAN identification schemes); camera crashes or model inference anomalies will directly affect CAN message transmission and reception through shared process address spaces, posing a fundamental obstacle to the engineering requirement of decoupling "the vision subsystem failing while the control subsystem continues to maintain a safe posture."

[0056] Defect 5: The "one-time enumeration of devices at startup + selection of the first hand according to the detection order" scheme described in Method 5 suffers from two major defects: strict startup order constraints, lack of support for hot-swapping, and confusion of the controlled object when both hands are present. First, this method requires the dexterous hand to be powered on and online before the vision process starts; otherwise, the vision process will exit directly. During on-site debugging or when the dexterous hand is powered on late, the vision process needs to be restarted, which is not production-friendly. Furthermore, it cannot automatically recover if the dexterous hand experiences an unexpected power outage and subsequent power-on. Second, this method selects the first hand according to the MediaPipe output order. When the operator places the other hand in the camera's field of view (e.g., supporting the dexterous hand with the left hand while making gestures with the right), the controlled object may not match the operator's expectations, and the selection result will jitter with posture changes.

[0057] In view of the above-mentioned defects in the driving technology of traditional dexterous hands, the present invention provides a data processing method and apparatus for real-time gesture mirroring of dexterous hands as in Embodiment 1, and a real-time gesture mirroring method and apparatus for dexterous hands as in Embodiment 2, in order to solve at least one of the above-mentioned technical problems and provide a real-time gesture mirroring process that is more cost-effective, more efficient and more accurate.

[0058] like Figures 3-8 As shown, the methods provided in Embodiments 1 and 2 of the present invention are specifically executed collaboratively by a visual perception process (hereinafter referred to as the camera process) deployed on the operator-side host computer and an execution control process (hereinafter referred to as the Dashboard process) deployed on the dexterous hand control-side host computer:

[0059] (1) The camera process is responsible for video acquisition, key point detection, target hand screening, ten-degree-of-freedom geometric calculation, joint-by-joint normalization, joint-by-joint differential EMA smoothing and logical value transmission via Hypertext Transfer Protocol (HTTP);

[0060] (2) The Dashboard process is responsible for exposing the device discovery interface to the outside world, managing the dexterous hand device, converting the joint logic value to the CAN message protocol and transmitting and receiving the CAN bus.

[0061] The two processes described above are decoupled via a REST API, and the camera process is completely unaware of the specific model, variant, or underlying CAN protocol of the dexterous hand. Therefore, the methods provided in Embodiments 1 and 2 of this invention can at least implement the operation process including the following steps S1 to S9.

[0062] Among them, such as Figure 3 and Figure 8 As shown in Table 1 below, for ease of understanding, the relevant modules, entities, and interfaces are listed in detail:

[0063] physical entities Monocular RGB Camera The example is a standard Universal Serial Bus (USB) camera, which does not require a depth sensor. S1 physical entities Dexterous Hand The embodiment is a bionic dexterous hand with 10 degrees of freedom, which can be in different models / variants. S7, S8 physical entities CAN bus Connecting the Dashboard process host to the dexterous hand's controller LAN S7 end segment (only reflected in the Dashboard process) Computational Entity Vision Host (Visual Perception Host) The computer running the camera process does not require a graphics processing unit (GPU). S1–S7, S9 Computational Entity Control Host (ControlHost) Computer running the Dashboard process S7, S8 Software Module Video capture module Video frame acquisition and horizontal flipping were performed using the video capture interface of the Open Source Computer Vision Library (OpenCV). S1 Software Module Landmark Module Instantiate MediaPipe Hands and perform detection and tracking for each frame. S2 Software Module HandednessSelector module Filter the target hand based on the MediaPipe handedness field configured by the user. S3 Software Module Geometry Solver module Solving 10 primitive geometric angles using pure vector geometry methods S4 Software Module Normalizer module Linear normalization and saturation trimming based on physiological range of motion joint by joint S5 Software Module Per-joint EMA Filter Smoothing is performed on each joint using an independent EMA coefficient. S6 Software Module HTTPDispatcher module Linear range mapping from 0–100 to 0–255, beat-based throttling, and HTTP POST delivery. S7 Software Module Device Discovery Module Periodically poll the Dashboard device discovery interface and filter matches by model / variant. S8 Software Module Preview Module (optional) OpenCV window overlay keypoint connection relationships; can be turned off S9 External Interface GET / api / hand / devices The Dashboard exposes a device list query interface; it outputs a device array containing fields such as model, variant, interface, and side. S8 External Interface POST / api / hand / joint / logical The Dashboard exposes an interface for distributing logical values; the input is {logical_values:{basic,swing},targets:[target]}; the output is the distribution result. S7

[0064] Table 1

[0065] In Table 1, "target hand" refers to the left or right hand specified by the user in the configuration; "target" refers to the dexterity hand instance description object obtained in step S8, containing fields such as model, variant, interface, and side, i.e., the target object. Steps S1–S7 and S9 within the camera process are connected via function calls and shared memory; the camera process and the Dashboard process communicate only through the two REST APIs mentioned above.

[0066] Specifically, such as Figure 3 As shown in the overall system architecture diagram of the methods in Embodiments 1 and 2 of this invention, the left side represents the visual perception host computer, which can include seven software modules: a video acquisition module, a key point detection module, a target hand screening module, a geometric calculation module, a normalization module, a differential smoothing module, and a distribution module. The solid rectangle on the right side represents the control host computer, which can include a Dashboard process and its subordinate device management, protocol conversion, and CAN message transmission and reception modules. The CAN message transmission and reception module distributes messages to the dexterous hand physical entity. In addition, the left and right host computers are connected by two REST interfaces marked with double-line single arrows: Hypertext Transfer Protocol requests (such as POST / api / hand / joint / logical) and device discovery polling (GET / api / hand / devices).

[0067] Among them, such as Figure 3 and Figure 8As shown, the legend indicates that "solid rectangles represent software modules, double-lined single arrows represent the data flow direction of REST interfaces, solid-lined single arrows represent in-process function calls, and dashed rectangles represent optional modules"; the coordinate direction is indicated by arrows from left to right, from "left operator - right dexterous hand", which does not depend on color information.

[0068] Figure 1 The illustration schematically depicts an application scenario of a data processing method for real-time gesture mirroring of a dexterous hand and a method, apparatus, device, medium, and program product for real-time gesture mirroring of a dexterous hand according to embodiments of the present invention.

[0069] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0070] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0071] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0072] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0073] It should be noted that the method provided in the embodiments of the present invention can generally be executed by server 105. Correspondingly, the apparatus provided in the embodiments of the present invention can generally be located in server 105. The method provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the apparatus provided in the embodiments of the present invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0074] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0075] Example 1:

[0076] The following will be based on Figure 1 The described scene, through Figures 2A to 9B The data processing method for real-time gesture mirroring of dexterous hands according to the disclosed embodiment 1 will be described in detail.

[0077] like Figure 2A As shown, one aspect of an embodiment of the present invention provides a data processing method for real-time gesture mirroring of a dexterous hand, comprising operations S211 to S214.

[0078] In operation S211, video frame data that has undergone image preprocessing is acquired;

[0079] In operation S212, target hand identification data of video frame data is obtained based on the hand key point detection model instantiated with pre-parameters;

[0080] In operation S213, target 3D keypoint data is matched based on target hand identifier data; and

[0081] In operation S214, the target three-dimensional key point data is subjected to multi-degree-of-freedom pure vector geometry calculation based on the preset angle primitive, and the original geometric angle vector data of the corresponding degree of freedom is output to realize the real-time gesture mirroring of the dexterous hand.

[0082] The original geometric angle vector data includes vector data corresponding sequentially to the preset joints of the dexterous hand, such as the thumb flexion angle, thumb deflection geometric measurement, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, little finger flexion angle, index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle.

[0083] like Figures 2A to 9AAs shown, according to an embodiment of the present invention, the process of obtaining video frame data after image preprocessing in operation S211 includes:

[0084] Open the preset camera index based on the preset video capture interface, and call and read each video frame in the first preset color space frame by frame;

[0085] Perform horizontal axis flipping processing on each video frame based on a preset image horizontal flipping operator to generate corresponding mirror view data;

[0086] The mirror view data is converted to a second preset color space to generate video frame data that conforms to the preset data input format of the hand key point detection model.

[0087] like Figures 3-8 As shown, step S1 mainly involves video acquisition and image preprocessing, specifically triggered by the video acquisition module in the main loop of the camera process on a frame-by-frame basis (typically the camera's own exposure rhythm, such as 30 frames per second); the upstream dependency is the RGB video stream driven by the camera hardware, and the downstream is consumed by keypoint detection in step S2. Step S1 includes:

[0088] (a) Camera opening and frame acquisition: Open the preset camera index (such as the camera index specified by the user, which can be specified by default) with default parameters through the preset video capture interface (such as the video capture interface of OpenCV); call the interface frame by frame to obtain a video frame in the first preset color space (such as blue-green-red, i.e., BGR color space) as each video frame of the present invention.

[0089] (b) Horizontal flip: Call the preset image horizontal flip operator for each frame (i.e. video frame) (in this example, the flip function of OpenCV is used to flip the horizontal axis) to construct a mirror view (i.e. mirror view data) that is consistent with the operator's visual intuition, so that when the operator moves his hand to the right, the image of the hand in the camera window also moves to the right, avoiding the operator's reliance on reverse mental mapping operation.

[0090] (c) Color space conversion: Convert the mirror view data of BGR frames that conform to the first preset color space into video frame data of the second preset color space. The second preset color space can be the Red-Green-Blue (RGB) color space, which is the frame image input format (i.e., the preset data input format) agreed upon by the downstream hand key point detection model (such as the MediaPipe Hands model).

[0091] Therefore, as Figure 3 and Figure 8The output of step S1 is a single-frame RGB image as video frame data. Its technical advantage lies in the fact that, compared to the traditional method four, which tightly couples acquisition and control into a single unit, the video frame data acquisition process in this embodiment only handles image preparation and does not involve any dexterous hand protocol or hardware abstraction; furthermore, by eliminating "mirror misalignment" through horizontal flipping, it provides a visually intuitive and consistent input premise compared to the "control object selection ambiguity" present in the traditional method five.

[0092] like Figures 2A to 9A As shown, according to an embodiment of the present invention, before operating S212 to obtain target hand identification data of video frame data based on the pre-parameter instantiated hand key point detection model, the method further includes:

[0093] Based on the instantiated parameters, including video stream mode parameters, parameters for detecting at most two hands, initial detection confidence threshold, and follow-up frame confidence threshold, the preset key point detection model is instantiated to generate a hand key point detection model.

[0094] like Figures 3-8 As shown, step S2 mainly involves the 3D detection and confidence filtering of hand keypoints using a hand keypoint detection model (such as MediaPipe). First, the model instantiation process for the hand keypoint detection model is performed. This is mainly done by instantiating a preset keypoint detection model using parameters such as video stream mode (static_image_mode=False), maximum number of hands to be detected (max_num_hands=2), initial detection confidence threshold (min_detection_confidence=0.7), and tracking frame confidence threshold (min_tracking_confidence=0.6). This process can be defined as detecting at most two hands in video stream mode, while simultaneously determining the confidence thresholds for the initial detection and tracking frame.

[0095] The preset keypoint detection model can be the initial model architecture for the hand keypoint detection model, which can be implemented based on the MediaPipe Hands model. The MediaPipe Hands model can be replaced with other detectors capable of outputting 3D hand keypoints (or capable of completing them into 3D keypoints), including but not limited to: OpenPoseHand, HRNet-Hand, hand keypoint models under the MMPose framework, self-trained "RGB to 3D hand keypoint" convolutional neural networks (CNNs) or Transformer networks. Alternatively, 3D keypoints can be directly obtained using an RGB-Depth camera; or keypoints can be obtained by geometric clustering after obtaining a sparse 3D point cloud of the hand using millimeter-wave radar. The boundary of the substitution is that the detector output must still be a sequence of 3D keypoints with fixed topological numbers (containing at least 12 keypoints in total, including WRIST, four-finger MCP / PIP / TIP and THUMB_MCP / IP / TIP). Therefore, as long as the core property of "constructing hand bone connection vectors and palm plane based on 3D keypoints with fixed topological numbers" is maintained, it can be regarded as an equivalent substitution for the keypoint detection model of the present invention.

[0096] like Figures 2A to 9A As shown, according to an embodiment of the present invention, the target hand identification data in the operation S212, which acquires video frame data based on the pre-parameter instantiated hand key point detection model, includes:

[0097] By calling the preset processing interface of the hand keypoint detection model, the video frame data is detected based on the parameters of at most two hands to determine the three-dimensional coordinate data and chirality classification label data of the hand keypoints.

[0098] Based on the initial detection confidence threshold, the target hand identification data is output, which includes the corresponding three-dimensional coordinate data of key hand points and the chiral classification label data.

[0099] like Figure 3 and Figure 8 As shown, 21 keypoint 3D detection is performed in step S2, specifically triggered synchronously by the keypoint detection module when each video frame data is output in step S1. Therefore, in step S2, its upstream dependency is the RGB frame video frame data output in step S1, and its downstream is further executed or processed by the target hand screening in step S3. In this embodiment of the invention, step S2 may include:

[0100] (b) Per-frame detection: For each frame (video frame data), the preset processing interface of the hand keypoint detection model is called to obtain the detection results of up to two hands. Each detection result may contain: 3D coordinates (x, y, z) of 21 hand keypoints as the 3D coordinate data of that hand keypoint and chirality classification label data. Among them, x and y in the 3D coordinates (x, y, z) of the hand keypoint are expressed in the image normalized coordinate system, z is the relative depth, the unit is the same as x and y, and the smaller the value, the closer to the camera. The chirality classification label data can be the chirality (handedness) classification result, including the label (value is Left or Right) and the confidence score (confidence value is [0, 1]).

[0101] (c) Confidence Filtering: Detection frames with a confidence level below the initial detection threshold can be directly discarded by the hand keypoint detection model (such as MediaPipe) without outputting target hand identification data. The target hand identification data can be the identification field data of multi_hand_landmarks. Step S3 is only continued if a non-empty field is received; otherwise, no control command is issued for this frame.

[0102] Therefore, the output in step S2 is a list of multi_hand_landmarks and multi_handedness (which can be empty) for each frame as target hand identification data. Compared with the wearable hardware solution used in Method 1, step S2 of this embodiment of the invention obtains 21 3D key points using only a regular monocular RGB camera and a general key point model, significantly reducing hardware costs and wearing burden; and even if the relative depth z output by the hand key point detection model is not an absolute distance, it is sufficient to support the geometric construction of the palm plane normal vector and thumb direction vector in the subsequent step S4, thereby solving the problem in the traditional Method 2 solution that cannot solve the thumb opposition degree of freedom because it only uses 2D projection.

[0103] like Figure 4 The illustrated implementation example shows a topology diagram of 21 hand keypoints output by MediaPipe. The 21 keypoints (numbered 0 to 20) are represented by solid circles (smaller diameter):

[0104] Number 0 is located at the wrist (WRIST); numbers 1 to 4, namely THUMB_CMC / MCP / IP / TIP, are located on the thumb from the palm to the fingertip (i.e., 1-THUMB_CMC is the thumb wrist-palmar joint, 2-THUMB_MCP is the thumb metacarpophalangeal joint, 3-THUMB_IP is the thumb interphalangeal joint, and 4-THUMB_TIP is the thumb fingertip); numbers 5 to 8, namely INDEX_MCP / PIP / DIP / TIP, are located on the index finger (e.g., 5-INDEX_MCP is the index finger metacarpophalangeal joint); numbers 9 to 12, namely MIDDLE_MCP / PIP / DIP / TIP, are located on the middle finger; numbers 13 to 16, namely RING_MCP / PIP / DIP / TIP, are located on the ring finger; and numbers 17 to 20, namely PINKY_MCP / PIP / DIP / TIP, are located on the little finger.

[0105] Among them, such as Figure 4 The thin solid line segments shown represent the connection relationship between adjacent key points as agreed by MediaPipe; in the figure, the THUMB_TIP–THUMB_MCP segment corresponding to the thumb direction vector in step S4 of the embodiment of the present invention is additionally marked with a thick solid line segment, and the WRIST–INDEX_MCP segment and WRIST–PINKY_MCP segment corresponding to the construction of the palm plane normal vector are additionally marked with double line segments, as can be seen in the following text.

[0106] like Figures 2A to 9A As shown, according to an embodiment of the present invention, in operation S213, matching target three-dimensional key point data based on target hand identifier data includes:

[0107] The preset target-side field is mapped to the target string corresponding to the hand keypoint detection model;

[0108] Based on the chiral classification label data corresponding to the target hand identifier data and the classification label of the target string, the target's three-dimensional key points are extracted by matching;

[0109] The target's 3D key points are encapsulated based on the preset key point numbers of the hand key point detection model to generate corresponding target 3D key point data.

[0110] like Figures 3-8As shown, step S3 mainly implements target hand selection based on chirality classification. This selection is triggered synchronously when the target hand selection module outputs a non-empty detection result (i.e., target hand identification data) in step S2. Therefore, the upstream dependency of step S3 can be the multi_handedness list data (chirality classification label data) of the target hand identification data output in step S2 and the user-preconfigured target side fields, i.e., preset target side fields such as side∈{left, right}. Downstream, the geometric calculation in step S4 further executes and processes the process. Step S3 may include:

[0111] (a) Target side label construction: The user-configured preset target side field is mapped to the Left or Right string agreed by the hand key point detection model (MediaPipe), i.e. the target string.

[0112] (b) Traversal Matching: The target string's classification label (i.e., classification[0].label) is compared item by item according to the index order of the chiral classification label data (multi_handedness). The first detection item that is equal to the target side label is found, and the 21 key points corresponding to the index in the target hand identification data (multi_hand_landmarks) are extracted as the target's 3D key points. If there is no matching item, no control command is issued in this frame, and the frame waits for the next frame.

[0113] (c) Key point export: The 21 key points of the matching item (i.e. target 3D key points) are encapsulated into an array indexed by the key point number (0 to 20) of the hand key point detection model (MediaPipe) as target 3D key point data and transmitted to the geometry solution module in step S4.

[0114] In this embodiment of the invention, step S3 outputs 21 target hand 3D key points, which are the three-dimensional key point data of the target hand. Therefore, compared with the existing solution of "selecting the first hand according to the detection order" described in traditional method five, the control object can be explicitly locked by the handedness classification field, eliminating the phenomenon of "the control object shaking with the posture switching" in the scenario of two hands coexisting, and avoiding miscontrol.

[0115] Regarding the target hand selection mechanism, this embodiment of the invention primarily uses the MediaPipe handedness classification field for target hand selection. Specifically, it can be replaced by selection based on keypoint spatial location (e.g., selecting keypoints whose average x-coordinate is located in the right half of the image), selection based on explicit operator gestures (the hand is locked after the operator raises it and completes a set of calibration gestures), selection based on visual identifiers worn by the operator (e.g., colored wristbands, visual cursors), or robust selection based on voting from multiple frames of the handledness field. The target hand selection result must be a unique target hand based on 21 3D keypoints. Therefore, as long as the core property of "explicitly locking a unique control source in a scenario with both hands present" is maintained, it can be considered an equivalent replacement for the target hand selection mechanism in this embodiment of the invention.

[0116] like Figures 3-8 As shown, in step S4, a 10-DOF pure vector geometric solution process can be performed on the 21 key points of the target 3D key point data. Specifically, this process can be triggered synchronously by the geometric solution module when the target hand key points are output in step S3; the upstream dependency is the 21 key points output in step S3, and the downstream is further processed by the joint-by-joint normalization in step S5.

[0117] like Figure 5B The diagram shown in this embodiment of the invention illustrates the primitive for the "three-point angle" between the flexion of one finger and the abduction of an adjacent finger. Specifically, small solid circles represent the three key points, thin solid line segments represent the bone connections, and small arcs represent the desired angle. In step S4, the preset angle primitive can be defined, such as... Figure 5B As shown, let the 3D coordinate vector of keypoint p be p=(x,y,z); the angle (in degrees) between the three points with vertex b and endpoints a and c can be defined as:

[0118] θ(a, b, c) = \arccos (clip (((a - b)·(c - b)) / (\lVert a - b \rVert·\lVert c - b \rVert +ε), -1, 1))·(180) / (π) (1)

[0119] Where a, b, and c are the position vectors of three 3D keypoints; · represents the dot product of three-dimensional vectors; \lVert·\rVert represents the Euclidean norm of the three-dimensional vector; ε is a numerical stability term to prevent division by zero (in this example, ε = 10). -8 `clip(u,-1,1)` means clipping `u` to the interval [-1,1] to ensure the domain of `arccos`. This preset angle primitive can be repeatedly called in subsequent geometric calculations of four-finger flexion, four-finger abduction, thumb flexion, and thumb abduction.

[0120] like Figures 2A to 9A As shown, according to an embodiment of the present invention, in operation S214, multi-degree-of-freedom pure vector geometry calculation of the target three-dimensional key point data based on a preset angle primitive is performed, and the original geometric angle vector data corresponding to the degrees of freedom is output, the following is included:

[0121] For any one of the index, middle, ring, and little fingers, based on the preset angle primitives, the corresponding index finger flexion angle, middle finger flexion angle, ring finger flexion angle, and little finger flexion angle are generated through the key points of the fingertip, the key points of the proximal interphalangeal joint, and the key points of the metacarpophalangeal joint.

[0122] like Figures 2A to 9A As shown, according to an embodiment of the present invention, in operation S214, which performs multi-degree-of-freedom pure vector geometric calculation on the target three-dimensional key point data based on a preset angle primitive and outputs the original geometric angle vector data corresponding to the degrees of freedom, the method further includes:

[0123] Based on the preset angle primitives, the corresponding thumb flexion angle is generated by the key points of the fingertip, interphalangeal joints and metacarpophalangeal joints of the corresponding thumb.

[0124] like Figures 2A to 9A As shown, according to an embodiment of the present invention, in operation S214, which performs multi-degree-of-freedom pure vector geometric calculation on the target three-dimensional key point data based on a preset angle primitive and outputs the original geometric angle vector data corresponding to the degrees of freedom, the method further includes:

[0125] For the index, ring, and little fingers, based on the preset angle primitives, the angle between the two metacarpophalangeal joint key points between two adjacent fingers and the wrist key point is formed to generate the corresponding index finger abduction angle, ring finger abduction angle, and little finger abduction angle.

[0126] like Figures 2A to 9A As shown, according to an embodiment of the present invention, in operation S214, which performs multi-degree-of-freedom pure vector geometric calculation on the target three-dimensional key point data based on a preset angle primitive and outputs the original geometric angle vector data corresponding to the degrees of freedom, the method further includes:

[0127] Based on the preset angle primitive, the corresponding thumb abduction angle is generated by forming the angle between the key points of the metacarpophalangeal joint of the corresponding thumb, the key point of the wrist, and the key point of the metacarpophalangeal joint of the index finger.

[0128] like Figures 2A to 9A As shown, according to an embodiment of the present invention, in operation S214, which performs multi-degree-of-freedom pure vector geometric calculation on the target three-dimensional key point data based on a preset angle primitive and outputs the original geometric angle vector data corresponding to the degrees of freedom, the method further includes:

[0129] The thumb deflection geometry is determined based on the palm plane normal vector with the wrist key point as the origin, and the thumb direction vector corresponding to the three-dimensional coordinate vectors of the thumb tip and the thumb metacarpophalangeal joint respectively.

[0130] like Figures 2A to 9A As shown, according to an embodiment of the present invention, in operation S214, which performs multi-degree-of-freedom pure vector geometric calculation on the target three-dimensional key point data based on a preset angle primitive and outputs the original geometric angle vector data corresponding to the degrees of freedom, the method further includes:

[0131] The original geometric angle vector data with multiple degrees of freedom are arranged in the following order: thumb flexion angle, thumb deflection geometric measure, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, little finger flexion angle, index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle.

[0132] Step S4 includes geometric sub-steps for four-finger flexion, four-finger abduction, and thumb flexion, thumb abduction, and thumb deflection:

[0133] (a) Four-finger flexion calculation (index finger, middle finger, ring finger, pinky finger): For each finger fi∈{index, middle, ring, pinky}, with the fingertip key point fi TIP Key points of the proximal interphalangeal joint fi PIP Key points of metacarpophalangeal joints fi MCP The three key points are used as inputs, and the aforementioned preset angle primitive is called to obtain the flexion angle θ at the proximal interphalangeal joint (PIP). flex,fi The degree of freedom of flexion for that finger (including the little finger) can specifically include the index finger flexion angle θ. flex, index θ, the flexion angle of the middle finger flex, middle θ, the flexion angle of the ring finger flex, ring and the little finger flexion angle θ flex, pinky Among them, TIP stands for fingertip, PIP stands for proximal interphalangeal joint, and MCP stands for metacarpophalangeal joint.

[0134] (b) Thumb flexion calculation: Using the three key points THUMB_TIP (thumb tip key point), THUMB_IP (thumb interphalangeal joint key point), and THUMB_MCP (thumb metacarpophalangeal joint key point) as input, the thumb flexion angle θ at the THUMB_IP joint is obtained by calling the preset angle primitive. flex,thumbIP stands for interphalangeal joint.

[0135] (c) Three-finger abduction calculation (index finger, ring finger, little finger): using the metacarpophalangeal joint key point MCP of adjacent two fingers and the wrist key point WRIST (the wrist is as follows) Figure 4 The key point (number 0 of MediaPipe shown) forms an angle, and the abduction angle is calculated based on the preset angle primitive:

[0136] (c1) For index finger abduction: Take the key point INDEX_MCP of the metacarpophalangeal joint of the index finger, the key point WRIST of the wrist, and the key point MIDDLE_MCP of the metacarpophalangeal joint of the middle finger, and generate the index finger abduction angle θ based on the preset angle primitive. abd,index ;

[0137] (c2) For ring finger abduction: Take the key point MIDDLE_MCP of the metacarpophalangeal joint of the middle finger, the key point WRIST of the wrist, and the key point RING_MCP of the metacarpophalangeal joint of the ring finger, and generate the ring finger abduction angle θ based on the preset angle primitive. abd,ring ;

[0138] (c3) For little finger abduction: Take the key point RING_MCP of the metacarpophalangeal joint of the ring finger, the key point WRIST of the wrist, and the key point PINKY_MCP of the metacarpophalangeal joint of the little finger, and generate the little finger abduction angle θ based on the preset angle primitive. abd,pinky .

[0139] Therefore, the three abduction angles mentioned above, namely the index finger abduction angle θ, can be obtained based on the preset angle primitive. abd,index Ring finger abduction angle θ abd,ring Little finger abduction angle θ abd,pinky .

[0140] (d) Thumb abduction calculation: The angle formed by the three key points THUMB_MCP of the thumb metacarpophalangeal joint, WRIST of the wrist, and INDEX_MCP of the index finger metacarpophalangeal joint is obtained by calling the preset angle primitive. abd,thumb .

[0141] (e) Thumb yaw (opposition of palms) calculation: e.g. Figure 5A As shown, first construct the normal vector of the palm plane with the wrist key point WRIST as the origin:

[0142] n = ((p INDEX_MCP - p WRIST )×(p PINKY_MCP - p WRIST )) / (\lVert (p INDEX_MCP - p WRIST )×(p PINKY_MCP- p WRIST )\rVert +ε) (2)

[0143] Where × represents the cross product of three-dimensional vectors; p INDEX_MCP p PINKY_MCP p WRIST These are the 3D coordinate vectors of the metacarpophalangeal joint of the index finger, the metacarpophalangeal joint of the little finger, and the wrist, respectively; ε is synonymous with the previous formula.

[0144] like Figure 5A The diagram shows the 10-DOF geometric solution principle. The 3D geometric construction of the palm plane normal vector and the thumb direction vector is shown (using a 3D oblique projection to depict the positions of five key points: WRIST, INDEX_MCP (index finger metacarpophalangeal joint key point), PINKY_MCP (little finger metacarpophalangeal joint key point), THUMB_MCP (thumb metacarpophalangeal joint key point), and THUMB_TIP (thumb fingertip key point), indicated by bold solid arrows). INDEX_MCP– p WRIST With p PINKY_MCP– p WRIST Two vectors are used: a double-lined arrow representing the normal vector n obtained by the cross product, and a bold dashed arrow representing the thumb direction vector t. The angle between n and t is marked with a small arc.

[0145] Therefore, the thumb direction vector can be represented as follows:

[0146] t = (p THUMB_TIP - p THUMB_MCP ) / (\lVert p THUMB_TIP - p THUMB_MCP \rVert +ε)(3)

[0147] Where, p THUMB_TIP p THUMB_MCP These are the 3D coordinate vectors of the thumb tip and the thumb metacarpophalangeal joint, respectively. Therefore, the geometric metric for thumb deflection can be defined as follows:

[0148] y thumb = 100 - \lvert t·n \rvert·100 (4)

[0149] Where t·n is the dot product of unit vector t and unit vector n, and its absolute value ranges from [0, 1]; when the thumb direction is parallel to the palm normal vector (i.e., the thumb is completely perpendicular to the palm, making a "palm-to-palm" posture), t·n ≈ 1, y thumb ≈0; when the thumb direction is within the palm plane (i.e., perpendicular to the normal vector, with the thumb and four fingers extended and coplanar) ≈0, y thumb≈100. This geometric metric can be stably solved at 3D keypoints and has invariance to the overall rotation of the hand in the camera coordinate system (that is, when the hand rotates around any axis, the normal vector defined by the cross product rotates with the palm, so that t·n still only reflects the orientation relationship of the thumb relative to the palm plane).

[0150] (f) Joint sequence convention: The above 10 original geometric angles are arranged in the following fixed order as a vector of length 10 (i.e., original geometric angle vector data): θ = (θ flex,thumb , y thumb , θ flex,index , θ flex,middle , θ flex,ring ,θ flex,pinky , θ abd,index , θ abd,thumb , θ abd,pinky , θ abd,ring Let its component be θ. i , i = 1,…, 10. This sequence is consistent with the interface convention of the downstream control host computer (such as Dashboard), so that steps S5, S6, and S7 can be directly mapped to specific joints by integer index.

[0151] Therefore, step S4 outputs a vector θ∈ℝ consisting of 10 original geometric angles. 10 In contrast to the existing heuristic mapping scheme of "finger-tip distance ratio + two-dimensional projection angle" used in the traditional method two, the above step S4 of this embodiment can be based on three primitives: "bending = angle between three PIP points", "abduction = angle between adjacent MCP key points and WRIST", and "thumb deflection = absolute value of the dot product of normal vector and thumb direction vector". This makes each dexterity hand degree of freedom have a clear and reproducible geometric definition. Among them, the normal vector measurement of thumb deflection utilizes the geometric information carried by the z component in the 3D key points. It cannot be directly observed under two-dimensional projection but can be stably solved on the 3D key points, which constitutes the key contribution of this invention to the thumb-palm degree of freedom.

[0152] In this embodiment of the invention, during the calculation of geometric quantities for angles, the preset angle primitive "three-point angle" used for four-finger flexion can be replaced with the primitive "angle between two bone vectors" (directly expressing flexion by the angle between the two vectors of the MCP-PIP segment and the PIP-TIP segment); the "absolute value of the dot product of the normal vector" metric used for thumb deflection can be replaced with "extraction of the rotational component from the local coordinate system of the thumb palmar key points to the palm coordinate system based on quaternions" or "fitting the palm plane based on Principal Component Analysis (PCA) and then projecting the thumb direction." The boundary of the substitution is that the new geometric quantity takes one extreme value in the thumb-palm posture and another extreme value in the extended posture, while maintaining approximately rotational invariance of the overall hand posture. Therefore, as long as the two core properties of "flexion and abduction being surrogate quantities of a certain 'two-bone angle'" and "thumb deflection being surrogate quantities of a certain 'thumb direction relative to the palm plane direction'" are maintained, it is considered an equivalent substitution in this embodiment of the invention.

[0153] Furthermore, regarding the process of constructing keypoint pairs using normals, as follows: Figure 5A The method of constructing the palm plane normal vector using the three key points (WRIST, INDEX_MCP, PINKY_MCP) can be replaced by fitting the least squares plane normal vector using PCA or Singular Value Decomposition (SVD) with four key points (INDEX_MCP, MIDDLE_MCP, RING_MCP, PINKY_MCP). Alternatively, it can be replaced by any non-collinear combination of three points (WRIST, MIDDLE_MCP, PINKY_MCP) that can determine the palm plane, with sign correction to ensure that the normal vector always points to the outer side of the back of the hand. The boundary of the substitution is that the selected combination of key points must be approximately coplanar and sufficient to determine a unique plane. Therefore, as long as the core property of "fitting the palm plane normal vector with three or more MCP key points (or an equivalent non-collinear combination including WRIST) in the palm region" is maintained, it is considered an equivalent substitution in the embodiments of the present invention.

[0154] Furthermore, regarding the number of degrees of freedom for the thumb and the four fingers abduction, in this embodiment of the invention, the thumb can have 3 degrees of freedom (flexion, abduction, and deflection) and the three fingers can have abduction (index, ring, and little fingers). Alternatively, it can be tailored to 2 degrees of freedom for the thumb (removing deflection and retaining only flexion and abduction) or extended to 4 degrees of freedom for the thumb including IP, depending on the dexterous hand mechanical structure. The four fingers abduction can be tailored to 1 degree of freedom for the little finger abduction or extended to 4 degrees of freedom for all four fingers including abduction. The corresponding geometric measures are increased or decreased synchronously according to the same primitives of "angle between two bones" or "normal + direction vector". Therefore, each degree of freedom in the new set of degrees of freedom can be solved by a subset of the 21 3D keypoints according to the primitives. Therefore, as long as the two core properties of "each joint degree of freedom has an interpretable pure geometric definition" and "all joint drive quantities are processed by a unified normalization-smoothing-downlink processing" are maintained, it is considered an equivalent substitution of this embodiment of the invention.

[0155] like Figures 2A to 9A As shown, according to an embodiment of the present invention, the data processing method applied to real-time gesture mirroring of a dexterous hand further includes:

[0156] Based on the preset baseline lower bound data and preset baseline range data corresponding to each joint of the original geometric angle vector data, the original geometric angle vector data is normalized to generate normalized vector data of the corresponding dimension. The normalized vector data is used to realize real-time gesture mirroring of the dexterous hand.

[0157] like Figures 3-8 As shown, step S5 is mainly used to implement joint-by-joint linear normalization and saturation clipping based on the range of physiological activity. The joint-by-joint normalization and saturation clipping are synchronously triggered by the normalization module corresponding to step S5 when the 10-dimensional original angle vector is output in step S4; the upstream dependency is the original geometric angle vector data θ output in step S4 and the lower bound and range parameters of the joint-by-joint baseline in the user configuration, and the downstream is further executed or processed by the differential smoothing in step S6 (see Embodiment 2 described later). In this embodiment of the invention, step S5 includes the following sub-steps:

[0158] (a) Joint-by-joint parameter pair convention: For each joint i = 1, …, 10, a lower baseline is defined. i (i.e., the preset baseline lower limit data, in degrees) and a range span i (i.e., preset baseline range data, in degrees). Among them, the parameters of the preset baseline lower limit data and the preset baseline range data reflect the physiological starting point of the corresponding joint in the human body (0 output when naturally extended) and the range of motion (100 output when fully bent or fully opened).

[0159] (b) Linear normalization: The original angles (i.e., the original geometric angle vector data) are mapped one by one to the range [0, 100] to form the following normalized vector data:

[0160] v i =clip((θ i - base i ) / (span i (5) × 100, 0, 100)

[0161] Where θ i The base is the i-th original geometric angle output in step S4 above. i ,span i The baseline lower bound and range of the joint; clip(u,0,100) means to saturate and clip u to the interval [0,100].

[0162] (c) Straight-through processing of thumb deflection: For thumb deflection y thumb Since step S4-(e) has directly produced values ​​in the range [0,100], this step sets its base=0 and span=100 and still processes it according to the above formula to maintain an interface consistent with the other nine degrees of freedom.

[0163] (d) Typical parameter values: Examples of parameters in the embodiments of the present invention are listed in Table 2 below, wherein the parameters can be self-tuned according to the operator's hand size, the specific model of the robot arm and the agreement of the downstream Dashboard.

[0164]

[0165] Table 2

[0166] In step S5, the output is a 10-dimensional local frame vector v∈[0,100]. 10 This refers to normalized vector data. Therefore, compared to the traditional approach of "unified linear mapping without joint-by-joint calibration" used in Method 2, the method described in this embodiment of the invention can independently map heterogeneous geometric angles (flexion and abduction, thumb and four fingers) to the range of [0,100] using joint-by-joint "baseline lower bound + range" parameter pairs. Furthermore, it removes out-of-bounds values ​​caused by abnormal detection or human hyperextension through saturation pruning, ensuring that the downstream dexterous hand controller only receives normalized and truncated valid logical values.

[0167] The "linear normalization + saturation clipping" process for normalized mapping can be replaced by: soft clipping mapping based on hyperbolic tangent (tanh) or sigmoid function; piecewise linear mapping in three levels: "natural straightening state - partially bent state - fully bent state"; lookup table method (indexed by the original angle, and the logical value is obtained by lookup table interpolation); or personalized mapping based on the operator's one-time calibration (using the operator's maximum straightening angle and maximum bending angle as the operator's base and base+span). The mapping result must fall within the range [0, 100]. That is, as long as the two core properties of "monotonic mapping of joint-independent parameters" and "output falling within the closed interval of 0–100" are maintained, it is considered an equivalent substitution of the embodiments of this invention.

[0168] Based on the above-described data processing method for real-time gesture mirroring of dexterous hands, this invention also provides a data processing device for real-time gesture mirroring of dexterous hands. The following will be combined with... Figure 9A The device is described in detail.

[0169] Figure 9A The diagram illustrates the structure of a data processing apparatus 910 for real-time gesture mirroring of a dexterous hand according to an embodiment of the present invention.

[0170] like Figure 9A As shown, the data processing device 910 for real-time gesture mirroring of dexterous hands in this embodiment includes a frame data acquisition module 911, an identifier data acquisition module 912, a key point data matching module 913, and a geometric calculation module 914.

[0171] The frame data acquisition module 911 is used to acquire video frame data that has undergone image preprocessing. In one embodiment, it can be used to perform the operation S211 described above, which will not be repeated here.

[0172] The identifier data acquisition module 912 is used to acquire target hand identifier data from video frame data based on a hand keypoint detection model instantiated with pre-parameters. In one embodiment, it can be used to perform the operation S212 described above, which will not be repeated here.

[0173] The key point data matching module 913 is used to match the target's three-dimensional key point data based on the target hand identification data. In one embodiment, it can be used to perform the operation S213 described above, which will not be repeated here.

[0174] The geometric solution module 914 is used to perform multi-degree-of-freedom pure vector geometric solution on the target's three-dimensional keypoint data based on a preset angle primitive, and outputs the original geometric angle vector data corresponding to the degrees of freedom to realize real-time gesture mirroring of the dexterous hand. In one embodiment, it can be used to execute the operation S214 described above, which will not be repeated here.

[0175] The original geometric angle vector data includes vector data corresponding sequentially to the preset joints of the dexterous hand, such as the thumb flexion angle, thumb deflection geometric measurement, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, little finger flexion angle, index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle.

[0176] According to embodiments of the present invention, any plurality of modules among the frame data acquisition module 911, the identifier data acquisition module 912, the key point data matching module 913, and the geometry calculation module 914 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the frame data acquisition module 911, the identifier data acquisition module 912, the key point data matching module 913, and the geometry calculation module 914 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in hardware or firmware, or in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the frame data acquisition module 911, the identifier data acquisition module 912, the key point data matching module 913, and the geometric solution module 914 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0177] Example 2:

[0178] The following will be based on Figure 1 The described scenario, combined with the data processing method and apparatus for the real-time gesture mirroring method of dexterous hands in Embodiment 1 above, is used through... Figures 2A to 9B The real-time gesture mirroring method for dexterous hands, as disclosed in Embodiment 2, will be described in detail.

[0179] like Figure 2B As shown, one aspect of an embodiment of the present invention provides a real-time gesture mirroring method for dexterous hands, which includes operations S221 to S223.

[0180] In operation S221, based on the target hand identification data output by the hand key point detection model, the target three-dimensional key point data corresponding to the target is generated, and normalized vector data of the corresponding dimension is generated.

[0181] In operation S222, a weighted moving average smoothing process is performed on the normalized vector data based on the pre-generated joint differentiation index to generate corresponding smoothed vector data; and

[0182] In operation S223, based on the smoothed vector data and the target object received through the preset device discovery interface, a hypertext transfer protocol request is generated and sent to the preset control host computer to realize real-time gesture mirroring applied to the dexterous hand.

[0183] Among them, the original geometric angle vector data corresponding to the normalized vector data includes vector data that correspond sequentially with the preset joints of the dexterous hand, such as the thumb flexion angle, thumb deflection geometric measurement, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, little finger flexion angle, index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle.

[0184] like Figure 3 and Figures 6-8 As shown, step S6 mainly implements the joint-weighted moving average smoothing process based on the joint differential index. Specifically, step S6 is triggered synchronously by the differential smoothing module when the normalized vector for this frame is output in step S5; the upstream dependency is the current frame vector v∈[0,100] output in step S5. 10 The smoothing vector s of the previous frame maintained by the differential smoothing module itself. (-1) The downstream process is further processed and executed by the distribution module in step S7. Step S6 may include sub-steps such as joint-by-joint EMA coefficient convention, joint-by-joint EMA recursion, and typical coefficient configuration.

[0185] like Figures 2B to 9B As shown, according to an embodiment of the present invention, before operation S222 performs weighted moving average smoothing on the normalized vector data according to the pre-generated joint differentiation index to generate corresponding smoothed vector data, the method further includes:

[0186] For each joint of the normalized vector data, an independent exponential moving average coefficient is generated as a joint differentiation index.

[0187] like Figure 3 As shown, in this embodiment of the invention, step S6 can be performed by defining the coefficients for each joint based on the rules governing the generation of the exponential moving average coefficients.

[0188] (a) Joint-by-joint coefficient (e.g., EMA exponential moving average) convention: assign an independent EMA coefficient f to each joint i = 1, …, 10. i∈(0, 1], serving as the joint differentiation index. This coefficient reflects the update weight of the joint in the current frame; the smaller the coefficient, the stronger the smoothness, the slower the response, and the more thorough the jitter suppression.

[0189] like Figures 2B to 9B As shown, according to an embodiment of the present invention, in operation S222, weighted moving average smoothing is performed on the normalized vector data according to the pre-generated joint differentiation index to generate corresponding smoothed vector data, including:

[0190] Based on the normalized value of the current frame corresponding to the joint in the normalized vector data, the smoothed value of the previous frame, and the corresponding joint difference index, the smoothed value of the current frame is updated; where the smoothed value of the current frame is used to construct the smoothed vector data.

[0191] like Figure 3 As shown, in step S6 of this embodiment of the invention, the following (b) joint-by-joint EMA recursive process is involved, mainly involving the following frame smoothing value s i Update:

[0192] s i = f i ·v i + (1 - f i )·s i (-1) (6)

[0193] Among them, v i The normalized value of the current frame is the i-th value output in step S5 above (i.e., the normalized value of the current frame); s i (-1) f is the smoothed value of the i-th frame in the previous frame (i.e., the smoothed value of the previous frame); i The joint differentiation index (such as the EMA coefficient) configured for this joint; s i This is the i-th smoothed value of the current frame (i.e., the smoothed value of the current frame), and it is written back as s for the next frame. i (-1) Among them, the first frame's s (-1) This allows us to initialize v itself, which is equivalent to setting f in the first frame. i =1.

[0194] like Figure 6 As shown, the joint differential index (EMA coefficient) is used for the following configurations. The horizontal axis represents the 10 joint indices i=1 to 10 from left to right (thumb flexion, thumb deflection, index finger flexion, middle finger flexion, ring finger flexion, little finger flexion, index finger abduction, thumb abduction, little finger abduction, and ring finger abduction in that order). The vertical axis represents the EMA coefficient f. i The values ​​are taken from 0 to 1 at equal intervals, with the main scale being 0.1; the f value for each joint is expressed in a bar chart. iThe columns for the eight active joints (i=1, 2, 3, 4, 5, 7, 8, 10) are represented without fill (height 0.4), while the columns for the two easily vibrating end joints (i=6, i=9) are represented with diagonal fill (height 0.2). Figure 6 The text indicates that "unfilled columns represent active joints, and diagonally filled columns represent easily shaking end joints"; the horizontal axis labels are labeled according to the semantic text of the joints.

[0195] Therefore, as an embodiment of the present invention, for (c) typical coefficient configuration, a specific embodiment can be configured as follows:

[0196] f = (0.4, 0.4, 0.4, 0.4, 0.4, 0.2, 0.4, 0.4, 0.2, 0.4) (7)

[0197] Specifically, smaller update weights f6=f9=0.2 are assigned to the distal joints with relatively high detection noise (as in the MediaPipe model) for the little finger flexion (i=6) and little finger abduction (i=9) keypoints, while larger update weights f=0.4 are assigned to the other 8 active joints. The above typical coefficient configuration is based on the fact that the hand keypoint detection model (MediaPipe) systematically has lower detection confidence for the little finger PIP and little finger MCP keypoints in self-occlusion or fast-moving scenarios than other fingers, thus easily causing greater tremor amplitude.

[0198] The output of step S6 above is a 10-dimensional smooth vector s∈[0,100]. 10 This refers to smoothing vector data. Therefore, compared to the traditional approach of using a single coefficient filter for all key points in Method 3, this step can decouple the EMA coefficient of each joint into an independent adjustable parameter. Smaller update weights are configured for easily jittery end joints to differentially suppress noise, while larger update weights are configured for active joints to maintain responsiveness. This allows "active joint responsiveness" and "end joint steady-state smoothness" to be achieved simultaneously under a single filter structure. As a result, downstream dexterous motors reduce repeated micro-movements and decrease mechanical wear.

[0199] Specifically, regarding the filter structure, an exponentially weighted moving average (EMA) filter differentiated by joint is required. This can be replaced by a one-euro filter differentiated by joint, a Kalman filter differentiated by joint, a Butterworth low-pass filter differentiated by joint, or a moving average or moving median filter differentiated by joint. The joint-differentiated strategy itself remains unchanged, but the filter must be a linear time-invariant or approximately linear time-invariant recursive or sliding window structure, and each joint must be configured with an independent set of parameters. Therefore, as long as the two core properties of "differentiated filter parameter configuration by joint" and "the filtering strength of easily jittered joints is strictly greater than the filtering strength of active joints" are maintained, this is considered an equivalent substitution of the embodiments of the present invention.

[0200] like Figures 2B to 9B As shown, according to an embodiment of the present invention, before operation S223 generates a Hypertext Transfer Protocol request to be sent to a preset control host computer based on smoothed vector data and the target object received through a preset device discovery interface, the method further includes:

[0201] Based on a preset hypertext transfer protocol, a polling request is initiated to the target device interface of a preset control host computer at a preset polling interval, and the corresponding target device list data is obtained.

[0202] Based on the pre-configured target field matching conditions, the target device list data is filtered and matched for each device object in the target device array, and the encapsulated field data of the corresponding device is read.

[0203] Perform field encapsulation on the encapsulated field data to generate a target object to be sent to the preset device discovery interface.

[0204] like Figure 3 , Figure 7 and Figure 8 As shown, step S8 of this embodiment is mainly used for automatic device discovery and hot-swappable startup based on polling and field filtering matching of the device discovery interface. Specifically, it can be triggered once by the device discovery module when the camera process starts and blocked before the main loop of steps S1 to S7 begins, waiting for successful matching. The upstream dependencies are the target model (such as "L10"), target variant (such as "ball_joint" or "worm_gear"), Dashboard basic Uniform Resource Locator (URL), and polling interval in the user configuration. The downstream is the target object obtained by consuming the matching through HTTPPOST in step S7. Among them, as shown Figure 3 , Figure 7 and Figure 8 As shown, step S8 includes the following sub-steps:

[0205] (a) Dashboard Device List Query: During the initialization process in step S70, after initializing parameters such as model / variant / side, the base URL, and the polling interval, in step S71, a device list request is initiated to the target device interface (GET / api / hand / devices interface) exposed by the Dashboard (i.e., the preset host computer) using the Hypertext Transfer Protocol to obtain the device list information provided by the corresponding preset host computer as the target device list data. When the Dashboard is unreachable or returns a non-200 status code, a fixed interval Δt is used. poll (i.e., the preset polling interval, which is 3 seconds in this example) and then retry.

[0206] (b) Field Filtering and Matching: For each device object in the target device array data[] in the response body, field matching is performed according to the user-configured target model and target variant (i.e., target field matching conditions), which is the field filtering process in step S72. Specifically, for each device object d∈data, a match is considered to be made if and only if d.model=target_model and d.variant=target_variant, which is the target field matching condition. This matching condition can be extended as needed to match by device serial number or by interface field, etc.

[0207] (c) Target Object Construction: In step S72 above, the fields such as interface (indicating the CAN interface name where the dexterous hand is located) and side (indicating whether the dexterous hand is left-handed or right-handed) are read from the matching device object as encapsulation field data. In step S73, this encapsulation field data is encapsulated into a target object, thus completing the construction of the target object. Further, the target object is passed to the sending module (HTTP POST) in step S7, entering the main loop of the camera process in step S74. When multiple matching devices exist, further filtering is performed according to the "target side" in the user configuration; if there are still multiple matching devices, the first one is selected, and all candidates are recorded in the log.

[0208] (d) Loop retry: When data[] is empty, the match is empty, or the Dashboard returns an error, retry according to the preset polling interval Δt. pollThe process retryes indefinitely until a match is found; the loop does not exit the main process. It should be noted that the loop also covers scenarios where the dexterous hand experiences an unexpected power outage and subsequent power restoration—if the HTTP POST request continues to fail beyond a preset threshold during the main loop, step S7 can call back to step S8 to trigger a rediscovery to recover the target object.

[0209] Therefore, step S8 of this embodiment outputs an available target object. Compared with the traditional approach of "enumerating devices once at startup or exiting" used in method five, this step eliminates startup order constraints with "periodic polling + field filtering and matching"—the dexterous hand can be powered on and online at any time before, after, or on-site before the vision process starts; the same set of vision algorithms can be reused across models and variants through model and variant field filtering, so that when adding a new dexterous hand variant, there is no need to modify the vision algorithm code, only to register a new variant identifier on the Dashboard.

[0210] like Figure 7 The diagram shows the automatic device discovery polling state, where rounded rectangle nodes represent states, solid single arrows represent state transitions, and diamond nodes represent decision branches. It includes eight main nodes: "Step S70 Initialization," "Step S71 Execute GET / api / hand / devices query in device list request," "Check if response status code is 200," "Check if data array is empty," "Step S72 Filter by model and variant fields," "Check if matching set is empty," "Step S73 Construct target object," and "Step S74 Enter main loop." The diagram returns "Wait Δt" for three different scenarios: a non-200 response, empty data, and empty filter match. poll The "Retry" node is followed by a dashed feedback arrow indicating the hot-plug recovery path: "HTTP POST failures in the main loop exceed the threshold → Re-enter device discovery".

[0211] The device discovery mechanism in this embodiment of the invention primarily utilizes a periodic polling mechanism of the Dashboard / api / hand / devices interface. Specifically, this can be replaced by a device online push mechanism based on Server-SentEvents (SSE) or WebSocket (where the Dashboard actively pushes device change events); network discovery based on multicast Domain Name System (mDNS) or ZeroConfiguration Networking (Zeroconf); or device discovery based on the Universal Plug and Play (UPnP) protocol. Device discovery requires that the camera process obtains a device description object containing the `model` and `variant` fields after the dexterous hand is online. Therefore, as long as the core property of "the camera process can start without prior assumption that the dexterous hand is online and can ultimately lock onto the correct dexterous hand" is maintained, it is considered an equivalent alternative to this embodiment of the invention.

[0212] like Figures 2B to 9B As shown, according to an embodiment of the present invention, in operation S223, generating a Hypertext Transfer Protocol request to be sent to a preset control host computer based on smoothed vector data and a target object received through a preset device discovery interface, the following steps are included:

[0213] Each logical value of the smoothed vector data is mapped component by component to the corresponding integer logical value of the preset program interface of the preset control host computer;

[0214] Arrange integer logical values ​​into the first dictionary data and the second dictionary data according to semantic grouping;

[0215] The first dictionary data includes integer logical values ​​related to the thumb flexion angle, thumb deflection geometry, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, and little finger flexion angle; the second dictionary data includes integer logical values ​​related to the index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle.

[0216] like Figure 3 , Figure 7 and Figure 8 As shown, in this embodiment of the invention, step S7 is mainly used to implement logical value range mapping, throttling by beat, and HTTP POST delivery. Specifically, it can be triggered synchronously by the delivery module when it outputs the smoothed vector data of this frame in step S6; the upstream dependency is the smoothed vector s∈[0, 100] of this frame output in step S6. 10The target object output in step S8 is the last distribution timestamp maintained by the distribution module itself. The downstream endpoint is the POST / api / hand / joint / logical interface provided by the Dashboard process via the REST API. Step S7 mainly includes the following sub-steps:

[0217] (a) Linear range mapping: Each logical value of the smoothed vector data [0, 100] is linearly mapped to the [0, 255] integer type required by the preset program interface (such as Dashboard API) of the preset control host computer. Specifically, the corresponding integer logical value is calculated component by component as follows:

[0218] u i = int ((s i ) / (100)·255) (8)

[0219] Among them, s i u is the i-th smoothed value output in step S6; i This is the mapped integer logical value; int(·) is for rounding down (the example uses Python's built-in int() for truncation).

[0220] (b) Assemble by semantic grouping: group 10 integer logic values ​​u i According to the default program interface (Dashboard interface) convention, the data is divided into two dictionaries, forming a first dictionary data "basic" and a second dictionary data "swing". The first dictionary data "basic" contains five entries related to flexion and thumb deflection ("thumb_rot"): thumb ("thumb"), index finger ("index"), middle finger ("middle"), ring finger ("ring"), and pinky finger ("pinky"). The second dictionary data "swing" contains four entries related to abduction: index finger abduction ("index_swing"), thumb abduction ("thumb_swing"), pinky finger abduction ("pinky_swing"), and ring finger abduction ("ring_swing"). This grouping allows the receiver of the default control host computer Dashboard to directly decode the data semantically.

[0221] like Figures 2B to 9B As shown, according to an embodiment of the present invention, in operation S223, which generates a Hypertext Transfer Protocol request to be sent to a preset control host computer based on smoothed vector data and the target object received through a preset device discovery interface, the method further includes:

[0222] Based on the preset control host computer that performs throttling at the minimum transmission interval, the first dictionary data, the second dictionary data, and the target object are used to generate a Hypertext Transfer Protocol request in a preset object identifier format.

[0223] like Figure 3 , Figure 7 and Figure 8 As shown, step S7 above also involves the preset control of the host computer's (c) rate limiting execution process. Specifically, the preset minimum transmission interval Δt min (In the example, Δt is taken) min =50 milliseconds, corresponding to a maximum transmission frequency of 20 Hz) to throttle the preset control host computer, putting it in a throttled state. The difference between the current frame timestamp and the previous transmission timestamp is less than this Δt. min When the current frame is executed, it only updates the internal state without sending it, waiting for the next frame to trigger; when the next frame is executed, the timestamp of the last sending is written back.

[0224] Further, under the aforementioned throttling state, a further (d) HTTP POST request is executed. Specifically, the comprehensive information {logical_values:{basic,swing},targets:[target]} based on the target object, the first dictionary data, and the second dictionary data can be used to generate an HTTP POST request conforming to the Hypertext Transfer Protocol (HTTP POST). This request is then sent via HTTP POST to the POST / api / hand / joint / logical interface exposed by the preset control host computer Dashboard. The target object can be the dexterous hand instance description object matched in step S8. Figure 3 , Figure 7 , Figure 8 As shown, the request body does not carry any specific CAN protocol fields; the protocol conversion and transmission to the CAN message can be completed internally by the preset control host computer dashboard. The JSON format can be a data transmission format of JavaScript Object Notation (JSON).

[0225] In this embodiment of the invention, step S7 outputs an HTTP POST request to the Dashboard. Compared to the tightly coupled monolithic solution described in Method 4, this step narrows the coupling between the camera process and the Dashboard process to a single HTTP interface and a single JSON data structure. The camera process is completely unaware of underlying details such as the CAN bus baud rate, the specific CAN identifier of the dexterous hand, and the quantization range of the joint drive, so that the same vision algorithm can be adapted to various dexterous hand models and variants. The throttling mechanism, compared to the direct transmission solution without throttling, prevents the Dashboard and CAN bus from being overwhelmed when the visual frame rate is higher than the controller's receiving capacity.

[0226] According to the embodiments of the present invention, the fixed minimum transmission interval Δtmin The 50ms throttling strategy can be replaced with adaptive throttling (dynamically adjusting Δt based on the Dashboard response latency). min The greater the response delay, the larger the interval; differential trigger throttling (when the L of the current frame's 10-dimensional vector and the previously sent vector... 2 (Send data only when the distance or L^∞ distance is greater than a threshold); or throttling with priority merging (high-change joints are sent immediately, low-change joints are merged for the next cycle). The throttling result must ensure that the sending frequency does not exceed the Dashboard's processing capacity limit. Therefore, as long as the core property of "throttling the sending frequency according to a strategy that matches the Dashboard's receiving capacity" is maintained, it is considered an equivalent substitution for the embodiments of this invention.

[0227] Inter-process communication (IPC) based on REST / HTTP, as defined by the inter-process communication protocol, can be replaced by binary streams based on Google Remote Procedure Call (gRPC); full-duplex long connections based on WebSocket; publish-subscribe or request-response based on ZeroMQ; lightweight messaging communication based on Message Queuing Telemetry Transport (MQTT); and topic communication based on Robot Operating System 2 (ROS 2). The message payload can be replaced with any binary encoding format such as Protocol Buffers or MessagePacks, instead of JSON. The communication protocol must be able to transmit a structured 10-dimensional logical value dictionary and a target description object. Therefore, as long as the core property of "decoupling the camera process and the control process through a structured message interface rather than function calls" is maintained, it is considered an equivalent alternative to the embodiments of this invention.

[0228] Among them, such as Figure 8 The diagram shows the end-to-end data flow timing. Four vertical lanes represent the four roles: operator, visual perception host computer, control host computer, and dexterous hand. The timeline progresses from top to bottom. Bold solid arrows horizontally indicate the direction of key message flow: "Operator → Visual Perception Host Computer" represents RGB video frames, "Visual Perception Host Computer → Control Host Computer" represents HTTP POST, and "Control Host Computer → Dexterous Hand" represents CAN messages. Thin solid lines on the right indicate typical timestamp cumulative values ​​(0 milliseconds for video frame acquisition, 33 milliseconds for MediaPipe detection completion, 34 milliseconds for steps S4–S6 calculation and smoothing completion, ≤50 milliseconds for HTTP POST being sent, and ≤60 milliseconds for the dexterous hand starting its action). The diagram also includes a note stating that each time interval represents a typical value and varies with hardware specifications.

[0229] Based on the above-described real-time gesture mirroring method for dexterous hands, this invention also provides a real-time gesture mirroring device for dexterous hands. The following will combine... Figure 9B The device is described in detail.

[0230] Figure 9B The diagram illustrates a structural block diagram of a real-time gesture mirroring device 920 for a dexterous hand according to an embodiment of the present invention.

[0231] like Figure 9B As shown, the real-time gesture mirroring device 920 for dexterous hands in this embodiment includes a vector data generation module 921, a differential smoothing module 922, and a request generation module 923.

[0232] The vector data generation module 921 is used to generate normalized vector data of the corresponding dimension based on the target three-dimensional keypoint data corresponding to the target hand identifier data output by the hand keypoint detection model. In one embodiment, it can be used to perform the operation S221 described above, which will not be repeated here.

[0233] The differential smoothing module 922 is used to perform weighted moving average smoothing on the normalized vector data according to the pre-generated joint differential index, to generate corresponding smoothed vector data. In one embodiment, it can be used to perform the operation S222 described above, which will not be repeated here.

[0234] The request generation module 923 is used to generate a Hypertext Transfer Protocol request to be sent to a preset control host computer based on the smoothed vector data and the target object received through the preset device discovery interface, so as to realize real-time gesture mirroring applied to the dexterous hand. In one embodiment, it can be used to perform the operation S223 described above, which will not be repeated here.

[0235] Among them, the original geometric angle vector data corresponding to the normalized vector data includes vector data that correspond sequentially with the preset joints of the dexterous hand, such as the thumb flexion angle, thumb deflection geometric measurement, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, little finger flexion angle, index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle.

[0236] According to embodiments of the present invention, any plurality of modules among the vector data generation module 921, the differential smoothing module 922, and the request generation module 923 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the vector data generation module 921, the differential smoothing module 922, and the request generation module 923 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the vector data generation module 921, the differential smoothing module 922, and the request generation module 923 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0237] like Figure 3 , Figure 7 and Figure 8 As shown, based on Embodiments 1 and 2 of the present invention, step S9 can be further provided to realize visual preview and turn-off display. Specifically, it can be triggered synchronously by the preview module when the key point detection results are output in step S2; the upstream dependency can be the current frame output in step S1 and the key point list output in step S2, as well as the show_preview∈{true, false} field in the user configuration. Step S9 can include the following sub-steps:

[0238] (a) Keypoint overlay drawing: When show_preview=true, 21 keypoints (represented by small solid circles) and their connection relationships (represented by thin solid line segments) are overlaid on the current frame using the keypoint connection relationship topology provided by MediaPipe, and the handledness classification label and confidence value of this frame are drawn in the upper left corner of the window.

[0239] (b) Window display and key abort: The superimposed frame is displayed using the OpenCV imshow function; waitKey(1) is polled every frame, and the entire camera process is aborted when the operator presses the q key.

[0240] (c) Preview shutdown branch: When show_preview = false, skip the imshow and waitKey calls to reduce the CPU usage and enable the relevant processes of Embodiment 1 and Embodiment 2 of the present invention to run in a deployment environment without a display device (such as an embedded industrial control computer).

[0241] In particular, the output of step S9 is a local visualization window (optional). As an optional auxiliary step, it can be excluded from the control link. However, in debugging and on-site demonstration scenarios, it can provide visualization of key points to verify the correctness of the detection. In unattended deployment scenarios, it can be shut down with one click to release computing resources. The shut-down design avoids the problem of "display device missing, i.e. the whole process is unavailable" that occurs when visualization and control are tightly coupled as described in method four.

[0242] like Figures 2A-8 As shown, the method provided in Embodiments 1 and 2 of this invention, with specific implementation examples, summarizes the overall process as follows: In step S1, RGB video frames are continuously acquired at camera beats and horizontal flipping and color space conversion are completed; in step S2, MediaPipeHands detection is performed on each frame and filtered by confidence; in step S3, a unique target hand is selected according to the handedness classification field; in step S4, 10 original geometric angles are calculated using pure vector geometry, where thumb deflection is measured by the absolute value of the dot product of the palm normal vector and the thumb direction vector; in step S5, the physiological activity range is linearly normalized and saturated to [0, 100] joint by joint; in step S6, the EMA coefficient is smoothed according to the joint differences; in step S7, the linear mapping is performed to [0, 255] integers and assembled into a {basic, swing} dictionary according to semantic grouping, throttled at a minimum transmission interval of 50 milliseconds, and sent to the Dashboard via HTTP POST; the Dashboard internally completes the protocol conversion to CAN message and the hardware distribution to the dexterous hand. In step S8, device discovery is completed and the process is blocked and waited for a successful match. After that, the camera process enters the main loop. In step S9, as an optional auxiliary step, an overlay display is provided for step S2.

[0243] Among them, the thumb deflection normal vector measurement in step S4 and the differential filtering in step S6 jointly ensure the solvability and stability of the thumb-palm motion; the REST API decoupling and polling discovery mechanism in steps S7 and S8 jointly ensure deployment robustness and cross-model reusability; and the handedness filtering in step S3 ensures the determinism of the controlled object in the scenario where both hands coexist.

[0244] Therefore, based on the descriptions of Embodiments 1 and 2 above, the relevant working mechanisms of Embodiments 1 and 2 of the present invention can be summarized as follows:

[0245] The reason why the method of this invention can restore 10 degrees of freedom of a dexterous hand using only a monocular RGB camera and a general keypoint model is that it fully utilizes the fact that the 3D keypoints output by MediaPipe (even though the z-component is relative depth rather than absolute distance) are sufficient to construct the key bone connection vectors of the hand and the palm plane. For the joint quantities of flexion and abduction, which are geometrically equivalent to the "angle between two bones", the angle can be extracted by using the vector dot product without changing the scale. For the rotation quantity of thumb opposition, which cannot be directly observed in two-dimensional projection, this invention instead measures the "parallelism between the thumb direction vector and the palm plane normal vector" - the geometric meaning of which is: parallel means that the thumb is perpendicular to the palm (opposition posture), and coplanar means that the thumb and palm are coplanar (extended posture); the normal vector defined by the cross product ensures that the measurement result does not depend on the overall posture of the hand in the camera coordinate system. Differential filtering is based on the empirical observation that "the detection confidence of distal keypoints (especially the little finger) is systematically lower than that of proximal keypoints." By decoupling the EMA coefficient of each joint into an independent adjustable parameter, "active joint responsiveness" and "distal joint steady-state smoothness" are achieved simultaneously within a single EMA structure. REST API decoupling and model / variant field filtering and matching together ensure that the vision algorithm and control stack remain independent during long-term engineering evolution, which is a key guarantee for the maintainability of this method across models and variants.

[0246] To further illustrate Embodiments 1 and 2 of the present invention fully, in conjunction with the following... Figures 3-8 The content shown is presented below, and a specific implementation method that can be applied to a visual following module product form is provided as follows:

[0247] The camera uses a standard USB monocular RGB camera with a resolution of 640×480 pixels and a frame rate of 30 frames per second. The visual perception host computer is a standard laptop computer (without a GPU). The target dexterous hand is model="L10", with one of two variants: variant ∈ {"ball_joint", "worm_gear"}. The target hand is explicitly specified by the user, with side ∈ {left, right}. The MediaPipe parameters are static_image_mode=False, max_num_hands=2, min_detection_confidence=0.7, and min_tracking_confidence=0.6. The joint-by-joint "baseline lower bound + range" parameters in step S5 are taken from the parameter table of the embodiment described in step S5-(d). The joint-by-joint EMA coefficient in step S6 is f=(0.4, 0.4, 0.4, 0.4, 0.4, 0.2, 0.4, 0.4, 0.2, 0.4). The minimum transmission interval in step S7 is Δt. min =50 milliseconds. The polling interval for step S8 is Δt. poll =3 seconds. The preview for step S9 is off by default and is enabled as needed by the debugging script.

[0248] The typical workflow between the above steps is as follows: After the camera process starts, it immediately enters step S8 for polling; when the Dashboard comes online and reports the device that matches the model and variant, the target object is constructed and the main loop of steps S1-S7 is entered; the operator makes gestures such as grasping, pinching, and clasping hands in front of the camera with the target hand, and the dexterous hand reproduces the operator's gestures within an end-to-end delay of about 50 milliseconds; if the dexterous hand accidentally loses power and then is powered on again, the failure counter of step S7 accumulates to exceed the threshold and triggers step S8 to rediscover the target, and the main loop is not interrupted.

[0249] The adjustable range of the embodiment includes: the detection and tracking confidence threshold of step S2 can be self-tuned according to the ambient lighting and background complexity (typical range [0.5, 0.85]); the joint-by-joint "baseline lower bound + range" parameter pair of step S5 can be self-tuned according to the operator's hand size and the specific model of the robot; the joint-by-joint EMA coefficient of step S6 can be tuned according to the measured noise spectrum of MediaPipe in this deployment environment (typical [0.1, 0.3] for the little finger-related joints, and [0.3, 0.6] for the other joints); the minimum transmission interval of step S7 can be self-tuned according to the Dashboard receiving capability and the CAN bus baud rate (in the embodiment, 50 milliseconds corresponds to 20 Hz); the polling interval of step S8 can be tuned according to the on-site online rhythm (typical [1, 10] seconds).

[0250] The key data structures, key algorithms, and formulas involved in Embodiments 1 and 2 of this invention can be summarized as follows:

[0251] 21 Keypoint Numbers: Integer numbers from 0 to 20 as defined by MediaPipe, including WRIST (0), THUMB_CMC / MCP / IP / TIP (1, 2, 3, 4), INDEX_MCP / PIP / DIP / TIP (5, 6, 7, 8), MIDDLE_MCP / PIP / DIP / TIP (9, 10, 11, 12), RING_MCP / PIP / DIP / TIP (13, 14, 15, 16), and PINKY_MCP / PIP / DIP / TIP (17, 18, 19, 20).

[0252] 10-DOF joint sequence: (θ flex,thumb , y thumb ,θ flex,index ,θ flex,middle ,θ flex,ring ,θ flex,pinky ,θ abd,index ,θ abd,thumb ,θ abd,pinky ,θ abd,ring ).

[0253] Three-point angle (original text) θ(a,b,c) = \arccos(clip(((ab)·(cb)) / (\lVerta-b\rVert\lVertc-b\rVert+ε),-1,1))·180 / π.

[0254] Palm plane normal vector: n = (p INDEX_MCP - p WRIST )×(p PINKY_MCP - p WRIST Unitization.

[0255] Thumb direction vector: t = (p THUMB_TIP - p THUMB_MCP Unitization.

[0256] Thumb deflection measurement: y thumb = 100 - \lvert t·n \rvert·100.

[0257] Joint-by-joint normalization: v i = clip((θ i - base i ) / span i• 100, 0, 100).

[0258] Joint-specific EMA:s i = f i ·v i + (1 - f i )·s i (-1) , f = (0.4, 0.4, 0.4, 0.4, 0.4,0.2, 0.4, 0.4, 0.2, 0.4).

[0259] Range mapping: u i = int(s i / 100·255).

[0260] Throttling condition: The difference between the current timestamp and the previous timestamp is ≥ Δt min =50 milliseconds.

[0261] Device detected pseudocode:

[0262] discover(dashboard_url, target_model, target_variant, side):

[0263] while True:

[0264] try:

[0265] data = http_get(dashboard_url + " / api / hand / devices").json()["data"]

[0266] except (NetworkError, NonOKStatus):

[0267] sleep(poll_interval); continue

[0268] candidates = [d for d in data

[0269] if d.model == target_model

[0270] and d.variant == target_variant

[0271] and d.side == side]

[0272] if candidates:

[0273] return build_target(candidates[0])

[0274] sleep(poll_interval)

[0275] Therefore, in response to the aforementioned shortcomings of existing dexterous hand teleoperation solutions, such as excessively high hardware requirements, non-standard semantic reconstruction of key points to joint space, inability of filtering coefficients to balance responsiveness and smoothness, tight coupling between vision and control, strict constraints on startup sequence, and confusion of control objects with both hands, Embodiments 1 and 2 of this invention propose a real-time dexterous hand gesture mirroring method. This method uses only a monocular RGB camera and a general hand key point detection model. It reconstructs the semantics of 21 3D key points into a 10-DOF joint space of a dexterous hand using a pure vector geometry method, uses an exponentially weighted moving average filter based on joint differentiation coefficients to balance responsiveness and steady-state smoothness, uses a characterization state transmission application programming interface to decouple visual perception and execution control, and uses a device discovery interface for polling in conjunction with left and right hand classification field filtering to achieve automatic device discovery and target hand locking.

[0276] It should be noted that Embodiments 1 and 2 of this invention mainly involve the interdisciplinary fields of computer vision, human-computer interaction (HCI), and robot teleoperation. Specifically, they involve a hand mirroring method that uses a monocular RGB camera as the acquisition source, takes 21 three-dimensional (3D) hand key points output by a general hand key point detection model (typically Google MediaPipeHands, hereinafter referred to as MediaPipe) as input, and after a set of pure vector geometric calculations and joint-based differential filtering, drives the operator's hand posture to the ten degrees of freedom (DOF) of a bionic dexterous hand in real time. In particular, it involves a method that measures the thumb bopposition (also known as thumb yaw) degree of freedom by the absolute value of the dot product of the palm plane normal vector and the thumb direction vector, and uses a visual perception module via a Representational State Transfer (REST) ​​application programming interface. A dexterous hand real-time gesture mirroring method that decouples the interface (API) from the execution control module, and uses polling of the device discovery interface with model and variant fields for filtering and matching to achieve automatic device discovery and hot-plug startup.

[0277] In summary, the technical objectives of Embodiments 1 and 2 of the present invention include:

[0278] (1) A pure vector geometric solution method is provided to reconstruct the semantics of 21 3D key points of MediaPipe into 10 degrees of freedom of dexterity hand (including 3 thumb degrees of freedom of thumb flexion, thumb abduction and thumb deflection, 4 four-finger flexion degrees of freedom of index finger, middle finger, ring finger and little finger, and 3 abduction degrees of freedom of index finger abduction, ring finger abduction and little finger abduction), so that each degree of freedom of dexterity hand has an interpretable and reproducible geometric definition;

[0279] (2) A method for solving the thumb deflection degree of freedom based on the absolute value of the dot product of the palm plane normal vector and the thumb direction vector is provided, so that the thumb-palm motion, which cannot be directly observed under two-dimensional projection, can be stably solved at 3D key points.

[0280] (3) Provides a joint-by-joint linear normalization and saturation trimming method based on the range of human physiological activities, which maps heterogeneous geometric angles with different physical dimensions to the 0-100 logic range required by the dexterous hand controller.

[0281] (4) A filtering method is provided that configures the EMA smoothing coefficient according to the joint difference. Smaller update weights are configured for the terminal joints with low detection confidence, such as little finger flexion and little finger abduction, while larger update weights are configured for active joints such as thumb and index finger, taking into account both responsiveness and steady-state smoothness.

[0282] (5) Provides a dexterous hand remote operating system architecture that decouples the visual perception module and the execution control module through REST API, so that the same vision solution can be adapted to multiple dexterous hand models and variants, and the vision algorithm and control stack can evolve independently, and the control subsystem is not affected when the vision subsystem fails;

[0283] (6) Provides an automatic device discovery and hot-plug startup method based on polling the device discovery interface of the host computer management service (hereinafter referred to as Dashboard) and filtering and matching the model field and variant field, eliminating startup order constraints;

[0284] (7) Provides a target hand screening method based on MediaPipe handedness classification results, so that when both hands appear in the camera's field of view at the same time, the control object can be explicitly locked by the user configuration to avoid accidental control.

[0285] In the description of the above embodiments of the present invention, the following key terms may be specifically involved:

[0286] 1. Monocular RGB Camera: A camera that uses only one image sensor to capture red, green and blue three-channel color images, without a matching depth sensor or a second lens; the embodiments of this invention use a common USB interface camera.

[0287] 2. Dexterous Hand: A multi-degree-of-freedom bionic robotic hand; the embodiment of this invention is a dexterous hand with 10 degrees of freedom, supporting different models and variations.

[0288] 3. Teleoperation: A human-computer interaction paradigm in which an operator drives a remote robot in real time through a certain sensing channel; the method described in this invention is vision-based teleoperation.

[0289] 4. Hand Mirroring: A teleoperation subtask in which the operator's hand posture is mirrored in real time on a remote robotic arm; the reproduction must maintain the same number of degrees of freedom and direction of motion.

[0290] 5. Landmark: Representative joint or phalangeal joint locations output by the hand detection model. MediaPipeHands outputs 21 landmarks.

[0291] 6. MediaPipe Hands: A hand detection and tracking model in the Google MediaPipe framework, outputting 21 3D key points of the hand and left and right hand classification labels.

[0292] 7. Degree of Freedom (DOF): The independently controllable motion dimension in a mechanical system; the dexterous hand of this invention has a total of 10 degrees of freedom.

[0293] 8. Flexion: The movement of bending a joint to bring the fingers inward; in this invention, it is measured by the angle between three points at the PIP joint.

[0294] 9. Abduction: The movement of adjacent fingers opening; in this invention, it is the angle between adjacent MCP key points and WRIST.

[0295] 10. Opposition: The ability of the thumb to deflect relative to the palm plane so that the tip of the thumb is opposite the tips of the other four fingers; in this invention, it is referred to as thumb deflection or thumb yaw, and is measured by the absolute value of the dot product of the palm normal vector and the thumb direction vector.

[0296] 11. Metacarpophalangeal joint (MCP): The joint connecting the metacarpal bones and the proximal phalanges of the hand.

[0297] 12. Proximal interphalangeal joint (PIP): The interphalangeal joint at the proximal end of the four fingers excluding the thumb.

[0298] 13. Distal Interphalangeal Joint (DIP): The distal interphalangeal joint of the four fingers excluding the thumb.

[0299] 14. Interphalangeal joint (IP): The only interphalangeal joint on the thumb.

[0300] 15. Carpometacarpal joint (CMC): The joint connecting the thumb metacarpal bone and the carpal bone; corresponding to keypoint 1 in MediaPipe.

[0301] 16. Wrist: MediaPipe keypoint number 0, which is the reference origin for all geometric primitives in this invention.

[0302] 17. Exponential Moving Average (EMA): A linear time-invariant recursive filter that calculates the moving average value by incrementing the moving average ... i = f i ·v i + (1 - f i )·s i (-1) Formal update; this invention configures an independent EMA coefficient f for each joint. i .

[0303] 18. handedness: The left and right classification results output by the MediaPipe model for each detected hand, including label (value "Left" or "Right") and score (confidence level).

[0304] 19. Representational State Transfer Application Programming Interface (REST API): An inter-process interface exposed via resource URLs and HTTP methods (GET, POST, etc.) based on the Hypertext Transfer Protocol; the camera process and the Dashboard process in this invention are decoupled through the REST API.

[0305] 20. Controller Area Network (CAN): Industrial control bus protocol; this invention can be used within the Dashboard process and is transparent to the camera process.

[0306] 21. Dashboard: This invention uses the internal code name for the host computer management service; it exposes two types of REST APIs externally: device discovery interface and joint logic value distribution interface, and internally it is responsible for dexterous hand device management and CAN communication.

[0307] 22. model field: A string field in the device description object returned by the Dashboard device discovery interface that identifies the dexterity hand model; used as the primary key for filtering and matching in this invention.

[0308] 23. variant field: A string field in the device description object returned by the Dashboard device discovery interface that identifies the dexterous hand mechanical variant (such as ball_joint variant, worm_gear variant, etc.); it is used as a subkey for filtering and matching in this invention.

[0309] 24. side field: A string field in the device description object returned by the Dashboard device discovery interface that identifies the body side (left / right) corresponding to the dexterous hand.

[0310] 25. interface field: A string field in the device description object returned by the Dashboard device discovery interface that identifies the name of the CAN interface where the dexterous hand is located.

[0311] 26. Open Source Computer Vision Library (OpenCV): An open-source computer vision library; this invention uses it to complete video frame acquisition, horizontal flipping, color space conversion, and preview window display.

[0312] 27. JavaScript Object Notation (JSON): A lightweight data exchange format; the HTTP POST payload of this invention is encoded in JSON.

[0313] 28. Hypertext Transfer Protocol (HTTP): The protocol that carries the REST API described in this invention.

[0314] 29. Rate Limiting (Throttling): An engineering mechanism to avoid overloading downstream processes by limiting the number of messages sent per unit time; this invention sets a minimum sending interval for HTTP POST requests.

[0315] 30. One-Euro Filter: An adaptive low-pass filter whose cutoff frequency adjusts adaptively with the rate of signal change; it can be listed as an alternative filter to the EMA.

[0316] 31. Kalman Filter: An optimal linear recursive filter based on a state-space model; it can be listed as an alternative filter to EMA.

[0317] 32. Convolutional Neural Network (CNN): Can be listed as an alternative detector to MediaPipe that can output 3D keypoints.

[0318] 33. Principal Component Analysis (PCA): Listed as an alternative geometric method for fitting the palm plane.

[0319] 34. Singular Value Decomposition (SVD): This can be listed as an alternative geometric method for fitting the palm plane.

[0320] 35. Remote Procedure Call (RPC): List the alternative communication protocols that can be used as REST APIs.

[0321] 36. Google Remote Procedure Call (gRPC): Listed as an alternative communication protocol to the REST API, carrying Protocol Buffers serialization.

[0322] 37. Server-Sent Events (SSE): Listed as alternative mechanisms to device discovery polling.

[0323] 38. Multicast Domain Name System (mDNS): Listed as an alternative mechanism for device discovery.

[0324] To further ensure that those skilled in the art can fully understand and appreciate Embodiments 1 and 2 of the present invention, Table 3, a table of Chinese and English abbreviations, is provided below:

[0325]

[0326] Table 3

[0327] In addition, the following supplementary technical explanations are provided:

[0328] 1. Hardware and Parameter Sources: The camera resolution in this embodiment is 640×480 pixels, the acquisition frame rate is 30 frames per second, the MediaPipe detection confidence threshold is 0.7 and the tracking confidence threshold is 0.6, the joint-by-joint normalized "baseline lower bound + range" parameter pair, the joint-by-joint EMA coefficient f = (0.4, 0.4, 0.4, 0.4, 0.4, 0.2, 0.4, 0.4, 0.2, 0.4, 0.4, 0.2, 0.4), and the minimum transmission interval Δt. min = 50 milliseconds, polling interval Δt poll = 3 seconds, target model string "L10", target variant string "ball_joint" / "worm_gear", etc. are typical values ​​for ease of implementation and do not constitute a limitation on the scope of protection of this invention; the values ​​can be self-tuned according to camera specifications, operator hand size, specific dexterity hand model and downstream Dashboard conventions. The 21 key point numbering topology of the MediaPipe Hands model and the handedness tag strings ("Left" / "Right") are external conventions of the MediaPipe framework itself and are not custom-defined by this invention; HTTP and REST, CAN bus baud rate and ID allocation scheme, USB camera acquisition API, etc. are industry-standard specifications and are irrelevant to the points protected by this invention.

[0329] 2. External Dependencies of MediaPipe and Dashboard: This invention does not impose specific constraints on the internal network structure, training dataset, or training hyperparameters of the MediaPipe Hands model; nor does it impose specific constraints on the specific implementation of the Dashboard process (such as the Web framework, CAN communication library, or device management data structure used). Implementers may choose the official MediaPipe release or a self-trained alternative model, and may choose any Dashboard implementation that supports REST API exposure. External dependencies serve as boundary conditions for the input and output of this invention, and their selection does not affect the coverage of the claims of this invention.

[0330] 3. Delay Budget Breakdown: Under the values ​​specified in the embodiment, the typical end-to-end delay (from the operator completing the gesture to the dexterous hand starting the action) can be broken down as follows: Step S1 acquisition and flipping (approximately 1 to 5 milliseconds); Step S2 MediaPipe inference (approximately 25 to 40 milliseconds without GPU, approximately 5 to 15 milliseconds with GPU); Steps S3 to S6 calculation, normalization, and smoothing (totaling less than 1 millisecond); Step S7 HTTP POST and Dashboard internal CAN message assembly and transmission (approximately 2 to 10 milliseconds, depending on the local loopback or LAN path); CAN bus transmission and dexterous hand motor response (approximately 5 to 15 milliseconds, depending on the motor controller implementation); the total is typically 35 to 70 milliseconds, on the same order of magnitude as the 50 millisecond throttling interval in Step S7. The time budget is illustrative, and the specific values ​​vary with hardware specifications, and do not constitute a limitation on the scope of protection of this invention.

[0331] 4. Language Independence Declaration: The core ideas of this invention are not limited to specific programming languages ​​and operating systems; the geometric calculation module, normalization module, differential smoothing module, distribution module, and device discovery module in the embodiments can be implemented using any general-purpose programming language such as Python, C++, Rust, and Go; the camera process can be deployed on Linux, Windows, or Unix-like operating systems; the Dashboard process is also not limited to a specific operating environment.

[0332] 5. Scope of Application: The methods and systems described in this invention are applicable to any engineering scenario where "monocular visual key points are used as input and a multi-degree-of-freedom bionic robotic hand is used as output," including but not limited to real-time teleoperation of dexterous hands, hand posture mirroring in gesture-driven robot teaching and embodied intelligent data acquisition; they can also be extended to the visual actuation of other multi-degree-of-freedom joint actuators (such as multi-finger prostheses and multi-finger rehabilitation training devices). The four abstractions—"pure vector geometric solution from 21 key points to 10 degrees of freedom," "thumb deflection measurement based on the absolute value of the dot product of the normal vector and the thumb direction vector," "EMA smoothing differentiated by joint," and "vision-control decoupling architecture based on polling of REST API and device discovery interface"—can all be directly reused in the above scenarios.

[0333] 6. Explanation of Relationship with Existing Infrastructure: The MediaPipe Hands model, OpenCV, HTTP / REST protocol, CAN bus protocol, common filters (EMA, Kalman, One-Euro, etc.), and common message serialization formats (JSON, Protocol Buffers, etc.) described in Chapter 2 of this invention are all existing public infrastructures in the industry. This invention does not propose protection for the above-mentioned infrastructure itself, but only for the combination and coordination mechanism of the six layers of abstraction on it: "pure vector geometry solution from 21 keypoints to 10 degrees of freedom (including thumb deflection normal vector measurement) + linear normalization and saturation pruning of physiological range of motion by joint + joint-differentiated EMA smoothing + visual-control dual-process architecture decoupled by REST API + hot-plug startup based on device discovery interface polling and model / variant field filtering and matching + target hand screening based on the handledness classification field".

[0334] Figure 10 The diagram illustrates an electronic device suitable for implementing a data processing method for real-time gesture mirroring of a dexterous hand and a real-time gesture mirroring method for a dexterous hand, according to an embodiment of the present invention.

[0335] The electronic device provided in the embodiments of the present invention includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to execute the above-described data processing method for real-time gesture mirroring of a dexterous hand and the real-time gesture mirroring method for a dexterous hand.

[0336] like Figure 10 As shown, an electronic device 1000 according to an embodiment of the present invention includes a processor 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1001 may also include onboard memory for caching purposes. The processor 1001 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0337] RAM 1003 stores various programs and data required for the operation of electronic device 1000. Processor 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Processor 1001 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 1002 and / or RAM 1003. It should be noted that the programs may also be stored in one or more memories other than ROM 1002 and RAM 1003. Processor 1001 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0338] According to an embodiment of the present invention, the electronic device 1000 may further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 may also include one or more of the following components connected to the I / O interface 1005: an input section 1006 including a keyboard, mouse, etc.; an output section 1007 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1010 as needed so that computer programs read from it can be installed into the storage section 1008 as needed.

[0339] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the above-described data processing method for real-time gesture mirroring of a dexterous hand and the real-time gesture mirroring method for a dexterous hand.

[0340] The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0341] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 1002 and / or RAM 1003 and / or one or more memories other than ROM 1002 and RAM 1003 described above.

[0342] Embodiments of the present invention also include a computer program product comprising a computer program that, when executed by a processor, implements the above-described data processing method for real-time gesture mirroring of a dexterous hand and the real-time gesture mirroring method for a dexterous hand.

[0343] The computer program includes program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the methods provided in the embodiments of the present invention.

[0344] When the computer program is executed by the processor 1001, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0345] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1009, and / or installed from a removable medium 1011. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0346] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from the removable medium 1011. When the computer program is executed by the processor 1001, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0347] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0348] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0349] Furthermore, all actions involving the acquisition of information, signals, or data in this invention are carried out in compliance with the relevant data protection laws, regulations, and policies of the country where the invention is located, and with the authorization granted by the owner of the corresponding device.

[0350] Those skilled in the art will understand that the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention can be combined or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0351] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of the invention is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A real-time gesture mirroring method applied to dexterous hands, characterized in that, include: Based on the target hand identifier data output by the hand key point detection model, the target 3D key point data corresponding to the target is generated, and normalized vector data of the corresponding dimension is generated. Based on the pre-generated joint differentiation index, a weighted moving average smoothing process is performed on the normalized vector data to generate corresponding smoothed vector data. This includes: updating the current frame smoothing value based on the normalized value of the current frame of the joint corresponding to the normalized vector data, the smoothing value of the previous frame, and the corresponding joint differentiation index; wherein the current frame smoothing value is used to construct the smoothed vector data. Based on the smoothed vector data and the target object received through the preset device discovery interface, a hypertext transfer protocol request is generated and sent to the preset control host computer to realize the real-time gesture mirroring applied to the dexterous hand; The original geometric angle vector data corresponding to the normalized vector data includes vector data that sequentially correspond to the thumb flexion angle, thumb deflection geometric measurement, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, little finger flexion angle, index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle of the preset joints of the dexterous hand. Before performing weighted moving average smoothing on the normalized vector data based on the pre-generated joint differentiation index to generate corresponding smoothed vector data, the method further includes: generating an independent exponential moving average coefficient for each joint of the normalized vector data as the joint differentiation index.

2. The method according to claim 1, characterized in that, Before generating a Hypertext Transfer Protocol request to be sent to a preset control host computer based on the smoothed vector data and the target object received through the preset device discovery interface, the method further includes: Based on a preset hypertext transfer protocol, a polling request is initiated to the target device interface of a preset control host computer at a preset polling interval, and the corresponding target device list data is obtained. Based on the pre-configured target field matching conditions, the target device list data is subjected to field filtering and matching for each device object in the target device array, and the encapsulated field data of the corresponding device is read. Perform field encapsulation on the encapsulated field data to generate a target object to be sent to the preset device discovery interface.

3. The method according to claim 1, characterized in that, The step of generating a hypertext transfer protocol request to be sent to a preset control host computer based on the smoothed vector data and the target object received through a preset device discovery interface includes: Each logical value of the smoothed vector data is mapped component by component to an integer logical value corresponding to a preset program interface of a preset control host computer; The integer logical values ​​are grouped semantically and assembled into the first dictionary data and the second dictionary data; The first dictionary data includes integer logical values ​​related to the thumb flexion angle, thumb deflection geometry, index finger flexion angle, middle finger flexion angle, ring finger flexion angle, and little finger flexion angle; the second dictionary data includes integer logical values ​​related to the index finger abduction angle, thumb abduction angle, little finger abduction angle, and ring finger abduction angle.

4. The method according to claim 3, characterized in that, The process of generating a hypertext transfer protocol request to be sent to a preset control host computer based on the smoothed vector data and the target object received through the preset device discovery interface further includes: Based on a preset control host computer that performs throttling at the minimum transmission interval, the first dictionary data, the second dictionary data, and the target object are used to generate the hypertext transfer protocol request in a preset object identifier format.

5. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method of any one of claims 1 to 4.

6. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 4.

7. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 4.

Images