Humanoid robot dexterous hand control method and system based on RGB image and medium

Abstract

The invention belongs to the technical field of dexterous hand control, and discloses a humanoid robot dexterous hand control method and system based on an RGB image and a medium, and the method comprises the steps: obtaining continuous action RGB image frames of a human hand; performing cutting and posture detection processing on the continuous action RGB image frame of the human hand to obtain a plurality of hand key points; in a pre-constructed 3D hand model, calculating a preset number of joint point angles of the 3D hand model based on the plurality of hand key points; and the joint point angle is input into a pre-constructed joint angle mapping model of the dexterous hand, the joint angle of the dexterous hand is calculated, and the joint angle of the dexterous hand is sent to a controller of the humanoid robot, so that the controller of the humanoid robot drives the humanoid robot to execute corresponding actions. The method has the advantages of high precision, high real-time performance, high adaptability and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of dexterous hand control, and in particular relates to a method, system and medium for controlling the dexterous hand of a humanoid robot based on RGB images. Background Art

[0002] Dexterous hands are an important component of humanoid robots. They belong to the intersection of cutting-edge technologies such as deep bionics, flexible perception, micro-electromechanical systems, and high-performance materials. They are a key breakthrough in achieving a technological revolution in intelligent robots.

[0003] Teleoperated dexterous hands enable operators to remotely control robotic arms and dexterous hands to complete complex tasks. Currently, the control systems of teleoperated dexterous hands mainly rely on hardware devices such as sensors, glove input devices, and inertial measurement units (IMUs) to control the movement of the robot hand by capturing the operator's hand movements. These systems usually use the following technologies: 1. Sensor-based teleoperation system Traditional teleoperation dexterous hand control systems rely on sensors (such as force sensors, displacement sensors, etc.) to accurately capture the operator's hand movements or force feedback. Common methods include wearing gloves (such as CyberGlove, data gloves, etc.) or sensor kits (such as Flex Sensors, inertial measurement units IMU). These devices can provide relatively accurate physical signal feedback, but are usually costly and impose certain restrictions on the operator's comfort and flexibility.

[0004] 2. Gesture recognition system based on video and depth image In recent years, with the advancement of computer vision technology, gesture recognition technology based on RGB images or depth images has been widely used in teleoperation systems. The operator's hand gestures are collected by the camera and converted into control signals for the robot's hands in combination with image processing algorithms. Commonly used technologies include deep learning methods (such as convolutional neural networks (CNNs)) and traditional image processing technologies (such as HOG, SIFT, etc.). This method effectively reduces the dependence on dedicated hardware and improves the flexibility of the system.

[0005] The existing teleoperated dexterous hand control system still has at least the following problems: 1. Strong hardware dependence and high cost: Many traditional teleoperation systems rely on expensive sensors or dedicated hardware (such as gloves, sensor belts, etc.). These devices are often bulky, non-portable, and have a significant impact on the operator's dexterity and comfort. For example, data gloves are not only expensive, but also may limit the operator's hand freedom if used for a long time. 2. Poor real-time performance and high latency: Although visual gesture recognition technology has gradually matured, many image-based gesture recognition systems still have a latency of more than 120ms due to the large amount of computation required by image processing algorithms, resulting in a detection error of more than ±2mm. This latency may lead to jerky movements or misoperation in teleoperation applications, especially in tasks requiring high precision and real-time performance (such as medical surgery, precision assembly, etc.). 3. Insufficient multi-frame consistency: Current gesture recognition methods based on RGB images usually rely only on a single frame image to determine the hand posture, and fail to fully utilize the multi-frame information in the time series; especially when the hand remains still, there is a lack of consistent processing of continuous frame data, which causes the traditional single-frame processing method to produce an angle fluctuation of ±2.5° under static gestures, resulting in unstable recognition results, affecting the accuracy and smoothness of the operation. Summary of the invention

[0006] The purpose of the present invention is to provide a humanoid robot dexterous hand control method, system and medium based on RGB images, so as to solve the problems of strong hardware dependence, high cost, poor real-time performance, high delay and insufficient multi-frame consistency in the prior art.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for controlling a dexterous hand of a humanoid robot based on an RGB image, the method comprising: Obtain continuous motion RGB image frames of human hands; The continuous motion RGB image frames of the human hand are cropped and processed for posture detection to obtain several key points of the hand; In a pre-built 3D hand model, angles of a preset number of joint points of the 3D hand model are calculated based on a number of hand key points; The joint point angles are input into a pre-built joint angle mapping model of the dexterous hand, the joint angles of the dexterous hand are calculated, and the joint angles of the dexterous hand are sent to the controller of the humanoid robot so that the controller of the humanoid robot drives the humanoid robot to perform corresponding actions.

[0008] Preferably, before inputting the joint point angles into a pre-built joint angle mapping model of a dexterous hand, the method further comprises: pre-processing the joint point angles, wherein the pre-processing comprises: sliding mean filtering and normalization.

[0009] Preferably, the joint point angles are preprocessed, including: Normalizing the joint point angles to obtain normalized joint point angles; The normalized joint angles are subjected to sliding mean filtering to obtain the joint angles after preprocessing, and the joint angles after preprocessing are used as input data for the joint angle mapping model of the dexterous hand.

[0010] Preferably, the continuous motion RGB image frames of the human hand are cropped and processed with posture detection to obtain several key points of the hand, including: Detecting the hand position in the continuous action RGB image frame based on a first preset algorithm, and cropping the continuous action RGB image frame according to the detected hand position to obtain a cropped image frame; The cropped image frame is subjected to posture detection based on a second preset algorithm to obtain a number of hand key points.

[0011] Preferably, the window length of the sliding mean filtering process is 5 frames.

[0012] Preferably, the first preset algorithm is a ViTPose algorithm, and the second preset algorithm is a HaMeR algorithm.

[0013] Preferably, the method further comprises: constructing a joint angle mapping model of a dexterous hand, comprising: Obtain angle sample data of human finger joints; The angle sample data of human finger joints are fitted based on the Lagrange interpolation method to obtain the joint angle mapping model of dexterous hands.

[0014] Preferably, the 3D hand model is constructed based on the MANO model.

[0015] In a second aspect, the present invention provides a humanoid robot dexterous hand control system based on RGB images, which is used to implement the above-mentioned humanoid robot dexterous hand control method based on RGB images, and the system comprises: An image acquisition module is used to acquire continuous motion RGB image frames of a human hand; The image processing module is used to perform cropping and posture detection processing on the continuous action RGB image frames of the human hand to obtain a number of key points of the hand; An angle calculation module, used to calculate the angles of a preset number of joint points of the 3D hand model based on a number of hand key points in constructing the 3D hand model; The angle mapping module is used to input the joint point angles into a pre-built joint angle mapping model of the dexterous hand, calculate the joint angles of the dexterous hand, and send the joint angles of the dexterous hand to the controller of the humanoid robot so that the controller of the humanoid robot drives the humanoid robot to perform corresponding actions.

[0016] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned RGB image-based humanoid robot dexterous hand control method when executing the computer program.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned RGB image-based humanoid robot dexterous hand control method.

[0018] Beneficial effects: 1. High precision: The present invention constructs a joint angle mapping model of a dexterous hand, and realizes high-precision mapping of human finger movements to robot finger movements through the joint angle mapping model of the dexterous hand; 2. Real-time performance: The present invention can process human finger joint data in real time and quickly calculate the joint angles of robot fingers and other data. The delay within 80ms is almost imperceptible. 3. Adaptability: The model parameters can be adjusted according to different robot palm structures and have good adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present invention, but do not constitute a limitation on the embodiments of the present invention. In the accompanying drawings: Figure 1 is a flow chart of a method for controlling a dexterous hand of a humanoid robot based on RGB images provided by one embodiment of the present invention; Figure 2 is a schematic diagram of key points of a hand provided by an embodiment of the present invention; Figure 3 It is a block diagram of a humanoid robot dexterous hand control system based on RGB images provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0021] Embodiment 1 Figure 1FIG. 1 is a flow chart of a method for controlling a humanoid robot's dexterous hand based on RGB images provided by an embodiment of the present invention. Figure 1 As shown, this embodiment provides a method for controlling a humanoid robot dexterous hand based on RGB images. The method can be executed by a computer device or a virtual machine with certain computing resources, such as a personal computer or a smart phone, or a virtual machine. The method includes: Step S10: Acquire continuous motion RGB image frames of human hands; In this embodiment, the continuous motion of human hands is captured by an RGB sensor, and the continuous motion of the hands needs to be collected to extract multiple frames of continuous motion RGB image frames.

[0022] In this embodiment, RGB represents the colors of three channels: red (R), green (G), and blue (B). The RGB sensor uses an RGB camera, which does not require the operator to wear any additional equipment, thus avoiding the limitation of the operator's flexibility in traditional systems. The operator only needs to make natural hand movements within the field of view of the camera, and the system can recognize and respond in real time, greatly improving the ease of use and convenience of the system.

[0023] Step S20: cropping and performing posture detection processing on the continuous motion RGB image frames of the human hand to obtain a number of hand key points.

[0024] In this embodiment, since the human hand only occupies a very small part of the entire human body, it is difficult to converge the loss function by directly using the image to train the model. Therefore, it is necessary to crop the acquired continuous action RGB image frames according to the hand image position of the human hand.

[0025] As a further optimization of this embodiment, the continuous motion RGB image frames of the human hand are cropped and processed with posture detection to obtain several key points of the hand, including: Step S201: detecting the hand position in the continuous action RGB image frame based on a first preset algorithm, and cropping the continuous action RGB image frame according to the detected hand position to obtain a cropped image frame.

[0026] Among them, the first preset algorithm is preferably the ViTPose algorithm. The ViTPose algorithm is a human / animal posture estimation algorithm based on the pure visual Transformer (ViT) architecture, developed by the ViTAE-Transformer team. The algorithm simplifies model design and utilizes large-scale pre-training strategies.

[0027] Introduction to ViTPose algorithm: 1. Core idea: 1. Pure ViT architecture: The ViTPose algorithm abandons the hybrid design of traditional CNN (convolutional neural network) and ViT (‌Vision Transformer, abbreviated as ViT, is a neural network model based on the Transformer architecture) and directly uses non-hierarchical ViT as the backbone network. The input image is cut into small patches, and global features are extracted through a multi-layer Transformer encoder after linear projection. This design breaks through the limitation of traditional methods that rely on CNN feature extraction, proving that pure ViT can achieve high-precision posture estimation with sufficient computing power.

[0028] 2. Lightweight decoder: Bilinear interpolation upsampling: directly interpolate the low-resolution feature map output by ViT to generate a key point heatmap; Transposed convolution decoding: Increase the resolution through 3 layers of transposed convolution (256×192→64×48→128×96→256×192); Experiments show that ViT's strong representation capability makes the performance of a simple decoder close to that of a complex design, with an accuracy improvement of 42% compared to similar CNN methods.

[0029] 2. Technical advantages 1. Strong scalability: Flexible model scale: supports four specifications of ViT-S / B / L / H, and the parameter volume is adjustable from 20M to 1B; Resolution adaptation: The input image size is compatible with 256×192 to 1024×768, and the feature map resolution supports dynamic adjustment from 1 / 8 to 1 / 16.

[0030] 2. Training flexibility: Pre-training strategy: Supports ImageNet supervised pre-training and MAE self-supervised pre-training, the latter only requires 500,000 pose data to achieve the same effect Knowledge transfer: Knowledge transfer from large models to small models is achieved through "knowledge tokens", which improves the performance of small models while reducing training resource consumption by 60%. 3. Multi-task adaptation: ViTPose+ upgrade: Introducing the Mixed Experts (MoE) mechanism, solving the conflict problem during joint training of multiple datasets (human / animal / hand key points) through shared + exclusive FFN modules, and improving cross-domain accuracy by up to 15% in 12 model variants; Supports 2D / 3D posture estimation: Through spherical polar coordinate system conversion, it can process sparse lidar point cloud data and achieve 0.5mm level accuracy in medical rehabilitation scenarios.

[0031] Step S201: performing posture detection on the cropped image frame based on a second preset algorithm to obtain a number of hand key points.

[0032] Among them, the second preset algorithm is preferably the HaMeR (Hand Mesh Regression, referred to as HaMeR) algorithm, which can quickly and efficiently extract accurate three-dimensional hand information from RGB images; the advantage of this algorithm lies in its efficient processing capability of complex hand movements, which can greatly reduce computing time and resource consumption while maintaining high recognition accuracy, and has significant low-latency performance in hand gesture recognition.

[0033] The HaMeR algorithm has an excellent inference speed. This embodiment achieves real-time performance of less than 80 milliseconds, ensuring the instant response and smoothness of gestures during teleoperation. Whether in precision operation scenarios such as medical surgery and industrial assembly, or in daily teleoperation, the system can capture and respond to the operator's hand movements in real time, ensuring smooth operation and low latency.

[0034] Step S30: In the pre-constructed 3D hand model, a preset number of joint point angles of the 3D hand model are calculated based on a number of hand key points.

[0035] In this embodiment, the 3D hand model is constructed using the MANO model, which is a 3D parametric model based on the human hand. The parameters of the MANO model include 21 key points, each of which has 3 degrees of freedom, 3 degrees of freedom of rotation at the wrist, and 10 parameters about the hand shape. According to these parameters, a 3D hand model can be generated based on the MANO model. The 3D hand model contains 778 vertices and 1538 faces, and based on the 21 key points, a complete hand chain, or a forward kinematic tree, is constructed.

[0036] This embodiment uses the HaMeR algorithm to detect the model parameters corresponding to the hand posture in the cropped image frame frame by frame, such as Figure 2 As shown, the 21 key points detected are used as the hand key points, and the set of hand key points K for , where n=21.

[0037] After detecting 21 key points of the hand, the angles of the joints of the 3D hand model are calculated. At this time, the number of joints of the 3D hand model is 15, and the preset number in this embodiment is 15. At this time, the joints can be represented by a set, which contains the indexes of all joints. Represents a set of joint points.

[0038] but ; Among them, it is assumed that the angle of the joint point is , where m=15.

[0039] but ; In the formula, Indicates i Joints angle, i=0,1,2,…,14, represents the cosine function, and For joint points The two key points connected, Indicates key points With joint points The vector formed, Indicates joint points With key points The vector composed.

[0040] Step S40: input the joint point angles into a pre-built joint angle mapping model of the dexterous hand, calculate the joint angles of the dexterous hand, and send the joint angles of the dexterous hand to the controller of the humanoid robot so that the controller of the humanoid robot drives the humanoid robot to perform corresponding actions.

[0041] In this embodiment, due to the structural differences between human fingers and robot fingers, a mapping model needs to be constructed; therefore, the method for constructing the joint angle mapping model of the dexterous hand is: Step a10: Obtain angle sample data of human finger joints; Step a20: fitting the angle sample data of the human finger joints based on the Lagrange interpolation method to obtain a joint angle mapping model of the dexterous hand.

[0042] The joint angle mapping model of the dexterous hand includes the mapping models of multiple fingers. Each finger has three joints, and the angle of each joint is , then the expression of the joint angle mapping model of a finger is: ; ; ; In the formula, , and They are all mapping functions, which can be obtained by fitting the angle sample data of the human finger joints using the Lagrange interpolation method.

[0043] Before the 15 joint point angles are input into the joint angle mapping model of the dexterous hand, the 15 joint point angles need to be subjected to sliding mean filtering and normalization processing.

[0044] As a further optimization of this embodiment, the joint point angles are preprocessed, including: Step b10: normalize the joint point angles to obtain normalized joint point angles.

[0045] In this embodiment, the normalization process adopts the minimum-maximum normalization process, and the function expression of the normalization process is: ; In the formula, is the normalized joint angle, θ Indicates any one of the 15 joint angles. and Respectively represent the data collected within a period of time θ The minimum and maximum values ​​of the joint angle.

[0046] Step b20: Perform sliding mean filtering on the normalized joint point angles to obtain the joint point angles after preprocessing, and use the joint point angles after preprocessing as input data for the joint angle mapping model of the dexterous hand.

[0047] In this embodiment, the normalized joint point angles need to be processed by sliding mean filtering, and the window length of the sliding mean filtering is 5 frames.

[0048] The window length used is , then the function expression of sliding mean filtering is: ; is the normalized joint point angle corresponding to the m-th cropped image frame, is the joint point angle of the mth frame output after sliding mean filtering, that is, the joint point angle after preprocessing. Indicates rounding down.

[0049] By inputting the joint point angles after preprocessing into the joint angle mapping model of the dexterous hand, the joint angles of the dexterous hand corresponding to the human hand movements can be calculated, and the joint angles of the dexterous hand are sent to the humanoid robot. The controller of the humanoid robot will drive the fingers of the robot's dexterous hand to perform the corresponding human hand movements.

[0050] As a further optimization of this embodiment, this embodiment also combines an adaptive mechanism to adjust the response of the teleoperation system in real time, ensuring that the system always maintains a high degree of stability and adaptability in a dynamic environment. The system collects the data of the finger opening and closing limits of the user's fists and palms for multiple times within a certain period of time (for example, within 10 seconds), analyzes the palm size and opening degree of different users, and then automatically optimizes the joint angle mapping model of the dexterous hand to adapt to individual differences. In this way, the system can accurately adjust the gesture mapping according to each user's palm characteristics and operating habits, thereby improving the operating accuracy and adaptability.

[0051] This adaptive mechanism enables the system to optimize in real time according to environmental changes and user operation requirements. Whether in complex environments or with changes in user operations, the system can dynamically adjust to ensure a smooth and accurate operation experience and improve overall reliability and response speed. To further improve stability, the system performs weighted averaging on multiple frames of data before sending instructions to the dexterous hand, thereby achieving stability in hand pose mapping and avoiding the instability caused by instantaneous data fluctuations.

[0052] Embodiment 2 Figure 3 FIG. 1 is a block diagram of a humanoid robot dexterous hand control system based on RGB images provided by one embodiment of the present invention. Figure 2 As shown, this embodiment provides a humanoid robot dexterous hand control system based on RGB images, which is used to implement the humanoid robot dexterous hand control method based on RGB images in embodiment 1. The system includes: An image acquisition module is used to acquire continuous motion RGB image frames of a human hand; The image processing module is used to perform cropping and posture detection processing on the continuous action RGB image frames of the human hand to obtain a number of key points of the hand; An angle calculation module, used to calculate the angles of a preset number of joint points of the 3D hand model based on a number of hand key points in constructing the 3D hand model; The angle mapping module is used to input the joint point angles into a pre-built joint angle mapping model of the dexterous hand, calculate the joint angles of the dexterous hand, and send the joint angles of the dexterous hand to the controller of the humanoid robot so that the controller of the humanoid robot drives the humanoid robot to perform corresponding actions.

[0053] This embodiment also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for controlling the dexterous hand of a humanoid robot based on RGB images in Embodiment 1 is implemented.

[0054] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for controlling the dexterous hand of a humanoid robot based on RGB images in the first embodiment is implemented.

[0055] The present invention constructs a joint angle mapping model of a dexterous hand, and realizes high-precision mapping of human finger movements to robot finger movements through the joint angle mapping model of the dexterous hand; the present invention can process human finger joint data in real time, and quickly calculate data such as the joint angles of the robot fingers, and the delay within 80ms is almost imperceptible to people; and the model parameters can be adjusted according to different robot palm structures, with good adaptability.

[0056] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0057] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0058] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A humanoid robot dexterous hand control method based on RGB image, characterized in that: The method comprises: Obtain continuous motion RGB image frames of human hands; The continuous motion RGB image frames of the human hand are cropped and processed for posture detection to obtain several key points of the hand; In constructing the 3D hand model, a preset number of joint point angles of the 3D hand model are calculated based on a number of hand key points; The joint point angles are input into a pre-built joint angle mapping model of the dexterous hand, the joint angles of the dexterous hand are calculated, and the joint angles of the dexterous hand are sent to the controller of the humanoid robot so that the controller of the humanoid robot drives the humanoid robot to perform corresponding actions.

2. The method for controlling a humanoid robot's dexterous hand based on RGB images according to claim 1, characterized in that: Before inputting the joint point angles into the pre-built joint angle mapping model of the dexterous hand, the method further comprises: pre-processing the joint point angles, wherein the pre-processing comprises: sliding mean filtering and normalization.

3. The method for controlling a humanoid robot's dexterous hand based on RGB images according to claim 2, characterized in that: Preprocess the joint angles, including: Normalizing the joint point angles to obtain normalized joint point angles; The normalized joint angles are subjected to sliding mean filtering to obtain the joint angles after preprocessing, and the joint angles after preprocessing are used as input data for the joint angle mapping model of the dexterous hand.

4. The method for controlling a humanoid robot's dexterous hand based on RGB images according to claim 3 is characterized in that: The window length of the sliding mean filtering process is 5 frames.

5. The method for controlling a humanoid robot dexterous hand based on RGB images according to claim 1, characterized in that: The continuous motion RGB image frames of the human hand are cropped and processed for posture detection to obtain several key points of the hand, including: Detecting the hand position in the continuous action RGB image frame based on a first preset algorithm, and cropping the continuous action RGB image frame according to the detected hand position to obtain a cropped image frame; The cropped image frame is subjected to posture detection based on a second preset algorithm to obtain a number of hand key points.

6. The method for controlling a dexterous humanoid robot hand based on RGB images according to claim 1, characterized in that: The method further comprises: constructing a joint angle mapping model of a dexterous hand, comprising: Obtaining angle sample data of human finger joints and joint angle sample data of dexterous hands; Based on the Lagrange interpolation method, the angle sample data of human finger joints and the joint angle sample data of dexterous hands are fitted to obtain the joint angle mapping model of dexterous hands.

7. The method for controlling a humanoid robot's dexterous hand based on RGB images according to claim 1, characterized in that: The 3D hand model is constructed based on the MANO model.

8. A humanoid robot dexterous hand control system based on RGB images, used to implement the humanoid robot dexterous hand control method based on RGB images according to any one of claims 1 to 7, characterized in that: The system comprises: An image acquisition module is used to acquire continuous motion RGB image frames of a human hand; The image processing module is used to perform cropping and posture detection processing on the continuous action RGB image frames of the human hand to obtain a number of key points of the hand; An angle calculation module, used to calculate the angles of a preset number of joint points of the 3D hand model based on a number of hand key points in constructing the 3D hand model; The angle mapping module is used to input the joint point angles into a pre-built joint angle mapping model of the dexterous hand, calculate the joint angles of the dexterous hand, and send the joint angles of the dexterous hand to the controller of the humanoid robot so that the controller of the humanoid robot drives the humanoid robot to perform corresponding actions.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for controlling a dexterous hand of a humanoid robot based on RGB images according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for controlling a dexterous hand of a humanoid robot based on RGB images described in any one of claims 1 to 7 is implemented.

Images