Method for accurately controlling dexterous hand of humanoid robot by utilizing visual identification technology

Abstract

The invention discloses a method for completing precise control of a dexterous hand of a humanoid robot by using a visual identification technology, and relates to the technical field of humanoid robot joint control, and the method comprises the following steps: S1, constructing a visual perception architecture: constructing a layered visual architecture through cooperation of a sensor to meet the perception demand of covering global to local, and meanwhile, the precision requirements of different operation scenes are met. According to the method for completing precise control over the dexterous hand of the humanoid robot through the visual recognition technology, global coverage and local fineness can be considered through the arranged layered visual perception architecture, full-link perception from operation space positioning to target grabbing point fine recognition and then to fingertip action high-speed monitoring is achieved, and the accuracy of the control over the dexterous hand of the humanoid robot is improved. The global space requirement of robot moving operation is met, the submillimeter-level grabbing precision of the dexterous hand is guaranteed, and through the design of closed-loop control and multi-mode fusion, self-adaptive high-precision grabbing of the humanoid robot is achieved, and real-time monitoring of fingertip deformation and self-adaptive adjustment of gripping force are achieved.

Description

Technical Field

[0001] This invention relates to the field of humanoid robot joint control technology, specifically a method for achieving precise control of a humanoid robot's dexterous hand using visual recognition technology. Background Technology

[0002] Humanoid robots are intelligent machines with human-like forms that can autonomously perform complex tasks in human environments. Their core lies in a closed-loop system of embodied intelligence, encompassing perception, decision-making, and execution. Currently, they are transitioning from laboratory settings to industrial and service applications, and are in a transitional phase from G2 (partially autonomous interaction) to G3 (complex scenario adaptation). The control units for humanoid robots primarily include language and body movements, with precise control of the hands requiring specialized methods. However, current methods for precise control of humanoid robot dexterity hands still have the following shortcomings: Existing methods for precise control of humanoid robot dexterous hands mostly rely on a single visual model for manipulation. This leads to incomplete observation of objects and susceptibility to environmental interference, which in turn interferes with the accurate grasping of subsequent objects. Furthermore, the inability to precisely control the grasping force results in low hand dexterity, making it prone to damaging objects during grasping. Additionally, the robot has low perception of the grasped objects when moving them, making it unable to adapt to moving scenarios. Moreover, it cannot predict changes in object deformation and contact state, resulting in a rigid grasping strategy. Summary of the Invention

[0003] The purpose of this invention is to provide a method for precise control of the dexterous hand of a humanoid robot using visual recognition technology, so as to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for precise control of a humanoid robot's dexterous hand using visual recognition technology, comprising the following steps: S1. Visual perception architecture construction: A layered visual architecture is built by using sensors to meet the perception needs from global to local coverage, while adapting to the accuracy requirements of different operating scenarios. The spatial coordinate transformation formula is as follows: ( : Extrinsic parameter matrix, (internal parameter matrix), where The expression: ( , Camera's x and y axis focal lengths , (Camera principal point coordinates). Due to the different sampling frequencies of the various sensors, time differences occur in the data, which requires compensation through linear interpolation. ( Completed data for the target time t. , Adjacent sampling times , (raw data); S2. Computational Model Building: A deep learning model is used to understand the target, providing accurate input for the subsequent control of the humanoid robot's dexterous hand; S3. Control Link Design: Construct a closed-loop control system from perception to planning to execution to feedback, and combine visual guidance and force correction to improve accuracy. After the control link design is completed, input simulation data to conduct pre-operation simulation training of the humanoid robot's multi-finger grasping postures to ensure that the humanoid robot can operate normally in multi-finger grasping postures during subsequent actual operations. Global visual localization of target object pose: Scene images are acquired via a camera, and the pose of the target object relative to the robot's base coordinate system is obtained based on a feature point matching algorithm. The calculation formula is as follows: Target object position coordinates: ( , : The coordinates of the feature points of the target object in the pixel coordinate system. , Camera optical center pixel coordinates Camera focal length The depth of the feature point from the camera coordinate system. (The inverse transformation matrix from the camera coordinate system to the robot base coordinate system), and the fingertip deformation distance detection formula: ( Visual calibration coefficients Pixel displacement of visual feature points (calibrated intercept); S4. Multimodal fusion: By establishing a mapping model of magnetic field strength, distance, and force, the fusion of vision and force perception is achieved, enabling the deformation of the humanoid robot's fingertips to be monitored by visual microsensors. Combined with force data, adaptive grasping is achieved, thereby automatically adjusting the grip force when the object slips. Contact distance derived from magnetic field Contact force derived from visual reasoning Establish a fusion model and contact distance The smaller the area, the larger the contact area, and the greater the actual gripping force for the same deformation. The corrected formula is: when = hour, = ;when When =0, =2 in: The initial distance between the fingertip and the object when the fingertip is not deformed (calibrated value). Sliding critical force, The distance between the fingertip and the object. Fingertip shape variable : Finger tip elasticity coefficient; Meanwhile, the critical condition for an object to slide is: the frictional force generated by the gripping force is greater than or equal to the component of the object's weight along the gripping surface, that is: ( The weight of an object : Capture angle, : coefficient of friction (Sliding critical force), and when the visual sensor detects that the object is sliding (relative displacement) When the current grip strength is less than the critical force, calculate the difference between the current grip strength and the critical force, and adjust the grip strength accordingly. ,in Adjustment coefficient ( =1.1 1.3, to prevent over-adjustment); S5. Real-time data acquisition and preprocessing: Real-time acquisition of visual data is carried out through the constructed visual perception system, and the acquired multi-angle visual data is cleaned and transformed to remove invalid data (such as: image blurring, duplication and measurement noise, etc.). S6. Motion Execution and Feedback Data Acquisition: The humanoid robot automatically completes the work action by pressing the command and collects feedback data from various parts during operation to ensure that in case of accidental failure, the source can be traced or the system can be upgraded (to improve the problem found).

[0005] Furthermore, the layered visual architecture in step S1 mainly consists of global vision (LiDAR, installed on the head or torso, with a positioning error of ≤5cm), local vision (binocular camera, installed on the wrist or palm, with a reconstruction accuracy of ≤0.5mm), and micro vision (miniature high-speed camera, installed on the side of the fingertip, with a frame rate of ≥200fps).

[0006] Furthermore, step S2 includes the following sub-steps: S21. Object Detection and Segmentation: Use the YOLOv8 model to output the target bounding box, semantic category and mask, so as to achieve comprehensive feature extraction in the scene (including feature extraction in occluded scenes). S22, 6D pose and depth estimation: Combining the PointNet++ algorithm, the three-dimensional position and pose of the target are calculated from the point cloud data, and the positioning error needs to be controlled within the sub-millimeter level; S23. Visual motion mapping: Key points of the human hand are extracted through OpenPose, then mapped to the dexterous hand joint space through motion redirection, and a reference trajectory is automatically generated. S24. Dynamic Prediction and Decision-Making: Introducing the ViTacFormer modal model, integrating visual and tactile temporal features, predicting changes in contact state, and optimizing the grasping strategy.

[0007] Furthermore, in step S3, the pre-operation generates the operation space path by global vision, and local vision completes the fine positioning of the target and the planning of the grasping point. At the same time, based on the target size, material and shape, the optimal multi-finger grasping posture is output through the large visual model.

[0008] Furthermore, in step S4, the IMU angular velocity and visual feature points are fused using the VINS-Fusion algorithm to control the dynamic positioning error to the centimeter level, so as to meet the operational requirements of humanoid robot movement.

[0009] Furthermore, in step S4, a pre-training and physical fine-tuning framework is also required, which uses human video data transfer learning to improve the model's ability to adapt to new objects or scenes.

[0010] Furthermore, in step S4, after the vision and force perception are fused, the deformation trend of the object within 100ms is automatically predicted, and the grasping strategy is adjusted in advance according to the predicted deformation trend to ensure that the object will not fall off when the humanoid robot hand moves the object.

[0011] Furthermore, in step S5, in addition to collecting visual information data, a magnetic tactile sensor is used to monitor the grasping contact state to ensure that the force exerted by the robot's fingertips when grasping objects can be adjusted in real time, thus avoiding damage caused by excessive force or loss of objects due to insufficient force when grasping flexible or fragile objects.

[0012] Furthermore, in step S6, when an offset (such as a grasping offset > 1 mm) is detected in the real-time feedback data from visual detection, joint fine-tuning and force control parameter updates are automatically triggered to ensure operational accuracy.

[0013] This invention provides a method for precise control of a humanoid robot's dexterous hand using visual recognition technology, which has the following beneficial effects: 1. This invention, through its layered visual perception architecture, enables the method to balance global coverage and local precision, achieving end-to-end perception from operation space positioning to precise identification of target grasping points, and then to high-speed monitoring of fingertip movements. This satisfies the global spatial requirements of robot mobile operations while ensuring sub-millimeter precision in dexterous hand grasping. Furthermore, the closed-loop control and multimodal fusion design enables adaptive high-precision grasping of humanoid robots, achieving real-time monitoring of fingertip deformation and adaptive adjustment of grip force. When the object slides or the grasping offset is greater than 1mm, joint fine-tuning and parameter updates are automatically triggered, effectively avoiding grasping failure.

[0014] 2. This invention introduces the VINS-Fusion algorithm to fuse IMU and visual feature points, controlling dynamic positioning errors to the centimeter level. It adapts to the operational needs of humanoid robots during movement. It adopts a pre-training and physical fine-tuning framework and uses human video data transfer learning to significantly improve the model's adaptability to new objects and scenes, reducing the deployment cost of new tasks. Furthermore, it monitors the grasping contact state through a magnetic tactile sensor and adjusts the fingertip force in real time, solving the pain points of traditional rigid control where excessive force damages flexible and fragile items, while insufficient force causes them to fall off. This expands the application scenarios of dexterous hands, such as precision electronic component assembly and fresh food sorting. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall operation process of a method for precise control of a humanoid robot's dexterous hand using visual recognition technology, according to the present invention. Detailed Implementation

[0016] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0017] like Figure 1 As shown, a method for precise control of a humanoid robot's dexterous hand using visual recognition technology includes the following steps: S1. Visual Perception Architecture Construction: A layered visual architecture is constructed using sensors to meet the perception requirements from global to local coverage, while adapting to the accuracy requirements of different operating scenarios. The layered visual architecture in step S1 mainly consists of global vision (LiDAR, installed on the head or torso, with a positioning error of ≤5cm), local vision (binocular camera, installed on the wrist or palm, with a reconstruction accuracy of ≤0.5mm), and micro-vision (miniature high-speed camera, installed on the side of the fingertip, with a frame rate of ≥200fps); the spatial coordinate transformation formula is as follows: ( : Extrinsic parameter matrix, (internal parameter matrix), where The expression: ( , Camera's x and y axis focal lengths , (Camera principal point coordinates). Due to the different sampling frequencies of the various sensors, time differences occur in the data, which requires compensation through linear interpolation. ( Completed data for the target time t. , Adjacent sampling times , (raw data); S2. Computational Model Building: A deep learning model is used to achieve target understanding, providing precise input for the subsequent control of the humanoid robot's dexterous hand. Step S2 includes the following sub-steps: S21. Object Detection and Segmentation: Use the YOLOv8 model to output the target bounding box, semantic category and mask, so as to achieve comprehensive feature extraction in the scene (including feature extraction in occluded scenes). S22, 6D pose and depth estimation: Combining the PointNet++ algorithm, the three-dimensional position and pose of the target are calculated from the point cloud data, and the positioning error needs to be controlled within the sub-millimeter level; S23. Visual motion mapping: Key points of the human hand are extracted through OpenPose, then mapped to the dexterous hand joint space through motion redirection, and a reference trajectory is automatically generated. S24. Dynamic Prediction and Decision-Making: Introducing the ViTacFormer modal model, integrating visual and tactile temporal features, predicting changes in contact state, and optimizing the grasping strategy; S3. Control Link Design: Construct a closed-loop control system from perception to planning to execution to feedback, and combine visual guidance and force correction to improve accuracy. After the control link design is completed, input simulation data to conduct pre-operation simulation training of the humanoid robot's multi-finger grasping postures to ensure that the humanoid robot can operate normally in multi-finger grasping postures during subsequent actual operations. In step S3, the pre-operation generates the operation space path by global vision, and local vision completes the fine positioning of the target and the planning of the grasping point. At the same time, based on the target size, material and shape, the optimal multi-finger grasping posture is output through the large visual model. Global visual localization of target object pose: Scene images are acquired via a camera, and the pose of the target object relative to the robot's base coordinate system is obtained based on a feature point matching algorithm. The calculation formula is as follows: Target object position coordinates: ( , : The coordinates of the feature points of the target object in the pixel coordinate system. , Camera optical center pixel coordinates Camera focal length The depth of the feature point from the camera coordinate system. (The inverse transformation matrix from the camera coordinate system to the robot base coordinate system), and the fingertip deformation distance detection formula: ( Visual calibration coefficients Pixel displacement of visual feature points (calibrated intercept); S4. Multimodal Fusion: By establishing a mapping model of magnetic field strength, distance, and force, the fusion of vision and force perception is achieved. This allows the deformation of the humanoid robot's fingertips to be monitored by visual microsensors, combined with force data to achieve adaptive grasping. This enables automatic adjustment of grip force when the object slips. In step S4, the IMU angular velocity and visual feature points are fused using the VINS-Fusion algorithm to control the dynamic positioning error to the centimeter level, in order to meet the operational needs of the humanoid robot during movement. Step S4 also requires the use of a pre-training and physical fine-tuning framework, through transfer learning from human video data, to improve the model's adaptability to new objects or scenes. In step S4, after the fusion of vision and force perception, the deformation trend of the object within 100ms is automatically predicted, and the grasping strategy is adjusted in advance based on the predicted deformation trend to ensure that the object does not fall off when the humanoid robot hand moves the object. Contact distance derived from magnetic field Contact force derived from visual reasoning Establish a fusion model and contact distance The smaller the area, the larger the contact area, and the greater the actual gripping force for the same deformation. The corrected formula is: when = hour, = ;when When =0, =2 in: The initial distance between the fingertip and the object when the fingertip is not deformed (calibrated value). Sliding critical force, The distance between the fingertip and the object. Fingertip shape variable : Finger tip elasticity coefficient; Meanwhile, the critical condition for an object to slide is: the frictional force generated by the gripping force is greater than or equal to the component of the object's weight along the gripping surface, that is: ( The weight of an object : Capture angle, : coefficient of friction (Sliding critical force), and when the visual sensor detects that the object is sliding (relative displacement) When the current grip strength is less than the critical force, calculate the difference between the current grip strength and the critical force, and adjust the grip strength accordingly. ,in Adjustment coefficient ( =1.1 1.3, to prevent over-adjustment); S5. Real-time data acquisition and preprocessing: The visual perception system is built to acquire visual data in real time and clean and transform the acquired multi-angle visual data to remove invalid data (such as image blurring, duplication, and measurement noise). In step S5, in addition to acquiring visual information data, a magnetic tactile sensor is used to monitor the grasping contact state to ensure that the force applied by the robot's fingertips when grasping objects can be adjusted in real time, avoiding the situation where the robot applies too much force to grasp flexible or fragile objects, causing damage, or applies too little force to grasp them, causing them to fall off. S6. Motion Execution and Feedback Data Acquisition: The humanoid robot automatically completes the operation by pressing the command and collects feedback data from various parts during operation to ensure that the source can be traced or the system can be upgraded (to improve the problem found) in case of accidental failure. In step S6, when the deviation is found in the real-time feedback data of visual detection (such as grasping deviation > 1mm), the joint fine adjustment and force control parameter update are automatically triggered to ensure the operation accuracy.

[0018] Example: S1. Layered Vision Architecture Setup and Calibration: The LiDAR is installed on the robot's head gimbal, covering the entire production line workspace (radius 3m); the binocular camera is fixed to the wrist, with a field of view covering the dexterous hand's grasping range (diameter 30cm); miniature high-speed cameras are symmetrically installed on the sides of the thumb and index fingertips, focusing on the contact area between the fingertips and the egg. After the LiDAR, binocular camera, and miniature high-speed camera are all installed, accuracy calibration is performed to ensure that the LiDAR positioning error is stable at 2.8cm (≤5cm requirement); the binocular camera's 3D reconstruction accuracy reaches 0.25mm (≤0.5mm requirement); the miniature high-speed camera maintains a frame rate of 250fps (≥200fps requirement) to meet the needs of high-speed dynamic contact monitoring. S2. Computational Model Training and Deployment: S21. Object detection and segmentation: Collect 5000 images of eggs with different placement postures and lighting conditions (including scenes partially occluded by egg cartons), train the YOLOv8 model. After the model is trained, the model achieves a bounding box detection accuracy of 99.8% for eggs and a mask segmentation IoU value of 0.98, which can accurately identify eggs nested in egg cartons. S22, 6D Pose and Depth Estimation: The point cloud from the LiDAR and the point cloud from the binocular camera are fused and input into the PointNet++ algorithm for 3D pose calculation. The test results show that the 3D position positioning error of the egg is 0.3mm and the attitude angle error is 0.1°, which meets the sub-millimeter level accuracy requirements. S23. Visual motion mapping: Collect 100 sets of video footage of human hand movements when grasping eggs, extract the coordinates of key points of the palm and fingers using OpenPose, and map them to the 15 joint spaces of the dexterous hand using a motion relocation algorithm to automatically generate reference trajectories with a trajectory smoothness of 95%; S24. Dynamic Prediction and Decision-Making: Based on the ViTacFormer model, which integrates visual (high-speed camera images from fingertips) and tactile (temporal data of contact force) features, a contact state prediction model is trained. The model can predict the sliding trend of the egg and fingertip 50ms in advance with a prediction accuracy of 99.2%, providing a basis for optimizing the grasping strategy. S3. Closed-loop control link construction and pre-operation simulation: Global vision plans the motion path from the robot's initial position to the egg tray; local vision completes the fine positioning of the egg grasping point (selecting the largest diameter point in the middle of the egg as the grasping point); the large vision model outputs the optimal multi-finger grasping posture based on the egg's size (major axis approximately 5.5cm), material (fragile, flexible shell), and shape (ellipsoidal), namely: the thumb and index finger lightly touch the sides of the eggshell at a 45° angle, with the middle, ring, and little fingers assisting in lifting, and the fingertip contact area ≥1.5cm². Then, 1000 sets of simulation data (including different egg sizes, placement angles, and egg tray gap parameters) are input, and multi-finger grasping posture pre-training is performed in the simulation environment. After training, the dexterous hand achieves a 100% grasping success rate in the simulated scene, with no simulated breakage or slippage. S4. Multimodal Fusion and Adaptive Grasping Implementation: A mapping model is established for magnetic field strength (output of the magnetic tactile sensor), distance between fingertip and egg (micro-visual measurement), and contact force. When the micro-visual system detects slight deformation of the eggshell at the fingertip (deformation displacement ≥ 0.1 mm), the system automatically reduces the grip force from the initial 1.2 N to 0.8 N; when the eggshell is detected to be slipping, the grip force is increased by 0.3 N to achieve adaptive adjustment. Next, the VINS-Fusion algorithm is used to fuse IMU angular velocity and visual feature points. During the robot's movement (speed 0.5 m / s), the dynamic positioning error of the egg is stabilized at 4.2 cm (≤ centimeter level), meeting the requirements for grasping while moving. Then, based on video data of humans grasping similar ellipsoidal objects such as apples and tomatoes, transfer learning is performed. The adaptation time for new items (such as duck eggs) is shortened from the original 2 hours to 10 minutes, and the grasping success rate reaches 98.5% after adaptation. S5. Real-time Data Acquisition and Preprocessing: During system operation, the LiDAR acquires 10 frames of global point cloud data per second, the binocular camera acquires 30 frames of local images per second, and the miniature high-speed camera acquires 250 frames of fingertip contact images per second; the magnetic tactile sensor acquires 1000 sets of contact force data per second. Then, a filtering algorithm is used to remove image blur, duplicate frames, and force data noise (Gaussian filtering is used to remove image noise, thresholding is used to remove duplicate images of the same posture, and moving average filtering is used to smooth force data fluctuations). After preprocessing, the data efficiency is improved to 99.5%. When the magnetic tactile sensor detects a grip force exceeding 1.0N, the system immediately issues a command to reduce the joint torque; when the grip force is below 0.5N, the torque is automatically increased to ensure the safety of grasping flexible and fragile items. S6. Motion Execution and Feedback Correction: The robot moves to the top of the egg tray according to the planned path. The dexterous hand completes the egg grasping according to the pre-trained trajectory and then transfers it to the target tray. The whole process takes 3 seconds. At this time, visual positioning data, joint angle data, and contact force data are collected and stored simultaneously for subsequent fault tracing and system upgrades. If the vision detects a grasping deviation of 0.8mm during a grasping process (not exceeding the 1mm threshold), the system does not trigger a large adjustment. When the deviation reaches 1.2mm, the system automatically triggers joint fine-tuning and force control parameter updates (thumb joint angle fine-tuned by 0.5°, grip force increased by 0.1N, and the deviation reduced to 0.3mm after correction) to ensure operational accuracy.

[0019] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.

Claims

1. A method for precise control of a humanoid robot's dexterous hand using visual recognition technology, characterized in that, The method for precise control of a humanoid robot's dexterous hand includes the following steps: S1. Visual perception architecture construction: A layered visual architecture is built by using sensors to meet the perception needs from global to local coverage, while adapting to the accuracy requirements of different operating scenarios. The spatial coordinate transformation formula is as follows: ( : Extrinsic parameter matrix, (internal parameter matrix), where The expression: ( , Camera's x and y axis focal lengths , (Camera principal point coordinates). Due to the different sampling frequencies of the various sensors, time differences occur in the data, which requires compensation through linear interpolation. ( Completed data for the target time t. , Adjacent sampling times , (raw data); S2. Computational Model Building: A deep learning model is used to understand the target, providing accurate input for the subsequent control of the humanoid robot's dexterous hand; S3. Control Link Design: Construct a closed-loop control system from perception to planning to execution to feedback, and combine visual guidance and force correction to improve accuracy. After the control link design is completed, input simulation data to conduct pre-operation simulation training of the humanoid robot's multi-finger grasping postures to ensure that the humanoid robot can operate normally in multi-finger grasping postures during subsequent actual operations. Global visual localization of target object pose: Scene images are acquired via a camera, and the pose of the target object relative to the robot's base coordinate system is obtained based on a feature point matching algorithm. The calculation formula is as follows: Target object position coordinates: ( , : The coordinates of the feature points of the target object in the pixel coordinate system. , Camera optical center pixel coordinates Camera focal length The depth of the feature point from the camera coordinate system. (The inverse transformation matrix from the camera coordinate system to the robot base coordinate system), and the fingertip deformation distance detection formula: ( Visual calibration coefficients Pixel displacement of visual feature points (calibrated intercept); S4. Multimodal fusion: By establishing a mapping model of magnetic field strength, distance, and force, the fusion of vision and force perception is achieved, enabling the deformation of the humanoid robot's fingertips to be monitored by visual microsensors. Combined with force data, adaptive grasping is achieved, thereby automatically adjusting the grip force when the object slips. Contact distance derived from magnetic field Contact force derived from visual reasoning Establish a fusion model and contact distance The smaller the area, the larger the contact area, and the greater the actual gripping force for the same deformation. The corrected formula is: when = hour, = ;when When =0, =2 in: The initial distance between the fingertip and the object when the fingertip is not deformed (calibrated value). Sliding critical force, The distance between the fingertip and the object. Fingertip shape variable : Finger tip elasticity coefficient; Meanwhile, the critical condition for an object to slide is: the frictional force generated by the gripping force is greater than or equal to the component of the object's weight along the gripping surface, that is: ( The weight of an object : Capture angle, : coefficient of friction (Sliding critical force), and when the visual sensor detects that the object is sliding (relative displacement) When the current grip strength is less than the critical force, calculate the difference between the current grip strength and the critical force, and adjust the grip strength accordingly. ,in Adjustment coefficient; S5. Real-time data acquisition and preprocessing: Real-time acquisition of visual data is carried out through the established visual perception system, and the acquired multi-angle visual data is cleaned and transformed to remove invalid data. S6. Motion Execution and Feedback Data Acquisition: The humanoid robot automatically completes the operation by pressing the command and collects feedback data from various parts during operation to ensure that the source can be traced or the system can be upgraded in case of accidental failure.

2. The method for precise control of a humanoid robot's dexterous hand using visual recognition technology according to claim 1, characterized in that, The hierarchical visual architecture in step S1 mainly consists of global vision, local vision, and micro vision.

3. The method for precise control of a humanoid robot's dexterous hand using visual recognition technology according to claim 1, characterized in that, Step S2 includes the following sub-steps: S21. Object Detection and Segmentation: Use the YOLOv8 model to output the object bounding box, semantic category and mask, thereby achieving comprehensive feature extraction in the scene; S22, 6D pose and depth estimation: Combining the PointNet++ algorithm, the three-dimensional position and pose of the target are calculated from the point cloud data, and the positioning error needs to be controlled within the sub-millimeter level; S23. Visual motion mapping: Key points of the human hand are extracted through OpenPose, then mapped to the dexterous hand joint space through motion redirection, and a reference trajectory is automatically generated. S24. Dynamic Prediction and Decision-Making: Introducing the ViTacFormer modal model, integrating visual and tactile temporal features, predicting changes in contact state, and optimizing the grasping strategy.

4. The method for precise control of a humanoid robot's dexterous hand using visual recognition technology according to claim 1, characterized in that, In step S3, the pre-operation generates the operation space path by global vision, and local vision completes the fine positioning of the target and the planning of the grasping point. At the same time, based on the target size, material and shape, the optimal multi-finger grasping posture is output through the large visual model.

5. The method for precise control of a humanoid robot's dexterous hand using visual recognition technology according to claim 1, characterized in that, In step S4, the IMU angular velocity and visual feature points are fused using the VINS-Fusion algorithm to control the dynamic positioning error to the centimeter level, so as to meet the operational requirements of humanoid robot movement.

6. The method for precise control of a humanoid robot's dexterous hand using visual recognition technology according to claim 5, characterized in that, In step S4, a pre-training and physical fine-tuning framework is also required, which uses human video data transfer learning to improve the model's ability to adapt to new objects or scenes.

7. A method for precise control of a humanoid robot's dexterous hand using visual recognition technology according to claim 6, characterized in that, In step S4, after the vision and force perception are fused, the deformation trend of the object within 100ms is automatically predicted, and the grasping strategy is adjusted in advance according to the predicted deformation trend to ensure that the object will not fall off when the humanoid robot hand moves the object.

8. The method for precise control of a humanoid robot's dexterous hand using visual recognition technology according to claim 1, characterized in that, In step S5, in addition to collecting visual information data, a magnetic tactile sensor is used to monitor the grasping contact state to ensure that the force exerted by the robot's fingertips when grasping objects can be adjusted in real time, thus avoiding damage caused by excessive force or loss of objects due to insufficient force when grasping flexible or fragile objects.

9. A method for precise control of a humanoid robot's dexterous hand using visual recognition technology according to claim 1, characterized in that, In step S6, when an offset is detected in the real-time feedback data from visual inspection, joint fine-tuning and force control parameter updates are automatically triggered to ensure operational accuracy.

Images