Dexterous hand three-dimensional pose detection system based on digital grating projection
The three-dimensional posture detection of the skilled fingertips is achieved through the digital grating projection system, which solves the problem of lack of effective detection methods in the existing technology, and achieves high-precision, high-speed and strong stability detection effects, which are suitable for skilled hand performance evaluation.
Patent Information
- Application Number
- CN202411787423.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-01-07
AI Technical Summary
The existing technology lacks effective three-dimensional position detection methods for fingertips, making it difficult to evaluate the performance of robotic skilled hands, especially in real-time synchronous detection.
Adopting a smart hand three-dimensional posture detection system based on digital grating projection, the binocular stereo structure light measurement is realized through two cameras and a digital grating projector. The projector projects a multi-color coded pattern to the smart hand, and the camera acquires image pixel coordinates, combining RGB component separation and multi-step phase shift method obtain the three-dimensional coordinates and posture information of the fingers, and use the fusion classification method of image threshold and three-dimensional model for finger segmentation and data fitting.
It realizes high-precision, high-speed and strong stability of three-dimensional posture detection at the skilled fingertips, and can quickly acquire multiple phase shift images, meet the needs of skilled hand performance detection, and improves detection speed and resolution.
Smart Images
Figure CN119260802A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a dexterous hand posture detection system, in particular to a dexterous hand three-dimensional posture detection system based on digital grating projection. Background Art
[0002] The robot's dexterous hand is the end effector for the interaction between the embodied robot and the environment. It is an important component of the humanoid robot. It belongs to the intersection of cutting-edge technologies such as deep bionics, flexible perception, micro-electromechanical systems, and high-performance materials. It is a key breakthrough in realizing the technological transformation of intelligent robots. The dexterous hand imitates the structure and function of the human hand and plays a key role in the interaction between the robot and the environment. The robot's dexterous hand refers to the human hand in terms of structure and function. It can flexibly manipulate objects and achieve flexible grasping of objects to meet a variety of work needs. The most common number of fingers in a multi-finger dexterous hand is 3-5, each finger has 3 joints, and the kinematic pairs of the finger joints are all revolute pairs.
[0003] Dexterity, grasping ability, and reliability are the three major factors that measure the performance of dexterous hands, and each application scenario has its own emphasis.
[0004] The dexterity of a dexterous hand is mainly determined by its degrees of freedom and the number of actuators. The higher the number of degrees of freedom, the more diverse grasping actions the dexterous hand can achieve. The existing dexterous hand solutions mainly realize the rotational degrees of freedom of each finger joint and the degree of freedom of the thumb closing inward. The number of actuators determines the number of degrees of freedom that can be independently controlled.
[0005] The reliability of a dexterous hand refers to the performance stability and failure rate characteristics of the dexterous hand under long-term operation and various conditions. It is also a reflection of the comprehensive performance of the various mechanisms of the dexterous hand. Taking the rope drive method currently used as an example, its functional realization and service life under large loads are far behind those of practical human hands. To achieve high reliability, more actuators or more complex sensing solutions are needed.
[0006] However, there are no standards or specifications for evaluating the performance of dexterous hands at home and abroad, and there are no technical means and equipment that can meet the requirements of dexterous hand performance testing. Among them, the real-time synchronous detection of the three-dimensional position and posture of each fingertip is the biggest technical difficulty. Summary of the invention
[0007] Aiming at the current situation that the existing robot dexterous hand industry lacks a fingertip three-dimensional posture detection method that meets the requirements, the present invention proposes a dexterous hand three-dimensional posture detection system based on digital grating projection. The detection system uses two cameras and a digital grating projector to realize binocular stereo structured light measurement, the projector is centered, and the two cameras are arranged on the left and right. The projector continuously projects a multi-color coded pattern with phase shift to the dexterous hand, and the left and right cameras synchronously obtain the dexterous hand area image. The image pixel coordinates of each point on the dexterous hand are obtained by RGB component separation and multi-step phase shift method. Pixel-height mapping is realized according to the principle of structured light triangulation to obtain the spatial three-dimensional coordinate value of each point. The image threshold and three-dimensional model fusion classification method are adopted to segment each finger and extract the three-dimensional contour of each finger. The three-dimensional contour model of each fingertip is extracted by data fitting method, and the three-dimensional position and posture information of each fingertip are obtained by calculation. Finally, the three-dimensional position and posture information of each fingertip obtained by the left and right camera images are fused to obtain the final three-dimensional position and posture information of each fingertip, so as to realize the flexibility and stability detection of the dexterous hand. Since multiple phase-shift images can be obtained by taking one image, three-dimensional detection of the target can be achieved, so it has the outstanding advantages of fast speed, high resolution and strong stability. The present invention is achieved through the following technical solutions: The special feature of the three-dimensional posture detection system of the dexterous hand based on digital grating projection of the present invention is that the detection system includes two parts, a detector and a controller, which are connected by a cable. The detector includes a camera, a lens, a projector, a filter, and a synchronizer, wherein: The cameras are industrial-grade color array cameras, of which there are two, located on the left and right sides of the detector, with an external trigger function, responsible for acquiring color images within the field of view; The lenses are industrial-grade lenses, there are two of them, which are installed on the left and right cameras and are responsible for imaging the field of view onto the image plane of the camera. The projector is an industrial-grade digital grating projector that can project a phase-shift coded pattern of multiple colors, with each color corresponding to a phase shift, so as to achieve structured light three-dimensional detection; The filter is a multi-bandpass filter, the number of passbands is consistent with the color of the phase-shift coding pattern projected by the projector, and the central wavelength of each passband is consistent with the central wavelength of the color of the phase-shift coding pattern projected by the projector, so as to filter out ambient light interference and protect, seal, prevent dust and waterproof the detector; The synchronizer is a pure logic circuit, which synchronously triggers the left and right cameras to capture images, thereby reducing synchronization errors; The controller is a high-performance integrated industrial computer with a touch screen. It sends phase-shift coded pattern information of multiple colors to the projector and synchronously reads image data from the left and right cameras. The three-dimensional contours and posture information of all fingers are obtained through RGB component separation and data processing. Finally, the three-dimensional position and posture information of each fingertip obtained from the left and right camera images are fused to obtain the final three-dimensional position and posture information of each fingertip.
[0008] The working process of the three-dimensional posture detection system of the dexterous hand based on digital grating projection of the present invention is as follows: First, under the control of the controller, the projector projects a phase-shifted coding pattern of multiple colors to the area where the dexterous hand is located. Then, the controller controls the synchronizer to trigger the left and right cameras at the same time, and synchronously transmits the field of view image to the controller. Thereafter, the controller obtains the three-dimensional contours and posture information of all fingers respectively through RGB component separation and structured light phase shift contour principle. Finally, the three-dimensional position and posture information of each fingertip obtained from the left and right camera images are fused to obtain the final three-dimensional position and posture information of each fingertip.
[0009] The special feature of the dexterous hand three-dimensional posture detection system based on digital grating projection of the present invention is that the filter is a multi-bandpass filter, the number of passbands is consistent with the color of the phase-shift coding pattern projected by the projector, and the central wavelength of each passband is consistent with the central wavelength of the color of the phase-shift coding pattern projected by the projector, so as to filter out ambient light interference and protect, seal, dustproof and waterproof the detector.
[0010] The present invention proposes a method for detecting the three-dimensional posture of the dexterous fingertips based on the above-mentioned three-dimensional posture detection system of the dexterous hand based on digital grating projection, which is specifically as follows: (1) Under the control of the controller, the projector projects a multi-color phase-shift coded pattern onto the area where the dexterous hand is located, where each color corresponds to a phase shift; (2) The controller controls the synchronizer to trigger the left and right lenses simultaneously, synchronously imaging the field of view onto the image planes of the two cameras and transmitting the images to the controller; (3) The controller extracts the images captured by the left and right cameras respectively, extracts the RGB component data, and obtains an image of the Duofu phase shift coding pattern; (4) The controller uses a multi-step phase shift method to obtain the image pixel coordinates of each feature point on the dexterous hand; (5) The controller implements pixel-height mapping based on the principle of structured light triangulation to obtain the spatial three-dimensional coordinate value of each point; (6) The controller uses the image threshold and 3D model fusion classification method to segment each finger and extract the 3D contour of each finger; (7) The controller uses a data fitting method to extract the three-dimensional contour model of each fingertip and obtains the three-dimensional position and posture information of each fingertip through calculation; (8) The three-dimensional position and posture information of each fingertip obtained from the left and right camera images are averaged to obtain the final three-dimensional position and posture information of each fingertip. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 It is a schematic diagram of the composition of the three-dimensional posture detection system of the dexterous hand of the present invention; Figure 2 is a schematic diagram of a multi-color phase-shift coding pattern of the present invention; Figure 3 It is a schematic diagram of the principle of three-dimensional posture detection of the dexterous hand of the present invention; Figure 4 It is a schematic diagram of the principle of the multi-bandpass filter of the present invention.
[0012] In the figure, 1-detector, 2-controller, 3-camera, 4-lens, 5-projector, 6-filter, 7-synchronizer, 8-dexterous hand. DETAILED DESCRIPTION
[0013] The following is a detailed description of an embodiment of the present invention in conjunction with the accompanying drawings. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and a specific operation process, but the protection scope of the present invention is not limited to the following embodiment.
[0014] The special feature of the three-dimensional posture detection system of the dexterous hand based on digital grating projection of the present invention is that the detection system includes two parts, a detector 1 and a controller 2, which are connected by a cable. The detector 1 includes a camera 3, a lens 4, a projector 5, a filter 6, and a synchronizer 7. Figure 1 As shown, where: The camera 3 is an industrial-grade color array camera, and there are two of them, namely 3-1 and 3-2, which are located on the left and right sides of the detector 1, respectively, and have an external trigger function, responsible for acquiring a color image in the field of view; The lens 4 is an industrial-grade lens, and there are two of them, namely 4-1 and 4-2, which are respectively installed on the left and right cameras 3-1 and 3-2, and are responsible for imaging the field of view onto the image planes of the cameras 3-1 and 3-2; The projector 5 is an industrial-grade digital grating projector that can project a phase-shift coding pattern in multiple colors, such as red, green, and blue. Each color corresponds to a phase-shift coding pattern. Figure 2 Pr, Pg, and Pb are shown in order to realize structured light three-dimensional detection; The filter 6 is a multi-bandpass filter, the number of passbands is consistent with the color of the phase-shift coding pattern projected by the projector 5, and the center wavelength of each passband is consistent with the center wavelength of the color of the phase-shift coding pattern projected by the projector, so as to filter out the interference of ambient light and protect, seal, dustproof and waterproof the detector. For example, the projector 5 projects the phase-shift coding pattern in three colors: red, green and blue. Figure 4 As shown, the central wavelengths are 580nm, 520nm, and 435nm respectively, so the filter 6 should be a three-bandpass filter, and the central wavelengths should also be 580nm, 520nm, and 435nm respectively; The synchronizer 7 is a pure logic circuit, that is, it is composed of logic circuits, which synchronously trigger the left and right cameras 3-1 and 3-2 to capture images, thereby reducing synchronization errors; The controller 2 is a high-performance integrated industrial computer with a touch screen. It sends phase-shift coded pattern information of multiple colors to the projector 5, and synchronously reads the image data of the left and right cameras 3-1 and 3-2. The three-dimensional contours and posture information of all fingers a, b, c, d, e of the dexterous hand 8 are obtained through RGB component separation and data processing. Finally, the three-dimensional position and posture information of each fingertip a, b, c, d, e of the dexterous hand 8 obtained from the images of the left and right cameras 3-1 and 3-2 are fused to obtain the final three-dimensional position and posture information of each fingertip a, b, c, d, e.
[0015] The working process of the dexterous hand three-dimensional posture detection system based on digital grating projection of the present invention is as follows: Figure 3 As shown, first, under the control of the controller 2, the projector 4 projects a phase shift coding pattern of multiple colors, such as red, green, and blue, to the area where the dexterous hand 8 is located, corresponding to the three phase shift patterns Pr, Pg, and Pb, respectively. Then, the controller 2 controls the synchronizer 7 to simultaneously trigger the left and right cameras 3a and 3b, and synchronously transmit the field of view image to the controller 2. Thereafter, the controller 2 obtains the three-dimensional contours and posture information of all fingers a, b, c, d, and e of the dexterous hand 8 respectively through the RGB component separation and structured light phase shift profile principle. Finally, the three-dimensional position and posture information of each fingertip a, b, c, d, and e obtained from the images of the left and right cameras 3a and 3b are fused to obtain the final three-dimensional position and posture information of each fingertip a, b, c, d, and e.
[0016] The special feature of the dexterous hand three-dimensional posture detection system based on digital grating projection of the present invention is that the filter 6 is a multi-bandpass filter, such as Figure 4As shown, the number of passbands is consistent with the number of colors of the phase shift coding patterns Pr, Pg, and Pb projected by the projector 5, which are all 3. The central wavelength of each passband is consistent with the central wavelength of the color of the phase shift coding patterns Pr, Pg, and Pb projected by the projector 5, which are all 580nm, 520nm, and 435nm, so as to filter out ambient light interference and protect, seal, dustproof, and waterproof the detector.
[0017] The present invention proposes a method for detecting the three-dimensional posture of the dexterous fingertips based on the above-mentioned three-dimensional posture detection system of the dexterous hand based on digital grating projection, which is specifically as follows: (1) Under the control of the controller 2, the projector 5 projects a phase shift coding pattern of multiple colors to the area where the dexterous hand 8 is located, and each color corresponds to a phase shift, for example, red, green, and blue, and each color corresponds to a phase shift coding pattern Pr, Pg, and Pb; (2) The controller 2 controls the synchronizer 7 to simultaneously trigger the left and right cameras 3 - 1 and 3 - 2 to synchronously transmit the field of view images to the controller 2; (3) The controller 2 extracts the images captured by the left and right cameras 3-1 and 3-2 respectively, extracts the RGB component data, and obtains the image of the multi-phase shift coding pattern. For example, for the above-mentioned red, green, and blue color phase shift coding patterns Pr, Pg, and Pb, three phase shift images Ir, Ig, and Ib can be separated; (4) Controller 2 uses a multi-step phase shift method to obtain the image pixel coordinates Fk(ik,jk) (k=1,2,…,n) of each feature point on the dexterous hand; (5) Controller 2 implements pixel-height mapping based on the principle of structured light triangulation to obtain the spatial three-dimensional coordinate value Fk(xk,yk,zk) (k=1,2,…,n) of each feature point; (6) Controller 2 uses the image threshold and three-dimensional model fusion classification method to segment each finger a, b, c, d, e and extract the three-dimensional contour of each finger; (7) Controller 2 uses a data fitting method to extract the three-dimensional contour model of each fingertip, and obtains the three-dimensional position and posture information Pk(xk, yk, zk, ak, bk, ck) of each fingertip by calculation (k = 1, 2, 3, 4, 5); (8) The controller 2 averages the three-dimensional position and posture information of each fingertip obtained from the images of the left and right cameras 3 - 1 and 3 - 2 to obtain the final three-dimensional position and posture information of each fingertip. Compared with the prior art, the beneficial effects of the dexterous hand three-dimensional posture detection system based on digital grating projection of the present invention are: (1) The three-dimensional posture detection system of the dexterous hand based on digital grating projection of the present invention realizes the detection of the three-dimensional posture of each finger of the dexterous hand through the principle of digital grating projection. It has the advantages of high precision, strong versatility and high speed, and can fully meet the needs of dexterous hand performance detection.
[0018] (2) The three-dimensional posture detection system of the dexterous hand based on digital grating projection of the present invention constructs a digital grating projection system by a binocular stereo method, which can effectively avoid the inevitable occlusion problem between the fingers of the dexterous hand and is very suitable for the need of online detection of the dexterous hand during work.
[0019] (3) In the three-dimensional posture detection system of the dexterous hand based on digital grating projection of the present invention, the projector projects a phase-shifted coding pattern of multiple colors onto the dexterous hand. The two cameras only need to capture one image to obtain multiple phase-shifted coding pattern images, which greatly improves the detection speed and is very suitable for the performance detection of the dexterous hand under the condition of rapid movement. It can be seen that the technical solution of the present invention has very outstanding advantages compared with traditional detectors.
Claims
1. A three-dimensional posture detection system for dexterous hands based on digital grating projection is special in that the detection system includes two parts, a detector and a controller, which are connected by a cable. The detector contains a camera, a lens, a projector, a filter, and a synchronizer, wherein: The cameras are industrial-grade color array cameras, of which there are two, located on the left and right sides of the detector, with an external trigger function, responsible for acquiring color images within the field of view; The lenses are industrial-grade lenses, there are two of them, which are installed on the left and right cameras and are responsible for imaging the field of view onto the image plane of the camera. The projector is an industrial-grade digital grating projector that can project a phase-shift coded pattern of multiple colors, with each color corresponding to a phase shift, so as to achieve structured light three-dimensional detection; The filter is a multi-bandpass filter, the number of passbands is consistent with the color of the phase-shift coding pattern projected by the projector, and the central wavelength of each passband is consistent with the central wavelength of the color of the phase-shift coding pattern projected by the projector, so as to filter out ambient light interference and protect, seal, prevent dust and waterproof the detector; The synchronizer is a pure logic circuit, which synchronously triggers the left and right cameras to capture images, thereby reducing synchronization errors; The controller is a high-performance integrated industrial computer with a touch screen. It sends phase-shift coded pattern information of multiple colors to the projector and synchronously reads image data from the left and right cameras. The three-dimensional contours and posture information of all fingers are obtained through RGB component separation and data processing. Finally, the three-dimensional position and posture information of each fingertip obtained from the left and right camera images are fused to obtain the final three-dimensional position and posture information of each fingertip.
2. The three-dimensional posture detection system for dexterous hands based on digital grating projection according to claim 1 is further characterized by: The filter is a multi-bandpass filter, the number of passbands is consistent with the color of the phase-shift coding pattern projected by the projector, and the central wavelength of each passband is consistent with the central wavelength of the color of the phase-shift coding pattern projected by the projector.
3. The present invention proposes a detection method based on the above-mentioned dexterous hand three-dimensional posture detection system based on digital grating projection, which is specifically as follows: (1) Under the control of the controller, the projector projects a multi-color phase-shift coded pattern onto the area where the dexterous hand is located, where each color corresponds to a phase shift; (2) The controller controls the synchronizer to trigger the left and right lenses simultaneously, synchronously imaging the field of view onto the image planes of the two cameras and transmitting the images to the controller; (3) The controller extracts the images captured by the left and right cameras respectively, extracts the RGB component data, and obtains an image of the Duofu phase shift coding pattern; (4) The controller uses a multi-step phase shift method to obtain the image pixel coordinates of each feature point on the dexterous hand; (5) The controller implements pixel-height mapping based on the principle of structured light triangulation to obtain the spatial three-dimensional coordinate value of each point; (6) The controller uses the image threshold and 3D model fusion classification method to segment each finger and extract the 3D contour of each finger; (7) The controller uses a data fitting method to extract the three-dimensional contour model of each fingertip and obtains the three-dimensional position and posture information of each fingertip through calculation; (8) The three-dimensional position and posture information of each fingertip obtained from the left and right camera images are averaged to obtain the final three-dimensional position and posture information of each fingertip.