Multimodal dexterous hand fusing visual-haptic and palm eye and application thereof

Abstract

This invention belongs to the field of robotics technology and specifically discloses a multimodal dexterous hand integrating visual-tactile and palm-eye perception and its applications. It includes a hand body, a visual-tactile sensor, a palm-eye module, and a processor. The visual-tactile sensor includes a skeleton, an elastomer, RGB and NIR cameras. The skeleton has a finger-like structure, and the skeleton and hand body form the hand structure. The elastomer covers the outside of the skeleton, and its surface is coated with a diffuse reflection coating. The RGB and NIR cameras are installed inside the skeleton to acquire RGB and NIR images when a first object contacts the elastomer. The palm-eye module is installed in the middle of the hand body and includes a second RGB camera and a TOF camera to acquire the RGB image and sparse depth of a second object. The processor determines the 3D point cloud of the first object's surface and the depth map of the second object, achieving pose alignment between the two. This invention enables multimodal perception of touch, vision, and depth, improving the accuracy and robustness of the robotic hand.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, and more specifically, relates to a multimodal dexterous hand that integrates visual-tactile and palm-eye perception and its applications. Background Technology

[0002] With the increasing demand for dexterous robot operation in fields such as industrial automation and medical surgery, existing rigid manipulators are unable to meet the multimodal high-precision perception requirements in high-precision and high-dynamic scenarios due to insufficient tactile perception accuracy and low depth estimation resolution.

[0003] Existing robotic arms mostly employ a single modality of vision or touch, making it difficult to simultaneously grasp both global environmental understanding and local contact characteristics: vision fails in occluded environments, while touch cannot perceive untouched areas. Furthermore, existing ToF cameras output highly sparse depth maps, relying on post-processing algorithms for completion, but traditional methods (such as Kalman filtering and SGM (Semi-Global Matching)) suffer from a sharp increase in root mean square error at low input densities. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a multimodal dexterous hand that integrates visual-tactile and palm-eye perception and its application. Its purpose is to realize multimodal perception of touch, vision and depth, and improve the working accuracy and robustness of the robotic hand.

[0005] To achieve the above objectives, according to one aspect of the present invention, a multimodal dexterous hand integrating visual-tactile sensing and palm-eye sensing is proposed, comprising a hand body, several visual-tactile sensors, a palm-eye module, and a processor, wherein:

[0006] The visual-tactile sensor includes a frame, an elastomer, an RGB camera, and a NIR camera. The frame is a finger-shaped structure and is mounted on the hand body. The frames of each visual-tactile sensor and the hand body together form a hand structure. The elastomer covers the outside of the frame and has a diffuse reflection coating on its surface. The RGB camera and NIR camera are mounted inside the frame and are used to acquire RGB and NIR images, respectively, when the first object comes into contact with the elastomer.

[0007] The palm-mounted eye module is installed in the middle of the hand body. The palm-mounted eye module includes a second RGB camera and a TOF camera. The second RGB camera and the TOF camera are used to acquire the RGB image and sparse depth of the second object, respectively.

[0008] The processor is used to reconstruct a three-dimensional point cloud of the surface of the first object based on the RGB image and NIR image of the first object when it is in contact with the elastic body, and to obtain a depth map of the second object based on the RGB image and sparse depth of the second object; and to align the poses of the first object and the second object based on the three-dimensional point cloud of the surface of the first object and the depth map of the second object.

[0009] According to another aspect of the present invention, an application of a multimodal dexterous hand that integrates visual-tactile and palm-eye perception is provided, comprising the following steps:

[0010] When the dexterous hand grasps the first object, it acquires RGB and NIR images of the first object when it comes into contact with the sensor using an RGB camera and a NIR camera, and obtains RGB and NIR pixel values ​​and pixel coordinates based on the RGB and NIR images; based on the RGB and NIR pixel values ​​and pixel coordinates, it obtains the predicted normal vector of the first object; based on the predicted normal vector and the image boundary depth, it obtains the depth field and realizes the reconstruction of the three-dimensional point cloud of the surface of the first object.

[0011] The second RGB camera and the ToF camera respectively acquire the RGB image and sparse depth of the second object. The sparse depth is then completed by using the RGB image to obtain the depth map of the second object.

[0012] Based on the 3D point cloud of the first object's surface and the depth map of the second object, the pose alignment of the first object and the second object is achieved.

[0013] As a further preferred method, the first object prediction normal vector is obtained based on RGB and NIR pixel values ​​and pixel coordinates, specifically as follows:

[0014] Input the RGB and NIR pixel values ​​and pixel coordinates into the normal vector prediction model to obtain the first object prediction normal vector;

[0015] The method for obtaining the normal vector prediction model includes:

[0016] A calibration object with characteristic protrusions is placed on the sensor surface, and the actual image is acquired through an RGB camera or a NIR camera; based on the CAD model of the calibration object, the image of the contact surface between the calibration object and the sensor is rendered to obtain the calibration object rendering image; the camera parameters corresponding to the feature patterns of the actual image and the calibration object rendering image are obtained; based on the camera parameters, the scene when the elastic body is not deformed is rendered to obtain the background depth and background normal vector, and the image boundary depth is determined.

[0017] The probe with a ball at its end is brought into contact with the sensor at different positions. RGB and NIR images of different areas of the surface are acquired by RGB and NIR cameras, i.e., contact images. The RGB or NIR images are used as calibration images. Based on the sensor CAD model, the contact surface image between the ball and the sensor is rendered to obtain the sphere rendering image. The scene depth is obtained when the calibration image and the sphere rendering image are completely superimposed.

[0018] The mask region is obtained by subtracting the scene depth from the background depth. The RGB and NIR pixel values ​​and pixel coordinates of the mask region in the contact image are extracted. The surface normal vector is calculated based on the depth of the mask region.

[0019] The neural network model is trained by taking background depth, background normal vector, RGB and NIR pixel values ​​and pixel coordinates as inputs and surface normal vector as output. The trained neural network model is then used as the normal vector prediction model.

[0020] As a further preferred embodiment, the depth field is obtained based on the predicted normal vector and the image boundary depth, including:

[0021] The problem of 3D object reconstruction is described as a Poisson equation. The Poisson equation is discretized using the central difference method to obtain the depth constraints of each pixel in the contact image. The depth constraints of all pixels are combined into a discrete linear system.

[0022] Using the image boundary depth as the depth prior z prior And the depth prior z is weighted by λ prior As an additional constraint, this extends the discrete linear system into an augmented sparse linear system;

[0023] The depth field is obtained by solving the augmented sparse linear system using the least squares method.

[0024] As a further preferred embodiment, the augmented sparse linear system is specifically represented as follows:

[0025]

[0026] Where A is the sparse matrix of encoded depth coefficients, z is a vector containing the depths of all pixels; b is the divergence term of the Poisson equation, obtained by calculating the depth gradient from the predicted normal vector; I prior It is a diagonal matrix.

[0027] As a further preferred option, the sparse depth is completed by using the RGB image to obtain the depth map of the second object. Specifically, the RGB image and sparse depth of the second object are aligned and then input into the depth correction model to obtain the depth map of the second object.

[0028] The deep correction model includes ResNet34-Unet and a dynamic spatial propagation network, wherein:

[0029] ResNet34-Unet obtains an initial depth map V based on the aligned RGB image and sparse depth. 0 Unweighted affinity matrix Initial similarity matrix Attention weight And confidence mask C;

[0030] The dynamic spatial propagation network consists of multiple sub-modules, based on the initial depth map V. 0 Each submodule updates the depth map in turn, and the final corrected depth map is obtained after all submodules have completed the update.

[0031] Where, for the t-th iteration, t = 1, 2…N, and N is the number of submodules:

[0032] Based on attention weights and the initial similarity matrix Calculate the dynamic similarity matrix W t ;

[0033] From the dynamic similarity matrix W t and Generate affinity matrix A t :

[0034] Based on affinity matrix A t And the confidence mask C for the current depth map V t-1 The update is performed to obtain the updated depth map V. t .

[0035] As a further optimization, the current depth map V t-1 The formula for updating is:

[0036] V t =(D -1 A t V t-1 ·(1-C)+V sparse ·C

[0037] Among them, (D) -1 A t V t-1 For depth propagation, it is activated in regions with low confidence mask values, and D is the affinity matrix A. t Rows and matrices; V sparse =V 0 It is a sparse depth constraint term that is activated in regions with high confidence mask values.

[0038] As a further preferred option, the depth map is not updated in the object edge region, i.e., edge region V t=V t-1 .

[0039] As a further preferred option, attention weight The determination method is as follows:

[0040] For the current depth map V at the t-th iteration t-1 The pixel at position (i,j) is... This represents the attention weight of this pixel to its neighboring pixels at a distance of k. As k increases, the attention weight... Gradually decrease.

[0041] As a further optimization, the unweighted affinity matrix Neighborhood-based propagation in dynamic spatial networks Update:

[0042]

[0043] in, express The value at position (i,j) and its neighborhood range The method for determining it is as follows:

[0044]

[0045] Where (p,q) is the neighborhood offset. represents the offset field predicted by the deformable convolutional network, and represents the parameters of the deformable convolutional network.

[0046] In summary, compared with the prior art, the above-described technical solutions conceived by this invention mainly possess the following technical advantages:

[0047] 1. This invention replaces rigid fingertips with finger-shaped visual-tactile sensors and embeds a palm-mounted eye module equipped with an RGB camera and a ToF camera in the palm. Through their collaboration, it can overcome the limitations of single-modal information and achieve multimodal perception of "touch-vision-depth". This invention fills the technological gap in the field of multimodal perception for robotic arms and provides a highly robust solution for scenarios requiring precise perception, such as precision assembly and minimally invasive surgery.

[0048] 2. To address the challenges of obtaining the true value of normal vectors, limited accuracy of normal vector reconstruction, and cumulative errors in normal vector integration during 3D reconstruction of complex curved surfaces using visual-tactile sensing, this invention proposes a multispectral depth-photometric stereo 3D reconstruction method based on contact surface rendering and boundary priors. This method obtains the true value of surface normals through rendering, increases the constraints for solving photometric stereo normal vectors by introducing NIR spectroscopy, and reduces the cumulative error of normal vector integration by introducing boundary priors. Ultimately, it achieves high-precision and robust estimation of curved surface visual-tactile normal vectors and depth reconstruction.

[0049] 3. This invention proposes a dynamic spatial propagation network based on adaptive weights, dynamic paths, and diffusion suppression strategies. It integrates dense RGB color information with sparse depth information to achieve high-resolution operational region depth estimation. Details are as follows:

[0050] 1) By weighting the static affinity matrix with a similarity matrix, dynamic decoupling of the propagation process is achieved, breaking through the static affinity limitation of traditional spatial propagation networks;

[0051] 2) An adaptive weighting strategy is proposed to decouple the neighborhood by distance, assign higher weights to nearby neighbors, and reduce the weights of distant neighbors with distance, thereby enhancing the depth accuracy of texture and edge details;

[0052] 3) Introduce a diffusion suppression operation to stop depth updates in the edge region, avoid cross-edge diffusion, and preserve the clarity of the object outline;

[0053] 4) A dynamic path strategy is proposed, which predicts the neighborhood offset through deformable convolution and dynamically generates the propagation path. This allows the neighborhood to break through the fixed grid, expands the path to accelerate filling in flat areas, and shrinks into a self-loop trigger in edge areas. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the overall structure of the multimodal dexterous hand according to an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the palm-sized eye module structure according to an embodiment of the present invention;

[0056] Figure 3 This is a schematic diagram of the visual-tactile sensor structure according to an embodiment of the present invention;

[0057] Figure 4 This is a schematic diagram of the multispectral depth photometric stereoscopic three-dimensional reconstruction method according to an embodiment of the present invention;

[0058] Figure 5 This is a flowchart illustrating the calibration process of the visual-tactile sensor according to an embodiment of the present invention.

[0059] Figure 6 This is a diagram showing the experimental results of three-dimensional reconstruction in an embodiment of the present invention;

[0060] Figure 7 This is a diagram illustrating the overall framework of the depth correction model in an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0062] This invention provides a multimodal dexterous hand that integrates visual-tactile and palm-eye perception, such as... Figure 1 and Figure 2 As shown, it includes a hand body, several visual and tactile sensors, and a palm-mounted eye module, wherein:

[0063] The visual-tactile sensor has a complex curved surface structure resembling the shape of a human finger, which can produce continuous deformation when in contact with an object. Specifically, it includes a skeleton, an elastomer, an RGB camera, and a NIR camera. The skeleton, being a finger-shaped structure, is mounted on the hand body, and the skeletons of each visual-tactile sensor together with the hand body form the hand structure. The elastomer covers the outside of the skeleton, and its surface is coated with a diffuse reflection coating. The RGB and NIR cameras are mounted inside the skeleton and are used to acquire RGB and NIR images, respectively, when the first object comes into contact with the elastomer.

[0064] The palm-mounted eye module is installed in the middle of the hand body. This module includes a second RGB camera and a ToF camera, which are used to acquire RGB images and sparse depth data of a second object, respectively. It also includes a ToF laser and a status indicator light.

[0065] Furthermore, it also includes RGB tri-color light sources, NIR light sources, and beam splitters, such as... Figure 3 As shown, the RGB tri-color light source and the NIR light source consist of three RGB LEDs and one NIR LED, evenly distributed on the inner surface of the frame. The series resistor value is adjusted to achieve suitable brightness for the LEDs, resulting in uniform illumination. A beam splitter enables simultaneous sensing by both cameras. It splits the incident light into two identical and perpendicular beams, which enter the RGB camera and the NIR camera respectively, achieving multispectral imaging. The RGB and NIR cameras are equipped with filters for different wavelengths, filtering out visible and near-infrared light within specific wavelength ranges. This allows the cameras to independently sense visible and near-infrared light and capture each type of light separately. To establish a controlled lighting environment, the camera's automatic white balance and automatic exposure correction functions are disabled. Furthermore, a transparent acrylic panel on the frame provides a viewing window for the camera; changing the size of the acrylic panel adjusts the camera's field of view.

[0066] Furthermore, the skeleton is made of photosensitive resin by 3D printing, and its shape is coordinated with the finger-shaped surface. It is used for elastomer forming and positioning, and serves as a container for other parts.

[0067] Furthermore, the silicone elastomer is made from a mixture of Smooth-On's Solaris curing silicone and Slacker softener. A higher proportion of softener results in a softer elastomer. When fabricating the finger-shaped elastomer, the mixture is first placed in a vacuum pump to remove air bubbles, preventing them from affecting imaging. Then, the mixture is poured into a mold with a finger-shaped inner surface and allowed to solidify. To avoid parting lines on the elastomer surface affecting imaging, a top-to-bottom mold separation method is used. To improve the smoothness of the elastomer surface, the mold is machined and polished from acrylic. The diffuse reflection coating is made from a mixture of silicone, silver powder, and thinner, and is sprayed onto the elastomer surface using a spray gun. The silicone mixture ensures adhesion between the coating and the elastomer, the silver powder imparts diffuse reflection properties, and the thinner prevents the coating from solidifying too quickly.

[0068] This invention provides an application of a multimodal dexterous hand that integrates visual-tactile sensing and palm-eye sensing, including multispectral depth-photometric stereoscopic reconstruction based on a visual-tactile sensor, depth map acquisition based on a palm-eye module, and the combined use of the two. Specifically, the dexterous hand grasps a first object, and the visual-tactile sensor acquires RGB and NIR images of the first object when it is in contact with an elastic body, thus realizing a three-dimensional point cloud of the first object's surface; the palm-eye module acquires RGB images and sparse depth of a second object at a distance, thereby obtaining a depth map of the second object; the pose alignment of the first and second objects can be achieved based on the three-dimensional point cloud of the first object's surface and the depth map of the second object.

[0069] (1) Multispectral depth-photometric stereo 3D reconstruction based on visual-tactile sensors aims to obtain the true value of surface normals through rendering, and to increase the constraints for solving the photometric stereo normal vector by introducing NIR spectroscopy. That is, the surface normal vector of the object is solved by the light intensity reflected from the object surface under multiple light source illumination. Three RGB light sources provide three normal vector constraint equations, and the introduction of NIR light sources provides a fourth equation to form an overdetermined system of equations, reducing the influence of random errors. By introducing boundary priors, the cumulative error of normal vector integration is reduced, and finally, high-precision and robust surface visual-tactile normal vector estimation and depth reconstruction are achieved. Figure 4 As shown, the details are as follows:

[0070] (1.1) The affine transformation parameters of the RGB camera and the NIR camera are obtained through a special structure calibration object, so as to align the obtained RGB and NIR images.

[0071] A calibration object with characteristic protrusions, conforming to the surface of the elastomer, is placed over the sensor surface. The calibration object is pressed to ensure full contact with the elastomer, and RGB and NIR images are acquired simultaneously. Affine transformation parameters of the RGB image relative to the NIR image are obtained using feature point matching, including scaling factor, rotation angle, and translation amount.

[0072] (1.2) Calibrate camera intrinsic and extrinsic parameters through rendering, such as... Figure 5 As shown, the background depth, background normal vector, and image boundary depth are obtained.

[0073] A special structural calibrator is placed on the surface of the sensor, and the actual image is acquired by an RGB camera or a NIR camera (since the images acquired by the two cameras are already aligned, either image can be used).

[0074] Based on the CAD model of the calibration object, an image of the contact surface between the calibration object and the sensor is rendered to obtain a rendered image of the calibration object. In this embodiment, the rendering scene is created using the OpenGL-based 3D rendering engine pyrender. The trimesh mesh is converted into a pyrender mesh and added to the rendering scene, creating a renderer with a size of 640×480 pixels (the same as the captured image).

[0075] Initialize the camera's intrinsic and extrinsic parameters, where the extrinsic parameters include the rotation matrix R and the translation vector T, and the intrinsic parameters include the focal lengths (f) along the image's X and Y axes. x ,f y ), principal point coordinates (c x ,c y The process involves pre-capturing multi-angle images of known geometric patterns (such as a checkerboard pattern) and calculating the intrinsic parameters of the two cameras using the Zhang Zhengyou calibration method. The extrinsic parameter matrix is ​​then calculated using the camera extrinsic parameters, and camera nodes are created in the rendering scene. In this embodiment, the camera extrinsic parameters are initialized, with the 3D coordinates initialized to [0,0,10] and the pitch / yaw / roll angles initialized to 0. The initial extrinsic parameter matrix is ​​calculated based on the initialization parameters. Specifically, the angles are converted to radians, X, Y, and Z axis rotation matrices are constructed, and the rotation matrix R is combined in the order of Z, Y, and X. A translation component T is added to construct the complete extrinsic parameter matrix, and the coordinate system is transformed from OpenCV to OpenGL. The RGB camera intrinsic parameters (f...) are then aligned using data... x1 ,f y1 ,c x1 ,c y With NIR camera intrinsic parameters (f x2 ,f y2 ,c x2 ,c y2 ) unified as (f x ',f y ',c x ',c y '), by camera intrinsic parameters (f x ',f y ',c x ',c y '), extrinsic parameters (R,T) create a camera node and add it to the rendering scene, add light sources to the scene, and import the actual calibration object image captured.

[0076] The system reads pixels and depth from the rendering scene, overlays the actual image and the rendered image, adjusts the camera parameters (extrinsic parameters), updates the camera's extrinsic parameter matrix in real time and synchronizes it to the rendering scene, until the feature patterns of the actual image and the rendered image coincide, and obtains the camera parameters at this point.

[0077] Read the CAD model of the sensor surface, import the camera parameters obtained from calibration, render the scene when the elastic body is not deformed, obtain the background depth and background normal vector, extract a certain width of the image edge (10 pixels in this implementation) as the image boundary, and obtain the image boundary depth.

[0078] (1.3) Obtain the true value of the normal vector by rendering an image of a ball with a known radius, and then train the normal vector prediction model.

[0079] The sensor is fixed on the base, and computer numerical control technology is used to control the probe with a small ball of known radius at the end to contact the sensor at different positions. Contact images of different areas of the surface are uniformly collected by RGB and NIR cameras, and the RGB or NIR images are used as calibration images.

[0080] Based on the CAD model of the sensor's outer surface, render the image of the sphere's contact surface with the sensor, input the calibrated camera parameters to add a camera node, input the known radius to create a small sphere mesh and add materials, combine the mesh and materials into a renderable object, add it to the scene, and create a sphere node.

[0081] The calibration image and the rendered image are overlaid. The 3D coordinates of the sphere are adjusted so that the calibration image and the rendered image completely overlap. The scene depth at this time is read from the renderer. The contact area mask is obtained by subtracting the scene depth from the background depth. The RGB and NIR pixel values ​​and pixel coordinates of the mask area in the contact image are extracted.

[0082] The ground truth value of the surface normal vector is determined based on the depth of the mask region, and then a training set is constructed by combining the corresponding RGB and NIR pixel values ​​and pixel coordinates; specifically:

[0083] The gradient of the depth map (scene depth) in the x and y directions of the image coordinate system is calculated using central difference:

[0084]

[0085] The true value of the surface normal vector n is calculated as follows:

[0086]

[0087] The neural network model is trained using a training set with background depth, background normal vector, RGB and NIR pixel values ​​and pixel coordinates as inputs, and surface normal vector as output. The trained neural network model is then used as the normal vector prediction model.

[0088] In this embodiment, the training set is specifically created as follows: RGB images are processed using affine transformation parameters to align them with NIR image data; the X-axis coordinates are normalized to [-1, 1], the Y-axis coordinates are normalized to H / W (H is the image height, W is the image height), and the Z-axis coordinate is set to 0. The image is processed using a ball contact area mask to preserve the pixel values ​​and coordinates of the contact area. Random numbers are generated, and contact data is loaded with a 50% probability, while background data is loaded with a 50% probability, allowing the model to learn to distinguish between contact and non-contact states. When training with NIR, RGB and NIR data are concatenated, and the input channels are expanded to 6.

[0089] The neural network model adopts the MLP model. The input of the MLP network is the difference between the foreground and background pixels and the positional encoding. If only RGB training is used, the number of input channels is 3; if NIR training is enabled, the number of input channels is 6; if positional encoding is enabled, the number of input channels is 6; if both NIR and positional encoding are enabled, the number of input channels is 9. In this embodiment, the specific structure of the MLP model is shown in Table 1. The 1×1 convolutional layer is used for cross-channel feature interaction and fusion of multimodal information; the batch normalization layer standardizes the activation values ​​and accelerates training convergence; the ReLU activation function layer introduces nonlinear factors to improve the model's expressive power; the DropOut layer randomly masks 40% of neurons to prevent overfitting due to tactile noise; the output layer compresses the features to 3 channels, representing the components of the normal vector in the three directions.

[0090] Table 1 MLP Model

[0091]

[0092] The model training configuration is as follows: GPU acceleration is used for training, with an accuracy set to FP32, an initial learning rate of 0.01, a batch size of 16, 300 epochs, and a weight decay coefficient of 0.0001. Weight decay is applied to ordinary convolutional layers to suppress overfitting; to avoid interfering with batch normalization statistics calculations and because the bias term has a relatively small impact on model complexity, weight decay is not applied to batch normalization layers and bias terms. Since adaptive learning rate characteristics are suitable for scenarios with large gradient changes in tactile data, the Adam optimizer is used. The learning rate is reduced to 0.1 every 100 epochs to achieve rapid initial convergence and fine-tuning later. `num_workers` is set to 8 to fully utilize multi-core CPUs; `persistent_workers` is set to True to avoid the overhead of repeatedly creating and destroying workers. The L1 loss function is selected to monitor the gradient error calculated from the normal vector.

[0093] (1.4) Integrate depth priors and solve the depth field by integrating the normal vector to achieve 3D reconstruction.

[0094] When the dexterous hand grasps the first object to be tested, RGB and NIR images of the first object in contact with the sensor are acquired by RGB and NIR cameras, and RGB and NIR pixel values ​​and pixel coordinates are obtained based on the RGB and NIR images. Based on the RGB and NIR pixel values ​​and pixel coordinates, as well as the background depth and background normal vector, the predicted normal vector of the object surface is obtained through the normal vector prediction model.

[0095] Then, based on the predicted normal vector and the image boundary depth, the depth field is obtained, specifically including:

[0096] Assuming the surface depth is z(x,y), its gradient (p,q) and normal vector n(x,y,z) are related as follows:

[0097]

[0098] The 3D reconstruction problem can be described as solving the following Poisson equation:

[0099]

[0100] Discretizing the Poisson equation using the central difference, the depth divergence of pixel (i,j) can be represented by the neighborhood depth:

[0101]

[0102] For all valid pixels in the image, combining the constraints of all pixels into a sparse matrix format yields the following discrete linear system:

[0103] Az = b

[0104] Where A is a sparse matrix of encoding depth coefficients (1 or -1), z is a vector containing the depth of all pixels, and b is the divergence term of the Poisson equation, which is obtained by calculating the depth gradient from the normal vector.

[0105] Using the image boundary depth as the depth prior z prior By using weights λ as additional constraints, the above sparse matrix is ​​extended to obtain an augmented sparse linear system:

[0106] A total z = b total

[0107] in I prior It is a diagonal matrix, where the diagonal elements are 1 at the positions corresponding to the prior pixels with effective depth.

[0108] Solve the weighted system of linear equations using the least squares method, i.e., find:

[0109] min z ||A total zb total || 2

[0110] Equivalent to solving:

[0111] A total T A total z = A total T b total

[0112] Use the sparse matrix solvers sparseqr or spsolve for fast solutions. Sparseqr is based on QR decomposition using Householder reflection, which is highly stable and suitable for large-scale problems. Spsolve is based on LU decomposition, which has a lower memory footprint but is slower, making it suitable for small-scale applications.

[0113] Let the camera intrinsic parameter be the focal length (f). x ,f y ), principal point (c x ,c y ), depth z i,j The corresponding 3D points are:

[0114]

[0115] Z i,j =z i,j

[0116] This completes the reconstruction of the three-dimensional point cloud on the object's surface.

[0117] It should be noted that each visual-tactile sensor performs the above-mentioned three-dimensional reconstruction process separately, thereby reflecting the shape of the object from different orientations.

[0118] (2) Based on the depth map acquisition of the palm eye module, the second RGB camera and the ToF (Time of Flight) camera acquire the RGB image and sparse depth of the second object respectively. A dynamic spatial propagation network based on adaptive weights, dynamic path and diffusion suppression strategy is designed to construct a depth correction model. Through this depth correction model, combined with the RGB image and sparse depth, a depth map is obtained.

[0119] Accurate and dense depth measurements are crucial for robot perception; however, achieving both accuracy and density in depth maps is often difficult. Considering the high resolution of RGB images and the accuracy of direct depth measurements, the Image-guided Depth Completion (IGDC) method generates high-density, high-precision depth maps through sparse depth measurements and corresponding dense RGB images. This invention proposes a dynamic spatial propagation network for image-guided sparse depth completion. It decomposes pixel affinity through a nonlinear propagation model and combines adaptive weights, dynamic paths, and diffusion suppression strategies to achieve dynamic propagation and optimization of depth values. Specifically:

[0120] like Figure 7 As shown, the deep correction model includes ResNet34-Unet and a dynamic spatial propagation network:

[0121] (2.1) ResNet34-Unet obtains the initial depth map V based on the aligned RGB image and sparse depth. 0 Unweighted affinity matrix Initial similarity matrix Attention weight And confidence mask C.

[0122] Specifically, a ToF camera acquires sparse depth measurements, while an RGB camera captures high-resolution color images, and the two datasets are aligned. The RGB image provides scene color and texture information to guide depth propagation; the sparse depth map serves as seed points for depth propagation, containing a small number of known depth values.

[0123] The aligned RGB image and sparse depth data are input into ResNet34-Unet to generate the initial depth map V. 0 Attention weight ((i,j) are pixel coordinates, k is the distance between the pixel and its neighbors), unweighted affinity matrix Initial similarity matrix And a confidence mask C. The initial depth map is a coarsely estimated depth value. The elements in the attention weights represent the attention that should be given to the neighbors at iteration step t and a specific distance k. The affinity matrix represents the connectivity of the pixel neighborhood. The initial similarity matrix represents the static similarity between the pixel and its neighborhood based on the RGB image prediction. The confidence mask marks sparse depth reliable regions, with 1 indicating that the original value is retained and 0 indicating that propagation calculation is required.

[0124] (2.2) The dynamic spatial propagation network consists of N sub-modules, based on the initial depth map V. 0 Each submodule updates the depth map in turn. After N iterations, all submodules have completed the update and the depth map is completed, resulting in the final corrected depth map.

[0125] For the t-th submodule, i.e., in the t-th iteration (t = 1, 2, ..., N), affinity is decomposed using a nonlinear propagation model, and then a deep update is performed, specifically as follows:

[0126] Unweighted affinity matrix Specifically as follows:

[0127]

[0128] The affinity matrix is ​​a 0-1 matrix that labels the neighborhood. The pixel pairs within the control determine the propagation path.

[0129] Attention weight Reflects the importance of neighboring pixels at different distances k to the depth propagation of the current pixel (i,j); based on attention weights. and the initial similarity matrix Calculate the dynamic similarity matrix W t Specifically based on the elements of the initial similarity matrix. Calculate the dynamic similarity matrix W t element

[0130]

[0131] in Let be the similarity weight between pixel (i,j) and its neighbor with offset (a,b) at iteration step t.

[0132] From the dynamic similarity matrix W t right Perform weighted summation to generate affinity matrix A t A t Integrating dissemination paths and intensity serves as the core matrix for deep dissemination:

[0133]

[0134] Based on affinity matrix A t And the confidence mask C for the current depth map V t-1 The update is performed to obtain the updated depth map V. t :

[0135] V t =(D -1 A t V t-1 ·(1-C)+V sparse ·C

[0136] Where D is the affinity matrix A t The rows and matrices (diagonal elements are A)t (The sum of the corresponding row elements) normalizes the depth propagation weights of each pixel; (D -1 A t V t-1 For depth propagation, it is activated in regions with low confidence mask values ​​and is used to interpolate and update depth values ​​based on neighborhood information; V sparse =V 0 This is a sparse depth constraint term that is activated in regions with high confidence mask values, used to re-inject high-precision initial depth values.

[0137] Furthermore, attention weights The update method is as follows:

[0138] In the t-th iteration, for the current depth map V t-1 For the pixel at position (i,j), let k represent the distance between the pixel and its neighboring pixels. ResNet34-Unet is trained to generate a series of attention weights related to k: for nearest neighbors (e.g., k=1), a higher attention weight is output, which makes the nearest neighbor pixels propagate depth first; for distant neighbors (e.g., k≥2), a lower attention weight is output, and the attention weight decays with increasing distance, reducing its propagation contribution, thereby avoiding the introduction of irrelevant depth information that would cause ambiguity.

[0139] This adaptive weighting strategy ensures that neighboring pixels with high similarity are filled first, while distant neighboring pixels with low similarity contribute less, thus ensuring accurate local details and avoiding blurring.

[0140] Furthermore, edge diffusion suppression is incorporated into the dynamic spatial propagation network:

[0141] The depth update process includes a diffusion suppression step to handle pixels in object edge regions, where the weights of the connection between the target pixel and itself are significantly higher than the weights of its connections with neighboring pixels, thus reducing the affinity matrix A. t The row corresponding to this pixel is approximately a self-loop matrix, and the propagation operator D is formed based on this self-loop matrix. -1 A t If it approximates the identity matrix, then V t =V t-1 This inhibits deep cross-edge propagation and updates.

[0142] Specifically, during the iterative update of the depth map, the Dynamic Spatial Propagation Network utilizes its attention module to dynamically identify edge regions based on the rich gradient information provided by the RGB image (especially where gradient changes are significant at object boundaries). For the identified edge pixels, the network learns to output specific attention in the current or subsequent iterations. This specific attention manifests as follows: for the edge pixel itself (i.e., its neighborhood distance k = 0), its attention weight... The attention weights are significantly boosted, approaching a large value, while for all other neighboring pixels (k>0), the attention weights are... The affinity is suppressed and approaches 0, which makes the overall affinity matrix A t The propagation operator D degenerates into a self-loop matrix with non-zero values ​​only at the diagonal positions on the rows corresponding to edge pixels. At this point, since the diagonal elements of the rows and matrix D are equal to the weights of this self-loop, the normalized propagation operator D... - 1 A t This becomes a unit vector on that line. Therefore, in the edge region without initial sparse points (C=0), the above depth update formula becomes:

[0143] V t =(D -1 A t )·V t-1 =V t-1

[0144] That is, the depth no longer expands.

[0145] Furthermore, dynamic path updates are employed in dynamic spatial propagation networks:

[0146] The dynamic path strategy optimizes the depth propagation path by dynamically adjusting the connectivity of pixel neighborhoods, reducing unnecessary computations and improving edge preservation. Neighborhood offset field. According to ResNet34-Unet prediction, the neighborhood Dynamically updated using the following formula:

[0147]

[0148] Where (p, q) represents the offset predicted by deformable convolution. Dynamically predicting neighborhood offsets via deformable convolution allows the neighborhood to break through a fixed grid, adaptively capturing long-distance dependencies or edge structures. In flat regions, the offset expands the neighborhood, including more distant neighbors and accelerating depth filling; in edge regions, the offset shrinks the neighborhood to only the center pixel.

[0149] (3) Based on the 3D point cloud of the first object's surface and the depth map of the second object, spatial pose alignment of the first and second objects is achieved, providing crucial perceptual support for the precise operation of the dexterous hand (such as precision assembly, grasping and control, or path planning). For example, during precision assembly, the 3D point cloud of the electronic component's surface is reconstructed using a visual-tactile sensor, and the palm-eye module acquires the sparse depth of the environment and completes it into a high-resolution depth map, achieving sub-millimeter pose alignment between the assembled component and the target substrate. During minimally invasive surgery, the surface of a precision instrument is reconstructed using a visual-tactile sensor, and combined with the palm-eye module, the depth map of the affected area is completed under liquid interference, generating a safe operation path that avoids vital organs.

[0150] To address the issues of insufficient tactile perception accuracy and low depth estimation resolution in existing rigid robotic hands, this invention proposes a multimodal three-finger dexterous hand that integrates visual-tactile sensing and a palm-mounted eye. By replacing rigid fingertips with finger-shaped visual-tactile sensors, a multispectral depth-photometric stereoscopic reconstruction method is proposed, enabling the robotic hand to possess high spatial resolution tactile perception capabilities and finely perceive surface textures. Furthermore, by embedding a palm-mounted eye module equipped with a monocular RGB camera and a ToF camera in the palm of the robotic hand, a dynamic spatial propagation network combining adaptive weights, dynamic paths, and diffusion suppression strategies is proposed. This network fuses dense RGB color information with sparse depth information to achieve high-resolution depth estimation of the operating area.

[0151] This invention enables high-precision, high-resolution tactile perception (normal vector estimation error of 0.013, surface texture spatial resolution of 0.01mm); when the sparse input is 0.05%, the root mean square error of depth estimation is reduced to 0.089, and the average relative error is reduced to 0.012. Due to its high-precision and robust multimodal visual and tactile perception capabilities, this invention can be applied to fields such as precision assembly, minimally invasive surgery, and human-machine collaboration.

[0152] The following are specific examples:

[0153] Example 1

[0154] The three-dimensional reconstruction method proposed in this invention is used to reconstruct three-dimensional images of four types of objects: a screwdriver, a fingertip, a grid pen holder, and a color-coded resistor. For example... Figure 6 As shown, observing the normal vector and the 3D point cloud reconstruction results, compared with using only the RGB channel, the subtle texture changes of the test object after adding NIR imaging can be more clearly displayed, proving the effectiveness of the method proposed in this invention in improving the accuracy of curved surface visual-tactile 3D reconstruction.

[0155] Example 2

[0156] Applications of multimodal three-finger dexterous hands in precision electronic component assembly:

[0157] In semiconductor chip assembly scenarios, it is necessary to accurately place miniature capacitors measuring 0.5mm × 0.3mm onto circuit board pads. This operation requires a robotic arm with sub-millimeter positioning accuracy, while also needing to perceive the surface texture of the component and the three-dimensional structure of the circuit board to avoid assembly errors caused by visual obstruction or tactile noise. The operation process is as follows:

[0158] 1) Initialization and calibration of visual and tactile sensors.

[0159] Structural configuration: Finger-shaped multispectral visual tactile sensors are installed on the fingertips of three fingers. Each sensor integrates an RGB camera (640×480 pixels), an NIR camera, a beam splitter prism, and a silicone elastomer (Shore hardness 20A). A palm eye module (including a ToF camera, depth resolution 160×120, sparse input density 0.5%) is embedded in the palm.

[0160] Calibration process: By rendering the CAD model of the calibration object, the camera's extrinsic parameters (rotation matrix, translation vector) are interactively adjusted to align the rendered image with the features of the actual calibration image, and the camera's intrinsic parameters (focal length, principal point) are obtained. A small ball probe with a radius of 3mm is used to contact the sensor surface, and the true value of the normal vector is obtained through rendering matching (error ≤ 0.013rad). An MLP model is then trained to predict the normal vector of the contact area.

[0161] 2) Component grasping and surface sensing.

[0162] Contact sensing: When three fingers are closed to grasp the capacitive element, the silicone elastomer deforms (maximum deformation 0.1mm), and the RGB / NIR camera simultaneously captures the contact image.

[0163] 3D Reconstruction: Surface normal vectors are calculated using a multispectral photometric stereo method, and NIR spectroscopy is fused to improve the accuracy of normal vectors in weakly textured regions (such as smooth surfaces of components). Boundary priors (10-pixel width and depth values ​​at image edges) are introduced to solve the Poisson equation, reconstruct the point cloud of the component surface, and obtain the component's pose and surface curvature distribution.

[0164] 3) Deep completion of dynamic spatial propagation network.

[0165] Input data: The palm-sized ToF camera provides sparse depth of the circuit board pad area (input density 0.05%), and the RGB camera outputs a 1920×1080 pixel color image.

[0166] Deep completion process:

[0167] Adaptive weighting strategy: Pixel neighborhoods are decoupled into near-neighbor regions (e.g., 3×3 area) and far-neighbor regions (e.g., 5×5 area) based on distance. A learnable attention mechanism dynamically assigns weights to regions at different distances and at different iteration steps. This mechanism tends to give higher weights to far-neighbor regions in the early stages of iteration for faster filling, and higher weights to near-neighbor regions in the later stages of iteration for finer details. It can also effectively learn to suppress noise interference from far-neighbor regions (e.g., circuit board background).

[0168] Dynamic Path Strategy: This strategy utilizes deformable convolution to dynamically predict a sparse set of neighborhood offsets (e.g., offsets within ±2 pixels) for each pixel. By interacting with only a few (e.g., 3-5) key neighbors, this strategy significantly reduces computational complexity over 6 iterations.

[0169] Diffusion suppression: In the identified object boundary regions (such as the junction of components and pads), attention or path adjustment is used to locally degenerate the propagation weights of pixels in that region into a self-looping structure. This causes the propagation matrix to form unit vectors in the rows of corresponding boundary pixels, thereby terminating the cross-edge propagation of depth and ensuring the boundary sharpness of the final depth map.

[0170] 4) Precise assembly execution.

[0171] Path planning: Based on the completed dense depth map (resolution 1920×1080), an obstacle avoidance trajectory is generated (avoiding component pins and capacitors around pads), with end effector positioning accuracy ≤ ±0.05mm.

[0172] Pose adjustment: Component and pad poses reconstructed by visual-tactile sensors are used to adjust the three-finger gripping angle and position in real time. Sub-millimeter pose alignment (error ≤ 0.1mm) is achieved through multispectral image matching and depth completion data.

[0173] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal dexterous hand integrating visual-tactile and palm-eye perception, characterized in that, It includes the main body of the hand, several visual and tactile sensors, a palm-mounted eye module, and a processor, among which: The visual-tactile sensor includes a frame, an elastomer, an RGB camera, and a NIR camera. The frame is a finger-shaped structure and is mounted on the hand body. The frames of each visual-tactile sensor and the hand body together form a hand structure. The elastomer covers the outside of the frame and has a diffuse reflection coating on its surface. The RGB camera and NIR camera are mounted inside the frame and are used to acquire RGB and NIR images, respectively, when the first object comes into contact with the elastomer. The palm-mounted eye module is installed in the middle of the hand body. The palm-mounted eye module includes a second RGB camera and a TOF camera. The second RGB camera and the TOF camera are used to acquire the RGB image and sparse depth of the second object, respectively. The processor is used to reconstruct a three-dimensional point cloud of the surface of the first object based on the RGB image and NIR image of the first object when it is in contact with the elastic body, and to obtain a depth map of the second object based on the RGB image and sparse depth of the second object; and to align the poses of the first object and the second object based on the three-dimensional point cloud of the surface of the first object and the depth map of the second object.

2. An application of the multimodal dexterous hand that integrates visual-tactile and palm-eye perception as described in claim 1, characterized in that, Includes the following steps: When the dexterous hand grasps the first object, it acquires RGB and NIR images of the first object when it comes into contact with the sensor using an RGB camera and a NIR camera, and obtains RGB and NIR pixel values ​​and pixel coordinates based on the RGB and NIR images; based on the RGB and NIR pixel values ​​and pixel coordinates, it obtains the predicted normal vector of the first object; based on the predicted normal vector and the image boundary depth, it obtains the depth field and realizes the reconstruction of the three-dimensional point cloud of the surface of the first object. The second RGB camera and the ToF camera respectively acquire the RGB image and sparse depth of the second object. The sparse depth is then completed by using the RGB image to obtain the depth map of the second object. Based on the 3D point cloud of the first object's surface and the depth map of the second object, the pose alignment of the first object and the second object is achieved.

3. The application of the multimodal dexterous hand integrating visual-tactile and palm-eye perception as described in claim 2, characterized in that, Based on the RGB and NIR pixel values ​​and pixel coordinates, the first object prediction normal vector is obtained as follows: Input the RGB and NIR pixel values ​​and pixel coordinates into the normal vector prediction model to obtain the first object prediction normal vector; The method for obtaining the normal vector prediction model includes: A calibration object with characteristic protrusions is placed on the sensor surface, and the actual image is acquired through an RGB camera or a NIR camera; based on the CAD model of the calibration object, the image of the contact surface between the calibration object and the sensor is rendered to obtain the calibration object rendering image; the camera parameters corresponding to the feature patterns of the actual image and the calibration object rendering image are obtained; based on the camera parameters, the scene when the elastic body is not deformed is rendered to obtain the background depth and background normal vector, and the image boundary depth is determined. The probe with a ball at its end is brought into contact with the sensor at different positions. RGB and NIR images of different areas of the surface are acquired by RGB and NIR cameras, i.e., contact images. The RGB or NIR images are used as calibration images. Based on the sensor CAD model, the contact surface image between the ball and the sensor is rendered to obtain the sphere rendering image. The scene depth is obtained when the calibration image and the sphere rendering image are completely superimposed. The mask region is obtained by subtracting the scene depth from the background depth. The RGB and NIR pixel values ​​and pixel coordinates of the mask region in the contact image are extracted. The surface normal vector is calculated based on the depth of the mask region. The neural network model is trained by taking background depth, background normal vector, RGB and NIR pixel values ​​and pixel coordinates as inputs and surface normal vector as output. The trained neural network model is then used as the normal vector prediction model.

4. The application of the multimodal dexterous hand integrating visual-tactile and palm-eye perception as described in claim 3, characterized in that, Based on the predicted normal vector and the image boundary depth, the depth field is obtained, including: The problem of 3D object reconstruction is described as a Poisson equation. The Poisson equation is discretized using the central difference method to obtain the depth constraints of each pixel in the contact image. The depth constraints of all pixels are combined into a discrete linear system. Using image boundary depth as a depth prior And through weights Depth Prior As an additional constraint, this extends the discrete linear system into an augmented sparse linear system; The depth field is obtained by solving the augmented sparse linear system using the least squares method.

5. The application of the multimodal dexterous hand integrating visual-tactile and palm-eye perception as described in claim 4, characterized in that, The augmented sparse linear system is specifically represented as follows: in, For the sparse matrix of encoding depth coefficients, It is a vector containing the depth of all pixels; The divergence term of the Poisson equation is obtained by calculating the depth gradient from the predicted normal vector; It is a diagonal matrix. For depth priors.

6. The application of the multimodal dexterous hand integrating visual-tactile and palm-eye perception as described in any one of claims 2-5, characterized in that, The sparse depth is completed by using an RGB image to obtain the depth map of the second object. Specifically, the RGB image and sparse depth of the second object are aligned and then input into the depth correction model to obtain the depth map of the second object. The deep correction model includes ResNet34-Unet and a dynamic spatial propagation network, wherein: ResNet34-Unet obtains an initial depth map based on the aligned RGB image and sparse depth. Unweighted affinity matrix Initial similarity matrix Attention weight and confidence mask ; The dynamic spatial propagation network consists of multiple sub-modules, based on an initial depth map. Each submodule updates the depth map in turn, and the final corrected depth map is obtained after all submodules have completed the update. Among them, for the first t iteration t =1,2…N, where N is the number of submodules: Based on attention weights and the initial similarity matrix Calculate the dynamic similarity matrix ; From the dynamic similarity matrix and Generate affinity matrix : ; Based on affinity matrix and confidence mask For the current depth map Perform an update to obtain the updated depth map. .

7. The application of the multimodal dexterous hand integrating visual-tactile and palm-eye perception as described in claim 6, characterized in that, For the current depth map The formula for updating is: in, For depth propagation terms, activation occurs in regions with low confidence mask values. Affinity matrix Rows and matrices; It is a sparse depth constraint term that is activated in regions with high confidence mask values.

8. The application of the multimodal dexterous hand integrating visual-tactile and palm-eye perception as described in claim 7, characterized in that, The depth map is not updated in the edge region of an object, i.e., the edge region... .

9. The application of the multimodal dexterous hand integrating visual-tactile and palm-eye perception as described in claim 6, characterized in that, Attention weight The determination method is as follows: For the The current depth map at the next iteration middle The position of pixels, in This indicates that the distance between the pixel pair and its value is... k The attention weights of the neighboring pixels, as k Increase attention weight Gradually decrease.

10. The application of the multimodal dexterous hand integrating visual-tactile and palm-eye perception as described in claim 6, characterized in that, Unweighted affinity matrix Neighborhood-based propagation in dynamic spatial networks Update: in, express middle Location value, neighborhood range The method for determining it is as follows: in, This is the neighborhood offset. This represents the offset field predicted by a deformable convolutional network. These are the parameters of the deformable convolutional network.

Images