A 3D reconstruction method for visual-tactile sensor objects based on 3D point cloud stitching

Through the visual haptic sensor method based on three-dimensional point cloud splicing, combined with photometric three-dimensional method and CTI algorithm, the problem of large errors in visual and tactile sensors in robot operation is solved, high-resolution three-dimensional reconstruction in complex environments is realized, and the robot is improved Fine operation capability.

CN118799508BActive Publication Date: 2024-11-15DALIAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411287818.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-14
Publication Date
2024-11-15
Estimated Expiration
2044-09-14

AI Technical Summary

Technical Problem

The existing visual and tactile sensors have problems such as limited viewing angle, occlusion, complex background, resulting in large errors, complex production processes and high costs in the fine operation of robots, which affects the accuracy of the refined operation of robots.

Method used

Using a visual haptic sensor method based on three-dimensional point cloud splicing, the three-dimensional shape of the object is reconstructed by installing a visual haptic sensor at the end of the continuous robotic arm, combining photometric three-dimensional method and improved CTI algorithm.

Benefits of technology

It realizes the reconstruction of the three-dimensional shape of an object with high resolution and fast real-time under visual obstruction, and improves the robot's fine operation ability in complex environments, and is suitable for aerospace and medical fields.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118799508B_ABST
    Figure CN118799508B_ABST
Patent Text Reader

Abstract

A method for three-dimensional reconstruction of objects using visual-tactile sensors based on three-dimensional point cloud stitching belongs to the field of three-dimensional reconstruction of objects operated by robots. First, a visual-tactile sensor is installed at the end of a continuous robotic arm; second, a driving box is controlled by a controller to make the continuous robotic arm touch the object to be measured; third, the visual-tactile sensor collects several local touch images; fourth, the position information of multiple contact points in the world coordinate system is obtained through the kinematics of the continuous robotic arm; finally, the above data is input into a computer to reconstruct the three-dimensional shape of the contact object. The present invention can conveniently and quickly reconstruct the three-dimensional shape of the touched object, so it can effectively use a high-resolution visual-tactile sensor to completely reconstruct the three-dimensional shape of the contact object under visual obstruction, without relying on limited lighting environments, and lays the foundation for robots to implement on-orbit assembly or surgical collaboration in the fields of aerospace, deep sea, or medical fields, similar to tasks that require refined operations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of three-dimensional reconstruction of objects operated by robots, and relates to a three-dimensional reconstruction method of visual-tactile sensor objects based on three-dimensional point cloud splicing. Background Art

[0002] With the development of the times and the progress of science, human society is gradually moving towards automation and intelligence. Robot precision operation has become an urgent need in the fields of scientific research, medical equipment, home services, and assistance for the elderly and the disabled. In the process of realizing robot precision operation, 3D reconstruction of the operated object is necessary. People usually reconstruct 3D shapes through vision.

[0003] Generally speaking, vision-based 3D reconstruction technologies include laser scanning, binocular vision, and multi-eye vision. For example, in the patent for visual-tactile sensor and its 3D reconstruction method, system, device, and storage medium (publication number CN116129037 A), it is proposed to use a binocular camera to capture images, which can realize stereo matching process for image regions, and then complete the 3D reconstruction of multi-curvature objects. However, when the robot performs specific fine operation tasks, there are limitations such as limited viewing angle, occlusion, and complex background, which have an impact on the above-mentioned vision-based methods and produce large errors, which greatly affects the robot's fine operation.

[0004] In recent years, people have used tactile sensors to perform three-dimensional reconstruction. For example, in a method and system for reconstructing anthropomorphic touch objects based on tactile sensors (publication number CN 112802182 A), a six-axis force-torque sensor was used to propose an anthropomorphic touch object reconstruction method based on cluster analysis. However, the manufacturing process of tactile sensors is complex, the spatial resolution is low, and the cost is high. There are still great difficulties in putting them into practical application in industry. They can also obtain very little tactile information of objects, which also has large errors and thus affects the refined operation of robots.

[0005] In view of the above-mentioned problems, the present invention proposes a three-dimensional reconstruction method of a visual-tactile sensor object based on three-dimensional point cloud stitching to improve the robot's refined operation capability. Summary of the invention

[0006] The purpose of the present invention is to provide a three-dimensional reconstruction method of a visual-tactile sensor object based on three-dimensional point cloud stitching to solve the deficiencies in the background technology.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A method for 3D reconstruction of visual tactile sensor objects based on 3D point cloud splicing is implemented based on a drive box, a continuous type mechanical arm, a visual tactile sensor, a controller, a voltage regulator, and a computer. The controller is integrated into the computer. The overall process of the method for 3D reconstruction of visual tactile sensor objects of the present invention is as follows: Figure 1 As shown, the three-dimensional reconstruction method first installs the visual tactile sensor at the end of the continuous robot arm; secondly, the controller controls the driving box to make the continuous robot arm touch the object to be measured; thirdly, the visual tactile sensor collects several local touch images; fourthly, the position information of multiple contact points in the world coordinate system is obtained through the kinematics of the continuous robot arm; finally, the above data is input into the computer to reconstruct the three-dimensional shape of the contact object. Specifically, it includes the following steps:

[0009] The first step is to install the visual tactile sensor at the end of the continuous robot arm as follows:

[0010] 1.1) Using commercial visual tactile sensors, GelSight Mini's tactile intelligence technology digitizes touch with the sensitivity and high resolution of human touch. It is a tactile sensor with ultra-high resolution that can digitally map tactile 2D and 3D. Its contact surface is made of gel material. When it touches an object, the built-in camera captures the deformation of the gel that touches the object with the help of LED lights, and maps the gel deformation information with the tactile information through computer vision algorithms.

[0011] 1.2) Fix the GelSight Mini visual tactile sensor to the end of the continuous robot arm with bolts and connect it to the computer via a USB interface cable to prepare for subsequent data communication and transmission.

[0012] The second step is to control the drive box through the controller to make the continuous type robot arm touch the object to be measured, as follows:

[0013] 2.1) The controller is integrated into the computer and communicates with the drive box through cables to control the operation of the drive box.

[0014] 2.2) The continuous manipulator is a tensegrity structure with a driving rope and spring unit inside. The rope is a pulling structure that runs through the continuous manipulator, causing the continuous manipulator to deform and thus move, so that the visual tactile sensor moves when it contacts an object. The spring is installed between the unit cells of the continuous manipulator.

[0015] 2.3) The continuous type manipulator and the rope are connected to the drive box, which is composed of six array motors and a winding wheel. As a preferred embodiment, the motor composition The output end of the motor is connected to the winding wheel, and the winding wheel rotates under the drive of the motor, and the winding wheel is used to wind the rope.

[0016] 2.4) Fix the position and direction of the touched object, and the controller sends a command to the drive box to rotate the motor and drive the continuous robot arm to move. The command causes the end of the continuous robot arm to move along a plane perpendicular to the normal direction of the touched object to collect real-time data. Starting from the initial point position, keep a sufficiently small uniform sampling interval and move forward. When the end of the continuous robot arm sounds empty in the motion path, the end of the continuous robot arm returns to the initial point and moves downward by one-tenth of the length of the touched object to continue the above action until the entire touched object is scanned.

[0017] In the third step, the visual tactile sensor collects several local touch images, as follows:

[0018] When scanning the touched object, the visual tactile sensor makes multiple contacts with the touched object. The built-in camera of the visual tactile sensor captures the deformation of the gel when it contacts the object with the assistance of LED light, and generates several touch images through the stereo vision theory and image processing algorithm within its commercial system.

[0019] The fourth step is to obtain the position information of multiple contact points in the world coordinate system through continuous robot arm kinematics, as follows:

[0020] 4.1) The kinematics of the continuous robot arm can be divided into the mapping relationship of three spaces. The change of rope length is defined as the drive space, the joint angle is defined as the joint space, and the end position of the continuous robot arm is defined as the task space. If the joint angle is known, the current end position of the continuous robot arm can be solved, which is positive kinematics.

[0021] 4.2) The continuous manipulator is a serial robot and is often modeled using the DH coordinate method.

[0022] 4.3) The relative position and posture relationship between the joints of the robot arm is described by DH parameters. By multiplying the homogeneous change matrices describing the coordinate posture relationship of each joint in sequence, the continuous robot arm end posture can be obtained, and the position information of the contact point in the world coordinate system can be obtained.

[0023] The fifth step is to input the above data into a computer and use the technical solution of the present invention to reconstruct the three-dimensional shape of the contact object, as follows:

[0024] 5.1) The visual tactile sensor collects data in real time and transmits the collected touch images to the computer via a USB interface cable.

[0025] 5.2) Photometric stereo is used to generate several local height images from the collected local touch images.

[0026] The photometric stereo method is a technology in computer vision. Its core idea is to calculate the normal vector of the object surface at each point under different lighting conditions. The photometric stereo method is based on the basic assumptions: each point in the image corresponds to a point on the sensor surface; the color of each point in the image is a function of the normal of the sensor contact surface; on the sensor contact surface, the reflectivity is constant. The process of obtaining the height image by the photometric stereo method is as follows:

[0027] For a surface with Lambertian reflection characteristics, the brightness I of a point (x, y) under the illumination of light source i is i (x,y) can be expressed as: i (x, y)=ρ(x, y)(s i •n(x, y)), where: ρ(x, y) is the reflectivity of the surface (assumed to be constant or known), s i is the direction vector of light source i; n(x, y) is the normal vector of the object surface at point (x, y).

[0028] Assuming there are N different light sources, we get N equations:

[0029] [ I 1 ( x , y ) I 2 ( x , y ) ⋮ I N ( x , y ) ] = ρ ( x , y ) [ s 1 s 2 ⋮ s N ] n ( x , y )

[0030] Simplified to:

[0031]

[0032] Where I(x,y) is an N×1 brightness vector, S is an N×3 light source matrix, and each row is a light source direction vector. The normal vector n(x,y) is obtained by solving the linear equation Sn(x,y)=I(x,y), and then the height map of the object is further solved.

[0033] Then the normal vector n at a certain point = (n x ,n y ,n z ) and the gradient of the object surface are as follows:

[0034]

[0035] Among them, n x ,n y ,n z They are the components of the normal vector at that point along the three mutually perpendicular axes, and the gradient can be calculated:

[0036]

[0037] Among them, p and q represent the surface gradient at that point respectively, and then the height image z(x,y) is restored from the gradient by integration:

[0038]

[0039] The photometric stereo method calculates the normal vector of the local surface of the touched object by utilizing the relationship between the light source direction and the local surface brightness of the touched object, and then derives the surface gradient through the normal vector to obtain a local height image of the touched object.

[0040] 5.3) Convert several local height images into several local point cloud images.

[0041] 5.4) The position information of the end of the continuous robot arm in the world coordinate system obtained in 4.3) is combined with the obtained several local point cloud images to obtain several local point cloud images with position information in the world coordinate system.

[0042] 5.5) The improved CTI (Closest Tactile Imprint) algorithm of the present invention is used to stitch and fuse several local point cloud images together, so as to reconstruct its global shape.

[0043] The core idea of ​​the improved CTI algorithm described in the present invention is to achieve the alignment and stitching of point clouds by finding the closest touch imprint between point clouds. The touch imprint can be understood as a feature point or a small surface in the point cloud. These feature points or feature areas are selected from the point cloud and can reflect some important information of the surface. Its goal is to align and splice several different point cloud image sets to form a unified whole. The touch imprint has a certain uniqueness and stability, so that different point cloud images can be correctly identified and matched, and it has significant local features in the point cloud. Its specific process is as follows:

[0044] 5.5.1) Extracting tactile impressions from several point cloud images:

[0045] The point cloud images obtained in step 5.4) Input to the PointNet model, where p 1 ,p 2 ,…,p N PointNet is a 3D point cloud classification and segmentation model based on deep learning. It can stably extract point set features even if the point cloud is fluctuating, noisy or missing.

[0046] Extract local features of each point cloud image through the multi-layer perceptron (MLP) in the PointNet model;

[0047] The global features of each point cloud image are extracted through the Max Pool layer in the PointNet model;

[0048] The local features and the global features are combined to obtain the final feature description, that is, to obtain several local point cloud images with tactile imprints.

[0049] 5.5.2) Align the point cloud images with tactile imprints obtained in step 5.5.1):

[0050] Aligning several local point cloud images with tactile imprints in order of acquisition time, with two groups of point cloud images as one group;

[0051] For example, for two sets of local point cloud images A and B, initialize the transformation matrix T as the unit matrix and set the maximum number of iterations N. max And convergence threshold ε; for each point a in point cloud A i A, find the point b closest to it in point cloud B i B:

[0052] , where b is a traversal point in point cloud B.

[0053] Calculate all matching point pairs (a i , b i )’s mean square error:

[0054] , where a i , b i They are the closest point in point cloud A to point cloud B and the closest point in point cloud B to point cloud A respectively.

[0055] Compute the centroids of two point clouds:

[0056]

[0057] De-centre the two point clouds:

[0058]

[0059] Calculate the covariance matrix H:

[0060] , where the matrix For the matrix The transpose of .

[0061] Perform singular value decomposition on the covariance matrix H:

[0062] , where the matrices U and V are two orthogonal matrices, and the matrix V Tis the transpose of the orthogonal matrix V.

[0063] According to the result of singular value decomposition, calculate the rotation matrix R and translation vector t:

[0064] , where the matrix U T is the transpose of the orthogonal matrix U.

[0065] Use the transformation matrix T to transform the point cloud A:

[0066]

[0067] Update point cloud A to transformed point cloud A', repeat the above steps until the error E is less than the convergence threshold ε or reaches the maximum number of iterations N max .

[0068] Through the above iterations, the distance between point clouds is minimized, the optimal transformation matrix is ​​found, and the point clouds are aligned;

[0069] 5.5.3) Fusion and post-processing of several aligned local point cloud images:

[0070] Initialize an empty global point cloud container and add the point cloud image obtained in step 5.5.2) to the global point cloud container;

[0071] The merged point cloud may have overlapping areas, resulting in duplicate points. In order to remove duplicate points, the point cloud is voxelized and only one representative point is retained in each voxel. For each point, the local surface is fitted using its neighborhood points, and the position of the point is recalculated to achieve a smoothing effect.

[0072] Based on the above steps, the three-dimensional shape of the object is reconstructed.

[0073] The beneficial effects of the present invention are as follows:

[0074] The present invention is a method for 3D reconstruction of objects using visual-tactile sensors based on 3D point cloud stitching, which uses commercial visual-tactile sensors with high resolution, fast real-time performance and multi-modality, combined with a continuous manipulator to touch the object being measured multiple times. The improved algorithm of the present invention is used to conveniently and quickly reconstruct the 3D shape of the touched object, so it can effectively use high-resolution visual-tactile sensors to completely reconstruct the 3D shape of the contact object under visual obstruction conditions, independent of limited lighting environments, which lays the foundation for robots to perform on-orbit assembly or surgical collaboration in the fields of aerospace, deep sea or medical fields, and similar tasks that require refined operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a schematic diagram of the method of the present invention;

[0076] Figure 2 This is a schematic diagram of an embodiment of a continuous robot of the present invention;

[0077] Figure 3 This is a schematic diagram of the continuous type mechanical arm of the present invention;

[0078] Figure 4 It is a schematic diagram of the drive box of the present invention;

[0079] Figure 5 It is a schematic diagram of the continuous robot arm motion control process of the present invention;

[0080] Figure 6 The present invention reconstructs a depth map of a local object based on the photometric stereo method;

[0081] Figure 7 A flowchart of the improved CTI algorithm of the present invention;

[0082] Figure 8 The reconstructed 3D point cloud of the object. DETAILED DESCRIPTION

[0083] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0084] A method for 3D reconstruction of visual tactile sensor objects based on 3D point cloud splicing is implemented based on a drive box, a continuous type mechanical arm, a visual tactile sensor, a controller, a voltage regulator, and a computer. The controller is integrated into the computer. The overall process of the method for 3D reconstruction of visual tactile sensor objects of the present invention is as follows: Figure 1 As shown: a visual-tactile sensor is installed at the end of a continuous robotic arm; a driving box is controlled by a controller to make the continuous robotic arm touch the object to be measured; the visual-tactile sensor collects several local touch images; the position information of multiple contact points in the world coordinate system is obtained through the kinematics of the continuous robotic arm; the above data is input into a computer to reconstruct the three-dimensional shape of the contact object.

[0085] The first step is to install the visual tactile sensor at the end of the continuous robot arm:

[0086] The tactile sensor used in the present invention is a commercial visual tactile sensor - GelSightMini. The tactile intelligence technology of GelSight Mini digitizes touch with the sensitivity and high resolution of human touch. It is a tactile sensor with ultra-high resolution that can map tactile 2D and 3D digitally. Its contact surface is made of gel material. When it contacts an object, the built-in camera captures the deformation of the gel that contacts the object, and maps the gel deformation information with the tactile information through the algorithm of computer vision. The GelSight Mini visual tactile sensor is fixed to the end of the continuous robot arm by bolts, and connected to the computer through a USB interface cable for data communication and transmission.

[0087] The second step is to control the drive box through the controller to make the continuous robot arm touch the object to be measured:

[0088] The controller is integrated into the computer and communicates with the drive box through cables to control the operation of the drive box. Figure 2 As shown in the figure, from right to left are the visual tactile sensor, the continuous type robot arm, and the drive box. The continuous type robot arm is a flexible arm. The flexible arm is installed on the drive box and driven by the driver array, and the driver array is installed inside the drive box. The visual tactile sensor is installed at the end of the flexible arm to perform the detection task.

[0089] The continuous type mechanical arm is a tensegrity structure, such as Figure 3 As shown. Six driving ropes are arranged inside the continuous robot, and the driving ropes are connected to the driving box. The driving ropes are of a pull-type structure. The driving ropes pass through the continuous robot arm and are coupled and stretched to deform the continuous robot arm, thereby realizing the spatial movement of the continuous robot arm and moving the tactile sensor. The continuous robot arm also includes a stiffness spring installed between each unit cell, and the stiffness spring is connected to each unit cell through a hanging hole.

[0090] The drive box is as Figure 4 As shown, it includes a motor array and a winding wheel. The motor array is installed in a staggered form inside the drive box to form a compact integrated system. As a preferred embodiment, the motor array includes an array of 6 motors forming a 2*3 formation. The winding wheel is connected to the output end of the motor and rotates under the drive of the motor, and the winding wheel is used to wind the rope.

[0091] In order to enable the end of the continuous robot arm to reach the target position smoothly, it is necessary to control the continuous robot arm to move to the target position. Figure 5The figure shows the process of continuous robot arm motion control. During the motion process, the subsequent part of the continuous robot arm moves along the path of the previous part, starting from the initial point position, and moving forward with a sufficiently small sampling interval. When the end of the continuous robot arm sounds empty in the motion path, the end of the continuous robot arm returns to the initial point and moves downward by one tenth of the length of the touched object to continue the above action until the entire touched object is scanned. The voltage stabilizer supplies power to the drive box.

[0092] In the third step, the visual tactile sensor collects several local touch images:

[0093] When scanning the touched object, the visual tactile sensor makes multiple contacts with the touched object. The built-in camera of the visual tactile sensor captures the deformation of the gel when it contacts the object with the assistance of LED light, and generates several touch images through the stereo vision theory and image processing algorithm within its commercial system.

[0094] The fourth step is to obtain the position information of multiple contact points in the world coordinate system through continuous robot arm kinematics:

[0095] The kinematics of the continuous robot arm can be divided into the mapping relationship of three spaces. The change of rope length is defined as the driving space, the joint angle is defined as the joint space, and the end posture of the continuous robot arm is defined as the task space. If the joint angle is known, the current end posture of the continuous robot arm can be solved, which is positive kinematics.

[0096] A continuous manipulator is a serial robot that is often modeled using the DH coordinate method.

[0097] The relative position and posture relationship between the joints of the robot arm is described by DH parameters. By multiplying the homogeneous change matrices describing the coordinate posture relationship of each joint in sequence, the continuous robot arm end posture can be obtained, and the position information of the contact point in the world coordinate system is obtained.

[0098] Step 5: Input the above data into a computer and use the technical solution of the present invention to reconstruct the three-dimensional shape of the contact object:

[0099] The visual tactile sensor collects data in real time and transmits the collected touch images to the computer via a USB interface cable.

[0100] Photometric stereo is used to generate several height images from several collected local touch images.

[0101] The photometric stereo method is a technology in computer vision. Its core idea is to calculate the normal vector of the object surface at each point under different lighting conditions. The photometric stereo method is based on the basic assumptions: each point in the image corresponds to a point on the sensor surface; the color of each point in the image is a function of the normal of the sensor contact surface; on the sensor contact surface, the reflectivity is constant. The process of obtaining the height image by the photometric stereo method is as follows:

[0102] For a surface with Lambertian reflection characteristics, the brightness I of a point (x, y) under the illumination of light source i is i (x,y) can be expressed as: i (x, y)=ρ(x, y)(s i •n(x, y)), where: ρ(x, y) is the reflectivity of the surface (assumed to be constant or known), s i is the direction vector of light source i; n(x, y) is the normal vector of the object surface at point (x, y).

[0103] Assuming there are N different light sources, we get N equations:

[0104] [ I 1 ( x , y ) I 2 ( x , y ) ⋮ I N ( x , y ) ] = ρ ( x , y ) [ s 1 s 2 ⋮ s N ] n ( x , y )

[0105] Simplified to:

[0106]

[0107] Where I(x,y) is an N×1 brightness vector, S is an N×3 light source matrix, and each row is a light source direction vector. The normal vector n(x,y) is obtained by solving the linear equation Sn(x,y)=I(x,y), and then the height map of the object is further solved.

[0108] Then the normal vector n at a certain point = (n x ,n y ,n z ) and the gradient of the object surface are as follows:

[0109]

[0110] where n x ,n y ,n z They are the components of the normal vector at that point along the three mutually perpendicular axes, and the gradient can be calculated:

[0111]

[0112] Where p and q represent the surface gradient at that point, respectively, and then the height image z(x, y) is restored from the gradient by integration:

[0113]

[0114] The photometric stereo method calculates the normal vector of the local surface of the touched object by using the relationship between the light source direction and the local surface brightness of the touched object, and then derives the surface gradient through the normal vector to obtain the local height image of the touched object. Figure 6 As shown, this is the top layout height image of the scissors object in this case.

[0115] The local height images are converted into a number of local point cloud images through a computer program.

[0116] By combining the position information of the end of the continuous robot arm in the world coordinate system obtained above with the obtained several local point cloud images, several local point cloud images with position information in the world coordinate system can be obtained.

[0117] The improved CTI (Closest Tactile Imprint) algorithm of the present invention is used to stitch and fuse several local point cloud images together, so as to reconstruct its global shape. The process is as follows Figure 7 As shown in the figure, tactile marks are extracted from several local point cloud images; the point cloud images with tactile marks obtained in the previous step are aligned; the aligned point cloud images are fused; and finally, post-processing is performed to remove duplicate points in the global point cloud image, filter out noise and smooth the image.

[0118] The core idea of ​​the improved CTI algorithm described in the present invention is to achieve the alignment and stitching of point clouds by finding the closest touch imprint between point clouds. The touch imprint can be understood as a feature point or a small surface in the point cloud. These feature points or feature areas are selected from the point cloud and can reflect some important information of the surface. Its goal is to align and splice several different point cloud image sets to form a unified whole. The touch imprint has a certain uniqueness and stability, so that different point cloud images can be correctly identified and matched, and it has significant local features in the point cloud. Its specific process is as follows:

[0119] Extract tactile impressions from several point cloud images:

[0120] The several point cloud images obtained previously Input to the PointNet model, where p 1 ,p 2 ,…,p NPointNet is a 3D point cloud classification and segmentation model based on deep learning. It can stably extract point set features even if the point cloud is fluctuating, noisy or missing.

[0121] Extract local features of each point cloud image through the multi-layer perceptron (MLP) in the PointNet model;

[0122] The global features of each point cloud image are extracted through the Max Pool layer in the PointNet model;

[0123] The local features and the global features are combined to obtain the final feature description, that is, to obtain several local point cloud images with tactile imprints.

[0124] Align several point cloud images with tactile imprints obtained in the previous step:

[0125] Aligning several local point cloud images with tactile imprints in order of acquisition time, with two groups of point cloud images as one group;

[0126] For example, for two sets of local point cloud images A and B, initialize the transformation matrix T as the unit matrix and set the maximum number of iterations N. max And convergence threshold ε; for each point a in point cloud A i A, find the point b closest to it in point cloud B i B:

[0127] , where b is a traversal point in point cloud B.

[0128] Calculate all matching point pairs (a i , b i )’s mean square error:

[0129] , where a i , b i They are the closest point in point cloud A to point cloud B and the closest point in point cloud B to point cloud A respectively.

[0130] Compute the centroids of two point clouds:

[0131]

[0132] De-centre the two point clouds:

[0133]

[0134] Calculate the covariance matrix H:

[0135] , where the matrix For the matrix The transpose of .

[0136] Perform singular value decomposition on the covariance matrix H:

[0137] , where the matrices U and V are two orthogonal matrices, and the matrix V T is the transpose of the orthogonal matrix V.

[0138] According to the result of singular value decomposition, calculate the rotation matrix R and translation vector t:

[0139] , where the matrix U T is the transpose of the orthogonal matrix U.

[0140] Use the transformation matrix T to transform the point cloud A:

[0141]

[0142] Update point cloud A to transformed point cloud A', repeat the above steps until the error E is less than the convergence threshold ε or reaches the maximum number of iterations N max .

[0143] Through the above iterations, the distance between point clouds is minimized, the optimal transformation matrix is ​​found, and the point clouds are aligned;

[0144] Fusion and post-processing of several aligned local point cloud images:

[0145] Initialize an empty global point cloud container and add the point cloud image obtained in the previous step to the global point cloud container;

[0146] The merged point cloud may have overlapping areas, resulting in duplicate points. In order to remove duplicate points, the point cloud is voxelized and only one representative point is retained in each voxel. For each point, the local surface is fitted using its neighborhood points, and the position of the point is recalculated to achieve a smoothing effect.

[0147] Based on the above steps, the global point cloud image of the touched object scissors in this case is reconstructed as follows Figure 8 As shown, it has a good similarity to its real three-dimensional appearance.

[0148] It should be noted that in various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for three-dimensional reconstruction of visual tactile sensor objects based on three-dimensional point cloud splicing, characterized in that: The three-dimensional reconstruction method The following steps are involved: The first step is to install the visual tactile sensor on the end of the continuous robot arm; The second step is to control the drive box through the controller to make the continuous type robot arm touch the object to be measured, as follows: First, the controller is integrated into the computer to communicate with the drive box; the continuous type robot arm and the rope are connected to the drive box; Secondly, the position and direction of the touched object are fixed, and the controller sends a command to the drive box to rotate the motor, driving the continuous robot arm to move until the entire touched object is scanned; In the third step, when scanning the touched object, the visual tactile sensor collects several local touch images; The fourth step is to obtain the position information of multiple contact points in the world coordinate system through continuous robot arm kinematics, as follows: Step 4.1, the kinematics of the continuous manipulator is divided into the mapping relationship of three spaces, the rope length change of the continuous manipulator is defined as the driving space, the joint angle is defined as the joint space, and the end position of the continuous manipulator is defined as the task space; the current end position of the continuous manipulator is solved by knowing the joint angle; Step 4.2: The continuous robot arm is modeled using the DH coordinate method; Step 4.3, the relative position and posture relationship between the joints of the continuous manipulator is described by DH parameters, and the homogeneous change matrices describing the coordinate posture relationship of each joint are multiplied in sequence to obtain the end posture of the continuous manipulator, and the position information of the contact point in the world coordinate system is obtained; The fifth step is to input the position information into the computer to reconstruct the three-dimensional shape of the contact object, as follows: Step 5.1, transmitting a number of local touch images collected in real time by the visual tactile sensor to a computer; Step 5.2, using photometric stereo method to generate a number of local height images from the collected local touch images; Step 5.3, converting the local height images obtained in step 5.2 into a number of local point cloud images; Step 5.4, combining the position information of the end of the continuous robot arm in the world coordinate system obtained in step 4.3 with the several local point cloud images obtained in step 5.3, to obtain several local point cloud images with position information in the world coordinate system; Step 5.5, using the improved CTI algorithm to stitch and fuse the several local point cloud images obtained in step 5.4 together to reconstruct its global shape; the CTI algorithm realizes the alignment and stitching of point clouds by finding the closest touch mark between point clouds; the touch mark is a feature point or feature area in the point cloud; the goal is to align and stitch several different point cloud image sets to form a unified whole.

2. The method for three-dimensional reconstruction of visual-tactile sensor objects based on three-dimensional point cloud splicing according to claim 1, characterized in that: In the second step, the continuous robotic arm is a serial robot composed of multiple continuous robotic arm cells; it is a tensile structure with a driving rope and a spring unit inside; the rope is a pulling structure, which runs through the continuous robotic arm, causing the continuous robotic arm to deform and move, and the visual tactile sensor to move when it contacts an object; the spring is installed between the continuous robotic arm cells.

3. The method for three-dimensional reconstruction of visual-tactile sensor objects based on three-dimensional point cloud splicing according to claim 1, characterized in that: In the second step, the command causes the end of the continuous robotic arm to move in a plane perpendicular to the normal direction of the touched object to collect real-time data; starting from the initial point position, it moves forward while maintaining a uniform sampling interval. When the end of the continuous robotic arm detects empty space in the motion path, the end of the continuous robotic arm returns to the initial point and moves downward by one tenth of the length of the touched object to continue the above action until the entire touched object is scanned.

4. The method for three-dimensional reconstruction of visual-tactile sensor objects based on three-dimensional point cloud splicing according to claim 1, characterized in that: In the third step, when scanning the touched object, the visual tactile sensor makes multiple contacts with the touched object. The built-in camera of the visual tactile sensor captures the deformation of the gel when it contacts the object with the assistance of LED light, generating several touch images.

5. The method for three-dimensional reconstruction of visual-tactile sensor objects based on three-dimensional point cloud splicing according to claim 1, characterized in that: In step 5.2, the process of obtaining the height image by photometric stereo method is as follows: For a surface with Lambertian reflection characteristics, the brightness I of a point (x, y) under the illumination of light source i is i (x, y) is expressed as: i (x, y)=ρ(x, y)(s i •n(x, y)), Where: ρ(x, y) is the reflectivity of the surface of the object, s i is the direction vector of light source i; n(x, y) is the normal vector of the object surface at point (x, y); Assuming there are N different light sources, we get N equations: Simplified to: Where I(x,y) is an N×1 brightness vector, S is an N×3 light source matrix, and each row is a light source direction vector. The normal vector n(x,y) is obtained by solving the linear equation system Sn(x,y)=I(x,y), and the height map of the object is further solved. Then the normal vector n at a certain point = (n x ,n y ,n z ) and the gradient of the object surface are as follows: Among them, n x ,n y ,n z The components of the normal vector at that point along the three mutually perpendicular axes are calculated respectively, and the gradient is: Among them, p and q represent the surface gradient at that point respectively, and then the height image z(x,y) is restored from the gradient by integration: The photometric stereo method calculates the normal vector of the local surface of the touched object by using the relationship between the light source direction and the local surface brightness of the touched object, and then derives the surface gradient through the normal vector to obtain the local height image of the touched object.

6. The method for three-dimensional reconstruction of visual-tactile sensor objects based on three-dimensional point cloud splicing according to claim 1, characterized in that: The specific process of step 5.5 is as follows: Step 5.5.1: Extract tactile impressions of several point cloud images: The point cloud images obtained in step 5.4 Input to the PointNet model, where p1, p2, ..., p N are several local point cloud images with position information in the world coordinate system; The local features of each point cloud image are extracted through the multi-layer perceptron in the PointNet model; The global features of each point cloud image are extracted through the maximum pooling layer in the PointNet model; The local features and the global features are combined to obtain the final feature description, that is, to obtain several local point cloud images with tactile imprints; Step 5.5.2: Align the point cloud images with tactile imprints obtained in step 5.5.1: Aligning several local point cloud images with tactile imprints in order of acquisition time, with two groups of point cloud images as one group; For example, for two sets of local point cloud images A and B, initialize the transformation matrix T as the unit matrix and set the maximum number of iterations N. max And convergence threshold ε; for each point a in point cloud A i A, find the point b closest to it in point cloud B i B: , where b is the traversed point in point cloud B; Calculate all matching point pairs (a i , b i )’s mean square error: , where a i , b i They are the closest point in point cloud A to point cloud B and the closest point in point cloud B to point cloud A respectively; Compute the centroids of two point clouds: De-centre the two point clouds: Calculate the covariance matrix H: , where the matrix For the matrix The transpose of Perform singular value decomposition on the covariance matrix H: , where the matrices U and V are two orthogonal matrices, and the matrix V T is the transpose of the orthogonal matrix V; According to the result of singular value decomposition, calculate the rotation matrix R and translation vector t: , where the matrix U T is the transpose of the orthogonal matrix U; Use the transformation matrix T to transform the point cloud A: Update point cloud A to transformed point cloud A', repeat the above steps until the error E is less than the convergence threshold ε or reaches the maximum number of iterations N max ; Through the above iterations, the distance between point clouds is minimized, the optimal transformation matrix is ​​found, and the point clouds are aligned; Step 5.5.3: Fusion and post-processing of the aligned local point cloud images: Initialize an empty global point cloud container and add the point cloud image obtained in step 5.5.2 to the global point cloud container; The merged point cloud may have overlapping areas, resulting in duplicate points. In order to remove duplicate points, the point cloud is voxelized and only one representative point is retained in each voxel. For each point, the local surface is fitted using its neighborhood points, and the position of the point is recalculated to achieve a smoothing effect. Based on the above steps, the three-dimensional shape of the object is reconstructed.