Active three-dimensional reconstruction method for visual and tactile fusion

Abstract

The invention belongs to the technical field of computer vision, computer graphics, robots and algorithm design, and discloses an active three-dimensional reconstruction method for visual and tactile fusion. The system specifically comprises the following two modules: a three-dimensional reconstruction module and a strategy estimation module. The two modules carry out the following steps: module input, module architecture, a training method and a prediction process. According to the method, a diffusion model technology and a reinforcement learning technology are utilized to combine a vision mode and a touch mode, the three-dimensional shape of a target is actively sensed and reconstructed, and the problem that local details of a reconstruction result are not accurate due to the fact that a traditional vision-based three-dimensional reconstruction method is limited by factors such as illumination and shielding is solved; the problems that an active reconstruction method is large in observation space and low in efficiency due to the fact that the dimension of information needed by decision making is high are solved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, computer graphics, robotics and algorithm design technology, and in particular to an active three-dimensional reconstruction method of vision-tactile fusion. Background Art

[0002] 3D reconstruction has long been a hot topic in computer vision and graphics. Especially in today's era of big data and large models, acquiring 3D data is more challenging than 2D images and text, and dataset collection is expensive. Therefore, high-quality 3D reconstruction capabilities are crucial. 3D reconstruction from incomplete information has potential applications in areas such as autonomous driving and robotic interaction, where environmental occlusion and camera measurement errors are common.

[0003] Currently, research on 3D shape reconstruction mainly focuses on the field of visual perception. However, due to the high dimensionality of the observation space and interference from factors such as occlusion and illumination, training computer vision algorithms to perform object manipulation faces huge challenges. Recently, some researchers have proposed strategies based on high-resolution tactile perception as a complementary solution to target exploration and reconstruction problems. Compared with camera-based perception, this perception method can obtain richer contact information and still perform well in the face of occlusion, homogeneous surfaces, and transparent or reflective materials. In addition, due to the smaller observation space required, tactile perception also simplifies the transition from simulation tasks to real scenes. In recent years, some methods have combined visual perception with tactile perception to explore new three-dimensional reconstruction technologies.

[0004] These methods have the following characteristics:

[0005] 1. This type of method uses the end sensor of the robotic arm to contact the target and obtain a tactile image. It then uses deep learning technology to predict the local point cloud corresponding to the tactile image. By combining all the predicted local point clouds, it reconstructs the global shape of the target.

[0006] 2. These methods process tactile signals by analogy with vision-based deep learning methods. However, they ignore the differences between vision and touch, especially how to fully utilize the complementary characteristics of visual and tactile information in global context and local details.

[0007] 3. The active reconstruction of this type of method plans the location of the next tactile perception by judging the areas with high uncertainty in the current reconstruction results. Therefore, each interaction requires the reconstruction of the complete shape of the target, which is inefficient and wastes resources.

[0008] Based on the above analysis, an active three-dimensional reconstruction method that can fully take into account the global context prediction ability of visual information and the local structure analysis ability of tactile information, and can effectively utilize information during the reconstruction process, is a technical difficulty that needs to be overcome at present. This is also the motivation for the invention of this invention. Summary of the Invention

[0009] The purpose of this invention is to combine vision and touch modalities using diffusion modeling and reinforcement learning techniques to actively perceive and reconstruct the three-dimensional shape of an object. This approach addresses the limitations of traditional vision-based 3D reconstruction methods, such as lighting and occlusion, which lead to inaccurate local details in the reconstruction results. It also addresses the inefficiency of active reconstruction methods due to the large observation space and the high dimensionality of information required for decision-making. This method utilizes both visual and tactile perception to autonomously reconstruct an object in three dimensions.

[0010] The technical solution of the present invention is as follows: an active 3D reconstruction method of visual-tactile fusion obtains a single-perspective visual image, and predicts the 3D global information of the target through a visual-tactile conditional coding module and a 3D reconstruction module; based on the 3D global information, the optimal interaction posture of the tactile sensor at the end of the next robotic arm is determined through a strategy estimation module, and the robotic arm is driven to contact the target according to the optimal interaction posture and obtain a tactile image; the visual image and the tactile image are encoded and feature-fused using the visual-tactile conditional coding module, a low-resolution 3D feature map is obtained through the 3D reconstruction module, and the strategy estimation module is continued to be used to determine the next optimal interaction posture, and it is iterated until the termination condition is met; finally, the low-resolution 3D feature map is reconstructed into a 3D volume through a 3D decoding network, and then a 3D shape is generated by a voxel-level reconstruction method.

[0011] The 3D reconstruction module includes a 3D autoencoder network and a 3D denoising network; the 3D autoencoder network and the 3D diffusion model are trained separately by a stochastic gradient descent method;

[0012] During the training process, the training data set contains a 3D model and the corresponding visual and tactile images. The 3D model is processed into a 3D volume represented by a signed distance field (SDF).

[0013] First, the 3D autoencoder network is trained to calculate the error L between the predicted 3D volume and the true value of the 3D volume in the training dataset. ed , and calculate the gradient of the error, and gradually update the parameters of the three-dimensional autoencoder network along the direction of gradient descent according to the back propagation of the three-dimensional autoencoder network, and iterate until convergence;

[0014] L ed =|X′-X| (1)

[0015] After training, the 3D encoding network in the 3D autoencoder network extracts the feature maps of all 3D bodies in the training data set, iteratively adds noise to the feature maps, and uses the 3D denoising network to predict the noise. The error loss L between the predicted noise and the actual added noise is calculated according to formula (2): diff , reversely update the network parameters of the 3D denoising network according to the error loss;

[0016] L diff =||N θ (z t ,t,F)-∈ t ||2 (2)

[0017] ∈ t The noise added for iteration t.

[0018] The three-dimensional self-encoding network includes a three-dimensional encoding network and a three-dimensional decoding network;

[0019] The three-dimensional coding network is a three-dimensional convolutional neural network, the input is a three-dimensional volume X represented by a signed distance field SDF, and the encoding is a low-resolution, compressed three-dimensional feature map; the three-dimensional coding network is defined as E s , then the low-resolution, compressed three-dimensional feature map z is:

[0020] z=E s (X) (3)

[0021] The three-dimensional decoding network is a three-dimensional deconvolution neural network; a VQ quantization operation is performed on the low-resolution, compressed three-dimensional feature map z, and then decoded into a high-resolution three-dimensional volume through the three-dimensional deconvolution neural network; the three-dimensional decoding network is defined as D s , then the decoded three-dimensional body X′ is:

[0022] X′=D s (VQ(z)) (4)

[0023] The three-dimensional denoising network inputs the conditional code F output by the visual-tactile conditional coding module, and adds noise to the low-resolution, compressed three-dimensional feature map z through the diffusion model for t iterations to obtain the three-dimensional feature map z after adding noise. t , and then use the three-dimensional denoising network N θ According to formula (5), the added noise ∈′ is predicted t , the three-dimensional feature map z after adding noise t Remove noise ∈′ t Then generate a low-resolution three-dimensional feature map z′ t ;

[0024] ∈′ t =N θ (z t,t,F) (5).

[0025] The input of the visual-tactile conditional encoding module has two modes, the first is a single visual image input, and the other is a visual-tactile fusion input; in the first input mode, the input visual image is extracted through the image feature extraction network to extract m labeled visual features V = {v1, v2, ..., v m In the second input mode, in addition to extracting the visual features of a single image, the depth information corresponding to the perceived tactile image is predicted through the depth estimation network, projected into a three-dimensional point cloud according to the interactive posture, and the three-dimensional point cloud is sent to the point cloud feature extraction network to obtain the point cloud features corresponding to each tactile image. Each point cloud feature is regarded as a labeled tactile feature. Assuming there are n tactile images, a total of n labeled tactile feature sets C = {c1, c2, ..., c n}; In the single visual image input mode, since there are only visual features, the final conditional encoding F=V={v1,v2,...,v m}; In the visual-tactile fusion mode, the conditional encoding F=V∪C={v1,v2,...,v m ,c1,c2,...,c n}.

[0026] During the prediction process, the three-dimensional feature map is initialized to z using random noise t , using the conditional code F, according to formula (5) through the trained three-dimensional denoising network iteratively predict the noise ∈ ′ t ,, remove noise ∈′ from the 3D feature map t And generate shape feature z′ t Finally, the output 3D volume X′ is obtained through the trained 3D decoding network, and then the 3D shape is generated by the voxel-level reconstruction method.

[0027] The strategy estimation module determines the optimal interaction posture by predicting the value of each predefined interaction posture, drives the robotic arm to reach the corresponding optimal interaction posture, completes the interaction and obtains a new tactile image.

[0028] The input of the strategy estimation module is two parts. The first part is the low-resolution 3D feature map generated by the trained 3D denoising network. The second part is the set of predefined interaction poses. Assuming that the predefined poses performed by the robot arm contain k, the set of predefined interaction poses is represented as A = {a1, a2, ..., a k};

[0029] The strategy estimation module is a neural network mainly composed of a multi-layer perceptron. It connects the input features and the predefined interaction posture a, inputs them into the multi-layer perceptron, and predicts the value of the predefined posture. The value corresponding to the interaction posture a is formula (6):

[0030] value a =Q(z′ t ,a) (6)

[0031] Q is the Q network;

[0032] The strategy estimation module is trained using a reinforcement learning method. The entire process is iterative. In one round, assuming the current number of steps is s, all the tactile input features currently acquired are {c1, c2, ..., c s}; Calculate the reward according to formula (7):

[0033]

[0034] After accumulating the rewards at each time step, the network is trained through Q learning. The loss function is calculated according to formula (8), and then the loss function is optimized to obtain a network that can predict the value of the interaction pose.

[0035] L rl =[R+γmaxQ(z′ t ,a s+1 )-Q(z′ t ,a s )] 2 (8)

[0036] Where γ is the discount factor.

[0037] After training the strategy estimation module, the shape feature z′t generated by the 3D reconstruction module is input into the prediction process. The value of each predefined interaction posture is calculated by the Q network according to formula (6). The interaction posture with the highest value is selected. The robot arm contacts the target according to the interaction posture and obtains a tactile image. The obtained new tactile image is added to the tactile condition feature set C. The 3D reconstruction module is used to generate a new shape feature z′t based on the updated feature set. The previous interaction posture judgment and tactile perception are iteratively repeated until the termination condition is met and the iteration stops.

[0038] Beneficial effects of the present invention: The proposed active three-dimensional reconstruction method of visual-tactile fusion can take advantage of the strong global context prediction ability of visual information, guide the exploration of touch through the information provided by vision, and then further plan the interactive posture through the accumulation of touch, and iteratively optimize the reconstruction results. The difference between this method and other three-dimensional reconstruction methods based on deep learning is that it effectively plays the complementary advantages of vision and touch, and realizes the modal collaboration of vision and touch through interaction, breaking through the limitations of previous methods that the reconstruction results are only similar globally but ignore the local details of the reconstruction. In addition, since only low-resolution features generated by the diffusion model are used in the strategy estimation, only the complete shape needs to be generated in the end, avoiding the waste of resources that need to predict the complete shape in each iterative process. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Schematic diagram of active 3D reconstruction for visual-tactile fusion;

[0040] Figure 2 This is a network diagram of the 3D reconstruction module;

[0041] Figure 3 Network diagram of the policy estimation module. DETAILED DESCRIPTION

[0042] An active 3D reconstruction method using visual and tactile fusion includes a 3D reconstruction module and a strategy estimation module. A conventional camera is used to capture a single-view visual image of a target. The 3D global information of the target is predicted using a visual and tactile conditional encoding module and a 3D reconstruction module. Based on this 3D global information, the strategy estimation module is used to determine the optimal interaction pose of the tactile sensor at the end of the robotic arm. The robotic arm is then driven to the optimal interaction pose to generate a tactile image. The visual and tactile images are then fused and passed through the 3D reconstruction module to obtain a low-resolution 3D feature map. The strategy estimation module is then used to determine the next optimal interaction pose, and the process is repeated until a termination condition is met. Finally, the low-resolution 3D feature map is reconstructed into a 3D volume using a 3D decoding network, and then a voxel-level reconstruction method is used to generate a 3D shape.

[0043] (1) Visual-tactile conditional encoding module

[0044] The input of the visual-tactile conditional encoding module has two modes: the first is a single visual image input, and the other is a visual-tactile fusion input. In the first input mode, the input visual image is extracted through the image feature extraction network to extract m labeled visual features V = {v1, v2, ..., v mIn the second input mode, in addition to extracting the visual features of a single image, the depth information corresponding to the perceived tactile image is predicted through the depth estimation network, projected into a three-dimensional point cloud according to the interactive posture, and the three-dimensional point cloud is sent to the point cloud feature extraction network to obtain the point cloud features corresponding to each tactile image. Each point cloud feature is regarded as a labeled tactile feature. Assuming there are n tactile images, a total of n labeled tactile feature sets C = {c1, c2, ..., c n}; In the single visual image input mode, since there are only visual features, the final conditional encoding F=V={v1,v2,...,v m}; In the visual-tactile fusion mode, the conditional encoding F=V∪C={v1,v2,...,v m ,c1,c2,...,c n}.

[0045] (2) 3D reconstruction module

[0046] The 3D reconstruction module contains two neural networks. The first neural network is a 3D autoencoder network, which consists of a 3D encoding network and a 3D decoding network; the second neural network is a 3D denoising network.

[0047] The three-dimensional encoding network is a three-dimensional convolutional neural network that encodes high-dimensional three-dimensional input into low-resolution, compressed three-dimensional feature maps. The three-dimensional encoding network is defined as E s , the three-dimensional input is a three-dimensional volume X represented by a conforming distance field (SDF), then the low-resolution, compressed three-dimensional feature map z is:

[0048] z=E s (X) (1)

[0049] The three-dimensional decoding network first performs VQ quantization on the low-resolution, compressed three-dimensional feature map z, and then decodes it into a high-resolution three-dimensional volume. The three-dimensional decoding network is defined as D s , then the decoded three-dimensional body X′ is:

[0050] X′=D s (VQ(z)) (2)

[0051] The three-dimensional denoising network takes the conditional features F encoded by the visual-tactile conditional encoding module as input, iteratively adds noise to the low-resolution, compressed three-dimensional feature map z, and then performs network prediction on the added three-dimensional noise to generate three-dimensional shape features. The denoising network is defined as N θ , the current number of iterations is t, z t The noise feature after adding noise in the tth iteration, the noise predicted by the network ∈ ′ t for:

[0052] ∈′ t =N θ (z t ,t,F) (3)

[0053] z t Removing the noise ∈′ can generate the three-dimensional shape feature z′ under the tth iteration t .

[0054] Both the 3D autoencoder network and the 3D denoising network are trained using the stochastic gradient descent method. The training dataset contains paired 3D models, corresponding visual images, and tactile images. First, the 3D autoencoder network is trained and the error L between the predicted 3D volume and the true value of the database is calculated according to formula (4): ed , and calculate the gradient of the error, and gradually update the network parameters in the direction of gradient descent according to the back propagation of the neural network, and iterate until convergence.

[0055] L ed =|X′-X| (4)

[0056] Then, the 3D encoding network in the trained 3D autoencoder network is used to extract the features of all 3D models in the training dataset, noise is iteratively added to the features, and the 3D denoising network is used to predict the noise. The error L between the predicted noise and the actual added error is calculated according to formula (5): diff :

[0057] L diff =||N θ (z t ,t,F) -∈ t ||2 (5)

[0058] After training the 3D autoencoder network and the 3D denoising network, during the prediction process, the 3D feature map is initialized to z using random noise. t , using the conditional feature F input by the visual-tactile conditional encoding module, the trained three-dimensional denoising network is used to iteratively predict the noise ∈′ through formula (3) t , the three-dimensional feature map z after adding noise t Remove noise ∈′ t Then generate a low-resolution three-dimensional feature map z′ t Finally, the trained 3D decoding network is used to predict the 3D volume using formula (2).

[0059] (3) Strategy Estimation Module

[0060] The strategy estimation module determines the optimal interaction posture by predicting the value of each predefined interaction posture, drives the robotic arm to the corresponding position, completes the interaction and obtains a new tactile image.

[0061] The input of the strategy estimation module consists of two parts. The first part is the low-resolution 3D feature map z′ generated by the trained 3D denoising network. t The second part is the set of predefined interactive postures. Assuming that the robot arm may perform predefined postures containing k, the set can be expressed as A = {a1, a2, ..., a k}.

[0062] The strategy estimation module is a neural network composed mainly of multi-layer perceptrons. It connects the input features and the predefined interaction posture a and inputs them into the multi-layer perceptron to predict the value of the predefined posture. The value corresponding to the interaction posture a is formula (6):

[0063] value a =Q(z′ t ,a) (6)

[0064] Q is the Q network;

[0065] The strategy estimation module is trained using reinforcement learning. The whole process is iterative. In one round, assuming the current number of steps is s, all the tactile input features currently acquired are {c1, c2, ..., c s}. Calculate the reward according to formula (7):

[0066]

[0067] After accumulating the rewards at each time step, the network is trained through Q learning. Specifically, the loss function is calculated according to formula (8), and then the loss function is optimized to obtain a network that can reasonably predict the value of the interaction pose.

[0068] L rl =[R+γmaxQ(z′ t ,a s+1 )-Q(z′ t ,a s )] 2 (8)

[0069] Where γ is the discount factor.

[0070] After training the strategy estimation module, the shape feature z′ generated by the 3D reconstruction module is input during the prediction process. t , calculate the value of each predefined interaction posture through the Q network according to formula (6), select the interaction posture with the highest value, drive the robot arm to the corresponding position, touch the target through the end tactile sensor and obtain the tactile image, and add the new tactile image to the tactile condition feature set C. Use the 3D reconstruction module to generate a new z′ based on the updated feature set t, and iteratively repeat the previous interactive posture judgment and tactile perception until the termination condition (such as a fixed number of steps) is completed and the iteration stops.

[0071] The present invention will be further described in detail below with reference to specific embodiments, but the present invention is not limited to the specific embodiments.

[0072] An active 3D reconstruction method based on visual and tactile fusion includes database construction, network model building, network training and autonomous reconstruction.

[0073] 1. Database construction

[0074] Build a large database containing 3D mesh models of different shapes. For each 3D mesh model, perform the following operations:

[0075] ① Normalize the nodes of the grid and then calculate the signed distance field (SDF) in a three-dimensional volume to obtain an N×N×N SDF volume.

[0076] ② Use blender or pyrender to render a visual image of the 3D mesh model from a certain perspective.

[0077] ③ Establish a simulation environment for the robotic arm to simulate the operation of the robotic arm and the collision between the end tactile sensor and the target.

[0078] ④ Load the 3D mesh model into the simulation environment, and evenly select a fixed number (e.g. 50) of positions and directions on the sphere centered on the target as predefined interaction poses.

[0079] ⑤ Drive the simulated robotic arm to the predefined positions in sequence, and then drive the simulated robotic arm to move along the direction of the interactive posture until it collides with the target, rendering the tactile image generated by the collision of the end tactile sensor.

[0080] 2. Network model construction

[0081] The present invention includes two modules, a 3D reconstruction module and a strategy estimation module. The 3D reconstruction module contains two network models. The first is a 3D autoencoder network, which includes a 3D encoding network and a 3D decoding network. The second neural network is a 3D denoising network, which together with the 3D decoding network in the 3D autoencoder network forms a 3D diffusion model. Figure 2 Let’s build two network models.

[0082] The strategy estimation module contains a network model mainly composed of multi-layer perceptrons. Figure 3 To build a network model.

[0083] 3. Network training

[0084] Divide the constructed dataset into training sets, transfer them to the network model to be trained in batches through multiple threads, and complete the network training by following the steps below:

[0085] ① Inputting the SDF Volume (X in formula (1)) into the 3D autoencoder network can obtain the output X′. According to formula (4), the error and gradient are calculated. According to the back propagation of the neural network, the network parameters are gradually updated along the direction of gradient descent. Iterate until convergence to complete the training of the 3D encoding network and 3D decoding network in the 3D autoencoder network.

[0086] ② Use the 3D encoding network in the trained 3D autoencoder network to encode the SDF volume into a compressed feature z, input the visual image and a random number of tactile images through the visual-tactile conditional encoding module to obtain the conditional feature F, and then iteratively add noise to the feature z to obtain z t , predict the added noise according to formula (3), calculate the error according to formula (5), and calculate the gradient. According to the back propagation of the neural network, gradually update the network parameters along the direction of gradient descent, iterate until convergence, and complete the training of the three-dimensional denoising network in the three-dimensional diffusion model.

[0087] 4. Autonomous reconstruction process

[0088] For a target to be reconstructed, the autonomous reconstruction process is implemented according to the following iterative steps (such as Figure 1 shown):

[0089] ① Use the camera on the robotic arm to capture a single-view image of the target, and obtain the input feature F = V = {v1, v2, ..., v m}, input into the 3D denoising network in the trained 3D reconstruction module, and the 3D feature map is initialized to z using random noise t , use the trained three-dimensional denoising network to iteratively predict the noise ∈′ through formula (3) t , the three-dimensional feature map z after adding noise t Remove noise ∈′ t Then generate a low-resolution three-dimensional feature map z′ t .

[0090] ② Input the three-dimensional shape features and pre-set interaction poses into the trained network of the strategy estimation module. The value of each interaction pose is calculated according to Equation (6). The interaction pose with the highest value is then selected, and the robot arm is driven to move according to the interaction pose until it collides with the target. If the environment is simulated, the tactile image generated by the end-tactile sensor collision is rendered; if the environment is real, the tactile data sensed by the end-tactile sensor is read and the tactile image is generated.

[0091] ③ The tactile image is combined with the visual image in step ① and input into the module input of the 3D reconstruction module to obtain the conditional feature F=V∪C={v1,v2,...,v m ,c1,c2,...,c n}, the three-dimensional feature map is initialized to z using random noise t , use the trained three-dimensional denoising network to iteratively predict the noise ∈′ through formula (3) t , the three-dimensional feature map z after adding noise t Remove noise ∈′ t Then generate a low-resolution three-dimensional feature map z′ t Then continue with step ② and iterate until the termination condition is reached (such as completing a fixed number of steps), and then go to step ④.

[0092] ④ Input the generated 3D shape feature z′ into the 3D decoding network of the 3D diffusion model, and then generate the 3D volume X′ according to formula (2). Use the voxel-level reconstruction algorithm to extract the isosurface from the 3D volume, generate the 3D mesh, and complete the 3D reconstruction.

[0093] Compared with similar methods previously proposed internationally by Facebook AI, Berkeley University, etc., the method of the present invention has an average improvement of 80.6% in the accuracy of generated results under the same number of grasps; in terms of the efficiency of tactile exploration, the same reconstruction gain can be achieved using only 27.8% of the number of grasps, enhancing the efficiency of tactile perception and the quality of three-dimensional generation.

Claims

1. An active 3D reconstruction method based on visual and tactile fusion, characterized in that: A single-perspective visual image is acquired, and the three-dimensional global information of the target is predicted through the visual-tactile conditional encoding module and the three-dimensional reconstruction module; based on the three-dimensional global information, the optimal interaction posture of the tactile sensor at the end of the next robotic arm is determined through the strategy estimation module, and the robotic arm is driven to contact the target according to the optimal interaction posture to obtain a tactile image; the visual-tactile conditional encoding module is used to encode and fuse the visual image and the tactile image, and a low-resolution three-dimensional feature map is obtained through the three-dimensional reconstruction module, and the strategy estimation module is continued to be used to determine the next optimal interaction posture, and it is iterated until the termination condition is met; finally, the low-resolution three-dimensional feature map is reconstructed into a three-dimensional volume through a three-dimensional decoding network, and then a three-dimensional shape is generated by a voxel-level reconstruction method.

2. The active 3D reconstruction method of visual-tactile fusion according to claim 1, characterized in that: The 3D reconstruction module includes a 3D autoencoder network and a 3D denoising network; the 3D autoencoder network and the 3D diffusion model are trained separately by a stochastic gradient descent method; During the training process, the training data set contains a 3D model and the corresponding visual and tactile images. The 3D model is processed into a 3D volume represented by a signed distance field (SDF). First, the 3D autoencoder network is trained to calculate the error L between the predicted 3D volume and the true value of the 3D volume in the training dataset. ed , and calculate the gradient of the error, and gradually update the parameters of the three-dimensional autoencoder network along the direction of gradient descent according to the back propagation of the three-dimensional autoencoder network, and iterate until convergence; L ed =|X′-X| (1) After training, the 3D encoding network in the 3D autoencoder network extracts the feature maps of all 3D bodies in the training data set, iteratively adds noise to the feature maps, and uses the 3D denoising network to predict the noise. The error loss L between the predicted noise and the actual added noise is calculated according to formula (2): diff , reversely update the network parameters of the 3D denoising network according to the error loss; L diff =||N θ (z t ,t,F)-∈ t ||2 (2) ∈ t The noise added for iteration t.

3. The active 3D reconstruction method of visual-tactile fusion according to claim 2, characterized in that: The three-dimensional self-encoding network includes a three-dimensional encoding network and a three-dimensional decoding network; The three-dimensional coding network is a three-dimensional convolutional neural network, the input is a three-dimensional volume X represented by a signed distance field SDF, and the encoding is a low-resolution, compressed three-dimensional feature map; the three-dimensional coding network is defined as E s , then the low-resolution, compressed three-dimensional feature map z is: z=E s (X) (3) The three-dimensional decoding network is a three-dimensional deconvolution neural network; a VQ quantization operation is performed on the low-resolution, compressed three-dimensional feature map z, and then decoded into a high-resolution three-dimensional volume through the three-dimensional deconvolution neural network; the three-dimensional decoding network is defined as D s , then the decoded three-dimensional body X′ is: X′=D s (VQ(z)) (4) The three-dimensional denoising network inputs the conditional code F output by the visual-tactile conditional coding module, and adds noise to the low-resolution, compressed three-dimensional feature map z through the diffusion model for t iterations to obtain the three-dimensional feature map z after adding noise. t , and then use the three-dimensional denoising network N θ According to formula (5), the added noise ∈′ is predicted t , the three-dimensional feature map z after adding noise t Remove noise ∈′ t Then generate a low-resolution three-dimensional feature map z′ t ; ∈′ t =N θ (z t ,t,F) (5)。 4. The active 3D reconstruction method of visual-tactile fusion according to claim 1, characterized in that: The input of the visual-tactile conditional encoding module has two modes, the first is a single visual image input, and the other is a visual-tactile fusion input; in the first input mode, the input visual image is extracted through the image feature extraction network to extract m labeled visual features V = {v1, v2, ..., v m In the second input mode, in addition to extracting visual features from a single image, the depth information corresponding to the perceived tactile image is predicted through a depth estimation network. This is then projected into a 3D point cloud based on the interaction pose. This 3D point cloud is then fed into a point cloud feature extraction network to obtain the point cloud features corresponding to each tactile image. Each point cloud feature is considered a labeled tactile feature. Assume there are n tactile images, and a total of n labeled tactile feature sets C = {c1, c2, ..., c n }; In the single visual image input mode, since there are only visual features, the final conditional encoding F=V={v1,v2,...,v m }; In the visual-tactile fusion mode, the conditional encoding F=V∪C={v1,v2,...,v m ,c1,c2,...,c n }.

5. The active 3D reconstruction method of visual-tactile fusion according to claim 1, characterized in that: During the prediction process, the 3D feature map is initialized with random noise, and the conditional code F is used to iteratively predict the noise ∈′ through the trained 3D denoising network. t , remove noise ∈′ from the 3D feature map t And generate shape feature z′ t Finally, the output 3D volume X′ is obtained through the trained 3D decoding network, and then the 3D shape is generated by the voxel-level reconstruction algorithm.

6. The active 3D reconstruction method of visual-tactile fusion according to claim 1, characterized in that: The strategy estimation module determines the optimal interaction posture by predicting the value of each predefined interaction posture, drives the robotic arm to reach the corresponding optimal interaction posture, completes the interaction and obtains a new tactile image.

7. The active 3D reconstruction method of visual-tactile fusion according to claim 6, characterized in that: The input of the strategy estimation module is two parts. The first part is the low-resolution 3D feature map generated by the trained 3D denoising network. The second part is the set of predefined interaction poses. Assuming that the predefined poses performed by the robot arm contain k, the set of predefined interaction poses is represented as A = {a1, a2, ..., a k }; The strategy estimation module is a neural network mainly composed of a multi-layer perceptron. It connects the input features and the predefined interaction posture a, inputs them into the multi-layer perceptron, and predicts the value of the predefined posture. The value corresponding to the interaction posture a is formula (6): Q is the Q network; The strategy estimation module is trained using a reinforcement learning method. The entire process is iterative. In one round, assuming the current number of steps is s, all the tactile input features currently acquired are {c1, c2, ..., c s }; Calculate the reward according to formula (7): After accumulating the rewards at each time step, the network is trained through Q learning. The loss function is calculated according to formula (8), and then the loss function is optimized to obtain a network that can predict the value of the interaction pose. Where γ is the discount factor.

8. The active 3D reconstruction method of visual-tactile fusion according to claim 7, characterized in that: After training the strategy estimation module, the shape feature z′ generated by the 3D reconstruction module is input into the prediction process. The value of each predefined interaction posture is calculated by the Q network according to formula (6). The interaction posture with the highest value is selected. The robot arm contacts the target according to the interaction posture and obtains a tactile image. The obtained new tactile image is added to the tactile condition feature set C. The 3D reconstruction module is used to generate a new shape feature z′ based on the updated feature set. The previous interaction posture judgment and tactile perception are iteratively repeated until the termination condition is met and the iteration stops.

Images