Three-dimensional surface reconstruction method and device based on visual tactile image
By using multi-frequency Fourier feature mapping and lightweight deep separable networks, the contradiction between high-frequency detail reconstruction and model lightweighting in visual-tactile surface reconstruction methods is resolved, achieving high-precision, real-time 3D surface reconstruction.
Patent Information
- Application Number
- CN202610055031.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-15
- Publication Date
- 2026-05-19
AI Technical Summary
Existing visual-tactile surface reconstruction methods suffer from insufficient high-frequency detail reconstruction capabilities, large model capacity, low computational efficiency, and limited data utilization, making it difficult to simultaneously meet the requirements of high-fidelity morphology reconstruction and real-time deployment.
Images are acquired using a visual tactile sensor. Multi-frequency Fourier feature mapping and a lightweight deep separable network structure are used to generate a surface gradient field for 3D surface reconstruction, including position feature enhancement, lightweight network prediction, and 3D tactile surface reconstruction.
It achieves high-precision, real-time 3D surface reconstruction on resource-constrained platforms, improves the ability to perceive high-frequency details and the nonlinear mapping capability of the model, and reduces the number of parameters and computational load.
Smart Images
Figure CN122066853A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of image processing technology, and more specifically, relates to a method and apparatus for three-dimensional surface reconstruction based on visual-tactile images. Background Technology
[0002] With the development of robotic tactile perception and intelligent operation technologies, visual-tactile sensors, due to their high spatial resolution and visualization features, are widely used in surface topography reconstruction, contact force estimation, and object recognition. Visual-tactile sensors typically consist of a flexible, transparent elastomer (such as silicone) and a camera on the bottom. By recording images of surface deformation caused by contact with an object, they indirectly reflect the three-dimensional geometric information of the contact area. In existing technologies, visual-tactile surface reconstruction methods typically employ traditional multilayer perceptron (MLP) networks or convolutional neural networks (CNNs) to perform depth reconstruction of visual-tactile images. Specifically, the color channels (B, G, R) and their two-dimensional coordinates (x, y) of each pixel are extracted from the visual-tactile image, forming a five-dimensional input feature vector (B, G, R, x, y). These features are input into a multilayer perceptron network, and the network outputs the two-dimensional gradient component (g) corresponding to each pixel. x ,g The two-dimensional gradient component is used to represent the local tilt angle of the surface at the pixel. Then, the depth of the contact surface is reconstructed based on the two-dimensional gradient component, thus realizing the mapping from the visual-tactile image to the three-dimensional surface morphology.
[0003] Existing visual-tactile surface reconstruction methods have advantages such as simple structure and convenient implementation, but they still have the following main problems: 1. Insufficient high-frequency detail reconstruction capability. For example, traditional MLP inputs are only linearly normalized coordinates (x, y), which are difficult to effectively represent complex high-frequency spatial changes, resulting in blurring or smoothing of surface morphology at edges and fine structures. 2. Large model capacity and low computational efficiency. For example, to compensate for the network's insufficient representation, the number of hidden layers or the size of neurons are usually increased, leading to an increase in the number of model parameters, which is not conducive to deployment on resource-constrained embedded or edge computing platforms. 3. Limited data utilization. For example, the nonlinear relationship between the original BGR features and coordinates is not fully explored, and the network needs to learn these implicit patterns through a large number of samples, resulting in high training costs and limited generalization. 4. Limited data utilization. For example, the nonlinear relationship between the original BGR features and coordinates is not fully explored, and the network needs to learn these implicit patterns through a large number of samples, resulting in high training costs and limited generalization.
[0004] Therefore, there is a clear contradiction between reconstruction accuracy and model lightweighting in existing technologies, making it difficult to simultaneously meet the requirements of high-fidelity topography reconstruction and real-time deployment. Summary of the Invention
[0005] The main objective of this application is to provide a three-dimensional surface reconstruction method based on visual-tactile images, which can not only solve the obvious contradiction between the accuracy of three-dimensional surface topography reconstruction and the lightweight model in the existing technology, but also take into account the needs of high-fidelity topography reconstruction and real-time deployment, thereby improving the accuracy of three-dimensional surface reconstruction.
[0006] To achieve the above objectives, according to a first aspect of this application, a three-dimensional surface reconstruction method based on visual-tactile images is proposed. The method includes: acquiring a visual-tactile image of a target object using a visual-tactile sensor; for any target pixel in the visual-tactile image: acquiring the normalized coordinates and RGB information of the target pixel; performing position feature enhancement processing on the normalized coordinates of the target pixel to generate a multi-frequency position vector; concatenating the multi-frequency position vector with the RGB information to generate an enhanced feature; predicting the gradient field using a lightweight network structure based on the enhanced feature to generate the surface gradient corresponding to the target pixel; and performing three-dimensional tactile surface reconstruction based on the surface gradient corresponding to each target pixel in the visual-tactile image to obtain a three-dimensional surface topography map.
[0007] Optionally, for any target pixel in the visual-tactile image to be tested: obtain the normalized coordinates and RGB information of the target pixel; perform multi-frequency Fourier feature mapping on the normalized coordinates of the target pixel to generate a Fourier position vector; concatenate the Fourier position vector with the RGB information to generate an enhanced feature; based on the enhanced feature, use a lightweight depth separable model to predict the gradient field and generate the surface gradient corresponding to the target pixel.
[0008] Optionally, the method further includes: during the calibration process of the visual-tactile sensor, acquiring visual-tactile image samples of a standard ball indenter; for any target pixel in the visual-tactile image samples: determining the enhancement feature corresponding to the target pixel; using a depth-separable linear layer to perform linear transformation processing on each dimension feature in the enhancement feature to generate a transformed feature; using a point-by-point linear layer to fuse the transformed features corresponding to each dimension feature to generate a fused feature; The fused features are corrected and filtered based on the BatchNorm+ReLU structure; the filtered features are predicted based on the linear regression layer, and the predicted surface gradient is output; the model is trained in a supervised manner based on the predicted surface gradient and surface gradient label corresponding to each target pixel, and a lightweight deep separable model is generated.
[0009] Optionally, the step of performing multi-frequency Fourier feature mapping on the normalized coordinates of the target pixel to generate a Fourier position vector includes: generating a frequency sequence based on a preset bandwidth and according to preset rules. For any target frequency in the frequency sequence: based on the target frequency, the normalized coordinates of the target pixel are encoded to generate a position vector corresponding to the target frequency; based on the position vector corresponding to each target frequency, a Fourier position vector is obtained.
[0010] Optionally, the surface gradient label is obtained by the following method: obtaining the pixel physical ratio parameters of the visual tactile sensor; based on the pixel physical ratio parameters, converting the radius of the standard ball indenter used in the calibration process to the image coordinate scale; based on the image coordinate scale, constructing a pixel coordinate grid covering the tactile sensing area; for any target pixel in the visual tactile image sample of the target object: based on the pixel coordinate grid, calculating the distance between the target pixel and the center of the visual tactile image sample; and calculating the true surface gradient of the target pixel using the spherical equation; determining the true surface gradient as the surface gradient label; wherein, the center of the visual tactile image sample is used to indicate the center of the circular contact pattern generated by the ball indenter pressing on the surface of the visual tactile sensor.
[0011] Optionally, based on the surface gradient corresponding to each target pixel in the visual-tactile image to be tested, a three-dimensional tactile surface is reconstructed to obtain a three-dimensional surface topography map; this includes: calculating the discreteness based on the surface gradient corresponding to each target pixel in the visual-tactile image to be tested using a fast Poisson solver; performing a two-dimensional discrete cosine transform on the discreteness to obtain a three-dimensional depth image of the tactile surface; and reconstructing and filtering the three-dimensional depth image of the tactile surface based on a color threshold, a gradient threshold, and a depth threshold to generate a three-dimensional surface topography map.
[0012] Optionally, the step of visually reconstructing the three-dimensional depth image of the tactile surface based on a color threshold, a gradient threshold, and a depth threshold to generate a three-dimensional surface topography map includes: obtaining a color threshold, a gradient threshold, and a depth threshold based on the original diaphragm image of the visual-tactile sensor; performing masking processing on the visual-tactile image to be tested based on the color threshold and the gradient threshold to generate a mask image; and performing visual reconstruction based on the mask image and the three-dimensional depth image of the tactile surface to generate a three-dimensional surface topography map.
[0013] To achieve the above objectives, according to a second aspect of this application, a three-dimensional surface reconstruction device based on visual-tactile images is also provided. The device further includes: a first acquisition module, configured to acquire a visual-tactile image of a target object using a visual-tactile sensor; a mapping module, configured to, for any target pixel in the visual-tactile image under test: acquire the normalized coordinates and RGB information of the target pixel; perform position feature enhancement processing on the normalized coordinates of the target pixel to generate a multi-frequency position vector; concatenate the multi-frequency position vector with the RGB information to generate an enhanced feature; and, based on the enhanced feature, predict the gradient field using a lightweight network structure to generate the surface gradient corresponding to the target pixel; and a surface reconstruction module, configured to perform three-dimensional tactile surface reconstruction based on the surface gradient corresponding to each target pixel in the visual-tactile image under test to obtain a three-dimensional surface topography map.
[0014] To achieve the above objectives, according to a third aspect of this application, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the positioning method as described in the first aspect.
[0015] To achieve the above objectives, according to a fourth aspect of this application, a robot is also provided, which constructs a three-dimensional surface topography map based on the acquired visual-tactile images according to the method described in the first aspect.
[0016] The technical solutions provided by the embodiments of this application may include the following beneficial effects: This application provides a method for three-dimensional surface reconstruction based on visual-tactile images. The method includes: first, acquiring a visual-tactile image of a target object using a visual-tactile sensor; second, for any target pixel in the visual-tactile image: acquiring the normalized coordinates and RGB information of the target pixel; performing position feature enhancement processing on the normalized coordinates of the target pixel to generate a multi-frequency position vector; concatenating the multi-frequency position vector with the RGB information to generate an enhanced feature; based on the enhanced feature, predicting the gradient field using a lightweight network structure to generate the surface gradient corresponding to the target pixel; and finally, performing three-dimensional tactile surface reconstruction based on the surface gradient corresponding to each target pixel in the visual-tactile image to obtain a three-dimensional surface topography map. This embodiment enhances the position feature of the target pixel coordinates through position encoding, thereby expanding a smooth low-frequency signal into a high-dimensional feature vector containing rich spatial high-frequency components, effectively solving the problem of weak fitting ability of traditional MLPs for high-frequency details. By using a lightweight network structure to map enhanced features to a gradient field and then reconstructing a 3D tactile surface based on the gradient field, the model can still have a strong expressive ability for complex nonlinear mappings while significantly reducing the number of parameters and computational cost. This enables real-time and high-precision surface reconstruction on embedded platforms with limited parameters and computational cost. Attached Figure Description
[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of this application are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 A flowchart illustrating a three-dimensional surface reconstruction method based on visual-tactile images provided in this application; Figure 2 This is a schematic diagram of the structure of a three-dimensional surface reconstruction device based on visual-tactile images provided in this application. Detailed Implementation
[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.
[0019] like Figure 1 The diagram shown is a flowchart of a three-dimensional surface reconstruction method based on visual-tactile images provided in this application.
[0020] A method for reconstructing a three-dimensional surface based on visual-tactile images includes at least the following steps: S101, acquires the visual-tactile image of the target object through a visual-tactile sensor; S102, for any target pixel in the visual-tactile image to be tested: obtain the normalized coordinates and RGB information of the target pixel; perform position feature enhancement processing on the normalized coordinates of the target pixel to generate a multi-frequency position vector; concatenate the multi-frequency position vector with the RGB information to generate an enhanced feature; based on the enhanced feature, use a lightweight network structure to predict the gradient field and generate the surface gradient corresponding to the target pixel. S103, based on the surface gradient corresponding to each target pixel in the visual-tactile image to be tested, perform three-dimensional tactile surface reconstruction to obtain a three-dimensional surface topography map.
[0021] In S101, the target object is used to apply indentations of different depths and positions on the surface of the visual tactile sensor. During this process, the corresponding visual tactile images to be tested are acquired, resulting in several visual tactile images to be tested.
[0022] In S102, the method of location feature enhancement is not limited, as long as it can convert the coordinates of the target pixel into more fully expressive spatial features. The location feature enhancement method can be multi-frequency Fourier feature mapping, or it can be random feature encoding, learnable frequency encoding, kernel function encoding, multi-scale spatial encoding, or local region spatial encoding, etc. Therefore, through location feature enhancement, the coordinates of the target pixel can be converted into more fully expressive spatial features, thereby helping to improve the model's ability to perceive high-frequency details on the surface of the target object.
[0023] For example, multi-frequency Fourier feature mapping is performed on the normalized coordinates of the target pixel to generate a Fourier position vector. More specifically, based on a preset bandwidth, a frequency sequence is generated according to preset rules; for any target frequency in the frequency sequence: based on the target frequency, the normalized coordinates of the target pixel are encoded to generate a position vector corresponding to the target frequency; based on the position vector corresponding to each target frequency, a Fourier position vector is obtained.
[0024] For example, perform multi-frequency sinusoidal encoding on the normalized coordinates (x, y) of the target pixel.
[0025] Based on a preset bandwidth N, a frequency sequence is constructed according to preset rules. ; For any target frequency in the frequency sequence, the encoding process is performed according to the following formula: Equation (1); The final generated Fourier position vector is: .
[0026] The Fourier position vector is concatenated with the RGB information to generate enhanced features, as shown in Equation (2).
[0027] Equation (2).
[0028] Here, no limitations are placed on the lightweight network structure, as long as the network can complete the mapping from enhanced features to surface gradients based on limited computational resources. The lightweight network structure in this embodiment can be a lightweight depthwise separable model; it can also be other network forms with low parameter count and low computational cost but still possessing nonlinear modeling capabilities, such as: improved lightweight convolutional networks, lightweight Transformer structures, hybrid MLP structures, selective channel modeling structures, or small networks obtained based on model compression, pruning, or quantization, etc.
[0029] For example, the enhanced features are input into a lightweight network structure, which outputs a predicted quantity. This predicted quantity is then converted into a gradient field using mathematical relationships to obtain the surface gradient corresponding to the target pixel. Predicted quantities include, for example, normal vectors, predicted depth values, or predicted curvature.
[0030] Alternatively, the enhanced features can be input into a lightweight network structure, which directly outputs the gradient field to obtain the surface gradient corresponding to the target pixel.
[0031] This embodiment uses positional encoding to enhance the positional features of the target pixel coordinates. This expands the smooth low-frequency signal into a high-dimensional feature vector containing rich spatial high-frequency components, which helps improve the model's ability to learn high-frequency details and solves the problem of traditional MLP's weak fitting ability to high-frequency details.
[0032] In S103, in terms of 3D tactile surface reconstruction, in addition to using the Poisson integral solution method, other numerical solution methods, variational approximation methods, iterative PDE methods, or direct end-to-end depth regression methods can also be used, as long as they can restore the surface depth of the target object based on gradient-related information.
[0033] This embodiment maps enhanced features to a gradient field using a lightweight network structure and reconstructs three-dimensional tactile surfaces based on the gradient field. This enables the network to retain strong expressive power for complex nonlinear mappings while significantly reducing the number of parameters and computational cost, thereby achieving real-time and high-precision three-dimensional surface reconstruction.
[0034] In a preferred embodiment of this example, for any target pixel in the visual-tactile image to be tested: the normalized coordinates and RGB information of the target pixel are obtained; multi-frequency Fourier feature mapping is performed on the normalized coordinates of the target pixel to generate a Fourier position vector; the Fourier position vector is concatenated with the RGB information to generate an enhanced feature; based on the enhanced feature, a lightweight depth separable model is used to predict the gradient field and generate the surface gradient corresponding to the target pixel.
[0035] Specifically, the original two-dimensional coordinates are mapped to a higher dimension using a multi-frequency sine function or a multi-frequency cosine function in a multi-frequency Fourier transform. Then, the Fourier position vector generated by the high-dimensional mapping is fused with RGB information, and the fused result is input into a lightweight, deeply separable model. This allows the lightweight, deeply separable model to possess multi-scale spatial frequency information at the input layer, significantly improving the model's ability to perceive high-frequency structures such as protrusions, fine lines, and hard boundaries. This embodiment uses a lightweight, deeply separable model instead of a traditional fully connected network, which not only improves the accuracy of surface gradient prediction but also significantly reduces the number of model parameters and computational cost, making it suitable for embedded devices and real-time tactile sensing systems.
[0036] This invention combines multi-frequency Fourier feature mapping with a lightweight deep separable network, achieving a synergistic effect of high-frequency information enhancement at the input end and efficient network representation capabilities. This significantly improves the regression accuracy and detail recovery ability of surface gradients compared to traditional methods. Simultaneously, this invention achieves good results in terms of model size, computational speed, and ease of deployment, enabling stable and reliable visual-tactile surface reconstruction without relying on high-performance hardware. Therefore, this invention represents a significant technological advancement and substantial improvement in overall performance, possessing outstanding practical value.
[0037] In a preferred embodiment of this example, the method further includes: during the calibration process of the visual-tactile sensor, acquiring visual-tactile image samples of a standard ball indenter; for any target pixel in the visual-tactile image samples: determining the enhancement feature corresponding to the target pixel; using a depth-separable linear layer to perform linear transformation processing on each dimension feature of the enhancement feature to generate transformed features; using a point-by-point linear layer to fuse the transformed features corresponding to each dimension feature to generate fused features; correcting and filtering the fused features based on a BatchNorm+ReLU structure; predicting the filtered features based on a linear regression layer to output predicted surface gradients; and performing supervised model training based on the predicted surface gradient and surface gradient label corresponding to each target pixel to generate a lightweight depth-separable model.
[0038] For example, in the calibration process of a visual-tactile sensor, a standard ball indenter of known diameter is used to apply indentations of different depths and positions on the surface of the sensor, and corresponding tactile image samples are collected to obtain several visual-tactile image samples. For any target pixel in the visual-tactile image sample: the normalized coordinates and RGB information of the target pixel are obtained; the normalized coordinates of the target pixel are subjected to position feature enhancement processing to generate a multi-frequency position vector; the multi-frequency position vector is concatenated with the RGB information to generate the enhanced feature F.
[0039] By using a depthwise separable linear layer, a linear transformation is performed on each dimension of the enhanced features to generate transformed features. Specifically, as shown in equation (3).
[0040] Equation (3); Where Fi represents the dimensional feature; This represents the learnable linear weight parameter corresponding to the i-th dimension feature of the enhanced feature F, which is used to perform weighted mapping on the feature of that dimension.
[0041] By using a pointwise linear layer, the transformation features of different channels are fused to generate a fused feature. Specifically, as shown in equation (4).
[0042] Equation (4); in, This represents the learnable weight matrix in the pointwise linear layer, used to weight and combine the transformation features of each channel to achieve information fusion between different transformation features; The bias vector parameter corresponding to the pointwise linear layer is used to perform translation correction on the linear transformation result to enhance the expressive power and fitting flexibility of the model.
[0043] Residual and normalization structure: Based on the BatchNorm+ReLU structure, the fused features are corrected and filtered to obtain the filtered features h; as shown in Equation (5).
[0044] Equation (5); Therefore, the BatchNorm+ReLU structure can ensure the stability of model training, thereby accelerating model convergence.
[0045] Based on the linear regression layer, the selected features are predicted, and the predicted surface gradient is output. ; as shown in equation (6) for details.
[0046] Equation (6) in, represents the output weight matrix in the linear regression layer, used to map the filtered and fused features to the surface gradient prediction space, and b represents the bias vector parameter corresponding to the linear regression layer, used to perform numerical translation correction on the linear mapping result.
[0047] Using the surface gradient label of the target pixel as a reference, the predicted surface gradient and the surface gradient label are compared during model training to generate a loss function. When the loss function is minimized, a lightweight deep separable model is obtained. Thus, the color and Fourier position vector of the target pixel can be learned based on a lightweight deep separable network, thereby achieving a non-linear mapping of the surface gradient of the target pixel.
[0048] As further exemplarily, the surface gradient label is obtained through the following method: acquiring the pixel physical scale parameters of the visual-tactile sensor; based on the pixel physical scale parameters, converting the radius of the standard ball indenter used in the calibration process to the image coordinate scale; constructing a pixel coordinate grid covering the tactile sensing area based on the image coordinate scale; for any target pixel in the visual-tactile image sample of the target object: calculating the distance between the target pixel and the center of the visual-tactile image sample based on the pixel coordinate grid; calculating the true surface gradient of the target pixel using the spherical equation; and determining the true surface gradient as the surface gradient label; wherein, the center of the visual-tactile image sample is used to indicate the center of the circular contact pattern generated by the ball indenter pressing on the surface of the visual-tactile sensor. Thus, physically accurate calibration data can be constructed based on the standard ball indenter, thereby improving the accuracy of training lightweight deep splittable models.
[0049] This invention utilizes a lightweight, depth-separable network structure to decompose the feature extraction process into independent linear transformations along the depth direction and point-by-point linear fusion, significantly reducing the number of model parameters and computational complexity while maintaining sufficient nonlinear modeling capabilities. This structure not only makes the model training process more stable but also maintains high accuracy while possessing high inference efficiency, making it suitable for tactile sensing systems with high real-time requirements.
[0050] In a preferred embodiment of this invention, a three-dimensional tactile surface is reconstructed based on the surface gradient corresponding to each target pixel in the visual-tactile image to be tested, resulting in a three-dimensional surface topography map. This includes: calculating the discreteness based on the surface gradient corresponding to each target pixel in the visual-tactile image to be tested using a fast Poisson solver; performing a two-dimensional discrete cosine transform on the discreteness to obtain a three-dimensional depth image of the tactile surface; and reconstructing and filtering the three-dimensional depth image of the tactile surface based on a color threshold, a gradient threshold, and a depth threshold to generate a three-dimensional surface topography map.
[0051] For example: the surface gradient corresponding to each target pixel in the visual-tactile image to be tested, using fast Poisson's algorithm... The slack solver calculates the discreteness S: Equation (7); The depth value is obtained by performing a two-dimensional discrete cosine transform (DCT) on the dispersion S. : Equation (8); Construct the coefficient matrix of the Poisson equation: Equation (9); D ij This represents the eigenvalue of the two-dimensional discrete Laplace operator at the frequency index (i,j) in the frequency domain (DCT domain), used for normalizing and solving the frequency domain components during the solution of the Poisson equation.
[0052] W represents the pixel resolution or width of the reconstructed tactile surface 3D depth image in the horizontal direction, i.e., the size of the image in the x-axis direction.
[0053] H represents the pixel resolution or height of the reconstructed tactile surface 3D depth image in the vertical direction, i.e., the size of the image in the y-axis direction.
[0054] Finally, the three-dimensional depth image H of the tactile surface is obtained through inverse DCT: Equation (10).
[0055] For example, the step of visually reconstructing the three-dimensional depth image of the tactile surface based on a color threshold, a gradient threshold, and a depth threshold to generate a three-dimensional surface topography map includes: obtaining a color threshold, a gradient threshold, and a depth threshold based on the original diaphragm image of the visual-tactile sensor; performing masking processing on the visual-tactile image to be tested based on the color threshold and the gradient threshold to generate a mask image; and performing visual reconstruction based on the mask image and the three-dimensional depth image of the tactile surface to generate a three-dimensional surface topography map.
[0056] It should be noted that the data preprocessing, mask image generation, and feature combination methods of the present invention can also adopt other strategies with the same recognition and constraint effects, such as learning-based contact area detection, multimodal feature-based determination, or time-series change-based determination methods, all of which are reasonable variations of the present invention.
[0057] This embodiment uses the surface gradient corresponding to each target pixel in the visual-tactile image to obtain a continuous three-dimensional depth image of the tactile surface by solving the Poisson integral. It then combines color thresholding and depth thresholding methods to generate a contact mask, thereby achieving the retention of reconstruction results only in the effective contact area and improving the reconstruction accuracy of the three-dimensional surface topography.
[0058] The following section provides a detailed description of a three-dimensional surface reconstruction method based on visual-tactile images provided in this embodiment, using specific scenarios as examples.
[0059] A method for three-dimensional surface reconstruction based on visual-tactile images includes the following steps: S1, obtain the pixel physical ratio parameters of the visual tactile sensor; based on the pixel physical ratio parameters, convert the radius of the standard ball indenter used in the calibration process to the image coordinate scale; based on the image coordinate scale, construct a pixel coordinate grid covering the tactile sensing area; for any target pixel in the visual tactile image sample of the target object: based on the pixel coordinate grid, calculate the distance from the target pixel to the center of the visual tactile image sample; and calculate the true surface gradient of the target pixel using the spherical equation; determine the true surface gradient as the surface gradient label.
[0060] S2, during the calibration process of the visual-tactile sensor, acquire visual-tactile image samples of a standard ball indenter; for any target pixel in the visual-tactile image sample: determine the enhancement feature corresponding to the target pixel; use a depth-separable linear layer to perform linear transformation processing on each dimension feature of the enhancement feature to generate transformed features; use a point-by-point linear layer to fuse the transformed features corresponding to each dimension feature to generate fused features; use a BatchNorm+ReLU structure to correct and filter the fused features; use a linear regression layer to predict the filtered features and output the predicted surface gradient; Based on the predicted surface gradient and surface gradient label corresponding to each target pixel, supervised model training is performed to generate a lightweight, deep separable model.
[0061] S3 acquires the visual-tactile image of the target object through a visual-tactile sensor.
[0062] S4, for any target pixel in the visual-tactile image to be tested: obtain the normalized coordinates and RGB information of the target pixel; generate a frequency sequence according to a preset rule based on a preset bandwidth; for any target frequency in the frequency sequence: encode the normalized coordinates of the target pixel based on the target frequency to generate a position vector corresponding to the target frequency; obtain a Fourier position vector based on the position vector corresponding to each target frequency. Concatenate the Fourier position vector with the RGB information to generate an enhanced feature; based on the enhanced feature, predict the gradient field using a lightweight depthwise separable model to generate the surface gradient corresponding to the target pixel.
[0063] S5. Based on the surface gradient corresponding to each target pixel in the visual-tactile image to be tested, the discreteness is calculated using a fast Poisson solver; the discreteness is processed by a two-dimensional discrete cosine transform to obtain a three-dimensional depth image of the tactile surface; based on the original diaphragm image of the visual-tactile sensor, a color threshold, a gradient threshold, and a depth threshold are obtained; based on the color threshold and the gradient threshold, the visual-tactile image to be tested is masked to generate a mask image; based on the mask image and the three-dimensional depth image of the tactile surface, visualization reconstruction is performed to generate a three-dimensional surface topography map.
[0064] The solution in this embodiment can achieve the following technical effects: 1. When constructing tactile image input features, this invention introduces multi-frequency Fourier feature mapping technology to transform the original two-dimensional pixel coordinates into high-dimensional features containing multi-scale sine and cosine components. This makes the input space have richer spatial frequency information, which not only theoretically expands the function frequency band range that lightweight deep separable networks can fit, thus significantly enhancing the model's ability to express and learn high-frequency tactile features; but also overcomes the inherent defect of traditional methods' weak fitting ability for high-frequency signals, achieving accurate regression of gradients on complex tactile surfaces.
[0065] 2. This invention introduces a lightweight architecture combining depthwise separable linear layers and pointwise linear layers in the network structure, and employs residual and normalization structures. This allows the model to maintain a high-precision fitting capability for tactile image-surface gradient mapping while significantly reducing the number of parameters and computational cost. This structure has significant deployment advantages compared to traditional fully connected networks, and is particularly suitable for real-time operation on edge computing devices.
[0066] 3. This invention combines Fourier feature mapping with a lightweight network structure, allowing the input features to be fully expanded in spatial frequency, reflecting more detailed changes in visual-tactile images. Simultaneously, the network structure employs an efficient depthwise separable modeling approach, maintaining good nonlinear fitting capabilities while keeping the number of parameters and computational overhead low. This combination significantly improves the accuracy of surface gradient regression and makes the entire reconstruction method both lightweight and real-time. Therefore, this invention, through the combined design of "high-frequency feature enhancement + lightweight network structure," provides a structurally clear, performance-balanced, and easily deployable solution for visual-tactile surface reconstruction.
[0067] like Figure 2 The diagram shown is a structural schematic of a three-dimensional surface reconstruction device based on visual-tactile images provided in this application.
[0068] A three-dimensional surface reconstruction device based on visual-tactile images, the device 200 comprising: a first acquisition module Block 201 is used to acquire a test visual-tactile image of a target object using a visual-tactile sensor; Mapping module 202 is used for any target pixel in the test visual-tactile image to: acquire the normalized coordinates and RGB information of the target pixel; perform position feature enhancement processing on the normalized coordinates of the target pixel to generate a multi-frequency position vector; concatenate the multi-frequency position vector with the RGB information to generate an enhanced feature; based on the enhanced feature, use a lightweight network structure to predict the gradient field and generate the surface gradient corresponding to the target pixel; Surface reconstruction module 203 is used to perform three-dimensional tactile surface reconstruction based on the surface gradient corresponding to each target pixel in the test visual-tactile image to obtain a three-dimensional surface topography map.
[0069] In a preferred embodiment of this example, the mapping module is further configured to, for any target pixel in the visual-tactile image to be tested: acquire the normalized coordinates and RGB information of the target pixel; perform multi-frequency Fourier feature mapping on the normalized coordinates of the target pixel to generate a Fourier position vector; concatenate the Fourier position vector with the RGB information to generate an enhanced feature; and, based on the enhanced feature, predict the gradient field using a lightweight depth separable model to generate the surface gradient corresponding to the target pixel.
[0070] In a preferred embodiment of this invention, the device further includes: a second acquisition module, used to acquire visual-tactile image samples of a standard ball indenter during the calibration process of the visual-tactile sensor; a model training module, used for any target pixel in the visual-tactile image samples to: determine the enhancement feature corresponding to the target pixel; perform linear transformation processing on each dimension feature in the enhancement feature using a depth-separable linear layer to generate transformed features; perform fusion processing on the transformed features corresponding to each dimension feature using a point-by-point linear layer to generate fused features; perform correction and filtering processing on the fused features based on a BatchNorm+ReLU structure; predict the filtered features based on a linear regression layer to output predicted surface gradients; and a generation module, used to perform supervised model training based on the predicted surface gradient and surface gradient label corresponding to each target pixel to generate a lightweight depth-separable model.
[0071] In a preferred embodiment of this example, the mapping module includes: a generation unit, used to generate a frequency sequence according to a preset rule based on a preset bandwidth; an encoding processing unit, used to encode the normalized coordinates of the target pixel based on the target frequency for any target frequency in the frequency sequence, thereby generating a position vector corresponding to the target frequency; and an obtaining unit, used to obtain a Fourier position vector based on the position vector corresponding to each target frequency.
[0072] In a preferred embodiment of this invention, the surface gradient label is obtained by the following method: acquiring the pixel physical ratio parameters of the visual tactile sensor; based on the pixel physical ratio parameters, converting the radius of the standard ball indenter used in the calibration process to the image coordinate scale; based on the image coordinate scale, constructing a pixel coordinate grid covering the tactile sensing area; for any target pixel in the visual tactile image sample of the target object: calculating the distance between the target pixel and the center of the visual tactile image sample based on the pixel coordinate grid; calculating the true surface gradient of the target pixel using the spherical equation; and determining the true surface gradient as the surface gradient label.
[0073] In a preferred embodiment of this example, the surface reconstruction module includes: a calculation unit, used to calculate the discreteness based on the surface gradient corresponding to each target pixel in the tactile image to be tested, using a fast Poisson solver; a transformation processing unit, used to perform a two-dimensional discrete cosine transform on the discreteness to obtain a three-dimensional depth image of the tactile surface; and a filtering unit, used to reconstruct and filter the three-dimensional depth image of the tactile surface based on a color threshold, a gradient threshold, and a depth threshold to generate a three-dimensional surface topography map.
[0074] In a preferred embodiment of this example, the filtering unit includes: an acquisition subunit, used to acquire a color threshold, a gradient threshold, and a depth threshold based on the original diaphragm image of the visual-tactile sensor; a generation subunit, used to perform masking processing on the visual-tactile image to be tested based on the color threshold and the gradient threshold to generate a mask image; and a visualization reconstruction subunit, used to perform visualization reconstruction based on the mask image and the three-dimensional depth image of the tactile surface to generate a three-dimensional surface topography map.
[0075] The aforementioned three-dimensional surface reconstruction device can execute a three-dimensional surface reconstruction method based on visual-tactile images provided in an embodiment of the present invention, and possesses the corresponding functional modules and beneficial effects for executing a three-dimensional surface reconstruction method based on visual-tactile images. Technical details not described in detail in this embodiment can be found in the three-dimensional surface reconstruction method based on visual-tactile images provided in an embodiment of the present invention.
[0076] The present invention also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the three-dimensional surface reconstruction method based on visual-tactile images described in the present invention.
[0077] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this application described in the "Exemplary Methods" section above.
[0078] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of this application. The programming languages include object-oriented programming languages such as Java and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0079] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to the following embodiments of this application described in the "Exemplary Methods" section above.
[0080] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
[0081] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.
[0082] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.
[0083] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.
[0084] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.
[0085] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.
[0086] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.
[0087] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0088] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.
Claims
1. A method for three-dimensional surface reconstruction based on visual-tactile images, characterized in that, include: The visual-tactile image of the target object is acquired using a visual-tactile sensor. For any target pixel in the visual-tactile image to be tested: obtain the normalized value of the target pixel. Convert coordinates and RGB information; The normalized coordinates of the target pixel are subjected to position feature enhancement processing to generate a multi-frequency position vector; the multi-frequency position vector is concatenated with the RGB information to generate enhanced features; Based on the enhanced features, a lightweight network structure is used to predict the gradient field and generate the surface gradient corresponding to the target pixel. Based on the surface gradient corresponding to each target pixel in the visual-tactile image to be tested, a three-dimensional tactile... The surface is reconstructed to obtain a three-dimensional surface topography map.
2. The method according to claim 1, characterized in that, For any target pixel in the visual-tactile image to be tested: obtain the normalized coordinates and RGB information of the target pixel; perform multi-frequency Fourier feature mapping on the normalized coordinates of the target pixel to generate a Fourier position vector; concatenate the Fourier position vector with the RGB information to generate an enhanced feature; Based on the enhanced features, a lightweight depth separable model is used to predict the gradient field and generate the surface gradient corresponding to the target pixel.
3. The method according to claim 2, characterized in that, Also includes: During the calibration process of the visual-tactile sensor, visual-tactile image samples of a standard ball indenter are acquired; For any target pixel in the visual-tactile image sample: determine the enhancement feature corresponding to the target pixel; use a depth-separable linear layer to perform linear transformation processing on each dimension feature of the enhancement feature to generate a transformed feature; use a point-by-point linear layer to fuse the transformed features corresponding to each dimension feature to generate a fused feature; use a BatchNorm+ReLU structure to correct and filter the fused feature; use a linear regression layer to predict the filtered features and output the predicted surface gradient; Based on the predicted surface gradient and surface gradient label corresponding to each target pixel, supervised model training is performed to generate a lightweight, deep separable model.
4. The method according to claim 2, characterized in that, The step of performing multi-frequency Fourier feature mapping on the normalized coordinates of the target pixel to generate a Fourier position vector includes: Based on a preset bandwidth, a frequency sequence is generated according to preset rules; For any target frequency in the frequency sequence: based on the target frequency, the normalized coordinates of the target pixel are encoded to generate a position vector corresponding to the target frequency; Based on the position vector corresponding to each target frequency, a Fourier position vector is obtained.
5. The method according to claim 3, characterized in that, The surface gradient labels are obtained through the following method: Obtain the pixel physical ratio parameters of the visual tactile sensor; Based on the pixel physical ratio parameters, the radius of the standard ball indenter used in the calibration process is converted to the image coordinate scale; Based on the image coordinate scale, a pixel coordinate grid covering the tactile sensing area is constructed; For any target pixel in the visual-tactile image sample of the target object: calculate the distance between the target pixel and the center of the visual-tactile image sample based on the pixel coordinate grid; calculate the true surface gradient of the target pixel using the spherical equation; determine the true surface gradient as a surface gradient label; wherein, the center of the visual-tactile image sample is used to indicate the center of the circular contact pattern generated by the spherical indenter pressing on the surface of the visual-tactile sensor.
6. The method according to claim 1, characterized in that, Based on the surface gradient corresponding to each target pixel in the visual-tactile image under test, a three-dimensional tactile surface reconstruction is performed to obtain a three-dimensional surface topography map; including: Based on the surface gradient corresponding to each target pixel in the visual-tactile image to be tested, the discreteness is calculated using a fast Poisson solver. The discreteness is processed by a two-dimensional discrete cosine transform to obtain a three-dimensional depth image of the tactile surface; Based on color threshold, gradient threshold and depth threshold, the three-dimensional depth image of the tactile surface is reconstructed and filtered to generate a three-dimensional surface topography map.
7. The method according to claim 1, characterized in that, The step of visually reconstructing the three-dimensional depth image of the tactile surface based on color threshold, gradient threshold, and depth threshold to generate a three-dimensional surface topography map includes: Based on the original images from the visual-touch sensor, color threshold, gradient threshold, and depth threshold are obtained; Based on the color threshold and gradient threshold, a masking process is performed on the visual-tactile image to be tested to generate a mask image; Based on the mask image and the tactile surface 3D depth image, a visualization reconstruction is performed to generate a 3D surface topography map.
8. A three-dimensional surface reconstruction device based on visual-tactile images, characterized in that, include: The first acquisition module is used to acquire the visual-tactile image of the target object through a visual-tactile sensor; The mapping module is used to obtain the normalized coordinates and RGB information of any target pixel in the visual-tactile image to be tested. The normalized coordinates of the target pixel are subjected to position feature enhancement processing to generate a multi-frequency position vector; the multi-frequency position vector is concatenated with the RGB information to generate enhanced features; Based on the enhanced features, a lightweight network structure is used to predict the gradient field and generate the surface gradient corresponding to the target pixel. The surface reconstruction module is used to reconstruct the surface based on the surface corresponding to each target pixel in the visual-tactile image under test. The surface gradient is used to reconstruct the three-dimensional tactile surface, resulting in a three-dimensional surface topography map.
9. A computer-readable medium having a computer program stored thereon, the program being executed by a processor. The positioning method described in any one of claims 1 to 7 is implemented during the operation.
10. A robot, characterized in that, The robot constructs a three-dimensional surface topography map based on the acquired visual-tactile images according to any one of claims 1-7.