A dense normal distribution force field reconstruction method fusing force and visual tactile information
By fusing tactile images and normal forces through a neural network model based on the U-net framework, a dense normal force field is reconstructed, solving the problems of low resolution and high cost of array-type tactile sensors and achieving efficient and accurate robot tactile perception.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HUNAN UNIV
- Filing Date
- 2026-05-07
- Publication Date
- 2026-06-05
AI Technical Summary
Existing array-type tactile sensors have low resolution, are complex to manufacture, and are expensive, making them unable to meet the robot's need for precise perception of dense force fields.
We employ a neural network model based on the U-net framework, which integrates tactile images and normal forces. By reconstructing a dense normal force field through a training dataset, we simplify the hardware structure and reduce computational complexity.
It achieves high-resolution, dense force field perception, reduces hardware costs and computational complexity, and improves the accuracy and adaptability of robot tactile perception.
Smart Images
Figure CN122154490A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of force measurement technology, and specifically to a method for reconstructing a dense normal distribution force field that integrates force and visual / tactile information. Background Technology
[0002] With the emergence of embodied intelligence and the increasing demands of robotic tasks, single-point force sensing is no longer sufficient. Sensing dense force fields is gradually becoming a function that robots need to achieve. By sensing contact force fields, robots can determine the magnitude and location of applied forces, thereby completing more precise tasks.
[0003] Traditional array-based tactile sensors approximate force field sensing by arranging multiple sensing points in an array. However, they suffer from low resolution, complex manufacturing processes, and relatively high costs. With the rise of vision-based tactile sensors, new methods for reconstructing dense force fields have emerged. By acquiring tactile images of contact with an object, the local deformation of the elastic body on the sensor surface can be sensed, and this deformation contains force information. This force information can be acquired through methods based on physical models or data-driven approaches. Summary of the Invention
[0004] This invention provides a method for reconstructing a dense normal force field that integrates force and visual-tactile information, which can realize real-time dense normal force field reconstruction when the robot end effector comes into contact with an object.
[0005] To achieve the above technical objectives, the present invention adopts the following technical solution: A method for reconstructing a densely distributed force field that integrates force and visual-tactile information, comprising: A neural network model with dual-modal inputs was built based on the U-net framework; the dual-modal inputs include tactile images and normal force. Training data is generated to train the neural network model into a reconstructed model that outputs a dense normal force field. The reconstructed model is obtained by training the neural network model using the training dataset; Real-time acquisition of normal force and tactile images when the robot end effector comes into contact with an object, and prediction of dense normal force field distribution when the robot end effector comes into contact with an object using a trained reconstruction model.
[0006] Furthermore, the training data is prepared as follows: First, a normal force sensor A and a visual-tactile sensor are fixedly installed longitudinally, and a normal force sensor B is installed under a flat pressure head; Then, the longitudinally connected normal force sensor A and visual-tactile sensor are operated, causing the visual-tactile sensor to press on the planar pressure head, and during each pressing process... Multiple time points The resultant normal force of the contact surface is collected by normal force sensor A and recorded as follows: The tactile images of the contact surface are acquired through a visual-tactile sensor and recorded as... The resultant normal force at the contact surface is collected by normal force sensor B and recorded as follows: ; For each pressing process Extract the indentation area from its tactile image and construct a mask for this pressing process. Then Distributed evenly on the mask For each pixel, obtain the time point. Dense normal force field ; Each time point during each press Collected normal resultant force and tactile images As a training input to the reconstruction model, the corresponding dense normal force field will be obtained. As training label data for the reconstructed model Represents the coordinates of pixels in a tactile image.
[0007] Furthermore, from each pressing process Extract the indentation area from the tactile image and construct a mask for the pressing process. Specifically, this includes: from the pressing process From the tactile images at all time points, a number of tactile images were selected; then, the SAM model was used to segment the indentation region, and the corresponding segmented indentation regions from all the selected tactile images were merged to obtain the pressing process. mask .
[0008] Furthermore, Distributed evenly on the mask Each pixel is used to obtain a dense normal force field distribution. Specifically, it is expressed as: ; ; In the formula, This indicates the resultant force of the normal directions. The value that is evenly distributed across the white highlight area in the mask. To express summation, It is a tactile image and normal force The corresponding dense normal distribution force field label, Represents the coordinates of pixels in a tactile image. Indicates the pressing process Maximum time step.
[0009] Furthermore, multiple sets of training data are generated using planar indenters of various shapes, including any two or more of the following shapes: circles, triangles, squares, and hexagons.
[0010] Furthermore, the planar range of the flat pressure head is smaller than the acquisition surface of the visual-tactile sensor. For flat pressure heads of the same shape, a grid-based pressing method is adopted: the acquisition surface of the visual-tactile sensor is divided into grids, and a pressing process is performed on the flat pressure head with each grid as the center, and a set of training data is collected accordingly.
[0011] Furthermore, the reconstruction model includes a haptic image encoder, a normal force encoder, a feature fusion module, a decoder, and a final convolutional layer; The tactile image encoder performs multi-layer feature encoding on the input tactile image; The normal resultant force encoder performs multi-layer feature encoding on the input normal resultant force; The feature fusion module fuses the multi-layer feature codes of the two encoders to obtain multi-layer fused features; The decoder decodes the multi-layer fusion features layer by layer; The last convolutional layer performs channel mapping on the top-level decoded features and outputs a dense normal distribution force field.
[0012] Furthermore, the haptic image encoder employs several convolutional modules. The normal force encoder is composed of stacked downsampling layers and uses a fully connected layer. It is composed of stacked nonlinear activation functions, and multi-layer feature encoding is performed on the input tactile image and the resultant normal force, respectively, as follows: ; In the formula, Represents a haptic image encoder. The input image represents the haptic image encoder. Represents the normal resultant force encoder. The input image representing the normal resultant force encoder. These are the four layers of feature encoding output from the tactile image encoder. These are the four layers of feature codes output by the normal force encoder.
[0013] Furthermore, the feature fusion module fuses the multi-layer feature codes from the two encoders, specifically: First, the first three layers of features output by the tactile image encoder and the normal force encoder are fused using Hadamard product: ; in, The first output of the haptic image encoder represents the... Layer feature encoding, The first normal resultant force encoder output Layer feature encoding, It represents the Hadamah accumulation. Indicates the first Layer fusion features; Secondly, adaptive weighted fusion is performed on the feature encoding of the fourth layer: ; in, This represents adaptive weighting, specifically the process of encoding the fourth-layer features from the two encoders. and The concatenated features are then mapped to a 2-channel tensor via a convolution operation. The two channels of this tensor are... and The weights are then used to... and Perform weighted fusion.
[0014] Furthermore, the decoding process of the decoder includes: first, recording the fused features of the 4th layer as the decoded features of the 4th layer; then, upsampling the decoded features through deconvolution; finally, concatenating and fusing them with the fused features of the previous layer; and finally, using a... Layers and The convolution module implements feature channel mapping, specifically as follows: ; in, Indicates the first Features obtained from layer decoding ; Indicates the first Layer fusion features Indicates a deconvolution layer. Indicates feature channel splicing. This is a convolutional module.
[0015] Compared with existing technologies, the advantages of this invention are as follows: This invention provides a dense normal force field reconstruction method that integrates force and visual-tactile information. Compared with traditional array-type tactile sensors, its structural design is simpler, eliminating the need for complex array arrangements and significantly reducing hardware manufacturing costs. Furthermore, this invention uses a neural network to directly reconstruct the normal force distribution field. Compared with the traditional finite element method (FEM) for generating force distribution, this not only reduces computational complexity but also avoids discrepancies between simulation results and actual contact states. Through end-to-end fusion of visual-tactile and force information, this invention achieves high-resolution dense force field perception, enabling real-time and efficient reconstruction of the normal force distribution on the contact surface. This improves the accuracy and adaptability of robot tactile perception and makes it suitable for complex task environments. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the model for reconstructing dense normal force distribution in this embodiment; Figure 2 This is a diagram of the equipment used for reconstructing the dense normal force distribution in this embodiment; Figure 3 This is a schematic diagram illustrating the fabrication of the densely distributed normal force label in this embodiment; Figure 4 This is a diagram showing the prediction results of a reconstructed dense normal distribution force in this embodiment.
[0017] Reference numerals: 1-Three-dimensional moving platform, 2-Normal force sensor A, 3-Visual and tactile sensor, 4-Normal force sensor B, 5-Planar circular pressure head. Detailed Implementation
[0018] The embodiments of the present invention will be described in detail below. These embodiments are based on the technical solutions of the present invention and provide detailed implementation methods and specific operation processes to further explain the technical solutions of the present invention.
[0019] This embodiment provides a method for reconstructing a dense normal distribution force field that integrates force and visual-tactile information, including the following steps: Step 1: Build a neural network model with dual-modal input based on the U-net framework. The reconstructed model includes a tactile image encoder, a normal force encoder, a feature fusion module, a decoder, and a final convolutional layer. The dual-modal inputs include the tactile image and the normal force. The detailed structure of the neural network model is as follows: Figure 1 As shown.
[0020] The tactile image encoder employs several convolutional modules. The convolutional module is composed of stacked downsampling layers. Each convolutional module consists of two consecutive CBR blocks (Conv2D-BatchNorm2D-ReLU) and a Coordinate Attention (CA) mechanism. The downsampling module consists of a 2×2 max pooling layer and a Dropout layer. The output of each convolutional module is a layer of tactile image features. The multi-layer feature encoding of the input tactile image is as follows: ; In the formula, Represents a haptic image encoder. The input image represents the haptic image encoder. These are the four layers of feature encoding output by the tactile image encoder.
[0021] The normal force encoder uses a fully connected layer. It is composed of stacked nonlinear activation functions such as Softmax, with the output of each activation function representing a force feature layer. Multi-layer feature encoding is applied to the input normal resultant force, as shown below: ; In the formula, Represents the normal resultant force encoder. The input image representing the normal resultant force encoder. These are the four layers of feature codes output by the normal force encoder.
[0022] The feature fusion module fuses the multi-layer feature codes from the two encoders to obtain multi-layer fused features. Specifically: First, the first three layers of features output by the tactile image encoder and the normal force encoder are fused using Hadamard product: ; in, The first output of the haptic image encoder represents the... Layer feature encoding, The first normal resultant force encoder output Layer feature encoding, It represents the Hadamah accumulation. Indicates the first Layer fusion features; Secondly, adaptive weighted fusion is performed on the feature encoding of the fourth layer: ; in, This represents adaptive weighting, specifically the process of encoding the fourth-layer features from the two encoders. and The concatenated features are then mapped to a 2-channel tensor via a convolution operation. The two channels of this tensor are... and The weights are then used to... and Perform weighted fusion.
[0023] The decoder decodes the multi-layer fused features layer by layer: first, the fused features of the 4th layer are recorded as the decoded features of the 4th layer; then, the decoded features are upsampled by deconvolution and then concatenated and fused with the fused features of the previous layer, and finally fused through a... Layers and The convolution module implements feature channel mapping, specifically as follows: ; in, Indicates the first Features obtained from layer decoding , Indicates the first Layer fusion features Indicates a deconvolution layer. Indicates feature channel splicing. This is a convolutional module. Finally, a convolutional layer performs channel mapping on the top-level decoded features, converting the number of channels in the decoder's output features to a specified size for outputting a dense normal force field. This is represented as: ; in, This is the output of the neural network, which is the standardized normal force distribution field predicted by the reconstructed model. This represents a two-dimensional convolutional layer.
[0024] Step 2: Create training data to train the neural network model into a reconstructed model that outputs a dense normal distribution force field.
[0025] Step 2.1: Install the normal force sensor A and the visual-tactile sensor longitudinally, and install another normal force sensor B under a flat pressure head.
[0026] In this embodiment, reference Figure 2 As shown, a normal force sensor A and a visual-tactile sensor are longitudinally mounted on a three-axis moving platform. They are fixedly connected by an adapter flange. The visual-tactile sensor is located below the normal force sensor A, with its sampling surface facing downwards for contact with the object. Another normal force sensor B is longitudinally connected to a planar indenter and fixed to the test bench, with the planar indenter facing upwards. The normal force sensor A is model SBT674, the visual-tactile sensor is model GelSight, and the normal force sensor B is model ZNLBM.
[0027] Step 2.2: Using a three-axis moving platform, operate the longitudinally connected normal force sensor A and visual-tactile sensor, causing the visual-tactile sensor to press on the planar pressure head, and during each pressing process... Multiple time points The resultant normal force of the contact surface is collected by normal force sensor A and recorded as follows: The tactile images of the contact surface are acquired through a visual-tactile sensor and recorded as... The resultant normal force at the contact surface is collected by normal force sensor B and recorded as follows: .like Figure 3 As shown.
[0028] In a preferred embodiment, the planar range of the indenter is smaller than the acquisition surface of the visual-tactile sensor. For planar indenters of the same shape, a gridded pressing method is used: the acquisition surface of the visual-tactile sensor is divided into a grid, and a pressing process is performed on the planar indenter with each grid as the center, and a set of training data is collected accordingly. Specifically, the planar indenter is fixed on the test bench, and the grid is divided into 2mm sections along the x-axis and y-axis with the center of the GelSight sensor surface as the origin, so that each grid of the visual-tactile sensor contacts the pressing platform indenter. In the z-axis direction, a data acquisition device consisting of a normal force sensor A and a visual-tactile sensor is used to press the circular indenter, slowly loading from an initial force of 0N to a maximum contact force of 10N, and then gradually unloading until the pressure is completely released, completing one pressing process. Data is collected at several time steps during this pressing process. In this embodiment, a total of 7×11 sets of experimental data based on circular indentations were obtained, and 1 set of data without indentations was added, for a total of 27,851 samples.
[0029] In this embodiment, for the other three types of indenters, including triangular, square and hexagonal indenters, two pressing tests were randomly conducted for each type, resulting in 6 independent datasets as test sets, totaling 4602 samples, which were used to evaluate the model's generalization ability to unseen indenter shapes.
[0030] Step 2.3, for each pressing process Extract the indentation area from its tactile image and construct a mask for this pressing process. Then Distributed evenly on the mask For each pixel, obtain the time point. Dense normal force field .
[0031] Among them, from each pressing process Extract the indentation area from the tactile image and construct a mask for the pressing process. Specifically, this includes: from the pressing process From the tactile images at all time points, several images with clear indentations were selected. This selection could be done manually, generally with greater pressure resulting in clearer indentations. Then, the SAM model was used to segment the indentation regions, and the segmented indentation regions from all the selected tactile images were merged to obtain the pressing process. mask .
[0032] ; ; in, This indicates the sequence number of the data group collected during each press. Indicates the first The first of 10 binarized contact region images in the dataset indivual, This represents the union operation. Indicates the first A mask for group data.
[0033] For a planar indenter, the force on its contact area can be considered uniformly distributed. Therefore, a label with a densely distributed normal force field can be represented as a label whose resultant normal force is uniformly distributed over the white area of the mask. Specifically: ; ; In the formula, This indicates the resultant force of the normal directions. The value that is evenly distributed across the white highlight area in the mask. To express summation, It is a tactile image and normal force The corresponding dense normal distribution force field label, Represents the coordinates of pixels in a tactile image. Indicates the pressing process Maximum time step.
[0034] Step 2.4: Record each time point during each press. Collected normal resultant force and tactile images As a training input to the reconstruction model, the corresponding dense normal force field will be obtained. As training label data for the reconstructed model Represents the coordinates of pixels in a tactile image.
[0035] Step 3: Train the neural network model using the training dataset to obtain the reconstructed model.
[0036] Before training, the tactile images and dense normal force field labels were standardized. ; ; in, and Representing tactile images The corresponding statistical mean and standard deviation, and These represent densely distributed normal force fields. The corresponding statistical mean and standard deviation.
[0037] The model is then trained. During training, the desired output of the model is... With the standardized dense normal distribution force field in the label The goal is to minimize the error, while also ensuring that the resultant force corresponding to the output of the inverse-standardized model matches the normal resultant force in the label. The error should be minimized. Therefore, the training loss function... Defined as two parts: ; in, Represents the mean square error. The resultant normal force is collected by normal force sensor B. The weight corresponding to the loss of normal resultant force. for The dense normal force field obtained by inverse normalization .
[0038] The TINFnet training process in this embodiment is as follows: First, the training data is randomly divided into a training set and a validation set in an 8:2 ratio, and the model is trained using the following training parameters: The Adam optimizer is used, with an initial learning rate of 0.01, and the learning rate is gradually decayed to 10 using a cosine annealing strategy. -7 The entire training process lasted for 120 iterations. Results show that on the validation set, the mean structural similarity index (SSIM) of the densely distributed normal force field is 0.8548, the root mean square error (RMSE) of the resultant force is 0.2401 N, and the mean absolute error (MAE) is 0.1969 N; on the test set, the SSIM of the densely distributed normal force field is 0.8775, the RMSE of the resultant force is 0.3338 N, and the MAE is 0.2610 N.
[0039] Step 4: Acquire the normal force and tactile images when the robot end effector comes into contact with the object in real time, and use the trained reconstruction model to predict the dense normal force field when the robot end effector comes into contact with the object.
[0040] The trained reconstruction model, along with the longitudinally connected normal force sensor and visual-tactile sensor, are mounted on the end effector of the robotic arm and deployed onto the robot system platform. The devices are then activated to collect the resultant normal force in real time. and tactile images and tactile images The data is standardized, then input into the trained reconstruction model and passed through forward propagation, finally affecting the output of the reconstruction model. Inverse normalization is performed to reconstruct the predicted dense normal force field: ;
[0041] The dense normal force field reconstructed in this invention can be directly used for various robot perception tasks, such as object shape perception, multi-point contact position perception, and stiffness determination. An example of a reconstructed dense normal force field is shown below. Figure 4 As shown.
[0042] The above embodiments are preferred embodiments of this application. Those skilled in the art can make various changes or improvements based on them. Without departing from the overall concept of this application, these changes or improvements should fall within the scope of protection claimed in this application.
Claims
1. A method for reconstructing a densely distributed normal force field that integrates force and visual-tactile information, characterized in that, include: A neural network model with dual-modal inputs was built based on the U-net framework; the dual-modal inputs include tactile images and normal force. Training data is generated to train the neural network model into a reconstructed model that outputs a dense normal force field. The reconstructed model is obtained by training the neural network model using the training dataset; Real-time acquisition of normal force and tactile images when the robot end effector comes into contact with an object, and prediction of dense normal force field distribution when the robot end effector comes into contact with an object using a trained reconstruction model.
2. The method for reconstructing a densely distributed normal force field that integrates force and visual-tactile information according to claim 1, characterized in that, The training data was prepared as follows: First, a normal force sensor A and a visual-tactile sensor are fixedly installed longitudinally, and a normal force sensor B is installed under a planar pressure head; Then, the longitudinally connected normal force sensor A and visual-tactile sensor are operated, causing the visual-tactile sensor to press on the planar pressure head, and during each pressing process... Multiple time points The resultant normal force of the contact surface is collected by normal force sensor A and recorded as follows: The tactile images of the contact surface are acquired through a visual-tactile sensor and recorded as... The resultant normal force at the contact surface is collected by normal force sensor B and recorded as follows: ; For each pressing process Extract the indentation area from its tactile image and construct a mask for this pressing process. Then Distributed evenly on the mask For each pixel, obtain the time point. Dense normal force field ; Each time point during each press Collected normal resultant force and tactile images As a training input to the reconstruction model, the corresponding dense normal force field will be obtained. As training label data for the reconstructed model Represents the coordinates of pixels in a tactile image.
3. The method for reconstructing a densely distributed normal force field that integrates force and visual-tactile information according to claim 2, characterized in that, From each press process Extract the indentation area from the tactile image and construct a mask for the pressing process. Specifically, this includes: from the pressing process From the tactile images at all time points, a number of tactile images were selected; then, the SAM model was used to segment the indentation region, and the corresponding segmented indentation regions from all the selected tactile images were merged to obtain the pressing process. mask .
4. The method for reconstructing a densely distributed normal force field that integrates force and visual-tactile information according to claim 2, characterized in that, Will Distributed evenly on the mask Each pixel is used to obtain a dense normal force field distribution. Specifically, it is expressed as: ； ； In the formula, This indicates the resultant force of the normal directions. The value that is evenly distributed across the white highlight area in the mask. To express summation, It is a tactile image and normal force The corresponding dense normal distribution force field label, Represents the coordinates of pixels in a tactile image. Indicates the pressing process Maximum time step.
5. The method for reconstructing a densely distributed normal force field that integrates force and visual-tactile information according to claim 2, characterized in that, Multiple sets of training data are generated using planar indenters of various shapes, including any two or more of the following shapes: circles, triangles, squares, and hexagons.
6. The method for reconstructing a densely distributed normal force field that integrates force and visual-tactile information according to claim 2, characterized in that, The planar range of the flat pressure head is smaller than the acquisition surface of the visual-tactile sensor. For flat pressure heads of the same shape, a grid-based pressing method is adopted: the acquisition surface of the visual-tactile sensor is divided into grids, and a pressing process is performed on the flat pressure head with each grid as the center, and a set of training data is collected accordingly.
7. The method for reconstructing a densely distributed normal force field that integrates force and visual-tactile information according to claim 1, characterized in that, The reconstruction model includes a haptic image encoder, a normal force encoder, a feature fusion module, a decoder, and a last convolutional layer; The tactile image encoder performs multi-layer feature encoding on the input tactile image; The normal resultant force encoder performs multi-layer feature encoding on the input normal resultant force; The feature fusion module fuses the multi-layer feature codes of the two encoders to obtain multi-layer fused features; The decoder decodes the multi-layer fusion features layer by layer; The last convolutional layer performs channel mapping on the top-level decoded features and outputs a dense normal distribution force field.
8. The method for reconstructing a densely distributed normal force field that integrates force and visual-tactile information according to claim 7, characterized in that, The haptic image encoder employs several convolutional modules. The normal force encoder is composed of stacked downsampling layers and uses a fully connected layer. It is composed of stacked nonlinear activation functions, and multi-layer feature encoding is performed on the input tactile image and the resultant normal force, respectively, as follows: ； In the formula, Represents a haptic image encoder. The input image represents the haptic image encoder. Represents the normal resultant force encoder. The input image representing the normal resultant force encoder. These are the four layers of feature encoding output from the tactile image encoder. These are the four layers of feature codes output by the normal force encoder.
9. The method for reconstructing a densely distributed normal force field that integrates force and visual-tactile information according to claim 7, characterized in that, The feature fusion module fuses the multi-layer feature codes from the two encoders, specifically: First, the first three layers of features output by the tactile image encoder and the normal force encoder are fused using Hadamard product: ； in, The output of the haptic image encoder represents the first... Layer feature encoding, The first normal resultant force encoder output Layer feature encoding, It represents the Hadamah accumulation. Indicates the first Layer fusion features; Secondly, adaptive weighted fusion is performed on the feature encoding of the fourth layer: ； in, This represents adaptive weighting, specifically the process of encoding the fourth-layer features from the two encoders. and The concatenated features are then mapped to a 2-channel tensor via a convolution operation. The two channels of this tensor are... and The weights are then used to... and Perform weighted fusion.
10. The method for reconstructing a densely distributed normal force field that integrates force and visual-tactile information according to claim 7, characterized in that, The decoder's decoding process includes: first, recording the fused features of layer 4 as the decoded features of layer 4; then, upsampling the decoded features through deconvolution; finally, concatenating and fusing them with the fused features of the previous layer; and finally, using a... Layers and The convolution module implements feature channel mapping, specifically as follows: ； in, Indicates the first Features obtained from layer decoding ; Indicates the first Layer fusion features Indicates a deconvolution layer. Indicates feature channel splicing. This is a convolutional module.