A haptic force field reconstruction method and system based on global multi-directional force information
Patent Information
- Application Number
- CN202611108126.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-24
- Publication Date
- 2026-10-23
AI Technical Summary
此时，计算机视觉难以分辨出近似的接触特征，使由形变特征到三维力场的映射关系失效
[0015]本发明方法及系统的有益效果是：本发明通过获取视触觉传感器接触目标物体过程中的接触图像与对应的全局多向力信息，全局多向力信息能够反映传感器整体所受载荷状态；进而对接触图像进行预处理，提取传感器接触表面的局部形变特征；进一步的，将传感器接触表面的局部形变特征输入至前向力场估计模型进行触觉力场重建，生成初始接触力场，利用整体载荷信息参与模型训练和修正过程，因此有助于提高宽量程接触条件下力场重建方法的适用范围；最后融合初始接触力场、局部形变特征和全局多向力信息，基于力场修正模型，对初始接触力场进行修正，得到满足全局力学约束的修正接触力场，作为视触觉传感器接触区域内的重建力场，在利用局部形变信息估计接触力场的基础上，引入由多维力传感单元获取的全局多向力信息，对初始接触力场进行修正，在接触载荷增大、局部图像形变变化不明显或趋于非线性响应时，可为力场重建提供额外的载荷约束信息，从而有助于扩大视触觉传感器的力场重建有效量程，能够在较宽载荷变化范围内继续利用全局载荷信息参与力场修正，降低因局部形变特征变化不足而导致的力场重建受限问题，尤其有助于拓展法向压力方向的可测范围。
Smart Images
Figure CN122965290A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot perception technology, and in particular to a method and system for reconstructing tactile force fields based on global multi-directional force information. Background Technology
[0002] With the increasing demand for rich tactile feedback in robotic tasks, high spatial resolution tactile perception technology has become a significant research hotspot. In contact-rich tasks with a wide range of contact force variations, robots need to reconstruct force fields with finer spatial resolution, a wider range of variations, and physical consistency to reliably regulate interaction forces and avoid problems such as local overload and object damage. Based on the principle of visual-tactile sensing, robots can acquire color tactile images, extract deformation information of the elastic thin film on the sensor surface, and map it into a three-dimensional force field. However, this type of method faces the problems of nonlinear characteristics of the elastic thin film and ambiguity of the force field during the perception process. Specifically, when the load on the sensor exceeds a certain threshold, a large change in contact force is required to trigger a small increment in surface deformation. At this point, computer vision struggles to distinguish approximate contact features, causing the mapping relationship from deformation features to the three-dimensional force field to fail. Summary of the Invention
[0003] To address the aforementioned technical problems, the present invention aims to provide a tactile force field reconstruction method and system based on global multi-directional force information, which can expand the effective range of force field reconstruction of visual tactile sensors and extend the measurable range of normal pressure direction.
[0004] The first technical solution adopted in this invention is: a method for reconstructing a tactile force field based on global multi-directional force information, comprising the following steps: Acquire contact images and corresponding global multi-directional force information during the process of visual-tactile sensors contacting target objects; The contact image is preprocessed to extract the local deformation features of the sensor contact surface; The local deformation characteristics of the sensor's contact surface are input into the forward force field estimation model to reconstruct the tactile force field and generate the initial contact force field.
[0005] By integrating the initial contact force field, local deformation characteristics, and global multi-directional force information, and based on the force field correction model, the initial contact force field is corrected to obtain a corrected contact force field that satisfies global mechanical constraints, which serves as the reconstructed force field within the contact area of the visual-touch sensor.
[0006] Furthermore, the local deformation features include tangential deformation features, normal deformation features, or a combination of both, wherein: The tangential deformation feature is extracted by calculating the optical flow change based on the temporal variation features of the RGB contact image; The normal deformation feature is obtained by solving a set of depth relationship equations constructed from the color-gradient mapping relationship of the RGB contact image.
[0007] Furthermore, the forward force field estimation model is constructed based on a multilayer perceptron and is used to construct the mapping relationship between the local deformation features of the sensor contact surface and the initial contact force field.
[0008] Furthermore, the force field correction model includes a global information encoding module, a local feature encoding module, a conditional fusion module, a local feature decoding module, and a fusion output module. The global information encoding module is built based on an MLP array, and both the local feature encoding module and the local feature decoding module are built based on a U-Net network. The conditional fusion module is based on one or more of the feature-level linear modulation and attention mechanisms.
[0009] Furthermore, the step of fusing the initial contact force field, local deformation features, and global multi-directional force information, and correcting the initial contact force field based on the force field correction model to obtain a corrected contact force field that satisfies global mechanical constraints, which serves as the reconstructed force field within the contact area of the visual-touch sensor, specifically includes: Based on the global information encoding module of the force field correction model, the global multi-directional force information is encoded and constrained to obtain global constraint features; Based on the local feature encoding module of the force field correction model, feature extraction and fusion are performed on the initial contact force field and local deformation features to obtain local spatial features; Based on the conditional fusion module of the force field correction model, global constraint features are fused into local spatial features to obtain fused features constrained by global multi-directional force information. Based on the local feature decoding module of the force field correction model, the fusion features constrained by global multi-directional force information are decoded to recover the force field correction features corresponding to the spatial resolution of the contact area. The fusion output module based on the force field correction model generates a force field correction amount according to the force field correction characteristics, and applies the force field correction amount to the initial contact force field to obtain a corrected contact force field that satisfies global mechanical constraints, which serves as the reconstructed force field within the contact area of the visual-touch sensor.
[0010] Furthermore, the loss function of the force field correction model includes one or more of the following: point-by-point force field error term, resultant force constraint error term, resultant moment constraint error term, and non-contact region regularization term, so that the corrected contact force field output by the force field correction model has local distribution accuracy and global mechanical consistency.
[0011] Furthermore, the training datasets for the forward force field estimation model and the force field correction model are obtained through finite element simulation. The training datasets include local deformation features, global multi-directional force information, and the corresponding supervised contact force field.
[0012] Furthermore, the finite element simulation includes establishing a contact model of the indenter, rigid support, and elastic sensing layer, and generating deformation field, global load, and contact force field labels under different contact conditions by changing the position, orientation, and load range of the indenter.
[0013] Furthermore, the force field correction amount generated by the fusion output module includes a local distribution correction amount and a global amplitude correction amount. The local distribution correction amount corresponds to the force vector amplitude coefficient at each spatial location in the contact area, and is used to modulate the spatial distribution of the contact force field point by point to correct the local deviation of the initial contact force field estimation. The global amplitude correction amount corresponds to the compensation coefficient of the overall load state, and is used to compensate the load amplitude of the contact force field as a whole to correct the overall load deviation between the initial contact force field and the global multi-directional force information.
[0014] The second technical solution adopted in this invention is: a tactile force field reconstruction system based on global multi-directional force information, comprising: The first module is used to acquire contact images and corresponding global multi-directional force information during the process of the visual-touch sensor contacting the target object; The second module is used to preprocess the contact image and extract the local deformation features of the sensor contact surface; The third module is used to input the local deformation characteristics of the sensor contact surface into the forward force field estimation model to reconstruct the tactile force field and generate the initial contact force field. The fourth module is used to fuse the initial contact force field, local deformation features and global multi-directional force information. Based on the force field correction model, the initial contact force field is corrected to obtain a corrected contact force field that satisfies global mechanical constraints, which serves as the reconstructed force field within the contact area of the visual-touch sensor.
[0015] The beneficial effects of the method and system of this invention are as follows: This invention acquires contact images and corresponding global multi-directional force information during the process of a visual-tactile sensor contacting a target object. The global multi-directional force information can reflect the overall load state of the sensor. The contact images are then preprocessed to extract local deformation features of the sensor's contact surface. Furthermore, the local deformation features of the sensor's contact surface are input into a forward force field estimation model for tactile force field reconstruction, generating an initial contact force field. The overall load information is used in the model training and correction process, thus helping to improve the applicability of the force field reconstruction method under wide-range contact conditions. Finally, the initial contact force field, local deformation features, and global multi-directional force information are fused, and based on the force field correction model, the initial contact force field is adjusted. The contact force field is corrected to obtain a corrected contact force field that satisfies global mechanical constraints. This corrected contact force field serves as the reconstructed force field within the contact area of the visual tactile sensor. Based on the estimation of the contact force field using local deformation information, global multi-directional force information obtained by the multi-dimensional force sensing unit is introduced to correct the initial contact force field. When the contact load increases, or when the local image deformation changes are not significant or tend to be nonlinear, additional load constraint information can be provided for force field reconstruction. This helps to expand the effective range of force field reconstruction of the visual tactile sensor. It can continue to use global load information to participate in force field correction within a wider load variation range, reducing the problem of limited force field reconstruction caused by insufficient changes in local deformation characteristics. In particular, it helps to expand the measurable range in the normal pressure direction. Attached Figure Description
[0016] Figure 1 This is a flowchart of the steps of a tactile force field reconstruction method based on global multi-directional force information according to the present invention; Figure 2 This is a structural block diagram of a tactile force field reconstruction system based on global multi-directional force information according to the present invention; Figure 3 This is a schematic diagram of the overall framework of the force field reconstruction method provided in a specific embodiment of the present invention; Figure 4 This is a schematic diagram of the force field reconstruction algorithm structure provided in a specific embodiment of the present invention; Figure 5 This is a schematic diagram of a CAD model for data acquisition provided in a specific embodiment of the present invention; Figure 6 This is a schematic diagram of the modified contact force field output provided in a specific embodiment of the present invention. Detailed Implementation
[0017] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.
[0018] Reference Figure 1 This invention provides a method for reconstructing a tactile force field based on global multi-directional force information, the method comprising the following steps: S100: Acquire contact images and corresponding global multi-directional force information during the process of the visual-tactile sensor contacting the target object; S200. Preprocess the contact image to extract the local deformation features of the sensor contact surface; The local deformation features include tangential deformation features, normal deformation features, or a combination of both. The tangential deformation features are extracted by calculating optical flow changes through the temporal variation features of the RGB contact image. The normal deformation features are obtained by solving a set of depth relationship equations constructed from the color-gradient mapping relationship of the RGB contact image.
[0019] In this embodiment, the local deformation features include tangential deformation features, normal deformation features, or a combination of both. The image preprocessing module calculates optical flow changes using the temporal variation features of the RGB contact image to extract tangential deformation features; and uses the color-gradient mapping relationship of the RGB contact image to solve and calculate the normal deformation features.
[0020] S300: Input the local deformation characteristics of the sensor contact surface into the forward force field estimation model to reconstruct the tactile force field and generate the initial contact force field; The forward force field estimation model is based on a multilayer perceptron and is used to construct the mapping relationship between the local deformation features of the sensor contact surface and the initial contact force field.
[0021] S400 integrates the initial contact force field, local deformation characteristics, and global multi-directional force information. Based on the force field correction model, it corrects the initial contact force field to obtain a corrected contact force field that satisfies global mechanical constraints, which serves as the reconstructed force field within the contact area of the visual-touch sensor.
[0022] Specifically, the force field correction model includes a global information encoding module, a local feature encoding module, a conditional fusion module, a local feature decoding module, and a fusion output module. The global information encoding module is built based on an MLP array, and both the local feature encoding module and the local feature decoding module are built based on a U-Net network. The conditional fusion module is based on one or more of the feature-level linear modulation and attention mechanisms.
[0023] The global mechanical constraints include the deviation between the resultant force and / or resultant moment obtained by spatial integration of the modified contact force field and the global multi-directional force information satisfying a preset condition.
[0024] S410. The global information encoding module based on the force field correction model encodes and constrains the global multi-directional force information to obtain global constraint features; S420. Based on the force field correction model, the local feature encoding module extracts and fuses the initial contact force field and local deformation features to obtain local spatial features. S430. Based on the force field correction model, the condition fusion module fuses the global constraint features into local spatial features to obtain fused features constrained by global multi-directional force information. S440. The local feature decoding module based on the force field correction model decodes the fused features constrained by global multi-directional force information to recover the force field correction features corresponding to the spatial resolution of the contact area. S450. The fusion output module based on the force field correction model generates a force field correction amount according to the force field correction characteristics, and applies the force field correction amount to the initial contact force field to obtain a corrected contact force field that satisfies global mechanical constraints, which serves as the reconstructed force field within the contact area of the visual tactile sensor.
[0025] In this embodiment, the global information encoding module is built on an MLP array and is used to receive global multi-directional force information and encode the global multi-directional force information into global constraint features; the local feature encoding module is built on a U-Net network and is used to receive the initial contact force field and local deformation features, and extract local spatial features from the initial contact force field and the local deformation features; the conditional fusion module is based on one or more of feature-level linear modulation (FiLM) and attention mechanisms, and is used to receive the global constraint features and the local spatial features, and fuse the global constraint features into the local spatial features to obtain fused features constrained by global multi-directional force information; the local feature decoding module is built on a U-Net network and is used to decode the fused features to recover the force field correction features corresponding to the spatial resolution of the contact area; the fusion output module is used to generate a force field correction amount according to the force field correction features, and apply the force field correction amount to the initial contact force field to obtain a corrected contact force field.
[0026] The force field correction amount generated by the fusion output module includes a local distribution correction amount and a global amplitude correction amount. The local distribution correction amount corresponds to the force vector amplitude coefficient at each spatial location in the contact area and is used to modulate the spatial distribution of the contact force field point by point to correct the local deviation of the initial contact force field estimation. The global amplitude correction amount corresponds to the compensation coefficient of the overall load state and is used to compensate the load amplitude of the contact force field as a whole to correct the overall load deviation between the initial contact force field and the global multi-directional force information.
[0027] Furthermore, the loss function of the force field correction model includes one or more of the following: pointwise force field error term, resultant force constraint error term, resultant moment constraint error term, and non-contact region regularization term. This ensures that the corrected contact force field output by the force field correction model possesses both local distribution accuracy and global mechanical consistency. The training datasets for the forward force field estimation model and the force field correction model are obtained through finite element simulation. The training dataset includes local deformation features, global multi-directional force information, and the corresponding supervised contact force field. The finite element simulation involves establishing a contact model of the indenter, rigid support, and elastic sensing layer. By changing the indenter position, attitude, and load range, deformation fields, global loads, and contact force field labels under different contact conditions are generated.
[0028] More specifically, the force field correction module is used to fuse the initial contact force field, deformation characteristics, and global multi-directional force information to generate a corresponding force field correction amount, and then apply the force field correction amount to the initial contact force field to obtain the final contact force field. Optionally, the force field correction model includes a global information encoding module, a local feature extraction module, a conditional fusion module, a local feature decoding module, and a fusion output module.
[0029] The global information encoding module is built on an MLP array and encodes global multi-directional force information into an embedding vector as a global constraint feature. Optionally, the local feature encoding module constructs a U-Net structure through a convolutional network to extract local spatial features from the initial contact force field and local deformation features; The conditional fusion module is built based on FiLM and attention mechanisms, and fuses the global constraint features with the local spatial features; The local feature decoding module constructs a U-Net structure through a convolutional network to decode the fused features and recover the force field correction features corresponding to the spatial resolution of the contact area. The fusion output module is used to generate a force field correction amount based on the force field correction characteristics, and apply the force field correction amount to the initial contact force field to obtain the corrected contact force field.
[0030] The three parts of the force field reconstruction framework exhibit a hierarchical, interconnected structure, expressed as follows: In the formula, For image preprocessing, The initial untouched image, For contact image, The local deformation feature; This is the forward estimation process of the force field. This is the initial force field; For the force field correction process, For global multi-directional force information, To correct the force field.
[0031] During training, one or more of the following loss functions are used: point-by-point force field error, resultant force error, resultant moment error, and non-contact region regularization term, so that the model output can take into account both local distribution accuracy and global mechanical consistency requirements.
[0032] The training methods for the dataset include: 1) such as Figure 5 As shown, various pressure head models and sensor models were constructed using CAD software; 2) In finite element simulation software, contact simulation is performed based on the indenter model and sensor model to construct a dataset that includes local deformation characteristics, global multi-directional force information, and a supervised contact force field; 3) Train the forward force field estimation model and the force field correction model.
[0033] In summary, such as Figure 3 As shown, this embodiment of the invention acquires contact images from a visual-tactile sensor and corresponding global multi-directional force information; preprocesses the contact images to extract local deformation features of the contact surface; generates an initial contact force field based on the local deformation features; and fuses the initial contact force field, local deformation features, and global multi-directional force information to correct the initial contact force field, resulting in a corrected contact force field that satisfies global mechanical constraints. This method can improve the stability and physical consistency of three-dimensional force field reconstruction under contact conditions with a wide range of load variations, and is suitable for robot grasping, manipulation, and contact-sensitive scenarios.
[0034] Finally, the embodiments of the present invention will be explained and described in conjunction with the accompanying drawings: like Figure 4 As shown, a force field reconstruction algorithm framework is first established. This framework consists of three parts: an image preprocessing module, a forward force field estimation model, and a force field correction network. The image preprocessing module is used to preprocess the captured contact images and extract surface deformation information. The forward force field estimation model is used to establish the mapping from deformation information to the initial force field and generate the initial force field estimate in real time during actual operation. The force field correction network corrects the initial force field by fusing the initial force field information with global multi-directional force information, thereby achieving a balance between local accuracy and global physical consistency in the force field reconstruction results under large loads.
[0035] Furthermore, the local deformation features include tangential deformation features and normal deformation features. Tangential deformation features are obtained through optical flow estimation and characterize the displacement changes of the sensor surface in the image plane. This is achieved by capturing two adjacent image frames, constructing a gradient magnitude map, and accumulating the total pixel displacement relative to the device initialization reference frame. Normal deformation features are obtained through a photometric stereo method based on color-gradient mapping and characterize the deformation changes of the sensor surface along the normal direction. Specifically, this involves pre-constructing a color-gradient lookup table using a standard indenter, querying the image gradient distribution in real-time calculation, and establishing a system of linear equations to solve the depth map. Subsequently, the tangential and normal deformation features are fused to obtain the local deformation features used for force field estimation, denoted as... .
[0036] A multilayer perceptron is used as the forward force field estimation model to approximate the mapping relationship from deformation features to the initial force field. This estimation process is written as follows: Furthermore, such as Figure 6 As shown, the force field correction network includes a global information encoding module, a local feature encoding module, a conditional fusion module, a local feature decoding module, and a fusion output module. The global information encoding module encodes multi-directional forces into global constraint features; the local feature encoding module extracts local spatial features from the initial contact force field and local deformation features; the conditional fusion module fuses the global constraint features with the local spatial features; the local feature decoding module decodes the fused features; and the fusion output module outputs the force field correction amount. Applying the force field correction amount to the initial contact force field yields the corrected contact force field.
[0037] Global Information Encoder It consists of several lightweight MLP modules. It utilizes a six-dimensional force / torque sensor to record global multi-directional force information corresponding to the contact process. The global multi-directional force information is input into the global information encoder. First, a global mapping MLP is used to convert the global multi-directional force reference... Embedded as a latent condition vector Subsequently, A token-based MLP is input, along with a projected MLP, to generate features for conditional fusion. Features of global constraint correction in the decoding process The above process can be expressed as follows: The local feature encoder consists of hierarchical convolutional layers. Initial force field. With deformation characteristics Through encoders with the same structure and This enhances the representation of force field features. The structure of the local feature encoder is as follows: a convolution-normalization-activation function module, followed by three stacked convolutional layers: the first two layers each consist of two convolutional modules and one downsampling module; the last layer consists of only two convolutional modules. The hierarchical local feature encoding process can be expressed as: A conditional fusion module is constructed using FiLM and an attention mechanism to fuse global constraint features and local spatial features. This is achieved by utilizing the global latent vector. Feature-level linear modulation of local features can be expressed as: Subsequently, a cross-attention module is introduced. In this module, key-value pairs are defined by bottleneck features. With global conditional terms It is jointly encoded. First, [the following is...] Perform average pooling and reshaping, then with Concatenate to obtain key-value pairs , Query value Then by The features are obtained by flattening the surface. Cross-attention computation is then performed to obtain the fused features. , expressed as: In the formula, This represents a multi-head attention mechanism.
[0038] The local feature decoding module utilizes a local feature decoder with the same structure. and , respectively and Decoding is then performed. The decoder structure is as follows: the first layer contains two convolutional modules; the latter two layers each consist of an upsampling module, a convolution-normalization-activation function module, and a convolutional module. This process can be expressed as:
[0039] In the formula, and These are the features after decoding the force branch and deformation branch, respectively.
[0040] The force field decoding features and deformation decoding features are input into the feature fusion output module to generate a residual force field for correcting the initial contact force field. The feature fusion output module includes a feature fusion unit, a local residual generation unit, a global residual generation unit, and a force field synthesis unit.
[0041] The feature fusion unit uses a convolutional network to fuse and project force field decoding features and deformation decoding features to obtain a fused feature map. .
[0042] The local residual generation unit uses the fused feature map As input, local amplitude correction coefficients corresponding to the spatial locations of the contact area are generated. These local amplitude correction coefficients are used to modulate the force vector amplitude at each spatial location in the contact force field point by point, thereby obtaining the local residual terms. The local residual term is mainly used to correct local biases in the initial contact force field estimation.
[0043] The global residual generation unit uses the fused feature map and global constraint features As input, global amplitude correction coefficients related to the overall load state are generated. These global amplitude correction coefficients are determined based on the global multi-directional force consistency condition and are used for overall amplitude compensation of the contact force field, thereby obtaining the global residual term. The global residual term is mainly used to correct the overall load deviation between the initial contact force field and the global multi-directional force information.
[0044] Adding the initial force field to the residual terms above, we obtain the final corrected force field: Then, training data is generated to train the neural network model within the framework.
[0045] A contact model of the sensor is established in finite element simulation software. Models of the external indenter, acrylic plate, and sensor film are created, and boundary conditions for each model are defined according to the actual assembly relationship. Finite element meshes are generated based on the geometry of each component: tetrahedral elements for the indenter, and hexahedral elements for the acrylic plate and film. Rigid properties are assigned to the indenter and acrylic plate models, while hyperelastic properties are assigned to the film. Translational and rotational information of the indenter are defined, and corresponding displacement and rotational boundary conditions are applied to bring the indenter model into contact with the film. Contact information such as deformation field and force field is calculated based on the finite element method. The extracted contact information is post-processed according to the output characteristics of the image preprocessing module.
[0046] The forward force field estimation model is trained using the following loss function: In the formula, For a node in the dataset, This represents the total number of nodes in the training dataset. For the corresponding data in the dataset The force vector.
[0047] The force field correction network is trained using the following objective function: In the formula, This represents the point-by-point mean square error loss between the corrected force field and the monitoring force field. , Represents the overall load loss. This represents a sparse regularization term used to suppress spurious responses in non-contact regions. This represents the weighting coefficient for the corresponding loss term.
[0048] Finally, the force field reconstruction method described above was actually implemented.
[0049] A prototype sensor was fabricated for this embodiment. The prototype includes a visual acquisition unit for acquiring contact images, a light source unit for generating stable illumination, an elastic sensing layer for generating contact deformation, and a multidimensional force sensing unit for acquiring global multi-directional force information. The elastic sensing layer and the multidimensional force sensing unit are connected by a support structure, allowing the overall load acting on the elastic sensing layer to be transferred to the multidimensional force sensing unit, simultaneously acquiring the contact image and corresponding global multi-directional force information. Based on the range of the multidimensional force sensing unit, the force field reconstruction method can effectively improve the measurable range of force field reconstruction for the visual-tactile sensor.
[0050] Based on the sensor prototype, contact images and global multi-directional force information are acquired in real time, and the contact force field information is reconstructed using the trained force field reconstruction method.
[0051] Therefore, compared with the prior art, the embodiments of the present invention have the following advantages: 1) This invention, based on estimating the contact force field using local deformation information, introduces global multi-directional force information acquired by a multi-dimensional force sensing unit to correct the initial contact force field. Since the global multi-directional force information can reflect the overall load state of the sensor, it can provide additional load constraint information for force field reconstruction when the contact load increases, local image deformation changes are not obvious, or tend to be nonlinear, thereby helping to expand the effective range of force field reconstruction of the visual-tactile sensor.
[0052] 2) This invention generates a corresponding force field correction value and obtains the corrected contact force field by inputting the initial contact force field, local deformation characteristics, and global multi-directional force information into the force field correction model. Compared with methods that rely solely on local deformation characteristics for force field estimation, this approach can continue to utilize global load information to participate in force field correction over a wider range of load variations, reducing the problem of limited force field reconstruction caused by insufficient changes in local deformation characteristics. It is particularly helpful in expanding the measurable range in the normal pressure direction.
[0053] 3) This invention can introduce the resultant force and / or resultant moment constraints obtained by force field integration during the training process, establishing a correspondence between the corrected contact force field and the global multi-directional force information. This constraint does not rely on a single local image feature, but utilizes the overall load information to participate in the model training and correction process, thus helping to improve the applicability of the force field reconstruction method under wide-range contact conditions.
[0054] Reference Figure 2 A tactile force field reconstruction system based on global multi-directional force information includes: The first module 201 is used to acquire contact images and corresponding global multi-directional force information during the process of the visual tactile sensor contacting the target object; The second module 202 is used to preprocess the contact image and extract the local deformation features of the sensor contact surface; The third module 203 is used to input the local deformation characteristics of the sensor contact surface into the forward force field estimation model to reconstruct the tactile force field and generate the initial contact force field. The fourth module 204 is used to fuse the initial contact force field, local deformation features and global multi-directional force information. Based on the force field correction model, the initial contact force field is corrected to obtain a corrected contact force field that satisfies global mechanical constraints, which serves as the reconstructed force field within the contact area of the visual-touch sensor.
[0055] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0056] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this is not intended to limit the scope of the embodiments of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the embodiments of the present application.
Claims
1. A method for reconstructing a tactile force field based on global multi-directional force information, characterized in that, Includes the following steps: Acquire contact images and corresponding global multi-directional force information during the process of visual-tactile sensors contacting target objects; The contact image is preprocessed to extract the local deformation features of the sensor contact surface; The local deformation characteristics of the sensor's contact surface are input into the forward force field estimation model to reconstruct the tactile force field and generate the initial contact force field. By integrating the initial contact force field, local deformation characteristics, and global multi-directional force information, and based on the force field correction model, the initial contact force field is corrected to obtain a corrected contact force field that satisfies global mechanical constraints, which serves as the reconstructed force field within the contact area of the visual-touch sensor.
2. The tactile force field reconstruction method based on global multi-directional force information according to claim 1, characterized in that, The local deformation features include tangential deformation features, normal deformation features, or a combination of both, wherein: The tangential deformation feature is extracted by calculating the optical flow change based on the temporal variation features of the RGB contact image; The normal deformation feature is obtained by solving a set of depth relationship equations constructed from the color-gradient mapping relationship of the RGB contact image.
3. The tactile force field reconstruction method based on global multi-directional force information according to claim 2, characterized in that, The forward force field estimation model is based on a multilayer perceptron and is used to construct the mapping relationship between the local deformation features of the sensor contact surface and the initial contact force field.
4. The tactile force field reconstruction method based on global multi-directional force information according to claim 3, characterized in that, The force field correction model includes a global information encoding module, a local feature encoding module, a conditional fusion module, a local feature decoding module, and a fusion output module. The global information encoding module is built on an MLP array, and the local feature encoding module and the local feature decoding module are both built on a U-Net network. The conditional fusion module is based on one or more of the feature-level linear modulation and attention mechanisms.
5. The tactile force field reconstruction method based on global multi-directional force information according to claim 4, characterized in that, The step of fusing the initial contact force field, local deformation features, and global multi-directional force information, and correcting the initial contact force field based on a force field correction model to obtain a corrected contact force field that satisfies global mechanical constraints, and using this as the reconstructed force field within the contact area of the visual-touch sensor, specifically includes: The global information encoding module based on the force field correction model performs encoding constraint processing on the global multi-directional force information to obtain global constraint features; Based on the local feature encoding module of the force field correction model, feature extraction and fusion are performed on the initial contact force field and local deformation features to obtain local spatial features; Based on the conditional fusion module of the force field correction model, global constraint features are fused into local spatial features to obtain fused features constrained by global multi-directional force information. Based on the local feature decoding module of the force field correction model, the fusion features constrained by global multi-directional force information are decoded to recover the force field correction features corresponding to the spatial resolution of the contact area. The fusion output module based on the force field correction model generates a force field correction amount according to the force field correction characteristics, and applies the force field correction amount to the initial contact force field to obtain a corrected contact force field that satisfies global mechanical constraints, which serves as the reconstructed force field within the contact area of the visual-touch sensor.
6. The tactile force field reconstruction method based on global multi-directional force information according to claim 5, characterized in that, The loss function of the force field correction model includes one or more of the following: point-by-point force field error term, resultant force constraint error term, resultant moment constraint error term, and non-contact region regularization term, so that the corrected contact force field output by the force field correction model has local distribution accuracy and global mechanical consistency.
7. The tactile force field reconstruction method based on global multi-directional force information according to claim 6, characterized in that, The training datasets for the forward force field estimation model and the force field correction model were obtained through finite element simulation. The training datasets include local deformation features, global multi-directional force information, and the corresponding supervised contact force field.
8. The tactile force field reconstruction method based on global multi-directional force information according to claim 7, characterized in that, The finite element simulation includes establishing a contact model of the indenter, rigid support, and elastic sensing layer. By changing the position, orientation, and load range of the indenter, deformation field, global load, and contact force field labels under different contact conditions are generated.
9. The tactile force field reconstruction method based on global multi-directional force information according to claim 8, characterized in that, The force field correction amount generated by the fusion output module includes a local distribution correction amount and a global amplitude correction amount. The local distribution correction amount corresponds to the force vector amplitude coefficient at each spatial location in the contact area, and is used to modulate the spatial distribution of the contact force field point by point to correct the local deviation of the initial contact force field estimation. The global amplitude correction amount corresponds to the compensation coefficient of the overall load state, and is used to compensate the load amplitude of the contact force field as a whole to correct the overall load deviation between the initial contact force field and the global multi-directional force information.
10. A tactile force field reconstruction system based on global multi-directional force information, characterized in that, Includes the following modules: The first module is used to acquire contact images and corresponding global multi-directional force information during the process of the visual-touch sensor contacting the target object; The second module is used to preprocess the contact image and extract the local deformation features of the sensor contact surface; The third module is used to input the local deformation characteristics of the sensor contact surface into the forward force field estimation model to reconstruct the tactile force field and generate the initial contact force field. The fourth module is used to fuse the initial contact force field, local deformation features and global multi-directional force information. Based on the force field correction model, the initial contact force field is corrected to obtain a corrected contact force field that satisfies global mechanical constraints, which serves as the reconstructed force field within the contact area of the visual-touch sensor.