Tactile webbing method for end-to-end hysteresis compensation

Abstract

The application discloses a kind of soft touch fabric force perception methods for end-to-end lag compensation, which comprises: collecting monocular RGB image sequence of soft robot in time interval and its synchronous pulse width modulation signal;Based on the monocular RGB image sequence, extract multi-modal visual feature representation containing pseudo-three-dimensional geometric information and deformation region distribution;Based on the pulse width modulation signal, the high-dimensional driving signal characteristics consistent with the multi-modal visual feature representation dimension are obtained by feature dimension increasing;Through cross-modal attention fusion mechanism, the multi-modal visual feature representation and the high-dimensional driving signal characteristics are fused to obtain unified semantic feature vector sequence;The unified semantic feature vector sequence is input into backtracking time series memory network for time series modeling and lag compensation, and the three-dimensional contact force prediction vector sequence of the soft robot is output.The application can realize stable prediction in dynamic scene.

Description

Technical Field

[0001] This invention relates to the field of force sensing technology, and in particular to a flexible tactile mesh force sensing method for end-to-end hysteresis compensation. Background Technology

[0002] With the rapid development of technologies such as artificial intelligence, computer vision, soft robotics, and intelligent manufacturing, flexible sensing and force perception technologies based on visual information have shown enormous application potential in fields such as medical-assisted surgery, precision and flexible manipulation, bionic tactile sensing, and human-computer interaction. In these applications, soft robots, with their inherent high compliance and continuous deformability, can achieve safe and compliant interaction and manipulation in unstructured environments. However, the significant nonlinearity, hyperelasticity, and viscoelasticity exhibited by soft materials lead to obvious time-delay effects and spatial nonlinearities in their response to external forces. This makes the accurate perception and modeling of contact forces one of the key bottlenecks restricting the development of this field.

[0003] Existing force sensing solutions for soft robots primarily rely on embedding physical sensors, such as strain gauges, capacitive sensors, or fiber optic sensors, within the robot body. While these methods can directly measure local strain or force signals, they generally suffer from several inherent limitations: First, the integration and wiring of sensors are complex, and the introduction of rigid sensing elements often compromises the overall compliance, biocompatibility, and mechanical durability of the soft structure. Second, current manufacturing and packaging technologies present significant challenges in deploying such sensor arrays in high-density, microscale, or non-transparent media. Finally, the complex dynamic coupling between sensor signals and the overall geometric deformation of the soft structure, coupled with inherent signal transmission delays, makes it difficult to achieve fast and accurate global force estimation under time-varying loads.

[0004] In recent years, with breakthroughs in deep learning and computer vision technologies, researchers have begun to explore label-free force perception methods based on monocular vision. This method constructs the force estimation problem as an image regression task, directly predicting the magnitude and direction of contact forces from RGB images through an end-to-end deep learning network, thus avoiding many drawbacks of embedded sensors. However, existing vision-based force estimation methods still have significant shortcomings: First, relying solely on two-dimensional RGB images as input lacks depth information perception, failing to accurately capture and describe the complex deflection and deformation of soft robots in three-dimensional space, resulting in incomplete spatial dimension prediction of force vectors and insufficient stability. Second, network architecture designs often employ separate or static feature extraction methods, failing to effectively model the dynamic coupling relationship between driving signals (such as air pressure and hydraulic pressure) and visual deformation, ignoring the crucial influence of driving input on the soft body's state. Third, existing models generally lack explicit modeling of temporal correlations, making it difficult to compensate for the force-displacement hysteresis characteristics of soft materials due to viscoelasticity. This leads to problems such as response delays, unstable output, or force estimation drift in prediction results when dealing with dynamic loading or rapid motion scenarios.

[0005] In summary, current force perception technology for soft robots faces a dual challenge: on the one hand, methods based on embedded sensors have limitations in system integration, compliance, and measurement dimensions; on the other hand, emerging monocular vision methods are still immature in 3D perception, multimodal information fusion, and dynamic temporal modeling. Specifically, existing deep learning frameworks generally employ discrete or phased processing flows when handling visual and actuation information, failing to establish a unified multimodal feature representation space. Furthermore, frame-independent prediction strategies neglect the temporal correlation and historical cumulative effects of soft body deformation, limiting the accuracy and robustness of the model in complex dynamic environments. Summary of the Invention

[0006] This invention aims to at least solve one of the technical problems existing in the prior art. To this end, this invention proposes a flexible tactile mesh force sensing method for end-to-end hysteresis compensation.

[0007] A flexible tactile mesh force sensing method for end-to-end hysteresis compensation according to an embodiment of the present invention includes the following steps: S1. Acquire the monocular RGB image sequence of the soft robot within the time interval and its synchronous pulse width modulation signal; S2. Based on the monocular RGB image sequence, extract multimodal visual feature representations containing pseudo-3D geometric information and deformation region distribution; S3. Based on the pulse width modulation signal, obtain high-dimensional driving signal features consistent with the dimension of the multimodal visual feature representation through feature dimensionality upscaling; S4. By using a cross-modal attention fusion mechanism, the multimodal visual feature representation and the high-dimensional driving signal feature are fused to obtain a unified semantic feature vector sequence; S5. Input the unified semantic feature vector sequence into the backtracking temporal memory network for temporal modeling and hysteresis compensation, and output the three-dimensional contact force prediction vector sequence of the soft robot.

[0008] In some embodiments of the present invention, the extraction of multimodal visual feature representation based on monocular RGB image sequences specifically includes: The monocular RGB image is input into a pre-trained depth estimation network to generate a corresponding pseudo depth map; The monocular RGB image is input into a pre-trained fully convolutional segmentation network to generate a segmentation mask image of the soft robot and the background; The RGB image, pseudo-depth map, and segmentation mask map at the same time are input into a shared feature extractor. After convolution, pooling, and feature alignment operations, a multimodal visual feature representation with a unified dimension is obtained.

[0009] In some embodiments of the present invention, the shared feature extractor is a conjoined network structure, comprising three parallel branches with shared weights.

[0010] In some embodiments of the present invention, obtaining high-dimensional driving signal features based on pulse width modulation signals specifically includes: The pulse width modulation signal is input into a driving signal feature enhancement network. This network adopts an encoder-decoder structure and uses multi-layer fully connected and nonlinear transformations to enhance the low-dimensional driving signal to a high-dimensional driving signal feature with the same dimension as the multimodal visual feature representation.

[0011] In some embodiments of the present invention, the driving signal feature enhancement network and the shared feature extractor are structurally isomorphic.

[0012] In some embodiments of the present invention, the feature fusion via a cross-modal attention fusion mechanism specifically includes: The high-dimensional driving signal features are used as query vectors, and the multimodal visual features are used as key vectors and value vectors. The cross-modal attention weight matrix is ​​calculated using the dot product attention mechanism; The value vector is weighted and summed using the attention weight matrix to output the fused unified semantic feature vector.

[0013] In some embodiments of the present invention, the backtracking temporal memory network is a recurrent neural network structure with an input gate, a forget gate, and an output gate, used to model the long-term dependencies in the unified semantic feature vector sequence in order to compensate for the lag response of the software system.

[0014] In some embodiments of the present invention, after outputting the three-dimensional contact force prediction vector sequence, a model training and evaluation step is further included: The mean square error loss is calculated by using the sequence of actual force measurements collected by the force sensor and the sequence of force vectors predicted by the network. By minimizing the loss function, the parameters of the shared feature extractor, the driving signal feature enhancement network, the cross-modal attention fusion mechanism, and the backtracking temporal memory network are jointly optimized.

[0015] Compared with existing technologies, the flexible tactile mesh force sensing method for end-to-end hysteresis compensation provided by this invention has the following beneficial effects: 1. By introducing a pre-trained Depth Estimation Network (DPT) and a Fully Convolutional Segmentation Network (FCN), this invention successfully enhances two-dimensional RGB images into a multimodal visual representation that includes pseudo-depth information and deformable region masks. This design fundamentally solves the problems of incomplete force vector space dimension and unstable prediction caused by the lack of depth perception in existing monocular vision methods, realizing a perceptual leap from two-dimensional pixel space to three-dimensional geometric space, and providing a reliable geometric foundation for high-precision force estimation.

[0016] 2. This invention constructs a driving signal feature enhancement network and a cross-modal attention fusion mechanism. This not only upscales the low-dimensional PWM driving signal to a high-dimensional semantic space aligned with visual features, but also dynamically establishes an explicit relationship between "driving-deformation-force" through the attention mechanism. This fusion enables the model to understand how the driving input leads to visual deformation and ultimately generates contact force, greatly enhancing the physical interpretability of features and the model's generalization ability, and solving the core challenges of semantic inconsistency and decoupling difficulties in multimodal information.

[0017] 3. This invention employs a backtracking temporal memory network (such as LSTM, GRU, etc.) to model continuous temporal features. This network can dynamically memorize and update historical states, automatically learn and compensate for the lag response of the software system. Therefore, when facing dynamic scenarios such as periodic loading and rapid movement, it can effectively overcome the prediction delay and force drift problems of traditional methods, and achieve smooth, stable and responsive continuous force estimation.

[0018] 4. This invention completely eliminates the need for embedded force sensors, requiring only a monocular camera and existing drive control signals to operate. This avoids complex sensor wiring and packaging processes, as well as damage to the flexibility, integrity, and durability of the soft structure. It truly achieves non-invasive and low-cost force sensing, greatly reducing system complexity and hardware costs, and more completely preserving the core advantages of soft robots.

[0019] 5. This invention provides a complete end-to-end solution, from multimodal information input to three-dimensional force vector output, all steps of which can be jointly optimized. This method does not rely on complex external global geometric prior models, simplifying system deployment. It offers a high-performance, high-reliability technical path for fields with extremely high requirements for force sensing accuracy, safety, and real-time performance, such as medical-assisted surgery, precision flexible manipulation, safe human-computer interaction, and bionic tactile sensing, and has broad prospects for industrial application. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the flexible woven mesh force sensing method for end-to-end hysteresis compensation according to the present invention. Figure 2 This is a schematic diagram of the overall structure of the present invention; Figure 3 This is a schematic diagram of the network structure in this invention; Figure 4 This is a schematic diagram of the data acquisition device and sample in this invention; Figure 5 A comparison chart of the force estimation performance of each core module of the present invention after ablation.

[0021] In the picture: 1. Force sensor; 2. Contact point plane; 3. Soft robot; 4. Motor; 5. Monocular camera; 6. Industrial computer. Detailed Implementation

[0022] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0023] The following is for reference. Figures 1-5 A flexible woven mesh force sensing method for end-to-end hysteresis compensation according to an embodiment of the present invention includes the following steps: S1. Acquire the monocular RGB image sequence of the soft robot within the time interval and its synchronous pulse width modulation signal; S2. Based on the monocular RGB image sequence, extract multimodal visual feature representations containing pseudo-3D geometric information and deformation region distribution; S3. Based on the pulse width modulation signal, obtain high-dimensional driving signal features consistent with the dimension of the multimodal visual feature representation through feature dimensionality upscaling; S4. By using a cross-modal attention fusion mechanism, the multimodal visual feature representation and the high-dimensional driving signal feature are fused to obtain a unified semantic feature vector sequence; S5. Input the unified semantic feature vector sequence into the backtracking temporal memory network for temporal modeling and hysteresis compensation, and output the three-dimensional contact force prediction vector sequence of the soft robot.

[0024] In some embodiments of the present invention, the extraction of multimodal visual feature representation based on monocular RGB image sequences specifically includes: The monocular RGB image is input into a pre-trained depth estimation network to generate a corresponding pseudo depth map; The monocular RGB image is input into a pre-trained fully convolutional segmentation network to generate a segmentation mask image of the soft robot and the background; The RGB image, pseudo-depth map, and segmentation mask map at the same time are input into a shared feature extractor. After convolution, pooling, and feature alignment operations, a multimodal visual feature representation with a unified dimension is obtained.

[0025] Specifically, using a monocular camera 5 within a time interval The RGB image sequence of the soft robot 3 under the working scene, which was continuously acquired internally, is denoted as For any time in the time series Get the corresponding frame RGB image And perform the following two parallel processing steps respectively: convert the corresponding frame RGB image Input a pre-trained lightweight depth estimation network DPT to generate a corresponding pseudo-depth map. This is used to compensate for the missing three-dimensional spatial information in RGB images; it converts the corresponding frame RGB image... Input the pre-trained fully convolutional segmentation network FCN to generate a segmentation mask image of the soft robot 3 and the background. This guides subsequent networks to focus on local deformations in the soft structure region and suppress background interference.

[0026] At the same time The three corresponding image data The inputs are shared into a monocular image multi-feature extraction module. This module employs a shared feature extractor with a connected network structure, consisting of three parallel branches with shared weights. Finally, after convolution, pooling, and feature alignment operations, a unified-dimensional multimodal visual feature representation is obtained. .

[0027] In some embodiments of the present invention, obtaining high-dimensional driving signal features based on pulse width modulation signals specifically includes: The pulse width modulation signal is input into a driving signal feature enhancement network. This network adopts an encoder-decoder structure and uses multi-layer fully connected and nonlinear transformations to enhance the low-dimensional driving signal to a high-dimensional driving signal feature with the same dimension as the multimodal visual feature representation.

[0028] Specifically, each drive motor of the soft robot 3 is driven by a pulse width modulation (PWM) signal output from the controller. This signal is synchronously acquired in time with the RGB image sequence captured by the monocular camera 5 in S1. For any given moment... , will input The vector is fed into a pre-trained driving signal feature enhancement network, which employs an encoder-decoder structure. Then, through multiple fully connected layers and nonlinear transformations, the vector is... The vector is upgraded from a low-dimensional feature space to a high-dimensional representation space to obtain a high-dimensional driving signal feature with the same dimension as the multimodal visual feature representation output by S1. , in, This represents the feature enhancement mapping function. Consistent with the dimensions of visual features.

[0029] To ensure consistency with the multimodal visual feature representation output in step S1, the driving signal feature enhancement network is structurally isomorphic to the shared feature extractor of the monocular image multi-feature extraction module in step S1.

[0030] In some embodiments of the present invention, the feature fusion via a cross-modal attention fusion mechanism specifically includes: The high-dimensional driving signal features are used as query vectors, and the multimodal visual features are used as key vectors and value vectors. The cross-modal attention weight matrix is ​​calculated using the dot product attention mechanism; The value vector is weighted and summed using the attention weight matrix to output the fused unified semantic feature vector.

[0031] Specifically, although the multimodal visual feature representation output in step S1 and the high-dimensional driving signal feature output in step S2 are aligned in the feature space, they still differ semantically. Therefore, a cross-modal attention fusion mechanism is introduced to achieve feature interaction and fusion: for any given time... ,Will as query vector ,Will as a key vector Sum value vector Input attention calculation unit; Calculate the cross-modal attention weight matrix using the dot product attention mechanism: , in, Key vector The feature dimensions.

[0032] Final output time range Internal fusion feature sequence Each of them Indicates at time The following is a unified semantic feature vector obtained by fusing visual features and driving features. In some embodiments of the present invention, the backtracking temporal memory network is a recurrent neural network structure with an input gate, a forget gate, and an output gate, used to model the long-term dependencies in the unified semantic feature vector sequence in order to compensate for the lag response of the software system.

[0033] Specifically, the fused feature sequences within a continuous time period can be... Input-backtracking temporal memory network. Through input gates, forget gates, and output gates, the backtracking temporal memory network effectively models long-term dependencies in time series, thereby enabling the memory and lag compensation of the dynamic process of soft deformation within the feature space.

[0034] For any time The network calculation is as follows: , in, Indicates the state of the memory cell. Indicates the current hidden state.

[0035] Hide the output state The input is fed into a fully connected regression layer, and the predicted three-dimensional contact force vector at that moment is obtained through linear mapping: , in, and These are the weight matrix and bias term of the regression layer, respectively.

[0036] Final output time range Three-dimensional contact force prediction vector sequence Each of them , indicating at time Three-dimensional force estimation results for the end effector of the lower soft robot.

[0037] In some embodiments of the present invention, after outputting the three-dimensional contact force prediction vector sequence, a model training and evaluation step is further included: The mean square error loss is calculated by using the sequence of actual force measurements collected by force sensor 1 and the sequence of force vectors predicted by the network. By minimizing the loss function, the parameters of the shared feature extractor, the driving signal feature enhancement network, the cross-modal attention fusion mechanism, and the backtracking temporal memory network are jointly optimized.

[0038] Specifically, the input is the three-dimensional contact force prediction vector sequence obtained in step S4. and the sequence of actual force measurements collected by force sensor 1 at the same timestamp To achieve supervised learning between the predicted results and the actual force values, this module uses the mean squared error (MSE) as the main loss function.

[0039] For each moment The loss function is defined as follows: , in, The number of training samples, This is the actual force value measured by the sensor. This represents the network's predictive power value.

[0040] By minimizing this loss function, the network parameters of the shared feature extractor in step S1, the driving signal feature enhancement network in step S2, the cross-modal attention fusion mechanism in step S3, and the backtracking temporal memory network in step S4 are jointly optimized.

[0041] The training strategy uses a large-scale sequence dataset consisting of monocular camera images, synchronous PWM signals, and stress sensors. The dataset is divided into training and test sets. During the training phase, the Adam optimizer is used to iteratively update the parameters, and a learning rate decay strategy is combined to improve convergence stability.

[0042] To comprehensively evaluate prediction performance, this module employs multiple error metrics. In addition to MSE, this invention also introduces root mean square error (RMSE): , Furthermore, due to differences in manufacturing methods and materials, soft robots exhibit significant differences in load-bearing capacity; therefore, the maximum contact force is introduced... The mean relative error (MRE) is as follows: , in, The value is the maximum contact force observed in the experiment, used for normalization.

[0043] By calculating the above indicators on the test set, the prediction accuracy and robustness of the network under different contact states and dynamic working conditions can be quantitatively evaluated. Finally, the end-to-end model parameters after training are output for real-time prediction of the three-dimensional contact force of the soft robot during the deployment phase. The evaluation indicators are used to verify the overall accuracy and stability of the system.

[0044] Example 1 This invention achieves high-precision three-dimensional force perception of a soft robot 3 in dynamic scenes by fusing monocular vision and actuation signal information. The method constructs a module that includes a monocular image multi-feature extraction module, an actuation signal feature enhancement network, a cross-modal attention fusion mechanism, and a backtracking temporal reasoning and force vector regression module. It can directly infer the spatiotemporal distribution of force from monocular image sequences and actuation signals without relying on an external force sensor 1.

[0045] The monocular image multi-feature extraction module constructed in this invention, combined with a pre-trained lightweight depth estimation network (DPT) and a pre-trained fully convolutional segmentation network (FCN), can perform multimodal analysis on monocular images, generating pseudo-depth maps and segmentation mask maps respectively. This compensates for the missing three-dimensional spatial information in two-dimensional vision and guides subsequent networks to focus on local deformations in soft structure regions while suppressing background interference. Furthermore, by using a shared feature extractor to uniformly encode multi-source image features, geometric consistency at the visual level is achieved, laying the foundation for subsequent cross-modal fusion.

[0046] Building upon this foundation, this invention constructs a driving signal feature enhancement network and a cross-modal attention fusion mechanism. By upscaling the pulse width modulation (PWM) signal, a semantic representation space of equal dimension to visual features is established. Furthermore, the cross-modal attention fusion mechanism establishes explicit semantic connections between the two modalities. This "web-like" fusion structure can simulate the driving-deformation relationship of a soft robot during force application, strengthening the spatial constraints of the driving signal on visual features. This ensures that the fused features not only contain deformation information but also possess the physical interpretability of mechanical constraints.

[0047] Furthermore, this invention introduces a backtracking temporal inference and force vector regression module at the temporal level, employing a backtracking temporal memory network to dynamically model and update the fused features at consecutive moments, thereby achieving automatic compensation for the hysteresis characteristics of soft materials. This structure can capture the temporal correlation between frames, enabling the system to maintain stable output even under periodic or abrupt loads, thus achieving continuous and smooth three-dimensional force estimation in dynamic stress environments.

[0048] This invention introduces mean squared error (MSE) as the main loss function during the training process, and can combine feature constraints and temporal consistency loss to achieve end-to-end optimization, thus completing self-supervised training under limited sample conditions.

[0049] Example 2 This invention evaluates the proposed three-dimensional force estimation method on a cable-driven soft robot and experimentally verifies the proposed network modules through a series of ablation experiments, evaluating the contribution of each module in the network.

[0050] like Figure 4As shown, the method proposed in this invention was tested on a soft robot experimental platform driven by four cables. The main body of the soft robot 3 is made of silicone material, and its tip can apply a maximum force of approximately 1N. Each of the four drive cables is controlled by an independent motor 4, which is driven by a pulse width modulation (PWM) signal. The cable length is actively changed by adjusting the duty cycle of the PWM signal. To obtain realistic tip interaction forces, the experimental platform is equipped with a three-dimensional force sensor with a measurement frequency exceeding 1000Hz to provide high-precision mechanical calibration data. Furthermore, a fixedly mounted OAK-D-Pro-W camera was used to visually capture the experimental process at a frame rate of 60Hz and a resolution of approximately 12 megapixels, ensuring that the interaction between the soft robot and the contact plane can be recorded in real time and with high quality.

[0051] During the data acquisition phase, four independent drive motors were driven to randomly generated target sampling positions at preset constant operating speeds. Visual and mechanical data were recorded continuously and synchronously during sampling. The sampling frequency was set to 5Hz. A total of 4000 motion data sequences were recorded, each containing dozens of monocular images along with time-aligned drive signals and 3D force sensor data. Then, pre-trained DPT and FCN networks were used to generate depth estimation and segmentation images. To further enrich the visual input, the acquired RGB images were input into the pre-trained DPT depth estimation network and FCN segmentation network to generate corresponding pseudo-depth maps and soft robot segmentation masks. Figure 4 (b) shows an example of our dataset, in which the entire dataset is divided in an 8:2 ratio for model training and independent validation, with 20% (i.e. 800 data sequences) set as a separate test set for performance evaluation and generalization verification of the method of this invention.

[0052] In the ablation experiments, four network architectures were constructed: SCAlexNet (single-channel visual baseline model), MCAlexNet (multi-channel visual features), MCAlexNet-PWM (multi-channel visual features and PWM signals are directly fused), and TMCAlexPWMNet (the final model incorporating time series modeling and PWM features). All models were trained for 300 epochs on 80% of the dataset using the same training strategy. The results are as follows: Figure 5 .

[0053] Experimental results show that although SCAlexNet is the baseline model, it achieves stable force estimation performance, verifying the feasibility of single-channel image features for flexible force perception tasks. MCAlexNet further improves estimation accuracy by introducing depth and segmentation features. In contrast, MCAlexNet-PWM suffers significant performance degradation due to directly concatenating modal features of different scales and statistical distributions, with a total MRE of 7.93%, proving that unaligned cross-modal fusion is unsuitable for this task. Furthermore, MCAlexPWMNet adopts a unified feature representation strategy, enabling matching of visual features and driving signal features in the same high-dimensional space, significantly reducing its total MRE to 1.10%. Finally, TMCAlexPWMNet, which combines time-series modeling—the complete network of the proposed method—achieves the best results, further reducing its total MRE to 0.84%, becoming the only network structure with an error below 1%, and exhibiting the most stable force estimation capability at the contact and disengagement moments of the soft robot.

[0054] In summary, this invention has the following advantages: strong end-to-end feature modeling and hysteresis compensation capabilities; through temporal networks and cross-modal attention mechanisms, it can simultaneously model the nonlinear time-delay relationship between visual deformation and driving response, achieving continuous dynamic force prediction; no embedded sensors required, high structural integrity; three-dimensional force estimation can be achieved solely with monocular vision and driving signals, avoiding the wiring complexity and flexibility damage problems caused by traditional force sensors; excellent cross-modal fusion expression capabilities; through feature dimensionality enhancement and attention guidance mechanisms, a unified representation of visual and driving features is achieved, significantly improving the physical interpretability and generalization performance of the model; strong temporal robustness and dynamic adaptability; the use of a backtracking temporal memory network to achieve historical state memory and dynamic updates, maintaining stable response under periodic loads, abrupt inputs, and multi-contact scenarios; significant engineering feasibility and low-cost advantages; high-precision force perception can be achieved solely with monocular imaging and driving control signals, facilitating its application in the medical, manufacturing, and human-computer interaction fields.

[0055] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A flexible tactile mesh force sensing method for end-to-end hysteresis compensation, characterized in that, Includes the following steps: S1. Acquire the monocular RGB image sequence of the soft robot within the time interval and its synchronous pulse width modulation signal; S2. Based on the monocular RGB image sequence, extract multimodal visual feature representations containing pseudo-3D geometric information and deformation region distribution; S3. Based on the pulse width modulation signal, obtain high-dimensional driving signal features consistent with the dimension of the multimodal visual feature representation through feature dimensionality upscaling; S4. By using a cross-modal attention fusion mechanism, the multimodal visual feature representation and the high-dimensional driving signal feature are fused to obtain a unified semantic feature vector sequence; S5. Input the unified semantic feature vector sequence into the backtracking temporal memory network for temporal modeling and hysteresis compensation, and output the three-dimensional contact force prediction vector sequence of the soft robot. Extracting multimodal visual feature representations based on monocular RGB image sequences, specifically including: The monocular RGB image is input into a pre-trained depth estimation network to generate a corresponding pseudo depth map; The monocular RGB image is input into a pre-trained fully convolutional segmentation network to generate a segmentation mask image of the soft robot and the background; The RGB image, pseudo-depth map and segmentation mask map at the same time are input into a shared feature extractor. After convolution, pooling and feature alignment operations, a multimodal visual feature representation with a unified dimension is obtained. The high-dimensional driving signal features are obtained based on the pulse width modulation signal, specifically: The pulse width modulation signal is input into a driving signal feature enhancement network. This network adopts an encoder-decoder structure and enhances the low-dimensional driving signal to a high-dimensional driving signal feature with the same dimension as the multimodal visual feature representation through multi-layer fully connected and nonlinear transformation. Feature fusion is performed through a cross-modal attention fusion mechanism, specifically including: The high-dimensional driving signal features are used as query vectors, and the multimodal visual features are used as key vectors and value vectors. The cross-modal attention weight matrix is ​​calculated using the dot product attention mechanism; The value vector is weighted and summed using the attention weight matrix to output the fused unified semantic feature vector.

2. The flexible woven mesh force sensing method for end-to-end hysteresis compensation according to claim 1, characterized in that, The shared feature extractor is a conjoined network structure, consisting of three parallel branches with shared weights.

3. The flexible tactile mesh force sensing method for end-to-end hysteresis compensation according to claim 2, characterized in that, The driving signal feature enhancement network and the shared feature extractor are structurally isomorphic.

4. The flexible tactile mesh force sensing method for end-to-end hysteresis compensation according to claim 3, characterized in that, The backtracking temporal memory network is a recurrent neural network structure with an input gate, a forget gate, and an output gate. It is used to model the long-term dependencies in the unified semantic feature vector sequence in order to compensate for the lag response of the software system.

5. The flexible tactile mesh force sensing method for end-to-end hysteresis compensation according to claim 4, characterized in that, After outputting the three-dimensional contact force prediction vector sequence, the process also includes model training and evaluation steps: The mean square error loss is calculated by using the sequence of actual force measurements collected by the force sensor and the sequence of force vectors predicted by the network. The parameters of the shared feature extractor, the driving signal feature enhancement network, the cross-modal attention fusion mechanism, and the backtracking temporal memory network are jointly optimized by minimizing the loss function.

Images