A fingertip bionic tactile sensor and a multimodal perception and three-dimensional force decoupling method thereof

Abstract

The application discloses a fingertip bionic tactile sensor and a multimodal sensing and three-dimensional force decoupling method thereof, and belongs to the technical field of flexible tactile sensing and intelligent information processing. The sensor adopts a multilevel skin simulation design and comprises, from top to bottom, a PDMS skin packaging layer, a sensing layer, a flexible printed circuit board and a PDMS flexible substrate. The sensing layer comprises four porous conductive composite force sensing units arranged in a 2*2 symmetrical mode and a temperature sensing chip located in a central region. The sensor can synchronously acquire pressure, friction force and temperature information in a structure with the size of a fingertip. Based on the sensor, the application further provides a three-dimensional force decoupling method. After four-channel resistance signals are preprocessed, the signals are input into a three-dimensional force decoupling model, and two orthogonal tangential force components and one normal force component are output. The application has the advantages of simple and compact structure, high integration and high decoupling precision, and is suitable for the fields of robot tactile sensing, intelligent grabbing, human-computer interaction and wearable devices.

Description

Technical Field

[0001] This invention belongs to the field of flexible tactile sensing and intelligent information processing technology, specifically relating to a fingertip bionic tactile sensor and its multimodal sensing and three-dimensional force decoupling method. Background Technology

[0002] Tactile perception is a crucial source of information for precise contact control in robotic manipulation, human-computer interaction, intelligent prosthetics, and wearable devices. Unlike non-contact sensing methods such as vision, tactile perception can directly reflect the mechanical state, material differences, and temperature changes during contact, making it significant in applications such as grasping stability assessment, slip detection, and object attribute recognition.

[0003] Existing tactile sensors still face many challenges in acquiring multidimensional tactile information. In practical tasks, the contact between a robot and an object is often accompanied by the simultaneous action of pressure and friction, which are highly coupled in the sensing signal. This makes it difficult to extract accurate pressure and friction information separately. This coupling effect severely restricts the robot's accurate judgment of slip state, grasping stability, and object properties.

[0004] Current solutions mostly enhance the separability of pressure and friction in response through sensor structure design, thereby reducing signal coupling to some extent. Building upon this, analytical methods using mathematical models are further employed to decouple pressure and friction. This approach achieves good results in scenarios with regular loading and small-scale measurements. However, its decoupling accuracy, generalization ability, and dynamic adaptability remain limited when the sensor material itself exhibits significant nonlinear response characteristics, or when normal pressure and tangential friction change simultaneously and superimpose.

[0005] Meanwhile, tactile interaction in intelligent systems is typically a complex process, and acquiring more comprehensive tactile information during this interaction is a significant challenge. Therefore, a tactile sensor capable of acquiring more tactile information during interaction and a more precise and generalizable three-dimensional force decoupling method are needed. Summary of the Invention

[0006] This invention addresses the problems existing in tactile sensing technology, such as insufficient acquisition of multi-dimensional tactile information, obvious nonlinear coupling between normal force and tangential friction, easy interference of temperature information with mechanical perception, and insufficient perception accuracy in complex interactive scenarios. It proposes a fingertip bionic tactile sensor and its multimodal perception and three-dimensional force decoupling method.

[0007] To solve the above technical problems, the present invention provides the following technical solution: a fingertip bionic tactile sensor, comprising, from top to bottom, an encapsulation layer, a force sensing layer, a digital temperature sensing chip, a flexible printed circuit board, and a flexible substrate;

[0008] The force sensing layer includes four force-sensitive units arranged symmetrically in a 2×2 pattern. The force-sensitive units are electrically connected to planar electrodes on the flexible printed circuit board via conductive silver paste. The flexible printed circuit board integrates planar electrodes, signal traces, and a circuit structure connected to the digital temperature sensing chip.

[0009] The encapsulation layer is used to transmit the normal pressure, tangential friction force and temperature information generated by external contact to the force sensing layer and the digital temperature sensing chip; the four force-sensitive units are used to output multi-channel resistance response signals during the contact process, and the digital temperature sensing chip is used to output temperature signals.

[0010] Furthermore, the aforementioned encapsulation layer and flexible substrate are made of polydimethylsiloxane, which is formed by mixing the main agent and the curing agent at a mass ratio of 10:1, followed by degassing and curing.

[0011] Furthermore, the aforementioned encapsulation layer adopts a differentiated thickness design, with the polydimethylsiloxane layer above the force-sensitive unit having a greater thickness than the polydimethylsiloxane layer above the digital temperature sensing chip, in order to balance mechanical thermal insulation performance and temperature response speed.

[0012] Furthermore, the aforementioned force-sensitive unit is a carbon black, multi-walled carbon nanotube, or polyurethane porous conductive polymer force-sensitive unit, which is prepared by the following method: carbon black, multi-walled carbon nanotube, and deionized water are mixed to form a dispersion, a polyurethane sponge is immersed in the dispersion to allow the conductive filler to adhere and penetrate into the porous framework, and then vacuum dried to form a porous conductive polymer.

[0013] Furthermore, the mass ratio of carbon black, multi-walled carbon nanotubes, and deionized water in the aforementioned dispersion is 5:2:100.

[0014] Furthermore, the aforementioned force-sensitive unit has dimensions of 6mm × 6mm × 2mm.

[0015] Furthermore, the aforementioned digital temperature sensing chip is a BMP280, which is integrated on the flexible printed circuit board and located at the center of the four force-sensitive units, for outputting temperature data through a digital interface.

[0016] The present invention also provides a multimodal sensing and three-dimensional force decoupling method, applied to the aforementioned fingertip bionic tactile sensor, the method being as follows: S1, acquiring the multi-channel resistance response signal output by the fingertip bionic tactile sensor;

[0017] S2. Construct a three-dimensional force decoupling model, which adopts a network structure combining FT-Transformer and cross-feature residual fusion module CFRF.

[0018] S3. Input the multi-channel resistance response signal from step 1 into the three-dimensional force decoupling model, establish the nonlinear mapping relationship between the multi-channel resistance signal and the three-dimensional contact force, and output the three-dimensional contact force prediction results, including the tangential force components Fx and Fy in two orthogonal directions and the normal force component Fz.

[0019] Furthermore, the aforementioned step S3 includes the following sub-steps:

[0020] S3.1. The four-channel resistance response signals of the sensor are used to form a multi-channel input vector. The input vector is sent to the feature embedding layer. The original features of each channel are converted into high-dimensional feature representations through linear mapping. The corresponding channel feature parameters are superimposed to form the feature tokens of each channel. At the same time, a learnable global aggregation token is introduced and together with the feature tokens of each channel, they constitute the input sequence of the Transformer encoder.

[0021] S3.2 The input sequence is fed into a Transformer encoder composed of multiple stacked coding units for feature extraction. Each coding unit includes a multi-head self-attention sub-layer, a feedforward neural network sub-layer, a residual connection structure and a layer normalization structure to extract high-order correlations between different sensing channels.

[0022] S3.3 Input the channel features encoded by Transformer into the cross-feature residual fusion module. By performing transpose, layer normalization, linear mapping, nonlinear activation, interactive fusion and residual superposition operations on the feature dimensions, information enhancement and fusion between different sensing channels are realized, and the ability to model the coupling relationship between normal pressure and tangential friction is improved while retaining the original feature expression.

[0023] S3.4 Perform global average pooling on the fused channel features, and obtain the fused feature vector through the feature projection layer. Then, superimpose the fused feature vector with the output feature corresponding to the global aggregation token to form a global feature for regression prediction.

[0024] S3.5 Input the global features into the regression prediction layer and output the tangential force components Fx and Fy and the normal force component Fz in two orthogonal directions.

[0025] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0026] 1. This invention adopts a layered structure design that mimics human skin, which is compact and highly integrated, and can achieve simultaneous acquisition of pressure, friction, and temperature information in a relatively small size.

[0027] 2. By using 2×2 symmetrically arranged porous carbon black / multi-walled carbon nanotubes / polyurethane (CB / MWCNTs / PU) conductive polymer force-sensitive units, the ability to identify three-dimensional forces is enhanced. Through the coordinated response of multiple units and combined with algorithms, the effective acquisition of three-dimensional contact force information is achieved.

[0028] 3. By using a differentiated dimethylsiloxane (PDMS) encapsulation layer thickness design and an independent digital temperature sensing chip, the problems of force-temperature signal crosstalk and structural redundancy that sensors often face are effectively avoided, thus improving the reliability and stability of the system in dynamic contact scenarios.

[0029] 4. The three-dimensional force decoupling method based on FT-Transformer and CFRF module proposed in this invention can autonomously learn the complex nonlinear mapping relationship between multi-channel sensing signals and three-dimensional contact forces, avoiding the problem of insufficient accuracy of traditional linear analytical models in highly coupled scenarios, thereby achieving high-precision prediction of normal force and two orthogonal tangential force components.

[0030] 5. The overall structure and size of the sensor are similar to those of a human fingertip, and it has good flexibility, integration and application scalability. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the overall appearance and structure of the present invention;

[0032] Figure 2 This is a schematic diagram of the structural layers of the present invention;

[0033] Figure 3 This is a schematic cross-sectional view of the skin encapsulation layer structure of the present invention;

[0034] Figure 4 This is a schematic diagram of the force sensing layer structure of the present invention;

[0035] Figure 5 This is a schematic diagram of the FPC structure of the present invention;

[0036] Figure 6 This is a schematic diagram illustrating the working principle of the present invention;

[0037] Figure 7 This is a response diagram of an embodiment of the present invention under normal pressure;

[0038] Figure 8 This is a response diagram of an embodiment of the present invention under frictional force;

[0039] Figure 9 This is a response diagram of an embodiment of the present invention under temperature excitation;

[0040] Figure 10 This is a structural diagram of the three-dimensional force decoupling model of the present invention;

[0041] Figure 11 The graph shows the regression analysis results of the Fx component prediction results of the decoupling method of the present invention.

[0042] Figure 12 The graph shows the regression analysis results of the Fy component prediction results of the decoupling method of the present invention.

[0043] Figure 13 The graph shows the regression analysis results of the Fz component prediction of the decoupling method of this invention.

[0044] In the figure: 1-Dimethylsiloxane PDMS skin encapsulation layer; 2-Force sensing layer; 3-Digital temperature sensing chip; 4-Flexible printed circuit board; 5-Dimethylsiloxane PDMS flexible substrate. Detailed Implementation

[0045] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0046] In this invention, various aspects of the invention are described with reference to the accompanying drawings, which illustrate numerous illustrative embodiments. Embodiments of the invention are not limited to those shown in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed in this invention are not limited to any particular implementation. In the description of this invention, "above," "below," and "within" are understood to include the stated number. Where "first" and "second" are used, they are for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of the indicated technical features. Furthermore, some aspects of the invention disclosed can be used alone or in any suitable combination with other aspects of the invention disclosed.

[0047] The present invention provides a fingertip bionic tactile sensor, the overall appearance of which is as follows: Figure 1 As shown, a multi-layered skin-like structure design is adopted, such as... Figure 2 As shown, it includes a polydimethylsiloxane (PDMS) skin encapsulation layer 1, a force sensing layer 2, a digital temperature sensing chip 3, a flexible printed circuit board 4, and a polydimethylsiloxane (PDMS) flexible substrate 5.

[0048] The dimethylsiloxane PDMS skin encapsulation layer 1 and the force sensing layer 2 are fixed together by silicone adhesive; the force sensing layer 2 is fixed to a pre-set planar electrode on the flexible printed circuit board 4 by conductive silver paste; the digital temperature sensing chip 3 is fixed to the flexible printed circuit board 4 by solder; the flexible printed circuit board 4 and the polydimethylsiloxane PDMS flexible substrate 5 are fixed together by silicone adhesive.

[0049] The polydimethylsiloxane (PDMS) skin encapsulation layer 1 is disposed on the outermost side to transfer external contact loads to the internal sensitive structure and protect the internal components. The PDMS is obtained by mixing the main agent and the curing agent in a 10:1 ratio, followed by vacuum degassing and curing.

[0050] like Figure 3 As shown, the polydimethylsiloxane (PDMS) skin encapsulation layer features a differentiated thickness design. The PDMS layer above the force sensing unit is 2mm thick, effectively preventing the conduction of the contact object's temperature to the force sensing unit, thereby obtaining a stable resistance response. The PDMS skin encapsulation layer above the digital temperature sensing chip is 0.2mm thick to shorten the heat conduction path and ensure the temperature response speed and accuracy of the sensing chip during contact.

[0051] like Figure 4 As shown, the force-sensing layer 2 is composed of porous carbon black, multi-walled carbon nanotubes, or polyurethane (CB / MWCNTs / PU) conductive polymer arranged in a 2×2 symmetrical pattern, and is labeled as force-sensitive unit 1, force-sensitive unit 2, force-sensitive unit 3, and force-sensitive unit 4, respectively. Among them, the one located in the upper left is force-sensitive unit 1, the one located in the lower left is force-sensitive unit 2, the one located in the lower right is force-sensitive unit 3, and the one located in the upper right is force-sensitive unit 4. The preparation process of the porous carbon black / multi-walled carbon nanotube / polyurethane (CB / MWCNTs / PU) conductive polymer includes: mixing CB, multi-walled carbon nanotubes (MWCNTs), and deionized water at a mass ratio of 5:2:100 and stirring for 15 minutes to form a uniform dispersion; completely immersing the PU sponge in the dispersion for 1 hour to allow the conductive filler to fully adhere and penetrate into the porous framework; then drying the impregnated PU sponge in a vacuum drying environment at 70°C for 3 hours; finally, cutting it into small pieces of 6mm×6mm as four independent force-sensitive units.

[0052] like Figure 5 As shown, the flexible printed circuit board 4 is equipped with a planar electrode array, a digital temperature sensing chip, and a driving circuit. The array size is 15mm×15mm, and the size of a single electrode is 2mm×6mm. The digital temperature sensing chip is a BMP280 with a size of 2mm×2.5mm×0.95mm, which is located at the center of the planar electrode array and is structurally relatively independent.

[0053] like Figure 6 As shown, under normal pressure, the porous structure inside the four force-sensitive units of the force-sensing layer 2 is compressed, increasing the number of contact points and conductive pathways between the conductive fillers, thus reducing the overall resistance. The four units exhibit an approximately uniform response trend. Based on this mechanism, the average relative resistance change of the four channels can be used as the metric. Characterizes the magnitude of normal pressure.

[0054] In this embodiment, a pressure of 0-5N is applied to the sensor using a digital push-pull gauge with a step size of 0.1N to evaluate the sensor's pressure sensitivity S. The pressure response performance is as follows: Figure 7 As shown, S represents pressure sensitivity, and R² represents the coefficient of determination for sensitivity fitting. This represents the rate of change of relative resistance. Indicates the initial resistance value. This represents the change in resistance. This represents the average relative resistance change of the output signals of the four force-sensitive units.

[0055] like Figure 6 As shown, under the action of tangential friction, the four force-sensitive units of the force-sensing layer 2 undergo direction-dependent shear deformation. For a given friction direction, the two force-sensitive units located at the far end of that direction exhibit larger response amplitudes, while the responses of the other two units are relatively smaller. Based on this mechanism, the average relative resistance change of the two far-end high-response units can be used. It represents the magnitude of tangential frictional force.

[0056] In this embodiment, a frictional force of 0-5N is applied along the x-direction of the sensor using a digital push-pull gauge, with a step size of 0.1N, to evaluate the sensor's frictional sensitivity GF. The frictional response performance is as follows: Figure 8 As shown, GF represents the friction sensitivity, and represents the applied tangential friction force. This represents the rate of change of relative resistance. Indicates the initial resistance value. This represents the change in resistance. This represents the average relative resistance change of the two highly responsive force-sensitive units at the far end of the frictional force direction. This represents the average relative resistance change of two low-response force-sensitive units under the action of friction.

[0057] Figure 9 The sensor outputs the temperature value of the BMP280 sensor chip when the contact temperature changes. In this embodiment, the sensor temperature output value is not affected by the normal pressure and friction during the contact process.

[0058] like Figure 10 As shown, in this embodiment, the three-dimensional force decoupling method adopts a network structure that combines FT-Transformer with cross-feature residual fusion module CFRF to learn the nonlinear mapping relationship between multi-channel resistance signals and three-dimensional contact forces.

[0059] In one specific embodiment of the present invention, the three-dimensional force decoupling method requires applying multiple sets of known three-dimensional forces to the sensor and recording the resistance output signal corresponding to each set of known three-dimensional forces as model input.

[0060] Specifically, the three-dimensional force decoupling model uses a four-dimensional vector formed by collecting the resistance response signals of the four force-sensitive units during the contact process as input.

[0061] Furthermore, the input vector is fed into the feature embedding layer, and the original features of each channel are converted into high-dimensional feature tokens through linear mapping;

[0062] Furthermore, the high-dimensional feature token and the learnable global aggregation token are input into the Transformer encoder, and the high-order correlation between different channels is extracted through the multi-head self-attention mechanism;

[0063] Furthermore, the encoded features are input into the cross-feature residual fusion module to further interact and fuse information between different sensing channels, thereby enhancing the model's ability to model nonlinear coupling relationships.

[0064] Furthermore, the fused global features are input into the regression head to output three-dimensional force prediction results, including two orthogonal tangential force components Fx and Fy, and a normal force component Fz.

[0065] The Transformer encoder consists of multiple stacked coding units, each including a multi-head self-attention sublayer and a feedforward neural network sublayer, employing a residual connection and layer normalization structure. Through the multi-head self-attention mechanism, the model can adaptively allocate weights based on the correlation between different input features, thereby more fully mining the implicit global correlation information in the four-channel tactile signals. Compared to traditional multilayer perceptrons or linear analytical methods, this method is more suitable for handling complex nonlinear coupling relationships in multi-channel signals.

[0066] The cross-feature residual fusion module enhances cross-channel information while preserving the original feature representation by performing transpose, layer normalization, linear mapping, nonlinear activation, interactive fusion, and residual superposition operations on the feature dimensions. This module further improves the model's ability to model multi-channel coupling modes, thereby enhancing the accuracy of 3D force decoupling and prediction stability.

[0067] In this embodiment, regression analysis was performed on the three-dimensional force prediction results, and the three-dimensional force component results are as follows: Figure 11 , 12 As shown in Figure 13, R² represents the coefficient of determination, and RMSE represents the root mean square error.

[0068] The predicted values ​​and the true values ​​show a highly linear relationship with a small root mean square error (RMSE), indicating that the proposed three-dimensional force decoupling method can accurately learn the mapping relationship between the sensing signal and the three-dimensional force, and the prediction results have high fitting degree and stability.

[0069] This three-dimensional force decoupling model can learn the complex nonlinear relationships related to normal pressure and tangential friction in four-channel signals, thereby achieving stable decoupling of three-dimensional contact forces. Compared with traditional linear analytical methods, this data-driven model has stronger characterization capabilities and higher prediction accuracy for coupled signals.

[0070] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. A method for decoupling multimodal sensing and three-dimensional force, characterized in that, The method is as follows: S1. Acquire the multi-channel resistance response signal output by the fingertip bionic tactile sensor; S2. Construct a three-dimensional force decoupling model, which adopts a network structure combining FT-Transformer and cross-feature residual fusion module CFRF. S3. Input the multi-channel resistance response signal from step S1 into the three-dimensional force decoupling model, establish the nonlinear mapping relationship between the multi-channel resistance signal and the three-dimensional contact force, and output the three-dimensional contact force prediction result, including the tangential force components Fx and Fy in two orthogonal directions and the normal force component Fz; step S3 includes the following sub-steps: S3.

1. The four-channel resistance response signals of the sensor are used to form a multi-channel input vector. The input vector is sent to the feature embedding layer. The original features of each channel are converted into high-dimensional feature representations through linear mapping. The corresponding channel feature parameters are superimposed to form the feature tokens of each channel. At the same time, a learnable global aggregation token is introduced and together with the feature tokens of each channel, they constitute the input sequence of the Transformer encoder. S3.2 The input sequence is fed into a Transformer encoder composed of multiple stacked coding units for feature extraction. Each coding unit includes a multi-head self-attention sub-layer, a feedforward neural network sub-layer, a residual connection structure and a layer normalization structure to extract high-order correlations between different sensing channels. S3.3 Input the channel features encoded by Transformer into the cross-feature residual fusion module. By performing transpose, layer normalization, linear mapping, nonlinear activation, interactive fusion and residual superposition operations on the feature dimensions, information enhancement and fusion between different sensing channels are realized, and the ability to model the coupling relationship between normal pressure and tangential friction is improved while retaining the original feature expression. S3.4 Perform global average pooling on the fused channel features, and obtain the fused feature vector through the feature projection layer. Then, superimpose the fused feature vector with the output feature corresponding to the global aggregation token to form a global feature for regression prediction. S3.5 Input the global features into the regression prediction layer and output the tangential force components Fx and Fy and the normal force component Fz in two orthogonal directions.

2. A fingertip bionic tactile sensor, applied to the multimodal sensing and three-dimensional force decoupling method described in claim 1, characterized in that, It includes, from top to bottom, the following layers: encapsulation layer, force sensing layer, digital temperature sensing chip, flexible printed circuit board, and flexible substrate; The force sensing layer includes four force-sensitive units arranged symmetrically in a 2×2 pattern. The force-sensitive units are electrically connected to planar electrodes on the flexible printed circuit board via conductive silver paste. The flexible printed circuit board integrates planar electrodes, signal traces, and a circuit structure connected to the digital temperature sensing chip. The encapsulation layer is used to transmit the normal pressure, tangential friction force and temperature information generated by external contact to the force sensing layer and the digital temperature sensing chip. The four force-sensitive units are used to output multi-channel resistance response signals during the contact process, and the digital temperature sensing chip is used to output temperature signals.

3. The fingertip bionic tactile sensor according to claim 2, characterized in that, The surfaces of the encapsulation layer and the flexible substrate are made of polydimethylsiloxane, which is formed by mixing the main agent and the curing agent at a mass ratio of 10:1, followed by degassing and curing.

4. The fingertip bionic tactile sensor according to claim 2, characterized in that, The encapsulation layer adopts a differentiated thickness design, with the polydimethylsiloxane layer above the force-sensitive unit having a greater thickness than the polydimethylsiloxane layer above the digital temperature sensing chip, in order to balance mechanical thermal insulation performance and temperature response speed.

5. The fingertip bionic tactile sensor according to claim 2, characterized in that, The force-sensitive unit is a carbon black, multi-walled carbon nanotube, or polyurethane porous conductive polymer force-sensitive unit, which is prepared by the following method: carbon black, multi-walled carbon nanotube, and deionized water are mixed to form a dispersion, a polyurethane sponge is immersed in the dispersion to allow the conductive filler to adhere and penetrate into the porous framework, and then vacuum dried to form a porous conductive polymer.

6. The fingertip bionic tactile sensor according to claim 5, characterized in that, The mass ratio of carbon black, multi-walled carbon nanotubes and deionized water in the dispersion is 5:2:

100.

7. The fingertip bionic tactile sensor according to claim 2, characterized in that, The force-sensitive unit has dimensions of 6mm × 6mm × 2mm.

8. The fingertip bionic tactile sensor according to claim 2, characterized in that, The digital temperature sensing chip is integrated on the flexible printed circuit board and located at the center of the four force-sensitive units, and is used to output temperature data through a digital interface.

Images