A composite high-frequency multi-modal tactile sensing system and signal acquisition method

Abstract

This invention provides a composite high-frequency multimodal tactile sensing system and signal acquisition method. The system includes a composite high-frequency multimodal tactile sensor, comprising a composite high-frequency multimodal sensing unit, an image acquisition unit, a programmable LED strip, and an integrated housing, a processing circuit, and a host computer. This sensing system can simultaneously acquire visual-tactile image signals and electrical signals. Through an interpolation algorithm, the results of the high-frequency electrical signals are interpolated and fused with the low-frequency visual-tactile image signals in the time domain, significantly improving the sampling rate of the visual-tactile image signals. This system possesses enhanced tactile perception capabilities and is suitable for high-frequency tactile signal monitoring and the capture of complex mechanical information. It exhibits superior performance in fields such as humanoid robot perception and precision manipulation, greatly expanding the system's application scenarios and reliability.

Description

Technical Field

[0001] This invention relates to the field of flexible sensor technology, specifically to a composite high-frequency multimodal tactile sensing system and its fabrication method. Background Technology

[0002] Touch is one of the five essential human senses, the most basic and widely used, and a crucial channel for perceiving and interacting with the external environment. Human skin contains various mechanoreceptors that respond to tactile stimuli across different frequency ranges. For example, Pacinian corpora are most sensitive to rapid vibrations, typically responding to frequencies from 40Hz to 500Hz; while Messler corpora are sensitive to low-frequency vibrations, responding to frequencies from 10Hz to 50Hz. This wide frequency response of these receptors allows humans to react accurately in various tactile scenarios (such as recognizing object textures and sensing external forces).

[0003] To better simulate human tactile perception, tactile sensors need to have high-frequency response capabilities, enabling them to capture tactile signals ranging from low to high frequencies. However, while existing tactile sensors (such as piezoresistive, piezoelectric, and capacitive sensors) excel in high-frequency response, they still have limitations in reconstructing and recognizing the texture of the surface of the object being touched. This makes these sensors perform poorly in applications requiring high-resolution perception, especially in the fine manipulation tasks of humanoid robots, where they cannot accurately reproduce surface details and rapidly changing tactile stimuli.

[0004] Vision-haptic sensors are a novel type of tactile sensor that combines computer vision and tactile perception to capture high-resolution image signals and process multimodal data through algorithms to achieve multimodal tactile perception. Vision-haptic sensors can not only sense the force distribution of objects but also perform detailed analysis of surface texture. However, current vision-haptic sensors still have limitations in capturing high-frequency signals, primarily due to the limitations of camera frame rates. Common miniature cameras have frame rates of 30fps or 60fps, and this low sampling rate cannot effectively capture high-frequency signals such as vibrations, limiting their performance in rapidly changing scenes. While existing high-speed cameras can achieve higher frame rates, their large size and high cost limit their application in the field of vision-haptic sensors.

[0005] A search revealed that patent application CN118443085A discloses a fully flexible multimodal high spatiotemporal resolution visual-tactile sensor. While this sensor can simultaneously acquire piezoelectric signals and visual-tactile image signals, with a piezoelectric signal sampling rate of at least 1000Hz, its visual-tactile sensing principle remains limited by the aforementioned limitations. The acquisition frequency of the visual-tactile image signal is constrained by the camera frame rate. Furthermore, the relationship between the piezoelectric signal and the visual-tactile image signal is unclear, making it impossible to acquire a high-frequency visual-tactile image signal based on the high-frequency piezoelectric signal. Therefore, this fully flexible multimodal high spatiotemporal resolution visual-tactile sensor can only acquire high-frequency piezoelectric signals and cannot acquire high-frequency visual-tactile image signals.

[0006] In conclusion, it is particularly urgent to invent a sensor that can perform both high-frequency sampling and surface reconstruction and recognition of contacted objects. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this invention is to provide a composite high-frequency multimodal tactile sensing system and its fabrication method.

[0008] According to a first aspect of the present invention, a composite high-frequency multimodal tactile sensing system is provided, the system comprising:

[0009] A composite high-frequency multimodal tactile sensor that simultaneously acquires visual-tactile image signals and electrical signals, wherein the sampling rate of the electrical signals is greater than the sampling rate of the visual-tactile image signals;

[0010] The processing circuit is used to transmit control signals, control the composite high-frequency multimodal tactile sensor to acquire visual tactile image signals, process and integrate the electrical signals, and transmit the integrated electrical signals from multiple channels at the same time to the host computer.

[0011] The host computer uses an interpolation algorithm to interpolate the high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain, and displays the interpolated high-frequency visual-tactile image signal. The interpolation algorithm can generate a corresponding visual-tactile image signal by training the visual-tactile image signal and the electrical signal at the overlapping time.

[0012] Optionally, the composite high-frequency multimodal tactile sensor includes:

[0013] The composite high-frequency multimodal sensing unit, through multi-layer composite stacking, generates electrical signal changes while deforming upon contact with the outside world. The electrical signal is then transmitted to the host computer after being processed by the processing circuit.

[0014] The image acquisition unit acquires the deformation information of the composite high-frequency multimodal sensing unit to obtain a visual-tactile image signal, and directly transmits the visual-tactile image signal to the host computer.

[0015] A programmable LED strip is connected to the processing circuit and provides a light source for the image acquisition unit under the control of the processing circuit.

[0016] An integrated housing assembles the composite high-frequency multimodal sensing unit, the image acquisition unit, and the programmable light strip into one unit.

[0017] Optionally, the composite high-frequency multimodal sensing unit includes, from bottom to top, a transparent rigid support layer, a transparent elastomer layer, a tracer particle layer, a multi-channel lower electrode layer, a multi-channel force-sensitive layer, a multi-channel upper electrode layer, and a reflective layer stacked sequentially.

[0018] Optionally, when the composite high-frequency multimodal sensing unit comes into contact with the outside world and undergoes deformation, the multichannel force-sensitive layer undergoes changes in electrical signals, including piezoelectric signals and piezoresistive signals.

[0019] Optionally, the number of channels in the multi-channel force-sensitive layer is no more than the number of tracer particles in the tracer particle layer, and each channel in the multi-channel force-sensitive layer is covered by at least one tracer particle.

[0020] Optionally, the multi-channel lower electrode layer, the multi-channel force-sensitive layer, and the multi-channel upper electrode layer are colorless and transparent, with a transmittance of not less than 80%. When the image acquisition unit takes a picture from below the composite high-frequency multimodal sensing unit, the acquired visual-tactile image signal has complete information on the morphological changes of the reflective layer and the positional changes of the tracer particles.

[0021] Optionally, the reflective layer is stretchable and opaque, so that when the image acquisition unit takes a picture from below the composite high-frequency multimodal sensing unit, the reflective layer can completely block the external light source.

[0022] Optionally, the host computer, wherein the interpolation algorithm specifically includes:

[0023] Let T be the sampling time for acquiring visual-tactile image signals. v For T v_1 T v_2 ... T v_n The sampling rate f of the visual-tactile image signal v 1 / (T) v_n -T v_n-1 ); Sampling time T for acquiring electrical signals e For T e_1 Te_2 ... T e_i-1 T e_i ... T e_2i-1 T e_2i ... T e_(n-1)i-1 T e_(n-1)i ... T e_ni-1 T e_ni The sampling rate f of the electrical signal e 1 / (T) e_ni -T e_ni-1 );f e Greater than f v ;

[0024] The sampling time of the composite high-frequency multimodal tactile sensing system is T. e Interpolation to T v The obtained signal is obtained by interpolation, which adds i-1 interpolated visual-tactile image signals between the low-frequency visual-tactile image signals. When the composite high-frequency multimodal tactile sensor is subjected to high-frequency tactile stimulation, it can retain the sampling time T. v The system captures visual-tactile image signals and also obtains high-frequency electrical signals. More visual-tactile image signals can be obtained through time-domain interpolation.

[0025] According to a second aspect of the present invention, a method for acquiring high-frequency visual-tactile image signals is provided, comprising:

[0026] When the composite high-frequency multimodal sensing unit comes into contact with the outside world and undergoes deformation, it simultaneously acquires visual-tactile image signals and electrical signals, wherein the sampling rate of the electrical signals is greater than the sampling rate of the visual-tactile image signals.

[0027] The acquisition of visual and tactile image signals is controlled, and the electrical signals are processed and integrated to integrate the electrical signals from multiple channels at the same time.

[0028] An interpolation algorithm is used to interpolate the integrated high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain to obtain the interpolated high-frequency visual-tactile image signal; wherein, the interpolation algorithm can generate a corresponding visual-tactile image signal from the electrical signal by training the visual-tactile image signal and the electrical signal at the overlapping time.

[0029] Optionally, the method specifically includes:

[0030] A composite high-frequency multimodal sensing unit is used in each T e The electrical signal is obtained at the sampling time point, processed by the processing circuit, and then transmitted to the host computer. The sampling rate f of the electrical signal is... e ;

[0031] Image acquisition units are used in each T v The visual-tactile image signals are acquired at sampling time points and sent to the host computer, where the sampling rate f of the visual-tactile image signals is... v ;f e Greater than f v ;

[0032] Based on the interpolation algorithm and the electrical signals in the host computer, each T is obtained. e If the visual-tactile image signal at the sampling time point exists, T v Sampling time point and T e If the sampling time points coincide, then the sampling time point T is used. v The visual-tactile image signal obtained by the image acquisition unit shall be the standard.

[0033] Through the above steps, the sampling frequency of the visual-tactile image signal is increased from f v Improved to f through time-domain interpolation e The high-frequency visual and tactile image signals are then displayed on the host computer.

[0034] Optionally, the method further includes, before acquiring the visual-tactile image signal and the electrical signal:

[0035] (i) Perform mechanical pre-calibration on the composite high-frequency multimodal sensing unit, including:

[0036] Multiple known external forces are applied to the composite high-frequency multimodal sensing unit, while simultaneously acquiring electrical signals and visual-tactile image signals;

[0037] For the electrical signal, multiple tests were conducted and the average value was taken. The relationship curve between the electrical output response and multiple known forces was plotted. e ;

[0038] For visual-tactile image signals, a visual-tactile algorithm is used to track the displacement field changes of tracer particles under different external forces. Multiple tests are conducted, and the average value is taken to establish a curve L showing the relationship between the average displacement field change of the tracer particles and multiple known forces. v ;

[0039] (ii) Continuously apply forces of different directions and magnitudes to the composite high-frequency multimodal sensing unit, and then use the image acquisition unit and processing circuit to process each T... v and T e The visual-tactile image signals and electrical signals at the sampling time points are transmitted to the host computer; in the host computer, T... v Sampling time point and T eThe visual-tactile image signal and electrical signal at the overlapping sampling time points are used as the training set for pre-training of the interpolation algorithm. By training the visual-tactile image signal and electrical signal at the overlapping time points, it is expected to achieve the effect of generating the corresponding visual-tactile image signal from the electrical signal.

[0040] The pre-trained interpolation algorithm described above is used for subsequent formal signal interpolation processing.

[0041] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0042] 1. This invention integrates the sensing principles of visual-tactile sensors and electrical-tactile sensors, enabling simultaneous processing and analysis of data from different sensing mechanisms. This multimodal integration of visual-tactile and electrical-tactile sensing allows the system to not only acquire electrical signals but also obtain visual-tactile image signals for precise reconstruction and recognition of object surfaces. Therefore, this invention provides richer and more accurate perceptual information than traditional single-sensor systems.

[0043] 2. This invention, through an innovative time-domain interpolation method, can effectively improve the signal acquisition speed and signal reception frequency of visual-touch sensors.

[0044] 3. This invention uses a composite high-frequency multimodal sensing unit, which enables the sensor to perform time-domain interpolation on low-frequency visual-tactile image signals, thereby improving the sampling rate of visual-tactile image signals and overcoming the shortcomings of low sampling rate in traditional visual-tactile sensors.

[0045] 4. This invention, through a composite high-frequency multimodal sensing unit and time-domain interpolation, enables the system to accurately capture and quickly respond to subtle mechanical changes and high-frequency tactile stimuli, thereby improving the performance of the visual-tactile sensor in application scenarios requiring high-frequency monitoring. Attached Figure Description

[0046] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0047] Figure 1 This is a schematic diagram of the structure of a composite high-frequency multimodal tactile sensing system in one embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the structure of a composite high-frequency multimodal sensing unit in one embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram illustrating the principle of temporal interpolation of visual-tactile image signals by a composite high-frequency multimodal tactile sensing system in one embodiment of the present invention.

[0050] Figure 4This is a sensing flowchart of a composite high-frequency multimodal tactile sensing system according to an embodiment of the present invention;

[0051] The labels in the diagram represent: 1- Composite high-frequency multimodal sensing unit, 2- Programmable LED strip, 3- Integrated housing, 4- Image acquisition unit, 5- Composite high-frequency multimodal tactile sensor, 6- Processing circuit, 7- Host computer, 8- Transparent rigid support layer, 9- Transparent elastomer layer, 10- Tracer particle layer, 11- Multi-channel lower electrode layer, 12- Multi-channel force-sensitive layer, 13- Multi-channel upper electrode layer, 14- Reflective layer. Detailed Implementation

[0052] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0053] Reference Figure 1 As shown, a preferred embodiment of the present invention provides a composite high-frequency multimodal tactile sensing system, comprising a composite high-frequency multimodal tactile sensor 5, a processing circuit 6, and a host computer 7. The composite high-frequency multimodal tactile sensor 5 simultaneously acquires visual-tactile image signals and electrical signals when it comes into contact with the outside world and undergoes deformation. The sampling rate of the electrical signals is greater than that of the visual-tactile image signals. The processing circuit 6 transmits control signals to control the composite high-frequency multimodal tactile sensor 5 to acquire visual-tactile image signals and processes and integrates the electrical signals, combining the electrical signals from multiple channels at the same time and transmitting them to the host computer 7. The host computer 7 uses an interpolation algorithm to interpolate the high-frequency electrical signals into the low-frequency visual-tactile image signals in the time domain and displays the interpolated high-frequency visual-tactile image signals. The interpolation algorithm in the host computer 7, by training the visual-tactile image signals and electrical signals at overlapping times, can generate corresponding visual-tactile image signals from the electrical signals.

[0054] The aforementioned high-frequency and low-frequency terms are relative to the sampling rates of electrical signals and visual-tactile image signals. For example, the sampling time T for acquiring visual-tactile image signals... v For T v_1 T v_2 ... T v_n Therefore, the sampling rate f of the visual-tactile image signal v 1 / (T) v_n -T v_n-1 (Common visual-tactile image signals have a sampling rate of 30fps or 60fps); the sampling time T for acquiring electrical signals. e For T e_1T e_2 ... T e_i-1 T e_i ... T e_2i-1 T e_2i ... T e_(n-1)i-1 T e_(n-1)i ... T e_ni-1 T e_ni Therefore, the sampling rate f of the electrical signal e 1 / (T) e_ni -T e_ni-1 );f e Greater than f v .

[0055] In the above embodiments of the present invention, the system can simultaneously acquire high-frequency electrical signals and low-frequency visual-tactile image signals, and combine them with an interpolation algorithm to generate new visual-tactile image signals, thereby improving the sampling rate of visual-tactile image signals.

[0056] Reference Figure 1 As shown, the composite high-frequency multimodal tactile sensor 5 includes a composite high-frequency multimodal sensing unit 1, an image acquisition unit 4, a programmable LED strip 2, and an integrated housing 3. The composite high-frequency multimodal sensing unit 1, through multi-layered composite stacking, undergoes deformation upon contact with the external environment, resulting in changes in electrical signals. These electrical signals are acquired, processed by the circuitry, and then transmitted to the host computer 7. The image acquisition unit 4 acquires the deformation information of the composite high-frequency multimodal sensing unit 1, obtaining a visual-tactile image signal, which is directly transmitted to the host computer 7. The programmable LED strip 2 is connected to the processing circuit 6, providing a light source for the image acquisition unit 4 under the control of the processing circuit 6. The integrated housing 3 assembles the composite high-frequency multimodal sensing unit 1, the image acquisition unit 4, and the programmable LED strip 2 into a single unit, thus forming a composite high-frequency multimodal tactile sensor 5.

[0057] In some preferred embodiments, in the above-mentioned composite high-frequency multimodal tactile sensor 5, the integrated housing 3 serves to support and fix the other components of the composite high-frequency multimodal tactile sensor 5; the composite high-frequency multimodal sensing unit 1 is placed on top of the integrated housing 3, and is used to generate deformation through contact with the outside world, while generating high-frequency electrical signals, which are processed by the processing circuit 6 and transmitted to the host computer 7; the programmable light strip 2 can be placed around the composite high-frequency multimodal sensing unit 1 to provide a light source, and is connected to the processing circuit 6 and controlled by the processing circuit 6.

[0058] Reference Figure 1 and Figure 2As shown in some preferred embodiments, the image acquisition unit 4 is placed directly below the composite high-frequency multimodal sensing unit 1 to acquire visual-tactile image signals when the composite high-frequency multimodal sensing unit 1 comes into contact with the outside world and deforms. It is directly connected to the host computer 7 to transmit the signals to the host computer 7. In addition, in order to ensure the quality of the visual-tactile image signals, the integrated housing 3 is made of black opaque material, and the top reflective layer 14 of the composite high-frequency multimodal sensing unit 1 is also opaque, ensuring that the internal light source of the entire composite high-frequency multimodal tactile sensor 5 is provided only by the programmable light strip 2.

[0059] In practical implementation, the composite high-frequency multimodal sensing unit 1 comes into contact with the outside world, and while deforming, the electrical signal changes. This electrical signal is transmitted to the host computer 7 via the processing circuit 6. At the same time, the image acquisition unit 4 acquires the deformation information of the composite high-frequency multimodal sensing unit 1 and directly transmits the visual tactile image signal to the host computer 7. During this process, the programmable light strip provides a stable white light source for the entire composite high-frequency multimodal tactile sensor 5.

[0060] In one embodiment, the host computer of the composite high-frequency multimodal tactile sensing system may further include: a training module for training the interpolation algorithm; the training module uses forces of different directions and magnitudes to touch the composite high-frequency multimodal sensing unit, and processes the data through an image acquisition unit and a processing circuit. v and T e The visual-tactile image signals and electrical signals at the sampling time points are transmitted to the host computer; in the host computer, T... v Sampling time point and T e The visual-tactile image signal and electrical signal at the overlapping sampling time points are used as the training set for pre-training of the interpolation algorithm. By training the visual-tactile image signal and electrical signal at the overlapping time points, the algorithm can generate the corresponding visual-tactile image signal from the electrical signal.

[0061] Reference Figure 3 As shown, when the host computer 7 receives samples with a sampling rate of f v The visual-tactile image signal and the sampling rate are f e When processing electrical signals, an interpolation algorithm is used to generate more visual-tactile image signals between two adjacent frames of sampled visual-tactile image signals. If a visual-tactile image signal acquired by image acquisition unit 4 exists at the time point corresponding to the generated visual-tactile image signal, then the visual-tactile image signal obtained by image acquisition unit 4 at that sampling time point is used as the standard. Finally, the visual-tactile image signals generated by the trained interpolation algorithm are mapped one-to-one with each electrical sampling point, increasing the sampling rate of the visual-tactile image signals from f... v Upgraded to f eThis also ensures the continuity of the generated visual-tactile image signal in the time domain, resulting in a high-frequency visual-tactile image signal that is continuous in the time domain.

[0062] In the above embodiments of the present invention, the sampling time of the composite high-frequency multimodal tactile sensing system is T e Interpolation to T v Therefore, through interpolation, i-1 interpolated visual-tactile image signals will be added between low-frequency visual-tactile image signals. When the composite high-frequency multimodal tactile sensor 5 is subjected to high-frequency tactile stimulation, it can not only retain the sampling time T v In addition to the captured visual-tactile image signals, high-frequency electrical signals can also be obtained. Through time-domain interpolation, more visual-tactile image signals can be obtained, thereby obtaining more detailed multimodal tactile information such as changes in the shape and depth of the contact object.

[0063] In other preferred embodiments, refer to Figure 1 and Figure 2 As shown, the composite high-frequency multimodal sensing unit 1 is stacked from bottom to top as follows: a transparent rigid support layer 8, a transparent elastomer layer 9, a tracer particle layer 10, a multi-channel lower electrode layer 11, a multi-channel force-sensitive layer 12, a multi-channel upper electrode layer 13, and a reflective layer 14.

[0064] When the multi-channel force-sensitive layer 12 deforms upon contact with the external environment, it generates changes in electrical signals, including piezoelectric and piezoresistive signals. The location of these electrical signals corresponds to the location of the tracer particles in the visual-tactile signal. In subsequent interpolation algorithms, this correspondence is used to establish a connection, enabling better prediction of new visual-tactile image signals.

[0065] In other preferred embodiments, the number of channels in the multi-channel force-sensitive layer 12 is no more than the number of tracer particles in the tracer particle layer 10. Each channel in the multi-channel force-sensitive layer 12 is covered with at least one tracer particle to ensure that the relationship between the visual-tactile image signal and the electrical signal is more intuitive, thereby better generating a new visual-tactile image signal by interpolating the electrical signal in the time domain.

[0066] In other preferred embodiments, the transparent rigid support layer 8 is made of transparent acrylic sheet material cut from thin; the transparent elastomer layer 9 is made of polydimethylsiloxane (PDMS); tracer particles are printed on the surface of the transparent elastomer layer 9 by dispensing using a microelectronic printer to form a tracer particle layer 10; optionally, the tracer particles are made of silicone doped with red pigment.

[0067] In other preferred embodiments, piezoelectric materials are selected as the sensing materials for the multi-channel force-sensitive layer 12. Preferably, a transparent and flexible polyvinylidene fluoride (PVDF) film is used as the material for the multi-channel force-sensitive layer 12.

[0068] In other preferred embodiments, the multi-channel lower electrode layer 11, the multi-channel force-sensitive layer 12, and the multi-channel upper electrode layer 13 are colorless and transparent, with a transmittance of not less than 80%. When the image acquisition unit 4 captures images from below the composite high-frequency multimodal sensing unit 1, the acquired visual-tactile image signal has complete information on the morphological changes of the reflective layer and the positional changes of the tracer particles. In a specific embodiment, a structure with a sensitive material located between the upper and lower electrodes can be used to achieve the acquisition and transmission of electrical signals. Preferably, transparent indium tin oxide (ITO) with good conductivity is used as the electrode material. The ITO electrode is attached to the PDMS film, and the electrode is patterned by laser cutting. Depending on the cutting pattern, it serves as the multi-channel lower electrode layer 11 and the multi-channel upper electrode layer 13, respectively. Finally, the multi-channel lower electrode layer 11, the multi-channel force-sensitive layer 12, and the multi-channel upper electrode layer 13 are pasted together in sequence and then attached above the tracer particle layer 10.

[0069] In other preferred embodiments, the reflective layer 14 is stretchable and opaque, so that when the image acquisition unit 4 takes a picture from below the composite high-frequency multimodal sensing unit 1, the reflective layer 14 can completely block the external light source. In a specific embodiment, the raw material of the reflective layer 14 can be PDMS doped with silver powder, and a layer of reflective layer 14 is spin-coated by a spin coating process to achieve high-precision texture perception.

[0070] In other preferred embodiments, the programmable light strip 2 is a WS2812B light strip. Optionally, in order to further improve the quality of the visual and tactile image signal and improve the stability of the light source, a light guide plate is added between the programmable light strip 2 and the composite high-frequency multimodal sensing unit 1 to make the light uniformly distributed and improve the stability of the light source.

[0071] In other preferred embodiments, refer to Figure 1 As shown, the integrated shell 3 is made by 3D printing, using resin as the material to achieve an overall black and opaque effect.

[0072] In other preferred embodiments, refer to Figure 1As shown, the image acquisition unit 4 uses an HBVCAM-F2209HDV11 camera, which can capture images at a rate of 30fps. It is connected to the host computer 7 via a USB interface to realize the transmission of visual and tactile image signals.

[0073] In other preferred embodiments, the processing circuit 6 includes a multi-channel operational amplifier, an analog-to-digital converter, a central processing unit, and a signal transmission module. (Refer to...) Figure 1 As shown, the processing circuit 6 is fabricated using PCB technology based on the schematic diagram of the acquisition and control circuit. The multi-channel operational amplifier required for the piezoelectric signal acquisition circuit uses the MCP6002 chip. The analog-to-digital converter, central processing unit, and signal transmission module are provided by the ESP32-PICO-KIT development board, which ultimately realizes the acquisition, transmission, and control of electrical signals and the programmable light strip 2.

[0074] The system constructed using the composite high-frequency multimodal tactile sensor 5 in the above embodiments of the present invention can simultaneously acquire high-frequency electrical signals and low-frequency visual-tactile image signals in an integrated manner, and can also acquire high-frequency visual-tactile image signals through high-frequency electrical signals and interpolation methods. By interpolating and fusing the results of the high-frequency electrical signals with the low-frequency visual-tactile image signals in the time domain using an interpolation algorithm, the sampling rate of the visual-tactile image signals is significantly improved. This system has higher tactile perception capabilities, is suitable for high-frequency tactile signal monitoring and the capture of complex mechanical information, and exhibits superior performance in fields such as humanoid robot perception and precision operation, greatly expanding the system's application scenarios and reliability.

[0075] Based on the same technical concept, another embodiment of the present invention also provides a method for acquiring high-frequency visual-tactile image signals, comprising:

[0076] S101, when the composite high-frequency multimodal sensing unit 1 comes into contact with the outside world and undergoes deformation, it simultaneously acquires visual-tactile image signals and electrical signals, wherein the sampling rate of the electrical signals is greater than the sampling rate of the visual-tactile image signals.

[0077] S102, processes the visual-tactile image signal and the electrical signal to meet the interpolation requirements;

[0078] S103 uses an interpolation algorithm to interpolate the processed high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain to obtain the interpolated high-frequency visual-tactile image signal.

[0079] In a preferred embodiment, the method for acquiring the above-mentioned high-frequency visual-tactile image signal can be further represented by the following steps:

[0080] S201 employs a composite high-frequency multimodal sensing unit 1 in each T eThe electrical signal is obtained at the sampling time point, processed by the processing circuit 6, and then transmitted to the host computer 7. The sampling rate f of the electrical signal is... e ;

[0081] S202, using image acquisition unit 4 at each T v The visual-tactile image signals acquired at sampling time points are sent to the host computer 7, where the sampling rate f of the visual-tactile image signals is... v ;f e Greater than f v ;

[0082] S203, based on the interpolation algorithm and the electrical signals in the host computer 7, obtain each T e If the visual-tactile image signal at the sampling time point exists, T v Sampling time point and T e If the sampling time points coincide, then the sampling time point T is used. v The visual-tactile image signal obtained by the image acquisition unit 4 shall be the standard.

[0083] Through the above steps, the sampling frequency of the visual-tactile image signal is increased from f v Improved to f through time-domain interpolation e And finally, the high-frequency visual and tactile image signals are displayed on the host computer.

[0084] In a preferred embodiment, to better obtain the aforementioned high-frequency visual-tactile image signal, the high-frequency visual-tactile image signal acquisition method may further include mechanical pre-calibration of the composite high-frequency multimodal sensing unit 1 before starting. Specifically, the mechanical pre-calibration process for the composite high-frequency multimodal sensing unit 1 includes: applying multiple known external forces to the composite high-frequency multimodal sensing unit 1, while simultaneously acquiring electrical signals and visual-tactile image signals; for the electrical signals, performing multiple tests and taking the average value, and plotting the relationship curve L between the electrical output response and the multiple known applied forces. e For visual-tactile image signals, visual-tactile algorithms, such as optical flow, are used to track the displacement field changes of tracer particles under different external forces. Multiple tests are conducted, and the average value is taken to establish the relationship curve L between the average displacement field change of the tracer particles and multiple known forces. v .

[0085] Following the aforementioned mechanical pre-calibration process, in order to better complete the interpolation operation, in a preferred embodiment, it further includes: continuously touching the composite high-frequency multimodal sensing unit 1 with forces of different directions and magnitudes, and then using the image acquisition unit 4 and processing circuit 6 to process each T... v and T e The visual-tactile image signal and electrical signal at the sampling time point are transmitted to the host computer 7; in the host computer 7, T... vSampling time point and T e The visual-tactile image signal and electrical signal at the overlapping sampling time points are used as the training set for pre-training of the interpolation algorithm. By training the visual-tactile image signal and electrical signal at the overlapping time points, it is expected to achieve the effect of generating the corresponding visual-tactile image signal from the electrical signal.

[0086] After the above interpolation algorithm is pre-trained, the electrical signal and visual-tactile image signal are formally acquired and interpolated (using the pre-trained interpolation algorithm).

[0087] In another preferred embodiment, the aforementioned composite high-frequency multimodal tactile sensing system is used to acquire high-frequency visual-tactile image signals, referring to... Figure 4 As shown, the specific process includes:

[0088] S1. Before the test, a mechanical pre-calibration is performed on the composite high-frequency multimodal sensing unit 1. Multiple known external forces are applied to the composite high-frequency multimodal sensing unit 1, and electrical signals and visual-tactile image signals are acquired simultaneously. For the electrical signals, the average value is taken after multiple tests, and the relationship curve L between the electrical output response and the multiple known applied forces is plotted. e For visual-tactile image signals, preferably, the displacement field changes of tracer particles caused by different external forces are tracked using optical flow method. Multiple tests are conducted, and the average value is taken to establish the relationship curve L between the average displacement field change of the tracer particles and multiple known forces. v ;

[0089] S2. Before the test begins, the composite high-frequency multimodal sensing unit 1 is pressed repeatedly with forces of different directions and magnitudes. The image acquisition unit 4 and processing circuit 6 then process the data for each T... v and T e The visual-tactile image signals and electrical signals at the sampling time points are transmitted to the host computer 7; in the host computer 7, the visual-tactile image signals and electrical signals at the overlapping sampling time points are used as the training set for the interpolation algorithm for pre-training;

[0090] Preferably, an interpolation algorithm based on the U-Net structure is used to interpolate the visual-tactile image signal in the time domain. During the algorithm's pre-training, two adjacent frames of visual-tactile image signals are input. After multiple down-sampling operations, an electrical signal is added to the intermediate signal of the algorithm using a cross-attention method for training. Then, multiple up-sampling operations and stitching are performed to generate a new visual-tactile image signal. The generated image signal is then compared with the existing visual-tactile image signal at that moment, and this process is repeated to make the output visual-tactile image signal more closely approximate the real visual-tactile image signal.

[0091] S3. Test begins. Continuously press the composite high-frequency multimodal tactile sensor 5's composite high-frequency multimodal sensing unit 1 to generate tactile signals. At each T... e The electrical signals obtained at each sampling time point are processed by the processing circuit 6 and then transmitted to the host computer 7; simultaneously, the image acquisition unit 4 will... v The visual-tactile image signals obtained at the sampling time points are sent to the host computer 7;

[0092] The visual-tactile image signals of two adjacent frames and the electrical signals of the corresponding time and the time to be generated are input into the interpolation algorithm after pre-training to obtain a new visual-tactile image signal for the time to be generated; if there is already a visual-tactile image signal acquired by the image acquisition unit 4 at the time to be generated, the visual-tactile image signal obtained by the image acquisition unit 4 at that sampling time point shall be used as the standard.

[0093] Repeat the above steps to increase the sampling rate of the visual-tactile image signal from f v Improved to f through time-domain interpolation e And finally, the high-frequency visual and tactile image signals are displayed on the host computer.

[0094] S4. End the test by specifying an instruction or button on the host computer 7.

[0095] The composite high-frequency multimodal tactile sensing system provided in the above embodiments of the present invention possesses excellent high-frequency response capability and multimodal data fusion capability. This system can simultaneously capture electrical signals and visual-tactile image signals through the composite high-frequency multimodal sensing unit 1, and further improve the sampling rate of the visual-tactile image signals by using an interpolation algorithm in the host computer 7 to perform time-domain interpolation on the low-frequency visual-tactile image signals. This fusion of high-frequency and high-resolution information greatly enhances the application range of the composite high-frequency multimodal tactile sensing system, enabling it to perform better in the fine manipulation tasks of humanoid robots. Furthermore, the system is highly modular and integrated, easy to install and maintain, further improving its economy and practicality in real-world applications.

[0096] Unless otherwise specified in the above embodiments of the present invention, all parts can be implemented using existing technologies.

[0097] It should also be understood that the terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the invention. All accompanying drawings are merely for illustrative purposes, and the numbers, positions of components, relationships between components, and dimensions of components used in the preferred embodiments do not constitute a limitation on the technical solution itself. In the description of this specification, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for ease of description and simplification, not to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention.

[0098] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A composite high-frequency multimodal tactile sensing system, characterized in that, include: A composite high-frequency multimodal tactile sensor that simultaneously acquires visual-tactile image signals and electrical signals, wherein the sampling rate of the electrical signals is greater than the sampling rate of the visual-tactile image signals; The processing circuit is used to transmit control signals, control the composite high-frequency multimodal tactile sensor to acquire visual tactile image signals, process and integrate the electrical signals, and transmit the electrical signals from multiple channels at the same time to the host computer. The host computer uses an interpolation algorithm to interpolate the high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain, and displays the interpolated high-frequency visual-tactile image signal. The interpolation algorithm, by training the visual-tactile image signal and the electrical signal at overlapping times, can generate a corresponding visual-tactile image signal from the electrical signal. Specifically, this includes: The training module for training the interpolation algorithm uses an interpolation algorithm based on the U-Net structure to interpolate the visual-tactile image signal in the time domain. In the pre-training of the algorithm, two adjacent visual-tactile image signals are input. After multiple downsampling, an electrical signal is added to the intermediate signal of the algorithm through the cross-attention method for training. After multiple upsampling and stitching, a new visual-tactile image signal is generated. The generated visual-tactile image signal is then compared with the existing visual-tactile image signal at the same time. The training is repeated to make the output visual-tactile image signal more similar to the real visual-tactile image signal. The composite high-frequency multimodal sensing unit is touched by forces of different directions and magnitudes. The images are then processed by the image acquisition unit and processing circuit. v and T e The visual-tactile image signals and electrical signals at each sampling time point are transmitted to the host computer; simultaneously, the image acquisition unit will transmit the visual-tactile image signals and electrical signals at each T... v The visual-tactile image signals obtained at the sampling time points are sent to the host computer; the visual-tactile image signals of two adjacent frames, the electrical signals at the corresponding times and the time to be generated are input into the pre-trained interpolation algorithm to obtain a new visual-tactile image signal for the time to be generated; if a visual-tactile image signal already exists at the time to be generated, the visual-tactile image signal obtained by the image acquisition unit at that sampling time point is used; through the above interpolation operation, the sampling rate of the visual-tactile image signal is adjusted from... f v Improved to [the desired level] through time-domain interpolation. f e The high-frequency visual and tactile image signals are then displayed on the host computer.

2. The composite high-frequency multimodal tactile sensing system according to claim 1, characterized in that, The composite high-frequency multimodal tactile sensor includes: The composite high-frequency multimodal sensing unit, through multi-layer composite stacking, generates electrical signal changes while deforming upon contact with the outside world. The electrical signal is collected, processed by the processing circuit, and then transmitted to the host computer. The image acquisition unit acquires the deformation information of the composite high-frequency multimodal sensing unit to obtain a visual-tactile image signal, and directly transmits the visual-tactile image signal to the host computer. A programmable LED strip is connected to the processing circuit and provides a light source for the image acquisition unit under the control of the processing circuit. An integrated housing assembles the composite high-frequency multimodal sensing unit, the image acquisition unit, and the programmable light strip into one unit.

3. The composite high-frequency multimodal tactile sensing system according to claim 2, characterized in that, The composite high-frequency multimodal sensing unit includes, from bottom to top, a transparent rigid support layer, a transparent elastomer layer, a tracer particle layer, a multi-channel lower electrode layer, a multi-channel force-sensitive layer, a multi-channel upper electrode layer, and a reflective layer.

4. The composite high-frequency multimodal tactile sensing system according to claim 3, characterized in that, When the composite high-frequency multimodal sensing unit comes into contact with the outside world and deforms, the multi-channel force-sensitive layer undergoes changes in electrical signals, including piezoelectric signals and piezoresistive signals. The number of channels in the multi-channel force-sensitive layer is no more than the number of tracer particles in the tracer particle layer, and each channel in the multi-channel force-sensitive layer is covered by at least one tracer particle.

5. The composite high-frequency multimodal tactile sensing system according to claim 3, characterized in that, It also includes one or more of the following features: - The multi-channel lower electrode layer, multi-channel force-sensitive layer and multi-channel upper electrode layer are colorless and transparent, with a transmittance of not less than 80%. When the image acquisition unit takes a picture from below the composite high-frequency multimodal sensing unit, the acquired visual-tactile image signal has complete information on the morphological changes of the reflective layer and the positional changes of the tracer particles. - The reflective layer is stretchable and opaque. When the image acquisition unit takes a picture from below the composite high-frequency multimodal sensing unit, the reflective layer can completely block the external light source.

6. The composite high-frequency multimodal tactile sensing system according to claim 1, characterized in that, The host computer, wherein the interpolation algorithm specifically includes: Let T be the sampling time for acquiring visual-tactile image signals. v For T v_1 T v_2 ... T v_n Sampling rate of visual-tactile image signal f v 1 / (T) v_n -T v_n-1 ); Sampling time T for acquiring electrical signals e For T e_1 T e_2 ... T e_i-1 T e_i ... T e_2i-1 T e_2i ... T e_(n-1)i-1 T e_(n-1)i ... T e_ni-1 T e_ni Sampling rate of electrical signals f e 1 / (T) e_ni -T e_ni-1 ); f e Greater than f v ; The sampling time of the composite high-frequency multi-modal tactile sensing system is T e interpolated to T v , and through an interpolation algorithm, i-1 interpolated visual-tactile image signals are added between the low-frequency visual-tactile image signals, so that when the composite high-frequency multi-modal tactile sensor is subjected to high-frequency tactile stimulation, the visual-tactile image signals captured at a sampling time of T v can be retained, and high-frequency electrical signals can also be obtained, and more visual-tactile image signals are obtained through interpolation in the time domain.

7. A method for acquiring high-frequency visual-tactile image signals, characterized in that, include: When the composite high-frequency multimodal sensing unit comes into contact with the outside world and undergoes deformation, it simultaneously acquires visual-tactile image signals and electrical signals, wherein the sampling rate of the electrical signals is greater than the sampling rate of the visual-tactile image signals. The acquisition of visual and tactile image signals is controlled, and the electrical signals are processed and integrated to integrate the electrical signals from multiple channels at the same time. An interpolation algorithm is used to interpolate the integrated high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain, thereby obtaining the interpolated high-frequency visual-tactile image signal. Specifically, the interpolation algorithm, by training the visual-tactile image signal and the electrical signal at overlapping times, can generate a corresponding visual-tactile image signal from the electrical signal. The interpolation algorithm is trained as follows: an interpolation algorithm based on U-Net structure is used to interpolate the visual-tactile image signal in the time domain. In the pre-training of the algorithm, two adjacent visual-tactile image signals are input. After multiple downsampling, an electrical signal is added to the intermediate signal of the algorithm through the cross-attention method for training. After multiple upsampling and splicing, a new visual-tactile image signal is generated. The generated visual-tactile image signal is then compared with the existing visual-tactile image signal at the same time. The training is repeated to make the output visual-tactile image signal more similar to the real visual-tactile image signal. The composite high-frequency multimodal sensing unit is touched by forces of different directions and magnitudes. The composite high-frequency multimodal sensing unit is used in various T... e The electrical signal is obtained at the sampling time point, processed by the processing circuit, and then transmitted to the host computer. The sampling rate of the electrical signal is... f e Image acquisition units are used at each T v The visual-tactile image signals are acquired at sampling time points and sent to the host computer. The sampling rate of the visual-tactile image signals is... f v ; f e Greater than f v Simultaneously, the image acquisition unit will be located at each T... v The visual-tactile image signals obtained at the sampling time points are sent to the host computer; the visual-tactile image signals of two adjacent frames, the electrical signals at the corresponding times and the time to be generated are input into the pre-trained interpolation algorithm to obtain a new visual-tactile image signal for the time to be generated; if a visual-tactile image signal already exists at the time to be generated, the visual-tactile image signal obtained by the image acquisition unit at that sampling time point is used; through this interpolation operation, the sampling frequency of the visual-tactile image signal is adjusted from... f v Improved to [the desired level] through time-domain interpolation. f e The high-frequency visual and tactile image signals are then displayed on the host computer.

8. The method for acquiring high-frequency visual-tactile image signals according to claim 7, characterized in that, Before acquiring visual-tactile image signals and electrical signals, the process also includes: (i) Perform mechanical pre-calibration on the composite high-frequency multimodal sensing unit, including: Multiple known external forces are applied to the composite high-frequency multimodal sensing unit, while simultaneously acquiring electrical signals and visual-tactile image signals; For the electrical signal, multiple tests were conducted and the average value was taken. The relationship curve between the electrical output response and multiple known forces was plotted. e ; For visual-tactile image signals, a visual-tactile algorithm is used to track the displacement field changes of tracer particles under different external forces. Multiple tests are conducted, and the average value is taken to establish a curve L showing the relationship between the average displacement field change of the tracer particles and multiple known forces. v .

Images