Composite high-frequency multi-mode tactile sensing system and signal acquisition method

Abstract

The invention provides a composite high-frequency multi-mode tactile sensing system and a signal acquisition method, and the system comprises a composite high-frequency multi-mode tactile sensor which comprises a composite high-frequency multi-mode sensing unit, an image collection unit, a programmable lamp strip, an integrated housing, a processing circuit and an upper computer. The sensing system can collect the visual tactile image signals and the electrical signals at the same time, the result of the high-frequency electrical signals and the low-frequency visual tactile image signals are subjected to interpolation fusion in a time domain through an interpolation algorithm, and the sampling rate of the visual tactile image signals is remarkably improved. The system has higher tactile perception capability, is suitable for high-frequency tactile signal monitoring and complex mechanical information capturing, shows more excellent performance in the fields of humanoid robot perception, precision operation and the like, and greatly expands the application scene and the reliability of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of flexible sensors, and in particular to a composite high-frequency multi-modal tactile sensing system and a preparation method thereof. Background Art

[0002] Touch is one of the five important human senses. It is the most basic and widely used sense organ of humans, and an important channel for perceiving and interacting with the external environment. Human skin contains a variety of mechanical receptors that can respond to tactile stimuli in different frequency ranges. For example, Pacinian corpuscles are most sensitive to rapid vibrations, and the response frequency range is usually 40Hz to 500Hz; while Meissler corpuscles are sensitive to low-frequency vibrations, and the response range is 10Hz to 50Hz. The wide frequency response of these receptors enables humans to respond accurately in different tactile scenarios (such as identifying object textures, sensing external forces, etc.).

[0003] In order to better simulate human tactile perception, tactile sensors need to have high-frequency response capabilities and be able to capture tactile signals from low to high frequencies. However, existing tactile sensors (such as piezoresistive, piezoelectric, and capacitive sensors) perform well in high-frequency response, but still have limitations in reconstructing and identifying the surface texture of the contacted object. This makes these sensors perform poorly in applications that require high-resolution perception, especially in the fine manipulation tasks of humanoid robots, which cannot accurately reproduce surface details and rapidly changing tactile stimuli.

[0004] Visual tactile sensors are a new type of tactile sensor that can capture high-resolution image signals by combining computer vision and tactile perception, and process multimodal data through algorithms to achieve multimodal tactile perception. Visual tactile sensors can not only sense the force distribution of objects, but also perform detailed analysis of surface textures. However, current visual tactile sensors still have shortcomings in capturing high-frequency signals, which is mainly limited by the frame rate of the camera. Common micro cameras have a frame rate of 30fps or 60fps. This low sampling rate cannot effectively capture high-frequency signals such as vibrations, limiting its performance in processing rapidly changing scenes. Although existing high-speed cameras can achieve higher frame rates, such cameras are large in size and high in cost, which limits their application in the field of visual tactile sensors.

[0005] After searching, it was found that the invention patent application with publication number CN118443085A disclosed a fully flexible multimodal high spatiotemporal resolution visual-tactile sensor. Although the sensor can simultaneously acquire piezoelectric signals and visual-tactile image signals, and the sampling rate of the piezoelectric signal is not less than 1000Hz, the sensor still follows the above-mentioned visual-tactile sensing principle in terms of visual-tactile sensing. The acquisition frequency of the visual-tactile image signal is limited by the camera frame rate. In addition, the relationship between the piezoelectric signal and the visual-tactile image signal is not obvious, and it is impossible to obtain a high-frequency visual-tactile image signal based on the high-frequency piezoelectric signal. Therefore, the fully flexible multimodal high spatiotemporal resolution visual-tactile sensor can only acquire high-frequency piezoelectric signals, but cannot acquire high-frequency visual-tactile image signals.

[0006] In summary, it is particularly urgent to invent a sensor that can not only perform high-frequency sampling but also realize surface reconstruction and identification of contact objects. Summary of the invention

[0007] In view of the defects in the prior art, an object of the present invention is to provide a composite high-frequency multi-modal tactile sensing system and a preparation method thereof.

[0008] According to a first aspect of the present invention, there is provided a composite high-frequency multi-modal tactile sensing system, the system comprising:

[0009] A composite high-frequency multimodal tactile sensor, which simultaneously collects visual-tactile image signals and electrical signals, wherein the sampling rate of the electrical signal is greater than the sampling rate of the visual-tactile image signal;

[0010] A processing circuit, used for transmitting a control signal, controlling the composite high-frequency multi-modal tactile sensor to collect visual and tactile image signals, processing and integrating the electrical signals, integrating the electrical signals of multiple channels at the same time and transmitting them to the host computer;

[0011] The host computer uses an interpolation algorithm to interpolate the high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain, and displays the interpolated high-frequency visual-tactile image signal, wherein the interpolation algorithm can generate a corresponding visual-tactile image signal through the electrical signal by training the visual-tactile image signal and the electrical signal at the overlapping moment.

[0012] Optionally, the composite high-frequency multimodal tactile sensor comprises:

[0013] The composite high-frequency multi-modal sensing unit, through a multi-layer composite stack, generates a change in electrical signals while being deformed when in contact with the outside world, and the electrical signals are transmitted to the host computer after passing through the processing circuit;

[0014] An image acquisition unit, which acquires deformation information of the composite high-frequency multi-modal sensing unit to obtain visual-tactile image signals, and directly transmits the visual-tactile image signals to the host computer;

[0015] A programmable light strip, connected to the processing circuit, and providing a light source for the image acquisition unit under the control of the processing circuit;

[0016] The integrated housing assembles the composite high-frequency multi-modal sensing unit, the image acquisition unit and the programmable light strip into one body.

[0017] Optionally, the composite high-frequency multi-modal sensing unit includes a transparent hard support layer, a transparent elastomer layer, a tracer particle layer, a multi-channel lower electrode layer, a multi-channel force-sensitive layer, a multi-channel upper electrode layer and a reflective layer stacked in sequence from bottom to top.

[0018] Optionally, when the composite high-frequency multi-modal sensing unit is in contact with the outside world and deforms, the multi-channel force-sensitive layer generates a change in electrical signal, and the types of electrical signal include piezoelectric signal and piezoresistive signal.

[0019] Optionally, the number of channels of the multi-channel force-sensitive layer is no more than the number of tracer particles in the tracer particle layer, and each channel in the multi-channel force-sensitive layer is covered by at least one tracer particle.

[0020] Optionally, the multi-channel lower electrode layer, the multi-channel force-sensitive layer and the multi-channel upper electrode layer are colorless and light-transmitting, with a light transmittance of not less than 80%. When the image acquisition unit shoots upward from the bottom of the composite high-frequency multi-modal sensing unit, the collected visual-tactile image signal has complete information on the morphology changes of the reflective layer and the position changes of the tracer particles.

[0021] Optionally, the reflective layer is stretchable and opaque, and when the image acquisition unit shoots upward from below the composite high-frequency multi-modal sensing unit, the reflective layer can completely block the external light source.

[0022] Optionally, the host computer, wherein the interpolation algorithm specifically includes:

[0023] Assume that the sampling time of collecting visual and tactile image signals is T v T v_1 、T v_2 ,…,T v_n , the sampling rate of the visual-tactile image signal f v is 1 / (T v_n -T v_n-1 );Sampling time T for collecting electrical signals e T e_1 、Te_2 ,…,T e_i-1 , T e_i ,…,T e_2i-1 , T e_2i ,…,T e_(n-1)i-1 , T e_(n-1)i ,…,T e_ni-1 , T e_ni , the sampling rate of the electrical signal is f e is 1 / (T e_ni -T e_ni-1 );f e Greater than f v ;

[0024] The sampling time of the composite high-frequency multi-modal tactile sensing system is T e Interpolate to T v By interpolation, i-1 interpolated visual-tactile image signals are added between the low-frequency visual-tactile image signals, and when the composite high-frequency multimodal tactile sensor is subjected to high-frequency tactile stimulation, the sampling time can be retained as T v The visual and tactile image signals captured in time can also be obtained, and high-frequency electrical signals can be obtained at the same time, and more visual and tactile image signals can be obtained through interpolation in the time domain.

[0025] According to a second aspect of the present invention, a method for acquiring a high-frequency visual-tactile image signal is provided, comprising:

[0026] When the composite high-frequency multi-modal sensing unit is in contact with the outside world and deforms, visual-tactile image signals and electrical signals are collected simultaneously, wherein the sampling rate of the electrical signal is greater than the sampling rate of the visual-tactile image signal;

[0027] Controlling the collection of visual and tactile image signals, and processing and integrating the electrical signals, integrating the electrical signals of multiple channels at the same time;

[0028] An interpolation algorithm is used to interpolate the integrated high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain to obtain an interpolated high-frequency visual-tactile image signal; wherein the interpolation algorithm can generate a corresponding visual-tactile image signal through the electrical signal by training the visual-tactile image signal and the electrical signal at the overlapping moment.

[0029] Optionally, the method specifically includes:

[0030] A composite high-frequency multi-modal sensor unit is used in each T e The electrical signal is obtained at the sampling time point, and then transmitted to the host computer after being processed by the processing circuit. The sampling rate of the electrical signal is f e ;

[0031] The image acquisition unit is used at each T v The visual-tactile image signal is collected at the sampling time point and sent to the host computer, where the sampling rate of the visual-tactile image signal is f v ;f e Greater than f v ;

[0032] According to the interpolation algorithm and the electrical signal in the host computer, each T e The visual-tactile image signal at the sampling time point, if there is T v Sampling time point and T e If the sampling time points coincide, then the sampling time point T v The visual-tactile image signal obtained by the image acquisition unit is used as the basis;

[0033] Through the above steps, the sampling frequency of the visual-tactile image signal is changed from f v Increase to f by time domain interpolation e , and finally display the high-frequency visual and tactile image signals on the host computer.

[0034] Optionally, before collecting the visual-tactile image signal and the electrical signal, the method further includes:

[0035] (i) Mechanical pre-calibration of the composite high-frequency multi-modal sensing unit, including:

[0036] Applying multiple known external forces to the composite high-frequency multi-modal sensing unit, and simultaneously collecting electrical signals and visual and tactile image signals;

[0037] For electrical signals, multiple tests are performed to obtain the average value, and a relationship curve L between the electrical output response and multiple known forces is drawn. e ;

[0038] For the visual-tactile image signal, the visual-tactile algorithm is used to track the displacement field changes of the tracer particles caused by different external forces. The average value is taken after multiple tests, and the relationship curve L between the average displacement field change of the tracer particles and multiple known forces is established. v ;

[0039] (ii) Use forces of different directions and magnitudes to continuously touch the composite high-frequency multi-modal sensing unit, and use the image acquisition unit and processing circuit to v and T e The visual and tactile image signals and electrical signals at the sampling time point are transmitted to the host computer; in the host computer, T v Sampling time point and T eThe visual-tactile image signals and electrical signals at the time when the sampling time points coincide with each other are used as training sets for the interpolation algorithm for pre-training. By training the visual-tactile image signals and electrical signals at the time when the sampling time points coincide with each other, it is expected to achieve the effect of being able to generate corresponding visual-tactile image signals through electrical signals.

[0040] The above pre-trained interpolation algorithm is used for subsequent formal signal interpolation processing.

[0041] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0042] 1. The present invention can simultaneously process and analyze data from different sensing mechanisms by integrating the sensing principles of visual tactile sensors and electrical tactile sensors. This multimodal integration of visual tactile sensing and electrical tactile sensing enables the system to not only collect electrical signals, but also obtain visual tactile image signals for fine reconstruction and recognition of the surface of objects. Therefore, the present invention provides richer and more accurate perception information than traditional single sensors.

[0043] 2. The present invention can effectively improve the signal acquisition speed of the visual-tactile sensor and the signal receiving frequency of the visual-tactile sensor through an innovative time domain interpolation method.

[0044] 3. The present invention uses a composite high-frequency multi-modal sensing unit, and the sensor can perform time domain interpolation on low-frequency visual-tactile image signals, thereby improving the sampling rate of visual-tactile image signals and making up for the shortcoming of low sampling rate of traditional visual-tactile sensors.

[0045] 4. The present invention uses a composite high-frequency multimodal sensing unit and time domain interpolation to enable the system to accurately capture and quickly respond to subtle mechanical changes and high-frequency tactile stimulation, thereby improving the performance of visual tactile sensors in application scenarios that require high-frequency monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0047] Figure 1 is a schematic structural diagram of a composite high-frequency multi-modal tactile sensing system in one embodiment of the present invention;

[0048] Figure 2 is a schematic structural diagram of a composite high-frequency multi-modal sensing unit in one embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of the principle of implementing time domain interpolation of visual tactile image signals by a composite high-frequency multi-modal tactile sensing system in one embodiment of the present invention;

[0050] Figure 4is a sensing flow chart of a composite high-frequency multi-modal tactile sensing system in one embodiment of the present invention;

[0051] The marks in the figure are respectively represented as: 1-composite high-frequency multimodal sensing unit, 2-programmable light strip, 3-integrated housing, 4-image acquisition unit, 5-composite high-frequency multimodal tactile sensor, 6-processing circuit, 7-host computer, 8-transparent hard support layer, 9-transparent elastomer layer, 10-tracer particle layer, 11-multi-channel lower electrode layer, 12-multi-channel force-sensitive layer, 13-multi-channel upper electrode layer, 14-reflection layer. DETAILED DESCRIPTION

[0052] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0053] Reference Figure 1 As shown, a composite high-frequency multimodal tactile sensing system provided by a preferred embodiment of the present invention includes a composite high-frequency multimodal tactile sensor 5, a processing circuit 6 and a host computer 7, wherein the composite high-frequency multimodal tactile sensor 5 simultaneously collects visual-tactile image signals and electrical signals when it contacts with the outside world and generates deformation, wherein the sampling rate of the electrical signal is greater than the sampling rate of the visual-tactile image signal; the processing circuit 6 is used to transmit a control signal to control the composite high-frequency multimodal tactile sensor 5 to collect the visual-tactile image signal, and to process and integrate the electrical signal, and integrate the electrical signals of multiple channels at the same time and transmit them to the host computer 7; the host computer 7 uses an interpolation algorithm to interpolate the high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain, and displays the interpolated high-frequency visual-tactile image signal, wherein the interpolation algorithm in the host computer 7 can generate a corresponding visual-tactile image signal through the electrical signal by training the visual-tactile image signal and the electrical signal at the overlapping moment.

[0054] The high frequency and low frequency mentioned above are relative to the sampling rates of the electrical signal and the visual and tactile image signal. For example, the sampling time T for collecting the visual and tactile image signal is v T v_1 、T v_2 ,…,T v_n , so the sampling rate of the visual-tactile image signal is f v is 1 / (T v_n -T v_n-1 ) (the common sampling rate of visual and tactile image signals is 30fps or 60fps); the sampling time T for collecting electrical signals e T e_1, T e_2 ,…,T e_i-1 , T e_i ,…,T e_2i-1 , T e_2i ,…,T e_(n-1)i-1 , T e_(n-1)i ,…,T e_ni-1 , T e_ni , so the sampling rate of the electrical signal is f e is 1 / (T e_ni -T e_ni-1 );f e Greater than f v .

[0055] In the above embodiment of the present invention, the system can generate a new visual-tactile image signal by simultaneously collecting high-frequency electrical signals and low-frequency visual-tactile image signals and combining an interpolation algorithm, thereby improving the sampling rate of the visual-tactile image signal.

[0056] Reference Figure 1 As shown, the composite high-frequency multimodal tactile sensor 5 includes a composite high-frequency multimodal sensing unit 1, an image acquisition unit 4, a programmable light strip 2 and an integrated housing 3. The composite high-frequency multimodal sensing unit 1 is stacked in multiple layers, and changes in electrical signals occur when it is in contact with the outside world and deforms. The electrical signals are collected and transmitted to the host computer 7 after being processed by the circuit; the image acquisition unit 4 collects the deformation information of the composite high-frequency multimodal sensing unit 1 to obtain a visual tactile image signal, and directly transmits the visual tactile image signal to the host computer 7; the programmable light strip 2 is connected to the processing circuit 6, and provides a light source for the image acquisition unit 4 under the control of the processing circuit 6; the integrated housing 3 assembles the composite high-frequency multimodal sensing unit 1, the image acquisition unit 4 and the programmable light strip 2 into one, thereby forming a composite high-frequency multimodal tactile sensor 5.

[0057] As some preferred embodiments, in the above-mentioned composite high-frequency multimodal tactile sensor 5, the integrated housing 3 plays the role of supporting and fixing the remaining components of the composite high-frequency multimodal tactile sensor 5; the composite high-frequency multimodal sensing unit 1 is placed on the top of the integrated housing 3, and is used to contact with the outside world to generate deformation, and at the same time generate a high-frequency electrical signal, which is processed by the processing circuit 6 and transmitted to the host computer 7; the programmable light strip 2 can be placed around the composite high-frequency multimodal sensing unit 1, and is used to provide a light source, and is connected to the processing circuit 6 and controlled by the processing circuit 6.

[0058] Reference Figure 1 and Figure 2As shown, as some preferred embodiments, the image acquisition unit 4 is placed directly below the composite high-frequency multimodal sensing unit 1, and is used to obtain the visual and tactile image signal when the composite high-frequency multimodal sensing unit 1 contacts the outside world and produces deformation, and is directly connected to the host computer 7 to transmit the signal to the host computer 7; in addition, in order to ensure the quality of the visual and tactile image signal, the integrated housing 3 is made of a black opaque material as a whole, and the top reflective layer 14 of the composite high-frequency multimodal sensing unit 1 is also opaque, ensuring that the internal light source of the entire composite high-frequency multimodal tactile sensor 5 is only provided by the programmable light strip 2.

[0059] During the specific implementation, the composite high-frequency multimodal sensing unit 1 comes into contact with the outside world, and deformation occurs while the electrical signal changes, and the electrical signal is transmitted to the host computer 7 via the processing circuit 6; at the same time, the image acquisition unit 4 obtains the deformation information of the composite high-frequency multimodal sensing unit 1, and transmits the visual and tactile image signal directly to the host computer 7; in this process, the programmable light strip provides a stable white light source for the entire composite high-frequency multimodal tactile sensor 5.

[0060] In one embodiment, the host computer of the composite high-frequency multi-modal tactile sensing system may further include: a training module for training the interpolation algorithm; a training module for touching the composite high-frequency multi-modal sensing unit with forces of different directions and magnitudes, and converting each T v and T e The visual and tactile image signals and electrical signals at the sampling time point are transmitted to the host computer; in the host computer, T v Sampling time point and T e The visual-tactile image signals and electrical signals at the time when the sampling time points coincide with each other are used as training sets for the interpolation algorithm for pre-training. By training the visual-tactile image signals and electrical signals at the time when the sampling time points coincide with each other, the effect of being able to generate corresponding visual-tactile image signals through electrical signals is achieved.

[0061] Reference Figure 3 As shown, when the host computer 7 receives the sampling rate of f v The visual tactile image signal and the sampling rate are f e When an electrical signal is obtained, more visual-tactile image signals are generated between two adjacent frames of sampled visual-tactile image signals through an interpolation algorithm. When there is a visual-tactile image signal obtained by the image acquisition unit 4 at a time point corresponding to the generated visual-tactile image signal, the visual-tactile image signal obtained by the image acquisition unit 4 at the sampling time point shall prevail. Finally, the visual-tactile image signal generated by the trained interpolation algorithm corresponds to each electrical sampling point one by one, and the sampling rate of the visual-tactile image signal is changed from f to f. v Raised to f eAt the same time, the generated visual-tactile image signal is made continuous in the time domain, thereby obtaining a high-frequency visual-tactile image signal that is continuous in the time domain.

[0062] In the above embodiment of the present invention, the sampling time of the composite high-frequency multi-modal tactile sensing system is to set T e Interpolate to T v Therefore, through interpolation, i-1 interpolated visual and tactile image signals will be added between the low-frequency visual and tactile image signals. When the composite high-frequency multimodal tactile sensor 5 is subjected to high-frequency tactile stimulation, it can not only retain the sampling time T v The visual-tactile image signals captured in the time domain can also obtain high-frequency electrical signals. Through interpolation in the time domain, more visual-tactile image signals can be obtained, and then more detailed multimodal tactile information such as the morphological changes and depth changes of the contact objects can be obtained.

[0063] In other preferred embodiments, refer to Figure 1 and Figure 2 As shown, the composite high-frequency multi-modal sensing unit 1 is stacked from bottom to top in sequence as a transparent hard support layer 8, a transparent elastomer layer 9, a tracer particle layer 10, a multi-channel lower electrode layer 11, a multi-channel force-sensitive layer 12, a multi-channel upper electrode layer 13 and a reflective layer 14.

[0064] When the multi-channel force-sensitive layer 12 is deformed by contact with the outside world, the electrical signal changes, and the types of electrical signals include piezoelectric signals and piezoresistive signals. The position where the electrical signal is generated corresponds to the position of the tracer particles in the visual tactile signal. In the subsequent interpolation algorithm processing, the connection is established through the corresponding relationship at this position, which can better predict the new visual tactile image signal.

[0065] In other preferred embodiments, the number of channels of the multi-channel force-sensitive layer 12 is no more than the number of tracer particles in the tracer particle layer 10, and each channel in the multi-channel force-sensitive layer 12 is covered by at least one tracer particle to ensure that the relationship between the visual-tactile image signal and the electrical signal is more intuitive, and thus better generate a new visual-tactile image signal by interpolating the electrical signal in the time domain.

[0066] In other preferred embodiments, the transparent hard support layer 8 is cut from a transparent acrylic plate material; the transparent elastomer layer 9 is made of polydimethylsiloxane (PDMS); on the surface of the transparent elastomer layer 9, tracer particles are printed through a dispensing process of a microelectronic printer to form a tracer particle layer 10; optionally, the tracer particles are prepared from silica gel doped with red pigment.

[0067] In some other preferred embodiments, piezoelectric material is selected as the sensing material of the multi-channel force-sensitive layer 12 . Preferably, a transparent and flexible polyvinylidene difluoride (PVDF) film is used as the material of the multi-channel force-sensitive layer 12 .

[0068] In other preferred embodiments, the multi-channel lower electrode layer 11, the multi-channel force-sensitive layer 12 and the multi-channel upper electrode layer 13 are colorless and translucent, and the transmittance is not less than 80%. When the image acquisition unit 4 shoots upward from the bottom of the composite high-frequency multi-modal sensing unit 1, the collected visual and tactile image signals have complete information on the morphological changes of the reflective layer and the position changes of the tracer particles. In a specific embodiment, a structure in which a sensitive material is located between the upper and lower electrodes can be used to realize the collection and transmission of electrical signals. Preferably, transparent indium tin oxide (ITO) with good conductivity is used as the electrode material; the ITO electrode is attached to the PDMS film, and the electrode is patterned by a laser cutting process. According to different cutting patterns, they are used as the multi-channel lower electrode layer 11 and the multi-channel upper electrode layer 13 respectively; finally, the multi-channel lower electrode layer 11, the multi-channel force-sensitive layer 12 and the multi-channel upper electrode layer 13 are pasted together in order, and finally attached to the top of the tracer particle layer 10.

[0069] In other preferred embodiments, the reflective layer 14 is stretchable and opaque, and when the image acquisition unit 4 shoots upward from the bottom of the composite high-frequency multi-modal sensing unit 1, the reflective layer 14 can completely block the external light source. In a specific embodiment, the raw material of the reflective layer 14 can be PDMS doped with silver powder, and a layer of the reflective layer 14 is spin-coated by a spin coating process to achieve high-precision texture perception.

[0070] In other preferred embodiments, the programmable light strip 2 uses a WS2812B light strip. Optionally, in order to further improve the quality of visual and tactile image signals and improve the stability of the light source, a light guide plate is added between the programmable light strip 2 and the composite high-frequency multimodal sensing unit 1 to evenly distribute the light and improve the stability of the light source.

[0071] In other preferred embodiments, refer to Figure 1 As shown, the integrated housing 3 is made by 3D printing, and resin is selected as the preparation material to achieve an overall black and opaque effect.

[0072] In other preferred embodiments, refer to Figure 1As shown, the image acquisition unit 4 uses a HBVCAM-F2209HDV11 camera, which can capture images at a rate of 30fps. It is connected to the host computer 7 through a USB interface to achieve the transmission of visual and tactile image signals.

[0073] In other preferred embodiments, the processing circuit 6 includes a multi-channel operational amplifier, an analog-to-digital converter, a central processing unit and a signal transmission module. Figure 1 As shown, the processing circuit 6 is manufactured by designing the schematic diagram of the acquisition and control circuit using the PCB process; the multi-channel operational amplifier required for the piezoelectric signal acquisition circuit adopts the MCP6002 chip; the analog-to-digital converter, the central processing unit and the signal transmission module are provided by the ESP32-PICO-KIT development board, and finally the acquisition and transmission of electrical signals and the control of the programmable light strip 2 are realized.

[0074] The system constructed by the composite high-frequency multimodal tactile sensor 5 in the above-mentioned embodiments of the present invention can not only realize the synchronous acquisition of high-frequency electrical signals and low-frequency visual tactile image signals in an integrated manner, but also realize the acquisition of high-frequency visual tactile image signals through high-frequency electrical signals and interpolation methods. The results of high-frequency electrical signals are interpolated and fused with low-frequency visual tactile image signals in the time domain through the interpolation algorithm, which significantly improves the sampling rate of visual tactile image signals. The system has higher tactile perception capabilities, is suitable for high-frequency tactile signal monitoring and the capture of complex mechanical information, and shows more excellent performance in the fields of humanoid robot perception and precision operation, which greatly expands the application scenarios and reliability of the system.

[0075] Based on the same technical concept, in another embodiment of the present invention, a method for acquiring a high-frequency visual-tactile image signal is also provided, comprising:

[0076] S101, when the composite high-frequency multi-modal sensing unit 1 is in contact with the outside world and deforms, visual and tactile image signals and electrical signals are collected simultaneously, wherein the sampling rate of the electrical signal is greater than the sampling rate of the visual and tactile image signal;

[0077] S102, processing the visual and tactile image signal and the electrical signal to meet the interpolation requirement;

[0078] S103, using an interpolation algorithm to interpolate the processed high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain to obtain an interpolated high-frequency visual-tactile image signal.

[0079] In a preferred embodiment, the method for acquiring the high-frequency visual-tactile image signal can be further represented by the following steps:

[0080] S201, using a composite high-frequency multi-modal sensing unit 1 in each T eThe electrical signal is obtained at the sampling time point, and is processed by the processing circuit 6 and transmitted to the host computer 7. The sampling rate of the electrical signal is f e ;

[0081] S202, using the image acquisition unit 4 at each T v The visual-tactile image signal is collected at the sampling time point and sent to the host computer 7, wherein the sampling rate of the visual-tactile image signal is f v ;f e Greater than f v ;

[0082] S203, according to the interpolation algorithm and the electrical signal in the host computer 7, obtain each T e The visual-tactile image signal at the sampling time point, if there is T v Sampling time point and T e If the sampling time points coincide, then the sampling time point T v The visual-tactile image signal obtained by the image acquisition unit 4 is used as the basis;

[0083] Through the above steps, the sampling frequency of the visual-tactile image signal is changed from f v Increase to f by time domain interpolation e , and finally the high-frequency visual-tactile image signal is displayed on the host computer 7.

[0084] In a preferred embodiment, in order to better obtain the above-mentioned high-frequency visual-tactile image signal, before the above-mentioned high-frequency visual-tactile image signal acquisition method is started, it can further include mechanical pre-calibration of the composite high-frequency multi-modal sensor unit 1. Specifically, the mechanical pre-calibration process of the composite high-frequency multi-modal sensor unit 1 includes: applying multiple known external forces to the composite high-frequency multi-modal sensor unit 1, and collecting electrical signals and visual-tactile image signals at the same time; for the electrical signal, taking the average value of multiple tests, and drawing the relationship curve L between the electrical output response and multiple known forces e ; For visual-tactile image signals, visual-tactile algorithms, such as optical flow method, are used to track the displacement field changes of tracer particles caused by different external forces, and the average value is taken after multiple tests to establish the relationship curve L between the average displacement field change of tracer particles and multiple known forces v .

[0085] After the above mechanical pre-calibration process, in order to better complete the interpolation operation, in a preferred embodiment, it further includes: continuously using forces of different directions and magnitudes to touch the composite high-frequency multi-modal sensing unit 1, and converting each T v and T e The visual and tactile image signals and electrical signals at the sampling time point are transmitted to the host computer 7; in the host computer 7, T vSampling time point and T e The visual-tactile image signals and electrical signals at the time when the sampling time points coincide with each other are used as training sets for the interpolation algorithm for pre-training. By training the visual-tactile image signals and electrical signals at the time when the sampling time points coincide with each other, it is expected to achieve the effect of being able to generate corresponding visual-tactile image signals through electrical signals.

[0086] After the pre-training of the interpolation algorithm is completed, the electrical signal and the visual-tactile image signal are formally collected and difference processed (using the pre-trained interpolation algorithm).

[0087] In another preferred embodiment, the above-mentioned composite high-frequency multi-modal tactile sensing system is used to obtain high-frequency visual tactile image signals. Figure 4 As shown, the specific process includes:

[0088] S1. Before the test begins, perform mechanical pre-calibration on the composite high-frequency multi-modal sensing unit 1. Apply multiple known external forces to the composite high-frequency multi-modal sensing unit 1, and collect electrical signals and visual and tactile image signals at the same time; for the electrical signal, take the average value after multiple tests, and draw a relationship curve L between the electrical output response and multiple known forces e For visual-tactile image signals, preferably, the displacement field changes of tracer particles caused by different external forces are tracked by optical flow method, and the average value is taken after multiple tests to establish the relationship curve L between the average displacement field change of tracer particles and multiple known forces v ;

[0089] S2. Before the test begins, the composite high-frequency multi-modal sensing unit 1 is continuously pressed with forces of different directions and magnitudes, and each T v and T e The visual-tactile image signal and the electrical signal at the sampling time point are transmitted to the host computer 7; in the host computer 7, the visual-tactile image signal and the electrical signal at the same sampling time point are used as the training set of the interpolation algorithm for pre-training;

[0090] Preferably, an interpolation algorithm based on a U-Net structure is used to implement interpolation of visual-tactile image signals in the time domain. In the pre-training of the algorithm, two adjacent frames of visual-tactile image signals are input, and after multiple down-sampling (Down-Sampling), an electrical signal is added to the intermediate signal of the algorithm through the cross attention method (Cross Attention) for training, and then multiple up-sampling (Up-Sampling) and splicing are performed to generate a new visual-tactile image signal; then the generated image signal is compared with the existing visual-tactile image signal at that moment, and the training is repeated to make the output visual-tactile image signal more similar to the real visual-tactile image signal;

[0091] S3, the test starts, the composite high-frequency multi-modal sensing unit 1 of the composite high-frequency multi-modal tactile sensor 5 is continuously pressed to generate a tactile signal. e The electrical signal obtained at the sampling time point is processed by the processing circuit 6 and then transmitted to the host computer 7; at the same time, the image acquisition unit 4 will v The visual-tactile image signal obtained at the sampling time point is sent to the host computer 7;

[0092] Input the visual-tactile image signals of two adjacent frames and the electrical signals at the corresponding time and the time to be generated into the interpolation algorithm after pre-training to obtain a new visual-tactile image signal at the time to be generated; if there is a visual-tactile image signal collected by the image acquisition unit 4 at the time to be generated, the visual-tactile image signal obtained by the image acquisition unit 4 at the sampling time point shall prevail;

[0093] Repeat the above steps to increase the sampling rate of the visual-tactile image signal from f v Increase to f by time domain interpolation e , and finally display the high-frequency visual and tactile image signal on the host computer 7;

[0094] S4. The host computer 7 specifies instructions or buttons to end the test.

[0095] The composite high-frequency multimodal tactile sensing system provided by the above-mentioned embodiment of the present invention has excellent high-frequency response capability and multimodal data fusion capability. The system can simultaneously capture electrical signals and visual-tactile image signals through the composite high-frequency multimodal sensing unit 1, and by using the interpolation algorithm in the upper computer 7, it can realize the time domain interpolation of the low-frequency visual-tactile image signal, thereby improving the sampling rate of the visual-tactile image signal. This high-frequency and high-resolution information fusion greatly enhances the application scope of the composite high-frequency multimodal tactile sensing system, so that it has better performance in performing the fine operation tasks of humanoid robots. In addition, the system design is highly modular and integrated, easy to install and maintain, and further improves its economy and practicality in practical applications.

[0096] Parts not particularly described in the above embodiments of the present invention may be implemented using existing technologies.

[0097] It should also be understood that the terms used in this specification are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. Embodiments All drawings are only for the convenience of explaining the technical content of the present invention, and the technical features such as the numbers, the positions of the parts, the mutual relations between the parts, and the sizes of the parts used in the optimal implementation mode do not constitute a limitation on the technical solution itself. In the description of this specification, it should be understood that the orientation or positional relationship indicated by the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. is based on the orientation or positional relationship shown in the drawings, only for the convenience of description and simplified description, rather than indicating or suggesting that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0098] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A composite high-frequency multi-modal tactile sensing system, characterized in that: include: A composite high-frequency multimodal tactile sensor, which simultaneously collects visual-tactile image signals and electrical signals, wherein the sampling rate of the electrical signal is greater than the sampling rate of the visual-tactile image signal; A processing circuit, used for transmitting a control signal, controlling the composite high-frequency multi-modal tactile sensor to collect visual and tactile image signals, processing and integrating the electrical signals, integrating the electrical signals of multiple channels at the same time and transmitting them to the host computer; The host computer uses an interpolation algorithm to interpolate the high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain, and displays the interpolated high-frequency visual-tactile image signal, wherein the interpolation algorithm can generate a corresponding visual-tactile image signal through the electrical signal by training the visual-tactile image signal and the electrical signal at the overlapping moment.

2. The composite high-frequency multi-modal tactile sensing system according to claim 1, characterized in that: The composite high-frequency multi-modal tactile sensor comprises: The composite high-frequency multi-modal sensing unit, through a multi-layer composite stack, generates a change in electrical signals while generating deformation when in contact with the outside world, collects the electrical signals and transmits them to the host computer after passing through the processing circuit; An image acquisition unit, which acquires deformation information of the composite high-frequency multi-modal sensing unit to obtain visual-tactile image signals, and directly transmits the visual-tactile image signals to the host computer; A programmable light strip, connected to the processing circuit, and providing a light source for the image acquisition unit under the control of the processing circuit; The integrated housing assembles the composite high-frequency multi-modal sensing unit, the image acquisition unit and the programmable light strip into one body.

3. The composite high-frequency multi-modal tactile sensing system according to claim 2, characterized in that: The composite high-frequency multi-modal sensing unit comprises a transparent hard support layer, a transparent elastic layer, a tracer particle layer, a multi-channel lower electrode layer, a multi-channel force-sensitive layer, a multi-channel upper electrode layer and a reflective layer which are stacked in sequence from bottom to top.

4. The composite high-frequency multi-modal tactile sensing system according to claim 3, characterized in that: When the composite high-frequency multi-modal sensing unit is in contact with the outside world and deforms, the multi-channel force-sensitive layer generates a change in electrical signal, and the types of electrical signal include piezoelectric signal and piezoresistive signal; The number of channels of the multi-channel force-sensitive layer is no more than the number of tracer particles in the tracer particle layer, and each channel in the multi-channel force-sensitive layer is covered by at least one tracer particle.

5. The composite high-frequency multi-modal tactile sensing system according to claim 3, characterized in that: Also includes one or more of the following features: - the multi-channel lower electrode layer, the multi-channel force-sensitive layer and the multi-channel upper electrode layer are colorless and light-transmitting, with a light transmittance of not less than 80%. When the image acquisition unit shoots upward from the bottom of the composite high-frequency multi-modal sensing unit, the collected visual-tactile image signal has complete information on the morphological changes of the reflective layer and the position changes of the tracer particles; -The reflective layer is stretchable and opaque, and when the image acquisition unit shoots upward from below the composite high-frequency multi-modal sensing unit, the reflective layer can completely block the external light source.

6. The composite high-frequency multi-modal tactile sensing system according to claim 1, characterized in that: The host computer, wherein the interpolation algorithm specifically includes: Assume that the sampling time of collecting visual and tactile image signals is T v T v_1 , T v_2 ,…,T v_n , the sampling rate of the visual-tactile image signal f v is 1 / (T v_n -T v_n-1 );Sampling time T for collecting electrical signals e T e_1 , T e_2 ,…,T e_i-1 , T e_i ,…,T e_2i-1 , T e_2i ,…,T e_(n-1)i-1 , T e_(n-1)i ,…,T e_ni-1 , T e_ni , the sampling rate of the electrical signal is f e is 1 / (T e_ni -T e_ni-1 );f e Greater than f v ; The sampling time of the composite high-frequency multi-modal tactile sensing system is T e Interpolate to T v By using an interpolation algorithm, i-1 interpolated visual-tactile image signals are added between the low-frequency visual-tactile image signals. When the composite high-frequency multimodal tactile sensor is subjected to high-frequency tactile stimulation, the sampling time can be retained as T v The visual and tactile image signals captured in time can also be obtained, and high-frequency electrical signals can be obtained at the same time, and more visual and tactile image signals can be obtained through interpolation in the time domain.

7. The composite high-frequency multi-modal tactile sensing system according to claim 6, characterized in that: Also includes: A training module for training the interpolation algorithm; The training module uses forces of different directions and magnitudes to touch the composite high-frequency multi-modal sensing unit, and uses the image acquisition unit and processing circuit to v and T e The visual and tactile image signals and electrical signals at the sampling time point are transmitted to the host computer; in the host computer, T v Sampling time point and T e The visual-tactile image signals and electrical signals at the time when the sampling time points coincide are used as training sets for the interpolation algorithm for pre-training. By training the visual-tactile image signals and electrical signals at the time when the sampling time points coincide, the corresponding visual-tactile image signals can be generated through the electrical signals.

8. A method for acquiring a high-frequency visual-tactile image signal, characterized in that: include: When the composite high-frequency multi-modal sensing unit is in contact with the outside world and deforms, visual-tactile image signals and electrical signals are collected simultaneously, wherein the sampling rate of the electrical signal is greater than the sampling rate of the visual-tactile image signal; Controlling the collection of visual and tactile image signals, and processing and integrating the electrical signals, integrating the electrical signals of multiple channels at the same time; An interpolation algorithm is used to interpolate the integrated high-frequency electrical signal into the low-frequency visual-tactile image signal in the time domain to obtain an interpolated high-frequency visual-tactile image signal; wherein the interpolation algorithm can generate a corresponding visual-tactile image signal through the electrical signal by training the visual-tactile image signal and the electrical signal at the overlapping moment.

9. The method for acquiring high-frequency visual-tactile image signals according to claim 8, characterized in that: Specifically include: A composite high-frequency multi-modal sensor unit is used in each T e The electrical signal is obtained at the sampling time point, and then transmitted to the host computer after being processed by the processing circuit. The sampling rate of the electrical signal is f e ; The image acquisition unit is used at each T v The visual-tactile image signal is collected at the sampling time point and sent to the host computer, where the sampling rate of the visual-tactile image signal is f v ;f e Greater than f v ; According to the interpolation algorithm and the electrical signal in the host computer, each T e The visual-tactile image signal at the sampling time point, if there is T v Sampling time point and T e If the sampling time points coincide, then the sampling time point T v The visual-tactile image signal obtained by the image acquisition unit is used as the basis; Through the above steps, the sampling frequency of the visual-tactile image signal is changed from f v Increase to f by time domain interpolation e , and finally display the high-frequency visual and tactile image signals on the host computer.

10. The method for acquiring high-frequency visual-tactile image signals according to claim 9, characterized in that: Before collecting visual and tactile image signals and electrical signals, it also includes: (i) Mechanical pre-calibration of the composite high-frequency multi-modal sensing unit, including: Applying multiple known external forces to the composite high-frequency multi-modal sensing unit, and simultaneously collecting electrical signals and visual and tactile image signals; For electrical signals, multiple tests are performed to obtain the average value, and a relationship curve L between the electrical output response and multiple known forces is drawn. e ; For the visual-tactile image signal, the visual-tactile algorithm is used to track the displacement field changes of the tracer particles caused by different external forces. The average value is taken after multiple tests, and the relationship curve L between the average displacement field change of the tracer particles and multiple known forces is established. v ; (ii) Use forces of different directions and magnitudes to continuously touch the composite high-frequency multi-modal sensing unit, and use the image acquisition unit and processing circuit to v and T e The visual and tactile image signals and electrical signals at the sampling time point are transmitted to the host computer; in the host computer, T v Sampling time point and T e The visual-tactile image signals and electrical signals at the time when the sampling time points coincide with each other are used as training sets for the interpolation algorithm for pre-training. By training the visual-tactile image signals and electrical signals at the time when the sampling time points coincide with each other, it is expected to achieve the effect of being able to generate corresponding visual-tactile image signals through electrical signals.

Images