Ultrasonic enhanced tactile sensor system, sensor and sensing method

Abstract

The present invention discloses an ultrasonically enhanced tactile sensor system, sensor, and sensing method. The tactile sensor system includes a multimodal sensing unit and a processing unit. The multimodal sensing unit integrates an ultrasonic module and a visual tactile perception module, realizes ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure, and can synchronously collect ultrasonic echo signals and tactile deformation images. In the processing unit, the event trigger module detects contact events based on tactile images and generates switching signals. The adaptive processing module dynamically controls the ultrasonic mode accordingly, measures distance when non-contact, and switches to internal imaging of the object or material property identification when in contact. The three-level fusion processing module outputs multimodal perception results through signal-level synchronous acquisition, feature-level cross-modal fusion, and decision-level task switching. The system realizes multi-dimensional perception of proximity, surface texture, and deep information, improving the robot's ability to predict and accurately operate in complex scenarios.

Description

Technical Field

[0001] The present invention relates to the field of robot perception, and in particular to an ultrasonically enhanced tactile sensor system, a sensor, and a sensing method. Background Art

[0002] In the field of robotic perception, tactile sensors have become an important key component for achieving human-robot collaboration and high-precision manipulation. Today's tactile sensors include capacitive tactile sensors, resistive tactile sensors, piezoelectric tactile sensors, triboelectric tactile sensors, and visual tactile sensors.

[0003] Research on tactile sensors can be broadly and narrowly defined. The broad definition of tactile sensory perception includes touch, pressure, force, slip, and thermal sensation. The narrow definition of tactile sensory perception includes the force sensation at the interface between the manipulator and the object. Based on their functional characteristics, tactile sensors can be broadly categorized as contact sensors, force-torque sensors, pressure sensors, and slip sensors.

[0004] The current tactile sensing technologies mainly include:

[0005] (1) Capacitive tactile array sensor. Its principle is that external force changes the relative displacement between the plates, thereby changing the capacitance. The tactile force is measured by detecting the change in capacitance.

[0006] Capacitive tactile sensors have a large measurement range, good linearity, and low manufacturing cost; however, they are physically large and difficult to integrate; they are easily affected by noise and have poor stability.

[0007] (2) Inductive tactile sensor. It uses the principle of electromagnetic induction to convert pressure into changes in the self-inductance and mutual inductance of the coil, which are then converted by the circuit into a voltage or current change output.

[0008] Inductive tactile sensors have low manufacturing costs and a wide measurement range. However, they are difficult to control magnetic field distribution, have low resolution, and have poor consistency across different contact points.

[0009] (3) Photoelectric tactile sensor. It is developed based on the principle of total internal reflection and is usually composed of a light source and a photodetector. When the pressure applied to the interface changes, the reflection intensity of the sensor's sensitive element and the frequency of the light source will also change accordingly.

[0010] Photoelectric tactile sensors offer high sensitivity, fast response, high spatial resolution, and minimal electromagnetic interference. However, they suffer from poor linearity when multiple forces act together, lack real-time data quality, and are difficult to calibrate.

[0011] (4) Piezoresistive tactile sensor. This device is made based on the piezoresistive effect of semiconductor materials. Its substrate can be directly used as a measuring sensor element, and the diffused resistors are connected in the substrate to form a bridge. When the substrate is deformed by external force, the resistance values will change, and the bridge will produce a corresponding unbalanced output.

[0012] The piezoresistive type has high sensitivity and strong overload tolerance; however, the varistor leakage current stability is poor; it is large in size and difficult to miniaturize; it has high power consumption; it is easily affected by noise; and the contact surface is fragile.

[0013] (5) Piezoelectric tactile sensor. Under the action of pressure, a potential difference appears between the two end surfaces of the piezoelectric material; conversely, mechanical stress is generated when voltage is applied.

[0014] Piezoelectric tactile sensors have a wide dynamic range and good durability, but are susceptible to thermal response effects.

[0015] Visual tactile sensors (such as GelSight and DIGIT) rely on high-resolution cameras to capture images of the deformation of a flexible membrane upon contact with an object, thereby analyzing the surface texture, shape, and contact force distribution. These sensors have found widespread application in robotic grasping, object recognition, and force feedback control, providing intuitive and rich tactile information for precise manipulation.

[0016] However, in actual scientific research applications, traditional visual-tactile sensors still have obvious limitations. Due to their reliance on optical imaging technology, these sensors can only detect surface features such as texture and shape in the contact area and are unable to effectively identify and judge the object's deeper information (internal structure, material properties). For example, traditional systems are unable to achieve imaging detection inside objects, nor can they monitor the state changes of liquids, solids, and other substances inside containers. This brings obvious limitations when robots perform more complex tasks (such as distinguishing between types of substances and identifying the state of substances in containers). In addition, most visual-tactile systems must actually contact the object to obtain valid data, which is particularly unfavorable for task prediction and safe interaction in complex environments in actual scientific research tasks, such as the precise grasping and manipulation of fragile or sensitive objects by robots.

[0017] In recent years, the scientific research field has actively explored multimodal sensing technologies to make up for the blind spots of traditional tactile sensing methods. One of the important research directions is to achieve "proximity perception" before contact, that is, to obtain the position and contour information of the object before it actually contacts it. Existing studies have attempted to use a unilateral transparent film structure on the visual tactile sensor, combined with an internal light source for imaging, to indirectly determine whether the object is close to the contact area. This method has expanded the perception distance to a certain extent, but it is more sensitive to external lighting conditions, and the imaging range is limited. It is difficult to work stably in textureless, dark or complex backgrounds. In addition, such systems usually use time-sharing to multiplex tactile and distance imaging on the same camera, which complicates light source control and data processing, making it difficult to respond in real time.

[0018] To further enhance the stability and depth of proximity perception, ultrasonic sensing technology has been introduced as an effective supplement. It enables precise non-contact distance measurement and analyzes the echo signal to reveal the material and internal structure of the object being measured, significantly expanding the robot system's understanding of the environment.

[0019] Despite this, there are still many technical challenges in achieving the organic combination of visual touch and ultrasonic sensing in actual scientific research scenarios. On the one hand, there are significant difficulties in integrated structural design. How to ensure that the visual sensor has clear imaging capabilities while achieving efficient propagation of ultrasonic signals requires an innovative layout of the sensor structure. On the other hand, the development of acousto-optic coupling materials also faces bottlenecks. Traditional acoustic impedance matching materials often do not have good optical transparency, which will directly affect the image quality. Therefore, there is an urgent need to develop a composite material system with excellent acoustic and optical properties. In addition, at the data processing level, the real-time fusion and intelligent switching mechanism of multimodal data also need to be solved. How to dynamically judge the contact state of the sensor according to different task requirements and quickly switch between non-contact and contact modes while maintaining high-precision and high-timeliness data processing capabilities is an important prerequisite for the practical application of this technical system.

[0020] There are several ways for traditional tactile sensors to achieve proximity sensing:

[0021] 1. Utilizing a translucent, controllable internally illuminated skin to achieve mode switching (visual or tactile). This skin is essentially a flexible, selectively light-transmitting membrane. Before contact, the sensor's internal light source is dimmed, allowing the camera to see external objects using ambient light. Computer vision algorithms are then used to determine the target's location and distance to the object. After contact, the sensor's internal light source is increased, obscuring the camera's view of the external object and allowing it to observe only the membrane's deformation, thus achieving tactile sensing.

[0022] 2. Synchronous acquisition of tactile and close-range perception is achieved through an RGB camera and an infrared time-of-flight (ToF) depth camera. The soft elastomer membrane allows the selective transmission of visible and infrared light, ensuring that tactile and close-range perception do not interfere with each other.

[0023] The first solution uses a translucent elastic membrane and controllable internal lighting switching mode to achieve environmental perception based on external light sources before contact and capture tactile images after contact. However, this solution is highly dependent on ambient lighting, resulting in reduced perception capabilities under low light or complex lighting conditions. At the same time, because images are captured through the elastic membrane, image quality often suffers from blurring and distortion, which severely limits image recognition effectiveness. The second solution uses a combination of RGB cameras and ToF depth cameras to simultaneously acquire tactile and distance information through sub-band imaging of visible light and infrared. Although this improves the sensing dimension, the system is large and complex in structure, making it unsuitable for compact end effectors.

[0024] While traditional visual tactile sensors offer high resolution and the ability to extract rich surface information, they still face several key challenges. First, they can only acquire data after contact occurs, lacking the "proximity perception" required before contact. Second, they cannot perceive the material properties or internal structure of the object being measured, severely limiting robots' ability to perform precise manipulation in complex environments. Some research has attempted to expand their sensing range through visual means, but due to limitations in lighting conditions and imaging methods, they still cannot meet the robust, real-time, and multimodal application requirements.

[0025] It should be noted that the information disclosed in the above background technology section is only used to understand the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0026] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide an ultrasonic enhanced tactile sensor system, sensor and sensing method.

[0027] To achieve the above object, the present invention adopts the following technical solutions:

[0028] In a first aspect of the present invention, an ultrasonic enhanced tactile sensor system comprises:

[0029] A multimodal sensing unit comprising an ultrasonic module and a visual-tactile perception module, wherein the ultrasonic module transmits ultrasonic waves and receives ultrasonic echo signals, and the visual-tactile perception module synchronously collects tactile deformation images; wherein the ultrasonic module and the visual-tactile perception module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure;

[0030] A processing unit configured with:

[0031] an event triggering module, detecting a contact event based on the tactile deformation image and generating a mode switching signal;

[0032] An adaptive processing module dynamically controls the ultrasonic working mode according to the mode switching signal: performs distance measurement in a non-contact state, and switches to an object interior imaging or material property recognition mode in a contact state;

[0033] The three-level fusion processing module performs signal-level synchronous acquisition, feature-level cross-modal fusion, and decision-level task switching, and outputs multimodal perception results.

[0034] In a second aspect of the present invention, an ultrasonic enhanced tactile sensor for use in the system comprises:

[0035] A multimodal sensing unit comprising an ultrasonic module and a visual-tactile perception module, wherein the ultrasonic module transmits ultrasonic waves and receives ultrasonic echo signals, and the visual-tactile perception module synchronously collects tactile deformation images; wherein the ultrasonic module and the visual-tactile perception module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure;

[0036] The visual tactile perception module comprises an acousto-optic composite matching layer, a flexible deformation sensing layer, an acoustic matching layer, and a light source and camera assembly stacked in sequence, and the image of the contact deformation of the flexible deformation sensing layer is captured by the light source and camera assembly;

[0037] The ultrasonic module comprises a piezoelectric ceramic component and an ultrasonic backing layer arranged behind the acousto-optic composite matching layer, and is used to transmit and receive ultrasonic signals;

[0038] The acousto-optic composite matching layer acts as a light-blocking layer of the visual-tactile sensing module, allowing the light source to illuminate the sensor and allowing ultrasonic waves to penetrate;

[0039] The acoustic impedance matching between the flexible deformation sensing layer and the air is achieved through the acousto-optic composite matching layer, and the acoustic impedance matching between the piezoelectric ceramic component and the flexible deformation sensing layer is achieved through the acoustic matching layer.

[0040] In a third aspect of the present invention, a multimodal sensing method based on the system comprises the following steps:

[0041] S1, non-contact stage: the ultrasonic module is activated to perform distance measurement, and the tactile module remains on standby;

[0042] S2, contact detection stage: analyzing the tactile deformation image, determining the contact event and generating a mode switching signal;

[0043] S3, mode switching stage: according to the piezoelectric ceramic configuration, switching the ultrasonic wave to the material recognition mode or internal imaging mode;

[0044] S4, multimodal fusion stage: synchronously collect tactile surface information and ultrasonic deep information, and output the perception results through a three-level fusion framework.

[0045] The present invention has the following beneficial effects:

[0046] This invention proposes an innovative ultrasonically enhanced tactile sensor system, sensor, and sensing method. Through structural design, acousto-optically transparent composite materials, and adaptive signal processing methods, a multimodal sensing system is constructed, capable of sensing proximity, surface texture, and contact force, as well as acquiring deep-level object information (internal imaging and material property identification). Structurally, piezoelectric ceramic components are deployed within the visual tactile module, supporting either single-point or array configurations. They can operate in either monostatic (shared transmit and receive) or bistatic (separate transmit and receive) modes, enabling multiple ultrasonic functions such as distance sensing, material identification, and internal imaging, expanding and integrating the capabilities of traditional visual tactile sensors. Regarding materials, acoustically matched material design is incorporated into the traditional visual tactile sensor structure to optimize the acoustic impedance matching between the flexible membrane and the air and solid interfaces, improving ultrasonic transmission efficiency while ensuring optical imaging performance. System control utilizes a mode switching mechanism triggered by tactile feedback. When contact is absent, ultrasonic distance measurement is performed. Upon contact, the sensing mode switches based on the piezoelectric ceramic configuration: a single-point configuration is used for ultrasonic material property detection, while an array configuration initiates ultrasonic imaging to acquire internal structural images, achieving comprehensive multimodal perception. This solution enhances the dimensionality and depth of perception capabilities, strengthening the sensor's adaptability to changing environments. It can be widely used in robotics research and engineering scenarios requiring non-contact prediction and high-precision grasping. The visual-tactile sensor uses a camera to capture the deformation of an elastic membrane (PDMS layer) to analyze surface information. Its image sequence output facilitates spatial alignment with sensing modalities such as ultrasound, providing a natural interface and technical foundation for co-structural integration, shared sensing areas, and multimodal information fusion.

[0047] The innovative contributions of the present invention include the introduction of acoustic matching design on the basis of the traditional visual tactile sensor structure without changing it, so as to achieve efficient ultrasonic transmission and clear optical imaging; the construction of an event-driven mechanism triggered by tactile feedback, which automatically switches the ultrasonic function to enhance the perception consistency and adaptability; the integration of multi-dimensional perception functions of proximity perception, surface texture analysis and deep information acquisition, covering the complete information chain from object prediction to contact recognition; and the support of multiple combinations of ultrasound and visual touch, which can be flexibly configured and expanded according to the scene. The present invention realizes non-contact distance perception and proximity judgment, obtains surface and deep information at the same time after contact, has a compact structure suitable for end effectors and supports real-time mode switching, and has good ultrasonic function scalability. Compared with traditional technologies, the present invention overcomes the limitations of traditional visual tactile sensors that lack the ability to predict before contact and cannot detect the internal information of objects, solves the acoustic matching problem of ultrasonic and tactile modal fusion, supports multiple ultrasonic working modes and builds a dynamic switching mechanism, and improves operational intelligence and environmental adaptability.

[0048] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the overall sensor according to an embodiment of the present invention.

[0050] Figure 2 The sensor structure arrangement example 1 according to the embodiment of the present invention is shown.

[0051] Figure 3 The sensor structure arrangement example 2 according to the embodiment of the present invention is shown.

[0052] Figure 4 The sensor structure arrangement example 3 of the embodiment of the present invention is shown.

[0053] Figure 5 The sensor structure arrangement example 4 according to the embodiment of the present invention is shown.

[0054] Figure 6 The sensor structure arrangement example 5 according to the embodiment of the present invention is shown.

[0055] Figure 7 Schematic diagram of the principle of achieving multimodal perception through structural reuse of sensor structure arrangement example 1.

[0056] Figure 8 4 is a system workflow diagram of an embodiment of the present invention.

[0057] Figure 9 This is a diagram of the data processing framework of the three-level fusion embodiment of the present invention.

[0058] Figure 10 4 is a simplified flowchart of the sensing method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope of the present invention and its application.

[0060] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and coupling or communication.

[0061] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0062] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0063] The present invention aims to propose a multimodal integrated sensing system that integrates visual, tactile and ultrasonic sensing functions, and constructs a multimodal sensing framework with the ability to perceive proximity, surface texture and contact force, as well as the ability to obtain deep information about objects (internal imaging and material property recognition).

[0064] The innovative contributions and important features of the present invention include: 1. There is no need to change the structure of the traditional visual tactile sensor, and the acoustic matching design is introduced to enable the system to have both efficient ultrasonic transmission and clear optical imaging capabilities. 2. An event trigger mechanism based on tactile feedback is constructed, so that the system can automatically switch the ultrasonic function, enhancing the consistency and adaptability of the perception process. 3. The system integrates multi-dimensional perception functions such as proximity perception, surface texture analysis and deep information acquisition, covering the complete information chain from object prediction to contact recognition, and providing unified data support for complex operation tasks. 4. The architecture supports multiple combinations of ultrasound + visual touch, without limiting the number of piezoelectric ceramics, installation position, frequency or flexible interface materials, and can be flexibly configured and expanded according to different application scenarios.

[0065] The embodiment of the present invention provides an ultrasonic enhanced tactile sensor system, comprising a multimodal sensing unit and a processing unit. The specific structure and implementation principle of the system can be found in Figures 1 to 10 .

[0066] The multimodal sensing unit includes an ultrasonic module and a visual-tactile perception module. The ultrasonic module transmits ultrasonic waves and receives ultrasonic echo signals, and the visual-tactile perception module synchronously collects tactile deformation images. The ultrasonic module and the visual-tactile perception module simultaneously realize ultrasonic acoustic impedance matching and optical imaging transmission through an acoustic-optical matching structure.

[0067] The processing unit is configured with an event triggering module, an adaptive processing module, and a three-level fusion processing module. The event triggering module detects contact events based on the tactile deformation image and generates a mode switching signal. The adaptive processing module dynamically controls the ultrasonic operating mode based on the mode switching signal: performing distance measurement in a non-contact state and switching to an internal object imaging or material property identification mode in a contact state. The three-level fusion processing module performs signal-level synchronous acquisition, feature-level cross-modal fusion, and decision-level task switching, outputting multimodal perception results.

[0068] The system realizes multi-dimensional perception of proximity, surface texture and deep information, which can significantly improve the robot's prediction and precise operation capabilities in complex scenarios.

[0069] See Figures 1 to 6 In some embodiments, the visual-tactile sensing module includes an acousto-optic composite matching layer 1, a flexible deformation sensing layer 2, an acoustic matching layer 4, a light source 5, and a camera 7 stacked in sequence. The light source 5 (such as an LED light panel) and the camera 7 capture images of the contact deformation of the flexible deformation sensing layer 2. The ultrasonic module includes a piezoelectric ceramic component 6 (such as a piezoelectric ceramic ring, a piezoelectric ceramic array, etc.) and an ultrasonic backing layer 8 arranged behind the acousto-optic composite matching layer 1, which are used to transmit and receive ultrasonic signals. The acousto-optic composite matching layer 1 serves as a light-blocking layer for the visual-tactile sensing module, allowing the light source 5 to illuminate the sensor and allowing ultrasonic waves to penetrate. The acousto-optic composite matching layer 1 achieves acoustic impedance matching between the flexible deformation sensing layer 2 and air, and the acoustic matching layer 4 achieves acoustic impedance matching between the piezoelectric ceramic component 6 and the flexible deformation sensing layer 2.

[0070] In some embodiments, the acousto-optic composite matching layer 1 is a hybrid film of hollow glass microspheres containing a dye and PDMS (referred to as a HGM-PDMS hybrid film). The inclusion of the hollow glass microspheres reduces the acoustic impedance difference, bringing the acoustic impedance closer to that of air. The flexible deformation sensing layer 2 is a PDMS layer. The acoustic matching layer 4 is an acrylic sheet, with a thickness designed based on a quarter-wavelength acoustic matching formula to optimize the transmission efficiency of ultrasound between the piezoelectric ceramic component 6 and the PDMS layer.

[0071] In some embodiments, the multimodal sensing unit further includes a multiplexed mechanical carrier, which provides a mounting cavity and integrally carries the visual-tactile perception module and the ultrasonic module.

[0072] The piezoelectric ceramic component 6 is arranged around the camera 7 (see Figures 2 to 4 ), or disposed between the hollow glass microsphere and PDMS mixed film (acoustic-optical composite matching layer 1) and the PDMS layer (flexible deformation sensing layer 2) (see Figures 5 and 6 ), sharing the spatial sensing area with the visual-tactile sensing module, achieving structural reuse of ultrasonic and tactile modalities. Specifically, the piezoelectric ceramic assembly 6 may include: (a) a ring-shaped piezoelectric ceramic unit, sleeved around the periphery of the camera 7 and fixed to the lower fixed housing 9; or (b) an array of piezoelectric ceramic units, embedded between the HGM-PDMS hybrid film and the PDMS layer, or around the camera 7. The piezoelectric ceramic assembly 6 supports monostatic mode (shared transmission and reception) or bistatic mode (separate transmission and reception).

[0073] like Figures 1 to 6 As shown, in some embodiments, the reusable mechanical carrier includes an upper fixed shell 3 and a lower fixed shell 9, the upper fixed shell 3 is fixedly connected to the acrylic sheet and the PDMS layer, and the hollow glass microbeads and PDMS mixed film is arranged on the surface of the upper fixed shell 3; the lower fixed shell 9 fixes the camera 7 in the center, and the piezoelectric ceramic component 6 and the ultrasonic backing layer 8 are arranged on the periphery, and the bottom is externally connected to the LED light board; wherein, the upper fixed shell 3 and the lower fixed shell 9 are assembled to form an installation cavity.

[0074] In some embodiments, the mode switching mechanism of the adaptive processing module of the processing unit specifically includes: in a non-contact state, turning off tactile image acquisition and starting ultrasonic distance measurement; when a contact event is triggered, entering the corresponding mode according to the configuration of the piezoelectric ceramic: (a) internal object imaging mode under array configuration, performing high-resolution ultrasonic scanning to generate an image of the internal structure of the object; (b) material property identification mode under single-point configuration, extracting and analyzing echo frequency domain features to identify material properties.

[0075] In some embodiments, the three-level fusion processing module of the processing unit specifically includes: a signal-level fusion unit, which synchronously collects tactile images and ultrasonic echo signals and shares a spatial perception area; a feature-level fusion unit, which extracts tactile texture features through a neural network and combines the ultrasonic echo spectrum analysis results to perform cross-modal feature splicing; a decision-level fusion unit, which dynamically selects the ultrasonic task mode based on the contact event, outputs distance information when there is no contact, and fuses the tactile surface information and ultrasonic deep information after contact to output the object recognition result.

[0076] In a further embodiment, the processing mechanism of the decision-level fusion unit specifically includes: when there is no contact, the ultrasonic mode outputs proximity distance information; after the contact event is triggered, the ultrasonic mode switches to material recognition or internal imaging mode; and the tactile texture features and ultrasonic material / structure features are integrated to generate the final decision through a weighted or voting mechanism.

[0077] See Figures 1 to 7 An embodiment of the present invention further provides an ultrasonic enhanced tactile sensor for the tactile sensor system, comprising a multimodal sensing unit, wherein the multimodal sensing unit includes an ultrasonic module and a visual tactile perception module, wherein the ultrasonic module transmits ultrasonic waves and receives ultrasonic echo signals, and the visual tactile perception module synchronously collects tactile deformation images; wherein the ultrasonic module and the visual tactile perception module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure. As mentioned above, specifically, the visual-tactile perception module includes an acousto-optic composite matching layer 1, a flexible deformation sensing layer 2, an acoustic matching layer 4, a light source 5 and a camera 7 stacked in sequence, and the image of the contact deformation of the flexible deformation sensing layer 2 is captured by the light source 5 (such as an LED light board) and the camera 7; the ultrasonic module includes a piezoelectric ceramic component 6 (such as a piezoelectric ceramic ring, a piezoelectric ceramic array, etc.) and an ultrasonic backing layer 8 arranged behind the acousto-optic composite matching layer 1, which are used to transmit and receive ultrasonic signals; wherein the acousto-optic composite matching layer 1 serves as a light-blocking layer of the visual-tactile sensing module, allowing the light source 5 to illuminate the sensor and allowing ultrasonic waves to penetrate; wherein, the acoustic impedance matching of the flexible deformation sensing layer 2 with the air is achieved through the acousto-optic composite matching layer 1, and the acoustic impedance matching of the piezoelectric ceramic component 6 with the flexible deformation sensing layer 2 is achieved through the acoustic matching layer 4.

[0078] An embodiment of the present invention further provides a multimodal sensing method based on the tactile sensor system, comprising the following steps:

[0079] Step S1, non-contact stage: activating the ultrasonic module to perform distance measurement, and the tactile module remains in standby;

[0080] Step S2, contact detection phase: analyzing the tactile deformation image, determining the contact event and generating a mode switching signal;

[0081] Step S3, mode switching stage: switching the ultrasonic wave to the material recognition mode or the internal imaging mode according to the configuration of the piezoelectric ceramic;

[0082] Step S4, multimodal fusion stage: synchronously collect tactile surface information and ultrasonic deep information, and output the perception results through a three-level fusion framework.

[0083] Figures 8 to 10 The principle and specific implementation process of the multimodal sensing method are shown.

[0084] Specific embodiments of the present invention are further described below.

[0085] This paper proposes a multimodal integrated sensing system that integrates visual, tactile, and ultrasonic sensing capabilities. This system creates a multimodal sensing framework that combines proximity sensing, surface texture, and contact force sensing, with the ability to acquire deeper object information (internal imaging and material property identification). Structurally, the sensor deploys piezoelectric ceramic units within the visual-tactile module, allowing for either a single-site configuration or an array-based distribution. Depending on the task requirements, the piezoelectric ceramics can operate flexibly in either single-base (shared transmit and receive) or dual-base (separate transmit and receive) modes, supporting a variety of ultrasonic functions, including distance sensing, ultrasonic object property detection, and non-contact imaging. This expands and integrates the capabilities of traditional visual-tactile sensors. At the material level, the sensor incorporates an acoustically matched material design based on the existing structure, improving ultrasonic propagation efficiency while ensuring uncompromising imaging quality. The system incorporates an event-driven mechanism based on tactile feedback. In the non-contact state, the sensor defaults to distance measurement, automatically switching to more complex sensing tasks upon contact, including texture analysis, object property detection, or imaging. This overall enhances adaptive intelligence and perception capabilities for multi-stage tasks.

[0086] This invention integrates ultrasound functionality into a visual-tactile sensor. The sensor comprises an acoustic-optical composite matching layer 1 (which can be a silicone or PDMS mixed dye) acting as a light-blocking film, a flexible deformation sensing layer 2 (which can be a silicone or PDMS layer), a fixed housing, an acrylic sheet (acoustic matching layer 4), an LED light panel (light source 5), and a camera 7. The core component of the ultrasound function is a piezoelectric ceramic assembly 6, which can be embedded around the camera 7 or between the light-blocking film and the flexible deformation sensing layer 2. If the piezoelectric ceramic assembly 6 is positioned around the camera 7, its acoustic propagation path is through the acrylic sheet, the flexible deformation sensing layer 2, the light-blocking film, air, or an object. If it is positioned between the light-blocking film and the flexible deformation sensing layer 2, its propagation path is through the light-blocking film, air, or an object. The acoustic impedances of the piezoelectric ceramic assembly 6, the acrylic sheet, the flexible deformation sensing layer 2, the light-blocking film, and air differ significantly. According to the laws of acoustic propagation, a large number of sound waves are reflected at the interfaces, resulting in low acoustic propagation efficiency. In this regard, the present invention further improves the structure of the visual and tactile parts, such as the material composition of the light-blocking film, the thickness of the acrylic sheet, etc., and accordingly forms structures such as the acousto-optic composite matching layer 1 and the acoustic matching layer 4.

[0087] The acousto-optic composite matching layer 1 is preferably a hybrid film of hollow glass microspheres containing a dye and PDMS (hereinafter referred to as HGM-PDMS film). It has two functions: first, it serves as a light-blocking layer for the visual tactile sensor, allowing the sensor's internal light source to illuminate the sensor, facilitating camera 7's observation of elastic layer deformation and acquiring tactile modal information while preventing ambient light from interfering with the sensor's camera 7 imaging; second, it serves as an acoustic matching layer between the PDMS and air. By incorporating hollow glass microspheres into a layer of PDMS, the acoustic impedance difference between the PDMS and air is reduced, thereby improving the acoustic propagation efficiency of ultrasound waves.

[0088] In this invention, the acrylic layer not only supports the deformation of the PDMS layer but also serves as an acoustic matching layer. Based on the formula for a quarter-wavelength propagation of an acoustic propagation matching layer, an acrylic sheet of a specific thickness is customized to serve as an acoustic matching layer for ultrasound between the piezoelectric ceramic component 6 and the PDMS layer.

[0089] Figure 1 The overall structure of the sensor of the present invention is shown. The sensor structure layout of various embodiments is as follows:

[0090] like Figure 2As shown, the piezoelectric ceramic assembly 6, a piezoelectric ceramic ring, is deployed between the LED light board and camera 7, embedded around the camera 7. The system components, from top to bottom, are: a hollow glass microsphere and PDMS hybrid membrane (hereinafter referred to as HGM-PDMS membrane), a PDMS layer, an upper fixed housing 3, an acrylic sheet, an LED light board, camera 7, a piezoelectric ceramic assembly 6, a lower fixed housing 9, and an ultrasonic backing layer. It can be divided into two parts, upper and lower.

[0091] like Figure 3 and Figure 4 As shown, the piezoelectric ceramic assembly 6, a piezoelectric ceramic array, is deployed between the LED light board and camera 7, embedded around the camera 7. The system components, from top to bottom, are: HGM-PDMS membrane, PDMS layer, upper fixed housing 3, acrylic sheet, LED light board, camera 7, piezoelectric ceramic assembly 6, lower fixed housing 9, and ultrasonic backing layer. It can be divided into two parts, upper and lower.

[0092] like Figure 5 and Figure 6 As shown, the piezoelectric ceramic assembly 6 uses a piezoelectric ceramic array and is deployed between the HGM-PDMS membrane and the PDMS layer. The system components, from top to bottom, are: HGM-PDMS membrane, PDMS layer, upper fixed housing 3, acrylic sheet, LED light board, camera 7, piezoelectric ceramic assembly 6, lower fixed housing 9, and ultrasonic backing layer. It can be divided into two parts.

[0093] The sensor structure arrangement is not limited to the above-mentioned method. In fact, the piezoelectric ceramic component 6 can be arranged at any position of the sensor. It is only necessary to ensure that the acoustic matching work is fully considered in the acoustic propagation path of the piezoelectric ceramic. According to the acoustic impedance of the component in the structure, the thickness of the component is adjusted or other materials are added to achieve efficient ultrasonic propagation, thereby realizing the fusion of ultrasound with visual touch.

[0094] Figure 2 The sensor structure layout example 1 shown in the figure shows a typical ultrasonic + tactile multimodal sensor structure. The following uses the sensor structure layout example 1 as an example to introduce the sensor components and their functions in detail:

[0095] The upper part consists of a HGM-PDMS hybrid film, a PDMS layer, an upper fixed shell 3, and an acrylic sheet. The upper fixed shell 3 is hollow, with an acrylic sheet adhered to the bottom. PDMS is cast on the fixed shell, and a light-blocking layer is spin-coated on the cured PDMS.

[0096] The lower part consists of an LED light board, a camera 7, a piezoelectric ceramic ring, an ultrasonic backing layer 8, and a lower fixed shell 9. The center of the lower fixed shell 9 fixes the camera 7, the piezoelectric ceramic ring is sleeved on the camera 7, and the backing material is cast in the gap. The LED light board is placed on the outside of the shell.

[0097] The HGM-PDMS hybrid film has two functions: allowing ultrasound to penetrate and deforming to provide visual information. First, it acts as a light-blocking layer for the visual tactile sensor, allowing the sensor's internal light source to illuminate the sensor, allowing the camera to observe the deformation of the elastic layer and obtain tactile modal information while preventing external ambient light from affecting the sensor's camera imaging. Second, it acts as an acoustic matching layer between PDMS and air. By incorporating hollow glass microbeads into a layer of PDMS, the acoustic impedance difference between PDMS and air is reduced, thereby improving the acoustic propagation efficiency of ultrasound.

[0098] The PDMS layer is a component of the visual tactile sensor, used to generate deformation when in contact with an object and capture tactile modal information. The upper fixed shell 3 provides mechanical protection for the sensor. The acrylic sheet is a component of the visual tactile sensor, used to support the deformation of the PDMS layer, and also acts as an acoustic matching layer. Based on the formula for the quarter-wavelength propagation of the acoustic propagation matching layer, an acrylic sheet of a specific thickness is customized to make it an acoustic matching layer for ultrasonic waves between the piezoelectric ceramic component 6 and the PDMS layer. The LED light board serves as the light source 5 inside the visual tactile sensor. The camera 7 is used to observe the deformation of the PDMS layer and obtain tactile modal information. The piezoelectric ceramic ring serves as the ultrasonic emission source and operates in a single-base mode. The lower fixed shell 9 provides mechanical protection for the sensor. The ultrasonic backing layer 8 is a component of the ultrasonic sensor, which reduces the signal tailing of the ultrasonic emission wave and improves the working performance of the ultrasonic wave.

[0099] like Figure 7 As shown in the figure, the sensor components realize ultrasound + tactile fusion perception through structural reuse.

[0100] This paper proposes an event-driven ultrasonic mode switching mechanism. Using "contact event" signals generated by the visual-tactile module, the ultrasonic module automatically switches from a non-contact distance measurement mode to a contact mode for internal imaging or object property identification. This mechanism, combining hardware triggering with software control, enables dynamic functional reconfiguration of multimodal sensors at different task stages.

[0101] The whole process can be divided into the following stages:

[0102] Non-contact ultrasonic distance sensing: In the initial stage, before the sensor contacts an object, the system activates only the ultrasonic module, which continuously transmits ultrasonic signals and receives echoes. The system measures the propagation time of the sound waves to determine the relative position of the target object. Depending on the number of piezoelectric ceramics, the ultrasonic mode can operate in either monostatic or bistatic mode. Monostatic mode requires only one piezoelectric ceramic, reducing costs; bistatic mode reduces blind spots and offers a wider proximity sensing range. During this time, the tactile image module remains in a low-power standby state and does not participate in information collection, reducing system burden.

[0103] Contact event detection and mode switching triggering: When the sensor contacts the target object, the flexible imaging layer in the tactile module will undergo significant deformation. The system determines whether the triggering conditions for the contact event have been met by analyzing the amplitude and structural changes of the image deformation. Once the contact criteria are met, the tactile image path immediately sends a "contact event signal" to the main control system. This signal serves as a transition condition in the state machine, triggering the subsequent mode switching process, allowing the ultrasonic mode to obtain deep information about the object. Furthermore, depending on whether the sensor has a piezoelectric ceramic array or a single piezoelectric ceramic, the ultrasonic mode can operate in an internal imaging mode or an object material property identification mode.

[0104] Ultrasonic internal object imaging in contact: Upon receiving a tactile trigger signal, the system immediately switches the operating parameters of the ultrasonic module and enters internal object imaging mode. In this mode, the piezoelectric ceramic array is activated, and the system adjusts the ultrasonic emission parameters to achieve high-resolution scanning of the target area. By performing time-frequency analysis and feature extraction on the echo signal, the internal structural distribution of the object can be obtained. At the same time, the visual and tactile modules continue to operate, collecting external information such as texture deformation and geometric contours on the object's surface. The two types of perception modules structurally share the same acquisition area and are precisely aligned in space, providing a consistent foundation for subsequent internal and external information fusion and three-dimensional reconstruction.

[0105] Ultrasonic object material property identification during contact: Upon receiving a tactile trigger signal, the system immediately switches the operating parameters of the ultrasonic module and enters material property identification mode. In this mode, the system performs frequency domain transformation and feature extraction on the received ultrasonic echo signal to identify the object's material information. Simultaneously, the visual-tactile module is also active, collecting information such as the object's surface texture and shape. The two modules share a hardware acquisition area and are spatially aligned in their structural design to facilitate subsequent information fusion.

[0106] Information Fusion and Subsequent Operation Interface: After completing the mode switch, the system synchronously outputs sensory data from both ultrasonic and visual tactile sensors for use by the upper-level control system to support actions such as grasping strategy adjustment, object classification, and path correction. This switching mechanism is implemented in real time by the embedded processor, using a periodic timer and interrupt mechanism to achieve multi-threaded synchronization, ensuring excellent real-time and scalability.

[0107] This event-driven mode switching mechanism simplifies the workflow of multimodal perception systems, enabling more efficient coordination between perception tasks and mechanical actions. It is particularly well-suited for robotic hands implementing control logic that prioritizes judgment and manipulation when handling unknown objects. The system boasts a compact structure, high modular integration, and a clear signal acquisition and processing pathway. It offers excellent portability and compatibility, enabling easy integration into the end-user structure of multi-jointed dexterous hands. The sensor itself can be deployed directly on the fingertips or palm, enabling integration without modifying the manipulator's structure. The system actively provides distance and positioning information during the non-contact phase and automatically switches to a high-precision recognition mode upon contact, sensing the object's texture and material properties. This mechanism enables the dexterous hand to dynamically switch perception modes based on the stage of the task, significantly enhancing operational autonomy, intelligence, and environmental adaptability.

[0108] Figure 8 This system workflow diagram shows an embodiment of the present invention. The system is capable of simultaneously analyzing both surface details and underlying information, providing a unified platform for multimodal intelligent perception and manipulation. It has practical application value in various scenarios, including industrial inspection, flexible gripping, and biological tissue analysis.

[0109] The present invention constructs a three-level fusion data processing framework for coordinating the information collection and processing of the two modalities of vision, touch and ultrasound. The system is divided into three levels according to its function: "signal level fusion", "feature level fusion" and "decision level fusion". Figure 9 shown.

[0110] Signal-level fusion: The system uses a synchronization mechanism to simultaneously collect two types of raw signals within the sensor hardware: a visual image from the camera, which records the deformation of the elastic membrane during contact; and an ultrasonic echo signal from the piezoelectric ceramic, which reveals deeper information such as the internal structure of the target object or its material properties. These two signals are collected in sync and share the same spatial sensing area, ensuring a consistent foundation for subsequent information fusion.

[0111] Feature-level fusion: This stage primarily involves data preprocessing. The visual path uses neural networks to extract high-dimensional image features such as texture and shape. The ultrasonic path performs spectral analysis on the echo signal to extract characteristic parameters such as energy distribution and spectral shape that reflect material differences. The extracted feature vectors are concatenated, aligned, or weightedly fused using an attention mechanism to provide rich data representation for higher-level recognition tasks.

[0112] Decision-level fusion: This stage is a key component of the system. Unlike typical static fusion structures, this invention dynamically controls the ultrasonic operating mode through a touch event detection mechanism based on tactile images. During the pre-contact phase, the system defaults to "ranging mode," providing spatial location information for the target object. Upon detecting a touch event, the system immediately switches the ultrasonic mode to "material recognition mode," activating specific signal acquisition and analysis parameters to extract deep features for classification. The decision-level further fuses the recognition results of the two paths (e.g., through weighting or voting) to output the final object recognition or state determination result.

[0113] This three-level structure decouples signal acquisition, feature analysis and task decision-making and processes them collaboratively, giving the system the advantages of clear structure, strong scalability and high processing efficiency. It is particularly suitable for the perception needs of dexterous robotic hands to perform high-precision, multi-stage operation tasks on complex objects.

[0114] Figure 10 This is a simplified flowchart of the sensing method according to an embodiment of the present invention, showing that ultrasound and tactile information are closely linked and work together to achieve multimodal perception.

[0115] In summary, the present invention proposes a multimodal integrated sensing system and sensor that integrates visual, tactile, and ultrasonic sensing functions. Its main innovative contributions and features are: introducing an acoustic matching design without changing the structure of traditional visual tactile sensors, and achieving synergy between efficient ultrasonic transmission and clear optical imaging capabilities by optimizing the acoustic impedance matching between the flexible membrane and the air and solid interfaces; constructing an event trigger mechanism based on tactile feedback, so that the system can automatically switch the ultrasonic function according to the contact state, perform distance measurement in a non-contact state, and after contact, switch to material property identification or object internal imaging mode according to the single-point or array configuration of the piezoelectric ceramics, thereby enhancing the consistency and adaptability of the sensing process; the system integrates multidimensional sensing functions such as proximity sensing, surface texture analysis, and deep information acquisition, covering the complete information chain from object prediction to contact recognition, providing unified data support for complex operation tasks; the architecture supports multiple combinations of ultrasound and visual tactile, without restrictions on the number of piezoelectric ceramics, installation location, or flexible interface material, and can be flexibly configured and expanded according to different scenarios.

[0116] The main functions of this ultrasonic-enhanced tactile sensor system include: realizing distance perception and proximity judgment of target objects in a non-contact state, synchronously obtaining surface texture information and internal structure images or material property identification information of the object after contact, and having tactile and ultrasonic dual-modal perception capabilities; the compact structure is suitable for integration into the end effector and supports real-time working mode switching; it has good ultrasonic function scalability, and realizes internal structure imaging and material property analysis through the flexible deployment of piezoelectric ceramics, which is suitable for complex object recognition.

[0117] Compared with traditional technologies, the significant advantages of the present invention are reflected in: making up for the lack of pre-contact prediction capability of traditional visual tactile sensors and realizing non-contact distance perception; breaking through the limitation of traditional tactile inability to detect deep information of objects and expanding the perception depth; solving the acoustic matching problem in the fusion of ultrasound and tactile modalities to ensure that optical imaging is not affected; supporting multiple ultrasound working modes such as single base and dual base to adapt to different precision requirements; automatically adjusting the perception strategy according to the contact status through a dynamic switching mechanism, and improving the adaptability under multi-stage tasks.

[0118] The outstanding beneficial effects of the present invention include: enhancing the predictive ability by acquiring non-contact distance information through the ultrasonic module, realizing the integrated perception of the surface texture and the internal and external information of the ultrasonic echo in the contact state; the compact design is adapted to the end effector and supports real-time mode switching; a variety of ultrasonic working modes are adapted to diverse task requirements, covering the recognition and evaluation of objects with complex structures; the dynamic conversion mechanism of ultrasonic modes based on tactile triggering improves operational intelligence and environmental adaptability.

[0119] The above description further details the present invention in conjunction with specific / preferred embodiments, and the specific implementation of the present invention should not be construed as being limited to these descriptions. Persons skilled in the art will appreciate that, without departing from the spirit of the present invention, they may make various substitutions or modifications to the described embodiments, and these substitutions or modifications should be considered to fall within the scope of protection of the present invention. Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "preferred embodiments," "examples," "specific examples," or "some examples" indicates that the specific features, structures, materials, or characteristics described in conjunction with such embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Persons skilled in the art may combine and assemble the different embodiments or examples described in this specification, as well as features of different embodiments or examples, without conflicting opinions. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications may be made herein without departing from the scope of protection of the patent application.

Claims

1. An ultrasonically enhanced tactile sensor system, characterized in that: include: A multimodal sensing unit comprising an ultrasonic module and a visual-tactile perception module, wherein the ultrasonic module transmits ultrasonic waves and receives ultrasonic echo signals, and the visual-tactile perception module synchronously collects tactile deformation images; wherein the ultrasonic module and the visual-tactile perception module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure; A processing unit configured with: an event triggering module, detecting a contact event based on the tactile deformation image and generating a mode switching signal; An adaptive processing module dynamically controls the ultrasonic working mode according to the mode switching signal: performs distance measurement in a non-contact state, and switches to an object interior imaging or material property recognition mode in a contact state; The three-level fusion processing module performs signal-level synchronous acquisition, feature-level cross-modal fusion, and decision-level task switching to output multimodal perception results. The visual tactile perception module comprises an acousto-optic composite matching layer, a flexible deformation sensing layer, an acoustic matching layer, and a light source and camera assembly stacked in sequence, and the image of the contact deformation of the flexible deformation sensing layer is captured by the light source and camera assembly; The ultrasonic module comprises a piezoelectric ceramic component and an ultrasonic backing layer arranged behind the acousto-optic composite matching layer, and is used to transmit and receive ultrasonic signals; The acousto-optic composite matching layer acts as a light-blocking layer for the visual-tactile sensing module, allowing the light source to illuminate the sensor and allowing ultrasonic waves to penetrate. The acoustic impedance matching between the flexible deformation sensing layer and the air is achieved through the acousto-optic composite matching layer, and the acoustic impedance matching between the piezoelectric ceramic component and the flexible deformation sensing layer is achieved through the acoustic matching layer.

2. The tactile sensor system according to claim 1, wherein: The acousto-optic composite matching layer is a mixed film of hollow glass microspheres and PDMS. The acousto-optic composite matching layer reduces the acoustic impedance difference by incorporating hollow glass microspheres, making the acoustic impedance close to that of air. The flexible deformation sensing layer is a PDMS layer; The acoustic matching layer is an acrylic sheet, the thickness of which is designed based on a quarter-wavelength acoustic matching formula and is used to optimize the transmission efficiency of ultrasound between the piezoelectric ceramic component and the PDMS layer.

3. The tactile sensor system according to claim 2, wherein: The multimodal sensing unit further comprises: A multiplexed mechanical carrier, which provides a mounting cavity and integrates and carries the visual and tactile perception module and the ultrasonic module; The piezoelectric ceramic component is deployed around the camera or between the hollow glass microspheres and PDMS mixed film and the PDMS layer, sharing the spatial sensing area with the visual tactile sensing module to achieve structural multiplexing of ultrasonic and tactile modes.

4. The tactile sensor system according to claim 3, wherein: The multiplexed mechanical carrier comprises: An upper fixed shell is fixedly connected to the acrylic sheet and the PDMS layer, and a mixed film of hollow glass microbeads and PDMS is provided on the surface of the upper fixed shell; The lower part is fixed with the outer shell, the center is fixed with the camera, the outer periphery is provided with piezoelectric ceramic components and ultrasonic backing layer, and the bottom is externally connected with an LED light board; Wherein, the upper fixed shell and the lower fixed shell are assembled to form a mounting cavity.

5. The tactile sensor system according to any one of claims 1 to 4, characterized in that: The mode switching mechanism of the adaptive processing module includes: In the non-contact state, tactile image acquisition is turned off and ultrasonic distance measurement is started; When a contact event is triggered, the corresponding mode is entered according to the configuration of the piezoelectric ceramic: (a) Internal imaging mode in array configuration, performing high-resolution ultrasound scanning to generate images of the internal structure of an object; (b) Material property identification mode under single-point configuration, extracting and analyzing echo frequency domain features to identify material properties.

6. The tactile sensor system according to any one of claims 1 to 4, characterized in that: The three-level fusion processing module includes: A signal-level fusion unit that synchronously collects tactile images and ultrasonic echo signals and shares a spatial perception area; Feature-level fusion unit, which extracts tactile texture features through a neural network and combines them with ultrasonic echo spectrum analysis results for cross-modal feature splicing; The decision-level fusion unit dynamically selects the ultrasonic task mode based on the contact event, outputs distance information when there is no contact, and fuses the tactile surface information with the ultrasonic deep information to output the object recognition result after contact.

7. The tactile sensor system according to claim 6, wherein: The processing mechanism of the decision-level fusion unit includes: When there is no contact, the ultrasonic mode outputs proximity distance information; After a contact event is triggered, the ultrasound mode switches to material identification or internal imaging mode; The tactile texture features are integrated with the ultrasonic material or structural features to generate the final decision.

8. An ultrasonic enhanced tactile sensor for use in the tactile sensor system according to any one of claims 1 to 7, characterized in that: include: A multimodal sensing unit comprising an ultrasonic module and a visual-tactile perception module, wherein the ultrasonic module transmits ultrasonic waves and receives ultrasonic echo signals, and the visual-tactile perception module synchronously collects tactile deformation images; wherein the ultrasonic module and the visual-tactile perception module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure; The visual tactile perception module comprises an acousto-optic composite matching layer, a flexible deformation sensing layer, an acoustic matching layer, and a light source and camera assembly stacked in sequence, and the image of the contact deformation of the flexible deformation sensing layer is captured by the light source and camera assembly; The ultrasonic module comprises a piezoelectric ceramic component and an ultrasonic backing layer arranged behind the acousto-optic composite matching layer, and is used to transmit and receive ultrasonic signals; The acousto-optic composite matching layer acts as a light-blocking layer for the visual-tactile sensing module, allowing the light source to illuminate the sensor and allowing ultrasonic waves to penetrate. The acoustic impedance matching between the flexible deformation sensing layer and the air is achieved through the acousto-optic composite matching layer, and the acoustic impedance matching between the piezoelectric ceramic component and the flexible deformation sensing layer is achieved through the acoustic matching layer.

9. A multimodal sensing method based on the tactile sensor system according to any one of claims 1 to 7, characterized in that: The steps include: S1, non-contact stage: the ultrasonic module is activated to perform distance measurement, and the tactile module remains on standby; S2, contact detection stage: analyzing the tactile deformation image, determining the contact event and generating a mode switching signal; S3, mode switching stage: according to the piezoelectric ceramic configuration, switching the ultrasonic wave to the material recognition mode or internal imaging mode; S4, multimodal fusion stage: synchronously collect tactile surface information and ultrasonic deep information, and output the perception results through a three-level fusion processing module.

Images