Ultrasonic enhanced tactile sensor system, sensor and sensing method

By combining the multimodal sensor system of ultrasonic module and visual haptic module, the problem that traditional visual haptic sensors cannot obtain deep information of objects is solved, distance perception in non-contact state and multi-dimensional perception in contact state are realized, and the operation ability of the robot in complex environments is improved.

CN120352054AActive Publication Date: 2025-07-22TSINGHUA SHENZHEN INTERNATIONAL GRADUATE SCHOOL

Patent Information

Application Number
CN202510829945.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-20
Publication Date
2025-07-22
Estimated Expiration
2045-06-20

AI Technical Summary

Technical Problem

Traditional visual haptic sensors cannot obtain deep information of objects in non-contact states, and their perception capabilities are limited in complex environments, so they cannot effectively identify the internal structure and material properties of objects.

Method used

A multimodal sensor system that combines ultrasonic modules and visual haptic modules is adopted to realize ultrasonic acoustic impedance matching and optical imaging transmission through the acousto-optical matching structure. Combined with the event trigger module and the adaptive processing module, the ultrasonic working mode is dynamically switched to realize the internal imaging of the object in the contact state or the identification of material properties in the object in the contact state.

Benefits of technology

It realizes the distance perception of objects in non-contact state and the acquisition of surface texture and deep information in contact state, improves the robot's prediction and precise operation capabilities in complex scenarios, and enhances the adaptability and real-time perception.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120352054A_ABST
    Figure CN120352054A_ABST
Patent Text Reader

Abstract

The invention discloses an ultrasonic enhanced tactile sensor system, a sensor and a sensing method. The tactile sensor system comprises a multi-mode sensing unit and a processing unit. The multi-mode sensing unit integrates an ultrasonic module and a visual tactile sensing module, ultrasonic acoustic impedance matching and optical imaging transmission are achieved through an acousto-optic matching structure, and ultrasonic echo signals and tactile deformation images can be synchronously collected. In the processing unit, an event triggering module detects a contact event based on a tactile image and generates a switching signal, and a self-adaptive processing module dynamically controls an ultrasonic mode, distance measurement during non-contact, and switching to object internal imaging or material property identification during contact according to the switching signal. And the three-level fusion processing module outputs a multi-modal sensing result through signal-level synchronous acquisition, feature-level cross-modal fusion and decision-level task switching. According to the system, multi-dimensional perception of proximity sense, surface texture and deep information is realized, and the pre-judgment and accurate operation capability of the robot in a complex scene is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robot perception, and particularly to an ultrasonic-enhanced tactile sensor system, a sensor, and a sensing method. Background Art

[0002] In the field of robot perception, tactile sensors have become important key components for realizing human-robot collaboration and high-precision operation. Current tactile sensors include capacitive tactile sensors, resistive tactile sensors, piezoelectric tactile sensors, triboelectric tactile sensors, and visual tactile sensors, etc.

[0003] The research on tactile sensors has two categories: broad sense and narrow sense. Tactile in the broad sense includes touch, pressure sense, force sense, slip sense, cold and heat sense, etc. Tactile in the narrow sense includes the force sensation on the contact surface between the manipulator and the object. Classified from the perspective of function, tactile sensors can be roughly divided into contact sensors, force-torque sensors, pressure sensors, and slip sensors, etc.

[0004] Current tactile sensing technologies mainly include: (1) Capacitive tactile array sensors. Its principle is that an external force causes a change in the relative displacement between the plates, thereby changing the capacitance, and the tactile force is measured by detecting the change in capacitance.

[0005] Capacitive tactile sensors have a large measurement range; good linearity; low manufacturing cost; but large physical size, difficult to integrate; easily affected by noise, and poor stability.

[0006] (2) Inductive tactile sensors. It uses the principle of electromagnetic induction to convert the pressure action into changes in the self-inductance coefficient and mutual inductance coefficient of the coil, and then converts it into a change in voltage or current output by the circuit.

[0007] Inductive tactile sensors have low manufacturing cost; large measurement range. But the magnetic field distribution is difficult to control, with low resolution; poor consistency at different contact points.

[0008] (3) Photoelectric tactile sensors. It is developed based on the principle of total internal reflection and usually consists of a light source and a photodetector. When the pressure applied to the interface changes, the reflection intensity and light source frequency of the sensor's sensitive element will also change accordingly.

[0009] Photoelectric tactile sensors have high sensitivity; fast response; high spatial resolution; less affected by electromagnetic interference. But when multiple forces act together, the linearity is poor; the data real-time performance is poor; and the calibration is difficult.

[0010] (4)Piezo-resistive tactile sensor. It is a device made based on the piezoresistive effect of semiconductor materials. Its substrate can directly serve as the measurement and sensing element, and the diffused resistors are connected in a bridge form within the substrate. When the substrate is deformed under an external force, the resistance values of each resistor will change, and the bridge will produce a corresponding unbalanced output.

[0011] The piezo-resistive type has high sensitivity; strong overload tolerance; but the leakage current stability of the piezoresistor is poor; large volume, not easy to miniaturize; high power consumption; vulnerable to noise; and the contact surface is fragile.

[0012] (5)Piezoelectric tactile sensor. Under the action of pressure, a potential difference appears between the two end faces of the piezoelectric material; conversely, applying a voltage generates mechanical stress.

[0013] The piezoelectric tactile sensor has a wide dynamic range; good durability. But it is easily affected by the thermal response effect.

[0014] Vision-based tactile sensors (such as GelSight, DIGIT, etc.) mainly rely on high-resolution cameras to capture the deformed images of the flexible film after touching an object, so as to analyze the texture, shape and contact force distribution of the object surface. Such sensors have been widely used in robot grasping, object recognition and force feedback control, etc., and provide intuitive and rich tactile information for realizing fine manipulation.

[0015] However, in the actual scientific research application process, traditional vision-based tactile sensors still have obvious limitations. Due to relying on optical imaging technology, such sensors can only detect the surface texture, shape and other features of the contact area, and cannot effectively identify and judge the deep information of the object (internal structure, material properties). For example, traditional systems cannot achieve internal imaging detection of objects, nor can they monitor the state changes of substances such as liquids and solids inside containers. This brings obvious limitations when robots perform more complex tasks (such as distinguishing the types of substances, identifying the state of substances inside containers). In addition, most vision-based tactile systems must actually touch the object to obtain effective data, which is particularly unfavorable for task prediction and safe interaction in complex environments in actual scientific research tasks, such as the precise grasping and manipulation of fragile or sensitive objects by robots.

[0016] In recent years, the scientific research field has actively explored multi-modal sensing technologies to make up for the perceptual blind spots of traditional tactile sensing methods. One important research direction is to achieve "proximity sensing" before contact, that is, to obtain the position and contour information of an object before it actually comes into contact. Existing research has attempted to indirectly judge whether an object is approaching the contact area by using a single-sided light-transmitting film structure on a visual-tactile sensor and combining it with an internal light source for imaging. This method has extended the sensing distance to a certain extent, but it is sensitive to external light conditions, and the imaging range is limited, making it difficult to work stably in a textureless, low-light, or complex background. In addition, such systems usually use a time-division multiplexing method to multiplex tactile and distance imaging on the same camera, resulting in complex light source control and data processing and making it difficult to respond in real time.

[0017] To further improve the stability and sensing depth of proximity sensing, ultrasonic sensing technology has been introduced as an effective supplement. It can achieve accurate ranging in a non-contact state and reflect the material and internal structure image of the measured object by analyzing the echo signal, thus greatly expanding the understanding of the environment by the robot system.

[0018] However, there are still many technical challenges in realizing the organic combination of visual-tactile and ultrasonic sensing in actual scientific research scenarios. On the one hand, there are significant difficulties in the integrated structure design. How to ensure the clear imaging ability of the visual sensor while achieving the efficient propagation of ultrasonic signals requires an innovative layout of the sensor structure. On the other hand, the development of acousto-optic coupling materials also faces bottlenecks. Traditional acoustic impedance matching materials often do not have good optical transparency, which will directly affect the image quality. Therefore, there is an urgent need to develop a composite material system with excellent acoustic and optical properties. In addition, at the data processing level, the real-time fusion and intelligent switching mechanism of multi-modal data also need to be solved urgently. How to dynamically judge the contact state of the sensor according to different task requirements, quickly switch between non-contact and contact modes, and maintain high-precision and high-timeliness data processing capabilities is an important prerequisite for the practical application of this technology system.

[0019] There are several methods for traditional tactile sensors to achieve the proximity sensing function as follows: 1. Use a semi-transparent, controllable internal illumination skin to achieve mode switching (visual or tactile). This skin is actually an elastic selective light-transmitting film. Before contact, reduce the brightness of the light source inside the sensor and rely on ambient light illumination. The camera can see external objects, and computer vision algorithms are used to extract the target position and judge the distance to the object; after contact, increase the brightness of the light source inside the sensor, and the camera cannot see external objects and can only observe the deformation of the film to achieve the tactile function.

[0020] 2. Synchronously acquire tactile and close-range perception through an RGB camera and an infrared time-of-flight (ToF) depth camera. Through a soft elastomeric membrane, selectively transmit visible light and infrared light simultaneously to ensure that there is no interference between tactile and close-range perception.

[0021] The first solution realizes environmental perception relying on external light sources before contact and acquires tactile images after contact through a translucent elastic membrane and a controllable internal lighting switching mode. However, this solution highly depends on environmental light, resulting in a decline in perception ability under low light or complex lighting conditions. At the same time, due to image acquisition through the elastic membrane, problems such as blurred and distorted image quality often occur, severely restricting the image recognition effect. The second solution uses a combination of an RGB camera and a ToF depth camera to synchronously acquire tactile and distance information through sub-band imaging of visible light and infrared. Although the sensing dimension is improved, the system has a large volume and a complex structure and is not suitable for compact end effectors.

[0022] Although traditional visual tactile sensors have high resolution and rich surface information extraction capabilities, they still face several key problems: on the one hand, they can only acquire data after contact occurs and lack the "proximity sense" ability before contact; on the other hand, they cannot perceive the material properties or internal structure images of the measured object, severely restricting the fine manipulation ability of robots in complex environments. Some studies have tried to expand their perception range through visual means, but still cannot meet the robust, real-time, and multimodal application requirements due to limitations in lighting conditions and imaging methods.

[0023] It should be noted that the information disclosed in the above background art section is only used for understanding the background of the present application, and thus may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0024] The main object of the present invention is to overcome the defects existing in the above background art and provide an ultrasonic-enhanced tactile sensor system, sensor, and sensing method.

[0025] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect of the present invention, an ultrasonic-enhanced tactile sensor system includes: A multimodal sensing unit, including an ultrasonic module and a visual tactile perception module. The ultrasonic module emits ultrasonic waves and receives ultrasonic echo signals, and the visual tactile perception module synchronously acquires tactile deformation images. Among them, the ultrasonic module and the visual tactile perception module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure; A processing unit, which is configured to have: An event trigger module that detects contact events based on the tactile deformation image and generates a mode switching signal; An adaptive processing module that dynamically controls the ultrasonic working mode according to the mode switching signal: performing distance measurement in a non-contact state and switching to an internal imaging mode or a material property identification mode in a contact state; A three-level fusion processing module that performs signal-level synchronous acquisition, feature-level cross-modal fusion, and decision-level task switching, and outputs multi-modal perception results.

[0026] In a second aspect of the present invention, an ultrasonic enhanced tactile sensor for the system includes: A multi-modal sensing unit including an ultrasonic module and a visual tactile perception module. The ultrasonic module emits ultrasonic waves and receives ultrasonic echo signals, and the visual tactile perception module synchronously acquires tactile deformation images. Among them, the ultrasonic module and the visual tactile perception module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure; The visual tactile perception module includes an acousto-optic composite matching layer, a flexible deformation perception layer, an acoustic matching layer, and a light source and camera assembly stacked in sequence, and captures images of the contact deformation of the flexible deformation perception layer through the light source and camera assembly; The ultrasonic module includes a piezoelectric ceramic component and an ultrasonic backing layer arranged behind the acousto-optic composite matching layer, and is used for emitting and receiving ultrasonic signals; Among them, the acousto-optic composite matching layer serves as a light blocking layer of the visual tactile sensing module, allowing the light source to illuminate the sensor and allowing ultrasonic waves to penetrate; Among them, the acousto-optic composite matching layer realizes the acoustic impedance matching between the flexible deformation perception layer and the air, and the acoustic matching layer realizes the acoustic impedance matching between the piezoelectric ceramic component and the flexible deformation perception layer.

[0027] In a third aspect of the present invention, a multi-modal sensing method based on the system includes the following steps: S1. Non-contact stage: Activate the ultrasonic module to perform distance measurement, and the tactile module remains standby; S2. Contact detection stage: Analyze the tactile deformation image, determine the contact event and generate a mode switching signal; S3. Mode switching stage: Switch the ultrasonic wave to the material identification mode or the internal imaging mode according to the piezoelectric ceramic configuration; S4. Multi-modal fusion stage: Synchronously acquire tactile surface information and ultrasonic deep information, and output perception results through a three-level fusion framework.

[0028] The present invention has the following beneficial effects: The present invention proposes an innovative ultrasonic-enhanced tactile sensor system, sensor, and sensing method. Through structural design, acousto-optically transparent composite materials, and adaptive signal processing methods, a multi-modal sensing system with the capabilities of proximity sensing, surface texture and contact force perception, and the ability to obtain deep information of objects (internal imaging, material property identification) is constructed. At the structural level, piezoelectric ceramic components are deployed inside the visual-tactile module, supporting single-point or array distribution, and can work in single-base (shared transmit and receive) or double-base (separate transmit and receive) modes to achieve various ultrasonic functions such as distance sensing, material identification, and internal imaging, expanding and integrating the functions of traditional visual-tactile sensors. In terms of materials, the design of acoustic matching materials is introduced into the structure of traditional visual-tactile sensors to optimize the acoustic impedance matching at the interfaces of the flexible film with air and solids, improving the ultrasonic transmission efficiency while ensuring the optical imaging performance. In system control, a working mode switching mechanism triggered by tactile feedback is adopted. When there is no contact, ultrasonic distance measurement is performed. After contact, the sensing mode is switched according to the piezoelectric ceramic configuration: single-point configuration is used for ultrasonic detection of the material properties of objects, and array configuration starts ultrasonic imaging to obtain internal structure images, realizing multi-modal comprehensive perception. This solution improves the dimension and depth of perception ability, enhances the adaptability of the sensor to changing environments, and can be widely applied to robot research and engineering scenarios that require non-contact pre-judgment and high-precision grasping. The visual-tactile sensor uses a camera to capture the deformation of the elastic film (PDMS layer) to analyze the surface information of the object. Its image sequence output form is convenient for spatial alignment with other sensing modalities such as ultrasound, providing a natural interface and technical basis for co-structure integration, shared sensing areas, and multi-modal information fusion.

[0029] The innovative contributions of the present invention include introducing acoustic matching design without changing the structure of traditional visual-tactile sensors, achieving efficient ultrasonic transmission and clear optical imaging; constructing an event-driven mechanism triggered by tactile feedback to automatically switch ultrasonic functions to enhance perception coherence and adaptability; integrating multi-dimensional sensing functions of proximity sensing, surface texture analysis, and deep information acquisition, covering the complete information chain from object pre-judgment to contact recognition; supporting multiple combination methods of ultrasound and visual touch, and being flexibly configurable and expandable according to the scenario. The present invention realizes non-contact distance sensing and proximity judgment, simultaneously obtains surface and deep information after contact, has a compact structure suitable for end effectors and supports real-time mode switching, and has good ultrasonic function expandability. Compared with traditional technologies, the present invention overcomes the limitations of traditional visual-tactile sensors lacking pre-contact judgment ability and being unable to detect the internal information of objects, solves the acoustic matching problem of the fusion of ultrasonic and tactile modalities, supports multiple ultrasonic working modes and constructs a dynamic switching mechanism, improving operation intelligence and environmental adaptability.

[0030] Other beneficial effects in the embodiments of the present invention will be further described below. Brief Description of the Drawings

[0031] Figure 1 Overall schematic diagram of the sensor according to an embodiment of the present invention.

[0032] Figure 2 Example 1 of the sensor structure layout showing an embodiment of the present invention.

[0033] Figure 3 Example 2 of the sensor structure layout showing an embodiment of the present invention.

[0034] Figure 4 Example 3 of the sensor structure layout showing an embodiment of the present invention.

[0035] Figure 5 Example 4 of the sensor structure layout showing an embodiment of the present invention.

[0036] Figure 6 Example 5 of the sensor structure layout showing an embodiment of the present invention.

[0037] Figure 7 Principle block diagram for realizing multi-modal perception through structural reuse of Example 1 of the sensor structure layout.

[0038] Figure 8 System working flowchart according to an embodiment of the present invention.

[0039] Figure 9 Data processing framework diagram of the three-level fusion according to an embodiment of the present invention.

[0040] Figure 10 Flow diagram of the sensing method according to an embodiment of the present invention. Detailed implementation

[0041] The following gives a detailed description of the embodiments of the present invention. It should be emphasized that the following description is merely exemplary and not intended to limit the scope of the present invention and its applications.

[0042] It should be noted that when an element is referred to as "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be for fixing or for coupling or communicating.

[0043] It should be understood that the orientation or positional relationship indicated by terms such as "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present invention.

[0044] In addition, the terms "first" and "second" are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present invention, the meaning of "a plurality" is two or more, unless otherwise specifically defined.

[0045] The present invention aims to propose a multi-modal integrated sensing system that combines visual touch and ultrasonic sensing functions, and constructs a multi-modal sensing framework with the capabilities of proximity sensing, surface texture and contact force sensing, and the ability to obtain deep information of objects (internal imaging, material property identification ability).

[0046] The innovative contributions and important features of the present invention include: 1. Without changing the structure of the traditional visual touch sensor, an acoustic matching design is introduced to enable the system to simultaneously have high-efficiency ultrasonic transmission and clear optical imaging capabilities. 2. Construct an event-triggered mechanism based on tactile feedback, enabling the system to automatically switch the ultrasonic function and enhancing the coherence and self-adaptability of the sensing process. 3. The system integrates multi-dimensional sensing functions such as proximity sensing, surface texture analysis, and deep information acquisition, covering the complete information chain from object prediction to contact recognition, and providing unified data support for complex operation tasks. 4. This architecture supports various combination methods of ultrasonic + visual touch, does not limit the number of piezoelectric ceramics, installation positions, frequencies, or flexible interface materials, and can be flexibly configured and expanded according to different application scenarios.

[0047] An embodiment of the present invention provides an ultrasonic-enhanced tactile sensor system, including a multi-modal sensing unit and a processing unit. For the specific structural composition and implementation principle of the system, refer to Figures 1 to 10 .

[0048] The multi-modal sensing unit includes an ultrasonic module and a visual touch sensing module. The ultrasonic module emits ultrasonic waves and receives ultrasonic echo signals, and the visual touch sensing module synchronously acquires tactile deformation images. Among them, the ultrasonic module and the visual touch sensing module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure.

[0049] The processing unit is configured to have an event trigger module, an adaptive processing module, and a three-level fusion processing module. The event trigger module detects contact events based on the tactile deformation image and generates a mode switching signal. The adaptive processing module dynamically controls the ultrasonic working mode according to the mode switching signal: performing distance measurement in a non-contact state and switching to the object internal imaging or material property identification mode in a contact state. The three-level fusion processing module performs signal-level synchronous acquisition, feature-level cross-modal fusion, and decision-level task switching, and outputs multi-modal perception results.

[0050] This system realizes multi-dimensional perception of proximity sense, surface texture, and deep information, and can significantly improve the prediction and precise operation capabilities of the robot in complex scenarios.

[0051] Refer to Figures 1 to 6 , in some embodiments, the visual-tactile perception module includes an acousto-optic composite matching layer 1, a flexible deformation perception layer 2, an acoustic matching layer 4, a light source 5, and a camera 7 that are stacked in sequence. The image of the contact deformation of the flexible deformation perception layer 2 is captured by the light source 5 (such as an LED light board) and the camera 7; the ultrasonic module includes a piezoelectric ceramic component 6 (such as a piezoelectric ceramic ring, a piezoelectric ceramic array, etc.) and an ultrasonic backing layer 8 arranged behind the acousto-optic composite matching layer 1, which are used to transmit and receive ultrasonic signals; wherein, the acousto-optic composite matching layer 1 serves as a light-blocking layer of the visual-tactile sensing module, allowing the light source 5 to illuminate the sensor and allowing ultrasonic waves to penetrate; wherein, the acoustic impedance matching between the flexible deformation perception layer 2 and air is achieved through the acousto-optic composite matching layer 1, and the acoustic impedance matching between the piezoelectric ceramic component 6 and the flexible deformation perception layer 2 is achieved through the acoustic matching layer 4.

[0052] In some embodiments, the acousto-optic composite matching layer 1 is a mixture film of hollow glass microspheres containing dyes and PDMS (abbreviated as HGM-PDMS mixture film), and its acoustic impedance difference is reduced by incorporating hollow glass microspheres to make the acoustic impedance approach that of air. The flexible deformation perception layer 2 is a PDMS layer. The acoustic matching layer 4 is an acrylic sheet, and its thickness is designed based on the quarter-wavelength acoustic matching formula to optimize the transmission efficiency of ultrasonic waves between the piezoelectric ceramic component 6 and the PDMS layer.

[0053] In some embodiments, the multi-modal sensing unit further includes a multiplexed mechanical carrier, which provides an installation cavity and integrally carries the visual-tactile perception module and the ultrasonic module.

[0054] The piezoelectric ceramic component 6 is arranged around the camera 7 (see Figures 2 to 4 ), or is arranged between the mixture film of hollow glass microspheres and PDMS (acousto-optic composite matching layer 1) and the PDMS layer (flexible deformation perception layer 2) (see Figures 5 to 6), sharing the spatial perception area with the visual-tactile perception module to achieve structural reuse of the ultrasonic and tactile modalities. Specifically, the piezoelectric ceramic component 6 may include: (a) an annular piezoelectric ceramic unit sleeved on the outer periphery of the camera 7 and fixed to the lower fixed housing 9; or (b) an array-type piezoelectric ceramic unit embedded between the HGM-PDMS hybrid film and the PDMS layer or around the camera 7. And the piezoelectric ceramic component 6 supports a monostatic mode (sharing of transmission and reception) or a bistatic working mode (separate arrangement of transmission and reception).

[0055] As Figures 1 to 6 shown, in some embodiments, the reusable mechanical carrier includes an upper fixed housing 3 and a lower fixed housing 9. The upper fixed housing 3 is fixedly connected to the acrylic sheet and the PDMS layer, and the hollow glass microsphere and PDMS hybrid film are disposed on its surface; the camera 7 is fixedly centered in the lower fixed housing 9, the piezoelectric ceramic component 6 and the ultrasonic backing layer 8 are disposed on the outer periphery, and an LED light board is externally connected to the bottom; wherein, the upper fixed housing 3 and the lower fixed housing 9 are assembled to form an installation cavity.

[0056] In some embodiments, the mode switching mechanism of the adaptive processing module of the processing unit specifically includes: in the non-contact state, turning off the tactile image acquisition and starting the ultrasonic distance measurement; when a contact event is triggered, entering the corresponding mode according to the configuration of the piezoelectric ceramic: (a) the internal imaging mode of the object under the array configuration, performing high-resolution ultrasonic scanning to generate an internal structure image of the object; (b) the material property identification mode under the single-point configuration, extracting and analyzing the frequency domain characteristics of the echo to identify the material property.

[0057] In some embodiments, the three-level fusion processing module of the processing unit specifically includes: a signal-level fusion unit that synchronously acquires tactile images and ultrasonic echo signals and shares the spatial perception area; a feature-level fusion unit that extracts tactile texture features through a neural network and performs cross-modal feature stitching in combination with the ultrasonic echo spectrum analysis result; a decision-level fusion unit that dynamically selects the ultrasonic task mode based on the contact event, outputs distance information when not in contact, and fuses the tactile surface information and the ultrasonic deep information after contact to output an object recognition result.

[0058] In a further embodiment, the processing mechanism of the decision-level fusion unit specifically includes: when not in contact, the ultrasonic modality outputs proximity distance information; after a contact event is triggered, the ultrasonic modality switches to the material identification or internal imaging mode; fusing the tactile texture features and the ultrasonic material / structure features, and generating a final decision through a weighting or voting mechanism.

[0059] Refer to Figures 1 to 7, embodiments of the present invention also provide an ultrasonic enhanced tactile sensor for the tactile sensor system, including a multimodal sensing unit, the multimodal sensing unit includes an ultrasonic module and a visual tactile perception module, the ultrasonic module emits ultrasonic waves and receives ultrasonic echo signals, and the visual tactile perception module synchronously acquires tactile deformation images; wherein, the ultrasonic module and the visual tactile perception module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure. As described above, specifically, the visual tactile perception module includes an acousto-optic composite matching layer 1, a flexible deformation perception layer 2, an acoustic matching layer 4, a light source 5 and a camera 7 stacked in sequence, and captures images of the contact deformation of the flexible deformation perception layer 2 through the light source 5 (such as an LED light board) and the camera 7; the ultrasonic module includes a piezoelectric ceramic component 6 (such as a piezoelectric ceramic ring, a piezoelectric ceramic array, etc.) and an ultrasonic backing layer 8 arranged behind the acousto-optic composite matching layer 1, for emitting and receiving ultrasonic signals; wherein, the acousto-optic composite matching layer 1 serves as a light blocking layer of the visual tactile sensing module, allowing the light source 5 to illuminate the sensor and allowing ultrasonic waves to penetrate; wherein, the acoustic impedance matching between the flexible deformation perception layer 2 and the air is achieved through the acousto-optic composite matching layer 1, and the acoustic impedance matching between the piezoelectric ceramic component 6 and the flexible deformation perception layer 2 is achieved through the acoustic matching layer 4.

[0060] Embodiments of the present invention also provide a multimodal sensing method based on the tactile sensor system, including the following steps: Step S1, non-contact stage: Activate the ultrasonic module to perform distance measurement, and the tactile module remains standby. Step S2, contact detection stage: Analyze the tactile deformation image, determine the contact event and generate a mode switching signal. Step S3, mode switching stage: According to the piezoelectric ceramic configuration, switch the ultrasonic wave to the material identification mode or the internal imaging mode. Step S4, multimodal fusion stage: Synchronously collect tactile surface information and ultrasonic deep layer information, and output the perception result through a three-level fusion framework.

[0061] Figures 8 to 10 Illustrates the principle and specific implementation process of the multimodal sensing method.

[0062] The following further describes specific embodiments of the present invention.

[0063] The present invention proposes a multi-modal integrated sensing system that combines visual touch and ultrasonic sensing functions, and constructs a multi-modal sensing framework with the capabilities of proximity sensing, surface texture and contact force sensing, and the ability to obtain deep information of objects (internal imaging, material property recognition). At the structural level, the sensor deploys piezoelectric ceramic units inside the visual touch module, which can be configured in a single-point manner or in an array distribution. According to the task requirements, the piezoelectric ceramic can work flexibly in a monostatic (shared transmission and reception) or bistatic (separate transmission and reception) mode, thus supporting a variety of ultrasonic functions including distance sensing, ultrasonic detection of object properties, and non-contact imaging, realizing the functional expansion and integration of traditional visual touch sensors. At the material level, the sensor introduces the design of acoustic matching materials on the original structure, improving the ultrasonic propagation efficiency while ensuring that the imaging quality is not affected. In system control, an event-driven mechanism based on tactile feedback is introduced. By default, distance measurement is performed in a non-contact state, and automatically switches to more complex sensing tasks including texture analysis, object property detection, or imaging after contact, overall enhancing the adaptive intelligence and sensing capabilities under multi-stage tasks.

[0064] The present invention integrates ultrasonic functions into visual touch sensors. The visual touch sensor includes an acousto-optic composite matching layer 1 (which can be silicone or PDMS mixed with dyes) as a light-blocking film, a flexible deformation sensing layer 2 (which can be a silicone or PDMS layer), a fixed housing, an acrylic sheet (acoustic matching layer 4), an LED light board (light source 5), and a camera 7. The core device of the ultrasonic function is the piezoelectric ceramic component 6, whose position can be embedded around the camera 7 or between the light-blocking film and the flexible deformation sensing layer 2. If the piezoelectric ceramic component 6 is deployed around the camera 7, its acoustic propagation path is the acrylic sheet, the flexible deformation sensing layer 2, the light-blocking film, air or an object; if it is deployed between the light-blocking film and the flexible deformation sensing layer 2, its propagation path is the light-blocking film, air or an object. The acoustic impedance differences between the piezoelectric ceramic component 6, the acrylic sheet, the flexible deformation sensing layer 2, the light-blocking film, air and other objects are relatively large. According to the acoustic propagation law, a large amount of sound waves will be reflected at the interface, resulting in low acoustic propagation efficiency. In response to this, the present invention further improves the structure of the visual touch part, such as the material composition of the light-blocking film and the thickness of the acrylic sheet, and correspondingly forms structures such as the acousto-optic composite matching layer 1 and the acoustic matching layer 4.

[0065] The acoustic-optic composite matching layer 1 is preferably a mixed film of hollow glass microspheres containing dyes and PDMS (hereinafter referred to as HGM-PDMS film). It has two functions. First, as a light-blocking layer of the visual tactile sensor, it allows the internal light source of the visual tactile sensor to illuminate the sensor, facilitating the camera 7 to observe the deformation of the elastic layer and obtain tactile modality information, while preventing external ambient light from affecting the imaging of the camera 7 of the sensor. Second, as an acoustic matching layer between PDMS and air, by mixing hollow glass microspheres into a layer of PDMS, the acoustic impedance gap between PDMS and air is reduced, so the acoustic propagation efficiency of ultrasonic waves can be improved.

[0066] In the present invention, in addition to supporting the deformation of the PDMS layer, the acrylic layer also functions as an acoustic matching layer. Based on the formula of quarter-wavelength propagation of the acoustic propagation matching layer, an acrylic sheet with a specific thickness is customized to make it an acoustic matching layer between the piezoelectric ceramic component 6 and the PDMS layer for ultrasonic waves.

[0067] Figure 1 The overall structure of the sensor of the present invention is shown. The sensor structures of multiple embodiments are specifically arranged as follows: As Figure 2 shown, the piezoelectric ceramic component 6 uses a piezoelectric ceramic ring, which is deployed between the LED light board and the camera 7 and embedded around the camera 7. The system components from top to bottom are: a mixed film of hollow glass microspheres and PDMS (hereinafter referred to as HGM-PDMS film), a PDMS layer, an upper fixed housing 3, an acrylic sheet, an LED light board, a camera 7, a piezoelectric ceramic component 6, a lower fixed housing 9, and an ultrasonic backing layer. It can be divided into upper and lower parts.

[0068] As Figure 3 and Figure 4 shown, the piezoelectric ceramic component 6 uses a piezoelectric ceramic array, which is deployed between the LED light board and the camera 7 and embedded around the camera 7. The system components from top to bottom are: HGM-PDMS film, PDMS layer, upper fixed housing 3, acrylic sheet, LED light board, camera 7, piezoelectric ceramic component 6, lower fixed housing 9, and ultrasonic backing layer. It can be divided into upper and lower parts.

[0069] As Figure 5 and Figure 6 shown, the piezoelectric ceramic component 6 uses a piezoelectric ceramic array, which is deployed between the HGM-PDMS film and the PDMS layer. The system components from top to bottom are: HGM-PDMS film, PDMS layer, upper fixed housing 3, acrylic sheet, LED light board, camera 7, piezoelectric ceramic component 6, lower fixed housing 9, and ultrasonic backing layer. It can be divided into upper and lower parts.

[0070] The layout of the sensor structure is not limited to the above - mentioned method. In fact, the piezoelectric ceramic component 6 can be arranged at any position of the sensor, as long as the acoustic matching work is fully considered in the acoustic propagation path of the piezoelectric ceramic. According to the acoustic impedance of the components in the structure, adjusting the thickness of the components or incorporating other materials can achieve efficient ultrasonic propagation, and thus realize the integration of ultrasound and visual touch.

[0071] Figure 2 Example 1 of the shown sensor structure layout shows a typical ultrasonic + tactile multimodal sensor structure. Taking Example 1 of the sensor structure layout as an example, the component composition and functions of the sensor will be introduced in detail as follows: The upper part consists of an HGM - PDMS hybrid film, a PDMS layer, an upper fixed housing 3, and an acrylic sheet. The upper fixed housing 3 is hollow, with an acrylic sheet adhered to the bottom. PDMS is poured on the fixed housing, and a light - blocking layer is spin - coated on the cured PDMS.

[0072] The lower part consists of an LED light board, a camera 7, a piezoelectric ceramic ring, an ultrasonic backing layer 8, and a lower fixed housing 9. The camera 7 is fixed at the center of the lower fixed housing 9, the piezoelectric ceramic ring is sleeved on the camera 7, backing material is poured in the gap, and the LED light board is placed outside the housing.

[0073] The HGM - PDMS hybrid film has two functions. It allows ultrasonic waves to penetrate and can also deform to provide visual information. First, as a light - blocking layer of the visual - tactile sensor, it allows the internal light source of the visual - tactile sensor to illuminate the sensor, facilitating the camera to observe the deformation of the elastic layer and obtain tactile - modality information, while preventing external ambient light from affecting the camera imaging of the sensor. Second, as an acoustic matching layer between PDMS and air, by mixing hollow glass microspheres into a layer of PDMS, the acoustic impedance gap between PDMS and air is reduced, so the acoustic propagation efficiency of ultrasonic waves can be improved.

[0074] The PDMS layer is a component of the visual - tactile sensor and is used to contact the object to generate deformation and capture tactile - modality information. The upper fixed housing 3 provides mechanical protection for the sensor. The acrylic sheet is a component of the visual - tactile sensor, used to support the deformation of the PDMS layer and also serves as the function of an acoustic matching layer. Based on the formula of quarter - wavelength propagation of the acoustic propagation matching layer, an acrylic sheet with a specific thickness is customized to make it an acoustic matching layer between the piezoelectric ceramic component 6 and the PDMS layer. The LED light board serves as the light source 5 inside the visual - tactile sensor. The camera 7 is used to observe the deformation of the PDMS layer and obtain tactile - modality information. The piezoelectric ceramic ring serves as the ultrasonic wave emission source and operates in the monostatic mode. The lower fixed housing 9 provides mechanical protection for the sensor. The ultrasonic backing layer 8 is a component of the ultrasonic sensor, reducing the signal tail of the ultrasonic emission wave and improving the working performance of ultrasonic waves.

[0075] As shown Figure 7 in the figure, the sensor component realizes ultrasonic + tactile fusion perception through structural reuse.

[0076] The present invention proposes an event-driven ultrasonic mode switching mechanism. Through the "contact event" signal generated by the visual tactile module, the ultrasonic module is automatically driven to switch from the distance measurement mode in the non-contact state to the object internal imaging or object property recognition mode in the contact state. This mechanism combines hardware triggering and software control to achieve dynamic functional reconfiguration of the multimodal sensor at different task stages.

[0077] The whole process can be divided into the following stages: Ultrasonic distance perception in the non-contact state: In the initial stage, the sensor has not contacted the object, and the system only activates the ultrasonic module to continuously emit ultrasonic signals and receive echoes, and judges the relative position of the target object by measuring the acoustic wave propagation time. Moreover, according to the number of piezoelectric ceramics, the ultrasonic mode can work in the monostatic mode or the bistatic mode. The monostatic mode only requires one piezoelectric ceramic, which reduces the cost; the bistatic mode can reduce the measurement blind area, and the proximity perception range is larger than the former. At this time, the tactile image module remains in a low-power standby state and does not participate in information acquisition, reducing the system burden.

[0078] Contact event detection and mode switching trigger: When the sensor contacts the target object, the flexible imaging layer in the tactile module will undergo obvious deformation. The system judges whether the trigger condition of the contact event is reached by analyzing the deformation amplitude and structural changes of the image. Once the contact criterion is met, the tactile image path immediately sends a "contact event signal" to the main control system. This signal serves as a conversion condition in the state machine to trigger the subsequent mode switching process, enabling the ultrasonic mode to obtain deep object information. Further, according to whether the sensor has a piezoelectric ceramic array or a single piezoelectric ceramic, the ultrasonic mode can work in the object internal imaging mode or the object material property recognition mode.

[0079] Ultrasonic object internal imaging in the contact state: After the system receives the tactile trigger signal, it immediately switches the working parameters of the ultrasonic module to enter the object internal imaging mode. In this mode, the piezoelectric ceramic array is activated, and the system adjusts the ultrasonic emission parameters to achieve high-resolution scanning of the target area. By performing time-frequency analysis and feature extraction on the echo signal, the internal structure distribution of the object can be obtained. At the same time, the visual tactile module continues to work, collecting external information such as texture deformation and geometric contour on the object surface. The two types of perception modules share the acquisition area structurally and are precisely aligned spatially, providing a consistent basis for subsequent internal and external information fusion and three-dimensional reconstruction.

[0080] Ultrasonic object material property identification in contact state: After the system receives the tactile trigger signal, it immediately switches the working parameters of the ultrasonic module and enters the material property identification mode. In this mode, the system performs frequency-domain transformation and feature extraction on the received ultrasonic echo signal to identify the material information of the object. At the same time, the visual-tactile module is also activated to collect information such as the texture and shape of the object surface. The two modules share the acquisition area in hardware and have been spatially aligned in the structural design to facilitate subsequent information fusion.

[0081] Information fusion and subsequent operation interface: After completing the mode switch, the system synchronously outputs the perception data from ultrasonic and visual-tactile for the upper-layer control system to call, to support actions such as grasping strategy adjustment, object classification, and path correction. This switching mechanism is executed by the embedded processor for real-time scheduling, and realizes multi-thread synchronization based on the periodic timer and interrupt mechanism, with good real-time performance and scalability.

[0082] This event-driven mode switching mechanism simplifies the working process of the multi-modal perception system, making the cooperation between perception tasks and mechanical actions more efficient, and is especially suitable for the control logic of the robot hand to first judge and then operate when facing unknown objects. The system has a compact structure, high module integration, a clear signal acquisition and processing path, and good portability and compatibility, and can be easily embedded in the end structure of the multi-joint dexterous hand. The sensor body can be directly deployed at the fingertip or palm surface, and integration can be achieved without changing the structure of the manipulator. The system actively provides distance and positioning information in the non-contact stage, and automatically switches to the high-precision identification mode after contact occurs to perceive the texture and material properties of the object. With this mechanism, the dexterous hand can dynamically switch the perception mode according to the task stage when performing complex tasks such as grasping, sorting, and manipulation, significantly improving the autonomy, intelligence, and environmental adaptability of the operation.

[0083] Figure 8 The system working flow chart of the embodiment of the present invention is shown. The system has the ability to simultaneously analyze the surface details and deep information of the object, provides a unified platform for multi-modal intelligent perception and operation, and has practical promotion value in various scenarios such as industrial inspection, flexible grasping, and biological tissue analysis.

[0084] The present invention constructs a three-level fusion data processing framework for coordinating the information acquisition and processing of two modalities, visual-tactile and ultrasonic. The system is divided into three levels according to functions: "signal-level fusion", "feature-level fusion", and "decision-level fusion", as Figure 9 shown.

[0085] Signal-level fusion: The system simultaneously acquires two types of raw signals in the sensing hardware segment through a synchronization mechanism: visual images from the camera, which are used to record the deformation process of the elastic membrane during contact; ultrasonic echo signals from the piezoelectric ceramic, which are used to reflect deep information such as the internal structure imaging or material properties of the target object. The acquisition of the two signals has time synchronization and shares the same spatial sensing area, ensuring the consistency of the subsequent information fusion basis.

[0086] Feature-level fusion: This stage mainly performs data preprocessing. The visual path extracts high-dimensional image features such as texture and shape through a neural network; the ultrasonic path performs spectral analysis on the echo signal and extracts characteristic parameters that reflect material differences, such as energy distribution and spectral shape. The extracted feature vectors are spliced, aligned, or weighted and fused through an attention mechanism to provide rich data representations for the upper-level recognition task.

[0087] Decision-level fusion: This stage is the key link of the system. Different from the general static fusion structure, the present invention dynamically controls the ultrasonic working mode through the contact event judgment mechanism of the tactile image: in the non-contact stage, the system defaults that the ultrasonic is in the "ranging mode" to provide the spatial position information of the target object; once a contact event is detected, the system immediately switches the ultrasonic to the "material identification mode" and enables specific signal acquisition and analysis parameters to extract deep features for classification. The decision layer can further fuse the recognition results of the two paths (for example, through a weighting or voting mechanism) and output the final object recognition or status judgment result.

[0088] This three-level structure decouples and collaboratively processes signal acquisition, feature analysis, and task decision-making, making the system have the advantages of clear structure, strong scalability, and high processing efficiency, and is especially suitable for the sensing requirements of a robotic dexterous hand to perform high-precision and multi-stage operation tasks on complex objects.

[0089] Figure 10 It is a flow diagram of the sensing method of the embodiment of the present invention, showing the close connection and collaborative work between ultrasonic and tactile information to jointly achieve multi-modal perception.

[0090] In summary, the present invention proposes a multi-modal integrated sensing system and sensor that combines visual touch and ultrasonic sensing functions. Its main innovative contributions and features are as follows: An acoustic matching design is introduced without changing the structure of traditional visual touch sensors. By optimizing the acoustic impedance matching at the interfaces of the flexible film with air and solids, the synergy of efficient ultrasonic transmission and clear optical imaging capabilities is achieved; A tactile feedback-based event-triggering mechanism is constructed, enabling the system to automatically switch the ultrasonic function according to the contact state, perform distance measurement in the non-contact state, and switch to material property identification or internal object imaging mode respectively according to the single-point or array configuration of piezoelectric ceramics after contact, enhancing the coherence and self-adaptability of the sensing process; The system integrates multi-dimensional sensing functions such as proximity sensing, surface texture analysis, and deep information acquisition, covering the complete information chain from object pre-judgment to contact recognition, providing unified data support for complex operation tasks; The architecture supports various combination methods of ultrasound and visual touch, without restricting the number of piezoelectric ceramics, installation positions, and flexible interface materials, and can be flexibly configured and expanded according to different scenarios.

[0091] The main functions of this ultrasonic-enhanced tactile sensor system include: realizing distance perception and proximity judgment of the target object in the non-contact state, synchronously obtaining surface texture information and internal object structure images or material property identification information after contact, and having the dual-modal sensing capabilities of touch and ultrasound; being structurally compact and suitable for integration in the end effector, supporting real-time working mode switching; having good ultrasonic function expandability, realizing internal structure imaging and material property analysis through the flexible deployment of piezoelectric ceramics, and being suitable for complex object recognition.

[0092] Compared with traditional technologies, the significant advantages of the present invention are reflected in: making up for the deficiency of traditional visual touch sensors lacking pre-contact judgment ability and realizing non-contact distance perception; breaking through the limitation that traditional touch cannot detect deep information of objects and expanding the sensing depth; solving the acoustic matching problem in the fusion of ultrasonic and tactile modalities to ensure that optical imaging is not affected; supporting various ultrasonic working modes such as monostatic and bistatic, and adapting to different accuracy requirements; automatically adjusting the sensing strategy according to the contact state through a dynamic switching mechanism to improve the self-adaptive ability under multi-stage tasks.

[0093] The prominent beneficial effects of the present invention include: enhancing the pre-judgment ability by obtaining non-contact distance information through the ultrasonic module, and realizing the integrated perception of surface texture and ultrasonic echo information inside and outside in the contact state; the compact design is suitable for the end effector and supports real-time mode switching; various ultrasonic working modes adapt to diverse task requirements, covering the recognition and evaluation of complex structure objects; the dynamic conversion mechanism of the ultrasonic mode based on tactile triggering improves the operation intelligence and environmental adaptability.

[0094] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments. It cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention pertains, without departing from the concept of the present invention, they can also make several substitutions or modifications to these described embodiments, and these substitution or modification methods should all be regarded as belonging to the protection scope of the present invention. In the description of this specification, the descriptions with reference to the terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. Without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the scope of protection of the patent application.

Claims

1. An ultrasonic enhanced tactile sensor system, characterized in that, Comprising: A multi-modal sensing unit, including an ultrasonic module and a visual tactile perception module. The ultrasonic module emits ultrasonic waves and receives ultrasonic echo signals, and the visual tactile perception module synchronously acquires tactile deformation images. Among them, the ultrasonic module and the visual tactile perception module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure; A processing unit, which is configured to have: An event trigger module that detects contact events based on the tactile deformation images and generates a mode switching signal; An adaptive processing module that dynamically controls the ultrasonic working mode according to the mode switching signal: performing distance measurement in a non-contact state and switching to object internal imaging or material property identification mode in a contact state; A three-level fusion processing module that performs signal-level synchronous acquisition, feature-level cross-modal fusion, and decision-level task switching, and outputs multi-modal perception results.

2. The tactile sensor system according to claim 1, wherein The visual tactile perception module includes an acousto-optic composite matching layer, a flexible deformation perception layer, an acoustic matching layer, and a light source and camera assembly stacked in sequence. The image of the contact deformation of the flexible deformation perception layer is captured by the light source and camera assembly; The ultrasonic module includes a piezoelectric ceramic component and an ultrasonic backing layer arranged behind the acousto-optic composite matching layer, and is used for transmitting and receiving ultrasonic signals; Among them, the acousto-optic composite matching layer serves as a light blocking layer of the visual tactile sensing module, allowing the light source to illuminate the sensor and allowing ultrasonic waves to penetrate; Among them, the acoustic impedance matching between the flexible deformation perception layer and the air is achieved through the acousto-optic composite matching layer, and the acoustic impedance matching between the piezoelectric ceramic component and the flexible deformation perception layer is achieved through the acoustic matching layer.

3. The tactile sensor system according to claim 2, wherein The acousto-optic composite matching layer is a hollow glass microsphere and PDMS hybrid film (HGM-PDMS hybrid film), which reduces the acoustic impedance difference by incorporating hollow glass microspheres and makes the acoustic impedance approach that of air; The flexible deformation perception layer is a PDMS layer; The acoustic matching layer is an acrylic sheet, and its thickness is designed based on the quarter-wavelength acoustic matching formula to optimize the transmission efficiency of ultrasonic waves between the piezoelectric ceramic component and the PDMS layer.

4. The tactile sensor system according to claim 3, wherein, The multi-modal sensing unit further includes: A multiplexed mechanical carrier that provides an installation cavity and integrally carries the visual tactile perception module and the ultrasonic module; Among them, the piezoelectric ceramic component is deployed around the camera or between the hollow glass microsphere and PDMS hybrid film and the PDMS layer, sharing the space perception area with the visual tactile perception module to achieve structural multiplexing of ultrasonic and tactile modalities.

5. The tactile sensor system according to claim 4, wherein The multiplexed mechanical carrier includes: An upper fixed housing that fixedly connects the acrylic sheet and the PDMS layer, and the surface thereof is provided with the hollow glass microsphere and PDMS hybrid film; A lower fixed housing that fixedly fixes the camera at the center, piezoelectric ceramic components and ultrasonic backing layers are arranged on the outer periphery, and an LED light board is externally connected to the bottom; Among them, the upper fixed housing and the lower fixed housing are assembled to form an installation cavity.

6. The tactile sensor system according to any one of claims 1 to 5, characterized in that, The mode switching mechanism of the adaptive processing module includes: In the non-contact state, turn off the tactile image acquisition and start the ultrasonic distance measurement; When a contact event is triggered, enter the corresponding mode according to the configuration of the piezoelectric ceramic: (a) The internal object imaging mode under the array configuration, performing high-resolution ultrasonic scanning to generate an image of the internal structure of the object; (b) The material property identification mode under the single-point configuration, extracting and analyzing the frequency-domain characteristics of the echo to identify the material properties.

7. The tactile sensor system according to any one of claims 1 to 5, characterized in that, The three-level fusion processing module includes: The signal-level fusion unit, which synchronously acquires the tactile image and the ultrasonic echo signal and shares the spatial perception area; The feature-level fusion unit, which extracts the tactile texture features through a neural network and performs cross-modal feature splicing in combination with the ultrasonic echo spectrum analysis result; The decision-level fusion unit, which dynamically selects the ultrasonic task mode based on the contact event, outputs the distance information when not in contact, and fuses the tactile surface information and the ultrasonic deep-layer information to output the object recognition result after contact.

8. The tactile sensor system according to claim 7, wherein The processing mechanism of the decision-level fusion unit includes: When not in contact, the ultrasonic modality outputs the proximity distance information; After the contact event is triggered, the ultrasonic modality switches to the material identification or internal imaging mode; Fuse the tactile texture features and the ultrasonic material / structure features to generate the final decision.

9. An ultrasonic enhanced tactile sensor for the tactile sensor system according to any one of claims 1 to 8, characterized in that, It includes: The multi-modal sensing unit includes an ultrasonic module and a visual tactile perception module. The ultrasonic module emits ultrasonic waves and receives ultrasonic echo signals. The visual tactile perception module synchronously acquires the tactile deformation image. Among them, the ultrasonic module and the visual tactile perception module simultaneously achieve ultrasonic acoustic impedance matching and optical imaging transmission through an acousto-optic matching structure; The visual tactile perception module includes an acousto-optic composite matching layer, a flexible deformation perception layer, an acoustic matching layer, and a light source and camera assembly stacked in sequence. The image of the contact deformation of the flexible deformation perception layer is captured through the light source and camera assembly; The ultrasonic module includes a piezoelectric ceramic component and an ultrasonic backing layer arranged behind the acousto-optic composite matching layer, which are used to transmit and receive ultrasonic signals; Among them, the acousto-optic composite matching layer serves as a light-blocking layer of the visual tactile sensing module, allowing the light source to illuminate the sensor and allowing ultrasonic waves to penetrate; Among them, the acousto-optic composite matching layer realizes the acoustic impedance matching between the flexible deformation perception layer and the air, and the acoustic matching layer realizes the acoustic impedance matching between the piezoelectric ceramic component and the flexible deformation perception layer.

10. A multimodal sensing method based on the tactile sensor system according to any one of claims 1 to 8, characterized in that, It includes the following steps: S1. Non-contact stage: Activate the ultrasonic module to perform distance measurement, and the tactile module remains standby; S2. Contact detection stage: Analyze the tactile deformation image, determine the contact event and generate a mode switching signal; S3. Mode switching stage: According to the piezoelectric ceramic configuration, switch the ultrasonic wave to the material identification mode or the internal imaging mode; S4. Multi-modal fusion stage: Synchronously acquire the tactile surface information and the ultrasonic deep-layer information, and output the perception result through a three-level fusion framework.

Citation Information

Patent Citations

  • Remote tactile perception device and method and man-machine interaction method based on remote tactile perception

    CN117130474A

  • Multi-mode sensor and system with proximity sense and touch sense perception

    CN117309200A

  • Sensor

    EP0182466A1

  • Tactile sensor and tactile information detecting method

    JP2008197078A

Cited By

  • Visual tactile fusion multi-mode tactile sensor and three-dimensional pressure field reconstruction method

    CN121277350A

  • Low-power-consumption control method applied to visual touch sensor, visual touch sensor and dexterous hand

    CN121603634A

  • Low-power control method applied to tactile sensor, tactile sensor and dexterous hand

    CN121603634B

  • Decoupling ultrasonic tactile system, robot tactile system and robot

    CN122165461A

  • Grabbing control method and system based on touch sensing, dexterous hand and storage medium

    CN122442712A