Visual tactile sensor and detection method thereof

Abstract

The invention relates to a visual tactile sensor and a detection method thereof. The visual tactile sense detection method is applied to a visual tactile sense sensor capable of carrying out photometric stereo and volume phase scattering comprehensive calculation according to the distribution condition of scattering media in the visual tactile sense sensor, and comprises the following steps: configuring a light source to illuminate a soft elastomer in the visual tactile sense sensor; collecting a visual change image of the scattering medium in the soft elastomer caused by contact deformation of the soft elastomer; and performing photometric stereo and volume phase scattering comprehensive calculation on the visual change image to obtain mechanical characteristics and / or deformation characteristics of contact deformation of the soft elastomer. According to the application, the scattering medium in gradient distribution is constructed in the soft elastomer, and a vulnerable surface reflecting film is replaced by a volume phase scattering mechanism, so that the durability and environmental adaptability of the sensor are remarkably improved. And the gradient distribution of the scattering medium forms sensitivity layering, so that high-sensitivity and wide-range detection is realized.

Description

Technical Field

[0001] This invention relates to the field of sensor technology, and in particular to a visual-tactile sensor and its detection method. Background Technology

[0002] With the rapid development of robotics, industrial automation, and human-computer interaction, endowing machines with tactile perception capabilities similar to humans has become a research hotspot. Visual-tactile sensors, which convert the mechanical forces at the contact interface into optical image signals and use computer vision algorithms to extract the geometric shape and mechanical features of the contacted object, have attracted much attention due to their advantages of high spatial resolution and low cost.

[0003] In existing visual-tactile sensing technologies, surface reflection imaging is one of the mainstream approaches. For example, international patent application (PCT) publication number WO2009 / 155501A2 discloses a tactile sensor using elastic imaging (i.e., a GelSight sensor). This technology covers a reflective film on the surface of a transparent elastomer, illuminates the reflective film with a light source, and captures the normal change image of the reflective film surface when an external object presses on the reflective film, thereby reconstructing the 3D topography of the contact surface. However, this approach heavily relies on the reflective film layer on the surface. In long-term use, the reflective film is prone to wear, scratches, or peeling, leading to sensor failure and high maintenance costs. Furthermore, because the reflective film obscures the optical information inside the elastomer, this approach is mainly sensitive to surface geometric deformation. Without introducing additional markers, it is difficult to achieve decoupled measurement of normal pressure and tangential shear force, and it is difficult to adapt to complex contact environments such as oil stains.

[0004] On the other hand, sensing schemes based on changes in bulk scattered light intensity have also been proposed. For example, Chinese invention patent CN1138973C discloses a pressure sensor comprising a compressible wave energy carrier medium (such as translucent foam) with an outer boundary, in which wave energy scattering centers are scattered to form assimilation cavities. When external pressure causes the medium to compress, the density of scattering centers increases, resulting in an increase in the intensity of scattered light. Pressure is measured by detecting this intensity change. However, this scheme is usually based on fiber optic point acquisition or low-resolution light intensity detection. Although it can measure total pressure, it lacks high spatial resolution and cannot reconstruct the surface texture and three-dimensional shape of the contacting object as finely as a vision sensor.

[0005] Furthermore, in the field of optical materials, there have been some applications of controlling light propagation through doping particles. For example, Korean patent application KR1020020094846A describes a light guide plate structure containing light-scattering particles of various sizes and dispersed distributions. It even proposes a gradient distribution design of particle concentration or size to achieve uniform light emission or specific light-guiding effects. Although this document reveals the gradient distribution process of scattering particles, it is mainly applied to display backlighting or illumination, and does not address how to utilize this gradient scattering characteristic of elastomers in conjunction with photometric stereo vision algorithms to solve the problem of difficulty in simultaneously achieving geometric reconstruction and mechanical measurement, and the difficulty in decoupling them, in tactile sensing. Summary of the Invention

[0006] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0007] To achieve the above-mentioned objectives and other advantages of the present invention, according to a first aspect of the present invention, a visual-tactile detection method is provided, applied to a visual-tactile sensor capable of performing comprehensive calculations of photometric stereo and volume scattering based on the distribution of its internal scattering medium, comprising the following steps: Configure a light source to illuminate the soft elastomer in the visual-tactile sensor; Collect visual images of the changes in the scattering medium inside a soft elastomer due to contact deformation of the soft elastomer. Photometric stereo and volumetric scattering are combined to solve the visual change image to obtain the mechanical characteristics and / or deformation characteristics of the soft elastic body under contact deformation.

[0008] Optionally, the step of "configuring a light source to illuminate the soft elastomer in the visual-tactile sensor" specifically includes the following steps: At least two directional light sources are configured so that each directional light source radiates to the soft elastomer in a different direction, so as to provide multi-directional illumination that meets the requirements of photometric stereoscopic solution; wherein, the directional light source adopts a multi-band illumination mode, which is adapted to the scattering characteristics of the internal scattering medium of the soft elastomer, so as to achieve the separation of optical information between the photometric stereoscopic channel and the scattering channel.

[0009] Optionally, the step of "configuring a light source to illuminate the soft elastomer in the visual-tactile sensor" specifically includes the following steps: Configure at least two directional light sources with different radiation directions for the soft elastomer; A path-by-path activation control method is adopted for each directional light source to control the output of illumination light from each directional light source at different times, so as to obtain the independent optical response signal of the soft elastomer corresponding to each directional light source.

[0010] Optionally, the step of "acquiring visual images of the changes in the scattering medium inside the soft elastic body due to contact deformation of the soft elastic body" specifically includes the following steps: When a soft elastomer is subjected to external contact that causes a change in the state of its internal scattering medium, it captures the scattered light signal generated by the change in the state of the scattering medium. Based on the scattered light signal, a visual change image reflecting the change in the state of the scattering medium is generated.

[0011] Optionally, the soft elastomer is provided with a light-shielding layer and / or a marking layer; the marking layer is distributed on the surface and / or inside the soft elastomer; the step of "generating a visual change image reflecting the change in the state of the scattering medium based on the scattered light signal" specifically includes the following steps: For the scattered light signal, a masking process is performed based on the shading area of ​​the light-shielding layer or the distribution area of ​​the marker layer to remove invalid image areas related to the non-scattering medium and retain the effective scattered light signal that only reflects the internal scattering medium of the soft elastomer. Generate images showing visual changes.

[0012] Optionally, the step of "performing a comprehensive solution of photometric stereo and volumetric scattering on the visual change image to obtain the mechanical characteristics and / or deformation characteristics of the soft elastic body undergoing contact deformation" specifically includes the following steps: Extract first image features for photometric stereo calculation and second image features for scattering property analysis from visually changing images; Based on the illumination direction parameters of the light source and the first image features, the initial geometric information of the soft elastic body is determined by photometric stereo solution; Based on the intensity attenuation information of the second image features and using the initial geometric information as a path constraint, the initial scattering parameters of the soft elastic body are determined. The initial geometric information and initial scattering parameters are jointly optimized to obtain the corrected geometric information and corrected scattering parameters; Determine the deformation characteristics of the soft elastic body based on the corrected geometric information; Based on the preset calibration relationship, the corrected scattering parameters are mapped to the mechanical characteristics of the soft elastic body.

[0013] Optionally, the step of "jointly optimizing the initial geometric information and initial scattering parameters to obtain the corrected geometric information and corrected scattering parameters" specifically includes the following steps: A joint optimization model is established, which includes geometric deformation variables and scattering medium variation variables. Starting with the initial geometric information and initial scattering parameters, the joint optimization model is iteratively solved to obtain the corrected geometric information and corrected scattering parameters.

[0014] Optionally, the step of "iteratively solving the joint optimization model to obtain the corrected geometric information and corrected scattering parameters" specifically includes the following steps: For the current pixel in the visually changing image, determine the solution confidence index; wherein, the solution confidence index includes: constructing the number of effective light directions based on the illumination direction parameters; In response to the solution confidence index meeting the preset blind zone condition, the following operations are performed during the iterative solution process: stop updating the geometric information of the current pixel and determine the geometric information of the current pixel based on the geometric information of the neighboring pixels; and iteratively optimize the scattering parameters of the current pixel to obtain the corrected scattering parameters.

[0015] According to a second aspect of the present invention, a visual-tactile sensor is provided, comprising: A soft elastic body that responds to physical contact and undergoes contact deformation, with a scattering medium inside the soft elastic body, and the distribution or orientation of the scattering medium changes when the soft elastic body deforms. A light source for illuminating soft elastomers and internal scattering media, and providing the illumination required for photometric stereoscopic calculations; A photosensitive structure that receives light scattered by a scattering medium after being illuminated by a light source and generates a visual change image. The visual change image is used for comprehensive calculation of photometric stereo and volume scattering to obtain the mechanical characteristics and / or deformation characteristics corresponding to the contact deformation of the soft elastic body.

[0016] Optionally, the scattering medium is a particle or particle cluster that has scattering properties and is doped inside a soft elastic body.

[0017] Optionally, the scattering medium is non-uniformly distributed in the soft elastomer; the particle size of the scattering medium gradually decreases along the direction from the side of the soft elastomer in contact with the solid to the side away from the contact, and / or the density of the scattering medium gradually decreases.

[0018] Optionally, in a soft elastomer, the doping concentration of the scattering medium gradually decreases from the side in contact with the solid to the side away from that contact.

[0019] Optionally, a light-shielding layer is provided on the side of the soft elastomer that contacts the solid, the light-shielding layer being used to shield ambient light.

[0020] Optionally, the soft elastomer is provided with a marking layer, which is distributed on the surface and / or inside of the soft elastomer; when the soft elastomer undergoes contact deformation in response to solid contact, the marking layer is displaced or deformed.

[0021] The visual-tactile sensor and its detection method provided in this application have at least the following beneficial effects: This application incorporates a scattering medium within a soft elastomer, utilizing the bulk scattering effect of light within the medium to modulate optical signals. This eliminates the need for the easily worn and peeling surface reflective film structure found in traditional solutions (such as GelSight). This design not only eliminates the risk of sensor failure due to aging or damage to the reflective film, significantly reducing maintenance costs, but also, because the scattered signal originates from within the bulk, the sensor possesses inherent resistance to minor scratches and oil contamination on the contact surface, significantly improving long-term operational reliability in complex industrial environments.

[0022] This application constructs a gradient distribution of scattering medium concentration or particle size (e.g., high concentration on the contact side and low concentration on the away side) within a soft elastomer. This "pseudo-tomography" structural design achieves a stratified distribution of sensitivity: the high-concentration medium layer on the contact side can produce a significant light intensity response to minute touches (kPa level), ensuring high tactile sensitivity; while the low-concentration medium layer on the away side allows light to penetrate to a greater depth, ensuring that the scattered signal remains unsaturated and maintains a linear response under heavy loads (MPa level). Thus, a wide dynamic range detection spanning orders of magnitude is achieved within a single sensor hardware architecture.

[0023] This application utilizes a multi-band or multi-directional illumination strategy, combined with a joint optimization model, to iteratively correct the relationship between geometric deformation and mechanical response. This mechanism effectively addresses the limitations of traditional photometric stereoscopic techniques, which struggle to quantify contact forces (only able to observe shape) and traditional scattering techniques, which struggle to reconstruct fine surface textures (only able to measure resultant force). It achieves dual-channel, high-precision decoupled measurement of the three-dimensional geometric shape (normal, depth) and multi-dimensional mechanical characteristics (normal pressure, shear stress) of contact objects. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating the steps of a visual-touch detection method applied to a visual-touch sensor provided in an embodiment of this application; Figure 2 A flowchart illustrating the steps of "collecting visual images of the scattering medium inside a soft elastomer due to contact deformation of the soft elastomer" provided in this application embodiment; Figure 3 A flowchart illustrating the steps of "performing photometric stereoscopic calculation on a visually changing image to obtain the mechanical characteristics of contact deformation of a soft elastic body" provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of a visual-tactile sensor provided in an embodiment of this application; Figure 5 This is a schematic diagram of the concentration gradient distribution of the scattering medium in a soft elastomer provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a visual-tactile sensor with a light-shielding layer provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a visual-tactile sensor with a marker layer provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a visual-tactile sensor with a support layer provided in an embodiment of this application; Figure 9 This is a schematic diagram of another visual-tactile sensor provided in an embodiment of this application; Figure 10 This is a schematic diagram of the structure of a visual-tactile sensor with optical elements provided in an embodiment of this application; Figure 11 A schematic diagram of the simulation verification results provided for the application embodiment. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that although functional modules are divided in the system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic diagram or the order in the flowchart. The step numbers in the following embodiments are set only for ease of explanation and do not limit the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art. Without conflict, the embodiments described below or the technical features can be arbitrarily combined to form new embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that the use of terms such as "first," "second," etc., in the specification, claims, and drawings of this application is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the specification of this invention is for the purpose of describing specific embodiments only and is not intended to limit the invention.

[0028] The visual-tactile detection method for visual-tactile sensors provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, set-top box, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the visual-tactile detection method for visual-tactile sensors, etc., but is not limited to the above forms.

[0029] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0030] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0031] In the field of visual-tactile sensing, various existing optical coding schemes all have performance limitations: although the photometric stereo method can meet the basic requirements of high resolution, high sensitivity and lateral force calculation, it has problems such as angle calculation blind spots (insufficient effective incident light direction or low signal-to-noise ratio when grazing incidence leads to unstable normal calculation), spatial calculation blind spots (the tracking area of ​​the marker point cannot be adapted), dependence on reflective skin (easy to wear and contamination and affect calibration consistency), weak anti-contamination ability and tolerance sensitivity; the field of view compression method, total internal reflection method and compressible diffuse reflection layer scheme face the problems of difficulty in normal force calculation and low resolution, and the marker array scheme is also limited by insufficient resolution.

[0032] This application provides illumination for photometric stereoscopic calculations by setting a scattering medium inside a soft elastomer that can change its distribution or orientation with contact deformation, and configuring multi-directional, multi-band directional light sources. Combined with photometric stereoscopic calculation logic, optical signals used to extract geometric information (such as normal and depth) of the soft elastomer and analyze scattering characteristics are separated from the visual change images acquired by the photosensitive structure. Then, the geometric deformation and mechanical response are decoupled by jointly optimizing the geometric information and scattering parameters. Without relying on a reflective surface, the geometric and mechanical features corresponding to the contact deformation of the soft elastomer can be perceived, so as to adapt to the application scenarios that require high resolution, high sensitivity and lateral force calculation.

[0033] like Figure 1 The diagram shows a flowchart of a visual-tactile detection method applied to a visual-tactile sensor according to an embodiment of the present invention. (Refer to...) Figure 1 This invention provides a visual-tactile detection method applied to a visual-tactile sensor, specifically including the following steps: S12. Configure a light source to illuminate the soft elastomer in the visual-tactile sensor; S14. Collect visual images of the changes in the scattering medium inside the soft elastic body due to the contact deformation of the soft elastic body. S16. Perform photometric stereo and volumetric scattering comprehensive calculations on the visual change image to obtain the mechanical characteristics and / or deformation characteristics of the soft elastic body under contact deformation.

[0034] In an optional embodiment, the soft elastomer in step S12 refers to a flexible substrate capable of supporting a bulk scattering medium and possessing recoverable deformation capability. The substrate material of the soft elastomer is selected from optically grade transparent materials, specifically including but not limited to: polydimethylsiloxane (PDMS), thermoplastic elastomer (TPE), polyurethane (PU), or dual-network hydrogel. The Shore A hardness of the soft elastomer is configured between 5 and 90; as a preferred embodiment, the hardness is configured between 10 and 40. Furthermore, the shape and thickness of the soft elastomer are adapted to the overall structure of the sensor (e.g., finger-shaped, block-shaped, cylindrical), and the thickness is preferably at least 50 μm.

[0035] The soft elastomer is doped with a scattering medium possessing scattering properties. This scattering medium specifically refers to particles or particle clusters doped within the substrate material. The particle material is selected from at least one of titanium dioxide (TiO2), silicon dioxide (SiO2), zinc oxide, hollow glass microspheres, or polymer microspheres (such as PS, PMMA). As a specific parameter configuration, the particle size (D50) is 0.2 μm to 5 μm, and the difference (Δn) between the refractive index of the particles and the refractive index of the soft elastomer substrate material is not less than 0.1, with a volume fraction (φ) between 0.1% and 5%. To achieve a wide dynamic range of mechanical measurements, the scattering medium exhibits a gradient distribution within the soft elastomer. Specifically, the doping concentration of the scattering medium gradually decreases along the direction from the side of the soft elastomer in contact with the solid (upper surface) to the side away from the contact (lower surface). This gradient distribution structure, combined with the difference in penetration depth of light at different wavelengths, forms a pseudo-tomography sensitivity distribution, enabling the sensor to handle both kPa-level minute touches and MPa-level heavy load measurements.

[0036] The preparation process of soft elastomers may include: injecting a curable substrate material mixed with particles of different particle sizes into a mold, and using the natural sedimentation of the particle density difference during the curing process to form a gradient distribution; or, when the particles contain magnetic or conductive cores, achieving field-induced sedimentation distribution by applying an external magnetic field or electric field.

[0037] Specifically, the light source configured in step S12 is a directional light source module that meets the requirements of photometric stereoscopic calculation. This light source module includes at least two (preferably three or six) directional light sources, and the radiation directions of each directional light source onto the soft elastic body are different and non-coplanar. As a preferred embodiment, the principal axis of the light distribution of the directional light source illuminates the soft elastic body at an angle (e.g., elevation angle) between 60 and 120 degrees. In terms of control logic, the directional light source is configured in one of the following modes: Time-sharing lighting mode: Each directional light source is activated sequentially, allowing each directional light source to output illumination light in different time slots. Multi-band illumination mode: Configuring directional light sources in different directions to emit spectra in different bands (such as red light, green light, and blue light), and the spectral bands are adapted to the scattering characteristics of the internal scattering medium of the soft elastomer (for example, using red light to characterize surface geometry and using blue light to characterize bulk scattering).

[0038] In an optional embodiment, step S12 of "configuring a light source to illuminate the soft elastomer in the visual-tactile sensor" specifically includes the following steps: S121. Configure at least two directional light sources so that each directional light source radiates to the soft elastic body in a different direction, so as to provide multi-directional illumination that meets the requirements of photometric stereoscopic solution; Among them, the directional light source adopts a multi-band illumination mode, which is adapted to the scattering characteristics of the internal scattering medium of the soft elastomer, so as to realize the separation of optical information between the photometric three-dimensional channel and the scattering channel.

[0039] Specifically, in this embodiment, the "different radiation directions" mentioned in step S121 refer to the fact that the incident light directions of at least two directional light sources are not collinear or coplanar in space. This arrangement ensures that any target point within the sensing area of ​​the soft elastic body can be covered by at least three linearly independent illumination vectors, avoiding singularity in normal calculation caused by collinear or coplanar light directions, thus providing sufficient directional constraints for photometric stereo calculation. To ensure the directionality of the incident light and reduce tolerance sensitivity, directional LEDs with a half-power angle of no more than 15° are preferably used.

[0040] The adaptation logic for "multi-band illumination modes" needs to be determined based on the characteristics of the scattering medium within the soft elastomer. For example, for soft elastomers doped with titanium dioxide particles or silica microspheres, directional light sources in two bands—470 nm (blue light band) and 625 nm (red light band)—can be selected. The blue light band (shorter wavelength) has a larger scattering cross-section for small particles and higher sensitivity to scattering of high-concentration particles, making it suitable for signal acquisition in the scattering channel to characterize microscopic changes in the bulk phase. The red light band (longer wavelength) has a longer mean free path in the medium, resulting in more stable intensity modulation of surface geometric deformation, making it suitable for geometric information calculation in the photometric three-dimensional channel. This band division effectively separates the optical information of the two channels at the physical level, avoiding signal interference.

[0041] Furthermore, to optimize signal quality, the illumination mode in this embodiment is also equipped with polarization control. Specifically, a cross-polarization method is adopted, that is, a polarizer is set at the light source end and a polarizer with a perpendicular direction is set at the camera end. This design can effectively suppress specular reflection and stray light from the surface of the soft elastomer, highlight the bulk scattering signal after multiple scattering and depolarization, and further alleviate the tolerance sensitivity problem.

[0042] In terms of power adjustment, the light source driving module supports dynamic adjustment to adapt to different loads and medium concentrations. Considering the gradient distribution characteristics within the soft elastic body (concentration decreases from the contact side to the away side), the signal-to-noise ratio is optimized by adjusting the total power of the light source: increasing power under low concentration and low load (kPa level) conditions to ensure the intensity of the scattered signal; and appropriately reducing power under high concentration and high load (MPa level) conditions to avoid signal saturation. Based on the spatial layout in step S121, the power of each directional light source can be independently configured to compensate for optical path differences under different illumination angles, ensuring that the collected scattered signals are uniform and consistent.

[0043] In an optional embodiment, step S12 of "configuring a light source to illuminate the soft elastomer in the visual-tactile sensor" specifically includes the following steps: S125. Configure at least two directional light sources with different radiation directions for the soft elastomer; S126. A path-by-path activation control method is adopted for each directional light source to control each directional light source to output illumination light at different times, so as to obtain the independent optical response signal of the soft elastomer corresponding to each directional light source.

[0044] The "path-by-path activation" in step S126 refers to the time-division output of each directional light source through timing control. That is, at any given time, only one directional light source is illuminated, while the others remain off. This control logic aims to ensure that the light signal acquired by the camera comes only from the currently activated specific light direction, physically preventing the superposition of light signals from different directions within the soft elastic body or on the camera's image sensor, thereby guaranteeing the independence and accuracy of each illumination component in the photometric stereo solution.

[0045] Specifically, the timing control logic of S126 is configured to be strictly synchronized with the camera's exposure timing. As a specific implementation, the camera is configured in a high frame rate acquisition mode (e.g., a total acquisition frame rate of N×60fps, where N is the number of light sources). Within each single-frame exposure window, only one directional light source is triggered, and the illumination duration (e.g., 100μs-200μs) is controlled within the camera's exposure time range. The activation interval between each directional light source is configured to be greater than the attenuation time of the light signal in the medium and the dead time of the camera readout (e.g., an interval of 50μs-100μs) to avoid residual signals from the previous frame interfering with the next frame.

[0046] Step S126, the path-by-path activation control, is performed in conjunction with the power adjustment of the light source to compensate for optical path loss under different illumination angles. For directional light sources with larger incident angles (longer optical paths), the driver module outputs a higher drive current (e.g., corresponding to 250mW-300mW optical power); for directional light sources with smaller incident angles (shorter optical paths), the driver module outputs a lower drive current (e.g., corresponding to 150mW-200mW optical power). During the path-by-path activation process, the light source driver module maintains the stability of the optical power through constant current control to ensure the reliability of the calculated data.

[0047] Furthermore, when the light source system simultaneously possesses multi-band characteristics, the sequential activation of S126 can be performed in a "band-priority" order. For example, first, all blue light source bands are activated sequentially to obtain a set of independent optical response signals for the scattering channel; then, all red light source bands are activated sequentially to obtain a set of independent optical response signals for the photometric stereo channel. This activation order helps reduce the thermal drift effect caused by light source switching and ensures the consistency of similar signals.

[0048] In a specific embodiment, the per-path activation control of S126 is achieved by configuring the following parameters: the soft elastomer is made of optical-grade transparent polydimethylsiloxane with a Shore A hardness of 30 and a thickness of 1 mm, internally doped with titanium dioxide particles (particle size D50 = 1.0 μm, volume fraction linearly varying from 1.5% on the contact side to 0.5% away from the contact side); the light source is configured as two spatially anisotropic directional LEDs, one of which is a 470nm blue LED (corresponding to contact side sensitivity, incident elevation angle 30°), and the other is a 625nm red LED (corresponding to geometric calculation, incident elevation angle 50°); the camera frame rate is set to 120fps. During the first frame exposure period, the activated blue LED is lit for 150μs with a power set to 280mW; during the second frame exposure period, the activated red LED is lit for 150μs with a power set to 200mW. With the above configuration, the sensor can output two independent optical response images without superposition interference: the first blue light image mainly carries volume scattering information and is used for mechanical calculations; the second red light image mainly carries surface geometric information and is used for photometric stereoscopic calculations. The combination of the two signals enables mechanical and deformation detection over a wide dynamic range from kPa to MPa.

[0049] like Figure 2 The diagram shown is a flowchart of step S14 provided in an embodiment of the present invention. (Refer to...) Figure 2 As a further optional implementation, step S14, "acquiring visual images of the changes in the scattering medium inside the soft elastomer due to contact deformation of the soft elastomer," specifically includes the following steps: S141. When a soft elastomer is subjected to external contact, causing a change in the state of its internal scattering medium, the scattered light signal generated by the change in the state of the scattering medium is captured. S142. Based on the scattered light signal, generate a visual change image reflecting the change in the state of the scattering medium.

[0050] Specifically, the "scattering medium inside the soft elastomer" in step S141 refers to a substance with stable optical scattering properties pre-doped into the soft elastomer matrix, such as titanium dioxide particles, silicon dioxide particles, or hollow glass microspheres. When the soft elastomer is deformed by external contact (such as pressing or shearing), the internal stress field will cause changes in the spatial distribution density of the scattering medium or the relative spacing between particles. At this time, the optical acquisition unit needs to be configured to capture the scattered light signal generated by this state change in real time. To ensure signal quality, the optical acquisition unit can be equipped with a large aperture lens to increase the amount of light entering, or a narrowband filter matching the scattered light band can be set at the front of the lens to reduce ambient light interference. If dynamic contact deformation needs to be acquired, the frame rate of the optical acquisition unit needs to meet the sampling requirements of the dynamic signal to ensure that no key details of the state change of the scattering medium are missed.

[0051] The "visual change image" generated in step S142 above is a carrier of quantifiable optical features that convey information about the state of the scattering medium. Its format is adapted to subsequent calculation requirements: it can be a static single-frame grayscale image, where the grayscale values ​​at different pixel locations correspond to the intensity of scattered light at different depths within the volume, thus reflecting the density differences of the scattering medium; or it can be a dynamic multi-frame sequence image, where the grayscale change trend between frames reflects the dynamic evolution of the scattering medium state over time. The resolution and field of view of the visual change image must cover the effective sensing area of ​​the soft elastic body to ensure that all changes in the scattering medium within the sensing area are fully included in the image, avoiding the loss of local deformation information due to insufficient field of view. Furthermore, during image generation, the original optical signal can be preprocessed with noise reduction (e.g., Gaussian filtering) or contrast enhancement (e.g., histogram equalization) to amplify the feature differences between the changing and non-changing areas of the scattering medium, providing input data with a higher signal-to-noise ratio for subsequent calculations.

[0052] In an optional embodiment, the soft elastomer is provided with a light-shielding layer and / or a marking layer; the marking layer is distributed on the surface and / or inside the soft elastomer; the step S142 of "generating a visual change image reflecting the change in the state of the scattering medium based on the scattered light signal" specifically includes the following steps: S1421. For the scattered light signal, perform masking processing based on the shading area of ​​the light-shielding layer or the distribution area of ​​the marker layer, remove invalid image areas related to non-scattering medium and retain the effective scattered light signal that only reflects the internal scattering medium of the soft elastomer. S1422, Generate visual change images.

[0053] In this embodiment, the "mask processing" described in step S1421 aims to eliminate the interference of surface auxiliary structures on the analysis of bulk scattering signals. For the original scattered light signal captured in step S141, using the shading area of ​​the light-shielding layer or the distribution area of ​​the marker layer as the mask reference, an image processing algorithm identifies and removes invalid pixel areas, retaining only the effective scattered light signal that reflects the changes in the scattering medium inside the soft elastomer.

[0054] Specifically, the "invalid image region" mentioned in step S1421 refers to the pixel region in the acquired visual image where, due to physical obstruction or optical interference from the auxiliary structure (light-shielding layer or marker layer) on the surface of the soft elastomer, the bulk scattering signal cannot be received or the signal-to-noise ratio of the received signal is lower than the preset calculation threshold. For example, when the light-shielding layer on the surface of the soft elastomer is designed as a grid-like black matrix structure, the pixel region covered by the solid light-shielding lines in the black matrix is ​​determined to be an invalid image region because this region blocks the light path and only reflects the absorption characteristics of the light-shielding material rather than the internal medium state. Similarly, when there is an opaque or specifically reflective marker layer, the pixel region occupied by the marker entity is also considered an invalid image region because it mainly reflects the displacement or surface reflection of the marker. If it directly participates in the calculation of the bulk scattering parameter (Σ), it will cause model distortion.

[0055] Correspondingly, the "effective region" corresponding to the "effective scattered light signal" refers to the set of pixels corresponding to the "windowed area" or "transparent area" on the surface of the soft elastomer that is not blocked by the light-shielding layer or the marker layer. Within these regions, external illumination light can smoothly penetrate into the interior of the soft elastomer, interact with the doped scattering medium (multiple scattering, absorption), and carry information about internal optical path and density changes before being captured by the photosensitive structure. These signals directly carry the integral attenuation information of light along its propagation path within the medium, satisfying the physical constraints of the path integral scattering model. They are the only effective data source for subsequent inversion of scattering parameters and joint calculation of geometric and mechanical characteristics. Strictly distinguishing between these two types of regions through masking ensures the purity of the input data to the solution model and improves the robustness of the algorithm.

[0056] The masking process can be implemented using either a pre-calibration mode or a dynamic recognition mode. The pre-calibration mode is suitable for scenarios where the marker layer position is fixed. It establishes a coordinate index library of invalid regions using pre-acquired reference images, and then directly calls the mask template for filtering during subsequent processing. The dynamic recognition mode, on the other hand, uses image segmentation algorithms to identify high-contrast landmarks or occlusion edges in each frame of the image in real time, dynamically generating a mask matrix. After filtering in step S1421, step S1422 reconstructs a clean visual change image using the retained effective scattered light signals, ensuring that the image primarily carries the physical change information within the volumetric phase, providing a reliable data foundation for the joint calculation of photometric stereo and volumetric scattering.

[0057] like Figure 3 The diagram shown is a flowchart of step S16 provided in an embodiment of the present invention. (Refer to...) Figure 3 As an optional implementation, step S16, which involves performing photometric stereo and volumetric scattering comprehensive calculations on the visual change image to obtain the mechanical characteristics and / or deformation characteristics of the soft elastic body undergoing contact deformation, specifically includes the following steps: S161. Extract the first image features for photometric stereo calculation and the second image features for scattering characteristic analysis from the visual change image; S162. Based on the illumination direction parameters of the light source and the first image features, the initial geometric information of the soft elastic body is determined by photometric stereo calculation. S163. Based on the intensity attenuation information of the second image features and using the initial geometric information as a path constraint, determine the initial scattering parameters of the soft elastic body. S164. Perform joint optimization on the initial geometric information and initial scattering parameters to obtain the corrected geometric information and corrected scattering parameters; S165. Determine the deformation characteristics of the soft elastic body based on the corrected geometric information; S166. Based on the preset calibration relationship, the corrected scattering parameters are mapped to the mechanical characteristics of the soft elastic body.

[0058] Specifically, in step S161, for the visually changing image, feature extraction is performed based on the differences in the optical signal attributes carried by the image, resulting in two types of image features. The first type is a first image feature used for photometric stereoscopic calculation. This feature mainly characterizes the surface geometric deformation information of the soft elastomer (such as surface normal and depth changes). As a preferred approach, when the light source uses multi-band illumination, the image channel corresponding to the band with shallower penetration depth or stronger surface reflection (e.g., the 625nm red light band) is selected as the first image feature; or the gradient component in the image that produces significant brightness differences with changes in illumination direction is extracted as the first image feature. The second type is a second image feature used for scattering characteristic analysis. This feature mainly characterizes the optical response of the scattering medium with changes in load. As a preferred approach, the image channel corresponding to the band with deeper penetration depth or more sensitive to bulk scattering (e.g., the 470nm blue light band) is selected as the second image feature; or the intensity component in the image that produces overall brightness attenuation with changes in optical path is extracted as the second image feature. It should be noted that the first image feature and the second image feature may be coupled in terms of physical signal, and subsequent steps will be based on this coupling relationship for joint calculation.

[0059] In step S162, the first image features are processed using a photometric stereo algorithm. Based on pre-calibrated light source illumination direction parameters (i.e., light source vectors), a set of mapping equations between surface normals and image brightness (such as the Lambertian reflection model or the bidirectional reflectance distribution function (BRDF) model) is constructed. This set of equations is solved using the least squares method to determine the initial normal vector field (NormalMap) and relative depth map of the soft elastic body surface, serving as the initial geometric information of the soft elastic body.

[0060] In step S163, a path integral model based on bulk scattering is established. This model describes the relationship between the intensity attenuation of light propagating inside the soft elastic body and the optical path length and the density of the scattering medium. Using the initial geometric information (surface normal) obtained in step S162, the refraction path length of the light inside the soft elastic body is calculated, and this path length is introduced into the model as an optical path constraint. Based on the intensity attenuation information (such as logarithmic light intensity) of the second image features, the bulk path integral scattering parameter (Σ) characterizing the distribution state of the scattering medium is calculated and used as the initial scattering parameter of the soft elastic body.

[0061] In step S164, a joint optimization model is constructed, comprising a geometric deformation term and a scattering medium variation term. This model uses minimizing the residual between the observed light intensity and the theoretical model light intensity as the objective function, and utilizes the physical correspondence between the geometric deformation and mechanical response of the soft elastic body as a constraint. Starting with the initial geometric information and initial scattering parameters, the model is iteratively solved using either Alternating Optimization (AEM) or Alternating Directional Multiplier Method (ADMM). During the iteration process, the updated scattering parameters are used to correct the calculation of the geometric normal, and simultaneously, the updated geometric normal is used to correct the optical path constraint of the scattering parameters until the model converges, thereby obtaining the corrected geometric information (precise normal and depth) and the corrected scattering parameters (precise volume scattering integral).

[0062] In step S165, the three-dimensional morphology of the soft elastomer is directly reconstructed using the corrected geometric information to determine its deformation characteristics, such as the depth of the concavity and convexity of the contact area and the surface gradient distribution.

[0063] In step S166, based on a pre-established mechanical calibration relationship (such as the correspondence curve between scattering parameters and pressure obtained through indentation experiments), the corrected scattering parameters are mapped to the mechanical characteristics of the soft elastic body. Specifically, the normal pressure distribution is calculated using the corrected bulk scattering parameters; if combined with the displacement information of the marker layer, the tangential shear force distribution can also be calculated using the gradient distribution of the scattering parameters, thereby achieving decoupled output of geometric deformation and mechanical response.

[0064] In an optional embodiment, in step S162, a mapping relationship between light intensity and surface normal in the first image feature is established using a surface reflection model. Based on the pre-calibrated light source illumination direction parameters and the observed light intensity of the first image feature, a set of photometric solid equations is constructed.

[0065] Based on the pre-defined condition that the soft elastomer surface follows the Near Lambert reflection characteristics or the polynomial bidirectional reflection distribution function (BRDF) model, for any pixel in the image... , No. Observational light intensity from each light source direction With surface normal The relationship is represented as: in, Indicating the first image feature, the first... The observed light intensity value pointing downwards; This represents the flat field and gain calibration coefficients of the light beam pointing downwards; Indicates the first The illumination direction vector of each light source; Indicates surface albedo or BRDF term; This represents the surface normal vector to be solved. The surface normal vector represents the local spatial orientation of the soft elastomer surface at this pixel.

[0066] The photometric solid equations are solved using numerical algorithms (such as the least squares method) to determine the initial normal vector of the soft elastic body surface. Furthermore, based on the integrability constraint of the normal field, the initial normal vector... Perform two-dimensional integration to obtain the relative depth. (Relative Depth) represents the height variation of a soft elastomer's surface relative to a reference plane. Initial normal vector. With relative depth The initial geometric information that together constitutes the soft elastic body.

[0067] In step S163, the initial normal vector from the initial geometric information is used. As a priori constraint, the initial scattering parameters are solved based on the path integral scattering model. The path integral scattering model is used to characterize the nonlinear attenuation characteristics of light propagating inside a soft elastic body doped with a scattering medium.

[0068] Specifically, the observed light intensity of the scattering channel Integral scattering parameters with bulk path The relationship is constructed as follows: in, Indicating the second image feature, the first The intensity of the observed light beam pointing downwards; Represents the reflection component related to surface geometry; This represents the path-integrated scattering parameter at the pixel location along the thickness direction of the soft elastomer. This represents the angle of refraction of the incident ray inside the soft elastic body. In this step, the ray vector is used. With the initial normal vector Calculate the angle of refraction ,Will As an optical path constraint, the initial scattering parameters are calculated by performing robust regression on the logarithm of the above formula. .

[0069] In step S164, a joint optimization model is constructed, incorporating both geometric deformation and scattering medium variation terms. The objective function of the joint optimization model is configured to minimize the error between the observed light intensity and the theoretically predicted light intensity, with spatial smoothness constraints introduced. The objective function is constructed as follows: In the objective function: This represents the surface normal vector to be optimized; This represents the relative depth to be optimized, which is related to the surface normal vector. Satisfies the integrability constraint; This represents the path integral scattering parameters to be optimized; This represents the measured light intensity of the image; This represents the theoretically predicted light intensity calculated based on the current variables. The theoretically predicted light intensity combines the aforementioned surface reflection model and path integral scattering model. and This represents the regularization parameter, used to control the surface normal vector. With path integral scattering parameters Spatial smoothness; This represents the gradient operator.

[0070] With initial geometric information ( ) and initial scattering parameters To optimize the starting point, an alternating minimization (AO) or alternating direction multiplier method (ADMM) strategy is used to iteratively solve the joint optimization model: the scattering parameters are integrated along a fixed path. Under the condition of updating geometric information (i.e., surface normal vector) With relative depth (Under fixed geometric information, update the path integral scattering parameters) Iterate until the objective function converges, and output the corrected geometric information (including the corrected surface normal vector). With the corrected relative depth ) and the corrected scattering parameters .

[0071] In step S165, the corrected relative depth is used. The three-dimensional morphological characteristics of the soft elastic body are determined, which are the deformation characteristics.

[0072] In step S166, the corrected scattering parameters are adjusted using a pre-established calibration relationship. It is converted into mechanical characteristics.

[0073] Specifically, based on the power function model determined by the indentation calibration experiment, the scattering parameters are... Mapped to normal pressure : in, and This represents the coefficient determined through calibration.

[0074] Furthermore, the spatial gradient of the corrected scattering parameters is calculated. Based on the mapping function determined by the shear calibration experiment, the spatial gradient is... Mapped to shear stress .

[0075] In an optional embodiment, step S164, "jointly optimizing the initial geometric information and initial scattering parameters to obtain the corrected geometric information and corrected scattering parameters," specifically includes the following steps: S1641. Establish a joint optimization model, which includes geometric deformation variables and scattering medium variation variables. In step S1641, a joint optimization model is constructed, the objective function of which is configured to characterize the consistency between the observed image data and the physically rendered model. The geometric deformation variable is defined as the surface normal vector. The variable in the scattering medium is defined as the path integral scattering parameter. .

[0076] Objective function of joint optimization model The structure is as follows: in, This is a data fidelity term used to measure the changes in geometric deformation based on the current value. and scattering medium variation variables Calculated theoretical predicted light intensity and measured light intensity The residuals between them. This theoretical prediction of light intensity integrates the surface reflection model (which depends on...). ) and bulk scattering model (dependent on And optical path). Its mathematical expression is: For regularization constraints, including those on the normal direction Spatial smoothing constraints (such as the total variational TV norm) and the scattering parameters Spatial continuity constraints are used to improve the well-posedness of the solution.

[0077] S1642. Starting with the initial geometric information and initial scattering parameters, the joint optimization model is iteratively solved to obtain the corrected geometric information and corrected scattering parameters.

[0078] In step S1642, the initial normal vector obtained in the previous step is used. and initial scattering parameters As the initial value for iteration, an alternating iteration strategy (such as the alternating minimization method) is used to solve the objective function.

[0079] Furthermore, regarding the process of "iteratively solving the joint optimization model," in order to address the angular blind spots or singularities in photometric stereo solutions, this embodiment introduces a dynamic update mechanism based on pixel-level confidence, specifically including the following steps: S1642.1 Determine the confidence index for the current pixel in the visually changing image.

[0080] Specifically, a local illumination matrix is ​​constructed based on the effective illumination configuration of the current pixel. The confidence index for the solution includes: Effective Light Count. ): Counts the number of light sources that illuminate this pixel and are not occluded (not in shadow).

[0081] S1642.2 Determine whether the calculated confidence index meets the preset blind zone condition.

[0082] The blind zone condition is configured to identify ill-conditioned regions where the solution is unstable. This is achieved when the effective number of light directions is satisfied. Less than the minimum required number of photometric stereo solutions (e.g.) The statement indicates that there is a lack of sufficient lighting constraints to solve the 3D normal, and therefore it is determined that the blind zone condition is met.

[0083] S1642.3 In response to the solution confidence index satisfying the blind zone condition, an asymmetric update operation is performed during the iterative solution process.

[0084] When it is determined that the area has entered a blind zone, the following specific operations are performed: Stop updating the geometric information of the current pixel: In the current iteration, cut off the data fidelity term. normal to this pixel The gradient update path ensures that the geometric information of the point does not diverge with the change of the residual in the current round.

[0085] Determine the geometric information of the current pixel based on the geometric information of neighboring pixels: Utilize the normal information of pixels in the non-blind zone (high confidence) within the neighborhood, and infer and reset the normal of the current pixel through interpolation or integrability propagation. To maintain the spatial continuity of the geometric surface.

[0086] Iterative optimization of the scattering parameters of the current pixel: Based on fixing or repairing the geometric information (i.e., fixing the optical path constraint), continue to utilize the data fidelity term. Scattering parameters of this pixel Perform iterative updates to obtain the corrected scattering parameters. .

[0087] Through the above steps, in blind areas with insufficient lighting conditions (such as steeply tilted surfaces or shadow areas), the algorithm logic automatically switches from "bivariate joint optimization" to "geometrically constrained scattering single-variable optimization", utilizing the robustness of the scattering channel to fill the blind areas of the geometric channel, ensuring that the mechanical and deformation characteristics of the final output are complete and continuous.

[0088] like Figure 11 The diagram shown is a simulation verification result provided in this embodiment. To verify the solution accuracy and decoupling capability of the above-mentioned visual-tactile detection method when processing geometric deformation and volume scattering coupled signals, this embodiment constructs a simulation experiment for a typical tactile scenario.

[0089] In the simulation setup of this embodiment, the contact object is configured as a rigid sphere with a radius of R = 3 mm, and the loading method is set to the rigid sphere pressing perpendicularly onto the surface of the soft elastomer, with a pressing depth of 1 mm. The lighting system is configured as an RGB three-color light source mode, with the azimuth angles of the three light sources symmetrically distributed at 120 degrees (0 degrees, 120 degrees, and 240 degrees respectively), and the incident elevation angle is set to 45 degrees for each. In the construction of the material model, a Lambertian reflection model is adopted and volume scattering characteristics are introduced: the albedo of the soft elastomer surface (the scattering parameter affected by the concentration of the scattering medium in the physical model) is set to be non-uniformly distributed, and the albedo increases linearly with the pressing depth. This setting aims to simulate the physical phenomenon in real tactile sensors where the density of the internal scattering medium increases due to pressure on the soft elastomer, resulting in a significant brightening of the tactile image in the pressure area (i.e., brightening under pressure).

[0090] The solution method is based on the three-source photometric stereo algorithm. It separates geometric information from scattering information by constructing and solving a set of linear equations containing geometric normals and scattering densities, and finally reconstructs the three-dimensional shape through gradient integration. Figure 11 The images and solution results at each processing stage of the simulation are shown: like Figure 11 As shown in (a), this is a simulated RGB image. The image shown is the original visual change image acquired by the visual-tactile sensor under the above lighting conditions. The image exhibits obvious coupled characteristics influenced by both geometric deformation (shadows caused by changes in surface normal) and bulk scattering density (albedo changes caused by pressure-induced brightening), with the contact center region appearing brighter due to enhanced scattering.

[0091] like Figure 11 (b) shows the calculated scattering density distribution map (Rho Map). The image shows the scattering density distribution obtained using the algorithm described in this application. Figure 11 The scattering parameter distribution separated in (a) shows that the scattering density is highest at the center of the contact area (corresponding to the bright yellow area in the figure) and gradually decreases towards the edge (corresponding to the dark red area in the figure), accurately reflecting the stress concentration at the center and the compression law of the medium caused by the sphere pressing, proving the effectiveness of the algorithm in extracting scattering channel information.

[0092] like Figure 11 (c) shows the actual geometric shape (Ground Truth). It shows the actual three-dimensional depth distribution of the rigid sphere indentation in the simulation settings, serving as a benchmark reference for verifying the algorithm's accuracy.

[0093] like Figure 11 (d) shows the reconstructed geometric topography. The image shows the 3D topography reconstructed based on the separated geometric normal information and through gradient integration (eliminating the influence of density variations). (Comparison) Figure 11 (c) and Figure 11 (d) shows that the reconstructed geometry highly reproduces the real shape of the spherical cap, and the surface is smooth, without artifacts or depth distortion caused by uneven albedo (scattering density).

[0094] This simulation example demonstrates that the detection method provided in this application can effectively decouple the geometric normal information and scattering density information coupled in the visual image. While accurately obtaining the scattering density distribution that characterizes the mechanical features, it eliminates the interference of changes in the state of the scattering medium on geometric reconstruction, thus achieving high-precision three-dimensional morphology restoration.

[0095] like Figure 4 As shown, according to one aspect of this application, a visual-tactile sensor 10 is also provided, comprising: A soft elastic body 11 that responds to physical contact and generates contact deformation, with a scattering medium 111 inside the soft elastic body, and the distribution or orientation of the scattering medium changes when the soft elastic body deforms. A light source 12 that illuminates soft elastomers and internal scattering media and provides the illumination required for photometric stereoscopic calculation; A photosensitive structure 13 receives light scattered by a scattering medium after being illuminated by a light source 12 and generates a visual change image. The visual change image is used for comprehensive calculation of photometric stereo and volume scattering to obtain the mechanical characteristics and / or deformation characteristics corresponding to the contact deformation of the soft elastic body.

[0096] Specifically, the soft elastomer 11 serves as the tactile sensing body of the visual-tactile sensor 10 and is configured to produce recoverable deformation when contacted by an external entity.

[0097] The soft elastomer 11 is made of a flexible matrix material with optical transmittance, including but not limited to polydimethylsiloxane (PDMS), thermoplastic elastomer (TPE), polyurethane (PU), or dual-network hydrogel. The Shore A hardness of the soft elastomer 11 is typically between 5 and 90, preferably between 10 and 40. The thickness and shape of the soft elastomer 11 are adapted according to the testing requirements, for example, designed as a sheet, block, or curved structure suitable for a robot fingertip, with a thickness preferably at least 50 μm.

[0098] In an optional embodiment, the scattering medium 111 is a particle or particle cluster possessing scattering properties and doped within a soft elastic body. For example... Figure 4 As shown in the magnified view, the scattering medium 111 consists of particles dispersed within the soft elastomer 11 matrix. In terms of material selection, the scattering medium 111 is made of materials with optical scattering properties, specifically including titanium dioxide (TiO2), silicon dioxide (SiO2), zinc oxide, hollow glass microspheres, or polymer microspheres (such as PS, PMMA). Furthermore, the refractive index of the scattering medium 111 differs from that of the soft elastomer 11 matrix material (e.g., the difference is not less than 0.1) to ensure significant bulk scattering of light when passing through the soft elastomer 11. Regarding dimensional parameters, the particle size (D50) of the scattering medium 111 is preferably between 0.2 μm and 5 μm, and the volume fraction is between 0.1% and 5%.

[0099] In terms of distribution characteristics, the scattering medium 111 exhibits a gradient distribution within the soft elastic body 11. Specifically, the doping concentration or particle density of the scattering medium 111 gradually decreases from the side of the soft elastic body 11 in contact with the solid (upper surface) to the side away from the contact (lower surface). This gradient distribution structure, combined with the differences in penetration depth of different wavelength bands, forms a pseudo-tomographic sensitivity distribution, enabling the sensor to handle both kPa-level micro-touch measurements and MPa-level heavy-duty measurements. When the soft elastic body 11 deforms under external force, the internal scattering medium 111 shifts, aggregates, or changes orientation with the matrix, causing changes in the scattering density and optical path in local areas, thereby modulating the illumination light.

[0100] The light source 12 is located on the side of the soft elastic body 11 away from the contact surface, and is configured to provide directional illumination to the soft elastic body 11 and the internal scattering medium 111. It should be noted that, although... Figure 4Only one light source component is shown schematically, but in actual configuration, light source 12 includes at least two directional light sources (e.g., directional LEDs) with different spatial arrangements. The radiation directions of each directional light source to the soft elastic body 11 are different and non-coplanar, providing multi-directional light constraints to meet the requirements of photometric stereoscopic calculation. As a preferred embodiment, the principal axis of the light distribution of the directional light sources illuminates the soft elastic body at an angle (e.g., elevation angle) between 60 and 120 degrees. Light source 12 is configured to support multi-band illumination modes (e.g., emitting red and blue light) or time-division illumination modes to adapt to the scattering characteristics of the scattering medium 111, assisting in the separation of geometric and mechanical information channels.

[0101] The photosensitive structure 13 is located on the side of the soft elastomer 11 away from the contact surface, and its optical axis points towards the sensing area of ​​the soft elastomer 11. The photosensitive structure 13 is configured to receive light scattered by the scattering medium 111 after illumination by the light source 12, and generate a visual change image. The acquisition frame rate of the photosensitive structure 13 is synchronized with the timing control of the light source 12. When the soft elastomer 11 deforms, causing a change in the state of the scattering medium 111, the light intensity distribution of the image acquired by the photosensitive structure 13 changes. This visual change image carries the surface geometry information of the photometric stereo channel and the volume load information of the scattering channel, which is used for subsequent comprehensive calculations to obtain the mechanical characteristics and / or deformation characteristics corresponding to the contact deformation of the soft elastomer.

[0102] like Figure 5 The diagram shown is a schematic representation of the internal scattering medium distribution of a soft elastomer provided in an embodiment of this application. (Refer to...) Figure 5 In an optional embodiment, the scattering medium 111 is distributed in a non-uniform gradient within the soft elastomer 11 to accommodate detection requirements at different depths.

[0103] In an alternative embodiment, regarding the concentration distribution of the scattering medium, in the soft elastomer, the doping concentration of the scattering medium gradually decreases from the side in contact with the solid to the side away from that contact. Specifically, as... Figure 5As shown, the upper part of the soft elastomer 11 has a higher density of shadow dots, representing a high-concentration scattering region on the contact side; the lower part has a lower density of shadow dots, representing a low-concentration scattering region on the away side. As a specific parameter configuration, the volume fraction (φ) of the scattering medium 111 is not constant within the soft elastomer 11, but rather decreases linearly or exponentially monotonically from the contact side surface to the away side surface. For example, the volume fraction of the contact side surface layer is set to 1.5% to 5%, the volume fraction of the away side bottom layer is set to 0.1% to 0.5%, and the concentration gradient ratio between the contact side and the away side (i.e., the volume fraction ratio of the surface layer to the bottom layer) is controlled between 1.2 and 5. This concentration gradient design enables the contact side to have a highly sensitive light intensity suppression capability for small loads, while the away side provides signal linearity under large loads while ensuring the light penetration depth.

[0104] In an optional embodiment, such as Figure 5 As shown, the scattering medium 111 exhibits a specific non-uniform gradient distribution in the soft elastic body 11, thereby enabling the modulation of the optical response characteristics of the contact interface.

[0105] Along the direction from the side of the soft elastomer 11 that contacts the solid, i.e. the contact surface, to the side away from the contact, i.e. the bottom surface, the distribution characteristics of the scattering medium 111 satisfy at least one of the following conditions: the particle size of the scattering medium 111 gradually decreases, and / or the doping density or concentration of the scattering medium 111 gradually decreases.

[0106] Specifically, the distribution of the physical properties of particles mainly includes two aspects: particle size distribution and material density distribution.

[0107] Regarding the particle size distribution, on the side closer to the contact between the soft elastomer 11 and the solid, larger particles (e.g., D50 of 1.0 μm to 5 μm) are concentrated; while on the side away from this contact, smaller particles (e.g., D50 of 0.2 μm to 0.8 μm) are concentrated. This distribution utilizes the faster migration rate of large-diameter particles during settling or floating, causing them to preferentially accumulate on the contact surface, forming a sensitive layer with a high scattering cross-section.

[0108] Regarding material density distribution, if the scattering medium contains particles of different densities, particles with larger density differences (i.e., those with a large density difference from the matrix) will be concentrated on the side closer to the contact. For example, if particles with a density less than the matrix, such as hollow glass microspheres, are used, they will naturally float and aggregate on the contact side, i.e., the upper surface; if particles with a density greater than the matrix, such as titanium dioxide, are used, they will settle and aggregate on the contact side using gravity or centrifugal force through an inverted curing process.

[0109] In summary, the particles or particle clusters in the soft elastomer 11 exhibit a gradient distribution of particle size and / or density along the direction from the surface in contact with the solid to the surface away from the contact surface, and this gradient distribution matches the gradually decreasing doping concentration of the scattering medium. This means that on the contact side of the soft elastomer 11, not only is the doping concentration of the scattering medium highest, but this region is also mainly composed of large-diameter or specific-density particles, collectively forming a surface structure with high scattering intensity; on the away side, not only is the doping concentration lower, but it is also mainly composed of small-diameter particles, forming a bottom layer structure with low scattering intensity, thereby optimizing the light transmission performance of the deeper layers while ensuring surface sensitivity.

[0110] The aforementioned gradient distribution structure can be achieved through a specific fabrication process.

[0111] One method is the layered curing process, which divides the soft elastomer matrix into multiple parts, mixes particles of different concentrations and sizes in each part, and injects them into the mold in sequence from the contact side to the back side and then cures them.

[0112] The second method is controlled sedimentation, in which a matrix containing polydisperse particles is injected into a mold. Taking advantage of Stokes' law, which states that larger particles settle or float faster, a gradient distribution of large or high-concentration particles on the contact side and small or low-concentration particles on the opposing side is naturally formed during the curing process. Alternatively, when the particles contain magnetic or conductive cores, an external magnetic or electric field is applied to actively control the field-induced sedimentation, forming the aforementioned gradient structure in which the position, concentration, and particle size match.

[0113] In an optional embodiment, such as Figure 6 and Figure 7 As shown, the soft elastomer 11 is provided with an auxiliary functional layer, specifically including a light-shielding layer 14 and / or a marking layer 15.

[0114] In an optional embodiment, such as Figure 6 As shown, the visual-tactile sensor 10 includes a soft elastomer 11, a light-shielding layer 14, a light source 12, and a photosensitive structure 13. Regarding the configuration of the light-shielding layer 14, a light-shielding layer is provided on the side of the soft elastomer that contacts the solid, and the light-shielding layer is used to shield ambient light.

[0115] Specifically, the light-shielding layer 14 typically covers the upper surface of the soft elastomer 11. The light-shielding layer 14 is made of an elastic film containing light-absorbing components (such as carbon black or organic dyes), with a hemispherical reflectivity preferably less than 5% in the 450-700nm visible light band, and a thickness controlled between 5-100μm. This material and thickness configuration aims to ensure that the light-shielding layer 14 can effectively block external ambient light from entering the photosensitive structure 13, preventing stray light from interfering with the acquisition of bulk scattering signals. Simultaneously, because the light-shielding layer 14 is elastic and relatively thin, when an external object contacts the light-shielding layer 14, the load of the object can be smoothly transferred to the soft elastomer 11 through the light-shielding layer 14, causing the soft elastomer 11 to produce contact deformation matching the shape of the object. Regarding the distribution, the light-shielding layer 14 can cover the entire contact surface, or it can be designed as a windowed black matrix structure, blocking light only in non-sensing areas.

[0116] In an optional embodiment, such as Figure 7 As shown, the visual-tactile sensor 10 includes a soft elastomer 11, a marking layer 15, a light source 12, and a photosensitive structure 13. Regarding the configuration of the marking layer 15, the soft elastomer is provided with a marking layer, which is distributed on the surface and / or inside the soft elastomer; when the soft elastomer undergoes contact deformation in response to physical contact, the marking layer also undergoes displacement or deformation.

[0117] Specifically, the marker layer 15 contains several discretely distributed markers (such as...) Figure 7 (The block structure in the middle). The marker material is selected from non-directional reflective microdots, absorptive microdots, or fluorescent markers, with an apparent reflectivity preferably less than 10% and a total thickness not exceeding 30 μm. Regarding the distribution location, the marker layer 15 can be as follows: Figure 7 The markers, discretely attached to the upper surface of the soft elastomer 11 in contact with the solid, can also be suspended within the matrix of the soft elastomer 11 and flow with the matrix. When the soft elastomer 11 is deformed under stress, the markers in the marker layer 15, acting as passive tracers attached to the matrix, will precisely follow the displacement or minute deformation of the matrix. By capturing the motion trajectory of the marker layer 15 (e.g., extracting the displacement field using optical flow or DIC technology), the photosensitive structure 13 can obtain auxiliary information on tangential mechanical characteristics (such as shear stress), thereby cross-validating with the bulk scattering signal and improving the accuracy of the mechanical calculation.

[0118] It should be noted that the light-shielding layer 14 and the marking layer 15 can be set separately or in combination. For example, setting the fluorescent marking layer 15 in the window area of ​​the light-shielding layer 14 can achieve both ambient light shielding and provide a tracking beacon for shear displacement.

[0119] In an optional embodiment, such as Figure 8 As shown, the visual-tactile sensor 10 also includes a support layer 16. (Refer to...) Figure 8The visual-tactile sensor 10 includes a soft elastomer 11, a support layer 16, a light source 12, and a photosensitive structure 13. Regarding the configuration of the support layer, the support layer 16 is provided on the side of the soft elastomer 11 that faces away from the solid contact surface.

[0120] Specifically, the support layer 16 is located on the lower surface of the soft elastomer 11 and is made of a rigid or semi-rigid material with high light transmittance, such as optical glass, polymethyl methacrylate (PMMA), or polycarbonate (PC). This support layer 16 is configured to provide mechanical support for the soft elastomer 11, preventing it from collapsing or excessively deforming in non-measuring areas, thereby improving the structural stability and measurement range of the visual-tactile sensor 10. Simultaneously, the support layer 16 is optically transparent, not obstructing the illumination light emitted by the light source 12 from entering the interior of the soft elastomer 11, nor obstructing the scattered light modulated by the scattering medium from exiting to the photosensitive structure 13. Therefore, the photosensitive structure 13 can pass through the support layer 16 to capture the bulk scattering signal inside the soft elastomer 11, thereby calculating the mechanical characteristics of the entity acting on the soft elastomer 11.

[0121] In an optional embodiment, the soft elastomer assembly of the visual-touch sensor 10 employs a composite layered structure, such as... Figure 9 As shown, the visual-tactile sensor 10 includes a soft elastomer, a support layer 16, a light source 12, and a photosensitive structure 13; wherein, the soft elastomer is divided into sub-layers with different functions along the thickness direction, specifically including a first soft elastomer 111 close to the contact side and a second soft elastomer 112 away from the contact side.

[0122] Specifically, the first soft elastomer 111 is disposed on the surface of the soft elastomer assembly that contacts the external entity. It contains a scattering medium and is configured as the primary sensing layer, used to scatter and modulate incident light to generate a bulk scattering signal in response to contact deformation. The second soft elastomer 112 is disposed between the first soft elastomer 111 and the support layer 16. It does not contain a scattering medium and is an optically transparent elastomer layer, configured as both a light transmission layer and a mechanical buffer layer. The support layer 16 is disposed on the side of the second soft elastomer 112 away from the entity contact and is made of a rigid or semi-rigid material with optical transmittance, used to support the first soft elastomer 111 and the second soft elastomer 112.

[0123] By controlling the first soft elastomer 111 containing the scattering medium to a preset thickness, it is possible to ensure that the scattered light signal is concentrated on the contact surface, thereby improving the detection sensitivity. By setting a second soft elastomer 112 without the scattering medium, it is possible to increase the overall thickness of the soft elastomer to provide a greater mechanical deformation range, while avoiding excessive attenuation of light in the transmission path due to excessively thick scattering layer. This improves the flexible contact capability of the sensor while ensuring the optical signal strength.

[0124] It should be noted that this embodiment is only illustrated using a two-layer structure as an example. Without departing from the concept of this application, the first soft elastomer 111 and the second soft elastomer 112 can also transition using a gradual change in concentration. In an optional embodiment, such as... Figure 10 As shown, the visual-tactile sensor 10 also includes an optical element 19. (Refer to...) Figure 10 The visual-tactile sensor 10 includes a soft elastomer 11, an optical element 19, a light source 12, and a photosensitive structure 13. Regarding the configuration of the optical path structure, an optical element deflects the light path between the photosensitive unit and the soft elastomer. The optical element includes, but is not limited to, a mirror, a prism, and a lens.

[0125] Specifically, the optical element 19 is disposed in the optical path between the soft elastomer 11 and the photosensitive structure 13. In a preferred embodiment, the optical element 19 is a mirror or prism used to deflect the scattered light emitted from the soft elastomer 11 at a specific angle (e.g., 90 degrees). After the light source 12 illuminates the internal scattering medium of the soft elastomer 11 to generate scattered light, this scattered light is reflected by the optical element 19 to the photosensitive structure 13. By introducing the optical element 19 to fold the optical path, the thickness of the visual-tactile sensor 10 in the longitudinal direction (i.e., perpendicular to the surface of the soft elastomer) can be reduced, allowing it to be adapted to space-constrained installation environments, such as integration into a thin robotic fingertip structure. The photosensitive structure 13 receives the light signal deflected by the optical element 19 and then calculates the mechanical characteristics of the entity acting on the soft elastomer 11.

[0126] The various embodiments in this application specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the visual-touch sensor embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0127] The above description is merely an embodiment of this application and is not intended to limit the scope of one or more embodiments of this application. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A visual-tactile detection method, characterized in that, The application in visual-tactile sensors, which can perform comprehensive calculations of photometric stereo and volumetric scattering based on the distribution of their internal scattering media, includes the following steps: A light source is configured to illuminate the soft elastomer in the visual-tactile sensor; Collect visual images of the changes in the scattering medium inside the soft elastomer due to the contact deformation of the soft elastomer. The visual change image is subjected to photometric stereo and volume scattering integrated calculations to obtain the mechanical characteristics and / or deformation characteristics of the soft elastic body under contact deformation.

2. The method according to claim 1, characterized in that, The step of "configuring a light source to illuminate the soft elastomer in the visual-tactile sensor" specifically includes the following steps: At least two directional light sources are configured such that each directional light source radiates to the soft elastic body in a different direction, so as to provide multi-directional illumination that meets the requirements of photometric stereoscopic solution; wherein, the directional light source adopts a multi-band illumination mode, and the multi-band illumination mode is adapted to the scattering characteristics of the internal scattering medium of the soft elastic body, so as to realize the separation of optical information between the photometric stereoscopic channel and the scattering channel.

3. The visual-tactile detection method according to claim 1, characterized in that, The step of "configuring a light source to illuminate the soft elastomer in the visual-tactile sensor" specifically includes the following steps: Configure at least two directional light sources with different radiation directions to the soft elastomer; Each of the directional light sources is activated sequentially, and the directional light sources are controlled to output illumination light at different times, so as to obtain the independent optical response signal of the soft elastomer corresponding to each directional light source.

4. The method according to claim 1, characterized in that, The step of "acquiring visual images of the changes in the scattering medium inside the soft elastic body due to contact deformation of the soft elastic body" specifically includes the following steps: When a soft elastomer is subjected to external contact that causes a change in the state of its internal scattering medium, the scattered light signal generated by the change in the state of the scattering medium is captured. Based on the scattered light signal, a visual change image reflecting the change in the state of the scattering medium is generated.

5. The method according to claim 4, characterized in that, The soft elastomer is provided with a light-shielding layer and / or a marking layer; the marking layer is distributed on the surface and / or inside the soft elastomer; the step of "generating a visual change image reflecting the change in the state of the scattering medium based on the scattered light signal" specifically includes the following steps: The scattered light signal is masked according to the shading area of ​​the light-shielding layer or the distribution area of ​​the marker layer to remove invalid image areas related to the non-scattering medium and retain the effective scattered light signal that only reflects the internal scattering medium of the soft elastomer. Generate the visual change image.

6. The method according to claim 1, characterized in that, The step of "performing a comprehensive photometric stereo and volume scattering calculation on the visual change image to obtain the mechanical characteristics and / or deformation characteristics of the soft elastic body undergoing contact deformation" specifically includes the following steps: Extract a first image feature for photometric stereo calculation and a second image feature for scattering characteristic analysis from the visual change image; Based on the illumination direction parameters of the light source and the first image features, the initial geometric information of the soft elastic body is determined by photometric stereo calculation; Based on the intensity attenuation information of the second image features, and using the initial geometric information as a path constraint, the initial scattering parameters of the soft elastic body are determined. The initial geometric information and initial scattering parameters are jointly optimized to obtain the corrected geometric information and corrected scattering parameters; The deformation characteristics of the soft elastic body are determined based on the corrected geometric information; Based on a preset calibration relationship, the corrected scattering parameters are mapped to the mechanical characteristics of the soft elastomer.

7. The method according to claim 6, characterized in that, The step of "jointly optimizing the initial geometric information and initial scattering parameters to obtain the corrected geometric information and corrected scattering parameters" specifically includes the following steps: A joint optimization model is established, which includes geometric deformation variables and scattering medium variation variables; Using the initial geometric information and the initial scattering parameters as the starting point for optimization, the joint optimization model is iteratively solved to obtain the corrected geometric information and the corrected scattering parameters.

8. The method according to claim 7, characterized in that, The step of "iteratively solving the joint optimization model to obtain the corrected geometric information and the corrected scattering parameters" specifically includes the following steps: For the current pixel in the visual change image, a solution confidence index is determined; wherein, the solution confidence index includes: constructing the number of effective light directions based on the illumination direction parameters; In response to the solution confidence index satisfying the preset blind zone condition, the following operations are performed during the iterative solution process: stop updating the geometric information of the current pixel and determine the geometric information of the current pixel based on the geometric information of the neighboring pixels; and iteratively optimize the scattering parameters of the current pixel to obtain the corrected scattering parameters.

9. A visual-tactile sensor, characterized in that, include: A soft elastic body that responds to physical contact and undergoes contact deformation, wherein the soft elastic body contains a scattering medium, and the deformation of the soft elastic body causes the distribution or orientation of the scattering medium to change. A light source that illuminates the soft elastomer and its internal scattering medium and provides the illumination required for photometric stereoscopic calculation; A photosensitive structure receives light scattered by the scattering medium after being illuminated by the light source and generates a visual change image. The visual change image is used for comprehensive calculation of photometric stereo and volume scattering to obtain the mechanical characteristics and / or deformation characteristics corresponding to the contact deformation of the soft elastic body.

10. The visual-tactile sensor according to claim 9, characterized in that, The scattering medium is a particle or particle cluster that has scattering properties and is doped inside the soft elastic body.

11. The visual-tactile sensor according to claim 10, characterized in that, The scattering medium is non-uniformly distributed in the soft elastomer; along the direction from the side of the soft elastomer in contact with the solid to the side away from the contact, the particle size of the scattering medium gradually decreases, and / or the density of the scattering medium gradually decreases.

12. The visual-tactile sensor according to claim 9, characterized in that, In the soft elastomer, the doping concentration of the scattering medium gradually decreases from the side in contact with the solid to the side away from the contact.

13. The visual-tactile sensor according to claim 9, characterized in that, The soft elastomer has a light-shielding layer on one side of the surface that contacts the solid, and the light-shielding layer is used to shield ambient light.

14. The visual-tactile sensor according to claim 9, characterized in that, The soft elastomer is provided with a marking layer, which is distributed on the surface and / or inside of the soft elastomer; when the soft elastomer undergoes contact deformation in response to physical contact, the marking layer also undergoes displacement or deformation.

Images