Visual tactile sensor and sensing method thereof

Abstract

The invention relates to a visual tactile sensor and a sensing method thereof. The visual tactile perception method is applied to a visual tactile sensor provided with a soft elastic body, and comprises the following steps: configuring a light source, and enabling the light source to irradiate a reflecting medium which is arranged on one side of a contact object of the soft elastic body and has a non-directional reflection characteristic or a partial non-directional reflection characteristic through the soft elastic body; wherein the reflecting medium comprises a first reflecting medium acting on internal environment reflection of the soft elastomer and a second reflecting medium for marking; the albedo of the first reflecting medium is different from that of the second reflecting medium; collecting a visual change image of a local position and a normal line generated by the reflection medium because the reflection medium contacts the object with the soft elastomer; and performing photometric stereo calculation on the visual change image to perceive the mechanical characteristic data of the soft elastomer in the current state. According to the method, the first reflection medium and the second reflection medium with different reflectivity are arranged, so that a normal resolving blind area caused by the fact that a mark destroys a photometric stereo reflection model is eliminated.

Description

Technical Field

[0001] This invention relates to the field of sensor technology, and in particular to a visual-tactile sensor and its sensing method. Background Technology

[0002] Visual-tactile sensors, as high-precision sensors capable of simulating human tactile perception, capture the deformation information of an elastic body under stress through optical means, thereby inferring the mechanical characteristics of the contact interface. In fields such as dexterous robotic manipulation, medical palpation, and human-computer interaction, visual-tactile sensors have attracted significant attention due to their advantages such as high spatial resolution and resistance to electromagnetic interference.

[0003] Among existing visual-tactile sensing technologies, photometric stereo is one of the core technologies for achieving high-resolution 3D shape reconstruction. Its basic principle is to illuminate the surface of an elastic body using multi-angle light sources and reconstruct a continuous surface normal field based on the shading of reflected light intensity, thereby accurately solving for normal force and surface texture. On the other hand, to sense shear force or detect slippage, existing technologies typically require placing markers on or inside the surface of the elastic body, calculating shear deformation by tracking the displacement of these markers within the image plane.

[0004] However, in attempting to simultaneously achieve high-precision normal and tangential sensing, existing visual-tactile sensors face the technical challenge of "marker points disrupting the photometric stereo solution model." Although some literature has attempted to address this issue, none have been able to completely eliminate the "solution blind zone" introduced by marker points within the photometric stereo solution framework.

[0005] For example, European patent application EP3693139A1 proposes an optical tactile sensor that resolves the normal and lateral components of force by embedding a multi-layered array of markers in an elastic layer. This scheme uses a combination of partially transparent and opaque markers. However, this scheme does not consider the obstructive effect of the opaque markers on photometric stereo processing. During surface normal reconstruction, these opaque markers can block light transmission or exhibit reflective properties drastically different from the surrounding medium, preventing the algorithm from obtaining effective normal gradient information in the marked areas.

[0006] For example, international patent application WO2022 / 264472A1 discloses an optical tactile sensor with a wear-resistant surface layer. The back of the first layer is configured with a two-dimensional pattern (such as a grid or dot matrix printed on a black substrate) to detect deformation. While this approach improves the sensor's durability through its layered structure, the two-dimensional pattern used is typically designed with a light-absorbing material (such as black) that has high contrast with the background. Although this design is beneficial for two-dimensional tracking of feature points, for photometric stereo algorithms that rely on light reflection, these light-absorbing patterns cannot generate diffuse reflection intensity that varies with the normal direction, resulting in the loss of local normal information.

[0007] For example, Chinese patent application CN1842701A describes an optical tactile sensor for measuring large-area force vector distribution. It utilizes multiple colored markers (such as red and blue spherical markers) within a transparent elastic body, capturing the movement of these markers using multiple photographic devices to calculate the force vector. However, this approach is essentially a measurement method based on discrete feature points. The sensor can only acquire displacement data at sparsely distributed marker locations, lacking direct measurement information over the vast areas between the markers. More importantly, this approach does not address the reconstruction of a continuous surface using the photometric variations of the markers themselves.

[0008] However, existing technologies have not taken into account the "blind spot" problem caused by the introduction of markers, such as Figure 17 As shown, Figure 17 This is an example of images captured by a typical visual-tactile sensor in existing technology under different force conditions. Figure 17 In the image, the background area exhibits rich textures or color gradients that change with surface deformation. However, the markers distributed within it (the black dots shown in the image) consistently appear as uniform color blocks with low albedo. This is because existing technologies typically design the markers as light-absorbing materials with extremely low reflectivity (such as black light-absorbing dots) to enhance the contrast for position identification in order to achieve tangential force calculation.

[0009] This design ignores the destructive impact of the markers' optical properties on photometric stereo processing. Because these markers cannot follow the Lambertian diffuse reflection model, they cannot generate continuously varying brightness or color gradients with local normal tilt. Therefore, during normal force calculation, the system is forced to discard these marked areas as interfering data or invalid pixels. This directly results in dense processing blind spots (such as...) on the sensor surface. Figure 17As shown in the discrete black dotted areas, the reconstructed 3D topography exhibits breaks or voids at the marked points. This not only disrupts the spatial continuity between the normal and tangential fields, limiting the sensor's spatial resolution, but also poses a risk of complete loss of normal force information in scenarios with small contact areas (especially when the contact object falls entirely on the marked point). Summary of the Invention

[0011] The purpose of this invention is to at least partially solve one of the technical problems existing in the prior art.

[0012] To achieve the above-mentioned objectives and other advantages according to the present invention, a visual-tactile sensing method is provided according to a first aspect of the present invention, applied in a visual-tactile sensor provided with a soft elastomer, characterized by comprising the following steps: A light source is configured such that the light source shines through a soft elastic body onto a reflective medium with non-directional or partial non-directional reflective properties, which is placed on the side of the soft elastic body that is in contact with the object. The reflective medium includes a first reflective medium that reflects the internal environment of the soft elastic body and a second reflective medium for marking. The first and second reflective media have different albedoes. The second reflective medium has non-directional or partial non-directional reflective properties that participate in photometric stereoscopic calculation. The system acquires visual images of the changes in local position and normal of the reflective medium caused by the contact of a soft elastic body with an object; wherein the visual images include light intensity distribution information of the first and second reflective media as a function of the normal under illumination by a light source. Photometric stereoscopic calculation is performed on visually changing images, and the mechanical characteristic data of the soft elastic body in its current state are perceived based on the light intensity distribution information of the first and second reflecting media.

[0013] Optionally, the mechanical characteristic data includes the normal displacement and corresponding normal force of the soft elastic body, and the tangential displacement and corresponding tangential force; the step of "performing photometric stereoscopic calculation on the visual change image and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media" specifically includes the following steps: Based on the principle of photometric stereo, the three-dimensional contour features of the soft elastic body on the side in contact with the object are calculated by utilizing the light intensity distribution information of the first and second reflective media in the visual change image. Extract the displacement and / or deformation features of the second reflective medium from the visual change image; By combining the three-dimensional contour features with the displacement features and / or deformation features, the normal displacement, normal force, tangential displacement, and tangential force of the soft elastic body in its current state are calculated.

[0014] Optionally, the step of "calculating the three-dimensional contour features of the side of the soft elastic body in contact with the object based on the photometric stereo principle and utilizing the light intensity distribution information of the first and second reflecting media in the visual change image" further includes the following steps: Extract the difference in reflectivity between the first and second reflective media in the visually changing image; The interference of the albedo difference between the first and second reflective media on the photometric stereo solution is eliminated based on the difference in reflectivity characteristics.

[0015] Optionally, the step of "performing photometric stereoscopic calculation on the visually changing image and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media" specifically includes the following steps: When a soft elastomer is not in contact with an object, a non-contact reference image is obtained by reflection from the first and second reflective media; Based on non-contact reference images and visual change images, planar displacement information of the first and second reflective media is extracted to establish a global displacement field model of the soft elastic body. Photometric stereoscopic calculations are performed based on a displacement information model to perceive the mechanical characteristic data of the soft elastic body under its current state. The mechanical characteristic data of the soft elastic body includes the global sum of the normal displacement, normal force, tangential displacement, tangential force, and torque of the soft elastic body. The tangential displacement and tangential force are determined based on the planar displacement information of the second reflecting medium, and the torque is determined based on the divergence-free components in the global displacement field model.

[0016] Optionally, the step of "performing photometric stereoscopic calculation on the visually changing image and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media" specifically includes the following steps: A non-contact reference image is obtained by the light source being reflected by the first and second reflective media when the soft elastomer is not in contact with the object. Based on photometric stereo calculation, local normal variation features of the soft elastic body on the side in contact with the object are extracted from the non-contact reference image and the visual change image; Extract the difference data of position and reflection characteristics of the first and second reflective media in the non-contact reference image and the visual change image; By integrating local normal variation features with differences in location and reflection characteristics, comparison information between non-contact reference images and visually altered images is generated. The comparison information is input into the pre-trained computational neural network model to output the mechanical characteristic data of the soft elastic body; wherein, the mechanical characteristic data of the soft elastic body includes the distributed normal displacement, normal force, tangential displacement and tangential force of the soft elastic body.

[0017] Optionally, the step of "acquiring visual images of the changes in local position and normal caused by the reflective medium contacting the object of the soft elastic body" specifically includes the following steps: Continuously acquire images of reflections from the reflective medium to obtain a multi-frame image sequence including at least two sets of illumination angles; wherein, the independent light source illuminates the reflective medium using at least one of time-division illumination, illumination with different wavelengths, or illumination with different polarizations. A multi-frame image sequence is processed in a time-series synchronization manner to obtain a visual change image; wherein, the visual change image is obtained by separating the illumination data corresponding to different illumination directions based on the difference in acquisition time, the difference in light wavelength, or the difference in light polarization.

[0018] According to a second aspect of the present invention, a visual-tactile sensor is provided, characterized in that it comprises: A soft elastic body that deforms in response to external pressure, with a reflective medium having non-directional or partial non-directional reflective properties on the side of the soft elastic body that contacts the object; wherein, the reflective medium includes a first reflective medium that reflects the internal environment of the soft elastic body and a second reflective medium for marking; the first reflective medium and the second reflective medium have different albedoes, and the second reflective medium has non-directional or partial non-directional reflective properties that participate in photometric stereoscopic calculation. A light source for illuminating soft elastomers and providing the illumination required for photometric stereoscopic calculations; A photosensitive structure capable of receiving light rays after illumination by a light source, reflecting the changes in the local normal of the reflective medium, and generating a visual change image; wherein, the visual change image includes light intensity distribution information of the first and second reflective media, and the visual change image is used for photometric stereoscopic calculation to obtain mechanical characteristic data of the soft elastic body.

[0019] Optionally, the light source includes at least two sets of independent light sources, each set of independent light sources having a different illumination direction, which can illuminate the reflective medium from different directions.

[0020] Optionally, the light source includes at least two independent light sources, wherein the emission wavelengths of the at least two independent light sources are different.

[0021] Optionally, the first reflective medium is a reflective layer covering the surface of the soft elastomer on the force-bearing side; the second reflective medium is a block-shaped marker of the soft elastomer distributed on the reflective layer and / or on the side close to the reflective layer.

[0022] Optionally, the surface of the soft elastomer on the force-bearing side is covered with a light-shielding layer, which is used to block external ambient light from entering the photosensitive unit; the second reflective medium is a block-shaped mark of the soft elastomer distributed on the light-shielding layer and / or on one side close to the light-shielding layer.

[0023] Optionally, the first reflecting medium is a scattering particle layer disposed on the soft elastomer, the scattering particle layer being composed of one or more aggregated scattering particles.

[0024] Optionally, the second reflective medium is a block marker, and the block marker has different albedoes depending on its location in the distribution area of ​​the soft elastomer.

[0025] Compared to existing technologies, this application eliminates the blind spot in normal force calculation, achieving continuous perception across the entire field of view. Existing technologies, to achieve tangential displacement tracking, typically design markers as light-absorbing materials with extremely low reflectivity or possess specular reflective properties. This design severely disrupts the diffuse reflection model upon which photometric stereo methods rely, causing the marked area to fail to generate brightness gradients that vary with the normal direction. The system is forced to treat this as invalid data and discard it, resulting in a dense calculation blind spot on the sensor surface. In contrast, this application configures a second reflective medium with non-directional reflective properties that participate in photometric stereo calculation. This allows the marked area to no longer be an optical black hole under illumination, but rather to generate brightness distribution information that continuously varies with local normals, much like the background. The algorithm does not need to discard the marked area but incorporates it into a unified model for direct calculation, thereby obtaining a continuous and complete surface normal distribution across the entire field of view, including the marked area, significantly improving the spatial resolution and completeness of 3D topography reconstruction. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the embodiments of the present invention are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] In the attached diagram: Figure 1 A flowchart illustrating the steps of a visual-tactile perception method applied to a visual-tactile sensor provided in an embodiment of this application; Figure 2 A flowchart illustrating the steps of "acquiring visual images of changes in the local position and normal caused by the contact of a reflective medium with an object in the presence of a soft elastomer" provided in this application embodiment; Figure 3 The flowchart provided in this application embodiment describes the steps of "performing photometric stereoscopic calculation on a visually changing image and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media". Figure 1 ; Figure 4The flowchart provided in this application embodiment describes the steps of "performing photometric stereoscopic calculation on a visually changing image and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media". Figure 2 ; Figure 5 The flowchart provided in this application embodiment describes the steps of "performing photometric stereoscopic calculation on a visually changing image and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media". Figure 3 ; Figure 6 This is a schematic diagram of the structure of a visual-tactile sensor provided in an embodiment of this application; Figure 7 A three-dimensional structural schematic diagram of the visual-tactile sensor provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the visual-tactile sensor provided in the embodiments of this application after being subjected to force deformation. Figure 1 ; Figure 9 This is a schematic diagram of the structure of the visual-tactile sensor reflective layer and the block-shaped mark provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of the visual-tactile sensor provided in the embodiments of this application after being subjected to force deformation. Figure 2 ; Figure 11 This is a schematic diagram of the structure of a visual-tactile sensor with a light-shielding layer provided in an embodiment of this application; Figure 12 This is a schematic diagram of the structure of the visual-tactile sensor provided in the embodiments of this application after being subjected to force deformation. Figure 3 ; Figure 13 Schematic diagram of visual-tactile sensor structure with different albedo marker distributions provided in embodiments of this application Figure 1 ; Figure 14 Schematic diagram of visual-tactile sensor structure with different albedo marker distributions provided in embodiments of this application Figure 2 ; Figure 15 A schematic diagram of the finite element simulation displacement field provided in the embodiments of this application; Figure 16 This is a schematic diagram of the tangential force of the visual-tactile sensor provided in the embodiments of this application; Figure 17 The image is captured by a visual-touch sensor in the prior art. Figure 18 An image acquired by a visual-tactile sensor provided in an embodiment of this application. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. It should be noted that although functional modules are divided in the system schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system schematic diagram or the order in the flowchart. The step numbers in the following embodiments are set only for ease of explanation and do not limit the order between steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art. Without conflict, the embodiments described below or the technical features can be arbitrarily combined to form new embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] It should be noted that the use of terms such as "first," "second," etc., in the specification, claims, and drawings of this application is solely for the purpose of distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the specification of this invention is for the purpose of describing specific embodiments only and is not intended to limit the invention.

[0031] The visual-tactile perception method for visual-tactile sensors provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, set-top box, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the visual-tactile perception method for visual-tactile sensors, etc., but is not limited to the above forms.

[0032] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0033] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards of the relevant countries and regions. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirects to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data for the proper functioning of the embodiments of this application obtained.

[0034] In the field of visual-tactile sensing, photometric stereo methods have attracted much attention due to their ability to reconstruct high-resolution three-dimensional shapes. However, existing technologies have significant limitations when integrating photometric stereo with marker tracking techniques. For example... Figure 17As shown, photometric stereo methods rely on surfaces following a Lambertian reflection model or exhibiting homogeneous diffuse reflection characteristics. However, the markers introduced to achieve tangential force calculation are typically designed with extremely low reflectivity or specular reflectivity. This significant difference in reflectivity between the markers and the background disrupts the continuity of the photometric stereo model, resulting in ineffective shadow variations or illumination saturation in the marked areas. Therefore, existing systems are forced to treat the marked areas as interference or occlusion, estimating normal topography only in unmarked areas, thus creating dense solution blind zones on the sensor surface, such as... Figure 17 As shown by the discrete dark dotted regions, these areas have lost normal information. This not only disrupts the spatial continuity between the normal and tangential fields, limiting spatial resolution, but also poses a risk of complete loss of normal information in scenarios with small contact areas.

[0035] The purpose of this invention is to provide a blind-spot-free, three-dimensional force-calcifiable visual-tactile sensing solution. For example... Figure 18 As shown, by ensuring that both the marker (the second reflecting medium) and the background reflecting medium (the first reflecting medium) possess strong reflectivity, and that the local normal variations of both can be uniformly analyzed by the global algorithm, the following method is implemented. Figure 18 As can be seen, the marked area and the background area exhibit continuous changes in light and shadow gradients, allowing both the marked and background areas to participate in the normal calculation, eliminating the calculation blind spot, and achieving blind spot-free, high-resolution calculation of multi-dimensional mechanical characteristics such as normal force, tangential force, and torque, which is suitable for high-precision tactile sensing application scenarios.

[0036] like Figure 1 The diagram shows a flowchart of a visual-tactile perception method applied to a visual-tactile sensor with a soft elastomer provided in an embodiment of the present invention. (Refer to...) Figure 1 This invention provides a visual-tactile perception method applied to a visual-tactile sensor, specifically including the following steps: S12. Configure a light source so that the light source shines through the soft elastic body onto a reflective medium with non-directional or partial non-directional reflective properties, which is placed on the side of the soft elastic body that is in contact with the object; wherein, the reflective medium includes a first reflective medium that reflects the internal environment of the soft elastic body and a second reflective medium for marking; the first reflective medium and the second reflective medium have different albedoes; and the second reflective medium has non-directional or partial non-directional reflective properties that participate in photometric stereoscopic calculation. In existing technologies, to detect the tangential deformation of soft elastomers, it is usually necessary to place markers on their surface for position tracking. Traditional markers are typically designed as light absorbers with extremely low reflectivity, such as carbon black dots, or reflective dots with specular reflection properties. Because traditional markers lack diffuse reflection properties or have excessively low albedo, they cannot produce a diffuse reflection intensity distribution that continuously varies with the surface normal direction under illumination. As a result, during photometric stereo calculation, the local area where the marker is located cannot satisfy the assumptions of the Lambertian reflection model or the approximate Lambertian model, and cannot provide effective normal constraint information. This leads to a blind zone or failure area for normal force calculation on the surface of the soft elastomer.

[0037] To address the aforementioned technical issues, this embodiment configures the first and second reflective media to have different albedoes to achieve region segmentation and recognition in visual images. Simultaneously, the second reflective media is configured to possess non-directional reflective properties, i.e., diffuse reflective properties, or partially non-directional reflective properties, that participate in photometric stereoscopic calculation. This means that the second reflective media is configured to generate a diffuse reflective response to incident light. When the soft elastic body is deformed under force, the intensity of reflected light on the surface of the second reflective media changes regularly with the change in the angle of its local normal relative to the light source, thereby providing effective photometric stereoscopic calculation information.

[0038] As an optional implementation method, this step can be based on, for example... Figure 6 The visual-tactile sensor structure shown is configured, but this embodiment is not limited to a specific sensor configuration.

[0039] by Figure 6 Taking the structure shown as an example, firstly, a soft elastomer 21 is configured as both a light transmission channel and a force sensing carrier. Its material is a flexible material with optical transmittance, such as PDMS, to allow light from the light source 22 to pass through. Secondly, the light source 22 is configured as follows: Figure 6 As shown, at least two independent light sources are arranged on the non-contact side of the soft elastomer 21, and the illumination angle of the light source 22 is adjusted, for example, distributed along the circumference, so that it can illuminate the reflective medium 26 on the top of the soft elastomer 21 from different directions, thereby forming a differentiated illumination field.

[0040] The key to this method lies in the configuration of the reflective medium 26. For example... Figure 7 An example of a reflective medium distribution is shown, wherein the reflective medium 26 comprises a continuously distributed first reflective medium 261 and a discretely distributed second reflective medium 262. Optically, the first reflective medium 261 and the second reflective medium 262 have different albedoes.

[0041] In terms of spatial distribution, the reflective medium 26 can be flexibly configured according to application requirements. One optional distribution method is as follows: Figure 6 or Figure 7 As shown, the second reflective medium 262 is disposed on the surface of the first reflective medium 261 or coplanar with it. Another optional distribution method is as follows: Figure 9 As shown, to improve durability, the second reflective medium 262 is configured as block-shaped markings distributed inside the soft elastomer 21, located on the side close to the first reflective medium 261 (e.g., the surface reflective layer). In this configuration, light from the light source 22 passes through the transparent soft elastomer 21 and illuminates the internal second reflective medium 262, thus stimulating its diffuse reflection characteristics.

[0042] S14. Acquire visual images of the changes in local position and normal of the reflective medium caused by the contact of the soft elastic body with the object; wherein, the visual change images include light intensity distribution information of the first and second reflective media as the normal changes under the illumination of the light source. like Figure 2 The diagram shown is a flowchart of step S14 provided in an embodiment of the present invention. (Refer to...) Figure 2 As an optional implementation, step S14, "acquiring visual images of the changes in local position and normal caused by the reflective medium contacting the object with the soft elastic body," specifically includes the following steps: S141. Continuously acquire images reflected by the reflective medium to obtain a multi-frame image sequence including at least two sets of illumination angles; wherein, the independent light source illuminates the reflective medium in at least one of time-division illumination, different wavelength illumination, or different polarization illumination. S142. Perform time-series synchronization processing on the multi-frame image sequence to obtain visual change images; wherein, the visual change images are obtained by separating the illumination data corresponding to different illumination directions based on the differences in acquisition time, light wavelength, or light polarization.

[0043] To obtain the multi-directional illumination data required for photometric stereo rendering, the above steps can be implemented using a multi-channel illumination acquisition method: In one embodiment, a time-sharing illumination method can be adopted, that is, at least two independent light sources with different illumination directions are controlled to work in a time-sharing illumination manner, and the photosensitive structure synchronously acquires multiple frame image sequences. Each frame image corresponds to the illumination response data under an independent illumination direction. Then, the multiple frame image sequences are processed in a time-series synchronization manner to obtain visual change images.

[0044] In another embodiment, different wavelengths of illumination, i.e., heterochromatic illumination, can be used. At least two independent light sources with different illumination directions are controlled to work in different wavelength illumination modes, such as emitting red light, green light and blue light respectively. The photosensitive structure, such as a color image sensor, is used to collect images containing multispectral information, and the illumination data corresponding to different illumination directions is separated based on the differences in light wavelengths, such as RGB color channels.

[0045] In another embodiment, different polarization illumination methods can be used, that is, controlling light sources in different directions to work with different polarization illumination methods, and using the difference in light polarization to separate the illumination data corresponding to different illumination directions.

[0046] Step S14 aims to acquire optical data reflecting the mechanical state of the contact interface. Specifically, it involves acquiring visual images of the changes in the reflective medium's local position shift and normal caused by the contact of the soft elastomer with the object.

[0047] To more clearly illustrate the technical effects of this application, first refer to... Figure 17 Explain the limitations of existing technology. Figure 17 This demonstrates an image captured by a typical visual-tactile sensor in the prior art when subjected to force.

[0048] like Figure 17 As shown, the background area of ​​the sensor exhibits rich, colorful light and shadow textures that change with surface deformation; however, the black markers distributed within it consistently appear as uniform color blocks with low albedo, lacking the light and shadow details to reflect changes in surface normals. This is because existing technologies typically design the markers as light-absorbing materials with extremely low reflectivity to enhance the contrast for position recognition in order to achieve tangential force calculation. In the photometric stereo calculation model, these black marker areas cause interruptions in reflected light intensity data, preventing the algorithm from calculating the normal vectors at these points. Therefore, Figure 17 Each black dot in the graph actually constitutes a "blind spot" or "data hole" in the calculation of normal force, causing the reconstructed 3D shape to break at the marked point.

[0049] To address the aforementioned issues, embodiments of this application employ a specific configuration for the reflective medium. For example... Figure 8 The diagram illustrates the cross-sectional deformation of the sensor presented in this application under normal pressure from an external object. When the soft elastic body is compressed and indented, the reflective medium adhering to its surface deforms accordingly, causing a local deflection of the normal direction of the contact area relative to the initial state. Figure 17 Unlike other embodiments, the second reflective medium (marker) in this embodiment is configured to have non-directional reflection characteristics (i.e., diffuse reflection characteristics). Therefore, under illumination by a light source, Figure 8 The marker within the stress region is not an optical black hole; the intensity of light reflected from its surface changes regularly with the deflection of the local normal. This means that the marker region, like the background region, can provide an effective photometric stereo constraint equation, thus filling the gap. Figure 17 The solution blind zone is shown.

[0050] Based on the above mechanism, when the sensor is subjected to complex combined forces, the visual change image acquired by the photosensitive structure is as follows: Figure 18 As shown. Figure 18This intuitively demonstrates how this application simultaneously characterizes multidimensional mechanical features using a single frame image: on the one hand, Figure 18 The background and logo areas (dark dots) both display continuous and rich colored light and shadow. Figure 17 In stark contrast, Figure 18 The marker points within the model also retain light and shadow gradient information, without any interruptions in uniform color blocks. The algorithm can reconstruct the complete surface height field based on the light and shadow information across the entire field of view, and thus solve for the algorithmic force. On the other hand, in comparison... Figure 16 (Displacement vector diagram) Compared to a regular grid under no-stress conditions, planar displacement of the marker points and array distortion can be clearly observed. For example... Figure 16 As shown by the red arrow, the arrow visually represents the displacement vector field of the marker point after it is subjected to force.

[0051] This embodiment, by configuring a second reflective medium with non-directional reflective properties, enables the marked area to provide effective normal constraint information in photometric stereo calculation, thus filling the gap. Figure 17 The marker points create a blind spot for normal sensing; however, they retain the function of representing planar displacement and horizontal torsion. This configuration enables simultaneous sensing of normal deformation, tangential deformation, and torsional deformation in a single measurement.

[0052] Specifically, it involves capturing visual images of the changes in the local normal of a reflective medium caused by the contact of a soft, elastic body with an object. For example... Figure 8 and Figure 10 The diagram illustrates the typical state of the sensor under stress. When an external object 30 contacts and presses the soft elastomer 21, the soft elastomer 21 undergoes mechanical deformation, causing the contact area to indent.

[0053] If such as Figure 6 The surface distribution structure shown is subjected to the following stress state: Figure 8 As shown, the reflective medium 26 (including the first reflective medium 261 and the second reflective medium 262) attached to the surface undergoes normal deflection as the surface is recessed.

[0054] If such as Figure 9 The embedded structure shown is subjected to the following stress states: Figure 10 As shown, the first reflective medium 261 on the surface undergoes a local change in normal under pressure, while the second reflective medium 262 embedded inside the soft elastic body 21 shifts in displacement and orientation with the deformation of the substrate. Figure 10 As shown in the contact area, although the second reflective medium 262 does not directly contact the object 30, its position and surface orientation have changed relative to the initial state.

[0055] In this state, the light source 22 illuminates the deformed reflective medium 26 from different angles. Since the second reflective medium 262 is configured to have diffuse reflection characteristics, in... Figure 8 or Figure 10 Within the stress-bearing area shown, the reflected light intensity on the marker surface changes regularly with changes in the local normal or orientation. The photosensitive structure 23 receives the reflected light and generates a visual change image. This image simultaneously contains information about the light intensity distribution of the first reflective medium 261 and the second reflective medium 262 under illumination, varying with the normal.

[0056] S16. Perform photometric stereoscopic calculation on the visually changing image, and perceive the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media. Since the second reflecting media provides effective photometric response information, the algorithm no longer considers it as an interference region or invalid region and removes it. Specifically, based on the light intensity distribution information of the first and second reflecting media, combined with the pre-calibrated albedo difference, and using a unified illumination reflection model, such as the Lambert model, calculate the full field of view of the soft elastic body, including the local normal distribution of marked and non-marked regions, and then perceive mechanical characteristic data such as normal displacement and normal force.

[0057] Step S16 uses the acquired optical information to infer the mechanical characteristics. Photometric stereoscopic calculation is performed on the visually changing image. The algorithm no longer removes the region where the second reflecting medium 262 is located, but instead uses the light intensity distribution information of the entire field of view of the first reflecting medium 261 and the second reflecting medium 262 to jointly calculate the mechanical characteristic data of the soft elastic body in its current state. Specifically, using... Figure 8 or Figure 10 The gradient of light and dark variations shown in the figure is used to reconstruct the continuous surface normal field of the contact area, and then the normal displacement and normal force are obtained by integration; at the same time, using Figure 7 or Figure 9 The planar displacement of the discrete marker (i.e., the second reflecting medium 262) is used to calculate the tangential force. Using the above methodological framework, combined with… Figures 6 to 10 As shown in the hardware configuration and physical state, this embodiment achieves high-precision three-dimensional force sensing of the soft elastomer contact interface without blind spots.

[0058] This embodiment eliminates the damage to the photometric stereo model caused by the marker points through the above steps, and realizes continuous normal sensing of the soft elastomer contact area without blind spots.

[0059] In an optional embodiment, the mechanical characteristic data includes the normal displacement and corresponding normal force of the soft elastic body, and the tangential displacement and corresponding tangential force. For example... Figure 3As shown, step S16, which involves performing photometric stereoscopic calculation on a visually changing image and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media, specifically includes the following steps: S1613. Based on the principle of photometric stereo, the three-dimensional contour features of the soft elastic body on the side in contact with the object are calculated by utilizing the light intensity distribution information of the first and second reflecting media in the visually changing image.

[0060] Specifically, this step utilizes a photometric stereo algorithm to reconstruct the surface morphology. Based on the multi-channel illumination data acquired in the previous step, the visually changing image is substituted into an illumination reflection model, such as the Lambertian reflection model. By combining the physical relationship between reflected light intensity and the angle between the reflected light and the surface normal, multiple sets of reflected light intensity data under different illumination angles are used to construct a system of equations and calculate the local normal distribution across the entire field of view of the soft elastomer contact area. Subsequently, by performing gradient integration on the local normal distribution, the depth information or height field of the soft elastomer surface is reconstructed, thus obtaining the three-dimensional contour feature. Since the second reflective medium region also participates in the solution, this three-dimensional contour feature characterizes the macroscopic deformation of the soft elastomer surface.

[0061] S1614. Extract the displacement and / or deformation features of the second reflecting medium from the visual change image.

[0062] Specifically, this step extracts the deformation information of the second reflective medium using a feature matching algorithm. The photometric stereoscopic image is primarily sensitive to normal deformation, while the second reflective medium, acting as a marker, reflects the tangential flow and local compression state of the material through its planar movement and local distortion. Using optical flow or feature point matching algorithms, the planar position change of the second reflective medium relative to a non-contact reference image in the visually changing image is tracked, generating displacement features composed of multiple discrete displacement vectors or continuous interpolation fields. This feature supplements the internal compression and shear information of the material not described by a simple three-dimensional contour.

[0063] S1615. By integrating the three-dimensional contour features, displacement features, and / or deformation features, calculate the normal displacement, normal force, tangential displacement, and tangential force of the soft elastic body under the current state.

[0064] Specifically, this step involves the comprehensive processing of multi-dimensional features based on the physical properties of the soft elastic body. Since the soft elastic body is volumetrically incompressible under stress, its normal and tangential deformations are physically coupled. Therefore, during the solution process, the displacement features extracted in step S1614 can be optionally used to constrain and correct the integral boundary conditions of the three-dimensional contour features in step S1613 to reduce depth calculation errors; alternatively, depth information from the three-dimensional contour features can be used to correct the perspective projection errors of the displacement features. Finally, combining the material properties of the soft elastic body, such as the elastic modulus and shear modulus, the comprehensively corrected geometric features are converted into distributed normal and tangential force data through constitutive equations or pre-calibrated mapping relationships.

[0065] It should be noted that the division of steps S1613 to S1615 in this embodiment is only a description of logical functions. In practical applications, the above feature extraction and comprehensive solution process can also be implicitly completed by an end-to-end neural network model that inputs multi-channel data.

[0066] Furthermore, to address the photometric stereoscopic calculation error caused by the difference in albedo between the first and second reflecting media, image preprocessing is required. In an optional embodiment, such as... Figure 3 As shown, the step of "calculating the three-dimensional contour features of the side of the soft elastic body in contact with the object based on the photometric stereo principle and utilizing the light intensity distribution information of the first and second reflective media in the visual change image" also includes the following steps: S1611. Extract the difference in reflection characteristics between the first and second reflecting media in the visually changing image.

[0067] Specifically, the process of extracting the reflectance characteristic difference features can be achieved through grayscale statistical analysis of visually changing images. The diffuse reflection background region of the first reflective medium and the marker region of the second reflective medium are selected in the image, and the local grayscale mean values ​​of the two types of regions under the same illumination conditions are calculated. Subsequently, the difference between the grayscale mean values ​​of the first and second reflective media, or the ratio of the grayscale mean values ​​of the first and second reflective media, is used as the reflectance characteristic difference feature. This feature quantifies the difference in optical response between the two types of media.

[0068] S1612. Eliminate the interference of the albedo difference between the first and second reflective media on the photometric stereo solution based on the difference in reflectivity characteristics.

[0069] Specifically, the interference elimination process can be implemented using a brightness compensation algorithm or a normalization algorithm. Based on the reflection characteristic difference features extracted in step S1611, a brightness compensation coefficient is calculated. For example, this compensation coefficient can be equal to the average gray value of the first reflecting medium divided by the average gray value of the second reflecting medium. This compensation coefficient is then used to perform weighted compensation on the pixel brightness of the marker area of ​​the second reflecting medium in the visually altered image. After this processing, the reflected light intensity response of the marker area and the background area is unified to the same albedo benchmark, conforming to the unified reflection model assumption of photometric stereoscopic solution, thereby avoiding the algorithm from mistakenly identifying inherent differences in albedo as drastic changes in surface normals.

[0070] In an optional embodiment, for the need to detect the global stress state of the soft elastomer 21 (such as the overall force distribution assessment when a large object is pressed), step S16 may also use photometric stereoscopic calculation by comparing with a reference image and global modeling to perceive the mechanical characteristic data of the soft elastomer in the current state.

[0071] like Figure 4 As shown, the steps of "performing photometric stereoscopic calculation on visually changing images and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media" specifically include the following steps: S1621. Obtain a non-contact reference image obtained by reflection from the first and second reflective media when the soft elastic body is not in contact with the object. S1622. Based on the non-contact reference image and the visual change image, extract the planar displacement information of the first and second reflective media to establish a global displacement field model of the soft elastic body. S1623. Photometric stereoscopic calculation is performed based on the displacement information model to perceive the mechanical characteristic data of the soft elastomer in its current state. The mechanical characteristic data of the soft elastomer includes the global sum of normal displacement, normal force, tangential displacement, tangential force, and torque. The tangential displacement and tangential force are determined based on the planar displacement information of the second reflecting medium, and the torque is determined based on the divergence-free components in the global displacement field model. Specifically, this embodiment covers the technical framework of "establishing a global displacement field and calculating the sum of mechanical data based on a non-contact reference diagram." Its function is to quantify the global stress state of the soft elastomer 21, adapting to scenarios requiring overall force control, such as large workpiece assembly and flexible fixture clamping. The scope of protection of this embodiment is not limited to specific modeling or calculation methods; as long as a global model can be established based on a non-contact reference diagram, combined with the displacement information of the two types of reflecting media, and the sum of mechanical data can be calculated, it falls within the scope of this embodiment.

[0072] For step S1621, its execution must be under the same acquisition conditions as the visual change image (e.g., Figure 6(The illumination angle and intensity of the two light sources are consistent, and the resolution and frame rate of the image acquisition device are consistent) to obtain a non-contact reference image of the soft elastic body when it is not in contact with the object, so as to ensure the consistency of the benchmark for subsequent displacement calculation.

[0073] For step S1622, its execution first uses image processing algorithms (such as pixel-level registration, optical flow, or feature matching) to extract the planar displacement information of the first reflecting medium (background) and the second reflecting medium (marker) in the non-contact reference image and the visual change image, respectively. Since the marker points are usually discretely distributed, this step further utilizes interpolation algorithms (such as bilinear interpolation or radial basis function interpolation) to complete the data of the unmarked areas of the soft elastic body based on the discrete displacement information, thereby establishing a continuous global displacement field model covering the entire surface of the soft elastic body. This model intuitively describes the motion trend of each point on the surface of the soft elastic body in the plane.

[0074] The core of step S1623 lies in decoupling the displacement field using physical field theory. Specifically, the global displacement field model is decomposed using Helmholtz decomposition, resulting in irrotational components, divergence-free components, and harmonic components. Tangential displacement and tangential force are determined based on the planar displacement information of the second reflecting medium (i.e., the overall translational modulus or harmonic component of the displacement field) combined with the material shear modulus. Torque is determined based on the divergence-free component (i.e., the curl field) in the global displacement field model. This divergence-free component can isolate the pure rotational effect from complex mixed deformations, thus achieving independent perception of torque. Normal displacement and normal force can be calculated based on the decomposed irrotational component (divergence field) or combined with the normal gradient information from photometric stereoscopic solutions, using pre-calibrated characteristic quantities and physical force mapping relationships. Through the above steps, the final output is complete mechanical characteristic data containing the global sum of normal displacement, normal force, tangential displacement, tangential force, and torque of the soft elastic body.

[0075] In an optional embodiment, steps S1621-S1623 in step S16, "performing photometric stereoscopic calculation on the visual change image to perceive the mechanical characteristic data of the soft elastomer in its current state," can specifically employ the following physical modeling-based summative force calculation method to calculate the summative normal force, tangential force, and torque acting on the entire sensor surface.

[0076] Specifically, for step S1621, under the condition that the soft elastomer is not in contact with any object (unloaded), images of the first and second reflective media on its surface are captured using an image acquisition device, serving as an initial reference frame, i.e., a non-contact reference image. In this non-contact reference image, the initial position of each marker point on the second reflective medium is recorded. Specifically, this can be achieved using the formula... express.

[0077] in, This represents the initial position of the i-th marker point on the second reflecting medium. Let x be the x-coordinate of the i-th marker point in the image coordinate system; Let be the ordinate of the i-th marker in the image coordinate system; `i` is the index of the marker, i = 1,2, ..., N, where N is the total number of markers on the second reflecting medium.

[0078] Specifically, visual changes are captured when the soft elastomer comes into contact with an object.

[0079] Step S1622 above may specifically include the following steps: S16221. Calculate the displacement vector of the marker point: Using image processing algorithms (such as feature matching), find each marker point in the visually changing image that corresponds to the non-contact reference image, and record its current position. Calculate the displacement vector of each marker point from its initial position to its current position: In the above formula, Let i be the displacement vector of the i-th marker point; Let be the displacement component of the i-th marker point in the x-direction; Let be the displacement component of the i-th marker point in the y-direction; and and are the x and y coordinates of the i-th marker point in the visual change image, respectively.

[0080] S16222. Establish the global displacement field; since the marker points are discretely distributed, the continuous displacement field of the entire plane of the soft elastic body surface is obtained through interpolation calculations (such as bilinear interpolation, radial basis function interpolation, etc.). The displacement field It is a vector field that describes the displacement of every point on the surface.

[0081] S16223, Helmholtz decomposition: The obtained global displacement field vector... Perform Helmholtz decomposition, which decomposes the field into a superposition of irrotational, divergence-free, and harmonic fields: In the above formula, Let be the total displacement vector at a point on the surface of the soft elastic body; ∇D is the irrotational field component, whose displacement is generated by the normal force; J is the rotation operator (in a two-dimensional plane, J is usually represented as a rotation perpendicular to the plane, and its matrix form is a second-order matrix with 0 and -1 in the first row and 1 and 0 in the second row); ∇R is the gradient of the scalar function R; J∇R is the divergence-free field component, whose displacement is generated by the rotational torque; To harmonize the field components, which contribute relatively little to force and torque, the harmonizing field components may optionally not be added to the formula. .

[0082] Specifically, for step S1623 above, based on the results of Helmholtz decomposition, characteristic quantities related to normal force, tangential force and torque are calculated respectively, and then the actual mechanical quantities are solved through pre-calibrated mapping relationships.

[0083] Among them, normal characteristic quantity The formula is ,in, It is a normal characteristic quantity used to characterize the sum of displacement effects caused by normal force; Let be the total displacement vector of the i-th marker point on the second reflecting medium; The irrotational field component obtained from Helmholtz decomposition corresponds to the displacement component generated by the normal force. The component belonging to the irrotational field ∇D (i.e., the normal displacement component) in the total displacement vector of the i-th marker point; i is the index of the marker point, with a value range of 1, 2, ..., N, where N is the total number of marker points on the second reflecting medium.

[0084] Tangential features The formula is: ,in, It is a tangential characteristic quantity used to characterize the sum of displacement effects caused by tangential force; For the second reflecting medium on the first The total displacement vector of the marked points; This represents the global displacement field on the surface of a soft elastic body. This indicates that the total displacement vector of the i-th marker point belongs to the global displacement field. The tangential component (i.e., the displacement component parallel to the surface of the soft elastic body); i is the index of the marker point, with a value range of 1, 2, ..., N, where N is the total number of marker points on the second reflecting medium.

[0085] Torque characteristic quantity The formula is: ,in, This is a torque characteristic quantity, used to characterize the sum of rotational effects caused by torque; Let be the position vector pointing from the j-th reference point to the i-th marker point; j is the index of the reference point, ranging from 1, 2, ..., M, where M is the number of selected reference points (usually 1, i.e., the geometric center point of the soft elastic body); i is the index of the marker point, ranging from 1, 2, ..., N, where N is the total number of marker points on the second reflecting medium. Let i be the total displacement vector of the i-th marker point; For rotation operators (in matrix form, it is a second-order matrix with the first row being 0 and -1, and the second row being 1 and 0); scalar function The gradient; The divergence-free components obtained from Helmholtz decomposition correspond to the displacement components generated by the rotational torque. This indicates that the total displacement vector of the i-th marker point belongs to the divergence-free field. The component (i.e., the rotational displacement component).

[0086] Finally, establish the normal eigenvalues ​​of the eigenvalues. Tangential features Torque characteristic quantity With respect to actual physical force and normal force Tangential force Torque Mapping function: , , ; In the above formula, the normal force This refers to the total normal force acting on the soft elastic body. Normal force mapping function; tangential force This refers to the total global tangential force acting on the soft elastic body. For tangential force mapping function; torque This refers to the global torque acting on the soft elastic body. These are torque mapping functions. These mapping functions can be linear or nonlinear, and their specific form is determined by calibration experimental data. In actual measurements, the calculated actual normal force is... Actual tangential force Actual torque By substituting the corresponding mapping function, the total normal force, tangential force, and torque acting on the soft elastic body in the current state can be calculated.

[0087] In an optional embodiment, for the need for high-precision detection of distributed mechanical features of the soft elastomer 21 (such as micro-force sensors, biomechanical detection of biological tissues), step S16 can also employ photometric stereoscopic solution by fusing neural networks with multiple features: In an alternative embodiment, such as Figure 5 As shown, the steps of "performing photometric stereoscopic calculation on visually changing images and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media" specifically include the following steps: S1631. Obtain a non-contact reference image obtained by the light source reflecting through the first and second reflective media when the soft elastic body is not in contact with the object. S1632. Based on photometric stereo calculation, extract the local normal change features on the side of the soft elastic body in contact with the object in the non-contact reference image and the visual change image. S1633. Extract the difference data of position and reflection characteristics of the first and second reflective media in the non-contact reference image and the visual change image; S1634. By integrating local normal change features and differences in position and reflection characteristics, comparison information between non-contact reference images and visually changed images is generated. S1635. Input the comparison information into the pre-trained solution neural network model to output the mechanical characteristic data of the soft elastic body; wherein, the mechanical characteristic data of the soft elastic body includes the distributed normal displacement, normal force, tangential displacement and tangential force of the soft elastic body.

[0088] Specifically, this embodiment covers a technical framework of "fusing multiple types of features and solving distributed data through a pre-trained neural network," which enables refined mechanical distribution detection of soft elastomers at the 21-pixel level, adapting to high-precision scenarios such as precision instrument assembly and biomechanical analysis of biological tissues. The scope of protection of this embodiment is not limited to specific network architectures or feature types; any method that can fuse multiple differential features and solve distributed mechanical data through a pre-trained model falls within the scope of this embodiment.

[0089] In an optional embodiment, steps S1631 to S1635 of "performing photometric stereoscopic calculation on the visual change image to perceive the mechanical characteristic data of the soft elastic body in its current state" employ a distributed force calculation method based on a neural network to calculate the distributed normal displacement, normal force, tangential displacement, and tangential force of the soft elastic body. The specific implementation is as follows: When performing step S1631, a non-contact reference image is obtained by reflecting light from the first and second reflective media when the soft elastic body is not in contact with the object, denoted as This image is RGB three-channel data with a resolution of w*h (e.g., 1280×960 pixels), serving as the reference image for subsequent input data pairs.

[0090] During step S1632, visual images of the changes after the soft elastomer comes into contact with the object are acquired. The image is also RGB three-channel with a resolution of w*h. Meanwhile, it was simulated using finite element simulation software such as... Figure 15 As shown, Figure 15 To visualize the output displacement field, presenting the deformation distribution of the elastic body under contact load, simulate various contact scenarios and output the displacement field (ux, uy, uz) and stress field (fx, fy, fz) for each simulation step as the true value of the output tensor.

[0091] When executing step S1633, the mesh node data obtained from the finite element simulation is interpolated to the image pixel mesh (resolution w*h) to ensure spatial alignment and generate the output tensor. In the above formula, This is the output tensor used to store the actual displacement and force data for each pixel; w is the image width (pixels); h is the image height (pixels); This indicates that the output tensor is a real tensor with w rows and h columns and 6 channels; each pixel... ,in These represent the distributed displacements of the pixel in the x, y, and z directions, respectively. These represent the distributed forces of the pixel in the x, y, and z directions, respectively; i and j are the row and column indices of the pixel in the image.

[0092] When performing step S1634, the non-contact reference image is... and contact deformation images Perform channel concatenation to generate the input tensor. In the above formula, The input tensor serves as the input data for the neural network. For channel splicing operations; For non-contact reference image (3 channels); For contact deformation image (3 channels); w is image width (pixels); h is image height (pixels); This indicates that the input tensor is a real number tensor with w rows, h columns, and 6 channels.

[0093] During step S1635, a computational neural network model Θ is constructed (using the U-net architecture, which has an encoder-decoder structure and skip connections to preserve spatial detail information), and the input tensor is... Input model, output 6-channel tensor with the same resolution as the input. This is used to predict the distributed displacement and force of each pixel. To improve model accuracy and physical plausibility, a multi-task learning optimization strategy is adopted, with the loss function being... In the above formula, This is the loss function, used to measure the error between the model's predicted values ​​and the actual values; , These are weighting coefficients used to balance the accuracy requirements of displacement and force; The L2 norm (Euclidean distance); The data is from a real distributed displacement tensor (w×h×3). Figure 15 The displacement field shown in the finite element simulation; For the distributed displacement tensor predicted by the model; It is a real distributed force tensor (w×h×3), and its data comes from the stress field of finite element simulation. This refers to the distributed force tensor predicted by the model. During training, physical constraints (such as equilibrium equations) are introduced. ,in The stress tensor is used as the regularization term; the Adam optimizer is employed with a learning rate decay schedule; optionally, data augmentation techniques such as elastic deformation, brightness perturbation, and simulated marker occlusion are used to improve the model's generalization ability. After training, the model can output distributed mechanical feature data for each pixel on the surface of the soft elastic body, including normal displacement (uz), normal force (fz), tangential displacement (ux, uy), and tangential force (fx, fy).

[0094] like Figure 6 , Figure 7 , Figure 8 As shown, according to one aspect of this application, a visual-tactile sensor 20 is also provided, comprising: A soft elastic body 21 deforms in response to external pressure. A reflective medium 26 with non-directional or partial non-directional reflective properties is provided on the side of the soft elastic body 21 that contacts the object. The reflective medium 26 includes a first reflective medium 261 that reflects the internal environment of the soft elastic body 21 and a second reflective medium 262 for marking. The first reflective medium 261 and the second reflective medium 262 have different albedoes, and the second reflective medium 262 has non-directional or partial non-directional reflective properties that participate in photometric stereoscopic calculation. A light source 22 illuminates the soft elastic body 21 and provides the illumination required for photometric stereoscopic calculation; A photosensitive structure 23 can receive light rays reflected by the reflective medium 26 after illumination by the light source 22, resulting in changes in the local normal, and generate a visual change image. The visual change image includes light intensity distribution information of the first reflective medium 261 and the second reflective medium 262. This visual change image is used for photometric stereoscopic calculation to obtain the mechanical characteristic data of the soft elastic body 21. In an optional embodiment, the soft elastic body 21 is a flexible substrate with recoverable deformation capability. It not only needs to deform under external force contact but also needs to have a certain degree of optical transmittance to avoid excessively hindering light source penetration and affecting the acquisition of scattered signals.

[0095] Specifically, the material of the aforementioned soft elastomer can be selected from polydimethylsiloxane, thermoplastic elastomer, polyurethane, or dual-network hydrogel, and can be flexibly adjusted according to the detection scenario. In terms of hardness, its Shore A hardness is usually between 5 and 90, and can be adjusted to 10 to 40 in conventional applications. In addition, the shape and thickness of the aforementioned soft elastomer are not fixed and can be adjusted to fit the overall structure of the sensor (such as finger-shaped or block-shaped).

[0096] Furthermore, the first reflective medium 261, acting as a reflective medium for reflecting the internal environment of the soft elastomer, can optionally be a continuously distributed diffuse reflective layer. Its material can be at least one elastic transparent material such as silicone or polyurethane, and doped with scattering particles, such as at least one of titanium dioxide, alumina, or polystyrene microspheres, to achieve non-directional reflective properties. Specifically, the particle size of the scattering particles can be selected from 0.1 μm to 100 μm, the doping mass fraction is 0.1% to 20%, and it is formed on the surface of the soft elastomer 21 by coating, molding, or blending processes, with a thickness of 1 μm to 200 μm.

[0097] Optionally, the second reflective medium 262 serves as the reflective medium for marking, in the form of a discrete pattern, such as a dot matrix, stripes, geometric shapes, etc. (e.g. Figure 7 The circular dot matrix (in the first reflective medium 261) is distributed within or on the surface of the first reflective medium 261. Its material can be at least one elastic material compatible with the first reflective medium 261. By adding different types or amounts of reflectance-modifying components, such as carbon black, metal powder, pigments, etc., a difference in reflectivity between the second reflective medium 262 and the first reflective medium 261 can be achieved. Specifically, the individual mark size of the second reflective medium 262 is 0.01mm-5mm, the mark spacing is 0.05mm-10mm, and the molding process can be at least one of printing, dispensing, or molding.

[0098] It should be noted that "non-directional reflection characteristics or partially non-directional reflection characteristics" means that the reflection of incident light by the reflecting medium is not constrained by a fixed direction, and the reflected light is spatially diffused (diffuse reflection component accounts for ≥60%). This characteristic is an important optical property of the protection scope of this embodiment, ensuring the integrity of information acquired in subsequent visual image acquisition. Optionally, the albedo ratio of the first reflecting medium and the second reflecting medium is between 0.1 and 10 to ensure the grayscale distinction between the two in the visual image, providing a reliable basis for subsequent mechanical feature calculation.

[0099] Specifically, the light source 22 may be a solid-state light source, such as at least one LED or laser diode, and the quantity is at least one. The arrangement can be symmetrical, circular, or arrayed on the non-contact object side of the soft elastic body 21 (e.g., Figure 6 (The symmetrical dual-light source layout in the middle); the light from the light source 22 passes through the soft elastic body 21 and shines on the reflective medium 26 by direct illumination or by being guided by at least one optical element, such as a lens or a light guide plate, to ensure that the light covers the effective area of ​​the reflective medium 26.

[0100] Furthermore, the light source 22 can independently control the light intensity, wavelength and on / off state, and optionally supports a light intensity adjustment range of 100 lux-10000 lux and a wavelength range of 380 nm-780 nm; specifically, when multiple light sources are used, each light source can work in a time-sharing or simultaneous manner to adapt to the lighting requirements of different detection scenarios.

[0101] In one specific embodiment, the first reflective medium 261 is an organosilicon material doped with 5μm titanium dioxide particles (doped by 5% by mass), which is formed on the surface of the soft elastomer 21 by a coating process, with a thickness of 20μm and an albedo of 0.8; the second reflective medium 262 is an organosilicon material with 3% carbon black added by mass, which is formed into a circular dot matrix with a diameter of 1mm and a spacing of 2mm by a dispensing process (e.g., Figure 7 As shown), the albedo is 0.2; Light source 22: Two LED light sources with a wavelength of 550nm are selected, with a light intensity of 2000 lux, and are symmetrically distributed at a 60° angle below the soft elastic body 21. The light passes through the soft elastic body 21 (thickness 1mm, light transmittance 95%) and shines on the reflective medium 26.

[0102] In one optional embodiment, the light source includes at least two sets of independent light sources, each set having a different illumination direction, capable of illuminating the reflective medium from different directions. Specifically, as... Figure 6 , Figure 8 As shown, the light source 22 can be configured as two groups, illuminating the soft elastic body 21 from both sides at different angles, thereby providing multi-directional illumination information for photometric stereoscopic calculation to accurately calculate the local normal changes of the reflecting medium 26. It should be noted that the number of light source groups 22 is not limited to two groups; at least two groups can be set according to the calculation accuracy requirements, such as three or four groups, and the illumination directions of each group of light sources must be different to meet the illumination conditions for photometric stereoscopic calculation. These different configuration methods are all within the protection scope of this embodiment.

[0103] In an optional embodiment, the light source includes at least two sets of independent light sources, wherein the emission wavelengths of the at least two sets of independent light sources are different. For example, one set of light sources can emit red light, and the other set of light sources can emit green light. By illuminating the reflective medium 26 with light of different wavelengths, the difference in reflection of light of different wavelengths on the reflective medium can be utilized to further improve the accuracy and anti-interference capability of the photometric stereoscopic solution. It should be noted that the specific selection of the emission wavelength is not limited to red and green light. Other combinations of different wavelengths can be selected according to actual needs, as long as the condition that the emission wavelengths of at least two sets of independent light sources are different is met, all of which fall within the protection scope of this embodiment.

[0104] It should be noted that the specific parameters mentioned above are merely examples, and the scope of protection of this embodiment is not limited to these specific values. Any equivalent substitutions or simple modifications made to the material, structure, and size of the reflective medium, or the type, layout, and parameters of the light source, under the technical concept of "the reflective medium possessing non-directional or partially non-directional reflective characteristics, the first and second reflective media having different albedoes, and the light source being irradiated by a soft elastic body onto the reflective medium," are all within the scope of protection of this embodiment.

[0105] In an optional embodiment, the photosensitive structure 23 can be an image acquisition device such as a CMOS sensor or a CCD sensor, which can capture visual changes in the reflective medium 26 under different lighting conditions. Specifically, as shown... Figure 6 , Figure 8 As shown, the photosensitive structure 23 is disposed below the soft elastomer 21 and can receive light reflected by the reflective medium 26 and generate image data. For example, when an external object 30 comes into contact with the soft elastomer 21 (such as...), the photosensitive structure 23 can receive light reflected by the reflective medium 26 and generate image data. Figure 8 As shown, the soft elastomer 21 deforms, causing a change in the local normal of the reflective medium 26. The photosensitive structure 23 captures this change and generates a visual change image. After photometric stereoscopic calculation, the mechanical characteristic data of the soft elastomer, such as normal force, tangential force, and torque, can be obtained from this image. It should be noted that the specific type and installation position of the photosensitive structure 23 are not limited to the example above. As long as it can receive the light from the change in the local normal of the reflective medium 26 and generate a visual change image, it is acceptable. For example, the photosensitive structure can be placed on the side of the soft elastomer. These different implementation methods are all within the protection scope of this embodiment.

[0106] Furthermore, the photosensitive structure 23 can optionally be positioned on the side of the soft elastic body 21 away from the contact entity 23, and distributed on the same side or opposite side to the light source 22, such as... Figure 6 As shown, the light source 22 can be located on different sides of the soft elastic body 21 to form a reflected light receiving path. If the spatial layout is limited, at least one optical element among a mirror, prism, and lens can be used to deflect the reflected light path to ensure that the photosensitive structure 23 can clearly capture the visual changes of the reflective medium 26. At the same time, the photosensitive structure 23 can support static single-frame acquisition or dynamic multi-frame sequence acquisition. The frame rate of the dynamic acquisition mode can be optionally not less than 10 frames / second to adapt to the dynamic deformation process of the soft elastic body 21 caused by the pressure of the solid 30.

[0107] like Figure 8 As shown, the local normal change occurs when the entity 30 applies pressure, tension, or tangential force to the soft elastic body 21. Local areas of the soft elastic body 21 undergo deformation such as depression, bulge, stretching, or shearing, causing the surface normal of the reflective medium 26 in that area to shift from its initial unstressed state (e.g., a direction approximately perpendicular to the surface of the soft elastic body 21) towards the direction of the applied force, resulting in a local normal change. Figure 8 For example, when the entity 30 presses on the soft elastic body 21, the contact area of ​​the soft elastic body 21 undergoes concave deformation, causing the surface normal of the reflective medium 26 in this area to shift from the initial vertical direction to the direction of pressure applied by the entity 30. This change will alter the direction and intensity of the reflected light from the light source 22 by the reflective medium 26, thereby presenting differences in optical characteristics such as grayscale and texture in the visually altered image.

[0108] In a specific example, a CMOS image sensor with a resolution of 1200×800 pixels is selected as the photosensitive structure 23, and is set in... Figure 6 Below the soft elastomer 21, on the side furthest from the contact entity 23, it is distributed on opposite sides to the two sets of LED light sources 22 (the optical axis forms an angle of 60°-120° with the surface of the soft elastomer 21); when Figure 8 The medium entity 30 is a plastic ball with a diameter of 4 mm. When a pressure of 0.8 N is applied to the soft elastic body 21, the contact area of ​​the soft elastic body 21 is indented. The surface normal of the reflective medium 26 in this area is shifted by about 25° relative to the initial direction, forming a local normal change. At this time, the CMOS image sensor continuously acquires the image sequence of this process at a frame rate of 25 frames / second, and obtains a visual change image containing the change in diffuse reflection intensity of the first reflective medium 261 and the change in contrast of the marked area of ​​the second reflective medium 262.

[0109] It should be noted that the specific parameters mentioned above, such as the resolution, frame rate, and layout position of the photosensitive structure 23, the type and magnitude of the force applied by the entity 30, and the normal offset angle, are merely examples. Any equivalent substitutions or simple modifications made to the type, resolution, frame rate, layout, and optical path adjustment methods of the photosensitive structure 23, as well as the force form of the entity 30 and the deformation type of the soft elastic body involved in the "local normal change," under the technical concept of "acquiring images of visual changes in the reflective medium 26 caused by local normal changes in the soft elastic body 21 (caused by the action of the entity 30) through at least one photosensitive structure 23 in a matched reflective optical path manner," are all within the protection scope of this embodiment. The protection scope of "local normal change" covers changes in the normal direction of the surface of the soft elastic reflective medium caused by at least one force such as pressure, tension, or tangential force applied by the entity 30, and is not limited to a specific entity 30 or deformation form.

[0110] In an optional embodiment, the first reflective medium is a reflective layer covering the surface of the soft elastomer on the force-bearing side; the second reflective medium is a block-shaped marker of the soft elastomer distributed on the reflective layer and / or on the side close to the reflective layer.

[0111] In one specific embodiment, such as Figure 9 , Figure 10 As shown, the visual-tactile sensor 20 includes a soft elastic body 21, a reflective medium 26, a light source 22, and a photosensitive structure 23. The surface of the soft elastic body 21 on the force-bearing side is covered with a first reflective medium 261 (reflective layer). The second reflective medium 262 consists of block-shaped markers distributed within the soft elastic body on the reflective layer and / or near the reflective layer, specifically as shown below. Figure 9 As shown, these block-shaped markings are embedded inside the soft elastomer 21, located in the region below the reflective layer; when an external object 30 comes into contact with the soft elastomer 21 (such as...), Figure 10As shown), the soft elastic body 21 deforms, causing the surface morphology of the reflective layer (first reflective medium 261) to change, which in turn causes local normal changes. That is, the normal direction of each point on the surface of the reflective layer changes due to deformation, and at the same time, the position of the block mark (second reflective medium 262) also shifts relatively.

[0112] Specifically, the local normal change of the first reflective medium 261 will cause a change in the direction of its reflection of light from the light source 22, and the information such as the intensity and angle of the reflected light received by the photosensitive structure 23 will change accordingly. The second reflective medium 262, as a block-shaped marker, can be represented as the displacement of the marker point in the visual image by its position change inside the soft elastic body. Combined with the information of the local normal change, the mechanical characteristics of the soft elastic body can be calculated more accurately. It should be noted that the "local normal change" is caused by the deformation of the soft elastic body, and the degree of change is related to the magnitude and direction of the external force. This embodiment does not limit the specific deformation form of the soft elastic body (such as compression, tension, bending, etc.), as long as the normal direction of the reflective layer surface changes due to the external force. The "block-shaped" form of the second reflective medium 262 includes, but is not limited to, three-dimensional structures such as cubes, cylinders, and prisms. Its position in the soft elastic body can be distributed below the reflective layer, on the side, or partially embedded in the reflective layer. These different placement methods are all within the protection scope of this embodiment.

[0113] In an optional embodiment, the surface of the soft elastomer on the force-bearing side is covered with a light-shielding layer, which is used to block external ambient light from entering the photosensitive unit; the second reflective medium is a block-shaped mark of the soft elastomer distributed on the light-shielding layer and / or on one side close to the light-shielding layer.

[0114] In an alternative embodiment, the first reflective medium is a scattering particle layer disposed on a soft elastomer, the scattering particle layer being composed of one or more aggregated scattering particles.

[0115] Specifically, such as Figure 11 , Figure 12 As shown, the visual-tactile sensor 20 includes a soft elastomer 21, a light-shielding layer 27, a first reflective medium 264, a second reflective medium 262, a light source 22, and a photosensitive structure 23.

[0116] The light-shielding layer 27 covers the surface of the soft elastic body 21 on the force-bearing side (i.e., the side in contact with the external object), such as... Figure 11As shown, its main function is to block ambient light from entering the photosensitive unit, preventing interference from ambient light in subsequent optical signal acquisition. The material can be a coating containing light-absorbing filler, a black polymer film, a metal vapor-deposited layer, etc. The manufacturing process includes, but is not limited to, spraying, lamination, and deposition. Specific materials, processes, and thicknesses are not limited, as long as they effectively block ambient light. It should be noted that the protection scope of the light-shielding layer 27 is primarily focused on its function of isolating external ambient light interference; any structure possessing this function falls within the protection scope.

[0117] The first reflective medium 264 is a layer of scattering particles disposed inside the soft elastic body 21, such as Figure 12 As shown, the first reflective medium 264 is preferably arranged close to the light-shielding layer 27, consisting of one or more aggregated scattering particles. It possesses non-directional reflective properties and can form a scattered light field supporting photometric stereoscopic calculations under light source illumination. The materials of the scattering particles include, but are not limited to, materials with light-scattering capabilities such as titanium dioxide, alumina, and glass microspheres. The aggregation method, particle size, and distribution concentration in the soft elastomer are not limited, as long as they can provide non-directional reflected light that meets the requirements of photometric stereoscopic calculations. It should be noted that the protection range of the first reflective medium 264 is centered on the aggregated scattering particles possessing non-directional reflective properties, and is not limited to specific particle parameters or arrangement depth.

[0118] The second reflective medium 262 is a block-shaped marker, distributed within the light-shielding layer 27 and / or the soft elastic body 21 near the light-shielding layer 27, such as... Figure 11 As shown, it can produce detectable positional changes with the deformation of the soft elastomer, providing a positional reference for tangential force calculation. Its structural form includes, but is not limited to, blocky shapes such as cylinders, cubes, and prisms. The material must have a identifiable optical difference from the soft elastomer (e.g., different reflectivity, different colors). Specific shape, material, and distribution density are not limited, as long as it can produce detectable positional changes with the deformation of the soft elastomer. It should be noted that the protection range of the second reflective medium 262 is centered on its placement within the light-shielding layer and the nearby soft elastomer, and its detectable displacement with deformation, not limited to specific structural parameters.

[0119] like Figure 12 As shown, when an external object comes into contact with the soft elastic body 21, the deformation of the soft elastic body causes the first reflective medium 264 to produce a local change in normal (change in the scattered light reflection characteristics), and at the same time the second reflective medium 262 produces a position change. The optical signals of the two are captured synchronously by the photosensitive structure 23. Combined with the multi-dimensional illumination of the light source 22, the normal force and tangential force of the marked area and the non-marked area can be calculated together, eliminating the calculation blind spot in the prior art.

[0120] In an optional embodiment, the second reflective medium is a block-shaped marker distributed inside a soft elastomer on the side of the light-shielding layer and / or close to the light-shielding layer, with the block marker having different albedoes depending on its location within the soft elastomer.

[0121] In one specific embodiment, the visual-tactile sensor includes a soft elastomer 21, a first reflective medium 261, a second reflective medium 262, a light source 22, and a photosensitive structure 23. The soft elastomer 21 can deform in response to external pressure, and its shape can be designed according to the application scenario (e.g., having a curved structure to adapt to complex contact scenarios). The first reflective medium 261 has non-directional reflective properties, providing basic light field information for photometric stereoscopic calculation. The second reflective medium is a block-shaped marker distributed within the soft elastomer on the side of the light-shielding layer and / or close to the light-shielding layer, and the block marker has different albedo values ​​depending on its location within the soft elastomer.

[0122] Specifically, such as Figure 13 , Figure 14 As shown, the albedo of the second reflective media 262a, 262b, and 262c is different; the light source 22 is used to illuminate the soft elastomer, providing the illumination required for photometric stereoscopic calculation; the photosensitive structure 23 can receive the light reflected by the light source after illumination, generate a visual change image for photometric stereoscopic calculation, and obtain the mechanical characteristic data of the soft elastomer.

[0123] Furthermore, the block-shaped markings of the second reflective medium are set with different albedo values ​​according to the distribution area of ​​the soft elastomer 21, such as... Figure 13 Albedo 262a, 262b, and 262c are distributed in different locations in the soft elastomer and have different albedo values. Figure 14 Different shades of gray in the block-shaped markers correspond to different albedoes. The purpose of this setting is to: combine the shape of the soft elastomer 21 (such as...) Figure 13 As shown in the curved surface structure, the block-shaped markers in different locations have different albedoes, making them more distinct in the visual image and facilitating the accurate identification of the position and orientation of each marker by the photosensitive structure 23. When the soft elastomer 21 deforms due to external pressure, the displacement and orientation changes of the block-shaped markers with different albedoes can be captured more accurately, thereby assisting in photometric stereoscopic calculation and achieving multi-region, high-precision calculation of mechanical characteristics (such as normal force, tangential force, and torque). Especially when the soft elastomer has a complex shape (such as a curved surface), it can avoid calculation errors caused by confusion of marker areas.

[0124] It should be noted that the specific shape (e.g., circular, square, etc.), albedo value, distribution density, and area division method of the block-shaped markers of the second reflective medium are not limited. As long as "block-shaped markers with different albedo values ​​are set according to the distribution area of ​​the soft elastomer, and distributed in the light-shielding layer and / or inside the soft elastomer on the side close to the light-shielding layer," they all fall within the protection scope of this embodiment. For example, the albedo can be set according to the curvature of the soft elastomer, or the albedo can be distinguished according to the force-sensitive area and the non-sensitive area. These different implementation methods can all achieve the function of "improving the accuracy of marker recognition and assisting in the mechanical calculation of complex-shaped soft elastomers," and therefore are all within the protection scope.

[0125] The working process of this embodiment is as follows: Light source 22 illuminates the soft elastomer 21 from different directions. The non-directional reflected light from the first reflective medium 261 and the reflected light from the second reflective medium (block-shaped markers with different albedoes) are received by the photosensitive structure 23. When an external object comes into contact with the soft elastomer 21, the deformation of the soft elastomer causes a local change in the normal of the first reflective medium 261. Simultaneously, the block-shaped markers with different albedoes undergo changes in position and orientation. This change information is converted into a visual change image by the photosensitive structure 23. Through accurate identification and photometric stereoscopic calculation of the markers with different albedoes in the image, distributed mechanical characteristic data of each region of the soft elastomer can be obtained, even if the soft elastomer has complex shapes such as curved surfaces (e.g., Figure 13 Even curved surfaces can achieve blind-zone-free, high-precision force calculation.

[0126] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the visual-touch sensor embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0127] The above description is merely an embodiment of this specification and is not intended to limit the scope of one or more embodiments of this specification. Various modifications and variations can be made to one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of one or more embodiments of this specification.

Claims

1. A visual-tactile sensing method, applied in a visual-tactile sensor equipped with a soft elastic body, characterized in that, Includes the following steps: A light source is configured such that the light source shines through the soft elastic body onto a reflective medium with non-directional or partially non-directional reflective properties, which is disposed on the side of the soft elastic body that is in contact with the object. The reflective medium includes a first reflective medium that reflects the internal environment of the soft elastic body and a second reflective medium for marking. The first and second reflective media have different albedoes, and the second reflective medium possesses non-directional or partially non-directional reflective properties that participate in photometric stereoscopic calculations. The visual change image of the reflective medium caused by the contact of the soft elastic body with the object is acquired; wherein, the visual change image includes the light intensity distribution information of the first and second reflective media as the normal changes under the illumination of the light source; Photometric stereoscopic calculation is performed on the visual change image, and the mechanical characteristic data of the soft elastic body in its current state are perceived based on the light intensity distribution information of the first and second reflective media.

2. The method according to claim 1, characterized in that, The mechanical characteristic data includes the normal displacement and corresponding normal force, and the tangential displacement and corresponding tangential force of the soft elastic body; the step of "performing photometric stereoscopic calculation on the visual change image and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflecting media" specifically includes the following steps: Based on the principle of photometric stereo, the three-dimensional contour features of the soft elastic body on the side in contact with the object are calculated by utilizing the light intensity distribution information of the first and second reflective media in the visual change image. Extract the displacement and / or deformation features of the second reflective medium from the visual change image; By combining the three-dimensional contour features with the displacement features and / or deformation features, the normal displacement, normal force, tangential displacement, and tangential force of the soft elastic body in its current state are calculated.

3. The method according to claim 2, characterized in that, The step of "calculating the three-dimensional contour features of the side of the soft elastic body in contact with the object based on the photometric stereo principle and utilizing the light intensity distribution information of the first and second reflecting media in the visual change image" further includes the following step: Extract the difference in reflection characteristics between the first and second reflecting media in the visual change image; The interference of the albedo difference between the first and second reflective media on the photometric stereo solution is eliminated based on the difference in reflectivity characteristics.

4. The method according to claim 1, characterized in that, The step of "performing photometric stereoscopic calculation on the visual change image and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflective media" specifically includes the following steps: When the soft elastomer is not in contact with an object, a non-contact reference image is obtained by reflection from the first reflective medium and the second reflective medium. Based on the non-contact reference image and the visual change image, the planar displacement information of the first reflective medium and the second reflective medium is extracted to establish a global displacement field model of the soft elastic body. Photometric stereoscopic calculations are performed based on the displacement information model to perceive the mechanical characteristic data of the soft elastic body in its current state. The mechanical characteristic data of the soft elastic body includes the global sum of the normal displacement, normal force, tangential displacement, tangential force, and torque of the soft elastic body. The tangential displacement and tangential force are determined based on the planar displacement information of the second reflecting medium, and the torque is determined based on the divergence-free components in the global displacement field model.

5. The method according to claim 1, characterized in that, The step of "performing photometric stereoscopic calculation on the visual change image and perceiving the mechanical characteristic data of the soft elastic body in its current state based on the light intensity distribution information of the first and second reflective media" specifically includes the following steps: Obtain a non-contact reference image obtained by reflecting light through the first and second reflective media when the soft elastomer is not in contact with an object; Based on photometric stereoscopic calculation, the local normal variation features of the soft elastomer on the side in contact with the object are extracted from the non-contact reference image and the visual change image; Extract the difference data of position and reflection characteristics of the first and second reflective media in the non-contact reference image and the visual change image; By integrating the local normal variation features with the position and reflection characteristic difference data, comparison information between a non-contact reference image and a visually altered image is generated. The comparison information is input into a pre-trained computational neural network model to output the mechanical characteristic data of the soft elastic body; wherein, the mechanical characteristic data of the soft elastic body includes the distributed normal displacement, normal force, tangential displacement and tangential force of the soft elastic body.

6. The method according to claim 1, characterized in that, The step of "acquiring visual images of the changes in local position and normal of the reflective medium due to the contact of the soft elastic body with the object" specifically includes the following steps: Images reflected by the reflective medium are continuously acquired to obtain a multi-frame image sequence including at least two sets of illumination angles; wherein the independent light source illuminates the reflective medium using at least one of time-division illumination, illumination with different wavelengths, or illumination with different polarizations; The multi-frame image sequence is subjected to time-series synchronization processing to obtain visual change images; wherein, the visual change images are obtained by separating illumination data corresponding to different illumination directions based on differences in acquisition time, differences in light wavelength, or differences in light polarization.

7. A visual-tactile sensor, characterized in that, include: A soft elastomer that deforms in response to external pressure, wherein a reflective medium with non-directional or partially non-directional reflective properties is disposed on the side of the soft elastomer that contacts an object; wherein the reflective medium includes a first reflective medium that reflects the internal environment of the soft elastomer and a second reflective medium for marking; the first and second reflective media have different albedoes, and the second reflective medium has non-directional or partially non-directional reflective properties that participate in photometric stereoscopic calculation. A light source for illuminating the soft elastomer and providing the illumination required for photometric stereoscopic calculation; A photosensitive structure capable of receiving light rays that change due to local normal variations in the reflective medium after being illuminated by the light source, and generating a visual change image; wherein, the visual change image includes light intensity distribution information of the first and second reflective media, and the visual change image is used for photometric stereoscopic calculation to obtain the mechanical characteristic data of the soft elastic body.

8. The visual-tactile sensor according to claim 7, characterized in that, The light source includes at least two sets of independent light sources, each set of independent light sources having a different illumination direction, which can illuminate the reflective medium from different directions.

9. The visual-tactile sensor according to claim 7, characterized in that, The light source includes at least two independent light sources, wherein the emission wavelengths of the at least two independent light sources are different.

10. The visual-tactile sensor according to claim 7, characterized in that, The first reflective medium is a reflective layer covering the surface of the soft elastomer on the force-bearing side; the second reflective medium is a block-shaped marker of the soft elastomer distributed on the reflective layer and / or on one side close to the reflective layer.

11. The visual-tactile sensor according to claim 7, characterized in that, The surface of the soft elastomer on the force-bearing side is covered with a light-shielding layer, which is used to block external ambient light from entering the photosensitive unit; the second reflective medium is a block-shaped mark of the soft elastomer distributed on the light-shielding layer and / or on one side close to the light-shielding layer.

12. The visual-tactile sensor according to claim 7 or 11, characterized in that, The first reflective medium is a scattering particle layer disposed on a soft elastomer, the scattering particle layer being composed of one or more aggregated scattering particles.

13. The visual-tactile sensor according to claim 7, characterized in that, The second reflective medium is a block-shaped marker, which has different albedoes depending on its location within the soft elastomer distribution area.

Images