A visual-tactile sensor and a visual-tactile sensing method thereof

By using a geometric topology coding layer and frequency domain analysis in the vision sensor, the robustness problem of illumination changes to 3D shape reconstruction is solved, achieving stable perception in complex environments, which is suitable for robot operation and precision assembly.

CN123111127APending Publication Date: 2026-11-17INSPIRATION ROBOT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611185281.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-06
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing visual sensors lack robustness under varying lighting conditions and ambient light, making it difficult to achieve stable 3D shape reconstruction and contact deformation sensing in complex operating scenarios.

Method used

A geometric topological coding layer is attached to the surface of the elastic body. Deformation images are acquired through an imaging unit and frequency domain analysis is performed. The three-dimensional morphology is reconstructed using geometric phase and the tangential displacement vector field is obtained by combining spatial domain analysis. This enables three-dimensional morphology reconstruction and contact deformation sensing that are insensitive to changes in illumination.

Benefits of technology

Stable 3D topography reconstruction and contact deformation sensing were achieved under conditions of fluctuating light source or changing ambient light, improving the reliability and accuracy of sensing, and making it suitable for robot operation and precision assembly.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123111127A_ABST
    Figure CN123111127A_ABST
Patent Text Reader

Abstract

This application discloses a visual-tactile sensor and its visual-tactile sensing method. The method includes: acquiring a deformation image of a geometric topological coding layer under contact force, resulting in geometric distortion; performing frequency domain analysis on the deformation image, and reconstructing the three-dimensional morphology of the contact surface based on the phase change or frequency shift of the coding layer; and outputting the three-dimensional morphology. Addressing the problems of photometric stereo methods relying on optical energy coding and being sensitive to illumination changes, and the problem of the coding layer being separated from the measurement object in structured light projection methods and unable to sense contact deformation, this application extracts the phase change or frequency shift of the coding layer through frequency domain analysis to reconstruct the three-dimensional morphology. By using geometric phase as an information carrier, it achieves three-dimensional morphology reconstruction that is insensitive to illumination changes and can accurately sense contact surface deformation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of robot tactile perception technology, and in particular to a visual tactile sensor and a visual tactile sensing method thereof. Background Technology

[0002] Visual sensors are a new type of sensor that combines visual imaging with flexible tactile perception. They have important application value in fields such as robot dexterity, precision assembly, and human-computer interaction, and have become one of the mainstream technical paths for achieving high-resolution tactile perception.

[0003] Currently widely adopted technical solutions mainly include photometric stereoscopic light and shadow reconstruction methods. Photometric stereoscopic methods utilize the intensity gradient of reflected light to recover the surface normal vector to achieve 3D reconstruction. However, it is essentially an energy encoding method, and its reconstruction accuracy is highly dependent on the stability of the light source and the consistency of surface reflectivity, resulting in insufficient robustness under conditions of light source fluctuations or changes in ambient light. Furthermore, in the field of 3D topography measurement, there are also 3D reconstruction methods based on structured light construction. These methods transmit a fringe pattern with a specific spatial frequency onto the object's surface, acquire images of the deformed fringe patterns using imaging equipment, and extract the phase changes of the fringe patterns using methods such as Fourier transform to reconstruct the 3D topography. However, in this type of method, the encoded pattern exists in the form of projected light, and its encoding layer is separate from the measured object. That is, after the projected light is transmitted to the object's surface, the encoded pattern itself does not undergo physical distortion with the object's deformation, making it difficult to directly apply to sensing scenarios involving contact deformation.

[0004] Therefore, photometric stereo methods are sensitive to changes in illumination due to their reliance on energy encoding, while structured light projection methods cannot detect contact deformation because the encoding layer is separated from the object being measured. These issues limit the reliability and accuracy of visual-tactile sensors in complex operating scenarios. Summary of the Invention

[0005] This application provides a visual-tactile sensor and its visual-tactile sensing method, which solves the problems of photometric stereo sensing schemes that rely on optical energy encoding and are sensitive to changes in illumination and lack stable physical feature support, and the problems of structured light projection-based three-dimensional reconstruction methods that cannot sense contact deformation due to the separation of the encoding layer from the measurement object. It achieves the technical effect of using geometric phase as an information carrier to realize three-dimensional shape reconstruction that is insensitive to changes in illumination, and at the same time, it utilizes the physical coupling mechanism of the solid encoding layer attached to the surface of the elastic body deforming synchronously with the elastic body to accurately sense the deformation information of the contact surface.

[0006] This application provides a visual-tactile sensing method, the method comprising: A deformation image is acquired, which is generated by the geometric distortion of the geometric topology coding layer under the action of contact force; Frequency domain analysis is performed on the deformation image, and the three-dimensional morphology of the contact surface is reconstructed based on the phase change or frequency shift of the geometric topology coding layer. Output the three-dimensional shape.

[0007] Optionally, the step of reconstructing the three-dimensional morphology of the contact surface based on the phase change or frequency shift of the geometric topology coding layer includes: Perform a Fourier transform on the deformed image to obtain the frequency domain spectrum; The frequency domain spectrum is filtered and then subjected to inverse Fourier transform to extract phase information. The phase information is unwrapped to obtain the absolute phase distribution; The three-dimensional topography is reconstructed based on the mapping relationship between the absolute phase distribution and the height.

[0008] Optionally, the geometric topology coding layer is a multicolor stripe coding layer, and the step of reconstructing the three-dimensional topography includes: Separate the stripe images of the multicolor stripe coding layer in different color channels; Multi-step phase-shifting processing is performed on the stripe images of each color channel to extract the phase information of each channel; The three-dimensional topography is reconstructed based on the phase information of each channel.

[0009] Optionally, before the step of acquiring the deformed image, the following steps are included: Obtain a reference image of the geometric topology coding layer in a non-contact state; The coordinates of feature nodes in the reference image are extracted as reference coordinates, which are used to match the current coordinates to obtain the coordinate drift amount; Furthermore, Fourier transform and phase extraction are performed on the reference image to obtain reference phase information, which is used to perform differential calculation with the phase information of the deformed image to obtain the phase change amount, and the phase change amount is used to reconstruct the three-dimensional shape.

[0010] Optionally, the method further includes: Based on the output of the three-dimensional topography, further spatial domain analysis is performed on the deformation image, and the tangential displacement vector field is reconstructed according to the coordinate drift of the feature nodes of the geometric topology coding layer and the topological relationship between the feature nodes; Output the three-dimensional topography and the tangential displacement vector field.

[0011] Optionally, the step of reconstructing the tangential displacement vector field based on the coordinate drift of the feature nodes in the geometric topology coding layer and the topological relationship between the feature nodes includes: Corner detection is performed on the deformed image to locate the current coordinates of the feature nodes; Based on the topological connection relationship between the feature nodes, the current coordinates are matched with the reference coordinates; Based on the matching results, the coordinate drift of each feature node is obtained; The tangential displacement vector field is generated based on the coordinate drift.

[0012] Optionally, after the step of reconstructing the tangential displacement vector field, the following steps are included: Based on the displacement direction and magnitude of each feature node in the tangential displacement vector field, the slippage tendency of the contact surface is identified, and the slippage tendency information is output.

[0013] Optionally, the geometric topology coding layer is a color stripe or mixed frequency stripe coding layer, and the step of reconstructing the tangential displacement vector field includes: The deformed image is subjected to color channel separation to obtain independent image components corresponding to different color channels; Spatial domain analysis was performed on the independent image components of each color channel to obtain the displacement information corresponding to each color channel; The displacement information from different color channels is fused to generate the final tangential displacement vector field.

[0014] Optionally, the method further includes: Contact pressure distribution information is generated based on the density changes of feature nodes in each local region of the geometric topology coding layer. The contact pressure distribution information is fused with the tangential displacement vector field and / or the three-dimensional morphology to output full-dimensional tactile information.

[0015] Furthermore, to achieve the above objectives, embodiments of the present invention also provide a visual-tactile sensor, the visual-tactile sensor comprising: Imaging unit; The tactile sensing layer includes an elastic transparent layer and a geometric topology coding layer disposed on the elastic transparent layer. The imaging unit is disposed toward the tactile sensing layer and is used to acquire the deformation image generated by the geometric topology coding layer under the action of contact force. A light source is positioned around the imaging unit to illuminate the geometric topology coding layer; A processor for executing the visual-tactile sensing method as described above.

[0016] Optionally, the geometric topology coding layer is a bidirectional orthogonal grid, which has different spatial frequencies or periods in the two orthogonal directions to independently modulate the tangential displacements in the X-axis and Y-axis directions, respectively.

[0017] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages: 1. By attaching the geometric topology coding layer to the surface of the elastic body in a solid form, the coding layer deforms synchronously with the elastic body. The phase change or frequency shift of the coding layer is extracted by frequency domain analysis of the deformation image to reconstruct the three-dimensional shape. By using geometric phase instead of optical energy as the information carrier, the reconstruction process of the three-dimensional shape does not depend on the light intensity and surface reflectivity. It can still stably reconstruct the three-dimensional shape under the conditions of light source fluctuation or ambient light change, and achieve stable sensing performance without the need for a precise light source control system.

[0018] 2. By attaching the coded pattern directly to the surface of the elastic body in a physical form, it differs from the mechanism in conventional structured light projection where the coded pattern exists in the form of projected light and is separated from the measurement object. This creates an integrated physical coupling between the coded layer and the elastic body. When the elastic body deforms, the coded layer undergoes real physical distortion, thus enabling accurate perception of the deformation information of the contact surface. This overcomes the limitation of projection methods in being difficult to apply directly to contact deformation perception scenarios. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the visual-tactile sensor of this application; Figure 2 This is a schematic diagram of the geometric topology coding layer of this application; Figure 3 This is a schematic diagram of the mesh topology under compression distortion of the geometric topology coding layer in this application; Figure 4 This is a schematic diagram of the three-dimensional undulating structure of the geometric topology coding layer in this application; Figure 5 This is a schematic diagram of the visual-tactile sensing method of this application. Detailed Implementation

[0020] To address the issues of current visual-tactile sensors, such as the sensitivity to illumination changes caused by the reliance on optical energy encoding in wide-area stereo methods and the inability to sense contact deformation in structured light projection methods due to the separation of the encoding layer from the measured object, this application provides a visual-tactile sensor and its sensing method. By setting a geometric topological encoding layer with continuous topological constraints on the surface of an elastic body, an imaging unit acquires deformation images of the encoding layer under contact force, and through frequency domain analysis, reconstructs the three-dimensional morphology of the contact surface based on the phase change or frequency shift of the encoding layer. Utilizing geometric phase rather than optical energy as the information carrier, the reconstruction of the three-dimensional morphology is independent of illumination intensity and surface reflectivity, and can stably reconstruct the three-dimensional morphology even under conditions of fluctuating light sources or changes in ambient light. Furthermore, spatial domain analysis can be performed on the deformation images to reconstruct the tangential displacement vector field based on the coordinate drift of feature nodes in the encoding layer and the topological relationships between feature nodes, thereby obtaining the tangential displacement information of the contact surface.

[0021] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0022] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0023] Example 1 In this embodiment, a visual-tactile sensor is provided.

[0024] Reference Figure 1 The visual-tactile sensor in this embodiment includes an imaging unit 1, a light source 2, a tactile sensing layer, and a processor (not shown in the figure).

[0025] Imaging unit 1 is used to acquire images.

[0026] Alternatively, the imaging unit may be a camera, such as a miniature camera, with its optical axis oriented toward the tactile sensing layer.

[0027] Light source 2 is positioned around the imaging unit to provide illumination for the imaging unit, thereby illuminating the tactile sensing layer.

[0028] Optionally, the light source can be an LED light, such as a monochrome LED light. There can be one or more light sources arranged around the imaging unit to ensure uniform illumination of the tactile sensing layer.

[0029] Optionally, the light source and imaging unit can be arranged in a face-to-face manner, that is, the imaging unit is positioned facing the tactile sensing layer, and the light source is arranged around the imaging unit to illuminate the geometric topology coding layer from the facing direction.

[0030] Alternatively, the light source and imaging unit can be arranged in a sideways or non-directly facing manner. For example, the light source illuminates the geometric topology coding layer from the side, and the imaging unit acquires images from the side or at an angle, as long as it can illuminate and acquire the image information of the geometric topology coding layer. The sideways arrangement can reduce the interference of reflected light from the light source on imaging in the direct-facing arrangement, thus improving image quality. The placement and angle of the light source and imaging unit can be flexibly adjusted according to actual application requirements.

[0031] The tactile sensing layer includes an elastic transparent layer 31 and a geometric topology coding layer 32 disposed on the elastic transparent layer 31.

[0032] Optionally, the tactile sensing layer further includes a contact layer 33 and a rigid transparent layer 34. The tactile sensing layer, from bottom to top, includes a rigid transparent layer, an elastic transparent layer, a geometric topology coding layer, and a contact layer.

[0033] As an optional implementation, the rigid transparent layer 34 is located at the bottom of the tactile sensing layer and is made of a transparent rigid material, such as glass or transparent acrylic, to serve as the supporting base for the entire tactile sensing layer. It provides rigid support for the elastic transparent layer, ensuring that the tactile sensing layer will not undergo overall displacement when subjected to force, and that the deformation is concentrated in the elastic transparent layer.

[0034] As an optional implementation, the elastic transparent layer 32 is made of a transparent elastic material, such as silicone or polyurethane, which has the ability to deform under external force and return to its original shape after the external force is removed. The elastic transparent layer is the main layer that undergoes elastic deformation. When an external object comes into contact with the sensor, the contact force is transmitted to the elastic transparent layer through the contact layer, causing it to deform. The thickness of the elastic transparent layer can be adjusted according to sensitivity requirements; the thinner the layer, the more sensitive the response to minute deformations.

[0035] As an optional implementation, the geometric topology coding layer 32 is attached to the upper surface of the elastic transparent layer 31 and is the core implementation layer for the sensing function. It has a pre-set spatial feature pattern with specific geometric logic, as shown in the reference layer. Figure 2 From left to right, the patterns are: a grid, a digital stripe pattern with varying black and white values, and an analog stripe pattern with continuously varying grayscale values. The pattern of the geometric topology coding layer can be designed according to the actual sensing requirements. When it is necessary to acquire bidirectional tangential displacement along both the X and Y axes, a bidirectional orthogonal grid pattern can be used; when it is necessary to acquire only unidirectional tangential displacement, a unidirectional stripe pattern can be used.

[0036] Optionally, in addition to the grid and stripe patterns mentioned above, the geometric topology coding layer can also adopt the following pattern shapes: unidirectional stripes, variable-interval stripes, annular stripes, sinusoidal stripes with different grayscale distributions, color stripes with different color channels, or mixed-frequency stripes with different spatial frequencies. These different pattern forms can all utilize geometric topological features as carriers of tactile information, achieving stable sensing insensitive to changes in illumination. Specifically, when using variable-interval stripes, since different regions have different stripe densities, denser stripes can be set in areas requiring high-resolution measurement, while sparser stripes can be set in other areas, achieving different measurement resolutions in different regions. Annular stripes are suitable for scenarios where radial and tangential displacements of the contact surface are measured.

[0037] Optionally, color stripes refer to patterns in which stripes of different directions or frequencies are assigned different color channels. For example, when horizontal stripes are set to red and vertical stripes are set to blue, deformation information in the horizontal and vertical directions can be extracted separately through color channel separation. When high-frequency stripes are set to red and low-frequency stripes are set to green, high-frequency components and low-frequency components can be extracted separately through color channel separation. High-frequency components are sensitive to small deformations, while low-frequency components have better tracking capabilities for large deformations. Combining the two can simultaneously take into account both large deformation measurement and detail acquisition.

[0038] As an alternative implementation, the geometric topology coding layer is not limited to planar patterns, but can also be a three-dimensional structure with initial three-dimensional undulations, such as micro-boobs, prism arrays, or wavy textures. (See reference...) Figure 4 This demonstrates a sinusoidal three-dimensional structure superimposed in orthogonal directions. This three-dimensional coding layer, by introducing a physical prior in the depth direction, causes the three-dimensional structure to deform when contact force is applied to the tactile sensing layer. The resulting projection of this structure onto the imaging unit produces more significant geometric distortion, thereby enhancing the sensor's sensitivity to minute displacements in the Z-axis direction.

[0039] Alternatively, the geometric topology coding layer can also be fabricated on the elastic transparent layer using processes such as multi-color overprinting, direct printing, spraying, or photolithography. When the geometric topology coding layer is a color stripe, multi-color overprinting can be used to achieve precise alignment of different color channels; when the geometric topology coding layer is a three-dimensional structure, it can be fabricated using processes such as photolithography or molding.

[0040] Optionally, the geometric topological coding layer is located above the elastic transparent layer and below the contact layer, making it more sensitive to external contact deformation. The imaging unit observes the geometric topological coding layer from below through the hard transparent layer and the elastic transparent layer.

[0041] Optionally, the geometric topology coding layer is printed onto the elastic transparent layer using water transfer printing technology. Water transfer printing technology is suitable for forming coding layers with complex geometric topological features on the surface of elastomers, and can achieve high-precision pattern transfer while maintaining an extremely thin coding layer.

[0042] Alternatively, the geometric topology coding layer can also be prepared on the elastic transparent layer using processes such as multicolor overprinting, direct printing, spraying, or photolithography.

[0043] As an alternative implementation, the contact layer covers the geometric topology coding layer and is located on top of the tactile sensing layer. The contact layer directly contacts external objects and is used to transmit external contact forces to the underlying elastic transparent layer and geometric topology coding layer.

[0044] Optionally, the contact layer is made of a flexible transparent material with a certain degree of flexibility and wear resistance, which can adapt to the surface shape of different objects, fit closely to the object being contacted, and protect the underlying geometric topology coding layer from wear.

[0045] As an optional implementation, the imaging unit is used to acquire the deformation image generated by the geometric distortion of the geometric topology coding layer under the action of contact force.

[0046] Specifically, the imaging unit is positioned below the rigid transparent layer, observing the geometric topological coding layer from below through both the rigid and elastic transparent layers. When the imaging unit acquires an image of the geometric topological coding layer, its optical axis passes through the rigid and elastic transparent layers, focusing on the patterned surface of the geometric topological coding layer. When an external object comes into contact with the contact layer of the tactile sensing layer, the contact force is transmitted through the contact layer to the elastic transparent layer, causing deformation of the elastic transparent layer. Consequently, the geometric topological coding layer attached to the elastic transparent layer undergoes geometric distortion. The imaging unit acquires an image of the geometric topological coding layer through the rigid transparent layer; this image is the deformation image containing information about the geometric distortion.

[0047] Taking a two-way orthogonal grid as an example, when the contact layer is not subjected to external force, the imaging unit acquires a regular, undistorted orthogonal grid image; when the contact layer is subjected to force and deforms, the lines of the orthogonal grid bend, stretch, or compress, and the grid intersections experience coordinate drift. The imaging unit then acquires a deformed image containing this geometric distortion information. Figure 3 .

[0048] The processor, connected to the imaging unit, is used to receive the deformation image acquired by the imaging unit and execute the visual-tactile sensing method described in Embodiment 2.

[0049] Optionally, the processor can be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), or other programmable logic devices. By executing a visual-tactile sensing method, frequency domain analysis is performed on the deformation image. The three-dimensional morphology of the contact surface is reconstructed based on the phase change or frequency shift of the coding layer, and the three-dimensional morphology is output. Furthermore, spatial domain analysis can be performed on the deformation image to reconstruct the tangential displacement vector field based on the coordinate drift of feature nodes and topological relationships, and the tangential displacement vector field is output.

[0050] In this embodiment, the geometric topology coding layer employs a bidirectional orthogonal mesh image. The orthogonal mesh has different spatial frequencies or periods in the two orthogonal directions to independently modulate the tangential displacements in the X and Y axes, respectively. The intersections of the orthogonal meshes are feature nodes. When the elastic transparent layer deforms, the intersections of the orthogonal meshes experience coordinate drift. The processor can reconstruct the tangential displacement vector field by tracking these coordinate changes. Because the mesh lines provide continuous topological constraints, even when large deformations cause severe distortion of the local pattern, the processor can still perform accurate feature matching through the topological connectivity of the mesh, avoiding problems such as lost points or registration errors. Simultaneously, the spatial frequency and phase of the orthogonal mesh shift with the deformation of the elastic transparent layer. The processor extracts the phase change by performing a Fourier transform on the deformation image and then reconstructs the three-dimensional morphology of the contact surface based on the mapping relationship between the phase change and height.

[0051] Optionally, the processor is also used to generate contact pressure distribution information based on the density changes of feature nodes in various local regions of the geometric topology coding layer. Specifically, when contact force is applied to the tactile sensing layer, the elastic transparent layer deforms, and the density of feature nodes in local regions changes accordingly. In areas with greater pressure, the elastic transparent layer is compressed more significantly, and the density of feature nodes in that local region is higher. By statistically analyzing the density changes of feature nodes in various local regions, the processor can generate pressure distribution information characterizing the contact pressure distribution.

[0052] Optionally, the processor can also fuse the contact pressure distribution information with the tangential displacement vector field to output the fused tactile information.

[0053] Optionally, the processor can also identify the slippage tendency of the contact surface based on the displacement direction of each feature node in the tangential displacement vector field, and output the slippage tendency information. When the contact object slides, the displacement direction of each feature node in the tangential displacement vector field shows a consistent trend, and the processor can identify the slippage tendency by analyzing the displacement direction of each feature node.

[0054] In this embodiment, an encoding layer with preset geometric topological features is set on the surface of the elastomer. An imaging unit is used to acquire deformation images of the encoding layer under contact force, showing geometric distortion. Frequency domain analysis is then used to reconstruct the three-dimensional morphology of the contact surface based on the phase change or frequency shift of the encoding layer. Since the encoding layer uses geometric topological features as an information carrier, it is insensitive to changes in illumination and has good environmental adaptability. Furthermore, spatial domain analysis can be used to reconstruct the tangential displacement vector field based on the coordinate drift and topological relationship of the feature nodes to obtain the tangential displacement information of the contact surface.

[0055] Example 2 Based on Embodiment 1, another embodiment of this application is proposed: a visual-tactile sensing method, referring to... Figure 5 The method includes: Step S1: Obtain a deformation image, which is generated by the geometric distortion of the geometric topology coding layer under the action of contact force.

[0056] In this embodiment, when a contact force is applied to the tactile sensing layer, the elastic transparent layer deforms, and the geometric topological coding layer attached to it undergoes geometric distortion. The lines of the pattern bend, stretch, or compress, and the positions of the feature nodes shift. The imaging unit acquires an image of the geometric topological coding layer, which is the deformation image.

[0057] As an optional implementation, when the geometric topology coding layer is a bidirectional orthogonal grid, if the contact layer is not subjected to external force, the imaging unit acquires a regular orthogonal grid image without distortion; if the contact layer is subjected to force and deforms, the lines of the orthogonal grid bend, stretch or compress, and the grid intersections drift in coordinates, the imaging unit acquires a deformed image containing this geometric distortion information.

[0058] Step S2: Perform frequency domain analysis on the deformation image, and reconstruct the three-dimensional morphology of the contact surface based on the phase change or frequency shift of the geometric topology coding layer.

[0059] In this embodiment, frequency domain analysis refers to a method of transforming a deformed image from the spatial domain to the frequency domain for processing. The geometric topology coding layer has a preset spatial frequency and phase distribution. When the elastic transparent layer deforms, the spatial frequency and phase of the coding layer shift accordingly. By extracting the phase change through Fourier transform of the deformed image, and then based on the mapping relationship between the phase change and height, the three-dimensional shape of the contact surface can be reconstructed. Frequency domain analysis can achieve sub-pixel-level reconstruction accuracy and is insensitive to ambient light fluctuations.

[0060] As an optional implementation, a Fourier transform is performed on the deformed image to obtain the frequency domain spectrum; the frequency domain spectrum is filtered to extract the carrier frequency component, and an inverse Fourier transform is performed to extract the phase information; the extracted phase information is unwrapped to obtain the absolute phase distribution; and the three-dimensional morphology of the contact surface is reconstructed based on the mapping relationship between the absolute phase distribution and the height.

[0061] Optionally, before acquiring the deformed image, a reference image of the geometric topology coding layer in a non-contact state is also acquired. The reference image is subjected to Fourier transform and phase extraction to obtain reference phase information. In the subsequent frequency domain analysis, the phase information of the deformed image and the reference phase information are differentially calculated to obtain the phase change amount, and then height mapping is performed based on the phase change amount.

[0062] Optionally, when the geometric topological coding layer is a multicolor stripe coding layer, the stripe images of the multicolor stripe coding layer in different color channels are separated, and multi-step phase-shifting processing is performed on the stripe images of each color channel to extract the phase information of each channel. The three-dimensional morphology of the contact surface is reconstructed based on the phase information of each channel. The multi-step phase-shifting method performs phase analysis in the spatial domain, forming an alternative to the Fourier transform method, and can be flexibly selected according to the coding layer type.

[0063] Step S3: Output the three-dimensional shape.

[0064] In this embodiment, the output three-dimensional topography can be used for force feedback and control in scenarios such as robot operation and precision assembly. The output can be in the form of transmitting data to the robot's control system as a data stream, or displaying the three-dimensional topography of the contact surface as a visual image.

[0065] Based on this, the method can further perform spatial domain analysis on the deformed image.

[0066] Step S4: Perform spatial domain analysis on the deformed image, and reconstruct the tangential displacement vector field based on the coordinate drift of the feature nodes of the geometric topology coding layer and the topological relationship between the feature nodes.

[0067] In this embodiment, spatial domain analysis refers to the direct processing and analysis of image pixels or feature points in the spatial domain of the image. Feature nodes are identifiable points with distinct geometric features in the geometric topological coding layer, such as intersections of bidirectional orthogonal grids, corner points or endpoints in a striped pattern, etc. Topological relationships refer to the inherent connection relationships between feature nodes. In orthogonal grids, this manifests as a regular connection structure formed by grid lines connecting adjacent intersections; in striped patterns, it manifests as the spacing and arrangement order between adjacent stripes. By combining coordinate drift and topological relationships for reconstruction, a continuous displacement field covering the entire sensing area can be obtained. Furthermore, the topological relationships remain stable during deformation and can serve as constraints in the matching process, ensuring that each feature node can still be correctly tracked under large deformation conditions and avoiding mismatches.

[0068] As an optional implementation, corner detection is performed on the deformed image to locate the current coordinates of the feature nodes. Corner detection identifies the positions of feature nodes in the geometric topological coding layer. Commonly used corner detection algorithms include the Harris corner detection algorithm, the Shi-Tomasi algorithm, and the FAST corner detection algorithm.

[0069] Optionally, when the geometric topological coding layer is a bidirectional orthogonal grid, after acquiring the deformed image, the deformed image is preprocessed such as grayscale conversion and noise reduction, and then a corner detection algorithm is used to detect all intersections of horizontal and vertical lines in the grid, and the pixel coordinates (x, y) of each intersection in the image coordinate system are output, which are the current coordinates of the feature nodes.

[0070] Optionally, when the geometric topological coding layer is a color stripe or a mixed frequency stripe, the deformation image is first separated into color channels to obtain independent image components corresponding to different color channels. Spatial domain analysis is then performed on the independent image components of each color channel to obtain the displacement information corresponding to each color channel. The displacement information of different color channels is then fused to generate the final tangential displacement vector field. Through color channel separation, deformation information in different directions or at different frequencies can be extracted separately. For example, color stripes can be horizontal red stripes and vertical blue stripes, which can avoid crosstalk between displacement information in different directions. Mixed frequency stripes can be example, high-frequency red stripes and low-frequency green stripes, which can simultaneously accommodate large deformation measurement and detail acquisition.

[0071] As an optional implementation, the current coordinates are matched with reference coordinates based on the topological connections between the feature nodes; the coordinate drift of each feature node is obtained based on the matching result. Since the positions of feature nodes change when the elastic transparency layer deforms, a correspondence between feature nodes before and after the change is established through matching. After matching, the difference between the current coordinates and reference coordinates of the same feature node is the coordinate drift of that node.

[0072] Specifically, the detected feature nodes in the current frame are initially matched with feature nodes in the reference image. Methods such as nearest neighbor matching and feature descriptor matching can be used to initially establish the correspondence between the current node and the reference node. The initial matching results are then constrained and verified using the topological connectivity of an orthogonal grid. For each feature node, its relative positional relationship with its neighboring nodes is checked to ensure consistency with the topological relationship in the reference image. If the matching result of a node causes an anomaly in its relative position with its neighboring nodes, such as disordered node order or abrupt changes in relative distance, the matching result is deemed erroneous, and the correspondence of that node is corrected based on the matching results of its neighboring nodes. Topological constraints effectively prevent mismatches of feature points, especially when large deformations cause severe distortion of the local pattern. The topological relationship ensures that each feature node can still match its correct corresponding node, thus avoiding lost points or registration errors.

[0073] Optionally, when the geometric topological coding layer is a striped pattern, the feature nodes can be the endpoints of the stripes or sampling points on the stripes, and the topological relationship is reflected in the spacing and relative positional relationship between adjacent stripes. During the matching process, the relative positional relationship between the stripes is used to constrain the matching results. For example, adjacent stripes maintain the same arrangement order before and after deformation, and there will be no intersection or order exchange.

[0074] Optionally, the reference coordinates can be obtained before deformation occurs by acquiring a reference image of the geometric topology coding layer in a non-contact state and extracting the coordinates of the feature nodes in the reference image. Specifically, before the sensor is used for the first time or before each use, in the initial state where the tactile sensing layer is not subjected to any contact force, the imaging unit acquires an image of the geometric topology coding layer as a reference image. Corner detection is performed on the reference image to locate the initial position of each feature node in a non-contact state, and its coordinates are extracted and stored as reference coordinates.

[0075] As an optional implementation, the tangential displacement vector field is generated based on the coordinate drift. The coordinate drift of each feature node contains the displacement information of that node in the X and Y directions. Reconstruction based on coordinate drift and topological relationships can, on the one hand, obtain a continuous displacement field covering the entire sensing area; on the other hand, the topological relationships remain stable during deformation and can serve as constraints in the matching process, ensuring that each feature node can still be correctly tracked under large deformation conditions and avoiding mismatches. By arranging the coordinate drift of all feature nodes according to their spatial positions, the tangential displacement vector field covering the entire sensing area can be obtained.

[0076] Specifically, for orthogonal grids, the drift amounts of all intersection points are arranged according to their row and column positions in the grid to form a two-dimensional displacement vector field. For regions without deformation, the displacement vector is approximately zero; for regions with deformation, the magnitude and direction of the displacement vector reflect the distribution of tangential displacement in that region.

[0077] Optionally, the output tangential displacement vector field can be used for force feedback and control in scenarios such as robot operation and precision assembly.

[0078] As an optional implementation, contact pressure distribution information can be generated based on the density changes of feature nodes in local regions of the geometric topology coding layer. When contact force is applied to the tactile sensing layer, the elastic transparent layer deforms, and the density of feature nodes in the local region changes accordingly. The greater the pressure, the more significantly the elastic transparent layer is compressed, and the more feature nodes per unit area in that local region, i.e., the higher the feature node density.

[0079] Specifically, the deformed image is divided into local regions, the number of feature nodes in each local region is counted, the feature node density of each local region is calculated, and compared with the reference density in the non-contact state to obtain the density change of each local region. Since there is a positive correlation between the density change and the contact pressure, pressure distribution information characterizing the contact pressure distribution can be generated based on the density change of each local region.

[0080] Optionally, the contact pressure distribution information can be fused with the tangential displacement vector field and / or three-dimensional topography to output full-dimensional tactile information, thereby achieving multi-dimensional tactile perception of tangential displacement, three-dimensional topography, and pressure distribution. The contact pressure distribution information reflects the magnitude of the normal force, the tangential displacement vector field reflects the effect of the tangential force, and the three-dimensional topography reflects the geometric features of the contact surface. By fusing these three information, the normal force distribution, tangential displacement distribution, and geometric topography of the contact surface can be simultaneously characterized, providing more complete force information for robot grasping control.

[0081] As an optional implementation, multiple consecutive frames of deformation images can be acquired, and time-series analysis can be performed based on the tangential displacement vector field corresponding to each frame to output a tactile data stream characterizing the slippage trend. By performing time-series analysis on the tangential displacement vector field of multiple consecutive frames, the dynamic change trend of contact surface slippage can be obtained, providing richer temporal information for slippage early warning and force feedback control.

[0082] As an optional implementation, after reconstructing the tangential displacement vector field, the slippage tendency of the contact surface is identified based on the displacement direction and magnitude of each feature node in the tangential displacement vector field, and slippage tendency information is output. By statistically analyzing the displacement direction distribution of each feature node, it is determined whether a consistent displacement direction trend exists; simultaneously, the displacement magnitude of each feature node is statistically analyzed to determine whether it exceeds a preset displacement threshold. When a consistent displacement direction trend exists and the average displacement magnitude exceeds the preset threshold, a slippage tendency is determined to exist. Specifically, principal component analysis can be used to calculate the principal direction of the displacement vector field. When the proportion of energy in the principal direction to the total energy exceeds a preset direction consistency threshold, and the average displacement magnitude of each feature node exceeds the preset displacement threshold, a slippage tendency is determined to exist. The output slippage tendency information can be used for early warning or feedback control.

[0083] Step S5: Output the tangential displacement vector field and / or the three-dimensional shape.

[0084] In this embodiment, depending on the actual application requirements, only the tangential displacement vector field, only the three-dimensional shape, or both can be output. The output can be in the form of transmitting the data to the robot's control system as a data stream, or displaying the displacement distribution and three-dimensional shape of the contact surface as a visual image.

[0085] In this embodiment, by performing frequency domain analysis on the deformation image and reconstructing the 3D topography based on the phase changes or frequency shifts of the coding layer, and using geometric phase as the information carrier, the reconstruction process of the 3D topography is independent of illumination intensity and surface reflectivity, and can still stably reconstruct the 3D topography under conditions of light source fluctuations or ambient light changes. Based on this, further spatial domain analysis can be performed to reconstruct the tangential displacement vector field based on the coordinate drift and topological relationships of feature nodes to obtain the tangential displacement information of the contact surface. Frequency domain reconstruction does not require multi-frame image sequences or complex temporal registration, has high computational efficiency, and can meet the needs of real-time robot control; spatial domain tracking uses topological relationships as matching constraints, and can still achieve accurate feature matching even when large deformations cause severe distortion of local patterns. Simultaneously, since this method uses geometric topological features as the carrier of tactile information, it is insensitive to illumination changes and can achieve stable sensing performance without a sophisticated light source control system. Based on obtaining the tangential displacement vector field and 3D topography, pressure distribution sensing and slip tendency recognition functions can also be simultaneously extended, realizing an effective expansion from a single physical quantity to multi-dimensional tactile information.

[0086] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, third, etc., does not indicate any order. These words can be interpreted as names.

[0091] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0092] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of the invention. Therefore, if these modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A visual-tactile sensing method, characterized in that, The method includes: A deformation image is acquired, which is generated by the geometric distortion of the geometric topology coding layer under the action of contact force; Frequency domain analysis is performed on the deformation image, and the three-dimensional morphology of the contact surface is reconstructed based on the phase change or frequency shift of the geometric topology coding layer. Output the three-dimensional shape.

2. The method as described in claim 1, characterized in that, The step of reconstructing the three-dimensional morphology of the contact surface based on the phase change or frequency shift of the geometric topology coding layer includes: Perform a Fourier transform on the deformed image to obtain the frequency domain spectrum; The frequency domain spectrum is filtered and then subjected to inverse Fourier transform to extract phase information. The phase information is unwrapped to obtain the absolute phase distribution; The three-dimensional topography is reconstructed based on the mapping relationship between the absolute phase distribution and the height.

3. The method as described in claim 2, characterized in that, The geometric topology coding layer is a multicolor stripe coding layer, and the step of reconstructing the three-dimensional shape includes: Separate the stripe images of the multicolor stripe coding layer in different color channels; Multi-step phase-shifting processing is performed on the stripe images of each color channel to extract the phase information of each channel; The three-dimensional topography is reconstructed based on the phase information of each channel.

4. The method as described in claim 1, characterized in that, Prior to the step of acquiring the deformed image, the following steps are included: Obtain a reference image of the geometric topology coding layer in a non-contact state; The coordinates of feature nodes in the reference image are extracted as reference coordinates, which are used to match the current coordinates to obtain the coordinate drift amount; Furthermore, Fourier transform and phase extraction are performed on the reference image to obtain reference phase information, which is used to perform differential calculation with the phase information of the deformed image to obtain the phase change amount, and the phase change amount is used to reconstruct the three-dimensional shape.

5. The method as described in claim 1, characterized in that, The method further includes: Based on the output of the three-dimensional topography, further spatial domain analysis is performed on the deformation image, and the tangential displacement vector field is reconstructed according to the coordinate drift of the feature nodes of the geometric topology coding layer and the topological relationship between the feature nodes; Output the three-dimensional topography and the tangential displacement vector field.

6. The method as described in claim 5, characterized in that, The step of reconstructing the tangential displacement vector field based on the coordinate drift of the feature nodes in the geometric topology coding layer and the topological relationship between the feature nodes includes: Corner detection is performed on the deformed image to locate the current coordinates of the feature nodes; Based on the topological connection relationship between the feature nodes, the current coordinates are matched with the reference coordinates; Based on the matching results, the coordinate drift of each feature node is obtained; The tangential displacement vector field is generated based on the coordinate drift.

7. The method as described in claim 5, characterized in that, The geometric topology coding layer is a color stripe or mixed frequency stripe coding layer, and the step of reconstructing the tangential displacement vector field includes: The deformed image is subjected to color channel separation to obtain independent image components corresponding to different color channels; Spatial domain analysis was performed on the independent image components of each color channel to obtain the displacement information corresponding to each color channel; The displacement information from different color channels is fused to generate the final tangential displacement vector field.

8. The method as described in claim 5, characterized in that, The method further includes: Contact pressure distribution information is generated based on the density changes of feature nodes in each local region of the geometric topology coding layer. The contact pressure distribution information is fused with the tangential displacement vector field and / or the three-dimensional morphology to output full-dimensional tactile information.

9. A visual-tactile sensor, characterized in that, The visual-tactile sensor includes: Imaging unit; The tactile sensing layer includes an elastic transparent layer and a geometric topology coding layer disposed on the elastic transparent layer. The imaging unit is disposed toward the tactile sensing layer and is used to acquire the deformation image generated by the geometric topology coding layer under the action of contact force. A light source is positioned around the imaging unit to illuminate the geometric topology coding layer; A processor for performing the visual-tactile sensing method as described in any one of claims 1-8.

10. The visual-tactile sensor as described in claim 9, characterized in that, The geometric topology coding layer is a bidirectional orthogonal grid, which has different spatial frequencies or periods in the two orthogonal directions to independently modulate the tangential displacements in the X-axis and Y-axis directions, respectively.