An irregular curved surface visual-tactile sensor and control unit

By designing an irregular curved surface visual-tactile sensor, and combining it with a light-shielding contact layer, an elastic layer, a light-guiding layer, and an image sensing module, three-dimensional point cloud reconstruction of irregular curved surfaces was achieved. This solves the shortcomings of existing irregular curved surface visual-tactile sensors and improves the accuracy of contact position recognition and deformation depth estimation.

CN123114301APending Publication Date: 2026-11-17HUIGUANG INNOVATION (BEIJING) TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611302695.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-26
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing visual-tactile sensors lack effective solutions for applications involving irregular curved surfaces, and cannot effectively reconstruct the 3D point cloud of contact with irregular curved surfaces, resulting in inaccurate contact position recognition and deformation depth estimation.

Method used

An irregular curved surface visual-tactile sensor was designed, including a shell, a light-shielding contact layer, an elastic layer, a light-guiding layer, an illumination diffusion module, and an image sensing module. Combined with a 3D point cloud construction module, it realizes 3D position construction through pre-stored reference data and real-time grayscale images, adapting to the 3D point cloud reconstruction of contact with irregular curved surfaces.

Benefits of technology

It improves the accuracy of 3D point cloud reconstruction of irregular curved surfaces, meets the usage requirements of irregular curved surface application scenarios such as dexterous fingertips, and realizes the effective application of irregular curved surface visual and tactile sensors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114301A_ABST
    Figure CN123114301A_ABST
Patent Text Reader

Abstract

This application provides an irregular curved surface visual-tactile sensor and control unit. The visual-tactile sensor includes a housing, a light-shielding contact layer, an elastic layer, a light-guiding layer, an illumination diffusion module, an image sensing module, and a 3D point cloud construction module. The housing has a mechanical interface for connection with control components. The surface of the light-shielding contact layer is an irregular curved surface. The light-shielding contact layer, elastic layer, and light-guiding layer are arranged sequentially from the contact surface inwards. The illumination diffusion module provides a light source to the area within the light-shielding contact layer. The image acquisition area of ​​the image sensing module faces the light-guiding layer. The 3D point cloud construction module constructs the 3D position of each pixel in the third real-time grayscale image based on pre-stored reference data and the third real-time grayscale image acquired by the image sensing module. This application provides an irregular curved surface visual-tactile sensor solution for applications such as dexterous fingertips.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of visual-tactile sensor technology, and in particular to an irregular curved surface visual-tactile sensor and control unit. Background Technology

[0002] In tasks such as grasping, assembling, manipulating, twisting, and slip suppression, dexterous hands require fingertips to simultaneously possess the capabilities of compliant contact, surface conformation, contact position recognition, and deformation depth estimation. Visuotactic sensors, which observe image changes in flexible contact layers through cameras, can acquire dense tactile information within a small volume, making them suitable for installation at the fingertips of robots.

[0003] Existing visual-tactile solutions are mostly based on planar elastic bodies, regular spherical caps, or regular circular arc surfaces. Their image inversion typically relies on planar calibration, dot matrix marker displacement, and stereo or fixed geometric assumptions of color light source photometric properties. When a visual-tactile sensor is designed to fit the irregular curved surface of a dexterous fingertip, the thickness, normal, illumination incident angle, camera viewing angle, and initial grayscale are all different at various points on the surface. The same indentation depth may manifest different grayscale responses at different pixel locations. If the reconstruction logic of planar visual-tactile sensors is directly applied, it is easy to mix together the uneven grayscale of the initial curved surface, the differences in material thickness, and the actual contact deformation.

[0004] Therefore, existing technologies lack visual-tactile sensor solutions for applications with irregular curved surfaces. Summary of the Invention

[0005] In view of this, embodiments of this application provide an irregular curved surface visual-tactile sensor and control unit to solve the technical problem of the lack of visual-tactile sensor solutions for irregular curved surface application scenarios in the prior art.

[0006] To address the aforementioned technical issues, this solution offers the following alternatives:

[0007] The first aspect of this application provides an irregular curved surface visual-tactile sensor, including a housing, a light-shielding contact layer, an elastic layer, a light-guiding layer, an illumination diffusion module, an image sensing module, and a three-dimensional point cloud construction module, wherein the housing is provided with a mechanical interface for connecting to control components. The surface of the light-shielding contact layer is a contact surface and the contact surface is an irregular curved surface. The light-shielding contact layer, the elastic layer and the light-guiding layer are arranged sequentially along the contact surface to the inside. The light-shielding contact layer is used to isolate light entering the elastic layer. The illumination diffusion module is used to provide a light source to the area within the light-shielding contact layer, and the image acquisition area of ​​the image sensing module faces the light guide layer. The 3D point cloud construction module is connected to the image perception module and the illumination diffusion module respectively. The 3D point cloud construction module is used to construct the 3D position of each pixel in the third real-time grayscale image based on the pre-stored reference data and the third real-time grayscale image output by the image perception module. The reference data includes a reference grayscale image, a reference surface depth map, a coordinate ray map, and the position and depth coefficient of the preset point in the image perception module in a preset coordinate system.

[0008] A second aspect of this application provides a control unit, including a control component and a visual-tactile sensor connected to the control component, wherein the visual-tactile sensor is an irregular curved surface visual-tactile sensor as described in the first aspect.

[0009] Compared with the prior art, the beneficial effects that at least one technical solution adopted in the embodiments of this specification can achieve include at least: This application proposes an overall architecture for an irregular curved surface visual-tactile sensor. Furthermore, when the contact surface is an irregular curved surface, a 3D point cloud construction module is proposed. Based on a pre-stored reference grayscale image, a reference surface depth map, a coordinate ray map, and the position and depth coefficients of preset points in the image perception module within a preset coordinate system, this module effectively constructs the 3D position of each pixel in a third real-time grayscale image. This achieves the reconstruction of 3D point clouds from irregular curved surface contact, effectively improving the accuracy of 3D point cloud construction. It provides an irregular curved surface visual-tactile sensor solution for applications such as dexterous fingertips, which is beneficial for meeting the usage requirements of such applications. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the structure of an irregular curved surface visual-tactile sensor provided in an embodiment of this application.

[0012] Figure reference numerals: 1. Outer shell; 2. Light-shielding contact layer; 3. Elastic layer; 4. Light guide layer; 5. Light diffusion module; 6. Image sensing module; 11. Mechanical interface. Detailed Implementation

[0013] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0014] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] The first aspect of this application discloses an irregular curved surface visual-tactile sensor, wherein, as Figure 1 As shown, the irregular curved surface visual-tactile sensor includes a housing 1, a light-shielding contact layer 2, an elastic layer 3, a light-guiding layer 4, an illumination diffusion module 5, an image perception module 6, and a three-dimensional point cloud construction module.

[0016] The irregular curved surface visual-tactile sensor can be a dexterous finger, toe, or other irregular curved surface visual-tactile unit.

[0017] The outer shell can be a 3D-printed shell, an injection-molded shell, or a metal / resin skeleton. The shell provides an assembly reference for the light-shielding contact layer, elastic layer, light-guiding layer, illumination diffusion module, and image sensing module. An installation window is provided on the shell, which also provides a mechanical interface 11 for connection to the control components and defines the relative positions of the image sensing module 6, illumination diffusion module 5, and light-guiding layer 4 to form a sensor coordinate system used for reference surface depth maps and pixel mapping during the reconstruction of 3D point clouds from irregular curved surfaces. The control components can be a robot limb skeleton, such as a finger skeleton or toe skeleton.

[0018] The light-shielding contact layer 2 is a black flexible layer used to isolate light entering the elastic layer 3 and serve as a contact interface for external objects. It can be a black flexible coating, a thin layer of black silicone, a sprayed light-shielding layer, or an integrally molded black surface, ensuring that grayscale variations primarily originate from the response of internal lighting modulated by elastic deformation. The surface of the light-shielding contact layer is a contact surface, and this contact surface is an irregular curved surface. The light-shielding contact layer 2 can be achieved through spraying, dip coating, thin film lamination, black silicone casting, or integral molding with the elastic layer.

[0019] Specifically, when the irregular curved surface visual-tactile sensor is a dexterous fingertip visual-tactile sensor, the light-shielding contact layer 2 is used to simulate the structure of the human fingertip, with the irregular curved surface fingertip contact surface as the sensing interface. The irregular curved surface fingertip contact surface is any one of the following: a free surface, an eccentric spherical cap surface, and a multi-curvature transition surface that fits the human fingertip or dexterous fingertip. Alternatively, the irregular curved surface of the contact surface is a composite surface formed by the continuous transition of the fingertip surface, the lateral surface, and the fingertip surface. The elastic layer 3 covers the outside of the light guide layer 4 and forms the transition surface between the fingertip and the fingertip.

[0020] Elastic layer 3 covers the outer side of the light guide layer. To improve the sensitivity and mechanical range of grayscale response adjustment, elastic layer 3 can be made of scattering microparticles, milky white pigments, and flexible fillers. The elastic layer can also be a multi-layer structure. The thickness of elastic layer 3 is greater than that of light-shielding contact layer 2. Generally, the thickness of elastic layer 3 is 1.5-2.5 mm, and the thickness of light-shielding contact layer 2 is 0.1-0.4 mm. Light-shielding contact layer 2 needs to maintain sufficient flexibility to prevent it from becoming the main load-bearing layer. Elastic layer 3 can be a semi-transparent elastic layer; specifically, the transparency of the elastic layer can be controlled between 40% and 60%. Elastic layer 3 is used to generate localized compression, indentation, shearing, and curvature changes under stress, so as to convert contact deformation into observable grayscale attenuation, enhancement, gradient, or texture changes.

[0021] The light guide layer 4 is fixed to the mounting windows around the perimeter of the housing. The equivalent stiffness of the light guide layer 4 is greater than that of the elastic layer 3, causing the main deformation to occur in the outer contact area under stress. The light guide layer 4 is transparent and can be made of transparent acrylic, transparent resin, glass, polycarbonate, or high-hardness transparent silicone. The light guide layer is used to provide the inner observation boundary, guide illumination light, stabilize the surface geometry, reduce non-target deformation on the inner side, and make the grayscale-depth mapping more stable. Specifically, in one embodiment, the hardness of the light guide layer is Shore A 45A; the hardness of the elastic layer is Shore A 00 30.

[0022] Specifically, the light-shielding contact layer 2, the elastic layer 3, and the light-guiding layer 4 are arranged sequentially from the contact surface inwards. In particular, when the irregular curved surface visual-tactile sensor is a dexterous finger, the light-shielding contact layer 2, the elastic layer 3, and the light-guiding layer 4 are arranged sequentially from the fingertip to the nail. The elastic layer 3 and the light-guiding layer 4 can be fixed to the outer shell 1 using silicone adhesive and screws with positioning holes.

[0023] The lighting diffusion module 5 is mounted along the side wall of the housing 1. The lighting diffusion module can be a side-lit LED, a ring LED, a multi-zone LED, or a monochromatic / multi-color light source homogenized by the diffusion layer. The lighting diffusion module 5 is used to provide stable internal lighting to the elastic layer.

[0024] The image sensing module 6 is housed within the outer casing 1, with the image acquisition area facing the guide light layer. The image sensing module 6 is a camera or video camera, including a miniature camera (containing a wide-angle lens and an image sensor), and is fixed to the outer casing via a mounting bracket.

[0025] The 3D point cloud construction module is signal-connected to both the image perception module 6 and the illumination diffusion module 5. While controlling illumination, it constructs the 3D positions of each pixel in the third real-time grayscale image based on pre-stored reference data and the third real-time grayscale image output by the image perception module 6, thus reconstructing the 3D point cloud of an irregular curved surface. The reference data includes a reference grayscale image, a reference surface depth map, a coordinate ray map, and the positions and depth coefficients of preset points in the image perception module within a preset coordinate system.

[0026] In practice, before the irregular curved surface visual-tactile sensor performs 3D point cloud construction, relevant reference data can be calibrated using the following methods: Specifically, the reference data includes a reference grayscale image, a reference surface depth map, a coordinate ray map, and the position and depth coefficients of preset points in the image perception module within a preset coordinate system. The preset coordinate system can be the system coordinate system of the control unit. The preset points in the image perception module can be the center of the image perception module or other fixed points. A hand has multiple fingertip surface models, each finger corresponding to a different fingertip surface model, and different fingertip surface models correspond to different reference data.

[0027] In specific implementation, the calibration of parameters such as the position of preset points in the reference surface depth map, coordinate ray map, and image perception module in the preset coordinate system is as follows: For each fingertip surface model, a non-contact surface depth map is first rendered based on the CAD model, 3D scan model, or camera calibration results. This map is then used as a reference surface depth map. The pixel coordinate rays corresponding to each pixel and the positions of preset points in the image perception module within a preset coordinate system are calculated. Pixel coordinate rays corresponding to all pixels are then calculated. This constitutes a coordinate ray diagram. Specifically, the pixel coordinate ray corresponding to each pixel represents the unit direction from the preset point in the image perception module, passing through the pixel and pointing to the finger surface point, which is used to convert one-dimensional depth changes into three-dimensional coordinate changes.

[0028] In one specific implementation, the pixel coordinate ray corresponding to each pixel is determined by the following formula: , In the formula, This represents the normalized pixel coordinate ray of pixel point (u,v) in the coordinate system of the image perception module. For the transpose of a matrix or vector, These are intrinsic parameters of the image perception module. Let be the magnitude of the vector, where represents the Euclidean distance of the reprojection error, i.e., the square of the L2 norm. , These are the x and y coordinates of the pixel in the coordinate system of the image perception module, respectively.

[0029] In one specific embodiment, the reference surface depth corresponding to each pixel is determined by the following formula to form a reference surface depth map, wherein,

[0030]

[0031]

[0032] In the formula, This represents the normalized pixel coordinate ray of pixel point (u,v) in the basic coordinate system of the curved visual-tactile sensor. This is the rotation matrix from the coordinate system of the image perception module to the basic coordinate system of the curved visual-tactile sensor. Here are the coordinates of the optical center of the image perception module in the basic coordinate system of the curved visual-tactile sensor. The surface of the curved visual-tactile sensor in a non-contact state. For Starting from the pixel coordinate ray The nearest intersection with the surface to The distance is the depth of the reference surface corresponding to the pixel; min is the minimum value; t represents the distance of any point on the pixel coordinate ray relative to the starting point. The displacement is a ray parameter in non-negative real number form.

[0033] In a specific example of this application, the position of the preset point in the image perception module in the preset coordinate system, i.e. the position of the center point of the image perception module in the system coordinate system, is used to determine the spatial origin of the point cloud projection.

[0034] Furthermore, for the reference surface depth map, it provides the basic position of each pixel on the non-contact surface to form the reference surface depth map. When the reference surface depth map has invalid values, non-finite values ​​or abnormal intersection of model edges, it can be replaced with the statistical depth of the effective region. The statistical depth of the effective region can be the median depth of the effective region to avoid holes or abnormal jumps in the real-time point cloud.

[0035] Specifically, by referencing the surface depth map, coordinate ray map, and the position of the center point of the image perception module in the system coordinate system, each pixel has an independent spatial reference, making the initial curvature an explicit parameter. This ensures that the initial shape of irregular surfaces is no longer determined by the real-time grayscale model, avoiding the misinterpretation of irregular surface parts such as fingertips and finger pad sides as contact deformation in irregular surface tactile sensors.

[0036] For the reference grayscale image: During sensor operation, factors such as light source brightness, temperature, camera exposure, light-shielding layer status, and material rebound can cause non-contact grayscale baseline drift. Therefore, this application proposes, after confirming the non-contact state, using a multi-frame median reference image to describe the grayscale baseline under the current individual, current light source, and current exposure state. It supports online updates of the reference grayscale benchmark without regenerating surface geometry parameters. This separates the static grayscale background formed by differences in illumination, exposure, and material transmittance from the dynamic grayscale changes caused by contact, reducing the impact of drift on depth mapping. For example, after the entire system is assembled, the fingertip contact surface is kept in a non-contact state (i.e., without external force), and the light source brightness, camera exposure, gain, and white balance are kept constant. After warm-up and stabilization, multiple frames of grayscale images are acquired, and the pixel-wise median image, rather than a single frame image, is used as the reference grayscale image to reduce the impact of serial port transmission jitter, exposure fluctuations, and occasional noise.

[0037] For example, the image perception module outputs a single-channel grayscale image of 320×240 pixels. The image perception module can output compressed image frames via a serial port or an equivalent high-speed link. After the 3D point cloud construction module decodes the image frame, if the actual size is inconsistent with the target size, it will resample to a fixed size to ensure that the image resolution of each grayscale image frame is consistent.

[0038] Specifically, the image perception module acquires a reference grayscale image through the following steps.

[0039] Step S11: In response to receiving the calibration command, after a first preset number of sampling cycles of the image sensing modules, acquire multiple frames of the first real-time grayscale image collected by the image sensing modules in a contactless state.

[0040] The calibration command can be passively triggered via the recalibration interface during power-on or automatically triggered during system startup. Upon receiving the calibration command, a first preset number of warm-up frames are discarded. Here, the first preset number can be set to any value between 5 and 10, for example, it can be set to 8.

[0041] Step S12: Calculate the average gray level difference between two adjacent frames of the first real-time gray level image in the multi-frame first real-time gray level image.

[0042] Here, the number of frames of the first real-time grayscale image can be any value from 3 to 10.

[0043] Step S13: In response to the fact that the average grayscale difference between two adjacent frames of the first real-time grayscale image in the multi-frame first real-time grayscale image is less than the preset value or the image sensing module has reached the maximum number of warm-up frames after receiving the calibration command, acquire the multi-frame second real-time grayscale image collected by the image sensing module in the non-contact state.

[0044] The default value here can be set to any value between 1.0 and 4.0, for example, it can be set to 2.0.

[0045] When the average grayscale difference between two adjacent frames of the first real-time grayscale image in multiple consecutive frames is less than the preset value, or when the maximum number of warm-up frames is reached, real-time grayscale images can continue to be acquired to reconstruct the reference grayscale image, thereby improving the accuracy of the reference grayscale image.

[0046] The more real-time grayscale images there are, the higher the accuracy of the obtained reference grayscale image. However, a larger number of images will also lead to time consumption. Therefore, it is preferable that the number of frames of the second real-time grayscale image is 8 to 15 frames.

[0047] Step S14: Calculate the median or average value of each pixel in the multiple frames of the second real-time grayscale image to obtain the reference grayscale image. .

[0048] The reference grayscale image is determined based on the median. The specific method is as follows: , In the formula, To take the median, This is the second real-time grayscale image of the i-th frame. The total number of the second real-time grayscale images. .

[0049] By using the above method, warm-up and stability assessments are performed during the startup phase. After stabilization, a reference grayscale image is determined, which effectively improves the accuracy of the reference grayscale image.

[0050] In practice, the reference surface depth map, coordinate ray map, and reference grayscale image have the same image resolution, so that the corresponding depth, grayscale, and pixel coordinate ray can be found in each map by using the index of the same pixel point.

[0051] In practice, when changes occur in light source brightness, temperature, light-shielding layer status, or camera exposure, recalibration can update the reference grayscale image. This avoids mistaking baseline drift for contact deformation.

[0052] In practical implementation, the calibration of the depth coefficient is as follows: Standard indenters with known geometries, such as ball-head, cylindrical-head, flat-head, or fingertip-like indenters, can be used to press on different areas, directions, and depths of the fingertip's curved surface. Each press records the current grayscale image, indenter pose, indentation amount, and contact area. Pixel-level true deformation depth is calculated using the standard indenter geometry and a reference depth map. Then, the depth coefficient is obtained by fitting the relationship between the gray-level difference and the deformation depth.

[0053] In practice, after the irregular curved surface visual-tactile sensor is activated, it can load data such as surface geometric parameters and depth coefficients, and the image perception module can start to collect and output grayscale image data to perform the reconstruction process of the irregular curved surface contacting the three-dimensional point cloud.

[0054] Specifically, during the process of calculating the grayscale difference image based on the reference grayscale image and the third real-time grayscale image, the finger enters the contact state, and the image perception module acquires the third real-time grayscale image in real time. The difference calculation unit subtracts the grayscale value of each pixel in the reference grayscale image from the grayscale value of the corresponding pixel in the third real-time grayscale image and assigns a negative value to 0, thus obtaining the grayscale difference image. .

[0055] This grayscale difference image represents the grayscale response caused by contact indentation, local displacement of the light-shielding contact layer, and changes in the optical path of the elastic layer. Specifically, the grayscale value of each pixel in the grayscale difference image... for: , In the formula, clip is a value between 0 and 255. Values ​​exceeding 255 are truncated to 255, and values ​​below 0 are truncated to 0. This is the third real-time grayscale image.

[0056] By independently establishing a non-contact grayscale reference, i.e., a grayscale difference map, it is possible to interpret only the effective grayscale changes relative to the reference in real time.

[0057] In practice, during the process of calculating the local deformation depth map based on the grayscale difference map and the depth coefficient, the deformation depth calculation unit calculates the grayscale value of each pixel in the grayscale difference map. Through depth coefficient Convert to local deformation depth The depth coefficient is the depth change corresponding to a unit change in gray level.

[0058] In some low-computing-power or real-time display scenarios, the following linear model can be preferentially used to obtain the local deformation depth map. For example, the gray value of each pixel in the gray-level difference map is multiplied by the corresponding depth coefficient to obtain the local deformation depth of each pixel in the local deformation depth map. Specifically: .

[0059] Wherein, the depth coefficient k is a calibrable coefficient or an adjustable coefficient.

[0060] In practice, after obtaining the local deformation depth map, the 3D point cloud can be reconstructed based on the 3D point cloud construction unit for contact with irregular curved surfaces. For example, the real-time depth calculation unit determines the real-time depth of each pixel using the following formula. To generate a real-time depth map: , In the formula, The reference surface depth corresponding to each pixel; To mark the display scene, symbols can be set for display scenes where the contact pressing direction and the display direction are different. The distinction is made to switch between different display scenarios such as recessed display and raised display, so as to obtain the surface geometric parameters corresponding to each pixel in the current display scenario.

[0061] In practice, the 3D point cloud construction unit calculates the position of each pixel in the preset coordinate system using the following formula. It can update the dense point cloud reconstruction at the contact point of irregular curved surfaces in real time: , In the formula, This refers to the position of a preset point in the image perception module within a preset coordinate system. The pixel coordinate ray corresponding to each pixel point.

[0062] Furthermore, the position of each pixel in the third real-time grayscale image in a preset coordinate system can be used as a reference. and deformation depth Calculate the contact center The contact center is the equivalent position center of the irregular curved surface visual tactile sensor under contact state, and this equivalent position center can be used as the equivalent force center.

[0063] Specifically, contact center for: , In the formula, This is the preset depth.

[0064] The reconstruction of the 3D point cloud of the irregular surface contact is achieved by representing the contact deformation as a depth perturbation along the pixel coordinate rays, rather than simply superimposing it on a uniform plane height map. This method allows regions on the irregular surface with different normals, depths, and viewing angles to be back-projected into 3D point clouds using the same formula.

[0065] In practical implementation, each pixel can be colored according to its local deformation depth in the local deformation depth map to form a contact heat map. The maximum contact depth can be recorded. , as an indicator of contact strength.

[0066] Furthermore, the relative force vector can be determined based on the deformation depth map. Specifically, for pixels in the local deformation depth map where the local deformation depth is greater than a preset depth, the opposite direction of the pixel coordinate ray corresponding to that pixel is taken as the contact contribution direction, and the relative force vector is obtained by accumulating the values ​​with the deformation depth as the weight. : , In the formula, For display or control purposes, a proportional coefficient is used. This represents the local deformation depth of this pixel in the local deformation depth map. For preset depth, The pixel coordinate ray corresponding to each pixel point.

[0067] In one specific implementation, It can be set to 0.02 mm, a proportional coefficient for display or control. It can be set to 0.0024.

[0068] This relative force vector does not necessarily have to be directly equivalent to the absolute Newtonian force, but it can be used to display the contact direction, determine the grasping stability trend, or serve as input for a further force calibration model, which is used to convert image information / 3D reconstruction information into force information.

[0069] Furthermore, this irregular curved surface visual-tactile sensor may also include a processing and display module for updating the point cloud and displaying it in 3D. Specifically, the processing and display module can be implemented using a web service. The backend loads the geometry array and maintains its operating status, including the reference grayscale image, the latest grayscale difference image, serial port connection status, frame number, error messages, and recalibration events. The frontend obtains the grayscale difference image, reference depth map, and pixel ray array through a binary interface, and completes the point cloud update and 3D display in the browser. The relevant interfaces or parameters of the processing and display module are shown in Table 1 below.

[0070] Table 1

[0071] In Table 1, the two sets of geometric models involved in the model switching are the standard finger model and the thumb model. The standard finger model is a unified surface geometric model shared by the index, middle, ring, and little fingers in a dexterous hand, while the thumb model is a surface geometric model specifically for the thumb.

[0072] In practical implementation, the aforementioned irregular curved surface visual-tactile sensor has the following beneficial effects: Adapting to irregular curved surfaces for nimble fingertips: by referencing surface depth Pixel coordinate rays The position of preset points in the image perception module in the preset coordinate system It expresses the three-dimensional geometric relationship of each pixel, avoiding treating the initial curved surface of the fingertip as a plane.

[0073] No need for regular dot matrix markings: Using continuous grayscale differences as the reconstruction signal avoids dot matrix projection distortion, occlusion, and matching drift problems in areas with strong curvature.

[0074] Good real-time performance: The basic implementation only requires grayscale difference, linear depth mapping and ray projection along pixel coordinates, which is suitable for real-time point cloud display and online control.

[0075] The model has strong scalability: for non-regular curved surface visual tactile sensor applications such as standard fingers, thumbs or other fingertip shapes, different pre-calculated geometric files, i.e. files that record the above reference data, can be connected to the same grayscale reconstruction framework.

[0076] Better individual consistency: Through multi-frame median reference grayscale images and recalibration interfaces, it can compensate for light source brightness, temperature, camera status, and material baseline drift.

[0077] In a specific implementation, the second aspect of this application provides a control unit including a control component and a visual-tactile sensor connected to the control component, wherein the visual-tactile sensor is an irregular curved surface visual-tactile sensor as described in any of the first aspects.

[0078] The control unit can be a robot, robotic arm, or other control unit.

[0079] When the control unit is a robot, its irregular curved surface visual-tactile sensor can be connected to the robot's finger skeleton and / or toe skeleton through mechanical interface 11.

[0080] When the control unit is a robotic arm, it can be connected to the end of the robotic arm via mechanical interface 11.

[0081] For example, the aforementioned fingertip visual-tactile sensors are simultaneously installed on the standard fingers and thumb of the finger skeleton. In this case, each fingertip visual-tactile sensor constructs a 3D point cloud based on its corresponding reference data and a third real-time grayscale image.

[0082] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An irregular curved surface visual-tactile sensor, characterized in that, It includes a shell (1), a light-shielding contact layer (2), an elastic layer (3), a light-guiding layer (4), an illumination diffusion module (5), an image perception module (6), and a three-dimensional point cloud construction module, wherein the shell (1) is provided with a mechanical interface (11) for connecting with the control components. The surface of the light-shielding contact layer (2) is a contact surface and the contact surface is an irregular curved surface. The light-shielding contact layer (2), the elastic layer (3) and the light-guiding layer (4) are arranged sequentially along the contact surface to the inside. The light-shielding contact layer (2) is used to isolate the light entering the elastic layer (3). The illumination diffusion module (5) is used to provide a light source to the area within the light-shielding contact layer (2), and the image acquisition area of ​​the image sensing module (6) faces the light guide layer (4). The three-dimensional point cloud construction module is connected to the image perception module (6) and the illumination diffusion module (5) respectively. The three-dimensional point cloud construction module is used to construct the three-dimensional position of each pixel in the third real-time grayscale image based on the pre-stored reference data and the third real-time grayscale image output by the image perception module (6), so as to realize the reconstruction of the three-dimensional point cloud of the irregular curved surface contact. The reference data includes the reference grayscale image, the reference curved surface depth map, the coordinate ray map, and the position and depth coefficient of the preset point in the image perception module in the preset coordinate system.

2. The irregular curved surface visual-tactile sensor according to claim 1, characterized in that, The thickness of the elastic layer (3) is greater than the thickness of the light-shielding contact layer (2).

3. The irregular curved surface visual-tactile sensor according to claim 1, characterized in that, The equivalent stiffness of the light guide layer (4) is greater than that of the elastic layer (3), and the elastic layer (3) is a semi-transparent elastic layer.

4. An irregular curved surface visual-tactile sensor according to any one of claims 1 to 3, characterized in that, The 3D point cloud construction module includes: The differential calculation unit is used to calculate a grayscale difference map based on the reference grayscale image and the third real-time grayscale image. The third real-time grayscale image is a grayscale image output by the image perception module when the irregular curved surface visual-tactile sensor is in contact. The third real-time grayscale image and the reference grayscale image have the same size. The reference grayscale image is the grayscale image corresponding to the irregular curved surface visual-tactile sensor in non-contact state. The grayscale value of each pixel in the grayscale difference map is the difference between the grayscale value of the corresponding pixel in the reference grayscale image and the grayscale value of the corresponding pixel in the third real-time grayscale image. The deformation depth calculation unit is used to calculate a local deformation depth map based on the grayscale difference map and the depth coefficient, wherein the depth coefficient is the depth change corresponding to a unit grayscale change. The real-time depth calculation unit is used to determine the real-time depth of each pixel based on the local deformation depth of each pixel in the local deformation depth map and the reference surface depth of the pixel in the reference surface depth map corresponding to the display scene. The 3D point cloud construction unit is used to calculate the position of each pixel in the preset coordinate system based on the real-time depth of each pixel, the pixel coordinate ray in the coordinate ray map corresponding to the display scene, and the position of the preset point in the image perception module in the preset coordinate system. In this case, each pixel corresponds to different reference surface depth maps, coordinate ray maps, and the positions of the preset points in the image perception module in the preset coordinate system under different display scenes.

5. The irregular curved surface visual-tactile sensor according to claim 4, characterized in that, The three-dimensional point cloud construction module is also used to, in response to receiving a calibration instruction, acquire multiple frames of first real-time grayscale images collected by the image sensing module in a non-contact state after a first preset number of sampling cycles of the image sensing module before the grayscale difference map is calculated by the difference calculation unit. Calculate the average grayscale difference between two adjacent frames of the first real-time grayscale image in a multi-frame first real-time grayscale image; In response to the fact that the average grayscale difference between two adjacent frames of the first real-time grayscale image in multiple frames is less than a preset value or the image sensing module has reached the maximum number of warm-up frames after receiving the calibration command, the image sensing module acquires multiple frames of the second real-time grayscale image in the non-contact state. The reference grayscale image is obtained by calculating the median or average value of each pixel in the multi-frame second real-time grayscale image.

6. The irregular curved surface visual-tactile sensor according to claim 4, characterized in that, The difference calculation unit is used to subtract the gray value of each pixel in the reference grayscale image from the gray value of the corresponding pixel in the third real-time grayscale image and assign the negative value to 0 to obtain the grayscale difference image.

7. The irregular curved surface visual-tactile sensor according to claim 4, characterized in that, The deformation depth calculation unit is used to multiply the gray value of each pixel in the gray-level difference map with the corresponding depth coefficient to obtain the local deformation depth map.

8. The irregular curved surface visual-tactile sensor according to claim 4, characterized in that, The irregular curved surface visual-tactile sensor is a dexterous fingertip visual-tactile sensor, with the irregular curved surface fingertip contact surface as the sensing interface. The irregular curved surface fingertip contact surface is any one of the following: a free surface, an eccentric spherical cap surface, and a multi-curvature transition surface that fits the human fingertip or dexterous fingertip. Alternatively, the irregular curved surface of the contact surface is a composite surface formed by the continuous transition of the fingertip surface, the lateral surface, and the fingertip surface. The elastic layer (3) covers the outside of the light guide layer (4) and forms the transition surface between the fingertip and the fingertip.

9. The irregular curved surface visual-tactile sensor according to claim 4, characterized in that, The real-time depth calculation unit is used to determine the real-time depth of each pixel using the following formula. : , in, The reference surface depth corresponding to this pixel in the reference surface depth map; The local deformation depth of each pixel is defined as follows: the reference surface depth is the depth of the pixel when the irregular surface visual-tactile sensor is in an uncontacted state; the reference surface depth is the depth of the contact surface of the irregular surface visual-tactile sensor along the corresponding pixel coordinate ray in the uncontacted state; the pixel coordinate ray represents the unit direction in which a preset point in the image perception module passes through the pixel and points to the surface point; the pixel coordinate ray is used to convert the local deformation depth of each pixel into a three-dimensional local depth perturbation along the direction of the pixel coordinate ray; s is a marker for the display scene. The three-dimensional point cloud construction unit is used to calculate the position of each pixel in a preset coordinate system using the following formula. : , in, This refers to the position of a preset point in the image perception module within a preset coordinate system. The real-time depth of each pixel. The pixel coordinate ray corresponding to each pixel point.

10. A control unit, comprising a control component and a visual-tactile sensor connected to said control component, characterized in that, The visual-tactile sensor is an irregular curved surface visual-tactile sensor as described in any one of claims 1 to 9.