Curved surface visual tactile sensor three-dimensional point cloud construction method and device, equipment and medium

Abstract

The application provides a curved surface visual tactile sensor three-dimensional point cloud construction method, device, equipment and medium, wherein the method comprises the following steps: calculating a gray difference image according to a reference gray image and a third real-time gray image; calculating a local deformation depth image according to the gray difference image and a depth coefficient; determining the corresponding curved surface geometric parameters of each pixel point in the local deformation depth image, wherein the curved surface geometric parameters comprise a reference curved surface depth, a pixel coordinate ray and the position of a preset point in the image sensing module in a preset coordinate system; calculating the position of each pixel point in the preset coordinate system according to the local deformation depth of each pixel point in the local deformation depth image and the corresponding curved surface geometric parameters of each pixel point, and realizing the reconstruction of the curved surface contact three-dimensional point cloud. The method realizes the curved surface geometric parameter calibration of the curved surface visual tactile sensor and the reconstruction of the curved surface contact three-dimensional point cloud, and can effectively improve the accuracy of the three-dimensional point cloud construction.

Description

Technical Field

[0001] This application relates to the field of visual-tactile sensor technology, and in particular to a method, apparatus, device and medium for constructing a three-dimensional point cloud of a curved visual-tactile sensor. Background Technology

[0002] Visual-tactile sensors in robotic end effectors, dexterous fingertips, thumb modules, and curved haptic interfaces in consumer electronics are often non-planar or irregular curved surfaces. These sensors exhibit assembly errors between the camera, flexible surface, transparent support, housing reference, and lighting path; different regions of the curved surface also have different initial depths, normals, material thicknesses, and grayscale responses.

[0003] If the calibration assumptions of planar sensors are directly adopted, it is easy to mix up the initial height of the curved surface, camera attitude deviation, illumination baseline drift, and actual contact deformation. Moreover, existing calibration methods often emphasize camera intrinsic parameters, planar coordinate scale, or single height mapping, making it difficult to directly adapt to complex fingertip surfaces such as freeform surfaces, lateral curved surfaces, and thumb curved surfaces.

[0004] Furthermore, in tasks such as grasping, assembling, manipulating, twisting, and suppressing slippage, dexterous hands require fingertips to simultaneously possess the capabilities of compliant contact, surface conformation, contact position recognition, and deformation depth estimation. Visor-tactile sensors, which observe image changes in the flexible contact layer through a camera, can acquire dense tactile information within a small volume, making them suitable for installation at the ends of robot fingers.

[0005] Existing visual-tactile solutions are mostly based on planar elastic bodies, regular spherical caps, or regular circular arc surfaces. Their image inversion typically relies on planar calibration, dot matrix marker displacement, and stereo or fixed geometric assumptions of color light source photometric properties. When the sensor is designed to fit the irregular curved surface of a dexterous fingertip, the thickness, normal, illumination incident angle, camera viewing angle, and initial grayscale are all different at various points on the surface. The same indentation depth may manifest as different grayscale responses at different pixel locations. If the reconstruction logic of planar visual-tactile sensing is directly applied, it is easy to mix together the uneven grayscale of the initial curved surface, the differences in material thickness, and the actual contact deformation.

[0006] Therefore, existing technologies lack calibration methods for curved surface visual-tactile sensors and methods for constructing contact 3D point clouds. Summary of the Invention

[0007] In view of this, embodiments of this application provide a method, apparatus, device and medium for constructing a three-dimensional point cloud of a curved surface visual-tactile sensor, in order to solve the technical problem of the lack of calibration methods and contact three-dimensional point cloud construction methods for curved surface visual-tactile sensors in the prior art.

[0008] To address the aforementioned technical issues, this solution offers the following:

[0009] The first aspect of this application provides a method for constructing a 3D point cloud of a curved visual-tactile sensor, including the following steps: A grayscale difference image is calculated based on a reference grayscale image and a third real-time grayscale image. The third real-time grayscale image is the grayscale image output by the image perception module when the curved visual-tactile sensor is in contact. The third real-time grayscale image and the reference grayscale image have the same size. The reference grayscale image is the grayscale image corresponding to when the curved visual-tactile sensor is in non-contact. The grayscale value of each pixel in the grayscale difference image is the difference between the grayscale value of the corresponding pixel in the reference grayscale image and the grayscale value of the corresponding pixel in the third real-time grayscale image. A local deformation depth map is calculated based on the grayscale difference map and the depth coefficient, wherein the depth coefficient is the depth change corresponding to a unit grayscale change. For each pixel in the local deformation depth map, the surface geometric parameters corresponding to each pixel are determined. The surface geometric parameters include a reference surface depth, a pixel coordinate ray, and the position of a preset point in the image perception module in a preset coordinate system. The reference surface depth is the depth corresponding to each pixel when the surface visual tactile sensor is in an uncontacted state, representing the depth of the contact surface of the surface visual tactile sensor along the corresponding pixel coordinate ray in the uncontacted state. The pixel coordinate ray represents the unit direction in which the preset point in the image perception module passes through the pixel and points to the surface point. The pixel coordinate ray is used to convert the local deformation depth of each pixel into a three-dimensional local depth perturbation along the direction of the pixel coordinate ray. The surface geometric parameters correspond to different pixels in different display scenarios. Based on the local deformation depth of each pixel in the local deformation depth map and the surface geometry parameters corresponding to each pixel, the position of each pixel in the preset coordinate system is calculated, thereby realizing the reconstruction of the three-dimensional point cloud of surface contact.

[0010] A second aspect of this application provides a three-dimensional point cloud construction device for a curved surface visual-tactile sensor, comprising a grayscale difference calculation module, a deformation depth calculation module, a parameter determination module, and a three-dimensional point cloud construction module that are sequentially and communicatively connected. The grayscale difference calculation module is used to calculate a grayscale difference map based on a reference grayscale image and a third real-time grayscale image. The third real-time grayscale image is a grayscale image output by the image perception module when the curved visual-tactile sensor is in contact. The third real-time grayscale image and the reference grayscale image have the same size. The reference grayscale image is the grayscale image corresponding to when the curved visual-tactile sensor is in non-contact. The grayscale value of each pixel in the grayscale difference map is the difference between the grayscale value of the corresponding pixel in the reference grayscale image and the grayscale value of the corresponding pixel in the third real-time grayscale image. The deformation depth calculation module is used to calculate a local deformation depth map based on the grayscale difference map and the depth coefficient, wherein the depth coefficient is the depth change corresponding to a unit grayscale change. The parameter determination module is used to determine the surface geometric parameters corresponding to each pixel in the local deformation depth map. The surface geometric parameters include a reference surface depth, a pixel coordinate ray, and the position of a preset point in the image perception module in a preset coordinate system. The reference surface depth is the depth corresponding to each pixel in the non-contact state of the surface visual tactile sensor, representing the depth of the contact surface of the surface visual tactile sensor along the corresponding pixel coordinate ray in the non-contact state. The pixel coordinate ray represents the unit direction in which the preset point in the image perception module passes through the pixel and points to the surface point. The pixel coordinate ray is used to convert the local deformation depth of each pixel into a three-dimensional local depth perturbation along the direction of the pixel coordinate ray. The surface geometric parameters correspond to different pixels in different display scenarios. The 3D point cloud construction module is used to calculate the position of each pixel in a preset coordinate system based on the local deformation depth of each pixel in the local deformation depth map and the surface geometric parameters corresponding to each pixel, thereby realizing the reconstruction of the 3D point cloud of surface contact.

[0011] A third aspect of this application provides a computer device, including a memory and a processor connected in sequence, wherein the memory stores a computer program, and the processor is used to read the computer program and execute the method for constructing a three-dimensional point cloud of a curved visual-tactile sensor as described in the first aspect.

[0012] The fourth aspect of this application provides a computer-readable storage medium storing a method for constructing a three-dimensional point cloud of a curved visual-tactile sensor as described in the first aspect.

[0013] Compared with the prior art, the beneficial effects that the at least one technical solution adopted in the embodiments of this application can achieve include at least: This application uses multiple frames of grayscale images without contact as reference grayscale images, and uses the grayscale difference between the reference grayscale image and the current real-time grayscale image as a grayscale difference map. Then, based on the depth coefficient, the grayscale value of each pixel in the grayscale difference map is converted into local depth deformation, forming a local deformation depth map to characterize the contact intensity. Then, based on the local deformation depth of each pixel in the local deformation depth map and the surface geometric parameters corresponding to each pixel, the position of each pixel in the preset coordinate system is calculated, that is, the contact 3D point cloud is updated along the pixel coordinate ray corresponding to each pixel to realize the reconstruction of the surface contact 3D point cloud. This surface contact 3D point cloud reconstruction method is not only applicable to regular surface visual tactile sensors, but also to irregular surface visual tactile sensors. In irregular surface visual tactile sensors, it can effectively improve the accuracy of surface contact 3D point cloud construction. Furthermore, this application pre-encodes the surface geometry of the curved visual tactile sensor in the non-contact state into the surface geometric parameters corresponding to each pixel, thereby realizing the calibration of the curved visual tactile sensor. This ensures that each pixel has an independent spatial reference, and the initial shape of the surface is no longer determined by real-time grayscale. This avoids mistaking the surface substrate of the irregular curved part of the curved visual tactile sensor for contact deformation, which in turn helps to improve the accuracy of the construction of the three-dimensional point cloud of curved surface contact in irregular curved visual tactile sensors. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of the structure of a curved visual-tactile sensor provided in an embodiment of this application.

[0016] Figure 2 This is a flowchart of a method for constructing a three-dimensional point cloud of a curved surface visual-tactile sensor, provided in an embodiment of this application. Figure 3 This is a structural block diagram of a computer device provided in an embodiment of this application.

[0017] Figure reference numerals: 1. Outer shell; 2. Light-shielding contact layer; 3. Elastic layer; 4. Light guide layer; 5. Illumination diffusion module; 6. Image sensing module; 201. Memory; 202. Processor. Detailed Implementation

[0018] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. This application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The first aspect of this application discloses a method for constructing a 3D point cloud for a curved surface visual-tactile sensor. This method is applicable not only to regular curved surface visual-tactile sensors but also to irregular curved surface visual-tactile sensors. For example... Figure 1 As shown, the curved visual-tactile sensor includes a housing 1, a light-shielding contact layer 2, an elastic layer 3, a light-guiding layer 4, an illumination diffusion module 5, an image sensing module 6, and a three-dimensional point cloud construction device for the curved visual-tactile sensor.

[0021] The curved visual-tactile sensor can be a finger, toe, or other visual-tactile unit of a dexterous hand. The contact surface of the curved visual-tactile sensor, i.e., the surface of the light-shielding contact layer 2, can be an irregular curved surface or a regular curved surface.

[0022] The outer shell 1 can be a 3D printed shell, an injection molded shell, or a metal / resin skeleton. The outer shell 1 provides an assembly reference for the light-shielding contact layer, elastic layer, light-guiding layer, illumination diffusion module, and image sensing module. An installation window is provided on the outer shell, which also provides a mechanical interface 11 for connection to the control components. These control components can be robot limb skeletons, such as finger or toe skeletons, or robotic arms and other control components.

[0023] The light-shielding contact layer 2 is a black flexible layer used to block light entering the elastic layer 3. It can be a black flexible coating, a thin layer of black silicone, a sprayed light-shielding layer, or an integrally molded black surface. It serves to block light entering the elastic layer 3 and acts as a contact interface for external objects, ensuring that grayscale changes primarily originate from the response of internal illumination modulated by elastic deformation. The surface of the light-shielding contact layer is a contact surface, which can be a regular curved surface or an irregular curved surface. When the irregular curved surface visual-tactile sensor is a dexterous finger, the light-shielding contact layer simulates the structure of the human fingertip. It can be a free-form surface, an eccentric spherical cap surface, a multi-curvature transition surface, or a composite surface formed by the continuous transition of the fingertip surface, lateral surface, and fingertip surface. The light-shielding contact layer can be achieved through spraying, dip coating, thin film lamination, black silicone casting, or integral molding with the elastic layer.

[0024] Elastic layer 3 covers the light guide layer and can form a transition surface between the fingertip and the fingertip. To improve the sensitivity and mechanical range of grayscale response adjustment, elastic layer 3 can be made of scattering microparticles, milky white pigments, and flexible fillers. The elastic layer can also be a multi-layer structure. The thickness of the elastic layer is greater than that of the light-shielding contact layer; generally, the thickness of elastic layer 3 is 1.5-2.5 mm, and the thickness of the light-shielding layer is 0.1-0.4 mm. The light-shielding contact layer needs to maintain sufficient flexibility to prevent it from becoming the main load-bearing layer. The elastic layer is 0-degree or near-0-degree translucent; specifically, the transparency of the elastic layer can be controlled between 40% and 60%. Under stress, it produces local compression, indentation, shearing, and curvature changes, converting contact deformation into observable grayscale attenuation, enhancement, gradient, or texture changes.

[0025] The light guide layer 4 is fixed to the mounting windows around the perimeter of the housing. The equivalent stiffness of the light guide layer 4 is greater than that of the semi-elastic layer, causing the main deformation to occur in the outer contact area under stress. The light guide layer is transparent and can be made of transparent acrylic, transparent resin, glass, polycarbonate, or high-hardness transparent silicone. The light guide layer is used to provide the inner observation boundary, guide illumination light, stabilize the surface geometry, reduce inner non-target deformation, and make the grayscale-depth mapping more stable. Specifically, in one embodiment, the hardness of the light guide layer is Shore 45A, i.e., Shore A degree 45; the hardness of the elastic layer is Shore 00 30, i.e., Shore 00 type 30.

[0026] That is, the light-shielding contact layer 2, the elastic layer 3, and the light-guiding layer 4 are arranged sequentially from the outside to the inside.

[0027] The lighting diffusion module 5 is mounted along the side wall of the housing. The lighting diffusion module can be a side-lit LED, a ring LED, a multi-zone LED, or a monochromatic / multi-color light source homogenized by the diffusion layer. The lighting diffusion module 5 is used to provide stable internal lighting to the translucent elastic layer.

[0028] The image sensing module 6 is housed within the outer casing, with the image acquisition area facing the guide light layer. The image sensing module 6 can be a camera or a video camera, specifically including a miniature camera, a wide-angle lens, and an image sensor, and is fixed to the outer casing via a mounting bracket.

[0029] Furthermore, the curved surface tactile sensor may also include a processing and display module for updating the point cloud and displaying it in 3D. Specifically, the processing and display module can be implemented using a web service. The backend loads the geometry array and maintains its operating status, including the reference grayscale image, the latest grayscale difference image, serial port connection status, frame number, error messages, and recalibration events. The frontend obtains the grayscale difference image, reference depth map, and pixel ray array through a binary interface, and completes the point cloud update and 3D display in the browser.

[0030] The curved surface visual-tactile sensor 3D point cloud construction device is signal-connected to the image sensing module 6 and the illumination diffusion module 5 to construct a curved surface contact 3D point cloud according to the curved surface visual-tactile sensor 3D point cloud construction method provided in this solution. For example, Figure 2 As shown, the method for constructing a 3D point cloud for a curved visual-tactile sensor includes the following steps: Step S1: Calculate a grayscale difference image based on the reference grayscale image and the third real-time grayscale image. The third real-time grayscale image is the grayscale image output by the image perception module when the curved visual-tactile sensor is in contact. The third real-time grayscale image and the reference grayscale image have the same size. The reference grayscale image is the grayscale image corresponding to the curved visual-tactile sensor in non-contact state. The grayscale value of each pixel in the grayscale difference image is the difference between the grayscale value of the corresponding pixel in the reference grayscale image and the grayscale value of the corresponding pixel in the third real-time grayscale image. Step S2: Calculate the local deformation depth map based on the grayscale difference map and the depth coefficient, where the depth coefficient is the depth change corresponding to a unit grayscale change. Step S3: For each pixel in the local deformation depth map, determine the surface geometric parameters corresponding to each pixel. The surface geometric parameters include a reference surface depth, a pixel coordinate ray, and the position of a preset point in the image perception module in a preset coordinate system. The reference surface depth is the depth corresponding to each pixel when the surface tactile sensor is in an uncontacted state, representing the depth of the contact surface of the surface tactile sensor along the corresponding pixel coordinate ray in the uncontacted state. The pixel coordinate ray represents the unit direction in which the preset point in the image perception module passes through the pixel and points to the surface point. The pixel coordinate ray is used to convert the local deformation depth of each pixel into a three-dimensional local depth perturbation along the direction of the pixel coordinate ray. Each pixel corresponds to different surface geometric parameters in different display scenarios. Step S4: Based on the local deformation depth of each pixel in the local deformation depth map and the surface geometry parameters corresponding to each pixel, calculate the position of each pixel in the preset coordinate system to realize the reconstruction of the surface contact 3D point cloud.

[0031] In practice, before implementing the above-mentioned method for constructing a 3D point cloud of a curved surface visual-tactile sensor, the relevant parameters can be calibrated using the following methods: In practice, the intrinsic parameters and distortion of the image perception module, namely the camera imaging model and lens distortion, are calibrated. The intrinsic parameters and distortion of the image perception module are used for image correction, pixel coordinate ray generation, and pixel projection.

[0032] Select a benchmark sensor with stable structure and imaging state, and acquire multi-pose images using checkerboard, dot plate, or equivalent calibration methods to obtain the intrinsic parameters of the image sensing module and lens distortion parameters. If sensors of the same model use the same lens module and fixed focal length assembly, the camera intrinsic parameters can be used as common parameters for the structure; when the lens module, focal length, or imaging path changes, recalibration should be performed.

[0033] Specifically, at least four rigid reference marks are placed in the area where the curved visual-tactile sensor will not deform. These marks can be dots, pits, microstructures, reflective points, or other reliably identifiable visual features. The three-dimensional coordinates of each mark point in the image perception module's base coordinate system are measured using a coordinate measuring machine, a curved surface model, or tooling, and the corresponding pixel coordinates are extracted from the image.

[0034] In practice, the individual extrinsic parameters are calibrated, i.e., the rigid body transformation from the camera to the sensor coordinate system, to align the camera geometry with the sensor coordinates or robot coordinates.

[0035] Specifically, the extrinsic parameters of the image perception module relative to the basic coordinate system of the curved surface visual-tactile sensor are solved using Perspective-n-Point (PnP). When the three-dimensional coordinates of the rigid reference marker correspond one-to-one with its two-dimensional pixel coordinates, the initial rotation matrix and initial translation vector can be obtained first using PnP. When there may be false detections or occlusions in marker recognition, interior points can be screened using random sampling consistency. Subsequently, nonlinear least squares optimization is performed with reprojection error as the objective to obtain the extrinsic parameters of each image perception module. These extrinsic parameters are used to unify the curved surface model, camera rays, and real-time point clouds under the same sensor coordinate system. The rigid reference marker is only used as a coordinate reference and does not participate in the deformation measurement of the flexible surface, thus decoupling the pose error of the image perception module from the contact deformation.

[0036] The extrinsic parameters of the image perception module 6 are solved using the following method:

[0037] in, The scale factor corresponding to the i-th rigid datum mark. Let be the x-coordinate of the i-th rigid reference marker in the image. Let be the pixel ordinate of the i-th rigid reference marker in the image. Let X be the X-axis coordinate component of the i-th rigid reference mark in the coordinate system of the image sensing module. Let represent the Y-axis coordinate component of the i-th rigid reference mark in the coordinate system of the image sensing module. Let Z be the Z-axis coordinate component of the i-th rigid reference mark in the coordinate system of the image sensing module. To mark the three-dimensional points of the rigid reference in the sensor's base coordinate system, For the corresponding pixel in the image, These are intrinsic parameters of the image perception module. Let be the initial rotation matrix. To optimize the solution, the optimal rotation matrix is ​​obtained, where t is the initial translation vector. To optimize the solution and obtain the optimal translation vector, Indicates perspective projection. This represents a robust loss function used to reduce the impact of outliers. Let be the magnitude of the vector, where represents the Euclidean distance of the reprojection error, i.e., the square of the L2 norm.

[0038] In practice, the surface geometry parameters corresponding to each pixel are calibrated to describe the spatial position of each pixel on the non-contact surface. This involves generating the reference surface depth, pixel coordinate ray, and the position of preset points in the image sensing module within a preset coordinate system for each pixel. This makes the initial curvature an explicit parameter, avoiding the use of two-dimensional planar coordinates, a uniform zero-point height, or a global scale as a reference. The preset coordinate system can be the coordinate system of the surface visual-tactile sensor. The preset points in the image sensing module can be the center of the image sensing module or other fixed points.

[0039] In practical implementation, a multi-surface model adaptation method is provided, meaning that different multi-surface models are adapted to different surface geometric parameters, and independent sets of surface geometric parameters can be generated for each. Taking a dexterous hand as an example, for different surface models such as the standard finger, thumb, or other dexterous fingertip surfaces, independent sets of geometric parameters can be generated for each. During runtime, the surface model is switched to construct a 3D point cloud based on the corresponding set of surface geometric parameters. However, during the 3D point cloud construction process, the image acquisition, reference frame recalibration, grayscale difference, and point cloud generation processes for different surface models remain consistent. As a result, the 3D point cloud reconstruction of the surface visual-tactile sensor changes from planar scale calibration to pixel-level surface ray calibration, and unifies different curvature regions, different viewing angle occlusion relationships, and different illumination baselines into the same calibration framework.

[0040] Specifically, regarding the positions of preset points in the reference surface depth map, pixel ray direction map, and image perception module within the preset coordinate system: Based on the intrinsic parameters of the image perception module, the extrinsic parameters of each individual, and the surface model corresponding to the curved visual-tactile sensor (i.e., the 3D model of the curved visual-tactile sensor structure), camera rays are generated for each pixel. The intersection of each camera ray with the surface model in the non-contact state is calculated, and the depth of the intersection point is determined as the reference surface depth for that pixel. This yields a reference surface depth map, which represents the reference depth of the non-contact surface on each pixel's coordinate rays. This reference surface depth map is used to express the initial geometric depth of the irregular surface. Simultaneously, the camera ray direction is transformed to the fundamental coordinate system of the curved visual-tactile sensor, forming a pixel ray direction map, which represents the 3D spatial direction corresponding to each pixel. This pixel ray direction map is used to convert depth changes into 3D point displacement. When using a 3D scanned point cloud instead of a surface model, the scanned point cloud can be registered to the sensor coordinate system first, and then the nearest intersection point or interpolated depth can be calculated for each ray.

[0041] Specifically, the pixel coordinate ray in the pixel ray direction map represents the unit direction from a preset point in the image perception module that passes through the pixel and points to the finger surface point, which is used to convert one-dimensional depth changes into three-dimensional coordinate changes.

[0042] Specifically, the pixel coordinate ray corresponding to each pixel is determined by the following formula to form a pixel ray direction map, wherein, , In the formula, This represents the normalized pixel coordinate ray of pixel point (u,v) in the coordinate system of the image perception module. For the transpose of a matrix or vector, , These are the x and y coordinates of the pixel in the coordinate system of the image perception module, respectively.

[0043] The reference surface depth corresponding to each pixel is determined by the following formula to form a reference surface depth map, wherein...

[0044]

[0045]

[0046] In the formula, This represents the normalized pixel coordinate ray of pixel point (u,v) in the basic coordinate system of the curved visual-tactile sensor. This is the rotation matrix from the coordinate system of the image perception module to the basic coordinate system of the curved visual-tactile sensor. Here are the coordinates of the optical center of the image sensing module in the basic coordinate system of the curved visual-tactile sensor. The surface of the curved surface model in a non-contact state. For Starting from the pixel coordinate ray The nearest intersection with the surface to The distance is the depth of the reference surface corresponding to the pixel; min is the minimum value; t represents the distance of any point on the pixel coordinate ray relative to the starting point. The displacement is a ray parameter in non-negative real number form.

[0047] The reference surface depth map and pixel ray direction map obtained in this step are important results that distinguish this application from planar calibration: the planar method only requires a scale or planar equation, while this application saves different initial depths and three-dimensional directions for each pixel, thereby allowing the sensor surface to be convex, concave, free-form, or multi-curvature transition surfaces.

[0048] By establishing pixel-level depth values, the extrinsic parameter stage uses PnP, robust interior point filtering, and reprojection error optimization to form a closed loop from two-dimensional image points to three-dimensional surface coordinates.

[0049] In specific implementation, the position of the preset point in the image perception module in the preset coordinate system, i.e. the position of the center point of the image perception module in the system coordinate system, is used to determine the spatial origin of the point cloud projection.

[0050] Furthermore, for the reference surface depth map, it provides the basic position of each pixel on the non-contact surface. When the reference surface depth map has invalid values, non-finite values, or abnormal intersection of model edges, it can be replaced with the statistical depth of the effective region (such as the median depth of the effective region) to avoid holes or abnormal jumps in the real-time point cloud.

[0051] Specifically, the calibration of surface geometric parameters pre-encodes the non-contact surface geometry into a reference surface depth map, a pixel ray direction map, and the positions of preset points in the image perception module within a preset coordinate system. This gives each pixel an independent spatial reference and makes the initial curvature an explicit parameter. This ensures that the initial shape of the surface is no longer determined by the real-time grayscale model, preventing irregular curved surfaces such as fingertips and fingertip sides from being mistakenly identified as contact deformation by the surface substrate of the surface visual-tactile sensor.

[0052] In practice, the reference grayscale image is used as follows: During sensor operation, factors such as light source brightness, temperature, camera exposure, light-shielding layer status, and material rebound can cause non-contact grayscale baseline drift. Therefore, this application proposes, after confirming the non-contact state, using a multi-frame median reference image to describe the grayscale baseline under the current individual, current light source, and current exposure state. It supports online updating of the reference grayscale benchmark without regenerating the surface geometry parameters. This separates the static grayscale background formed by differences in illumination, exposure, and material transmittance from the dynamic grayscale changes caused by contact, reducing the impact of drift on depth mapping. For example, after the entire system is assembled, the curved visual-tactile sensor's contact surface is kept in a non-contact state (i.e., without external force), and the light source brightness, camera exposure, gain, and white balance are kept constant. After warm-up and stabilization, the curved visual-tactile sensor acquires multiple frames of grayscale images and uses the pixel-wise median instead of a single frame image as the reference grayscale image to reduce the impact of serial port transmission jitter, exposure fluctuations, and occasional noise.

[0053] In practice, the acquisition of the reference grayscale image includes steps S11 to S14.

[0054] Step S11: In response to receiving the calibration command, after a first preset number of sampling cycles of the image sensing module, acquire multiple frames of the first real-time grayscale image collected by the image sensing module in a non-contact state by the visual-tactile sensor.

[0055] The calibration command can be passively triggered via the recalibration interface during power-on or automatically triggered during system startup. Upon receiving the calibration command, a first preset number of warm-up frames are discarded. Here, the first preset number can be set to any value between 5 and 10, for example, it can be set to 8.

[0056] Step S12: Calculate the average gray level difference between two adjacent frames of the first real-time gray level image in the multi-frame first real-time gray level image.

[0057] Here, the number of frames of the first real-time grayscale image can be any value from 3 to 10.

[0058] Step S13: In response to the fact that the average gray level difference between two adjacent frames of the first real-time gray level images in the multi-frame first real-time gray level images is less than the preset value, or that the image perception module has reached the maximum number of warm-up frames after receiving the calibration command, acquire the multi-frame second real-time gray level images collected by the image perception module in the non-contact state of the visual tactile sensor.

[0059] The default value here can be set to any value between 1.0 and 4.0, for example, it can be set to 2.0.

[0060] When the average grayscale difference between two adjacent frames of the first real-time grayscale image in multiple consecutive frames is less than the preset value, it is considered that the light source, camera exposure and sensor image state have stabilized, or the maximum warm-up frame number has been reached, and real-time grayscale images can continue to be acquired to reconstruct the reference grayscale image in order to improve the accuracy of the reference grayscale image.

[0061] The more real-time grayscale images there are, the higher the accuracy of the obtained reference grayscale image. However, a larger number of images will also lead to time consumption. Therefore, it is preferable that the number of frames of the second real-time grayscale image is 8 to 15 frames.

[0062] Step S14: Calculate the median or average value of each pixel in the multiple frames of the second real-time grayscale image to obtain the reference grayscale image. When changes occur in light source brightness, temperature, shielding layer condition, or camera exposure, recalibration can be updated. This avoids mistaking baseline drift for contact deformation.

[0063] The reference grayscale image is determined based on the median. The specific method is as follows: , In the formula, To take the median, This is the second real-time grayscale image of the i-th frame. The total number of the second real-time grayscale images. .

[0064] By using the above method, warm-up and stability assessments are performed during the startup phase. After stabilization, a reference grayscale image is determined, which effectively improves the accuracy of the reference grayscale image.

[0065] In practice, the reference surface depth map, pixel ray direction map and reference grayscale image have the same image resolution, so that the corresponding depth, grayscale and pixel coordinate rays can be found in each map by the index of the same pixel point.

[0066] In practical implementation, the depth coefficient, i.e., the mapping relationship between grayscale changes and local deformation depth, is used to convert the grayscale difference of pixels into deformation depth: Standard indenters with known geometries, such as ball-head, cylindrical-head, flat-head, or fingertip-like indenters, can be used to press into different regions, directions, and depths of the curved surface. Each press records the current grayscale image, indenter pose, indentation amount, and contact area. Pixel-level true deformation depth is calculated using the standard indenter geometry and a reference depth map. Then, the depth coefficient is obtained by fitting the relationship between the gray-level difference and the deformation depth.

[0067] In practice, after calibrating the relevant parameters of the curved visual tactile sensor, the results can be verified and logically checked from five aspects: geometric consistency, grayscale baseline stability, depth mapping accuracy, real-time point cloud continuity, and consistency of multi-model switching, as shown in Table 1 below.

[0068] Table 1

[0069] If the inspection error does not meet the requirements, the cause should be located according to the error pattern: overall offset usually corresponds to external parameter or camera center error; curved background error usually corresponds to reference surface depth or ray direction generation error; contact depth ratio error usually corresponds to gray-scale-depth mapping error; contact hot area still appears in non-contact state usually corresponds to reference gray-scale reference drift or light shading abnormality or lighting abnormality.

[0070] The accuracy of 3D point cloud reconstruction is determined based on the root mean square error. Specifically,

[0071] In the formula, The root mean square error (RMSE) of point cloud reconstruction is used to characterize the accuracy of 3D point cloud reconstruction. The total number of point cloud points involved in the error calculation. For the reconstructed 3D coordinates of the j-th point cloud, Let be the reference 3D coordinates corresponding to the j-th point cloud.

[0072] In practice, after the curved surface visual-tactile sensor is activated, data such as curved surface geometric parameters and depth coefficients can be loaded, and the image perception module begins to collect and output grayscale image data to perform the reconstruction process of the three-dimensional point cloud of the curved surface contact.

[0073] Specifically, during the process of calculating the grayscale difference map based on the reference grayscale image and the third real-time grayscale image, the curved visual-tactile sensor enters a contact state. The image perception module acquires the third real-time grayscale image in real time, subtracts the grayscale value of each pixel in the reference grayscale image from the grayscale value of the corresponding pixel in the third real-time grayscale image, and assigns negative values ​​to 0, thus obtaining the grayscale difference map. .

[0074] This grayscale difference image represents the grayscale response caused by contact indentation, local displacement of the light-shielding contact layer, and changes in the optical path of the elastic layer. Specifically, the grayscale value of each pixel in the grayscale difference image... for: , In the formula, clip is a value between 0 and 255. Values ​​exceeding 255 are truncated to 255, and values ​​below 0 are truncated to 0. This is the third real-time grayscale image.

[0075] By independently establishing a non-contact grayscale reference, i.e., a grayscale difference map, it is possible to interpret only the effective grayscale changes relative to the reference in real time.

[0076] In practice, during the process of calculating the local deformation depth map based on the grayscale difference map and the depth coefficient, the grayscale value of each pixel in the grayscale difference map... Through depth coefficient Convert to local deformation depth The depth of deformation represents the magnitude of force at different locations.

[0077] In some low-computing-power or real-time display scenarios, the following linear model can be preferentially used to obtain the local deformation depth, specifically: .

[0078] Wherein, the depth coefficient k is a calibrable coefficient or an adjustable coefficient.

[0079] In one specific embodiment, k can be any value from 0.001 mm / gray to 0.05 mm / gray; for example, k is 0.01 mm / gray.

[0080] In practice, the process of calculating the local deformation depth map based on the gray-level difference map and the depth coefficient can use not only gray-level difference, but also normalized difference, local gradient, pixel coordinates, reference surface depth, ray direction, surface partitioning and historical frame information. This process can be implemented by models such as partitioning polynomial, radial basis function, lightweight convolutional network or small encoder-decoder network.

[0081] In specific implementation, after obtaining the local deformation depth map, the surface contact 3D point cloud can be reconstructed based on the surface geometry parameters corresponding to each pixel. For example, based on the local deformation depth of each pixel in the local deformation depth map and the surface geometry parameters corresponding to each pixel, the position of each pixel in a preset coordinate system is calculated, including: The real-time depth of each pixel is determined based on the local deformation depth of each pixel and the reference surface depth corresponding to each pixel. The position of each pixel in the preset coordinate system is calculated based on the real-time depth of each pixel, the pixel coordinate ray, and the position of the preset point in the image perception module in the preset coordinate system.

[0082] Specifically, the local deformation depth of each pixel can be determined using the following formula. The real-time depth of each pixel is determined by the reference surface depth corresponding to each pixel. To generate a real-time depth map: , In the formula, The reference surface depth corresponding to each pixel; To mark the display scene, symbols can be set for display scenes where the contact pressing direction and the display direction are different. The distinction is made to switch between different display scenarios such as recessed display and raised display, so as to obtain the surface geometric parameters corresponding to each pixel in the current display scenario.

[0083] In specific implementation, the position of each pixel in the preset coordinate system is calculated using the following formula based on the real-time depth of each pixel, the pixel coordinate ray, and the position of the preset point in the image perception module in the preset coordinate system. It can update the dense point cloud reconstruction at the surface contact point in real time: , In the formula, This refers to the position of a preset point in the image perception module within a preset coordinate system. The pixel coordinate ray corresponding to each pixel point.

[0084] Furthermore, the position of each pixel in the third real-time grayscale image in a preset coordinate system can be used as a reference. and local deformation depth Calculate the contact center The contact center is the equivalent position center of the curved visual tactile sensor under contact state, and this equivalent position center can be used as the equivalent force center.

[0085] Specifically, contact center for: , The reconstruction of the 3D point cloud of the surface contact is achieved by representing contact deformation as depth perturbation along pixel coordinate rays, rather than simply superimposing it on a uniform plane height map. This method allows regions on the surface with different normals, depths, and viewing angles to be back-projected into 3D point clouds using the same formula.

[0086] In practical implementation, each pixel can be colored according to its local deformation depth in the local deformation depth map to form a contact heat map. The maximum contact depth can be recorded. , as an indicator of contact strength.

[0087] In practice, the relative force vector can also be determined based on the local deformation depth map. Specifically, for pixels in the local deformation depth map where the local deformation depth is greater than a preset depth, the opposite direction of the pixel coordinate ray corresponding to that pixel is taken as the contact contribution direction, and the relative force vector is obtained by accumulating the values ​​with the deformation depth as the weight. : , In the formula, For display or control purposes, a proportional coefficient is used. This represents the local deformation depth of this pixel in the local deformation depth map. For preset depth, The pixel coordinate ray corresponding to each pixel point.

[0088]

[0089] In the formula, The coordinates of the three-dimensional centroid of the contact area are: The coordinates of the three-dimensional point cloud corresponding to the pixel are given.

[0090] In one specific implementation, It can be set to 0.02mm, a proportional coefficient for display or control. It can be set to 0.0024.

[0091] Specifically, the relative force vector does not necessarily have to be directly equivalent to the absolute Newtonian force, but it can be used to display the contact direction, determine the grasping stability trend, or serve as input for a further force calibration model, which is used to convert image information / 3D reconstruction information into force information.

[0092] In practical implementation, the above-mentioned method for constructing three-dimensional point clouds of curved surface visual-tactile sensors has the following beneficial effects: Adapting to irregular curved surfaces: By explicitly describing the geometry of non-contact curved surfaces using reference surface depth maps and pixel ray direction maps, the algorithm avoids mistaking the initial height of the surface for contact deformation, enabling it to cover irregular curvature areas such as fingertips, finger pads, sides, and thumbs.

[0093] Clearly defined calibration layers: Parameters such as camera intrinsics, individual extrinsic parameters, surface reference geometry, grayscale baseline, and grayscale-depth mapping (i.e., depth coefficients) are determined separately, reducing parameter coupling. Compared to regular planar methods, this application does not rely on a single planar scale or a uniform zero-height point, but instead uses a pixel-by-pixel surface reference as the starting point for 3D reconstruction.

[0094] Real-time deployment friendly: Calibration results can be solidified into parameter sets and mapping relationships. Real-time reconstruction only requires grayscale difference, depth mapping, and ray projection, making it suitable for applications such as display, embedded processing, and robot control. Complex surface calculations are mainly performed in the calibration stage, keeping the runtime lightweight.

[0095] Good individual adaptability: rigid markers are used to solve individual extrinsic parameters, multi-frame median reference maps are used to compensate for individual lighting and material states, and a small amount of standard pressing is used to update the depth map.

[0096] Highly scalable across multiple models: Standard finger, thumb, and other fingertip surfaces can be integrated into the same reconstruction process using different pre-calculated surface geometric parameters. Standard finger refers to any one or more of the index, middle, ring, and little fingers.

[0097] The protection boundaries are clearly defined: the innovation lies in the calibration of the surface geometric parameters of irregular curved surfaces and the reconstruction link of the three-dimensional point cloud of the surface contact, that is, the combination of surface geometric parameters per pixel, independent grayscale reference, grayscale-depth mapping, back projection of point cloud along ray and closed-loop error verification, rather than being limited to a certain front-end display program.

[0098] Based on the same inventive concept, this application also provides a three-dimensional point cloud construction device for a curved surface visual-tactile sensor, as described in the following embodiments. Since the principle of the three-dimensional point cloud construction device for a curved surface visual-tactile sensor is similar to that of the three-dimensional point cloud construction method for a curved surface visual-tactile sensor, the implementation of the three-dimensional point cloud construction device for a curved surface visual-tactile sensor can refer to the implementation of the three-dimensional point cloud construction method for a curved surface visual-tactile sensor; repeated details will not be elaborated further. As used below, the terms "unit" or "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0099] In a specific implementation, the three-dimensional point cloud construction device for a curved visual-tactile sensor provided in this application includes a gray-scale difference calculation module, a deformation depth calculation module, a parameter determination module, and a three-dimensional point cloud construction module that are connected in sequence via communication.

[0100] The grayscale difference calculation module is used to calculate a grayscale difference map based on a reference grayscale image and a third real-time grayscale image. The third real-time grayscale image is a grayscale image output by the image perception module when the curved visual-tactile sensor is in contact. The third real-time grayscale image and the reference grayscale image have the same size. The reference grayscale image is the grayscale image corresponding to when the curved visual-tactile sensor is in non-contact. The grayscale value of each pixel in the grayscale difference map is the difference between the grayscale value of the corresponding pixel in the reference grayscale image and the grayscale value of the corresponding pixel in the third real-time grayscale image.

[0101] The deformation depth calculation module is used to calculate a local deformation depth map based on the grayscale difference map and the depth coefficient, wherein the depth coefficient is the depth change corresponding to a unit grayscale change.

[0102] The parameter determination module is used to determine the surface geometric parameters corresponding to each pixel in the local deformation depth map. The surface geometric parameters include a reference surface depth, a pixel coordinate ray, and the position of a preset point in the image perception module in a preset coordinate system. The reference surface depth is the depth corresponding to each pixel in the non-contact state of the surface visual tactile sensor, representing the depth of the contact surface of the surface visual tactile sensor along the corresponding pixel coordinate ray in the non-contact state. The pixel coordinate ray represents the unit direction in which the preset point in the image perception module passes through the pixel and points to the surface point. The pixel coordinate ray is used to convert the local deformation depth of each pixel into a three-dimensional local depth perturbation along the direction of the pixel coordinate ray. The surface geometric parameters correspond to different pixels in different display scenarios. The 3D point cloud construction module is used to calculate the position of each pixel in a preset coordinate system based on the local deformation depth of each pixel in the local deformation depth map and the surface geometric parameters corresponding to each pixel, thereby realizing the reconstruction of the 3D point cloud of surface contact.

[0103] In specific implementation, this application also provides a computer device, such as... Figure 3 As shown, it includes a memory 201 and a processor 202 connected in sequence. The memory stores a computer program, and the processor is used to read the computer program and execute the method for constructing a three-dimensional point cloud of a curved visual-tactile sensor as described in the first aspect.

[0104] In specific implementations, this application also provides a computer-readable storage medium storing a method for constructing a three-dimensional point cloud of a curved surface visual-tactile sensor, as described in any of the first aspects. Specifically, the computer-readable storage medium includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined herein, computer-readable storage media does not include transient media, such as modulated data signals and carrier waves.

[0105] Obviously, those skilled in the art should understand that the modules or steps of the embodiments of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0106] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for constructing a three-dimensional point cloud for a curved surface visual-tactile sensor, characterized in that, include: A grayscale difference image is calculated based on a reference grayscale image and a third real-time grayscale image. The third real-time grayscale image is the grayscale image output by the image perception module when the curved visual-tactile sensor is in contact. The third real-time grayscale image and the reference grayscale image have the same size. The reference grayscale image is the grayscale image corresponding to when the curved visual-tactile sensor is in non-contact. The grayscale value of each pixel in the grayscale difference image is the difference between the grayscale value of the corresponding pixel in the reference grayscale image and the grayscale value of the corresponding pixel in the third real-time grayscale image. A local deformation depth map is calculated based on the grayscale difference map and the depth coefficient, wherein the depth coefficient is the depth change corresponding to a unit grayscale change. For each pixel in the local deformation depth map, the surface geometric parameters corresponding to each pixel are determined. The surface geometric parameters include a reference surface depth, a pixel coordinate ray, and the position of a preset point in the image perception module in a preset coordinate system. The reference surface depth is the depth corresponding to each pixel when the surface visual tactile sensor is in an uncontacted state, representing the depth of the contact surface of the surface visual tactile sensor along the corresponding pixel coordinate ray in the uncontacted state. The pixel coordinate ray represents the unit direction in which the preset point in the image perception module passes through the pixel and points to the surface point. The pixel coordinate ray is used to convert the local deformation depth of each pixel into a three-dimensional local depth perturbation along the direction of the pixel coordinate ray. The surface geometric parameters correspond to different pixels in different display scenarios. Based on the local deformation depth of each pixel in the local deformation depth map and the surface geometry parameters corresponding to each pixel, the position of each pixel in the preset coordinate system is calculated, thereby realizing the reconstruction of the three-dimensional point cloud of surface contact.

2. The method according to claim 1, characterized in that, Based on the local deformation depth of each pixel in the local deformation depth map and the corresponding surface geometry parameters of each pixel, the position of each pixel in a preset coordinate system is calculated, including: The real-time depth of each pixel is determined based on the local deformation depth of each pixel and the reference surface depth corresponding to each pixel. The position of each pixel in the preset coordinate system is calculated based on the real-time depth of each pixel, the pixel coordinate ray, and the position of the preset point in the image perception module in the preset coordinate system.

3. The method according to claim 1, characterized in that, Before calculating the grayscale difference map based on the reference grayscale image and the third real-time grayscale image, the method further includes: In response to receiving a calibration command, after a first preset number of sampling cycles of the image sensing module, the first real-time grayscale image of multiple frames output by the image sensing module in a non-contact state of the visual-tactile sensor is acquired. Calculate the average grayscale difference between two adjacent frames of the first real-time grayscale image in the multi-frame first real-time grayscale image; In response to the fact that the average grayscale difference between two adjacent frames of the first real-time grayscale image in the multi-frame first real-time grayscale image is less than a preset value or the image sensing module has reached the maximum number of warm-up frames after receiving a calibration command, the multi-frame second real-time grayscale image output by the image sensing module in the non-contact state of the visual tactile sensor is acquired. The median or average value of each pixel in the multi-frame second real-time grayscale image is calculated to obtain the reference grayscale image.

4. The method according to claim 1, characterized in that, For each pixel in the local deformation depth map, determine the surface geometry parameters corresponding to each pixel, including: The pixel coordinate ray corresponding to each pixel is determined by the following formula: ， In the formula, This represents the normalized pixel coordinate ray of pixel point (u,v) in the coordinate system of the image perception module. These are intrinsic parameters of the image perception module. For the transpose of a matrix or vector, Let the magnitude of the vector be . , These are the x and y coordinates of the pixel in the coordinate system of the image perception module, respectively; The depth of the reference surface corresponding to each pixel is determined by the following formula: ， ， ， In the formula, This represents the normalized pixel coordinate ray of pixel point (u,v) in the basic coordinate system of the curved visual-tactile sensor. Let be the rotation matrix from the coordinate system of the image perception module to the basic coordinate system of the curved visual-tactile sensor. Here are the coordinates of the optical center of the image sensing module in the basic coordinate system of the curved visual-tactile sensor. The surface of the curved surface model in a non-contact state. Let (u,v) be the reference surface depth, min be the minimum value, and t be the value of any point on the pixel coordinate ray relative to the starting point. The amount of displacement.

5. The method according to claim 2, characterized in that, The step of calculating the position of each pixel in the preset coordinate system based on the real-time depth of each pixel, the pixel coordinate ray, and the position of the preset point in the image perception module in the preset coordinate system includes: ， in, The position of each pixel in the preset coordinate system. This refers to the position of a preset point in the image perception module within a preset coordinate system. The real-time depth of each pixel. The pixel coordinate ray corresponding to each pixel point.

6. The method according to any one of claims 1 to 5, characterized in that, After calculating the local deformation depth map based on the grayscale difference map and the depth coefficient, the method further includes: For pixels in the local deformation depth map whose local deformation depth is greater than a preset depth, the pixel coordinate rays corresponding to the pixel are weighted and accumulated using the local deformation depth of the pixel as the weight to obtain the relative force vector. : ， in, For display or control purposes, a proportional coefficient is used. This represents the local deformation depth of this pixel in the local deformation depth map. For preset depth, The pixel coordinate ray corresponding to each pixel point.

7. The method according to any one of claims 1 to 5, characterized in that, The calculation of the local deformation depth map based on the grayscale difference map and depth coefficient includes: The local deformation depth map is obtained by multiplying the gray value of each pixel in the gray-level difference map with the corresponding depth coefficient.

8. A three-dimensional point cloud construction device for a curved surface visual-tactile sensor, characterized in that, It includes a grayscale difference calculation module, a deformation depth calculation module, a parameter determination module, and a 3D point cloud construction module, which are connected in sequence. The grayscale difference calculation module is used to calculate a grayscale difference map based on a reference grayscale image and a third real-time grayscale image. The third real-time grayscale image is a grayscale image output by the image perception module when the curved visual-tactile sensor is in contact. The third real-time grayscale image and the reference grayscale image have the same size. The reference grayscale image is the grayscale image corresponding to when the curved visual-tactile sensor is in non-contact. The grayscale value of each pixel in the grayscale difference map is the difference between the grayscale value of the corresponding pixel in the reference grayscale image and the grayscale value of the corresponding pixel in the third real-time grayscale image. The deformation depth calculation module is used to calculate a local deformation depth map based on the grayscale difference map and the depth coefficient, wherein the depth coefficient is the depth change corresponding to a unit grayscale change. The parameter determination module is used to determine the surface geometric parameters corresponding to each pixel in the local deformation depth map. The surface geometric parameters include a reference surface depth, a pixel coordinate ray, and the position of a preset point in the image perception module in a preset coordinate system. The reference surface depth is the depth corresponding to each pixel in the non-contact state of the surface visual tactile sensor, representing the depth of the contact surface of the surface visual tactile sensor along the corresponding pixel coordinate ray in the non-contact state. The pixel coordinate ray represents the unit direction in which the preset point in the image perception module passes through the pixel and points to the surface point. The pixel coordinate ray is used to convert the local deformation depth of each pixel into a three-dimensional local depth perturbation along the direction of the pixel coordinate ray. The surface geometric parameters correspond to different pixels in different display scenarios. The 3D point cloud construction module is used to calculate the position of each pixel in a preset coordinate system based on the local deformation depth of each pixel in the local deformation depth map and the surface geometric parameters corresponding to each pixel, thereby realizing the reconstruction of the 3D point cloud of surface contact.

9. A computer device comprising a memory and a processor connected in sequence and communicatively, wherein the memory stores a computer program, characterized in that: The processor is used to read the computer program and execute the method for constructing a three-dimensional point cloud of a curved visual-tactile sensor according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a method for constructing a three-dimensional point cloud of a curved visual-tactile sensor according to any one of claims 1 to 7.

Images