Detection method and device of visual tactile sensor, visual tactile sensor and medium

Abstract

The embodiment of the application discloses a kind of detection method, device, visual tactile sensor and medium of visual tactile sensor, method includes: when reflecting elastomer is contacted with target object and is pressed, first coordinate set corresponding to first mirror and second coordinate set corresponding to second mirror are extracted from the target image collected;From two coordinate sets, the target coordinate pair corresponding to the same mark point of multiple groups is extracted;The absolute three-dimensional world coordinates of corresponding mark point are calculated based on target coordinate pair;Based on the dense gradient field of reflecting elastomer surface and absolute three-dimensional world coordinates, energy functional is constructed, and dense depth map is obtained by solving energy functional.This application detects dense depth map by fusing absolute three-dimensional world coordinates and dense gradient field, realizes that visual tactile sensor carries out high-precision, depth map detection of fine texture between mark points after contact target object, compared with prior art, can realize high-precision absolute depth measurement, and measure fine texture between mark points.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of visual-tactile sensor technology, and in particular to a detection method, apparatus, visual-tactile sensor, and medium for a visual-tactile sensor. Background Technology

[0002] In the current field of embodied intelligence and highly dexterous robotics, visual-tactile sensors can simultaneously perceive microscopic geometry, contact force distribution, and sliding friction states. The underlying physical mechanism of visual-tactile sensors lies in using a high-resolution optical imaging system to observe the mechanical deformation of the elastomeric layer (usually a multi-layer silicone composite material) covering a transparent rigid substrate when it comes into contact with an external object. By inversely solving this deformation process, the physical state of the elastomeric layer contact interface can be reconstructed.

[0003] Existing surface reconstruction technologies based on visual-tactile sensors are mainly based on photometric theory or geometric stereo vision theory. Among them, the most typical applications are photometric stereo and binocular stereo vision. However, these two technical approaches have inherent contradictions and limitations in information acquisition: although photometric stereo can achieve high-resolution reconstruction of the fine texture of an object's surface, it is essentially a relative depth recovery and cannot perform high-precision absolute depth measurement; while binocular stereo vision can provide stable, drift-free absolute depth measurement with sub-millimeter accuracy, it is limited by the Nyquist sampling theorem of optical imaging and the limitations of marker physical manufacturing, and cannot effectively capture the micro-dense texture information between markers. Summary of the Invention

[0004] This application provides a detection method, device, sensor, and medium for a visual-tactile sensor, which enables the visual-tactile sensor to achieve high-precision absolute depth measurement while simultaneously measuring fine texture information between marker points on the contact surface.

[0005] In a first aspect, embodiments of this application provide a detection method for a visual-tactile sensor, applied to a visual-tactile sensor. The visual-tactile sensor includes a housing with a sensing window at the top, a monocular camera, a first reflector, a second reflector, a reflective elastomer with a marked dot array on its surface, and a transparent optical transmission component with multiple light source components arranged circumferentially. The monocular camera, the first reflector, and the second reflector are all located within the housing, with the first reflector and the second reflector forming an angle. The transparent optical transmission component is embedded within the sensing window. The side of the transparent optical transmission component facing the inside of the housing serves as a light refraction surface opposite to the reflector, while the side facing the outside of the housing is fixedly connected to the lower surface of the reflective elastomer. The surfaces of the first reflector and the second reflector are located within the field of view of the monocular camera. The method includes:

[0006] When the reflective elastomer comes into contact with the target object and is compressed, the target image is acquired by the monocular camera, and the first coordinate set of the marker point array imaged by the first reflector and the second coordinate set imaged by the second reflector are extracted from the target image respectively.

[0007] Multiple sets of target coordinate pairs are extracted from the first coordinate set and the second coordinate set. Each set of target coordinate pairs includes the first coordinate and the second coordinate corresponding to the same marker point in the marker point array. The first coordinate belongs to the first coordinate set and the second coordinate belongs to the second coordinate set.

[0008] Based on any of the target coordinate pairs, calculate the absolute three-dimensional world coordinates of the corresponding marker point;

[0009] The dense gradient field on the surface of the reflective elastomer is obtained based on the normal estimation model;

[0010] An energy functional is constructed based on the dense gradient field and the absolute three-dimensional world coordinates. The dense depth map is obtained by solving the energy functional. The dense depth map can characterize the three-dimensional absolute deformation of the contact surface of the reflective elastomer.

[0011] In some embodiments, acquiring target images via the monocular camera includes:

[0012] When the light source assembly includes a first light source assembly, a second light source assembly, a third light source assembly, and a fourth light source assembly, and all of the light source assemblies are monochromatic LED arrays, the first light source assembly, the second light source assembly, the third light source assembly, and the fourth light source assembly are lit sequentially;

[0013] Using a preset sliding time window, the monocular camera is controlled to capture a first original image, a second original image, a third original image, and a fourth original image with different timestamps, respectively. The first original image is the image captured when the first light source component is lit, the second original image is the image captured when the second light source component is lit, the third original image is the image captured when the third light source component is lit, and the fourth original image is the image captured when the fourth light source component is lit.

[0014] Subtract the sub-reference image captured in advance in an unloaded, contactless state when the corresponding light source component in the same orientation is lit from the first original image, the second original image, the third original image, and the fourth original image of each captured frame, respectively, to obtain the corresponding first difference image, second difference image, third difference image, and fourth difference image.

[0015] The first difference image, the second difference image, the third difference image, and the fourth difference image are determined as the target image;

[0016] or,

[0017] When the light source component includes a red LED array, a green LED array, and a blue LED array, and the red LED array, the green LED array, and the blue LED array are lit simultaneously, the image captured by the monocular camera is determined as the target image.

[0018] In some embodiments, extracting a first coordinate set of the marker point array imaged via a first mirror and a second coordinate set imaged via a second mirror from the target image includes:

[0019] Based on the pre-calibrated first and second effective detection polygons, the corresponding first left half image and first right half image are obtained from the target image;

[0020] Perform perspective correction transformations on the first left half image and the first right half image respectively to obtain the corresponding first front view and second front view;

[0021] The first front view and the second front view are sequentially subjected to nonlocal mean denoising, adaptive Gaussian threshold segmentation, morphological opening operation processing, and morphological closing operation processing to obtain the corresponding first binary image and second binary image.

[0022] Connectivity analysis is performed on the first binary image and the second binary image to determine multiple first connected components retained by the first binary image and multiple second connected components retained by the second binary image, wherein each first connected component or second connected component uniquely corresponds to one of the marked points;

[0023] Determine the first image moments of each first connected component, and calculate the corresponding first coordinates based on each first image moment;

[0024] Determine the second image moments for each of the second connected components, and calculate the corresponding second coordinates based on each of the second image moments.

[0025] In some embodiments, before extracting multiple sets of target coordinate pairs from the first coordinate set and the second coordinate set, the method further includes:

[0026] Centered on each reference node in the reference image corresponding to the marked point, a grayscale template of the first size is defined. The reference image is the image captured by the monocular camera when the reflective elastomer is not in contact with the target object and is under pressure. Each reference node corresponds to a reference topological coordinate.

[0027] Centered on each of the reference nodes, a search area of ​​a second size is defined, and the coordinate points corresponding to the first coordinates or the second coordinates that fall within the search area are taken as candidate points, wherein the first size is smaller than the second size;

[0028] For the grayscale template corresponding to the same marker point, NCC matching is performed one by one with the image block of the first size centered on each candidate point to obtain multiple matching results. When the NCC correlation coefficient is greater than the preset coefficient threshold, the matching result is determined to be a successful match, and the reference node and the candidate point are assigned one-to-one according to the NCC correlation coefficient from high to low.

[0029] When the number of successfully matched results is less than the number of reference nodes, local interpolation is performed on the first coordinates associated with the markers of successfully matched points adjacent to the failed match reference nodes based on a preset grid step vector to obtain the corresponding first predicted coordinates. Similarly, local interpolation is performed on the second coordinates associated with the markers of successfully matched points adjacent to the failed match reference nodes based on the grid step vector to obtain the corresponding second predicted coordinates. When invalid geometry exists in the neighborhood of any failed match reference node, a first homography matrix is ​​fitted based on the first coordinates associated with each successfully matched marker and the corresponding reference topological coordinates. This first homography matrix is ​​then used to... The first predicted coordinates are determined by a matrix. A second homography matrix is ​​obtained by fitting the second coordinates associated with the successfully matched markers and the corresponding reference topological coordinates of each matching result. The second predicted coordinates are determined by the second homography matrix. All the first predicted coordinates and the successfully matched first coordinates are combined to obtain a new first coordinate set. The reference topological coordinates of the reference nodes corresponding to each coordinate point in the new first coordinate set are determined as the first topological coordinates of the corresponding coordinate points themselves. All the second predicted coordinates and the successfully matched second coordinates are combined to obtain a new second coordinate set. The reference topological coordinates of the reference nodes corresponding to each coordinate point in the new second coordinate set are determined as the second topological coordinates of the corresponding coordinate points themselves.

[0030] When the number of successfully matched results is equal to the number of reference nodes, the original first coordinate set and second coordinate set are retained. The reference topological coordinates of the reference node corresponding to the coordinate point of the first coordinate are determined as the first topological coordinates of the coordinate point of the first coordinate itself. The reference topological coordinates of the reference node corresponding to the coordinate point of the second coordinate are determined as the second topological coordinates of the coordinate point of the second coordinate itself.

[0031] In some embodiments, each of the reference nodes of the reference image is obtained according to the following steps:

[0032] Based on the pre-calibrated first and second effective detection polygons, the second left half image and the second right half image are obtained from the reference image;

[0033] The second left half image and the second right half image are respectively subjected to perspective correction transformation to obtain the corresponding third front view and fourth front view;

[0034] The third front view and the fourth front view are sequentially subjected to nonlocal mean denoising, adaptive Gaussian thresholding, morphological opening operation, and morphological closing operation to obtain the corresponding third binary image and fourth binary image.

[0035] Connectivity analysis is performed on the third binary image and the fourth binary image to determine multiple third connected components retained by the third binary image and multiple fourth connected components retained by the fourth binary image, wherein each third connected component or the fourth connected component uniquely corresponds to a marker point;

[0036] Determine the third image moments of each of the third connected components, calculate the corresponding third coordinates based on each of the third image moments, and form a third coordinate set from all the third coordinates;

[0037] Determine the fourth image moments of each of the fourth connected components, calculate the corresponding fourth coordinates based on each of the fourth image moments, and form a fourth coordinate set from all the fourth coordinates;

[0038] Calculate the first nearest neighbor distance for each of the third coordinates, determine the median of the first nearest neighbor distance as the first reference distance, perform denoising processing on the third coordinate set based on the first reference distance, let the marker point corresponding to each first effective coordinate in the denoised third coordinate set find the nearest neighbor in the first target direction, construct multiple first pointer structures, take the coordinate point corresponding to the first pointer structure with the highest pointer connectivity as the root node of the graph and perform breadth-first search (BFS), assign a first reference topology coordinate in the form of row and column numbers along the first target direction to construct a first grid topology, and perform coordinate zeroing processing on the first grid topology, wherein the first nearest neighbor distance is the distance between any third coordinate and the corresponding first nearest coordinate, and the first nearest coordinate is the coordinate point in the third coordinate set that is closest to the third coordinate other than itself;

[0039] Calculate the second nearest neighbor distance for each fourth coordinate in the fourth coordinate set, and determine the median of the second nearest neighbor distance as the second reference distance. Based on the second reference distance, perform denoising processing on the fourth coordinate set. Let the marker point corresponding to each second effective coordinate in the denoised fourth coordinate set find its nearest neighbor in the second target direction, construct multiple second pointer structures, and use the coordinate point corresponding to the second pointer structure with the highest pointer connectivity as the root node of the graph to perform BFS. Assign second reference topology coordinates in the form of row and column numbers along the second target direction to construct a second grid topology, and perform coordinate zeroing processing on the second grid topology. The second nearest neighbor distance is the distance between any fourth coordinate and the corresponding second nearest coordinate, and the second nearest coordinate is the coordinate point in the fourth coordinate set that is closest to the fourth coordinate other than itself.

[0040] When the number of nodes in the first grid topology or the number of nodes in the second grid topology is less than the number of marker points, a third homography matrix is ​​fitted by using the correspondence between the third coordinate associated with the reference topology coordinates of all known nodes in the first grid topology and the corresponding reference topology coordinates. The third homography matrix is ​​then used to determine the third predicted coordinates corresponding to the missing nodes. All the third predicted coordinates are combined with the third coordinates of all known nodes in the first grid topology to obtain a fifth coordinate set. A fourth homography matrix is ​​fitted by using the correspondence between the fourth coordinate associated with the reference topology coordinates of all known nodes in the second grid topology and the corresponding reference topology coordinates. The fourth homography matrix is ​​then used to determine the fourth predicted coordinates corresponding to the missing nodes. All the fourth predicted coordinates are combined with the fourth coordinates of all known nodes in the second grid topology to obtain a sixth coordinate set.

[0041] When the number of nodes in the first grid topology and the number of nodes in the second grid topology are equal to the number of marker points, the coordinate points corresponding to all the first reference topology coordinates and all the second reference topology coordinates are determined as the reference nodes.

[0042] In some embodiments, extracting multiple sets of target coordinate pairs from the first coordinate set and the second coordinate set includes:

[0043] Determine the first number of coordinate points in all the second topological coordinates that have the same coordinates as all the first topological coordinates;

[0044] All reference coordinate point pairs are divided according to column offset to obtain multiple reference coordinate sets. Each pair of reference coordinate points in each reference coordinate set has the same column offset, and any pair of reference coordinate points consists of a first topological coordinate and a second topological coordinate with the same topological row number. The column offset is obtained by subtracting the column number of the first topological coordinate from the column number of the second topological coordinate of any pair of reference coordinate points in the corresponding reference coordinate set.

[0045] Calculate the horizontal disparity for each pair of reference coordinate points, wherein the horizontal disparity is obtained by subtracting the column pixel coordinates of the second topological coordinate from the column pixel coordinates of the first topological coordinate of the corresponding group;

[0046] Determine the horizontal pixel column distance between the coordinate points of adjacent columns in the first topological coordinates of the same topological row number, and determine the median of all the horizontal pixel column distances as the reference column distance;

[0047] Calculate the median of all horizontal disparities in each of the reference coordinate sets to obtain the median of each disparity;

[0048] Traverse all the column offsets, calculate the absolute value of the difference between each horizontal disparity associated with any column offset and the corresponding median disparity, and when the absolute value is less than the product of the reference column distance and a preset value, increase the number of inliers of the corresponding column offset by 1.

[0049] The column offset corresponding to the largest number of interior points is determined as the target column offset, and all second topological coordinates are column offset using the target column offset.

[0050] Determine the second number of coordinate points that have the same coordinates in the first topological coordinates and the second topological coordinates after the offset operation;

[0051] When the second quantity is greater than the first quantity, the coordinate point that is the same as the first topological coordinate and the second topological coordinate after the offset operation is taken as the target matching result. When the second quantity is less than or equal to the first quantity, the coordinate point that is the same as the first topological coordinate and the second topological coordinate before the offset operation is taken as the target matching result. The target matching result includes multiple sets of target coordinate pairs.

[0052] In some embodiments, calculating the absolute three-dimensional world coordinates of the corresponding marker point based on any of the target coordinate pairs includes:

[0053] The target coordinate pairs are mapped to the original camera image coordinate system of the monocular camera using the inverse perspective transformation matrix.

[0054] The Brown-Conrady distortion model is used to perform anti-distortion correction on the target coordinate pairs after mapping to obtain the distortion-free intermediate coordinate pairs.

[0055] The intermediate coordinates are back-projected into the first unit ray and the second unit ray in the camera coordinate system of the monocular camera;

[0056] Based on Snell's law, the first unit ray and the second unit ray are refraction corrected, and the corrected first unit ray and second unit ray are transformed into the corresponding first reference ray and second reference ray in the world coordinate system.

[0057] Solve for the target parameter that minimizes the distance between the first reference ray and the second reference ray, wherein the target parameter includes a first parameter and a second parameter;

[0058] A first target point is determined from the first reference ray based on the first parameter, and a second target point is determined from the second reference ray based on the second parameter, wherein the distance between the first reference ray and the second reference ray is the shortest at the position of the first target point, and the distance between the second reference ray and the first reference ray is the shortest at the position of the second target point.

[0059] The midpoint between the first target point and the second target point is determined as the absolute three-dimensional world coordinates of the marker point.

[0060] In some embodiments, obtaining the dense gradient field of the reflective elastomer surface based on a normal estimation model includes:

[0061] Dimensionality enhancement is performed on each pixel of the first front view to obtain a first feature image, wherein each pixel of the first feature image includes a vector of color dimension and a vector of spatial coordinate dimension.

[0062] The first feature image is input into the normal estimation model to perform normal estimation, and a first component and a second component are output. The first component is the tilt component of the normal vector of the reflective elastomer surface in the x-axis direction, and the second component is the tilt component of the normal vector of the reflective elastomer surface in the y-axis direction.

[0063] Based on the first component and the second component, the third component is obtained by inversely solving the normalization constraint condition of the unit normal vector, wherein the third component is the component of the normal vector of the reflective elastomer surface in the z-axis direction.

[0064] The third component is subjected to a lower cutoff process, and the dense gradient field is calculated pixel by pixel based on the first component, the second component, and the lower cutoff third component.

[0065] In some embodiments, obtaining the dense gradient field of the reflective elastomer surface based on a normal estimation model includes:

[0066] Based on a pre-set registration lookup table, the first right half image is registered pixel by pixel to the coordinate system of the first left half image by combining an image remapping algorithm and a bilinear interpolation algorithm to obtain a third right half image. The registration lookup table is used to indicate the floating-point pixel corresponding to each pixel in the first left half image in the first right half image.

[0067] A perspective correction transformation is performed on the effective area of ​​the third right half image to obtain the fifth front view;

[0068] Dimensionality enhancement is performed on each pixel of the first front view to obtain a first feature image, wherein each pixel of the first feature image includes a vector of color dimension and a vector of spatial coordinate dimension.

[0069] Dimension-enhancing processing is performed on each pixel of the fifth front view to obtain a second feature image, wherein each pixel of the second feature image includes a vector of color dimension and a vector of spatial coordinate dimension;

[0070] The first feature image and the second feature image are combined to obtain the third feature image;

[0071] The third feature image is input into the normal estimation model for normal estimation, and the first component and the second component are output. The first component is the tilt component of the normal vector of the reflective elastomer surface in the x-axis direction, and the second component is the tilt component of the normal vector of the reflective elastomer surface in the y-axis direction.

[0072] Based on the first component and the second component, the third component is obtained by inversely solving the normalization constraint condition of the unit normal vector, wherein the third component is the component of the normal vector of the reflective elastomer surface in the z-axis direction.

[0073] The third component is subjected to a lower cutoff process, and the dense gradient field is calculated pixel by pixel based on the first component, the second component, and the lower cutoff third component.

[0074] In some embodiments, before obtaining the dense gradient field of the reflective elastomer surface based on the normal estimation model, the method further includes:

[0075] After mapping the target coordinate pair to the original camera image coordinate system using the inverse perspective transformation matrix, the first coordinate in the mapped target coordinate pair is remapped back to the front view coordinate system where the first front view is located based on the inverse perspective transformation matrix to obtain the intermediate topological coordinates.

[0076] Calculate the third nearest neighbor distance for each of the intermediate topological coordinates, and determine the median of all the third nearest neighbor distances as the third reference distance;

[0077] Calculate the first radius based on the third reference spacing, and draw the binary occlusion mask corresponding to each of the intermediate topological coordinates using the first radius;

[0078] For the first front view, the Fast Marching Method (FMM) is used to fill in the corresponding binary occlusion mask layer by layer from the boundary inward to obtain the repaired first front view.

[0079] In some embodiments, an energy functional is constructed based on the dense gradient field and the absolute three-dimensional world coordinates, and the dense depth map is obtained by solving the energy functional, including:

[0080] Based on the variational Poisson optimization framework, the energy functional is constructed by combining the dense gradient field and the absolute three-dimensional world coordinates.

[0081] The objective constraint terms in the energy functional are extracted for variational extremum conditions and discretized to obtain multiple linear equations. All the linear equations are stacked to obtain a sparse matrix. Based on the sparse matrix, the objective normal matrix is ​​constructed, and the objective normal equation is solved using the preprocessed conjugate gradient method to obtain the dense depth map.

[0082] In some embodiments, the target constraint term includes a static constraint term, which includes a gradient constraint term and a reference constraint term. The gradient constraint term is used to constrain the gradient approximation of the dense depth map to the corresponding absolute dense gradient field. The reference constraint term is a smoothing constraint term or a material constraint term. The smoothing constraint term is used to constrain the local curvature of the dense depth map. The material constraint term is used to constrain the geometric deformation of the reflective elastomer to conform to real physical laws. The target constraint term in the energy functional is subjected to variational extremum conditions and discretization to obtain multiple linear equations. All the linear equations are stacked to obtain a sparse matrix, including:

[0083] With the current target image as the first frame, the static constraint terms are subjected to variational extreme value conditions and discretization processing to obtain multiple linear equations. All the linear equations are stacked to obtain a static sparse matrix. The corresponding static normal matrix is ​​calculated based on the static sparse matrix. The static sparse matrix and the static normal matrix are saved to the cache area.

[0084] If the current target image is not the first frame image, but the grid size and / or the constraint weights of the static constraint terms have changed, delete the existing data in the buffer, re-extract the variational extreme value conditions and discretize the static constraint terms to obtain multiple new linear equations, stack all the new linear equations to obtain a new static sparse matrix, calculate the corresponding new static normal matrix based on the new static sparse matrix, and save the new static sparse matrix and the new static normal matrix to the buffer. The grid size is the grid size formed by all the anchor points corresponding to the absolute three-dimensional world coordinates.

[0085] If the current target image is not the first frame image, and the grid size and the constraint weights of the static constraint terms have not changed, the static sparse matrix and the static normal matrix in the buffer are retained and reused.

[0086] In some embodiments, the energy functional further includes an anchor constraint term, which is used to constrain the depth value at any marker point in the dense depth map to approximate the absolute depth value of the corresponding absolute 3D world coordinates. A target normal matrix is ​​constructed based on a sparse matrix, and the target normal equation is solved using the preprocessed conjugate gradient method to obtain the dense depth map, including:

[0087] For the anchor point constraint term, bilinear interpolation discretization is performed based on the position of the corresponding anchor point in the grid to construct the anchor point sub-matrix, and the anchor point normal matrix corresponding to the anchor point sub-matrix is ​​calculated.

[0088] By superimposing the anchor point normal matrix with the valid static normal matrix in the buffer, the target normal matrix corresponding to the current target image is obtained;

[0089] Construct a diagonal Jacobi preprocessor based on the diagonal of the target matrix;

[0090] Based on the diagonal Jacobi preprocessor, perform preprocessing conjugate gradient iteration on the target normal matrix and record the number of iterations;

[0091] When the number of iterations reaches the number threshold, or when the relative residual of the solution for the current iteration is less than or equal to the first tolerance threshold, the iteration stops and a first intermediate depth map is output, wherein the relative residual is used to indicate the reduction ratio of the residual of the current solution of the conjugate gradient iteration relative to the initial residual;

[0092] The first intermediate depth map is adjusted according to the preset size to obtain the dense depth map.

[0093] In some embodiments, solving the energy functional to obtain a dense depth map includes:

[0094] The dense gradient field is scaled by a downsampling factor and the grid step size is increased to generate a reference grid, wherein the anchor points corresponding to the absolute three-dimensional world coordinates remain unchanged in their original coordinates;

[0095] The energy functional is iteratively solved on the reference grid. If the relative residual corresponding to the solution of the current iteration number is less than or equal to the second tolerance threshold, the first depth map is output.

[0096] The first depth map is bilinearly upsampled back to the original resolution to obtain the second depth map;

[0097] Using the second depth map as the initial solution, the energy functional is iteratively solved at the original resolution to obtain the first intermediate depth map;

[0098] The first intermediate depth map is adjusted according to the preset size to obtain the dense depth map.

[0099] In some embodiments, solving the energy functional to obtain a dense depth map includes:

[0100] When the distance between the absolute 3D world coordinates of any of the marked points in the current target image and the absolute 3D world coordinates of the target image in the previous frame is less than a preset distance threshold, the first intermediate depth map obtained by solving the energy functional of the target image in the previous frame is used as the second intermediate depth map. The second intermediate depth map is used as the initial solution, and the energy functional of the current target image is iteratively solved at the original resolution to obtain the first intermediate depth map of the current target image. The first intermediate depth map is then adjusted according to a preset size to obtain the dense depth map.

[0101] When the distance between the absolute 3D world coordinates of any marker point in the current target image and the absolute 3D world coordinates of the target image in the previous frame is greater than or equal to a preset distance threshold, the dense gradient field corresponding to the current target image is scaled according to a downsampling factor, and the grid step size is increased to regenerate a new reference grid. The energy functional is iteratively solved on the new reference grid, and if the relative residual corresponding to the solution of the current iteration number is less than or equal to a second tolerance threshold, a new first depth map is output. The new first depth map is bilinearly upsampled back to the original resolution to obtain a new second depth map. The energy functional is iteratively solved at the original resolution using the new second depth map as the initial solution to obtain a new first intermediate depth map. The new first intermediate depth map is adjusted according to a preset size to obtain the dense depth map.

[0102] Secondly, embodiments of this application provide a control device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the detection method of the visual-tactile sensor as described in the first aspect.

[0103] Thirdly, embodiments of this application provide a visual-tactile sensor, including the control device of the second aspect embodiment.

[0104] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a detection method for a visual-tactile sensor as described in the first aspect.

[0105] This application provides a detection method, device, sensor, and medium for a visual-tactile sensor. The method includes: when a reflective elastomer is in contact with and pressurized against a target object, acquiring a target image using a monocular camera, and extracting a first coordinate set and a second coordinate set from the target image, whereby the marker array is imaged by a first reflector and by a second reflector; extracting multiple sets of target coordinate pairs from the first and second coordinate sets, wherein each set of target coordinate pairs includes a first coordinate and a second coordinate corresponding to the same marker point in the marker array, the first coordinate belonging to the first coordinate set and the second coordinate belonging to the second coordinate set; calculating the absolute three-dimensional world coordinates of the corresponding marker point based on any one of the target coordinate pairs; obtaining a dense gradient field on the surface of the reflective elastomer based on a normal estimation model; constructing an energy functional based on the dense gradient field and the absolute three-dimensional world coordinates, and solving the energy functional to obtain a dense depth map, wherein the dense depth map can characterize the three-dimensional absolute deformation of the contact surface of the reflective elastomer. According to the solution provided in the embodiments of this application, the absolute depth calculated by the binocular stereo vision method is fused with the dense gradient field obtained based on the normal estimation model in a multimodal manner to achieve high-precision depth map detection of the visual-tactile sensor after contacting the target object, which preserves the fine texture between the marker points. Compared with the existing solutions that only use the binocular stereo vision method or the photometric stereo method, this solution effectively solves the problem of cumulative error in surface reconstruction of the photometric stereo method and the problem of texture loss between marker points in the binocular stereo vision method. Attached Figure Description

[0106] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0107] Figure 1 This is a schematic diagram of the structure of a visual-tactile sensor provided in one embodiment of this application;

[0108] Figure 2 Provided for another embodiment of this application Figure 1 The diagram shows a top view of the structure of the visual-tactile sensor.

[0109] Figure 3 Provided for another embodiment of this application Figure 1 The diagram shows a top view of the structure of the visual-tactile sensor.

[0110] Figure 4 A flowchart illustrating the steps of a detection method using a visual-tactile sensor, provided as another embodiment of this application;

[0111] Figure 5 A schematic diagram of a target image provided for another embodiment of this application;

[0112] Figure 6 A schematic diagram of a first front view provided for another embodiment of this application;

[0113] Figure 7 A schematic diagram of a second front view provided for another embodiment of this application;

[0114] Figure 8 This application provides a gradient reconstruction map that can display a dense gradient field, as another embodiment of the present application.

[0115] Figure 9 A 3D dense depth map provided for another embodiment of this application;

[0116] Figure 10 A 2D dense depth map with color swatches provided in another embodiment of this application;

[0117] Figure 11 This is a structural diagram of a control device provided in another embodiment of this application.

[0118] Figure label:

[0119] Visor-tactile sensor 100; housing 110; monocular camera 120; first reflector 130; second reflector 140; reflective elastomer 150; marker point 151; transparent optical transmission component 160; light source component 1610; red LED array 1611; green LED array 1612; blue LED array 1613; first virtual camera 171; second virtual camera 172. Detailed Implementation

[0120] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0121] It is understandable that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0122] In the current field of embodied intelligence and highly dexterous robotics, visual-tactile sensors can simultaneously perceive microscopic geometry, contact force distribution, and sliding friction states. The underlying physical mechanism of visual-tactile sensors lies in using a high-resolution optical imaging system to observe the mechanical deformation of the elastomeric layer (usually a multi-layer silicone composite material) covering a transparent rigid substrate when it comes into contact with an external object. By inversely solving this deformation process, the physical state of the elastomeric layer contact interface can be reconstructed.

[0123] Existing surface reconstruction technologies based on visual-tactile sensors are mainly based on photometric theory or geometric stereo vision theory. Among them, the most typical applications are photometric stereo and binocular stereo vision. However, these two technical approaches have inherent contradictions and limitations in information acquisition: although photometric stereo can achieve high-resolution reconstruction of the fine texture of an object's surface, it is essentially a relative depth recovery and cannot perform high-precision absolute depth measurement; while binocular stereo vision can provide stable, drift-free absolute depth measurement with sub-millimeter accuracy, it is limited by the Nyquist sampling theorem of optical imaging and the limitations of marker physical manufacturing, and cannot effectively capture the micro-dense texture information between markers.

[0124] To address the aforementioned problems, this application provides a detection method, apparatus, sensor, and medium for a visual-tactile sensor. The method includes: when the reflective elastomer comes into contact with and is compressed against a target object, acquiring a target image using a monocular camera, and extracting a first coordinate set and a second coordinate set from the target image, whereby the marker array is imaged by a first reflector and by a second reflector; extracting multiple sets of target coordinate pairs from the first and second coordinate sets, wherein each set of target coordinate pairs includes a first coordinate and a second coordinate corresponding to the same marker point in the marker array, the first coordinate belonging to the first coordinate set and the second coordinate belonging to the second coordinate set; calculating the absolute three-dimensional world coordinates of the corresponding marker point based on any one of the target coordinate pairs; obtaining a dense gradient field on the surface of the reflective elastomer based on a normal estimation model; constructing an energy functional based on the dense gradient field and the absolute three-dimensional world coordinates, and solving the energy functional to obtain a dense depth map, wherein the dense depth map can characterize the three-dimensional absolute deformation of the contact surface of the reflective elastomer. According to the solution provided in the embodiments of this application, the absolute depth calculated by the binocular stereo vision method is fused with the dense gradient field obtained based on the normal estimation model in a multimodal manner to achieve high-precision depth map detection of the visual-tactile sensor after contacting the target object, which preserves the fine texture between the marker points. Compared with the existing solutions that only use the binocular stereo vision method or the photometric stereo method, this solution effectively solves the problem of cumulative error in surface reconstruction of the photometric stereo method and the problem of texture loss between marker points in the binocular stereo vision method.

[0125] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0126] This application provides a detection method for a visual-tactile sensor, which is applied to, for example... Figure 1 The visual-tactile sensor shown includes a housing 110 with a sensing window at the top, a monocular camera 120, a first reflector 130, a second reflector 140, a reflective elastomer 150 with an array of marker points (including multiple marker points 151) on its surface, and a transparent optical transmission component 160 with multiple light source components 1610 arranged circumferentially. The monocular camera 120, the first reflector 130, and the second reflector 140 are all located inside the housing 110, with an angle formed between the first reflector 130 and the second reflector 140. The transparent optical transmission component 160 is embedded in the sensing window. The side of the transparent optical transmission component 160 facing the inside of the housing 110 serves as a light refraction surface, opposite to the reflectors (i.e., the first reflector 130 and the second reflector 140). The side of the transparent optical transmission component 160 facing the outside of the housing 110 is fixedly connected to the lower surface of the reflective elastomer 150. The surfaces of the first reflector 130 and the second reflector 140 are within the field of view of the monocular camera 120.

[0127] Specifically, the housing in this embodiment adopts a compact design, which can be seamlessly embedded into industrial grippers and humanoid robotic hands or arms. The housing may be provided with an open contact window for placing transparent optical transmission components and reflective elastomers, allowing the elastic sensing layer (i.e., reflective elastomers) to directly contact external objects. In addition, this embodiment does not limit the specific size of the housing; those skilled in the art can determine it according to actual needs. For example, it can be set to a size of 35mm (height) × 18mm (width) × 15mm (depth).

[0128] Specifically, in this embodiment, the monocular camera 120, the first reflector 130, and the second reflector 140 are all located inside the housing 110. These two reflectors are fixedly arranged inside the housing 110 at a preset angle. (Refer to...) Figure 1 A sensing window is provided on the top of the housing 110, and a transparent optical transmission component 160 is embedded in the sensing window. The side of the transparent optical transmission component facing the inside of the housing 110 serves as a light refraction surface, opposite to the first reflector 130 and the second reflector 140. The side of the transparent optical transmission component facing the outside of the housing 110 is fixedly connected to the lower surface of the reflective elastomer 150. The surfaces of the first reflector 130 and the second reflector 140 are located within the field of view of the monocular camera 120. Based on this structure, when the light source component 1610 is lit, the optical diffuse reflection signal on the reflective elastomer 150 can be refracted into the camera lens of the monocular camera 120. Thus, based on the principle of optical path reversibility, this catadioptric system is optically equivalent to placing two virtual cameras on both sides of the elastic layer (e.g., Figure 1 The first virtual camera 171 and the second virtual camera 172 are shown in the image. The two mirrors have a preset spatial angle and relative position, which can form a sufficient baseline length to ensure the depth resolution in binocular stereo vision, and ensure that the array of marker points 151 can present sufficient parallax displacement in a single frame image captured by the monocular camera 120 (the left half of the image corresponds to the first virtual camera, and the right half of the image corresponds to the second virtual camera).

[0129] It should be noted that this embodiment does not limit the specific specifications and materials of the first and second reflectors. For example, they can be aluminum-based or silver-based reflectors with a mirror flatness of < λ / 4, where λ is the wavelength of light.

[0130] It should be noted that the embodiments of this application do not limit the light source component 1610 disposed inside the transparent optical transmission component 160, and can be as follows: Figure 2As shown, it includes a blue LED array 1613, a green LED array 1612, and a red LED array 1611. Correspondingly, three different-dimensional illumination direction fields are encoded in the three independent RGB color channels of the monocular camera 120. This design satisfies the constraint condition of having at least three independent illumination vectors required for photometric stereo reconstruction of surface geometry; it can also be as follows... Figure 3 As shown, the transparent optical transmission component 160 is surrounded by independent monochromatic light source components 1610, which can be red LED arrays or white LED arrays, without the need for RGB color encoding on the monocular camera 120.

[0131] It should be noted that this embodiment does not limit the specific method of manufacturing the reflective elastomer. It can be done by pouring a 0.2mm thick silver silicone into the bottom of the shell, and after the silver silicone has cured, using an array mold and a spraying tool to form a regularly arranged array of black marker dots on the surface of the silver silicone (each marker dot has a diameter of 0.3mm and the spacing between adjacent marker dots is 1.5mm to 2mm).

[0132] It should be noted that this embodiment does not limit the specific material of the transparent optical transmission component. It can be a rigid transparent sheet or a transparent elastomer. In one embodiment, part of the optical transmission component is a transparent elastomer. In the visual-tactile sensor, selecting a transparent elastomer as the optical transmission component is equivalent to providing a soft base for the sensing layer. When the reflective elastomer comes into contact with an external object, it can absorb and disperse the contact force, avoiding local stress concentration. At the same time, due to the viscoelastic properties of the elastomer itself, its recovery process after compression and the hysteresis effect during lateral sliding can be accurately captured by the monocular camera. The bottom of the transparent elastomer is fixedly connected to the rigid transparent base. When an external object presses on the elastomer, the rigid transparent base provides physical support, ensuring the reproducibility of the elastomer's deformation.

[0133] It should be noted that this embodiment does not limit the specific method of manufacturing the transparent optical transmission component. It can be that an RGB three-color LED array is installed in an annular groove, and the red LED array is set to point at 0° azimuth angle / 0° elevation angle, the green LED array to point at 90° azimuth angle / 0° elevation angle, and the blue LED array to point at 180° azimuth angle / 0° elevation angle. Then, a transparent elastomer is poured in as a filler, and after curing, it is installed on a transparent plate inside the sensing window of the housing.

[0134] It is understood that the monocular camera in this embodiment can simultaneously acquire images from both the left and right perspectives, imaged by two mirrors. Since the image data from the left and right perspectives are captured by a single camera within the same exposure period, the system is naturally immune to the hardware clock synchronization error problem common in multi-camera systems.

[0135] It should be noted that when independently controllable monochromatic light source components are configured around the transparent optical transmission component (corresponding to the four directions of east, south, west, and north), a monochrome industrial camera without a Bayer filter can be used as the monocular camera. Using a monochrome industrial camera eliminates the attenuation of light energy caused by color filters, maximizing the photon collection rate of each photosensitive pixel, thereby significantly improving the system's physical resolution and light sensitivity, and effectively avoiding image edge blurring artifacts caused by RGB channel interpolation calculations. The target image corresponding to this scheme is obtained through time-division strobe illumination of the four monochromatic light source components and time-division acquisition by the monocular camera. To ensure timing matching, the visual-tactile sensor in this embodiment also includes a high-frequency clock controller (e.g., MCU or FPGA). The high-frequency clock controller outputs a synchronization pulse signal with microsecond precision to the camera and the monochrome light source components through a hardware trigger pin (such as GPIO). The synchronization pulse signal is used to control the global shutter of the monocular camera and the transient illumination of the monochrome light source on one side to achieve hardware-level synchronization, so that the effective emission period of the light source completely covers the shutter exposure period of the camera, and performs high-speed polling triggering on the four monochrome light source components according to a preset timing sequence.

[0136] refer to Figure 4 , Figure 4 This is a flowchart illustrating the steps of a method provided in one embodiment of this application. The method includes, but is not limited to, the following steps:

[0137] Step S10: When the reflective elastomer comes into contact with the target object and is compressed, the target image is acquired by a monocular camera, and the first coordinate set of the marker point array imaged by the first reflector and the second coordinate set imaged by the second reflector are extracted from the target image.

[0138] It is understandable that during the contact deformation of the reflective elastomer, the photosensitive target of the monocular camera will simultaneously capture two independent viewpoints after the light path is folded by the reflection and refraction. The first coordinate set and the second coordinate set belong to the conjugate projection of the same physical marker array under different viewpoints. This step provides a precise data basis for subsequent stereo parallax calculation and absolute depth alignment.

[0139] Specifically, in some embodiments, Figure 4 Step S10, which involves acquiring the target image using a monocular camera, includes, but is not limited to, the following steps:

[0140] Step S11: When the light source assembly includes a first light source assembly, a second light source assembly, a third light source assembly, and a fourth light source assembly, and all light source assemblies are monochromatic LED arrays, the first light source assembly, the second light source assembly, the third light source assembly, and the fourth light source assembly are lit sequentially.

[0141] Using a preset sliding time window, the monocular camera is controlled to capture a first original image, a second original image, a third original image, and a fourth original image with different timestamps. The first original image is the image captured when the first light source component is lit, the second original image is the image captured when the second light source component is lit, the third original image is the image captured when the third light source component is lit, and the fourth original image is the image captured when the fourth light source component is lit.

[0142] Subtract the sub-reference image captured in advance in an unloaded, non-contact state when the corresponding light source component in the same orientation is lit from the first, second, third, and fourth original images of each captured frame to obtain the corresponding first, second, third, and fourth difference images.

[0143] The first difference image, the second difference image, the third difference image, and the fourth difference image are determined as the target image;

[0144] or,

[0145] Step S12: When the light source component includes a red LED array, a green LED array, and a blue LED array, the red LED array, the green LED array, and the blue LED array are lit simultaneously, and the image captured by the monocular camera is determined as the target image.

[0146] It is understandable that, in this embodiment, in the case of... Figure 3 In the illustrated light source configuration, the light source consists of four discrete directional illumination channels. The monocular camera performs acquisition operations based on microsecond-level hardware hard trigger signals. The preferred length of the sliding time window is 4, which is constructed as a 'ring-shaped four-frame buffer window' at the firmware level.

[0147] Specifically, during the high-speed time-division stroboscopic acquisition process, the monocular camera performs frame-by-frame capture of monochrome shadow images with a global shutter cycle on the order of microseconds. (i.e., the first original image, the second original image, the third original image, and the fourth original image). A high-frequency global clock controller (MCU or FPGA) simultaneously outputs the illumination trigger pulse and adds a binary timing tag representing the current illumination orientation channel to each frame of the image in real time. The front-end algorithm parses the binary time sequence tag k and sequentially pushes the received raw images into the corresponding azimuth channel slots within the sliding time window. This data routing mechanism, based on hard triggering and strong binding of channel tags, effectively avoids the image-lighting direction matrix mismatch problem caused by hardware frame drops or misalignments that may occur during high-speed polling. When each newly captured image is loaded, the window advances one position and overwrites the oldest historical frame in that channel. Under the dense depth output requirement of 30 fps, this circular four-frame buffer window will continuously slide and update at a hardware physical capture beat of 120 Hz, ensuring that within any system time interval of 33.3ms, there is always a set of the latest target images containing complete four-azimuth observations available for retrieval.

[0148] Furthermore, this embodiment also includes a temporal dynamic motion artifact suppression mechanism: based on the viscoelastic damping characteristics of the elastic silicone sensing medium and the upper limit of the physical motion speed of the robotic arm, the geometric deformation increment of the sensing layer surface is extremely small and negligible within a single complete stroboscopic cycle window. Since the total pulse window is much smaller than the 33.3 ms interval of a 30 fps video frame, the deformation state of the contact surface is considered quasi-static within the stroboscopic window, and no additional motion compensation kinematic registration is required between the four original images.

[0149] Furthermore, as an adaptive fault-tolerant mechanism of the system, when the system detects high-frequency severe vibration or extreme transient movement of the robotic arm equipped with the visual-tactile sensor through differential energy detection, the front-end algorithm evaluates the differential energy integral between two adjacent frames within the window. If the differential energy integral exceeds the preset integration threshold, the system will automatically adjust the weight parameters in the energy functional to reduce the confidence of the sliding time window in the downstream dense reconstruction, thereby ensuring the robustness of the reconstruction results under extreme conditions.

[0150] Finally, for each frame within the circular four-frame buffer window... (i.e., the first original image, the second original image, the third original image, and the fourth original image) respectively subtract the corresponding non-contact reference image in the same orientation. , The system includes a first sub-reference image (the image captured by a monocular camera when the first light source component is lit, without the reflective elastomer being in contact with the target object and under pressure), a second sub-reference image (the image captured by a monocular camera when the second light source component is lit, without the reflective elastomer being in contact with the target object and under pressure), a third sub-reference image (the image captured by a monocular camera when the third light source component is lit, without the reflective elastomer being in contact with the target object and under pressure), and a fourth sub-reference image (the image captured by a monocular camera when the third light source component is lit, without the reflective elastomer being in contact with the target object and under pressure). After removing static background shadows through differential operations, the system outputs clean first, second, third, and fourth differential images. These images are then used as the final target image to provide a high signal-to-noise ratio input stream for the downstream overdetermined photometric stereo solution matrix.

[0151] It is understood that in this embodiment, when the light source component includes a red LED array, a green LED array, and a blue LED array, i.e., as shown... Figure 2 With the light source components set as shown, the target image is obtained according to step S12. Specifically, the red LED array, green LED array, and blue LED array are lit simultaneously, and the image captured by the monocular camera is determined as the target image. The target image is obtained after step S12 as shown. Figure 5 As shown.

[0152] Specifically, in some embodiments, Figure 4 Step S10, which involves extracting the first coordinate set of the marker point array imaged by the first mirror and the second coordinate set imaged by the second mirror from the target image, includes, but is not limited to, the following steps:

[0153] Based on the pre-calibrated first and second effective detection polygons, the corresponding first left half image and first right half image are obtained from the target image, respectively.

[0154] Perform perspective correction transformations on the first left half image and the first right half image respectively to obtain the corresponding first front view and second front view;

[0155] Nonlocal mean denoising, adaptive Gaussian thresholding, morphological opening operation, and morphological closing operation are performed sequentially on the first front view and the second front view to obtain the corresponding first binary image and second binary image.

[0156] Connectivity analysis is performed on the first binary image and the second binary image to determine multiple first connected components retained by the first binary image and multiple second connected components retained by the second binary image. Each first or second connected component uniquely corresponds to a marker point.

[0157] Determine the first image moments of each first connected component, and calculate the corresponding first coordinates based on each first image moment;

[0158] Determine the second image moments of each second connected component, and calculate the corresponding second coordinates based on each second image moment.

[0159] It is understood that, referring to the description of the above embodiments, different light source component structures correspond to different target images. In this embodiment, the method of extracting the corresponding first coordinate set and second coordinate set for target images under different conditions is the same.

[0160] It is understandable that, for any target image obtained under any circumstances, its left and right fields of view correspond to the effective detection polygons poly_L and poly_R (each a quadrilateral defined by four full-image coordinate vertices, i.e., the effective region) determined in advance during the calibration stage, respectively. The left and right effective regions are directly extracted from the target image according to poly_L and poly_R, respectively, and perspective projection transformation is performed to flatten the trapezoidal distortion caused by the camera's tilted viewpoint into a standard front view, resulting in the corresponding first and second front views. For example, in the case of... Figure 2 When the light source component is set in a certain way, the first front view corresponds to the following: Figure 6 As shown, the second front view is as follows Figure 7 As shown, subsequent operations such as coordinate detection and coordinate matching are all performed based on the front view. This can significantly reduce the coupling interference between camera attitude deviation and elastic body geometric distortion, thereby ensuring the accuracy of the extracted first and second coordinate sets.

[0161] Specifically, in this embodiment, the non-local mean denoising process for the first front view and the second front view is as follows: by performing a weighted average on similar image blocks within the search window, compressed texture and speckle noise are suppressed, while the edge sharpness of the marker points is preserved, and the corresponding denoised image is obtained.

[0162] Specifically, after the front view is denoised using nonlocal mean, adaptive Gaussian thresholding is performed on the denoised image to extract the dark markers as foreground white connected regions. Then, morphological opening operations (removing fine white noise and threshold spikes) and closing operations (filling in the holes inside the markers caused by highlights) are performed in sequence to obtain a clear marker binary mask (i.e., the first binary image and the second binary image).

[0163] Furthermore, connected component analysis is performed on the first and second binary images. Noise and unlabeled regions are filtered out according to the area interval [area_min, area_max], thereby determining multiple first connected components retained in the first binary image and multiple second connected components retained in the second binary image. Each first or second connected component uniquely corresponds to a labeled point. The first image moments of each first connected component are determined, and the corresponding first coordinates are calculated based on each first image moment. The second image moments of each second connected component are determined, and the corresponding second coordinates are calculated based on each second image moment.

[0164] It should be noted that the first or second coordinates calculated based on image moments are obtained according to the following formula:

[0165] ;

[0166] ;

[0167] in,( , () represents any first or second coordinate. and For a first-order moment, The zeroth moment is the first moment. Both the first and zeroth moments belong to the image moments. That is, for the first coordinate, the ratio of the first moment to the zeroth moment of the first image moment is taken as the first coordinate to calculate the corresponding first coordinate. For the second coordinate, the ratio of the first moment to the zeroth moment of the second image moment is taken as the second coordinate to calculate the corresponding second coordinate.

[0168] Step S20: Extract multiple sets of target coordinate pairs from the first coordinate set and the second coordinate set. Each set of target coordinate pairs includes the first coordinate and the second coordinate corresponding to the same marker point in the marker point array. The first coordinate belongs to the first coordinate set, and the second coordinate belongs to the second coordinate set.

[0169] It is understood that this embodiment extracts multiple sets of corresponding target coordinate pairs from the first coordinate set and the second coordinate set, providing an effective data foundation for subsequent calculation of the absolute three-dimensional world coordinates corresponding to each marker point.

[0170] Specifically, before performing step S20, the embodiments of this application also include, but are not limited to, the following steps:

[0171] A grayscale template of the first size is defined with each reference node corresponding to the marker point in the reference image as the center. The reference image is the image captured by a monocular camera when the reflective elastomer is not in contact with the target object and is under pressure. Each reference node has a corresponding reference topological coordinate.

[0172] Centered on each reference node, a search area of ​​the second size is defined. The coordinate points corresponding to the first coordinate or the second coordinate that fall within the search area are taken as candidate points, wherein the first size is smaller than the second size.

[0173] For the grayscale template corresponding to the same marker point, NCC matching is performed one by one with the first-size image block centered on each candidate point to obtain multiple matching results. When the NCC correlation coefficient is greater than the preset coefficient threshold, the matching result is determined to be a successful match, and the reference node and candidate point are assigned one-to-one according to the NCC correlation coefficient from high to low.

[0174] When the number of successfully matched results is less than the number of reference nodes, local interpolation is performed on the first coordinates associated with the markers of successfully matched points adjacent to the failed matching reference nodes based on a preset grid step vector to obtain the corresponding first predicted coordinates. Similarly, local interpolation is performed on the second coordinates associated with the markers of successfully matched points adjacent to the failed matching reference nodes based on the grid step vector to obtain the corresponding second predicted coordinates. When invalid geometric structures exist in the neighborhood of any failed matching reference node, the first homography matrix obtained by fitting the first coordinates associated with the markers of successfully matched points and the corresponding reference topological coordinates is used... A homography matrix is ​​used to determine the first predicted coordinates. A second homography matrix is ​​obtained by fitting the second coordinates associated with the successfully matched markers and the corresponding reference topological coordinates based on each matching result. The second homography matrix is ​​used to determine the second predicted coordinates. All the first predicted coordinates and the successfully matched first coordinates are combined to obtain a new first coordinate set. The reference topological coordinates of the reference nodes corresponding to each coordinate point in the new first coordinate set are determined as the first topological coordinates of the corresponding coordinate points themselves. All the second predicted coordinates and the successfully matched second coordinates are combined to obtain a new second coordinate set. The reference topological coordinates of the reference nodes corresponding to each coordinate point in the new second coordinate set are determined as the second topological coordinates of the corresponding coordinate points themselves.

[0175] When the number of successful matching results equals the number of reference nodes, the original first coordinate set and second coordinate set are retained. The reference topological coordinates of the reference node corresponding to the coordinate point of the first coordinate are determined as the first topological coordinates of the coordinate point of the first coordinate itself. The reference topological coordinates of the reference node corresponding to the coordinate point of the second coordinate are determined as the second topological coordinates of the coordinate point of the second coordinate itself.

[0176] It is understood that, before determining the target coordinate pair corresponding to the same marker point based on the first coordinate set and the second coordinate set, this embodiment acquires a reference image obtained by a monocular camera when the reflective elastomer is in a state of not contacting the object. The topological coordinates of each coordinate point corresponding to the first and second coordinate sets determined in step S10 are then matched with the reference nodes corresponding to the same marker point in the reference image using NCC (Network Control Classification). Specifically, the grayscale template of each reference node with marked topological coordinates in the reference image is pasted onto the target image of the current frame to find the most similar candidate point. After calculating the NCC correlation coefficient, a preset coefficient threshold (not limited in this embodiment, it can be 0.4) is used as the lower limit of similarity. Matching results with an NCC correlation coefficient less than 0.4 are directly discarded. Among the remaining valid matching results, a greedy one-to-one assignment is performed in descending order of the NCC coefficient value. The operation of filling missing nodes is triggered by determining whether the number of successfully matched results equals the number of reference nodes.

[0177] Specifically, when the number of successfully matched results is less than the number of reference nodes, the operation of filling missing nodes is triggered. The specific steps include: for each failed match reference node, based on the reference topological coordinates of the failed match reference node, local interpolation is performed based on the first coordinates of the adjacent successfully matched marker points and the grid step vector to obtain the corresponding first predicted coordinates; when local interpolation is unavailable (in the case of invalid geometry in the neighborhood of any failed match reference node), the first predicted coordinates of the missing node are predicted using the first homography matrix obtained by fitting all successfully matched first coordinates and their reference topological coordinates; all first predicted coordinates are combined with all successfully matched first coordinates to obtain a new first coordinate set. The right view obtains the second predicted coordinates and the new second coordinate set in the same way. Subsequently, the reference topological coordinates of the reference nodes corresponding to each coordinate point in the new first coordinate set and the new second coordinate set are determined as the first topological coordinates and second topological coordinates of that coordinate point, respectively.

[0178] Specifically, when the number of successfully matched results equals the number of reference nodes, the original first and second coordinate sets are retained. The reference topological coordinates of the reference nodes corresponding to the coordinate points of the first coordinate are determined as the first topological coordinates of the coordinate points of the first coordinate. The reference topological coordinates of the reference nodes corresponding to the coordinate points of the second coordinate are determined as the second topological coordinates of the coordinate points of the second coordinate. That is, the reference topological coordinates of the corresponding reference nodes are directly inherited, providing a data foundation for subsequent topological coordinate alignment and absolute three-dimensional world coordinate calculation.

[0179] It is understandable that this embodiment, by performing NCC matching between the candidate coordinate points of the left and right views extracted from the target image of the current frame and the reference nodes of the reference image, and directly inheriting the reference topological coordinates of the reference image based on the matching results, eliminates the dependence on geometric neighbor consistency compared to the scheme of constructing topological coordinates based on the candidate coordinate points of the target image itself. Even if the pressing deformation destroys the local quadruple connectivity, as long as the grayscale template features of the marker point itself are still identifiable, its topological coordinates can be stably inherited, fundamentally eliminating the label collapse problem caused by the spread of BFS along the wrong pointer in the strong deformation region, and ensuring the accuracy of the topological coordinates of the candidate coordinate points.

[0180] It should be noted that this embodiment does not limit the specific size of the grayscale template and the search area. For example, the grayscale template can be 21×21 pixels and the second size can be 81×81 pixels. Those skilled in the art can adjust them according to actual needs.

[0181] Additionally, it should be noted that the determination of each reference node in the reference image in this embodiment includes, but is not limited to, the following steps:

[0182] Based on the pre-calibrated first and second effective detection polygons, the second left half image and the second right half image are obtained from the reference image;

[0183] Perform perspective correction transformations on the second left half image and the second right half image respectively to obtain the corresponding third and fourth front views;

[0184] Nonlocal mean denoising, adaptive Gaussian thresholding, morphological opening operation, and morphological closing operation are performed sequentially on the third and fourth front views to obtain the corresponding third and fourth binary images.

[0185] Connectivity analysis is performed on the third and fourth binary images to identify multiple third connected components retained in the third binary image and multiple fourth connected components retained in the fourth binary image. Each third or fourth connected component uniquely corresponds to a marker point.

[0186] Determine the third image moments of each third connected component, calculate the corresponding third coordinates based on each third image moment, and form a third coordinate set from all the third coordinates.

[0187] Determine the fourth image moments of each fourth connected component, calculate the corresponding fourth coordinates based on each fourth image moment, and form a fourth coordinate set from all the fourth coordinates.

[0188] Calculate the first nearest neighbor distance for each third coordinate, and determine the median of the first nearest neighbor distance as the first reference distance. Based on the first reference distance, denoise the third coordinate set. Let the marker point corresponding to each first effective coordinate in the denoised third coordinate set find the nearest neighbor in the first target direction. Construct multiple first pointer structures. Take the coordinate point corresponding to the first pointer structure with the highest pointer connectivity as the root node of the graph and perform breadth-first search (BFS). Assign first reference topology coordinates in the form of row and column numbers along the first target direction to construct the first grid topology. Then, perform coordinate zeroing on the first grid topology. Here, the first nearest neighbor distance is the distance between any third coordinate and the corresponding first nearest coordinate. The first nearest coordinate is the coordinate point in the third coordinate set that is closest to the third coordinate other than itself.

[0189] Calculate the second nearest neighbor distance for each fourth coordinate in the fourth coordinate set, and determine the median of the second nearest neighbor distance as the second reference distance. Based on the second reference distance, denoise the fourth coordinate set. Let the marker point corresponding to each second effective coordinate in the denoised fourth coordinate set find its nearest neighbor in the second target direction, construct multiple second pointer structures, and use the coordinate point corresponding to the second pointer structure with the highest pointer connectivity as the root node of the graph to perform BFS. Assign second reference topology coordinates in the form of row and column numbers along the second target direction to construct the second grid topology, and perform coordinate zeroing on the second grid topology. Here, the second nearest neighbor distance is the distance between any fourth coordinate and the corresponding second nearest coordinate, and the second nearest coordinate is the coordinate point in the fourth coordinate set that is closest to the fourth coordinate other than itself.

[0190] When the number of nodes in the first grid topology or the number of nodes in the second grid topology is less than the number of marker points, the third homography matrix, which maps row and column numbers to image coordinates, is fitted using the correspondence between the third coordinates associated with the reference topological coordinates of all known nodes in the first grid topology and the corresponding reference topological coordinates. The third homography matrix is ​​then used to determine the third predicted coordinates corresponding to the missing nodes. All the third predicted coordinates are combined with the third coordinates of all known nodes in the first grid topology to obtain the fifth coordinate set. The fourth homography matrix is ​​fitted using the correspondence between the fourth coordinates associated with the reference topological coordinates of all known nodes in the second grid topology and the corresponding reference topological coordinates. The fourth homography matrix is ​​then used to determine the fourth predicted coordinates corresponding to the missing nodes. All the fourth predicted coordinates are combined with the fourth coordinates of all known nodes in the second grid topology to obtain the sixth coordinate set.

[0191] When the number of nodes in the first grid topology and the number of nodes in the second grid topology are equal to the number of marked points, the coordinate points corresponding to all the first reference topology coordinates and all the second reference topology coordinates are determined as reference nodes.

[0192] It is understood that in this embodiment, the step of obtaining the second left half image and the second right half image from the reference image based on the pre-calibrated first effective detection polygon poly_L and the second effective detection polygon poly_R is to perform a region of interest (ROL) segmentation operation on the reference image.

[0193] It is understood that in this embodiment, the steps of denoising the third coordinate set based on the first reference spacing to obtain the denoised third coordinate set, and denoising the fourth coordinate set based on the second reference spacing to obtain the denoised fourth coordinate set, include: using a preset coefficient... The product of the first reference spacing sp and the first reference distance sp is the reference radius (i.e. Within the circular neighborhood of the reference radius, isolated noise points with fewer than min_nb effective neighbors are iteratively removed to obtain the denoised third coordinate set (each point is denoted as the first effective coordinate). Here, the first nearest neighbor distance is the distance between any third coordinate and the first nearest coordinate, and the first nearest coordinate is the coordinate point in the third coordinate set that is closest to that third coordinate (excluding itself). The denoising process for the fourth coordinate set is the same as that for the third coordinate set, except that the reference distance used is the second reference distance, which will not be elaborated further here. Each point in the denoised fourth coordinate set is denoted as the second effective coordinate.

[0194] It should be noted that after obtaining the denoised third coordinate set, this embodiment further divides the neighboring points within the circular neighborhood corresponding to the reference radius into four 90° quadrants (upper, lower, left, and right) according to their azimuth angles. Each quadrant retains only the nearest neighbor point to any of the marker points corresponding to the first valid coordinates as a directed pointer, thereby constructing multiple first pointer structures. The first target direction in this embodiment corresponds to the aforementioned four 90° quadrants (upper, lower, left, and right).

[0195] Furthermore, the first pointer structure with the highest pointer connectivity is determined as the first target structure. A breadth-first search (BFS) is performed using the coordinates corresponding to the first valid coordinates associated with the first target structure as the root node of the graph. For each associated marker point along the first target direction, the search is performed incrementally along the direction. First reference topological coordinates are assigned to construct a first mesh topology, and the coordinates of the first mesh topology are zeroed out. The first nearest neighbor distance is the distance between any third coordinate and its corresponding first nearest coordinate, where the first nearest coordinate is the coordinate point in the set of third coordinates that is closest to the third coordinate (excluding itself). Furthermore, the topological coordinates in this embodiment are integer two-dimensional topological coordinates. .

[0196] It should be noted that the technical principle of constructing the second grid topology is similar to that of constructing the first grid topology, and will not be elaborated here. The difference between the two is that the second grid topology corresponds to the grid topology of the coordinate points corresponding to the second right half of the reference image, while the first grid topology corresponds to the grid topology of the coordinate points corresponding to the second left half of the reference image.

[0197] It should be noted that the reference topology coordinates in the above embodiments include the first reference topology coordinates and the second reference topology coordinates in the current embodiment.

[0198] Furthermore, after constructing the first and second grid topologies, this embodiment also determines whether to trigger the operation of filling missing nodes by judging whether the number of nodes in the two grid topologies is equal to the number of marker points, thereby determining the final reference node.

[0199] Specifically, when the number of nodes in the first grid topology or the number of nodes in the second grid topology is less than the number of marked points, the operation of filling missing nodes is triggered. The specific steps include: calculating the third homography matrix using all the first reference topology coordinates, and calculating the missing third predicted coordinates using the third homography matrix; combining the coordinates corresponding to all the third predicted coordinates and all the first reference topology coordinates to obtain the fifth coordinate set; calculating the fourth homography matrix using all the second reference topology coordinates, and calculating the missing fourth predicted coordinates using the fourth homography matrix; combining the coordinates corresponding to all the fourth predicted coordinates and all the second reference topology coordinates to obtain the sixth coordinate set; and determining the coordinates corresponding to the fifth and sixth coordinate sets as reference nodes.

[0200] Specifically, when the number of nodes in the first grid topology and the number of nodes in the second grid topology are equal to the number of marked points, the operation of filling missing nodes is not triggered, and the coordinate points corresponding to all the first reference topology coordinates and all the second reference topology coordinates are determined as reference nodes.

[0201] Specifically, in some embodiments, Figure 4 Step S20, which involves extracting multiple sets of target coordinate pairs from the first and second coordinate sets, includes, but is not limited to, the following steps:

[0202] Determine the first number of coordinate points in all second topological coordinates that have the same coordinates as all first topological coordinates;

[0203] All reference coordinate point pairs are divided according to column offset to obtain multiple reference coordinate sets. Each pair of reference coordinate points in each reference coordinate set has the same column offset, and any pair of reference coordinate points consists of a first topological coordinate and a second topological coordinate with the same topological row number. The column offset is obtained by subtracting the column number of the first topological coordinate from the column number of the second topological coordinate of any pair of reference coordinate points in the corresponding reference coordinate set.

[0204] Calculate the horizontal disparity for each pair of reference coordinate points, where the horizontal disparity is obtained by subtracting the column pixel coordinates of the second topological coordinate from the column pixel coordinates of the first topological coordinate of the corresponding group.

[0205] Determine the horizontal pixel column distance between the coordinates of adjacent columns in the first topological coordinates of the same topological row number, and determine the median of all horizontal pixel column distances as the reference column distance;

[0206] Calculate the median of all horizontal disparities in each reference coordinate set to obtain the median of each disparity;

[0207] Iterate through all column offsets, calculate the absolute value of the difference between each horizontal disparity associated with any column offset and the corresponding disparity median, and when the absolute value is less than the product of the reference column distance and the preset value, increase the number of inliers of the corresponding column offset by 1.

[0208] The column offset corresponding to the largest number of interior points is determined as the target column offset, and all second topological coordinates are offset by the target column offset.

[0209] Determine the second number of points in the first topological coordinates and the second topological coordinates after the offset operation that have the same coordinates;

[0210] When the second quantity is greater than the first quantity, the coordinate point that is the same as the first topological coordinate and the second topological coordinate after the offset operation is taken as the target matching result. When the second quantity is less than or equal to the first quantity, the coordinate point that is the same as the first topological coordinate and the second topological coordinate before the offset operation is taken as the target matching result. The target matching result includes multiple sets of target coordinate pairs.

[0211] Specifically, this embodiment does not limit the specific preset value, and the value can be 0.4. Those skilled in the art can select according to the actual situation.

[0212] It is understood that the binocular topology matching operation in this embodiment is implemented based on the column offset histogram voting method. The specific operation steps include: determining the first number of coordinate points in all the second topology coordinates that have the same coordinates as all the first topology coordinates; and matching the topology row numbers with the same ( = A first topological coordinate and a second topological coordinate are matched to form a reference coordinate point pair, and the column offset is dc. Group all reference coordinate point pairs to obtain multiple reference coordinate sets, each corresponding to a column offset dc. Calculate the horizontal disparity for each reference coordinate point pair. And correlate the column offset and horizontal parallax of the reference coordinate set in the same group;

[0213] Next, determine the horizontal pixel spacing between the coordinate points corresponding to two first topological coordinates in adjacent columns (topological column numbers differ by 1) under the same topological row number, and calculate the median of this horizontal pixel spacing to obtain the reference column distance dx; calculate the median of all horizontal disparities in each reference coordinate set to obtain the disparity median.

[0214] Iterate through all column offsets dc, and for each pair of reference coordinate points corresponding to the same column offset, calculate the horizontal disparity. ,calculate The absolute value of the difference from the median of the disparity in this group, when the absolute value is less than the product of the corresponding reference column distance and the preset value, increases the number of inliers in the column offset by 1; the column offset with the largest number of inliers is determined as the target column offset, and all second topological coordinates are offset by the target column offset.

[0215] Determine the second number of coordinate points that are the same in the first topological coordinates and the second topological coordinates after offset. If the second number is greater than the first number, take the coordinate points that are the same in the first topological coordinates and the second topological coordinates after offset as the target matching result. Otherwise, take the coordinate points that are the same in the first topological coordinates and the second topological coordinates before offset as the target matching result, so as to retain the scheme with more common nodes. The target matching result includes multiple sets of target coordinate pairs, and the two points in each set of target coordinate pairs are identified as physical homogeneous points.

[0216] Understandably, this method does not require pre-calibration of the deformation field and can still achieve high recall matching even when the array of marked points is severely distorted.

[0217] Step S30: Calculate the absolute three-dimensional world coordinates of the corresponding marker point based on any target coordinate pair.

[0218] It is understandable that this embodiment calculates the absolute three-dimensional world coordinates of the corresponding marker points based on the target coordinate pairs, which can provide an effective data foundation for subsequent calculation of the dense depth map of the reflective elastomer surface.

[0219] Specifically, in some embodiments, Figure 4 Step S30 includes, but is not limited to, the following steps:

[0220] The target coordinate pairs are mapped to the original camera image coordinate system of the monocular camera through the inverse perspective transformation matrix;

[0221] The Brown-Conrady distortion model is used to perform anti-distortion correction on the target coordinate pairs after mapping to obtain the distortion-free intermediate coordinate pairs.

[0222] The intermediate coordinates are back-projected into the first and second unit rays in the camera coordinate system of the monocular camera;

[0223] Based on Snell's law, the first and second unit rays are refracted and corrected, and the corrected first and second unit rays are transformed into the corresponding first and second reference rays in the world coordinate system.

[0224] Solve for the objective parameters that minimize the distance between the first reference ray and the second reference ray, where the objective parameters include the first parameter and the second parameter;

[0225] A first target point is determined from a first reference ray based on a first parameter, and a second target point is determined from a second reference ray based on a second parameter, wherein the distance between the first reference ray and the second reference ray is the shortest at the position of the first target point, and the distance between the second reference ray and the first reference ray is the shortest at the position of the second target point.

[0226] The midpoint between the first target point and the second target point is determined as the absolute three-dimensional world coordinates of the marker point.

[0227] It is understandable that the steps for calculating the absolute three-dimensional world coordinates of each marker point in this embodiment include: using the inverse perspective transformation matrix The target coordinate pairs calculated in the above embodiments are mapped to the original camera image coordinate system to obtain multiple original pixel coordinate pairs. Where N is the number of marker points; the target coordinate pairs after mapping are processed using the Brown-Conrady distortion model. Perform anti-distortion correction to obtain the distortion-free intermediate coordinate pair. The intermediate coordinates are back-projected into the first and second unit rays in the camera coordinate system of the monocular camera, considering the light rays originating from the silicone elastomer (refractive index...). (Refractive index) Passes through a flat interface into the air The refraction bending that occurs during refraction is strictly corrected for the first and second unit rays using Snell's law. The corrected first and second unit rays are then transformed into their corresponding first and second reference rays in the world coordinate system. The target parameters that minimize the distance between the first and second reference rays are determined, where the target parameters include the first parameter s and the second parameter t. The first and second reference rays are skew lines in the world coordinate system. Based on the first parameter, a first target point is determined from the first reference ray, and based on the second parameter, a second target point is determined from the second reference ray. The first reference ray is positioned at the first target point with the shortest distance to the second reference ray, and the second reference ray is positioned at the second target point with the shortest distance to the first reference ray. The midpoint between the first and second target points is defined as the absolute three-dimensional world coordinate of the marker point. Let i = 1, ..., N, where N is the number of marker points. The absolute 3D world coordinates corresponding to all marker points constitute a sparse depth anchor set. .

[0228] It should be noted that, in this embodiment, the expressions for the first reference ray and the second reference ray are as follows:

[0229] ;

[0230] ;

[0231] in, As the first reference ray, As the second reference ray, As the starting point of the first reference ray, Let be the starting point of the second reference ray, s be the first parameter, and t be the second parameter. Let be the unit direction vector of the first reference ray in the world coordinate system. The second reference ray is the unit direction vector in the world coordinate system. The two are obtained by normalizing the left and right view coordinates of the intermediate coordinate pair after back projection, Snell refraction correction and transformation to the world coordinate system.

[0232] Step S40: Obtain the dense gradient field of the reflective elastomer surface based on the normal estimation model.

[0233] It should be noted that this implementation does not limit the specific normal estimation model; it can be a learning model or a photometric mapping gradient lookup table. The learning model in this embodiment is a multilayer perceptron (MLP).

[0234] Understandably, the traditional method for obtaining the dense gradient field of an elastomer surface is the linear photometric stereo method, which assumes that the silicone surface exhibits perfect Lambertian diffuse reflection under LED illumination and solves a system of linear equations pixel by pixel. The dense gradient field calculated by extracting the surface normal is affected by a variety of nonlinear factors in practical applications: specular reflection components of the silicone surface, radial vignetting of the lens, actual deviation of LED color temperature and direction, CMOS channel crosstalk, and saturation in high-brightness areas. These factors cause significant deviations in the linear solution results in large deformation and high-contrast regions. In other words, the accuracy of the dense gradient field calculated by the traditional method is poor.

[0235] Based on this, in order to absorb the above-mentioned nonlinear effects, this embodiment uses a multilayer perceptron (MLP) trained offline with calibrated data to directly learn the nonlinear mapping from "pixel color + spatial position" to "surface normal". Then, the dense gradient field is derived from the normal field through a closed analytical expression, which can effectively ensure the accuracy of the dense gradient field.

[0236] Specifically, in execution Figure 4 Prior to step S40, the detection method of the visual-tactile sensor provided in this application embodiment also includes, but is not limited to, the following steps:

[0237] After mapping the target coordinate pair to the original camera image coordinate system through the inverse perspective transformation matrix, the first coordinate in the mapped target coordinate pair is remapped back to the front view coordinate system where the first front view is located based on the inverse perspective transformation matrix to obtain the intermediate topological coordinates.

[0238] Calculate the third nearest neighbor distance for each intermediate topological coordinate, and determine the median of all third nearest neighbor distances as the third reference distance;

[0239] Calculate the first radius based on the third reference spacing, and draw the binary occlusion mask corresponding to each intermediate topological coordinate using the first radius;

[0240] For the first front view, the Fast Moving Model (FMM) is used to fill in the corresponding binary occlusion mask layer by layer from the boundary inward to obtain the repaired first front view.

[0241] Understandably, since the physical markers on the reflective elastomer are opaque black blocks, their internal pixels cannot carry LED light reflection information. If the first and second front views are directly fed into MLP inference, directional "pseudo-gradients" will be generated, contaminating subsequent variational fusion and failing to guarantee an accurate and reliable dense depth map. Therefore, before performing the operation of calculating the dense gradient field on the surface of the reflective elastomer based on MLP, this embodiment performs image restoration on the occluded areas of the markers.

[0242] Understandably, the steps for image inpainting of occluded areas of marked points include: after mapping the target coordinate pairs to the original camera image coordinate system using an inverse perspective transformation matrix, remapping the first coordinate in the mapped target coordinate pairs back to the orthographic coordinate system based on the inverse perspective transformation matrix to obtain intermediate topological coordinates. ; Calculate the third nearest neighbor distance for each intermediate topological coordinate. The median of all third nearest neighbor spacings is determined as the third reference spacing; the first radius is calculated based on the third reference spacing, and the intermediate topological coordinates are plotted on an 8-bit single-channel blank mask of the same size as the gradient map using the first radius. The corresponding binary occlusion mask; for any pixel of the first front view, the Fast Moving Model (FMM) is used to fill the corresponding binary occlusion mask layer by layer from the boundary inward to obtain the repaired first front view.

[0243] Specifically, in this embodiment, the first radius is calculated based on the third reference spacing using the clip function clip(). ,in, The distance is the third nearest neighbor, and k is a preset distance ratio coefficient.

[0244] Specifically, in this embodiment, for any pixel of the first front view, the Fast Moving Model (FMM) is used to fill the corresponding binary occlusion mask layer by layer from the boundary inward to obtain the repaired first front view. In this process, the value of each pixel to be filled in the first front view is obtained by weighted averaging of the corresponding known neighboring pixel values. The weight of each pixel to be filled is determined by the geometric distance from the neighboring pixel to the current pixel, the angle with the boundary normal direction, and the distance to the horizontal set of the mask boundary.

[0245] Understandably, the repaired first front view can present a pseudo-LED response that smoothly transitions with the surrounding lighting background at the marked point location, serving as the actual input to the MLP.

[0246] Specifically, in some embodiments, Figure 4 Step S40 includes, but is not limited to, the following steps:

[0247] Step S401: Perform dimensionality enhancement processing on each pixel of the first front view to obtain a first feature image, wherein each pixel of the first feature image includes a vector of color dimension and a vector of spatial coordinate dimension.

[0248] Step S402: Input the first feature image into the normal estimation model to perform normal estimation, and output the first component and the second component, wherein the first component is the tilt component of the normal vector of the reflective elastomer surface in the x-axis direction, and the second component is the tilt component of the normal vector of the reflective elastomer surface in the y-axis direction.

[0249] Step S403: Based on the first and second components, the third component is obtained by inverse solving using the normalization constraint of the unit normal vector, wherein the third component is the component of the normal vector of the reflective elastomer surface in the z-axis direction.

[0250] Step S404: Perform a lower cutoff process on the third component, and calculate the dense gradient field pixel by pixel based on the first component, the second component, and the lower cutoff processed third component.

[0251] In this embodiment, each pixel (u,v) of the first front view is subjected to dimensionality enhancement processing to obtain a first feature image. Each pixel of the first feature image includes a vector of color dimension and a vector of spatial coordinate dimension. The first feature image corresponds to a 5-dimensional feature vector, and the expression of the first feature image is as follows:

[0252] ;

[0253] Among them, the first 3 dimensions ( ) is the normalized BGR three-channel color, the latter two dimensions ( () represents the normalized spatial coordinates, where H and W are the height and width of the front view, respectively, and the normalized values ​​range from [ ]. It is understandable that the introduction of spatial coordinates enables MLP to automatically learn and compensate for position-dependent nonlinearities such as radial vignetting and continuous changes in the LED incident angle with pixel position, which is equivalent to implicitly encoding a spatially varying backlighting model in the network.

[0254] It should be noted that this embodiment does not limit the specific network structure of the MLP. An MLP can include four fully connected layers, specifically in the structure "5→64→64→64→2", which includes an input layer receiving a 5-dimensional feature vector, three hidden layers (each with 64 neurons), and an output layer that outputs a 2-dimensional vector. Specifically, each hidden layer is connected in series with a ReLU activation layer and a regularization layer. The regularization layer corresponds to p... drop =0.05. After receiving the first feature image corresponding to the 5-dimensional feature vector at the input layer, the hidden layer extracts complex nonlinear mapping relationships from the input 5-dimensional data. Then, the output layer directly predicts the components of the normal vector of the reflective elastomer surface in the x-axis and y-axis directions, namely the first component and the second component. .

[0255] Furthermore, based on the first component Second component Using the normalization constraint of the unit normal vector The third component is obtained by inverse solution. The third component This represents the z-axis component of the normal vector of the reflective elastomer surface. This embodiment considers that in actual deep learning models, due to floating-point precision errors or network prediction biases, the output... and There may be a risk of exceeding the boundary, i.e., n x 2 +n y 2 >1. To avoid division by zero errors, this embodiment performs a lower cutoff on the third component, where the lower cutoff value can be 0.05. .

[0256] Specifically, in this embodiment, the dense gradient field is calculated based on the first component, the second component, and the third component after lower cutoff processing, and is obtained according to the following formula:

[0257] ;

[0258] in, It is a dense gradient field.

[0259] It should be noted that this embodiment does not limit the specific device executing the MLP inference process; it can be a CUDA device. Specifically, this embodiment will use the first feature image... Pixel stitching Tensors are fed into the network in batches according to batch size, and inference is performed in parallel on CUDA devices.

[0260] Specifically, in some embodiments, Figure 4 Step S40 includes, but is not limited to, the following steps:

[0261] Step S405: Based on the pre-set registration lookup table, the first right half image is registered pixel by pixel to the coordinate system of the first left half image by combining the image remapping algorithm and the bilinear interpolation algorithm to obtain the third right half image. The registration lookup table is used to indicate the floating-point pixel corresponding to each pixel in the first left half image in the first right half image.

[0262] Step S406: Perform perspective correction transformation on the effective area of ​​the third right half image to obtain the fifth front view;

[0263] Step S407: Perform dimensionality enhancement processing on each pixel of the first front view to obtain a first feature image, wherein each pixel of the first feature image includes a vector of color dimension and a vector of spatial coordinate dimension.

[0264] Step S408: Perform dimensionality enhancement processing on each pixel of the fifth front view to obtain a second feature image, wherein each pixel of the second feature image includes a vector of color dimension and a vector of spatial coordinate dimension.

[0265] Step S409: Combine the first feature image and the second feature image to obtain the third feature image;

[0266] Step S410: Input the third feature image into the normal estimation model to perform normal estimation, and output the first component and the second component, wherein the first component is the tilt component of the normal vector of the reflective elastomer surface in the x-axis direction, and the second component is the tilt component of the normal vector of the reflective elastomer surface in the y-axis direction.

[0267] Step S411: Based on the first and second components, the third component is obtained by inverse solving using the normalization constraint of the unit normal vector, where the third component is the component of the normal vector of the reflective elastomer surface in the z-axis direction.

[0268] Step S412: Perform a lower cutoff process on the third component, and calculate the dense gradient field pixel by pixel based on the first component, the second component, and the lower cutoff processed third component.

[0269] It should be noted that, in this embodiment, the schemes in steps S401 to S404 and steps S405 to S412 are two parallel schemes for calculating the dense gradient field. Each parallel scheme is applicable to different light source component situations (i.e., applicable to...). Figure 2 and Figure 3 (The situation).

[0270] It is understandable that steps S401 to S404 correspond to the following: Figure 2 As shown, in the case of an RGB three-color light source component, a scheme is used where features of a single view (i.e., the first front view) are used as input to the MLP to estimate two normals and calculate the dense gradient field based on the normals. This embodiment also provides another method for calculating the dense gradient field in the case of the same RGB three-color light source component, namely steps S405 to S412, which use features of a dual view as input to the MLP to estimate two normals and calculate the dense gradient field based on the normals. The dense gradient field output by the MLP is as follows: Figure 8 As shown.

[0271] Specifically, the steps for calculating the dense gradient field using a dual-view approach include: obtaining a pre-set registration lookup table, which indicates the floating-point pixel corresponding to each pixel in the first left half image in the first right half image; based on the registration lookup table, combining the image remapping algorithm (cv2.remap) and bilinear interpolation algorithm to perform pixel-by-pixel registration of the first right half image to the coordinate system of the first left half image to obtain a third right half image; further, performing perspective projection transformation on the effective area of ​​the third right half image to obtain a fifth front view; performing dimensionality enhancement on each pixel of the first front view to obtain a first feature image identical to step S407; performing dimensionality enhancement on each pixel of the fifth front view to obtain a second feature image, wherein each pixel of the second feature image includes a vector of color dimension and a vector of spatial coordinate dimension; and combining the first feature image and the second feature image to obtain a third feature image with dimension 8. Among them, left-eye Figure 3 VI BGR Color + Right View Figure 3 The BGR color and the shared 2D normalized spatial coordinates are used as inputs to the MLP. It is understood that this embodiment simultaneously utilizes the left and right optical paths to observe the same physical surface at different incident angles, theoretically improving the signal-to-noise ratio. The MLP's nonlinear gain further learns the mirror crosstalk and high-light saturation compensation modes of the dual-optical path, and is unaffected by the occlusion of the pressed object. Furthermore, in the dual-view calculation of the dense gradient field, the steps of estimating two normals based on the MLP and calculating the dense gradient field based on the normals, i.e., steps S410 to S412, are similar in principle to steps S402 to S404 in the above embodiment, except that the input feature map is different, which will not be elaborated here. After processing through steps S410 to S412, a dense gradient field with the same resolution as the dense gradient field output in step S404 can be calculated.

[0272] Specifically, targeting Figure 3 In the case of a monochromatic light source component, taking a dual-view approach as an example, the steps for calculating the dense gradient field in this embodiment include: dimensionalizing the corresponding first left half image (including the difference images of the four directions corresponding to the left view) to obtain the corresponding first feature image (including the left view...). Figure 4 Single-channel gray-level difference dimension vector in each direction and normalized coordinate dimension vector ), and the second feature image (including the right view) obtained by pixel registration of the corresponding first right half image (including the difference images of the four directions corresponding to the right view) through steps S405 to S406. Figure 4 Single-channel gray-level difference dimension vector in each direction and normalized coordinate dimension vector Furthermore, the first and second feature images are combined to obtain a third feature image with a dimension of 10. Then, the third feature image with a dimension of 10 is used as input to the subsequent MLP to estimate two normals and calculate the dense gradient field based on the normals. This scheme directly utilizes the nonlinear activation function and arbitrary function approximation capability of neural networks to directly learn the inherently complex nonlinear mapping from "8-dimensional spatiotemporal photometric response + 2-dimensional spatial coordinate dimension" to "high-precision two-dimensional surface partial derivative gradient (p, q)". The algorithm can adaptively eliminate reconstruction errors caused by highlights, shadows, and non-uniform light intensity without explicitly calibrating the near-field illumination field or fitting a complex bidirectional reflection distribution function. It is understandable that in the current embodiment, when the perception layer experiences extreme depth compression or strong distortion, some areas will inevitably have shadows or overexposed highlights. In linear algorithms, this can lead to ill-conditioned matrices or failure to invert; however, in the MLP architecture, the network implicitly learns this high-dimensional feature discarding and compensation mechanism during the training phase. When one or two dimensions of the 10-dimensional input fail due to extreme conditions (manifesting as extreme noise), the MLP can adaptively rely on the effective orthogonal photometric features of the remaining dimensions to perform highly reliable gradient inference, exhibiting extremely high robustness against overexposure and shadows, and ensuring the reconstruction continuity of the visual-tactile sensor under extreme dynamic contact.

[0273] It should be noted that the specific method used in this embodiment for image remapping and registration using pre-calculated coordinate mapping relationships (i.e., registration lookup tables) in conjunction with bilinear interpolation can be selected by those skilled in the art according to the actual situation, and no restrictions are imposed here.

[0274] It is understandable that, in this embodiment, the core advantages of the time-division multiplexing scheme (corresponding to the case of a single-color light source component + dual view) over the traditional spectral multiplexing scheme (corresponding to the case of a three-color light source component + single view) are as follows:

[0275] (1) The training target space is significantly simplified and the generalization ability is improved: In the traditional spectral multiplexing scheme in steps S401 to S404, the input features are limited by the color dimension of the color image. The normal estimation model must learn two coupled tasks at the same time: "silicone illumination → normal mapping" and "Bayer filter channel crosstalk, mosaic artifact compensation". The model capacity is forced to be allocated to the latter. In the time-division multiplexing scheme of this embodiment, the normal estimation model learns the mapping relationship between the pure gray response mode of each pixel in 8 different orientations and the surface normal gradient of the pixel. This physically interpretable nonlinear mapping means that, compared with the RGB spectral multiplexing scheme, the normal estimation model no longer needs to learn any compensation tasks related to "color", such as Bayer channel crosstalk, mosaic artifact, and deviation between LED color temperature and calibrated color temperature. Because the 8-dimensional photometric feature space is completely isolated in the time domain, it has a high degree of spatiotemporal orthogonality independence at the optical level, eliminating spectral crosstalk noise from the data source, providing data input with extremely high signal-to-noise ratio for the normal estimation model, and with lower generalization error and stronger cross-sensor transferability under the same network capacity.

[0276] (2) The input space is increased from 3 channels to 8 channels, and the information entropy increases significantly: the input feature dimension of the normal estimation model is increased from 3D of RGB (single view) to 8D of four-way monochrome (left and right views); at the same time, since there is an explicit orientation geometric relationship between the 8 channels, the hidden layer of the normal estimation model can learn the strong constraint feature of "four-way brightness ratio → surface gradient orientation", which is much more physically interpretable than the weak constraint feature of "color ratio → gradient" under RGB three channels;

[0277] (3) The natural redundancy of the channels enables the normal estimation model to have data-level robustness against overexposure and shadows: Since only 3 independent observation equations are needed to solve the two-dimensional surface gradient, the 8-dimensional photometric input constructed in this embodiment makes the system in a state of ultra-high data redundancy. During network training, a data augmentation strategy of randomly zeroing 1-2 azimuth channels ("Channel Dropout", which is different from the conventional normalization layer Dropout in steps S401 to S404, and only applies to the first 8 lighting channels) is used. The normal prediction learned by the normal estimation model can still maintain a robust output when any 1-2 azimuth channels are saturated or lost. This fundamentally eliminates the weakness of the normal estimation model in steps S401 to S404, where the normal direction estimation collapses in the specular highlight area or depth pressing blind area due to the saturation of any channel or self-shadowing.

[0278] (4) Effective data accuracy brought about by improved bit depth and resolution: The monocular camera in this scheme preferably adopts a monochrome global shutter camera, so that each photosensitive pixel is not affected by the attenuation of the color filter, thereby maximizing the photon collection rate, and there is no spatial resolution loss caused by Bayer interpolation. The equivalent effective bit depth of the same pixel is about 1 bit higher than that of the Bayer sensor; the difference image input to the normal estimation model The floating-point dynamic range before normalization is thus significantly expanded, and the upper limit of the angular resolution of the corresponding normal output is thus increased.

[0279] (5) Illumination calibration drift immunity: The normal estimation model of the scheme in steps S401 to S404 implicitly encodes the LED color temperature and orientation at the "calibration time". When the LED color temperature drifts due to long-term use or hardware replacement, the network must be retrained. However, the normal estimation model in the time-division multiplexing scheme of this embodiment learns the "brightness → orientation" mapping. The overall drift of the LED monochromatic luminous intensity can be absorbed by simply scaling the input difference map. There is no need to retrain the network, and the cost of long-term deployment in industrial sites is significantly reduced.

[0280] It should be noted that the above technical advantages are mainly due to the underlying hardware optical path architecture and trigger timing, which are physical gains that neural networks cannot fully compensate for by simply increasing the model capacity on RGB data.

[0281] Additionally, it should be noted that the MLP in this embodiment is trained via transfer learning: the weights of a pre-trained basic normal estimation network are used as initial values, and then fine-tuned based on the calibration data collected by this sensor. Specifically, two types of calibration datasets are collected on the binocular catadioptric hardware of this application: (1) a pressure sample dataset, i.e., pressure is applied to a spherical calibration piece with known curvature, and the true normal of each pixel is derived from the geometric model. (2) Non-contact background sample dataset, i.e., the normal of the sensor surface collected without any external force (its normal is approximately the zero-tilt normal perpendicular to the sensing surface). During training, the two datasets are mixed in a certain proportion to form a training batch, in which the background samples account for a certain proportion (e.g., about 25%), in order to suppress false gradient responses in non-pressed areas.

[0282] The training process optimizes the network parameters using a loss function that is primarily based on the normal prediction error. The loss function includes at least a deviation measure between the predicted normal and the true normal (e.g., mean square error), and optionally supplements it with a consistency constraint term between the gradient fields obtained from the normal conversion, as well as a sample weighting strategy for the contact edge region and / or background region, in order to improve the normal accuracy of the contact edge and further suppress spurious background responses.

[0283] The optimization process can employ an adaptive gradient optimizer (such as the AdamW adaptive moment estimator with weight decay) in conjunction with an adaptive learning rate decay strategy. The relevant hyperparameters, such as the number of training epochs, batch size, learning rate, and weight decay, can be set according to the data scale and hardware conditions. When the error index of the model on the validation set reaches the expected level, the trained multilayer perceptron model is saved.

[0284] Step S50: Construct an energy functional based on the dense gradient field and absolute three-dimensional world coordinates, and solve the energy functional to obtain a dense depth map, wherein the dense depth map can characterize the three-dimensional absolute deformation of the contact surface of the reflective elastomer.

[0285] It is understood that, referring to the description of the above embodiments, this embodiment uses a sparse depth anchor point set composed of all absolute three-dimensional world coordinates generated by the binocular stereo vision method. The dense gradient field, obtained by MLP estimation, is jointly optimized through variational energy functionals, which can output a dense depth map with both absolute scale accuracy and surface texture resolution.

[0286] Specifically, in some embodiments, Figure 4 Step S50 includes, but is not limited to, the following steps:

[0287] Step S501: Based on the variational Poisson optimization framework, an energy functional is constructed by combining a dense gradient field and absolute three-dimensional world coordinates.

[0288] Step S502: Extract variational extremum conditions and discretize the objective constraint terms in the energy functional to obtain multiple linear equations, and stack all the linear equations to obtain a sparse matrix.

[0289] Step S503: Construct the target normal matrix based on the sparse matrix, and solve the target normal equation using the preprocessing conjugate gradient method to obtain the dense depth map.

[0290] Specifically, the expression for the energy functional is as follows:

[0291] ;

[0292] in, This is the gradient weight map corresponding to the repaired first front view in the above embodiments. Let Z be the gradient of the depth map over the entire imaging region Ω. , For gradient constraint weights, Let i be the anchor frame, and i be the anchor point number within the anchor frame. Let Z be the depth value of the depth map at the marked point (x, y). For sparse depth anchor set The absolute depth value corresponding to the i-th anchor point. For depth-constrained weights, For the reference constraint terms, This is a reference constraint term; the reference constraint term has the following two parallel implementation methods:

[0293] (a) Smoothing constraint (curvature regularization): By penalizing the horizontal and vertical second-order differences (i.e., local curvature) of the depth map Z to approach zero, curvature regularization is provided for regions lacking reliable optical gradients.

[0294] (b) Material constraint terms (hyperelastic constitutive): In other embodiments, the Mooney-Rivlin hyperelastic constitutive model is used to structurally model the flexible silicone sensing layer, wherein... All are material constants. The first and second invariants of the right Cauchy-Green deformation tensor are calculated based on the spatial partial derivatives of the surface at depth Z.

[0295] It is understandable that, as can be seen from the above expression of the energy functional, the energy functional includes three constraint terms, the first of which is... This is a gradient constraint term used to constrain the ∇Z approximation density gradient field. The first constraint term introduces Due to the spike artifacts generated at the marker points by traditional algorithms, the weight of each marker pixel repaired by Telea in W is decayed to a configurable constant (exemplarily 0.2~0.6, for example 0.6). This operation mathematically "shields" the false optical gradients caused by abrupt changes in albedo, preventing erroneous information from entering the subsequent integral propagation chain, effectively ensuring the authenticity and reliability of the gradient values ​​of the depth map Z; the second constraint term Anchor point constraint term, used to constrain the depth value of depth map Z at the marker point position (x,y). Approximating the corresponding absolute depth value This is equivalent to "pinning" a photometric surface suspended in relative space to a real physical coordinate system, with weights... Controlling the system's confidence in absolute depth, this mechanism fundamentally eliminates the unavoidable low-frequency depth drift accumulation error of traditional Poisson integrals when dealing with large-scale curved surfaces, enabling the sensor to simultaneously possess micron-level surface texture resolution and sub-millimeter-level spatial macroscopic measurement capabilities. The third constraint term (i.e., the reference constraint term) is the aforementioned smoothing constraint term or material constraint term: In the first constraint term, the gradient information of the marker point region is intentionally masked. If only the first two terms (i.e., the gradient constraint term and the anchor point constraint term) are relied upon, the system will exhibit mathematical "flat interpolation" in this region, resulting in unnatural discontinuities on the contact surface. To address this, in some embodiments, this application uses a smoothing constraint term as the third constraint term, employing second-order difference curvature regularization to ensure a smooth transition in this region. In other embodiments, a material constraint term based on the Mooney-Rivlin hyperelastic constitutive model is used as the third constraint term, forcing the depth surface to transition in the region inside the marker point lacking optical gradients according to the nonlinear differential geometry of hyperelastic silicone under large deformations, thereby further restoring the true geometric curvature of the mechanical contact within the optical blind zone.

[0296] Specifically, the discretization process in step S502 is performed on the two-dimensional image grid. (Physical pixel spacing) Grid origin Perform finite difference discretization:

[0297] For the gradient constraint term, write the horizontal gradient equation for each pair of horizontally adjacent pixels (i, j) and (i, j+1). , and These are the depth values ​​corresponding to the horizontally adjacent pixel nodes (i, j) and (i, j+1), respectively. and These are the partial derivatives of the depths along the x (column) direction at the horizontally adjacent pixel nodes (i, j) and (i, j+1), respectively. Both sides of the equation are multiplied by the equality weights. ,in For gradient constraint weights, Right now This is a pixel-by-pixel gradient weight map (the repaired area is attenuated). Similarly, the vertical gradient equation is written for vertically adjacent pixels (i, j) and (i+1, j). , and These are the depth values ​​at the vertically adjacent pixel nodes (i, j) and (i+1, j), respectively. and These are the partial derivatives of the depth along the y (row) direction at the vertically adjacent pixel nodes (i, j) and (i+1, j), respectively. The sum of the two sets of equations... Lines form sparse blocks .

[0298] For anchor point constraints, for sparse depth anchor point sets Each sparse anchor point in Map its physical coordinates to grid-normalized coordinates The normalized grid coordinates are then rounded down to obtain the integer part, and the normalized grid coordinates are subtracted from the integer part to obtain the decimal part. ;exist The bilinear interpolation coefficients are filled into the four adjacent pixel columns of the i-th row. The entire row is multiplied by the weight of the equation. The right-hand item is The depth constraint is essentially a sub-pixel interpolation constraint. Compared to the "take the nearest integer pixel" operation, it avoids the discretization noise caused by quantizing the anchor point position on the grid, and is crucial for the floating-point sub-pixel coordinates commonly found in the triangulation results.

[0299] Regarding the reference constraint term, in embodiments employing smoothing constraint terms, when At that time, for each ternary pixel string (i,j-1), (i,j), (i,j+1) and (i-1,j), (i,j), (i+1,j), write it into the second-order central difference equation. Multiply the entire row Later merged This provides curvature regularization; in embodiments employing material constraints, the discrete form of the Mooney-Rivlin strain energy density with respect to the Z-space partial derivative of the depth surface is written into the corresponding sparse block to constrain the rationality of geometric deformation.

[0300] Specifically, in some embodiments, the target constraint term includes a static constraint term, which includes a gradient constraint term and a reference constraint term. The gradient constraint term is used to constrain the gradient approximation of the dense depth map to the corresponding absolute dense gradient field. The reference constraint term is a smoothing constraint term or a material constraint term. The smoothing constraint term is used to constrain the local curvature of the dense depth map, and the material constraint term is used to constrain the geometric deformation of the reflective elastomer to conform to real physical laws. Step S502 includes, but is not limited to, the following steps:

[0301] Step S5021: With the current target image as the first frame image, extract variational extreme value conditions and discretize the static constraint terms to obtain multiple linear equations. Stack all the linear equations to obtain a static sparse matrix. Calculate the corresponding static normal matrix based on the static sparse matrix and save the static sparse matrix and the static normal matrix to the cache area.

[0302] Step S5022: If the current target image is not the first frame image, but the grid size and / or the constraint weight of the static constraint term have changed, delete the existing data in the buffer, re-extract the variational extreme value conditions and discretize the static constraint term to obtain multiple new linear equations, stack all the new linear equations to obtain a new static sparse matrix, calculate the corresponding new static normal matrix based on the new static sparse matrix, and save the new static sparse matrix and the new static normal matrix to the buffer. The grid size is the grid size formed by the anchor points corresponding to all absolute three-dimensional world coordinates.

[0303] Step S5023: If the current target image is not the first frame image and the grid size and the constraint weights of the static constraint terms have not changed, retain and reuse the static sparse matrix and static normal matrix in the buffer.

[0304] It should be noted that this embodiment does not limit the specific capacity of the cache area, which can be 8; at the same time, it does not limit the specific structure of the cache area, which can be a global LRU structure.

[0305] Understandably, static constraints (including gradient constraints and reference constraints (smoothing constraints or material constraints)) depend only on the mesh size and constraint weights. Therefore, the corresponding static sparse matrix and static normal matrix can be cached and reused across frames, ensuring hits even if the marker anchor points drift between frames. The anchor point constraint is small; the corresponding anchor point normal matrix is ​​reconstructed each frame and superimposed on the static normal matrix to obtain the target normal matrix for the current frame. Thus, when the mesh size and constraint weights remain unchanged in subsequent frames, the constraint can be directly reused, avoiding the overhead of repeated static sparse matrix assembly. This saves 0.5 to 2 seconds of assembly time per frame on a 960×960 mesh.

[0306] Specifically, in some embodiments, the energy functional further includes an anchor constraint term, which is used to constrain the depth value at any marker point in the dense depth map to approximate the absolute depth value of the corresponding absolute three-dimensional world coordinates. Step S503 includes, but is not limited to, the following steps:

[0307] For the anchor point constraint term, bilinear interpolation discretization is performed based on the position of the corresponding anchor point in the grid to construct the anchor point submatrix, and the anchor point normal matrix corresponding to the anchor point submatrix is ​​calculated.

[0308] By superimposing the anchor point normal matrix with the valid static normal matrix in the buffer, the target normal matrix corresponding to the current target image is obtained;

[0309] Construct a diagonal Jacobi preprocessor based on the diagonal of the objective matrix;

[0310] Preprocessing conjugate gradient iterations are performed on the target normal matrix based on the diagonal Jacobi preprocessor, and the number of iterations is recorded;

[0311] When the number of iterations reaches the number threshold, or when the relative residual of the solution for the current iteration is less than or equal to the first tolerance threshold, the iteration stops and the first intermediate depth map is output. The relative residual is used to indicate the reduction ratio of the residual of the current solution of the conjugate gradient iteration relative to the initial residual.

[0312] Adjust the first intermediate depth map according to the preset size to obtain the dense depth map.

[0313] It should be noted that in this embodiment, a preconditional conjugate gradient solver is used to perform conjugate gradient iteration on the normal matrix. The iteration termination condition is that the number of iterations reaches a threshold, or the relative residual of the solution at the current iteration number is less than or equal to a first tolerance threshold. In this embodiment Values .

[0314] It should be noted that the preconditioning conjugate gradient solver in this embodiment can be a solver deployed on the CPU or a solver deployed on the GPU. Specifically, when deployed on the CPU, the solver that performs conjugate gradient iteration on the target normal matrix is ​​scipy.sparse.linalg.cg; when the system is configured with use_gpu=true and CuPy and CUDA drivers are available, the system migrates the normal matrix and right-hand vector to cupyx.scipy.sparse.csr_matrix on the GPU memory, and performs preprocessing conjugate gradient iteration on the GPU memory through cupyx.scipy.sparse.linalg.cg. The CSR structure on the GPU side is also cached in a separate GPU structure cache with the object identifier of the target normal matrix structure as the key. If a runtime mismatch is detected between the CuPy compilation kernel and the local graphics card architecture (e.g., a NO_BINARY_FOR_GPU exception occurs), the system automatically falls back to GPU conjugate gradient without preprocessors. That is, in this case, conjugate gradient iteration is performed on the normal matrix without preprocessors, and the first intermediate depth map is output.

[0315] It is understandable that after performing conjugate gradient iteration on the normal matrix to obtain the first intermediate depth map, the first intermediate depth map is adjusted to a preset size. Floating-point depth map.

[0316] Specifically, in some embodiments, Figure 4 The solution of the energy functional in step S50 to obtain the dense depth map includes, but is not limited to, the following steps:

[0317] Step S504: Scale the dense gradient field according to the downsampling factor and increase the grid step size to generate a reference grid, wherein the anchor points corresponding to the absolute three-dimensional world coordinates remain unchanged.

[0318] Step S505: Iteratively solve the energy functional on the reference grid. If the relative residual corresponding to the solution of the current iteration number is less than or equal to the second tolerance threshold, output the first depth map.

[0319] Step S506: Perform bilinear upsampling of the first depth map back to the original resolution to obtain the second depth map;

[0320] Step S507: Using the second depth map as the initial solution, iteratively solve the energy functional at the original resolution to obtain the first intermediate depth map;

[0321] Step S508: Adjust the first intermediate depth map according to the preset size to obtain the dense depth map.

[0322] Understandably, in this example, during the process of solving the energy functional to obtain the dense depth map, cold start refers to the algorithm not relying on any previous calculation results at all, that is, setting the initial solution as a zero vector (or a random guess), and then starting from scratch to iterate step by step to approximate the true solution; warm start refers to the algorithm using the information obtained from the previous (or historical) solution as the initial solution for this calculation. Because the initial solution is already very close to the true solution, the algorithm does not need to slowly explore from the origin, thus greatly shortening the number of iterations and time required to reach the preset accuracy.

[0323] Understandably, to shorten the cold start time, the system provides a two-stage multi-scale solution: first, the dense gradient field is scaled according to the downsampling factor s, and the grid step size is increased accordingly (the anchor points keep their original coordinates unchanged), generating a reference grid. The reference grid is a low-resolution coarse network (the default size limit is 240×180). Then, a stricter tolerance (i.e., a second tolerance threshold, for example) is applied to the reference grid. A coarse first depth map is obtained by solving the entire solution. Then, the first depth map is bilinearly upsampled back to the original resolution to obtain a second depth map. This second depth map is used as the initial value for solving the conjugate gradient of the fine mesh preprocessing. That is, the global skeleton is quickly calculated at low resolution first, and then it is enlarged as the starting point for high-resolution calculation. In this way, the subsequent process of solving the first intermediate depth map can be transformed into a warm start, which greatly reduces the time of cold start solving the first intermediate depth map and effectively improves the detection efficiency of the visual-tactile sensor 100.

[0324] Specifically, in some embodiments, Figure 4 The solution of the energy functional in step S50 to obtain the dense depth map includes, but is not limited to, the following steps:

[0325] Step S509: When the distance between the absolute 3D world coordinates of any marker point in the current target image and the absolute 3D world coordinates of the target image in the previous frame is less than a preset distance threshold, the first intermediate depth map obtained by solving the energy functional of the target image in the previous frame is used as the second intermediate depth map. The second intermediate depth map is used as the initial solution, and the energy functional of the current target image is iteratively solved at the original resolution to obtain the first intermediate depth map of the current target image. The first intermediate depth map is then adjusted according to the preset size to obtain the dense depth map.

[0326] Step S510: When the distance between the absolute 3D world coordinates of any marker point in the current target image and the absolute 3D world coordinates of the target image in the previous frame is greater than or equal to a preset distance threshold, the dense gradient field corresponding to the current target image is scaled according to the downsampling factor and the grid step size is increased to regenerate a new reference grid. The energy functional is iteratively solved on the new reference grid. If the relative residual corresponding to the solution of the current iteration number is less than or equal to the second tolerance threshold, a new first depth map is output. The new first depth map is bilinearly upsampled back to the original resolution to obtain a new second depth map. The energy functional is iteratively solved at the original resolution using the new second depth map as the initial solution to obtain a new first intermediate depth map. The new first intermediate depth map is adjusted according to the preset size to obtain a dense depth map.

[0327] It is understood that steps S509 to S510 in this embodiment correspond to solving the dense depth map using a time-series hot-start strategy. Since the depth change between adjacent frames is usually very small (the deformation rate of the elastic body is much lower than the frame rate), the temporal consistency between video frames is utilized, and the solution result z_{t-1} (i.e., the second intermediate depth map) of the target image in the previous frame is used as the initial guess of the conjugate gradient solver in the current frame. Referring to the description of the above embodiments, and combining the assembly overhead skipped after a static sparse structure cache hit, it is possible to achieve near real-time dense depth reconstruction. When the anchor point position of a marker point changes beyond a threshold between consecutive frames (i.e., when the distance between the absolute 3D world coordinates of any marker point in the current target image and the absolute 3D world coordinates of the target image in the previous frame is greater than or equal to the distance threshold), the triggering source may be a violent contact or rapid slippage between the reflective elastomer and the target object. At this time, the system automatically resets the hot start and falls back to multi-scale coarse-fine solution to ensure global convergence. That is, the dense gradient field corresponding to the current target image is scaled according to the downsampling factor and the grid step size is increased accordingly to regenerate a new reference grid. The energy functional is iteratively solved on the new reference grid. When the relative residual corresponding to the solution of the current iteration number is less than or equal to the second tolerance threshold, a new first depth map is output. The new first depth map is bilinearly upsampled back to the original resolution to obtain a new second depth map. The energy functional is iteratively solved at the original resolution using the new second depth map as the initial solution to obtain a new first intermediate depth map. The new first intermediate depth map is adjusted according to the preset size to obtain a dense depth map.

[0328] It is understood that the first intermediate depth map in steps S509 to S510 of this embodiment is adjusted to a preset size. The floating-point depth map Z_fused (i.e., the final dense depth map) is in float32 format.

[0329] It should be noted that the dense depth map output in this embodiment is as follows: Figure 9 or Figure 10 The dense depth heatmap shown.

[0330] It should be noted that the schemes of steps S5021 to S5023, steps S504 to S508, and steps S509 to S510 in this embodiment are parallel schemes for solving the depth dense graph. They correspond to the sparse structure caching mechanism, the multi-scale coarse-fine solution strategy, and the temporal hot start strategy between video frames, respectively, and together achieve real-time dense depth reconstruction.

[0331] like Figure 11 As shown, Figure 11This is a structural diagram of a control device provided in one embodiment of this application. The present invention also provides a control device 200, including: a processor 210, which can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing relevant programs to implement the technical solutions provided in the embodiments of this application; and a memory 220, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 220 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 220 and is called and executed by the processor 210 to execute the detection method of the visual tactile sensor 100 in the embodiments of this application. The input / output interface 230 is used to realize information input and output. The communication interface 240 is used to realize communication interaction between this device and other devices. Communication can be realized by wired means (e.g., USB, network cable, etc.) or by wireless means (e.g., mobile network, WIFI, Bluetooth, etc.). The bus 250 transmits information between various components of the device (e.g., processor 210, memory 220, input / output interface 230 and communication interface 240). The processor 210, memory 220, input / output interface 230 and communication interface 240 realize communication connection between each other within the device through the bus 250.

[0332] In addition, this application embodiment also provides a visual-tactile sensor 100, which includes the control device 200 of the above embodiment.

[0333] In addition, this application embodiment also provides a storage medium, which is a computer-readable storage medium, storing a computer program that, when executed by a processor, implements the detection method of the visual-tactile sensor 100 described above.

[0334] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof. The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0335] Those skilled in the art will understand that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or temporary media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies. Digital multifunction discs (DVDs) or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically include computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0336] The above provides a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A detection method for a visual-tactile sensor, characterized in that, An application is made to a visual-tactile sensor, which includes a housing with a sensing window at the top, a monocular camera, a first reflector, a second reflector, a reflective elastomer with a marked dot array on its surface, and a transparent optical transmission component with multiple light source components arranged circumferentially. The monocular camera, the first reflector, and the second reflector are all located within the housing, with the first and second reflectors forming an angle. The transparent optical transmission component is embedded within the sensing window. The side of the transparent optical transmission component facing the inside of the housing serves as a light refraction surface opposite to the reflector, while the side facing the outside of the housing is fixedly connected to the lower surface of the reflective elastomer. The surfaces of the first and second reflectors are located within the field of view of the monocular camera. The method includes: When the reflective elastomer comes into contact with the target object and is compressed, the target image is acquired by the monocular camera, and the first coordinate set of the marker point array imaged by the first reflector and the second coordinate set imaged by the second reflector are extracted from the target image respectively. Multiple sets of target coordinate pairs are extracted from the first coordinate set and the second coordinate set. Each set of target coordinate pairs includes the first coordinate and the second coordinate corresponding to the same marker point in the marker point array. The first coordinate belongs to the first coordinate set and the second coordinate belongs to the second coordinate set. Based on any of the target coordinate pairs, calculate the absolute three-dimensional world coordinates of the corresponding marker point; The dense gradient field on the surface of the reflective elastomer is obtained based on the normal estimation model; An energy functional is constructed based on the dense gradient field and the absolute three-dimensional world coordinates. The dense depth map is obtained by solving the energy functional. The dense depth map can characterize the three-dimensional absolute deformation of the contact surface of the reflective elastomer.

2. The detection method of the visual-tactile sensor according to claim 1, characterized in that, Acquiring target images using the monocular camera includes: When the light source assembly includes a first light source assembly, a second light source assembly, a third light source assembly, and a fourth light source assembly, and all of the light source assemblies are monochromatic LED arrays, the first light source assembly, the second light source assembly, the third light source assembly, and the fourth light source assembly are lit sequentially; Using a preset sliding time window, the monocular camera is controlled to capture a first original image, a second original image, a third original image, and a fourth original image with different timestamps, respectively. The first original image is the image captured when the first light source component is lit, the second original image is the image captured when the second light source component is lit, the third original image is the image captured when the third light source component is lit, and the fourth original image is the image captured when the fourth light source component is lit. Subtract the sub-reference image captured in advance in an unloaded, contactless state when the corresponding light source component in the same orientation is lit from the first original image, the second original image, the third original image, and the fourth original image of each captured frame, respectively, to obtain the corresponding first difference image, second difference image, third difference image, and fourth difference image. The first difference image, the second difference image, the third difference image, and the fourth difference image are determined as the target image; or, When the light source component includes a red LED array, a green LED array, and a blue LED array, and the red LED array, the green LED array, and the blue LED array are lit simultaneously, the image captured by the monocular camera is determined as the target image.

3. The detection method of the visual-tactile sensor according to claim 2, characterized in that, Extracting a first coordinate set from the target image of the marker point array via the first mirror and a second coordinate set via the second mirror, including: Based on the pre-calibrated first and second effective detection polygons, the corresponding first left half image and first right half image are obtained from the target image; Perform perspective correction transformations on the first left half image and the first right half image respectively to obtain the corresponding first front view and second front view; The first front view and the second front view are sequentially subjected to nonlocal mean denoising, adaptive Gaussian threshold segmentation, morphological opening operation processing, and morphological closing operation processing to obtain the corresponding first binary image and second binary image. Connectivity analysis is performed on the first binary image and the second binary image to determine multiple first connected components retained by the first binary image and multiple second connected components retained by the second binary image, wherein each first connected component or second connected component uniquely corresponds to one of the marked points; Determine the first image moments of each first connected component, and calculate the corresponding first coordinates based on each first image moment; Determine the second image moments for each of the second connected components, and calculate the corresponding second coordinates based on each of the second image moments.

4. The detection method of the visual-tactile sensor according to claim 2 or 3, characterized in that, Before extracting multiple sets of target coordinate pairs from the first coordinate set and the second coordinate set, the method further includes: Centered on each reference node in the reference image corresponding to the marked point, a grayscale template of the first size is defined. The reference image is the image captured by the monocular camera when the reflective elastomer is not in contact with the target object and is under pressure. Each reference node corresponds to a reference topological coordinate. Centered on each of the reference nodes, a search area of ​​a second size is defined, and the coordinate points corresponding to the first coordinates or the second coordinates that fall within the search area are taken as candidate points, wherein the first size is smaller than the second size; For the grayscale template corresponding to the same marker point, NCC matching is performed one by one with the image block of the first size centered on each candidate point to obtain multiple matching results. When the NCC correlation coefficient is greater than the preset coefficient threshold, the matching result is determined to be a successful match, and the reference node and the candidate point are assigned one-to-one according to the NCC correlation coefficient from high to low. When the number of successfully matched results is less than the number of reference nodes, local interpolation is performed on the first coordinates associated with the markers of successfully matched points adjacent to the failed match reference nodes based on a preset grid step vector to obtain the corresponding first predicted coordinates. Similarly, local interpolation is performed on the second coordinates associated with the markers of successfully matched points adjacent to the failed match reference nodes based on the grid step vector to obtain the corresponding second predicted coordinates. When invalid geometry exists in the neighborhood of any failed match reference node, a first homography matrix is ​​fitted based on the first coordinates associated with each successfully matched marker and the corresponding reference topological coordinates. This first homography matrix is ​​then used to... The first predicted coordinates are determined by a matrix. A second homography matrix is ​​obtained by fitting the second coordinates associated with the successfully matched markers and the corresponding reference topological coordinates of each matching result. The second predicted coordinates are determined by the second homography matrix. All the first predicted coordinates and the successfully matched first coordinates are combined to obtain a new first coordinate set. The reference topological coordinates of the reference nodes corresponding to each coordinate point in the new first coordinate set are determined as the first topological coordinates of the corresponding coordinate points themselves. All the second predicted coordinates and the successfully matched second coordinates are combined to obtain a new second coordinate set. The reference topological coordinates of the reference nodes corresponding to each coordinate point in the new second coordinate set are determined as the second topological coordinates of the corresponding coordinate points themselves. When the number of successfully matched results is equal to the number of reference nodes, the original first coordinate set and second coordinate set are retained. The reference topological coordinates of the reference node corresponding to the coordinate point of the first coordinate are determined as the first topological coordinates of the coordinate point of the first coordinate itself. The reference topological coordinates of the reference node corresponding to the coordinate point of the second coordinate are determined as the second topological coordinates of the coordinate point of the second coordinate itself.

5. The detection method of the visual-tactile sensor according to claim 4, characterized in that, Each of the reference nodes in the reference image is obtained according to the following steps: Based on the pre-calibrated first and second effective detection polygons, the second left half image and the second right half image are obtained from the reference image; The second left half image and the second right half image are respectively subjected to perspective correction transformation to obtain the corresponding third front view and fourth front view; The third front view and the fourth front view are sequentially subjected to nonlocal mean denoising, adaptive Gaussian thresholding, morphological opening operation, and morphological closing operation to obtain the corresponding third binary image and fourth binary image. Connectivity analysis is performed on the third binary image and the fourth binary image to determine multiple third connected components retained by the third binary image and multiple fourth connected components retained by the fourth binary image, wherein each third connected component or the fourth connected component uniquely corresponds to a marker point; Determine the third image moments of each of the third connected components, calculate the corresponding third coordinates based on each of the third image moments, and form a third coordinate set from all the third coordinates; Determine the fourth image moments of each of the fourth connected components, calculate the corresponding fourth coordinates based on each of the fourth image moments, and form a fourth coordinate set from all the fourth coordinates; Calculate the first nearest neighbor distance for each of the third coordinates, determine the median of the first nearest neighbor distance as the first reference distance, perform denoising processing on the third coordinate set based on the first reference distance, let the marker point corresponding to each first effective coordinate in the denoised third coordinate set find the nearest neighbor in the first target direction, construct multiple first pointer structures, take the coordinate point corresponding to the first pointer structure with the highest pointer connectivity as the root node of the graph and perform breadth-first search (BFS), assign a first reference topology coordinate in the form of row and column numbers along the first target direction to construct a first grid topology, and perform coordinate zeroing processing on the first grid topology, wherein the first nearest neighbor distance is the distance between any third coordinate and the corresponding first nearest coordinate, and the first nearest coordinate is the coordinate point in the third coordinate set that is closest to the third coordinate other than itself; Calculate the second nearest neighbor distance for each fourth coordinate in the fourth coordinate set, and determine the median of the second nearest neighbor distance as the second reference distance. Based on the second reference distance, perform denoising processing on the fourth coordinate set. Let the marker point corresponding to each second effective coordinate in the denoised fourth coordinate set find its nearest neighbor in the second target direction, construct multiple second pointer structures, and use the coordinate point corresponding to the second pointer structure with the highest pointer connectivity as the root node of the graph to perform BFS. Assign second reference topology coordinates in the form of row and column numbers along the second target direction to construct a second grid topology, and perform coordinate zeroing processing on the second grid topology. The second nearest neighbor distance is the distance between any fourth coordinate and the corresponding second nearest coordinate, and the second nearest coordinate is the coordinate point in the fourth coordinate set that is closest to the fourth coordinate other than itself. When the number of nodes in the first grid topology or the number of nodes in the second grid topology is less than the number of marker points, a third homography matrix is ​​fitted by using the correspondence between the third coordinate associated with the reference topology coordinates of all known nodes in the first grid topology and the corresponding reference topology coordinates. The third homography matrix is ​​then used to determine the third predicted coordinates corresponding to the missing nodes. All the third predicted coordinates are combined with the third coordinates of all known nodes in the first grid topology to obtain a fifth coordinate set. A fourth homography matrix is ​​fitted by using the correspondence between the fourth coordinate associated with the reference topology coordinates of all known nodes in the second grid topology and the corresponding reference topology coordinates. The fourth homography matrix is ​​then used to determine the fourth predicted coordinates corresponding to the missing nodes. All the fourth predicted coordinates are combined with the fourth coordinates of all known nodes in the second grid topology to obtain a sixth coordinate set. When the number of nodes in the first grid topology and the number of nodes in the second grid topology are equal to the number of marker points, the coordinate points corresponding to all the first reference topology coordinates and all the second reference topology coordinates are determined as the reference nodes.

6. The detection method of the visual-tactile sensor according to claim 4, characterized in that, Multiple sets of target coordinate pairs are extracted from the first coordinate set and the second coordinate set, including: Determine the first number of coordinate points in all the second topological coordinates that have the same coordinates as all the first topological coordinates; All reference coordinate point pairs are divided according to column offset to obtain multiple reference coordinate sets. Each pair of reference coordinate points in each reference coordinate set has the same column offset, and any pair of reference coordinate points consists of a first topological coordinate and a second topological coordinate with the same topological row number. The column offset is obtained by subtracting the column number of the first topological coordinate from the column number of the second topological coordinate of any pair of reference coordinate points in the corresponding reference coordinate set. Calculate the horizontal disparity for each pair of reference coordinate points, wherein the horizontal disparity is obtained by subtracting the column pixel coordinates of the second topological coordinate from the column pixel coordinates of the first topological coordinate of the corresponding group; Determine the horizontal pixel column distance between the coordinate points of adjacent columns in the first topological coordinates of the same topological row number, and determine the median of all the horizontal pixel column distances as the reference column distance; Calculate the median of all horizontal disparities in each of the reference coordinate sets to obtain the median of each disparity; Traverse all the column offsets, calculate the absolute value of the difference between each horizontal disparity associated with any column offset and the corresponding median disparity, and when the absolute value is less than the product of the reference column distance and a preset value, increase the number of inliers of the corresponding column offset by 1. The column offset corresponding to the largest number of interior points is determined as the target column offset, and all second topological coordinates are column offset using the target column offset. Determine the second number of coordinate points that have the same coordinates in the first topological coordinates and the second topological coordinates after the offset operation; When the second quantity is greater than the first quantity, the coordinate point that is the same as the first topological coordinate and the second topological coordinate after the offset operation is taken as the target matching result. When the second quantity is less than or equal to the first quantity, the coordinate point that is the same as the first topological coordinate and the second topological coordinate before the offset operation is taken as the target matching result. The target matching result includes multiple sets of target coordinate pairs.

7. The detection method of the visual-tactile sensor according to claim 3, characterized in that, Based on any of the target coordinate pairs, calculate the absolute three-dimensional world coordinates of the corresponding marker point, including: The target coordinate pairs are mapped to the original camera image coordinate system of the monocular camera using the inverse perspective transformation matrix. The Brown-Conrady distortion model is used to perform anti-distortion correction on the target coordinate pairs after mapping to obtain the distortion-free intermediate coordinate pairs. The intermediate coordinates are back-projected into the first unit ray and the second unit ray in the camera coordinate system of the monocular camera; Based on Snell's law, the first unit ray and the second unit ray are refraction corrected, and the corrected first unit ray and second unit ray are transformed into the corresponding first reference ray and second reference ray in the world coordinate system. Solve for the target parameter that minimizes the distance between the first reference ray and the second reference ray, wherein the target parameter includes a first parameter and a second parameter; A first target point is determined from the first reference ray based on the first parameter, and a second target point is determined from the second reference ray based on the second parameter, wherein the distance between the first reference ray and the second reference ray is the shortest at the position of the first target point, and the distance between the second reference ray and the first reference ray is the shortest at the position of the second target point. The midpoint between the first target point and the second target point is determined as the absolute three-dimensional world coordinates of the marker point.

8. The detection method of the visual-tactile sensor according to claim 3, characterized in that, The dense gradient field of the reflective elastomer surface is obtained based on the normal estimation model, including: Dimensionality enhancement is performed on each pixel of the first front view to obtain a first feature image, wherein each pixel of the first feature image includes a vector of color dimension and a vector of spatial coordinate dimension. The first feature image is input into the normal estimation model for normal estimation, and a first component and a second component are output, wherein the first component is the normal vector of the reflective elastomer surface. x The tilt component in the axial direction, the second component being the normal vector of the reflective elastomer surface in... y Inclination component in the axial direction; Based on the first and second components, the third component is obtained by inverse solving using the normalized constraint condition of the unit normal vector, wherein the third component is the normal vector of the reflective elastomer surface at... z Components along the axial direction; The third component is subjected to a lower cutoff process, and the dense gradient field is calculated pixel by pixel based on the first component, the second component, and the lower cutoff third component.

9. The detection method of the visual-tactile sensor according to claim 3, characterized in that, The dense gradient field of the reflective elastomer surface is obtained based on the normal estimation model, including: Based on a pre-set registration lookup table, the first right half image is registered pixel by pixel to the coordinate system of the first left half image by combining an image remapping algorithm and a bilinear interpolation algorithm to obtain a third right half image. The registration lookup table is used to indicate the floating-point pixel corresponding to each pixel in the first left half image in the first right half image. A perspective correction transformation is performed on the effective area of ​​the third right half image to obtain the fifth front view; Dimensionality enhancement is performed on each pixel of the first front view to obtain a first feature image, wherein each pixel of the first feature image includes a vector of color dimension and a vector of spatial coordinate dimension. Dimension-enhancing processing is performed on each pixel of the fifth front view to obtain a second feature image, wherein each pixel of the second feature image includes a vector of color dimension and a vector of spatial coordinate dimension; The first feature image and the second feature image are combined to obtain the third feature image; The third feature image is input into the normal estimation model for normal estimation, and a first component and a second component are output, wherein the first component is the normal vector of the reflective elastomer surface. x The tilt component in the axial direction, the second component being the normal vector of the reflective elastomer surface in... y Inclination component in the axial direction; Based on the first and second components, the third component is obtained by inverse solving using the normalized constraint condition of the unit normal vector, wherein the third component is the normal vector of the reflective elastomer surface at... z Components along the axial direction; The third component is subjected to a lower cutoff process, and the dense gradient field is calculated pixel by pixel based on the first component, the second component, and the lower cutoff third component.

10. The detection method of the visual-tactile sensor according to claim 7, characterized in that, Before obtaining the dense gradient field of the reflective elastomer surface based on the normal estimation model, the method further includes: After mapping the target coordinate pair to the original camera image coordinate system using the inverse perspective transformation matrix, the first coordinate in the mapped target coordinate pair is remapped back to the front view coordinate system where the first front view is located based on the inverse perspective transformation matrix to obtain the intermediate topological coordinates. Calculate the third nearest neighbor distance for each of the intermediate topological coordinates, and determine the median of all the third nearest neighbor distances as the third reference distance; Calculate the first radius based on the third reference spacing, and draw the binary occlusion mask corresponding to each of the intermediate topological coordinates using the first radius; For the first front view, the Fast Marching Method (FMM) is used to fill in the corresponding binary occlusion mask layer by layer from the boundary inward to obtain the repaired first front view.

11. The detection method of the visual-tactile sensor according to claim 1, characterized in that, An energy functional is constructed based on the dense gradient field and the absolute three-dimensional world coordinates. Solving the energy functional yields a dense depth map, including: Based on the variational Poisson optimization framework, the energy functional is constructed by combining the dense gradient field and the absolute three-dimensional world coordinates. The target constraint terms in the energy functional are extracted for variational extremum conditions and discretized to obtain multiple linear equations. All the linear equations are stacked to obtain a sparse matrix. The target normal matrix is ​​constructed based on the sparse matrix, and the target normal equation is solved using the preprocessed conjugate gradient method to obtain the dense depth map.

12. The detection method of the visual-tactile sensor according to claim 11, characterized in that, The target constraint terms include static constraint terms, which in turn include gradient constraint terms and reference constraint terms. The gradient constraint terms constrain the gradient approximation of the dense depth map to the corresponding absolute dense gradient field. The reference constraint terms are either smoothing constraint terms or material constraint terms. The smoothing constraint terms constrain the local curvature of the dense depth map, and the material constraint terms constrain the geometric deformation of the reflective elastomer to conform to real physical laws. Variational extremum conditions are extracted from the target constraint terms in the energy functional, and the data is discretized to obtain multiple linear equations. All of these linear equations are then stacked to obtain a sparse matrix, including: With the current target image as the first frame, the static constraint terms are subjected to variational extreme value conditions and discretization processing to obtain multiple linear equations. All the linear equations are stacked to obtain a static sparse matrix. The corresponding static normal matrix is ​​calculated based on the static sparse matrix. The static sparse matrix and the static normal matrix are saved to the cache area. If the current target image is not the first frame image, but the grid size and / or the constraint weights of the static constraint terms have changed, delete the existing data in the buffer, re-extract the variational extreme value conditions and discretize the static constraint terms to obtain multiple new linear equations, stack all the new linear equations to obtain a new static sparse matrix, calculate the corresponding new static normal matrix based on the new static sparse matrix, and save the new static sparse matrix and the new static normal matrix to the buffer. The grid size is the grid size formed by all the anchor points corresponding to the absolute three-dimensional world coordinates. If the current target image is not the first frame image, and the grid size and the constraint weights of the static constraint terms have not changed, the static sparse matrix and the static normal matrix in the buffer are retained and reused.

13. The detection method of the visual-tactile sensor according to claim 12, characterized in that, The energy functional also includes an anchor constraint term, which is used to constrain the depth value at any marker point in the dense depth map to approximate the absolute depth value of the corresponding absolute 3D world coordinates. The objective normal matrix is ​​constructed based on a sparse matrix, and the objective normal equation is solved using the preprocessed conjugate gradient method to obtain the dense depth map, which includes: For the anchor point constraint term, bilinear interpolation discretization is performed based on the position of the corresponding anchor point in the grid to construct the anchor point sub-matrix, and the anchor point normal matrix corresponding to the anchor point sub-matrix is ​​calculated. By superimposing the anchor point normal matrix with the valid static normal matrix in the buffer, the target normal matrix corresponding to the current target image is obtained; Construct a diagonal Jacobi preprocessor based on the diagonal of the target matrix; Based on the diagonal Jacobi preprocessor, perform preprocessing conjugate gradient iteration on the target normal matrix and record the number of iterations; When the number of iterations reaches the number threshold, or when the relative residual of the solution for the current iteration is less than or equal to the first tolerance threshold, the iteration stops and a first intermediate depth map is output, wherein the relative residual is used to indicate the reduction ratio of the residual of the current solution of the conjugate gradient iteration relative to the initial residual; The first intermediate depth map is adjusted according to the preset size to obtain the dense depth map.

14. The detection method of the visual-tactile sensor according to claim 1, characterized in that, Solving the energy functional yields a dense depth map, including: The dense gradient field is scaled by a downsampling factor and the grid step size is increased to generate a reference grid, wherein the anchor points corresponding to the absolute three-dimensional world coordinates remain unchanged in their original coordinates; The energy functional is iteratively solved on the reference grid. If the relative residual corresponding to the solution of the current iteration number is less than or equal to the second tolerance threshold, the first depth map is output. The first depth map is bilinearly upsampled back to the original resolution to obtain the second depth map; Using the second depth map as the initial solution, the energy functional is iteratively solved at the original resolution to obtain the first intermediate depth map; The first intermediate depth map is adjusted according to the preset size to obtain the dense depth map.

15. The detection method of the visual-tactile sensor according to claim 14, characterized in that, Solving the energy functional yields a dense depth map, including: When the distance between the absolute 3D world coordinates of any of the marked points in the current target image and the absolute 3D world coordinates of the target image in the previous frame is less than a preset distance threshold, the first intermediate depth map obtained by solving the energy functional of the target image in the previous frame is used as the second intermediate depth map. The second intermediate depth map is used as the initial solution, and the energy functional of the current target image is iteratively solved at the original resolution to obtain the first intermediate depth map of the current target image. The first intermediate depth map is then adjusted according to a preset size to obtain the dense depth map. When the distance between the absolute 3D world coordinates of any marker point in the current target image and the absolute 3D world coordinates of the target image in the previous frame is greater than or equal to a preset distance threshold, the dense gradient field corresponding to the current target image is scaled according to a downsampling factor, and the grid step size is increased to regenerate a new reference grid. The energy functional is iteratively solved on the new reference grid, and if the relative residual corresponding to the solution of the current iteration number is less than or equal to a second tolerance threshold, a new first depth map is output. The new first depth map is bilinearly upsampled back to the original resolution to obtain a new second depth map. The energy functional is iteratively solved at the original resolution using the new second depth map as the initial solution to obtain a new first intermediate depth map. The new first intermediate depth map is adjusted according to a preset size to obtain the dense depth map.

16. A control device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the detection method of the visual-tactile sensor as described in any one of claims 1 to 15.

17. A visual-tactile sensor, characterized in that, Includes the control device as described in claim 16.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the detection method of the visual-tactile sensor as described in any one of claims 1 to 15.

Images