Three-dimensional haptic reconstruction and calibration method based on single-event camera reflection virtual binoculars

Abstract

The application provides a three-dimensional tactile reconstruction and calibration method based on a single-event camera reflection virtual binocular, which comprises collecting visual pictures of a plurality of mark points reflected by a mirror group from two different angles through a single visual detection mechanism to obtain binocular virtual pictures corresponding to two virtual angles; using a distorted binary grid initial frame generated by an initial visual picture as a structured reference of each region, performing PnP solving based on a regular point array of the binocular virtual pictures to obtain equivalent internal parameters, distortion parameters and relative poses of the two virtual angles; establishing a one-to-one correspondence relationship of homonymous mark points between the two pictures of the binocular virtual pictures according to a row and column topology of the regular point array; performing binocular triangulation on the binocular virtual pictures based on the corresponding relationship of the homonymous mark points to calibrate three-dimensional coordinates of each mark point and reconstruct a three-dimensional displacement field of the elastic contact medium surface.

Description

Technical Field

[0001] This application relates to the technical field of robot tactile perception, specifically to a method for three-dimensional tactile reconstruction and calibration based on a single-event camera reflection virtual binoculars. Background Technology

[0002] Existing robotic tactile sensing technologies typically use visual-tactile sensors to identify, match, and track the displacement of marked points on the surface of an elastic body to deduce contact position, normal deformation, tangential slip, shear force, and contact dynamics. However, while using binocular cameras provides more comprehensive images with more varied features, it is also more expensive. Using monocular cameras, on the other hand, results in images with fewer varied features, making it impossible to achieve 3D tactile reconstruction and calibration, and thus difficult to accurately realize tactile perception. Summary of the Invention

[0003] The purpose of this application is to overcome the shortcomings and deficiencies of the existing technology and provide a three-dimensional tactile reconstruction and calibration method based on a single-event camera reflection virtual binocular vision, applied to a dual-mirror tactile sensing device. The dual-mirror tactile sensing device includes a housing, a plane mirror group, and a monocular vision detection mechanism. The bottom of the housing is provided with an elastic contact medium, and the side of the elastic contact medium facing the plane mirror group has a regular dot matrix including multiple marker points. The plane mirror group is used to reflect the visual image of the multiple marker points from two different angles. The method includes:

[0004] The monocular vision detection mechanism acquires visual images of the multiple marker points reflected from two different angles by the plane mirror group, thereby obtaining binocular virtual images corresponding to the two virtual viewpoints;

[0005] Using the initial frame of the distorted binary mesh generated from the initial visual image as the structured reference for each region, PnP is solved based on the regular dot matrix of the binocular virtual image to obtain the equivalent intrinsic parameters, distortion parameters and relative pose of the two virtual viewpoints.

[0006] Based on the row and column topology of the rule dot matrix, a one-to-one correspondence of the same-name marker points is established between the two images of the binocular virtual screen;

[0007] Based on the correspondence of the same-named marker points, the binocular virtual image is triangulated to calibrate the three-dimensional coordinates of each marker point in order to reconstruct the three-dimensional displacement field of the elastic contact medium surface.

[0008] As one implementation method, the steps of performing PnP solving based on the regular pixel matrix of the binocular virtual image to obtain the equivalent intrinsic parameters, distortion parameters, and relative poses of the two virtual viewpoints include:

[0009] The equivalent intrinsic parameters and distortion parameters for each virtual viewpoint are obtained using the following formulas:

[0010] min_{K_s,d_s,R_s,t_s}Σ_i||p_{s,i}-pi(K_s,d_s,R_sX_i+t_s)||_2^2

[0011] Where K_s is the equivalent intrinsic parameter, d_s is the distortion parameter, R_s and t_s are the extrinsic parameters from the regular point matrix coordinate system to the virtual view coordinate system, pi is the projection function containing distortion, p_{s,i} is the pixel coordinate of the marker point in the corresponding virtual view coordinate system, and X_i is the coordinate of the marker point in the regular point matrix coordinate system.

[0012] In one implementation, the relative pose includes rotation parameters, translation parameters, and a baseline.

[0013] As one implementation method, the step of performing binocular triangulation on the binocular virtual image based on the correspondence of corresponding marker points, and calibrating the three-dimensional coordinates of each marker point to reconstruct the three-dimensional displacement field of the elastic contact medium surface includes:

[0014] Within each event time window, locate the same-named marker points in two separate screens;

[0015] The left and right observation rays are obtained by using calibration parameters to remove distortion and normalize pixel coordinates.

[0016] Based on the center C_L, C_R and relative pose in the equivalent intrinsic parameters of the two virtual viewpoints, find the nearest point between the two rays C_L+lambda d_L and C_R+mu d_R, and take the midpoint of the line connecting the nearest points as the real-time three-dimensional coordinates of the marker point.

[0017] Based on the initial anchor point position of the marker point in the initial visual image and the real-time three-dimensional coordinates, the displacement parameters of the marker point are calculated, and the three-dimensional displacement field of the surface of the elastic contact medium is formed by organizing the points into rows and columns according to the topology of the dot matrix.

[0018] As one implementation method, the step of locating the same-named marker points in two frames within each event time window includes:

[0019] Based on the event time window, the continuous asynchronous event streams of the two screens are fused to obtain a fused state diagram;

[0020] The initial anchor point position and the position of the marker point in the fused state diagram are mapped to a regular dot matrix coordinate system as dot matrix grid points;

[0021] Starting from the center of the lattice points at the previous moment, update the center and radius of the current lattice points in combination with the gating constraints;

[0022] Based on the real-time center, radius, confidence level, and displacement data of the current grid points, locate the corresponding marker points.

[0023] As one implementation, the gating constraints include at least one of search gating, soft / hard displacement gating, neighborhood regularization, velocity clipping, and anchor point regression.

[0024] As one implementation method, after the step of performing binocular triangulation on the binocular virtual image based on the correspondence of the same-named marker points and calibrating the three-dimensional coordinates of each marker point to reconstruct the three-dimensional displacement field of the elastic contact medium surface, the method further includes:

[0025] Based on the preset displacement-force mapping relationship, the contact three-dimensional force information corresponding to the three-dimensional displacement field is obtained.

[0026] In one implementation, the displacement-force mapping relationship is achieved through a neural network; wherein the neural network is trained by taking the three-dimensional displacement sample data of each marked point as input and the corresponding three-dimensional force sample as output.

[0027] In one implementation, the neural network is a multilayer perceptron.

[0028] As one implementation method, the displacement-force mapping relationship is realized by an elastic mechanical model based on finite element analysis; wherein, the elastic mechanical model is constructed based on the three-dimensional displacement sample data and three-dimensional force sample calibration of each marked point.

[0029] Compared to related technologies, this application provides a method for 3D tactile reconstruction and calibration based on a single-event camera reflection virtual binocular system. This method uses a monocular vision detection mechanism to acquire visual images of multiple marker points reflected from two different angles by a plane mirror group, obtaining binocular virtual images corresponding to two virtual viewpoints. Using the initial frame of the distorted binary mesh generated from the initial visual image as a structured reference for each region, PnP solving is performed based on the regular point matrix of the binocular virtual image to obtain the equivalent intrinsic parameters, distortion parameters, and relative poses of the two virtual viewpoints. Based on the row and column topology of the regular point matrix, in... A one-to-one correspondence of corresponding marker points is established between the two images of the binocular virtual image. Based on the correspondence of corresponding marker points, the binocular virtual image is triangulated to calibrate the three-dimensional coordinates of each marker point in order to reconstruct the three-dimensional displacement field of the elastic contact medium surface. This method is suitable for application scenarios where the visual images of multiple marker points are reflected from two different angles by a plane mirror group, and the visual images of two different angles are acquired by a monocular vision detection mechanism. Targeted calibration and reconstruction optimizations have been performed for this application scenario, effectively reducing hardware costs and improving the accuracy and real-time performance of three-dimensional tactile perception.

[0030] To provide a clearer understanding of this application, the specific embodiments of this application will be described below in conjunction with the accompanying drawings. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a three-dimensional tactile reconstruction and calibration method based on a single-event camera reflection virtual binoculars, according to an embodiment of this application.

[0032] Figure 2 This is a cross-sectional view of a dual-mirror tactile sensing device according to an embodiment of this application.

[0033] Figure 3 This is an exploded view of a dual-mirror tactile sensing device according to an embodiment of this application.

[0034] 1. Housing; 11. Receiving cavity; 13. Elastic contact medium; 15. Illumination mechanism; 17. Light-transmitting protective plate; 101. Upper housing; 103. Lower housing; 105. Rear cover; 3. Plane mirror group; 5. Monocular vision inspection mechanism. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0036] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0037] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances. The singular forms "a," "the," and "the" used in this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. The word "if" as used herein can be interpreted as "when," "when," or "in response to determination."

[0038] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0039] Please see Figure 1 This is a schematic diagram of a three-dimensional tactile reconstruction and calibration method based on a single-event camera reflection virtual binoculars according to the first embodiment of this application. The method is applied to a dual-mirror tactile sensing device.

[0040] Please see Figure 2 and Figure 3 The dual-mirror tactile sensing device includes a housing 1, a plane mirror group 3, and a monocular vision detection mechanism 5; wherein, the monocular vision detection mechanism 5 may be a monocular event camera.

[0041] The housing 1 is provided with a receiving cavity 11, and the plane mirror group 3 and the monocular vision detection mechanism 5 are both arranged in the receiving cavity 11; the bottom of the housing 1 is provided with an elastic contact medium 13, and the side of the elastic contact medium 13 facing the plane mirror group 3 is provided with a dot matrix.

[0042] The plane mirror group 3 is disposed in the accommodating cavity 11. The plane mirror group 3 includes two plane mirrors. The two plane mirrors are both inclinedly disposed above the elastic contact medium 13, and a preset clamping angle is left between the two plane mirrors.

[0043] The monocular vision detection mechanism 5 is disposed in the accommodating cavity 11 and faces the plane mirror group 3. It is used to collect two visual images of the elastic contact medium 13 reflected by the two plane mirrors from different angles.

[0044] The two plane mirrors are symmetrically arranged, with a preset angle between them, which is set by the user. Therefore, the dot matrix of the elastic contact medium 13 reflects two visual images through the two plane mirrors, allowing the monocular vision detection mechanism 5 to also acquire both visual images. This eliminates the need for a binocular vision detection mechanism; the monocular vision detection mechanism 5 alone can achieve a virtual binocular stereoscopic perception effect, thereby simplifying the hardware structure and reducing costs while effectively improving the device's integration and image feature acquisition accuracy. Specifically, the two mirrors are tilted inwards, forming a preset angle (e.g., 145°, 140°, 135°, etc.) between them, ensuring that both reflected images (two visual images) fall within the effective field of view of the event camera.

[0045] The dual-mirror tactile sensing device of this embodiment can be integrated into the fingers of a dexterous hand to serve as a tactile unit for the fingertips of a dexterous hand.

[0046] In one feasible embodiment, an illumination mechanism 15 is provided on the inner wall of the housing 1. For example, the illumination mechanism 15 is two light strips, which are respectively arranged on two opposite inner walls of the housing 1 to provide illumination light from opposite directions. In other embodiments, the illumination mechanism 15 may also be a ring light strip or a dot matrix LED light source to provide a uniform and shadow-free lighting environment, ensuring the clarity and consistency of the images acquired by the monocular vision inspection mechanism 5.

[0047] In a feasible embodiment, the accommodating cavity 11 of the housing 1 is further provided with a light-transmitting protective plate 17, which is disposed between the plane mirror assembly 3 and the elastic contact medium 13. The light-transmitting protective plate 17 is made of transparent acrylic or transparent glass. Therefore, the two plane mirrors can reflect the real-time state of the dot matrix on the elastic contact medium 13 normally through the light-transmitting protective plate 17, while also serving as a dustproof, scratch-proof, and isolation layer. This prevents the elastic contact medium 13 from deforming too much and directly contacting the plane mirrors, thereby protecting the two plane mirrors and preventing them from changing their tilt or being damaged by contact with the elastic contact medium 13.

[0048] In one feasible embodiment, the housing 1 includes an upper housing 101, a lower housing 103, and a rear cover 105;

[0049] The connection between the upper housing 101 and the lower housing 103 forms an accommodating cavity 11 including an opening; the rear cover 105 is connected to the upper housing 101 or the lower housing 103 from the opening to encapsulate the housing 1.

[0050] The method for three-dimensional tactile reconstruction and calibration based on single-event camera reflection virtual binocular vision includes:

[0051] S1: The monocular vision detection mechanism acquires the visual images of the multiple marker points reflected from two different angles by the plane mirror group, and obtains binocular virtual images corresponding to the two virtual viewpoints.

[0052] S2: Using the initial frame of the distorted binary mesh generated from the initial visual image as the structured reference for each region, PnP is solved based on the regular dot matrix of the binocular virtual image to obtain the equivalent intrinsic parameters, distortion parameters and relative pose of the two virtual viewpoints.

[0053] Preferably, the equivalent intrinsic parameters and distortion parameters of each virtual viewpoint are obtained using the following formula:

[0054] min_{K_s,d_s,R_s,t_s}Σ_i||p_{s,i}-pi(K_s,d_s,R_sX_i+t_s)||_2^2

[0055] Where K_s is the equivalent intrinsic parameter, d_s is the distortion parameter, R_s and t_s are the extrinsic parameters from the regular point matrix coordinate system to the virtual view coordinate system, pi is the projection function containing distortion, p_{s,i} is the pixel coordinate of the marker point in the corresponding virtual view coordinate system, and X_i is the coordinate of the marker point in the regular point matrix coordinate system.

[0056] The relative pose includes rotation parameters, translation parameters, and a baseline;

[0057] The left-right relative pose is: R = R_RR_L^T;

[0058] The translation parameters and baseline are obtained using the following formula:

[0059] T = t_R - Rt_L;

[0060] b = ||T||;

[0061] Where T is the translation parameter and b is the baseline.

[0062] S3: Based on the row and column topology of the rule dot matrix, establish a one-to-one correspondence between the same-name marker points between the two screens of the binocular virtual screen;

[0063] S4: Based on the correspondence of the same-named marker points, the binocular virtual image is triangulated to calibrate the three-dimensional coordinates of each marker point in order to reconstruct the three-dimensional displacement field of the elastic contact medium surface.

[0064] Preferably, step S4 includes:

[0065] S41: Locate the same-named marker points in two separate screens within each event time window;

[0066] S42: Use calibration parameters to remove distortion and normalize pixel coordinates to obtain two observation rays, left and right;

[0067] S43: Based on the center C_L, C_R and relative pose in the equivalent intrinsic parameters of the two virtual viewpoints, find the nearest point between the two rays C_L+lambda d_L and C_R+mu d_R, and take the midpoint of the line connecting the nearest points as the real-time three-dimensional coordinates of the marker point.

[0068] S44: Based on the initial anchor point position of the marker point in the initial visual image and the real-time three-dimensional coordinates, calculate the marker point displacement parameters and organize them into a three-dimensional displacement field of the surface of the elastic contact medium according to the matrix row and column topology.

[0069] In a feasible embodiment, S41: the step of locating the same-named marker points in two frames respectively within each event time window includes:

[0070] S411: Based on the event time window, merge the continuous asynchronous event streams of the two screens to obtain a merged state diagram;

[0071] S412: Map the initial anchor point position and the marker point position in the fused state diagram to a regular dot matrix coordinate system as dot matrix grid points;

[0072] S413: Starting from the center of the grid points at the previous moment, update the center and radius of the current grid points in combination with the gating constraints;

[0073] Specifically, for each grid point, the system uses the center of the previous time step and the velocity prediction reference center:

[0074] c_ij^pred=c_ij(t-1)+clip(v_ij(t-1),-k_pred*p,k_pred*p)

[0075] Where c_ij^pred is the reference center of the grid point, c_ij(t-1) is the center of the grid point at the previous time step, v_ij(t-1) is the velocity of the grid point at the previous time step, clip indicates that the velocity term is restricted to the range of [-k_pred*p, k_pred*p] according to each coordinate component, k_pred is the prediction clipping coefficient, and the function of k_pred is to limit the influence of the velocity term at the previous time step on the current prediction center, so as to prevent the prediction center from crossing adjacent grid points or exceeding the local search gate. Optionally, k_pred=0.38, p is the grid spacing.

[0076] Here, c_ij^pred is not the final output center, but rather the prediction reference center before the current update. Subsequent search gating, event attribution, displacement gating, radius consistency verification, and neighborhood constraints are all performed around this reference center. This avoids blindly searching from the previous center when local events are sparse, noisy, or moving at high speeds, which could lead to mismatches.

[0077] Then, the observation center is calculated only within the local patch corresponding to that grid point. If a local state graph weighted centroid is used, it can be written as:

[0078] c_ij^obs=sum(w_k*q_k) / sum(w_k)

[0079] Where c_ij^obs is the local state map weight centroid of the grid point, q_k=(u_k,v_k) represents the two-dimensional coordinates of the k-th foreground pixel or event response point within the local search region of the current grid point. u_k and v_k represent the x and y coordinates of the point in the image coordinate system or the local point matrix coordinate system, respectively; w_k represents the weight of the point, which can be determined by the fused state map intensity, ring weight, or distance weight, and sum is...

[0080] The radius of a grid point can be estimated by the weighted distance from the local point to the center:

[0081] r_ij^obs=clip(sum(w_k*||q_k-c_ij^obs||) / sum(w_k),r_min,r_max)

[0082] Where r_ij^obs is the radius of the grid points, r_min is the lower limit of the marker point radius, and r_max is the upper limit of the marker point radius. If the estimated radius is less than r_min, the candidate region is considered to be noise; if the estimated radius is greater than r_max, the candidate region is considered to be multiple marker points stuck together or an anomalous cluster, which can be used to constrain the reasonable range of the observation radius.

[0083] Then the observation center is written back to the grid point state:

[0084] c_ij(t)=c_ij(t-1)+alpha*(c_ij^obs-c_ij(t-1))

[0085] Wherein, alpha is the fusion coefficient. During normal tracking, alpha can be 0.72, and during recovery / refresh, alpha can be 0.88.

[0086] If the current displacement exceeds the soft threshold but not the hard threshold, the fusion coefficient is reduced: alpha' = alpha * D_soft / d. Here, alpha' is the reduced fusion coefficient, alpha is the original fusion coefficient, d = ||c_ij^obs - c_ij(t-1)||, and D_soft = k_soft * p, where D_soft is the soft displacement threshold, k_soft is the soft displacement threshold coefficient, and p is the grid spacing. When the current original observed displacement d is greater than the soft displacement threshold D_soft but not the hard displacement threshold D_hard, the observation is not directly rejected. Instead, the fusion coefficient of the current observation is reduced by D_soft / d, so that larger displacements are deweighted before participating in the update. As an optional implementation, k_soft = 0.18.

[0087] Updated and rewritten:

[0088] state.pos_uv[i,j]=c_ij(t)

[0089] state.radius[i,j]=r_ij^obs

[0090] state.confidence[i,j]=confidence

[0091] state.missed[i,j]=0

[0092] state.active[i,j]=True

[0093] The code written back described above is merely an example. Its function is as follows: if the observation center of the current grid point passes the search gating, displacement gating, and radius consistency check, the current grid point state is updated to a valid tracking state, the current center, radius, confidence level, and displacement data are recorded, and the number of consecutive non-matches for that grid point is reset to zero; if the check fails, the grid point state is maintained or predicted, and its confidence level is reduced. In other feasible embodiments, the above functions can be implemented using other forms of code.

[0094] The gating constraints include at least one of search gating, soft / hard displacement gating, neighborhood regularization, velocity clipping, and anchor point regression.

[0095] Specifically, when updating the center and radius of the current grid points, the thresholds for various gating constraints are not fixed pixel values, but are determined proportionally based on the grid pitch of the regular grid. The grid pitch is obtained by dividing the span of each side of the grid by the corresponding number of rows / columns. As a result, the search range, single-frame displacement limit, prediction clipping amount, and velocity limit all adaptively scale with the grid density, naturally conforming to the topology.

[0096] Specifically, for each ROI (Region of Interest), let the length of its quadrilateral's top side be W_top, its bottom side be W_bottom, its left side be H_left, its right side be H_right, and the regular point matrix be rowsxcols, then:

[0097] p_u=0.5*(W_top+W_bottom) / (cols-1);

[0098] Where p_u represents the estimated horizontal grid spacing of the ROI. W_top is the length of the top edge of the ROI, W_bottom is the length of the bottom edge of the ROI, and cols is the number of regular point arrays. The average horizontal grid spacing can be obtained by dividing the average of the top and bottom edge lengths by the column spacing cols-1.

[0099] p_v=0.5*(H_left+H_right) / (rows-1);

[0100] Where p_v represents the estimated vertical grid spacing of the ROI. H_left is the length of the left side of the ROI, H_right is the length of the right side of the ROI, and rows is the number of rows in the regular grid. The average vertical grid spacing can be obtained by dividing the average length of the left and right sides by the row spacing rows-1.

[0101] p = max(p_min, 0.5 * (p_u + p_v));

[0102] Where p is the estimated uniform grid spacing for the current ROI or the current regular point matrix. It is obtained by averaging the horizontal grid spacing p_u and the vertical grid spacing p_v, and a minimum lower limit is set by p_min to avoid the gating threshold being too small when the point matrix is ​​too small or the calibration is abnormal. This grid spacing p is subsequently used to adaptively determine parameters such as the search range, predicted clipping amount, single-frame displacement upper limit, velocity upper limit, and radius upper limit.

[0103] The search range, single-frame displacement limit, prediction cropping amount, and velocity limit are all scaled by p, for example:

[0104] Normal search range: R_search=max(2,0.56p)

[0105] Refresh / restore search range: R_refresh=max(2,0.72p)

[0106] Inactive or refreshed grid search range: R_inactive ≈ 1.18 * R_search or 1.18 * R_refresh

[0107] Predicted cut amount: D_pred=0.38p

[0108] Normal single-cycle displacement limit: D_frame=0.18p

[0109] Restore / refresh single-cycle displacement limit: D_recovery=0.28p

[0110] Maximum speed: V_max = max(0.75, 0.16p)

[0111] Upper limit of candidate radius: R_marker_max=min(13.0,0.52p)

[0112] With this limitation, the search range automatically increases when the pixel size is large and automatically decreases when the pixel size is small, avoiding hardcoding the parameters to a certain pixel scale.

[0113] Search gating: Normal tracking and refresh / recovery use different search range ratios. The search gate and single-frame displacement limit during refresh / recovery are greater than those during normal tracking. The single-frame displacement limit can also be slightly widened or tightened depending on the current grid point confidence, whether it is an edge grid point, and the stretching state relative to the initial anchor point.

[0114] The relaxation measures include: 1. Relaxing the search gate and single-cycle displacement limit in refresh / recovery mode; 2. Relaxing the limit for edge grid points, as edge points are more susceptible to occlusion, mirror boundaries, and perspective errors; 3. Relaxing the limit when the current grid point has been significantly stretched relative to the initial anchor point; 4. Appropriately relaxing the limit when the current grid point has low confidence or has just recovered, in order to recapture.

[0115] Tightening occurs in the following situations: 1. Tightening when a grid point is stably activated and has high confidence, to prevent jumping to adjacent grid points; 2. Tightening or rejecting when a candidate point's displacement relative to the previous center is too large; 3. Rejecting when a candidate point is closer to the predicted position of an adjacent grid point; 4. Rejecting or downweighting when the candidate's movement direction clearly conflicts with the movement trend of the neighborhood; 5. Pruning when a candidate point's displacement relative to the initial anchor point exceeds the anchor point limit.

[0116] Soft / Hard Displacement Gate: When a grid point is stably activated and its confidence level is higher than the threshold, and the original displacement exceeds the hard threshold, the update is rejected directly. When the original displacement exceeds the soft threshold but does not reach the hard threshold, it is not rejected. Instead, the fusion coefficient of the observation is reduced by "soft threshold ÷ original displacement" before being written back (large displacements are downweighted rather than discarded).

[0117] Stable activation refers to a situation where the grid point is currently valid, has not been lost in the most recent processing cycle, and the tracking confidence has reached a stable threshold, i.e., active=True, missed=0, and confidence>=0.72.

[0118] It's important to note that the confidence score is not the one output by the event camera hardware, but rather the tracking confidence score maintained by the algorithm for each grid point. It is derived from a combination of factors including candidate point quality, roundness, fill rate, number of supporting pixels, local peak intensity, and offset from the prediction center. This can be written as:

[0119] confidence=clip(a0+a1*circularity+a2*extent+a3*support_ratio+a4*peak_ratio-a5*step_penalty,0,1)

[0120] Where, circularity is the roundness of the candidate points; extent is the fill rate, which is the ratio of the area of ​​the candidate connected region to the area of ​​its bounding rectangle, and can be written as fill rate extent = area / (bbox_w * bbox_h), where area is the area of ​​the candidate connected region, bbox_w and bbox_h are the width and height of the bounding rectangle of the candidate connected region, respectively, extent_min is the minimum fill rate threshold; support_ratio is the proportion of local effective support points; peak_ratio is the proportion of local peak intensity; step_penalty is the penalty term for the deviation of the observation center from the prediction center.

[0121] Neighborhood regularization: The median of the displacements of adjacent grid points is taken as the target displacement. When the number of adjacent grid points is not less than the set value, the current confidence level is not higher than the set value, and the difference between the target and the measured displacement of this grid point exceeds the set threshold, the displacement of this grid point is corrected to the target displacement according to the mixing coefficient, and the correction amount is clipped to the range determined by the grid spacing proportionally; the radius can also be slightly closer to the neighborhood radius.

[0122] The "mixing coefficient" is not a completely fixed constant. More accurately, it is a coefficient that is dynamically calculated based on a preset base value as an upper limit or base weight, and then according to the number of neighbors, the confidence level of the current grid point, and the degree of deviation.

[0123] In this algorithm, the mixing coefficients for neighborhood correction are written as:

[0124] lambda=clip(lambda0*neighbor_factor*confidence_factor*trigger_factor,0,1)

[0125] Wherein, lambda0 is the base coefficient for the correction value, optionally lambda0=0.18; neighbor_factor is the first correction value, which is related to the number of effective neighbors; confidence_factor is the second correction value, which is related to the confidence of the current grid point, and the lower the confidence, the stronger the correction; trigger_factor is the third correction value, which is related to the degree to which the deviation exceeds the trigger threshold.

[0126] The neighborhood correction formula is: d_corr=d_meas+lambda*(d_neighbor-d_meas), and the maximum correction amount is limited to: ||d_corr-d_meas||≤0.07p;

[0127] Where d_meas is the measured displacement of the current grid point, d_neighbor is the median or weighted average of the displacements of neighboring grid points, d_corr is the corrected displacement after neighborhood regularization, and lambda is the neighborhood correction mixing coefficient. This formula means that the measured displacement of the current grid point is aligned with the uniform displacement of the neighborhood according to a certain proportion.

[0128] d_neighbor-d_meas represents the deviation between the neighborhood reference displacement and the current grid point's measured displacement. The larger the deviation, the more inconsistent the motion trend of the current grid point is with the surrounding grid points; when the trigger condition is met, the current grid point's displacement can be corrected by neighborhood regularization based on this deviation.

[0129] d_corr-d_meas represents the correction amount introduced by neighborhood regularization to the current measured displacement. To avoid the neighborhood constraint excessively altering the true local deformation, this correction amount needs to be capped, for example, ||d_corr-d_meas||≤0.07p, where p is the grid spacing.

[0130] The radius can also slightly converge towards the radius of its neighboring region:

[0131] r_corr=(1-beta_r)*r_meas+beta_r*r_neighbor

[0132] The above formula represents a neighborhood smoothing correction for the radius of the current grid point. Here, r_meas is the measured radius of the current grid point, r_neighbor is the median or weighted average of the radii of adjacent grid points, r_corr is the corrected radius, and beta_r is the radius correction coefficient. This correction is used to suppress radius jitter when local observation quality is low.

[0133] Velocity clipping and anchor point regression: The grid velocity state is clipped to an upper limit determined proportionally by the grid spacing; hysteresis memory is enabled only when the displacement is small, close to the initial anchor point, and is indeed regressing to the anchor point, making the area near the anchor point more stable and less prone to jitter.

[0134] "Small displacement" means that the movement of the updated center relative to the center at the previous time step does not exceed max(0.5, 0.08p); "closer to the initial anchor point" means that the distance between the updated center and the initial anchor point of this grid does not exceed max(0.75, 0.18p). Hysteresis memory is enabled only when the current confidence is not lower than 0.45, the update is a small displacement, the updated center is closer to the initial anchor point, and the distance to the anchor point after the update is reduced by at least 0.20 compared to before the update, so as to make the marker more stable when returning to the vicinity of the initial anchor point.

[0135] In this embodiment, each grid point is initialized with an initial anchor point: a_ij=(u_ij^0,v_ij^0). The center of the current grid point at the previous time step is c_ij(t-1), and the candidate center or fused center after this update is c_new. "Small displacement" refers to a small update step size, i.e., ||c_new-c_prev||≤D_small, where D_small=max(0.5,0.08p), and p is the grid spacing of the regular point matrix. It should be noted that c_new corresponds to the center obtained after the candidate observation and fusion, but it is not necessarily the final output c_ij(t), as it may be corrected by a hysteresis memory term to obtain c_out. The center at the previous time step is recorded as c_ij(t-1), and the candidate fused center is c_ij(t). If the hysteresis memory term is not enabled, the current center c_ij(t) = c_ij(t); if the hysteresis memory term is enabled, the final output center is c_ij(t) = c_out.

[0136] "Closer to the initial anchor point" means that the center of the updated point is closer to the initial anchor point, i.e., ||c_new-a_ij||≤D_anchor_near, where: D_anchor_near=max(0.75,0.18p).

[0137] Furthermore, enabling hysteresis not only requires "small displacement" and "closeness to the initial anchor point," but also requires that the current movement is indeed regressing to the anchor point. That is, the distance to the anchor point after the update should be less than the distance to the anchor point before the update: ||c_prev-a_ij||-||c_new-a_ij||>=D_progress, where D_progress=0.20, and the current observation confidence needs to meet the minimum requirement: confidence>=0.45. Therefore, the hysteresis enabling condition can be written as:

[0138] confidence>=0.45

[0139] ||c_new-c_prev||≤max(0.5,0.08p)

[0140] ||c_new-a_ij||≤max(0.75,0.18p)

[0141] ||c_prev-a_ij||-||c_new-a_ij||>=0.20

[0142] Only when the above conditions are met is the grid point considered to be in a state of "small displacement, close to the initial anchor point, and returning to the anchor point", thus enabling the hysteresis memory term. Its function is to prevent the marker from bouncing back and forth slightly due to event noise or binarization jitter when approaching the initial position.

[0143] The corresponding hysteresis correction can be written as: c_mem = c_prev + gamma * (c_new - c_prev), where gamma = 0.42, and then mixed according to the degree of proximity to the anchor point:

[0144] eta=1-||c_new-a_ij|| / D_anchor_near;

[0145] Where eta is the hysteresis mixing coefficient near the anchor point, used to control the mixing ratio between candidate centers and memory centers. It can be written as eta = clip(1 - ||c_new - a_ij|| / D_anchor_near, 0, 1). The closer the current center is to the initial anchor point a_ij, the larger eta is, and the stronger the effect of the hysteresis memory term; when the current center is far from the initial anchor point, eta approaches 0, and the hysteresis memory term does not participate or participates weakly.

[0146] D_anchor_near is a distance threshold to the initial anchor point, used to determine whether the current grid point has returned to the vicinity of the initial anchor point. As an optional implementation, D_anchor_near = max(0.75, 0.18p), where p is the grid spacing. When ||c_new - a_ij|| ≤ D_anchor_near, the current center can be considered to be near the initial anchor point.

[0147] c_out=(1-eta)*c_new+eta*c_mem;

[0148] Where c_out is the final output center after correction by the hysteresis memory term. When conditions such as small displacement, proximity to the initial anchor point, and regression to the anchor point are met, the memory center c_mem = c_prev + gamma * (c_new - c_prev) can be calculated first, and then the final output center can be obtained by c_out = (1 - eta) * c_new + eta * c_mem. When the hysteresis memory term is enabled, the current grid center is updated to c_ij(t) = c_out; when it is not enabled, c_ij(t) = c_new can be directly taken; eta is restricted to between 0 and 1.

[0149] Optionally, the constraint gate may also include other optional constraints, such as motion consistency, return direction gate, spatial guard, etc.

[0150] S414: Locate the corresponding marker point based on the real-time center, radius, confidence level, and displacement data of the current grid points.

[0151] In a feasible embodiment, after the step of performing binocular triangulation on the binocular virtual image based on the correspondence of the same-named marker points and calibrating the three-dimensional coordinates of each marker point to reconstruct the three-dimensional displacement field of the elastic contact medium surface, the method further includes:

[0152] S5: Based on the preset displacement-force mapping relationship, obtain the contact three-dimensional force information corresponding to the three-dimensional displacement field.

[0153] In one feasible embodiment, the displacement-force mapping relationship is implemented through a neural network; wherein the neural network is trained by taking the three-dimensional displacement sample data of each marked point as input and the corresponding three-dimensional force sample as output. Preferably, the neural network is a multilayer perceptron; in other embodiments, the displacement-force mapping relationship can also be implemented by a point cloud neural network acting on a set of three-dimensional displacement points.

[0154] In one feasible embodiment, the displacement-force mapping relationship is realized by an elastic mechanical model based on finite element analysis; wherein the elastic mechanical model is constructed based on three-dimensional displacement sample data and three-dimensional force sample calibration of each marked point.

[0155] Compared to related technologies, this application provides a method for 3D tactile reconstruction and calibration based on a single-event camera reflection virtual binocular system. This method uses a monocular vision detection mechanism to acquire visual images of multiple marker points reflected from two different angles by a plane mirror group, obtaining binocular virtual images corresponding to two virtual viewpoints. Using the initial frame of the distorted binary mesh generated from the initial visual image as a structured reference for each region, PnP solving is performed based on the regular point matrix of the binocular virtual image to obtain the equivalent intrinsic parameters, distortion parameters, and relative poses of the two virtual viewpoints. Based on the row and column topology of the regular point matrix, in... A one-to-one correspondence of corresponding marker points is established between the two images of the binocular virtual image. Based on the correspondence of corresponding marker points, the binocular virtual image is triangulated to calibrate the three-dimensional coordinates of each marker point in order to reconstruct the three-dimensional displacement field of the elastic contact medium surface. This method is suitable for application scenarios where the visual images of multiple marker points are reflected from two different angles by a plane mirror group, and the visual images of two different angles are acquired by a monocular vision detection mechanism. Targeted calibration and reconstruction optimizations have been performed for this application scenario, effectively reducing hardware costs and improving the accuracy and real-time performance of three-dimensional tactile perception.

[0156] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0157] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function selected in one or more boxes.

[0158] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function selected in one or more boxes.

[0159] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0160] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0161] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0162] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0163] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for three-dimensional tactile reconstruction and calibration based on a single-event camera reflection virtual binoculars, characterized in that, An application is made to a dual-mirror tactile sensing device, the dual-mirror tactile sensing device comprising a housing, a plane mirror assembly, and a monocular vision detection mechanism. The bottom of the housing is provided with an elastic contact medium, and the side of the elastic contact medium facing the plane mirror assembly has a regular dot matrix including multiple marked points. The plane mirror assembly is used to reflect the visual image of the multiple marked points from two different angles; the method includes: The monocular vision detection mechanism acquires visual images of the multiple marker points reflected from two different angles by the plane mirror group, thereby obtaining binocular virtual images corresponding to the two virtual viewpoints; Using the initial frame of the distorted binary mesh generated from the initial visual image as the structured reference for each region, PnP is solved based on the regular dot matrix of the binocular virtual image to obtain the equivalent intrinsic parameters, distortion parameters and relative pose of the two virtual viewpoints. Based on the row and column topology of the rule dot matrix, a one-to-one correspondence of the same-name marker points is established between the two images of the binocular virtual screen; Based on the correspondence of the same-named marker points, the binocular virtual image is triangulated to calibrate the three-dimensional coordinates of each marker point in order to reconstruct the three-dimensional displacement field of the elastic contact medium surface.

2. The method for three-dimensional tactile reconstruction and calibration based on single-event camera reflection virtual binoculars according to claim 1, characterized in that, The steps for performing PnP solving based on the regular pixel array of the binocular virtual image to obtain the equivalent intrinsic parameters, distortion parameters, and relative poses of the two virtual viewpoints include: The equivalent intrinsic parameters and distortion parameters for each virtual viewpoint are obtained using the following formulas: min_{K_s,d_s,R_s,t_s}Σ_i||p_{s,i}-pi(K_s,d_s,R_sX_i+t_s)||_2^2 Where K_s is the equivalent intrinsic parameter, d_s is the distortion parameter, R_s and t_s are the extrinsic parameters from the regular point matrix coordinate system to the virtual view coordinate system, pi is the projection function containing distortion, p_{s,i} is the pixel coordinate of the marker point in the corresponding virtual view coordinate system, and X_i is the coordinate of the marker point in the regular point matrix coordinate system.

3. The method for three-dimensional tactile reconstruction and calibration based on single-event camera reflection virtual binoculars according to claim 2, characterized in that, The relative pose includes rotation parameters, translation parameters, and a baseline.

4. The method for three-dimensional tactile reconstruction and calibration based on single-event camera reflection virtual binoculars according to claim 1, characterized in that, The steps of performing binocular triangulation on the binocular virtual image based on the correspondence of corresponding marker points, and calibrating the three-dimensional coordinates of each marker point to reconstruct the three-dimensional displacement field of the elastic contact medium surface include: Within each event time window, locate the same-named marker points in two separate screens; The left and right observation rays are obtained by using calibration parameters to remove distortion and normalize pixel coordinates. Based on the center C_L, C_R and relative pose in the equivalent intrinsic parameters of the two virtual viewpoints, find the nearest point between the two rays C_L+lambdad_L and C_R+mu d_R, and take the midpoint of the line connecting the nearest points as the real-time three-dimensional coordinates of the marker point. Based on the initial anchor point position of the marker point in the initial visual image and the real-time three-dimensional coordinates, the displacement parameters of the marker point are calculated, and the three-dimensional displacement field of the surface of the elastic contact medium is formed by organizing the points into rows and columns according to the topology of the dot matrix.

5. The method for three-dimensional tactile reconstruction and calibration based on single-event camera reflection virtual binoculars according to claim 4, characterized in that, The steps for locating the same-named marker points in two frames within each event time window include: Based on the event time window, the continuous asynchronous event streams of the two screens are fused to obtain a fused state diagram; The initial anchor point position and the position of the marker point in the fused state diagram are mapped to a regular dot matrix coordinate system as dot matrix grid points; Starting from the center of the lattice points at the previous moment, update the center and radius of the current lattice points in combination with the gating constraints; Based on the real-time center, radius, confidence level, and displacement data of the current grid points, locate the corresponding marker points.

6. The method for three-dimensional tactile reconstruction and calibration based on single-event camera reflection virtual binoculars according to claim 5, characterized in that, The gating constraints include at least one of search gating, soft / hard displacement gating, neighborhood regularization, velocity clipping, and anchor point regression.

7. The method for three-dimensional tactile reconstruction and calibration based on single-event camera reflection virtual binoculars according to claim 1, characterized in that, After the step of performing binocular triangulation on the binocular virtual image based on the correspondence of the same-name marker points, and calibrating the three-dimensional coordinates of each marker point to reconstruct the three-dimensional displacement field of the elastic contact medium surface, the method further includes: Based on the preset displacement-force mapping relationship, the contact three-dimensional force information corresponding to the three-dimensional displacement field is obtained.

8. The method for three-dimensional tactile reconstruction and calibration based on single-event camera reflection virtual binoculars according to claim 7, characterized in that, The displacement-force mapping relationship is realized through a neural network; wherein the neural network is trained by taking the three-dimensional displacement sample data of each marked point as input and the corresponding three-dimensional force sample as output.

9. A method for three-dimensional tactile reconstruction and calibration based on a single-event camera reflection virtual binoculars according to claim 8, characterized in that, The neural network is a multilayer perceptron.

10. A method for three-dimensional tactile reconstruction and calibration based on a single-event camera reflection virtual binoculars according to claim 7, characterized in that, The displacement-force mapping relationship is realized by an elastic mechanical model based on finite element analysis; wherein, the elastic mechanical model is constructed based on the three-dimensional displacement sample data and three-dimensional force sample calibration of each marked point.

Images