Refraction self-calibration method of binocular visual tactile sensor

Abstract

The invention discloses a refraction self-calibration method of a binocular visual tactile sensor, and belongs to the technical field of refraction calibration of visual tactile sensors. The problem that at present, three-dimensional reconstruction errors are difficult to directly serve as optimization targets is solved. The method comprises the following steps: initializing a representation nL of a normal vector of a refraction interface under a first camera coordinate system by adopting coplanar constraint, and initializing a distance d from an optical center of a first camera to an air-support plate interface by adopting collinear constraint and a false point; calculating initial values of a rotation matrix and translation vectors LRO and LtO from the world coordinate system to the first camera coordinate system by combining the initial values of nL and d; based on the refractive index of the supporting plate, the refractive index of the elastomer, nL, d, LRO and LtO initial values, reconstruction errors in three directions are used as a target function F, and refraction system parameters are obtained through minimization.

Description

Technical Field

[0001] This invention belongs to the field of refractive calibration technology of visual-tactile sensors, and relates to a refractive self-calibration method for binocular visual-tactile sensors. Background Technology

[0002] A schematic diagram of the structure of a binocular vision-touch sensor is shown below. Figure 1 As shown, it includes a binocular vision system (left camera 1 and right camera 2), a transparent elastomer 5 as a sensing element, a transparent support plate 4, and a fixed bracket 3. The binocular cameras record the positions of marker points 6 on the free surface of the transparent elastomer and reconstruct the marker points in three dimensions. Contact information is calculated by the displacement of the marker points before and after the object comes into contact with the transparent elastomer. When a pair of matching points on the left and right contact images are given, the marker points are mapped from the two-dimensional image plane to the three-dimensional Cartesian space using the RSRT model. The refractive system parameter set of the binocular tactile sensor is {μ1, μ2, n}. L ,d}. Where μ1 is the refractive index of the support plate, μ2 is the refractive index of the elastomer, and n L Let d represent the normal vector of the refraction interface in the coordinate system of the left camera, where d is the distance from the optical center of the left camera to the air-support plate interface.

[0003] Nonlinear distortion introduced by light refraction can reduce the accuracy of 3D reconstruction in binocular visual tactile sensors. Binocular tactile sensors often employ Refractive Stereo Ray Tracing (RSRT) models to eliminate refraction effects. The essence of the RSRT model is to simulate the path of light propagating through multiple media in an imaging system. With the RSRT model, the sensing accuracy of binocular tactile sensors is highly dependent on model parameters, including camera parameters and refraction system parameters. Existing refraction calibration methods rely on multiple contact depth refraction images obtained from repeated pressing of a calibration plate, which is difficult to apply to scenarios with high sensor hardness and self-calibration scenarios.

[0004] Existing refraction calibration methods calculate refraction system parameters using differential evolution algorithms. The difference between these methods lies in the construction of their objective functions. The MCD method uses contact depth error as the objective function, but this optimization problem is ill-conditioned when the contact depth is a single value. The UMMRC method's objective function is a linear combination of feature point distance error and perpendicularity error. None of the existing objective functions include 3D reconstruction error as the objective function, although the impact of imaging error on 3D sensing is considered. Using reconstruction error as the objective function would introduce additional unknown parameters, namely the coordinate transformation parameters between the camera coordinate system and the world coordinate system. L R O and L t O .when L R O and L tO When used as optimization parameters for the refraction model, the coordinate transformation parameters and refraction parameters are coupled together, so the 3D reconstruction error cannot be directly used as the objective function. Summary of the Invention

[0005] This invention aims to address the problem that current methods struggle to directly use 3D reconstruction errors as optimization targets.

[0006] A refractive self-calibration method for a binocular tactile sensor includes:

[0007] Applying coplanar constraints to n L Initialization is performed using collinearity constraints and pseudopoints to initialize d; combined with n L Given the initial value of d, calculate the pose parameters. L R O and L t O Initial values; based on the refraction system parameters μ1, μ2, n L ,d and L R O , L t O The initial value is set, and the reconstruction error in the three directions is used as the objective function F. This is achieved by minimizing... Obtain the parameters of the refractive system; μ1 is the refractive index of the support plate, μ2 is the refractive index of the elastomer, and n L Let d be the representation of the refractive interface normal vector in the first camera coordinate system, where d is the distance from the optical center of the first camera to the air-support plate interface. L R O and L t O Let be the rotation matrix and translation vector from the world coordinate system to the first camera coordinate system, where the first camera is one of the cameras in the binocular tactile sensor.

[0008] Furthermore, the target function The corresponding 3D point reconstruction error e(e x e y e z )as follows:

[0009]

[0010] Among them, e x e y e z The three coordinate axis components of the reconstruction error e of the three-dimensional points; P O Let be the representation of feature point P in the world coordinate system; d is the distance from the origin of the camera ray to the support plate. For the thickness of the support plate, For the thickness of the elastomer, This represents the dot product operation of two vectors; These are the representations of the camera ray direction vector v0, the refracted ray direction vector v1, and the refracted ray direction vector v2 in the first camera coordinate system, respectively.

[0011] Furthermore, the aforementioned The following is true:

[0012]

[0013]

[0014]

[0015] Where K is the intrinsic parameter matrix of the first camera. These are the generalized coordinates of the image point m of the first camera; , .

[0016] Furthermore, coplanar constraints are applied to n L The initialization process includes:

[0017] For any first camera ray v0, we have:

[0018] (8)

[0019] in, L R O and L t O These are the rotation matrix and translation vector from the world coordinate system to the first camera coordinate system; × represents the cross product operation; E=n L × L R O ;s=n L × L t O ;

[0020] Stacking the equations corresponding to N feature points yields a linear system:

[0021] (9)

[0022] in, This represents the Kronecker product; N is the number of feature points; - Represents the corresponding N feature points ;

[0023] Singular value decomposition is used to solve equation (9) to recover E and s, and then n is obtained. L n L Let E be the left zero singular vector.

[0024] Furthermore, the d obtained by using collinearity constraints and pseudo-point initialization is as follows:

[0025]

[0026] Here, the superscript i in the parameter represents the parameter corresponding to the i-th feature point. Represents the i-th feature point corresponding to , Represents the i-th feature point corresponding to .

[0027] or,

[0028]

[0029] Here, the superscript i in the parameter represents the parameter corresponding to the i-th feature point. Represents the i-th feature point corresponding to , Represents the i-th feature point corresponding to .

[0030] Furthermore, the process of initializing d using collinear constraints and pseudopoints includes:

[0031] The pseudopoint P corresponding to the feature point v For the first camera light O l P0 and the second camera light O r P 0r The intersection point, the second camera is another camera in the binocular tactile sensor; according to the collinearity constraint, P v The relationship between P and P is:

[0032] (10)

[0033] in, Represented as pseudopoint P v In the first camera coordinate system; θ0 represents the angle between the camera ray v0 and the normal to the refraction interface; θ1 represents the angle between the refracted ray v1 and the normal to the refraction interface; θ2 represents the angle between the refracted ray v2 and the normal to the refraction interface.

[0034] According to Snell's law, formula (10) is rewritten, and then the initialization formula of d is derived by least squares method.

[0035] Alternatively, based on this, when the transparent elastomer is a plate with a thickness of d2, This simplifies to the initialization formula for d.

[0036] Further, calculate pose parameters L R O andL t O The initialization process includes:

[0037] Recover from E using the five-point method L R O ,according to recover L t O , The representation of a pseudopoint in the world coordinate system; satisfying L t O The correct result is obtained after the third component is greater than d. L R O and L t O Initial value.

[0038] Furthermore, according to recover L t O During the process It is obtained through the following steps:

[0039] Point P v The line containing P is collinear with the normal vector n of the refraction interface. O =[x t y t , z t ] T and P v O =[x v y v , z v ] T The following relationship exists: x v =x t y v =y t ; combination Determine the relationship P v O .

[0040] Beneficial effects:

[0041] This invention can achieve L R O and L t O As an optimization parameter of the refraction model, the 3D reconstruction error can be directly used as the objective function. Therefore, this invention effectively solves the problem that the 3D reconstruction error is difficult to use directly as the optimization objective, thereby effectively controlling the 3D reconstruction error and reducing the impact of imaging error on 3D sensing.

[0042] Before refraction calibration, the reprojection error of the sensor was 51.239 pixels. After refraction calibration using the method of the present invention, the reprojection error of the sensor was 0.110 pixels, which shows that the present invention has effectively compensated for the influence of light refraction. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of a binocular vision-tactile sensor structure.

[0044] Figure 2 This is an overall flowchart of the refractive self-calibration method for binocular tactile sensors.

[0045] Figure 3 This is a schematic diagram of the reverse reconstruction of the imaging optical path of the left camera in the RSRT model.

[0046] Figure 4 This is a schematic diagram of the optical path for multi-medium binocular refractive imaging.

[0047] Figure 5 This is a schematic diagram of the rotational relationship between the camera and the object obtained from E.

[0048] Figure 6 This is a schematic diagram of the refractive calibration process of a binocular tactile sensor. Detailed Implementation

[0049] To address the problem that 3D reconstruction errors are difficult to directly use as optimization targets, this invention proposes a refraction self-calibration method for binocular tactile sensors to obtain refraction system parameters. This method only requires the refraction image of the binocular tactile sensor when it is not pressed, eliminating the need for multiple presses using a calibration plate. The method initializes the refraction system parameters using coplanar constraints, collinear constraints, and pseudopoints, and optimizes the refraction calibration results based on the 3D reconstruction error.

[0050] Specific implementation method one: Combining Figure 2 This implementation method is described below.

[0051] This embodiment proposes a refractive self-calibration method for a binocular visual tactile sensor, used to calibrate refractive system parameters. It uses the 3D reconstruction error as the objective function and employs refractive system parameters... L R O and L t O The initial value is used to start the optimization process.

[0052] This invention uses prior knowledge to set the initial values ​​of μ1 and μ2, and uses coplanar constraints on n. L Initialization is performed using collinearity constraints and pseudopoints to initialize d; combined with n L Given the initial value of d, calculate the pose parameters. L R O andL t O The initial values ​​are: μ1 is the refractive index of the support plate, μ2 is the refractive index of the elastomer, and n... L Let d be the representation of the refractive interface normal vector in the left camera coordinate system, where d is the distance from the optical center of the left camera to the air-support plate interface; L R O and L t O Let be the rotation matrix and translation vector from the world coordinate system to the left camera coordinate system.

[0053] The objective function is constructed as follows:

[0054] according to Figure 3 For any left camera ray v0, the corresponding feature point P (the marked point covered by the elastic surface of the binocular tactile sensor) is represented in the left camera coordinate system as P L Represented as:

[0055] (1)

[0056] Where d is the distance from the starting point of the camera ray to the support plate. For the thickness of the support plate, For the thickness of the elastomer, This represents the dot product operation of two vectors; These are the representations of the camera ray direction vector v0, the refracted ray direction vector v1 (inside the support plate), and the refracted ray direction vector v2 (inside the elastic body) in the left camera coordinate system, respectively.

[0057] The calculation formula is as follows:

[0058] (2)

[0059] Where K is the intrinsic parameter matrix of the left camera. These are the generalized coordinates of the image point m of the left camera.

[0060] (3)

[0061] in, .

[0062] (4)

[0063] in, .

[0064] The reconstruction error of the corresponding three-dimensional points e(e x e y e z ) expressed as

[0065] (5)

[0066] Among them, e x e y e z The three coordinate axis components of the reconstruction error e of the three-dimensional points; P O Here is the representation of feature point P in the world coordinate system, where the z-axis is perpendicular to the incident and refraction interface Π1 of the support plate. In this custom world coordinate system, P... O The quantity is known.

[0067] For N feature points, the final objective function is a linear combination of the mean values ​​of the absolute values ​​of the reconstruction errors in the three directions:

[0068] (6)

[0069] Here, N represents the number of feature points. Each feature point corresponds to only one camera ray, one v1 refracted ray, and one v2 refracted ray. Therefore, N can be N feature points, N camera rays, N v1 refracted rays, and N v2 refracted rays.

[0070] Thus, the optimization functions for the refraction system parameters and pose parameters are obtained as follows:

[0071] (7)

[0072] To avoid getting trapped in local optima, the fmincon function and the GlobalSearch algorithm are combined in Matlab to optimize equation (7). Specifically, when optimizing using equation (7), the initial values ​​of each parameter in equation (7) are determined through the following steps:

[0073] (1) n L initialization:

[0074] The coplanar constraint is defined as follows: in a refracted light path, the incident ray, the corresponding outgoing ray, and the normal to the refracted interface lie in the same plane. Therefore, the refraction parameter n L This can be calculated using coplanar constraints. For any left camera ray v0, we have:

[0075] (8)

[0076] in, L R O and L t O These are the rotation matrix and translation vector from the world coordinate system to the left camera coordinate system; × represents the cross product operation; E=n L × L R O ;s=nL × L t O .

[0077] By stacking the equations corresponding to N feature points, we can obtain a linear system to calculate n. L :

[0078] (9)

[0079] in, This represents the Kronecker product. N is the number of feature points (N satisfies N>11); - Represents the corresponding N feature points .

[0080] Singular value decomposition is used to solve equation (9). After recovering E and s, n can be obtained. L n L Let be the left zero singular vector of E. Formula (9) is an equation of the form Ax=0, where A is a known matrix; the value of form x can be obtained through singular value decomposition, and the value of x includes E and s.

[0081] (2) d initialization:

[0082] In the multi-media RSRT model, a pseudopoint P is defined corresponding to a feature point P. v For the left camera light O l P0 and right camera light O r P 0r The intersection point. Collinearity constraint is defined as the intersection point of lines P. v The line containing point P is collinear with the normal vector n of the refraction interface; the collinearity constraint is determined by… Figure 4 It is easy to prove that Figure 4 In the diagram, P is a feature point. v It is a pseudopoint; Ω l Ω is the plane containing any incident and refracted light rays from the left camera. r It is the plane containing the incident and refracted rays in the right camera.

[0083] Collinearity constraints are used to initialize d. Taking the view from the left camera as an example ( Figure 4 According to the collinearity constraint, P v The relationship between P and P is:

[0084] (10)

[0085] in, Represented as pseudopoint P vIn the left camera coordinate system: θ0 represents the angle between the camera ray v0 and the normal of the refraction interface; θ1 represents the angle between the refracted ray v1 (inside the support plate) and the normal of the refraction interface; θ2 represents the angle between the refracted ray v2 (inside the elastic body) and the normal of the refraction interface.

[0086] According to Snell's Law, equation (10) can be further written as:

[0087] (11)

[0088] in, , .

[0089] The formula for calculating d is derived using the least squares method:

[0090] (12)

[0091] Here, the superscript i in the parameter represents the parameter corresponding to the i-th feature point. Represents the i-th feature point corresponding to , Represents the i-th feature point corresponding to .

[0092] When the transparent elastomer is a plate with a thickness of d2 Equation (12) simplifies to:

[0093] (13)

[0094] (3) Initialization of pose parameters:

[0095] L R O and L t O The initialization process is as follows:

[0096] The five-point method (an efficient solution to the five-point relative pose problem) is used to recover from E. L R O Recover according to formula (14) L t O ,satisfy L t O The third component (physically defined as the z-direction distance from the camera center to the object) is greater than d, thus obtaining... L R O and L t O Initial value.

[0097] Lt O The calculation formula is:

[0098] (14)

[0099] in, This represents the pseudopoint in the world coordinate system.

[0100] The rotational relationship between the camera and the object obtained from E. L R O have Figure 5 Of the four types, (1) and (3) or (2) and (4) are chosen based on the displacement vector between the two cameras (which is known); the displacement vector between the camera and the object restricts the... L t O Then, determine the unique solution from the two selected cases.

[0101] Because point P v The line containing P is collinear with the normal vector n of the refraction interface. O =[x t y t , z t ] T and P v O =[x v y v , z v ] T The following relationship exists: x v =x t y v =y t In calculating P v O During the process, z can be derived from equation (11). v The expression for P is given by equation (15), and P can be obtained through equation (15). v O Thus, P v O Used for calculating formula (14).

[0102] (15)

[0103] Example

[0104] The structure of the binocular visual tactile sensor in this experiment is as follows: Figure 1 As shown. The support plate is a 5mm thick tempered glass plate. The elastomer is a 10mm thick acrylic plate. Figure 6 As shown, the specific steps for self-calibration are as follows:

[0105] (a) Phase 1: Camera calibration. The camera's internal parameters were calibrated in air using the Zhang Zhengyou calibration method and a checkerboard calibration board.

[0106] (b) Stage 2: Refraction parameter calibration. The binocular camera simultaneously records images of the marked points on the surface of the visual-tactile sensor.

[0107] (c) After collecting images of the marker points, we removed the distortion from these images and then detected the positions of the marker point images in the images.

[0108] (d) Based on prior knowledge such as material properties, the refractive index of the support plate is set to 1.52 and the refractive index of the elastomer is set to 1.49.

[0109] (e) According to n L Initialization, d initialization, and pose parameter initialization steps determine the initial values ​​of each parameter in equation (4) for calculating the refraction system parameters {μ1, μ2, d, n}. L} and pose parameters { L R O , L t O The initial value of}.

[0110] (f) The unknown parameters are nonlinearly optimized according to equation (4) to obtain the refraction calibration results, as shown in Table 1.

[0111] Table 1 Refraction Calibration Results

[0112]

[0113] Before refraction calibration, the reprojection error of the sensor was 51.239 pixels, and after refraction calibration, the reprojection error of the sensor was 0.110 pixels. This shows that the proposed refraction correction method has effectively compensated for the influence of light refraction, and verifies the effectiveness of the proposed refraction correction method.

[0114] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A refractive self-calibration method for a binocular visual tactile sensor, characterized in that, include: Applying coplanar constraints to n L Initialization is performed using collinearity constraints and pseudopoints to initialize d; combined with n L Given the initial value of d, calculate the pose parameters. L R O and L t O Initial values; based on the refraction system parameters μ1, μ2, n L ,d and L R O , L t O The initial value is set, and the reconstruction error in the three directions is used as the objective function F. This is achieved by minimizing... Obtain the parameters of the refractive system; μ1 is the refractive index of the support plate, μ2 is the refractive index of the elastomer, and n L Let d be the representation of the refractive interface normal vector in the first camera coordinate system, where d is the distance from the optical center of the first camera to the air-support plate interface. L R O and L t O Let be the rotation matrix and translation vector from the world coordinate system to the first camera coordinate system, where the first camera is one of the cameras in the binocular tactile sensor.

2. The refractive self-calibration method for a binocular visual tactile sensor according to claim 1, characterized in that, Target Letter The corresponding 3D point reconstruction error e(e x e y e z )as follows: Among them, e x e y e z The three coordinate axis components of the reconstruction error e of the three-dimensional points; P O Let be the representation of feature point P in the world coordinate system; d is the distance from the origin of the camera ray to the support plate. For the thickness of the support plate, For the thickness of the elastomer, Represents the dot product operation of two vectors; These are the representations of the camera ray direction vector v0, the refracted ray direction vector v1, and the refracted ray direction vector v2 in the first camera coordinate system, respectively.

3. The refractive self-calibration method for a binocular visual tactile sensor according to claim 2, characterized in that, The The following is true: Where K is the intrinsic parameter matrix of the first camera. These are the generalized coordinates of the image point m of the first camera; , .

4. The refractive self-calibration method for a binocular visual tactile sensor according to claim 2, characterized in that, Applying coplanar constraints to n L The initialization process includes: For any first camera ray v0, we have: (8) in, L R O and L t O These are the rotation matrix and translation vector from the world coordinate system to the first camera coordinate system; × represents the cross product operation; E=n L × L R O ;s=n L × L t O ; Stacking the equations corresponding to N feature points yields a linear system: (9) in, This represents the Kronecker product; N is the number of feature points; - Represents the corresponding N feature points ; Singular value decomposition is used to solve equation (9) to recover E and s, and then n is obtained. L n L Let E be the left zero singular vector.

5. The refractive self-calibration method for a binocular visual tactile sensor according to claim 4, characterized in that, The following is the result of initializing d using collinearity constraints and pseudopoints: Here, the superscript i in the parameter represents the parameter corresponding to the i-th feature point. Represents the i-th feature point corresponding to , Represents the i-th feature point corresponding to .

6. The refractive self-calibration method for a binocular visual tactile sensor according to claim 5, characterized in that, The process of initializing d using collinearity constraints and pseudopoints includes: The pseudopoint P corresponding to the feature point v For the first camera light O l P0 and the second camera light O r P 0r The intersection point, the second camera is another camera in the binocular tactile sensor; according to the collinearity constraint, P v The relationship between P and P is: (10) in, Represented as pseudopoint P v In the first camera coordinate system; θ0 represents the angle between the camera ray v0 and the normal to the refraction interface; θ1 represents the angle between the refracted ray v1 and the normal to the refraction interface; θ2 represents the angle between the refracted ray v2 and the normal to the refraction interface. According to Snell's law, formula (10) is rewritten, and then the initialization formula of d is derived by least squares method.

7. The refractive self-calibration method for a binocular visual tactile sensor according to claim 4, characterized in that, The following is the result of initializing d using collinearity constraints and pseudopoints: Here, the superscript i in the parameter represents the parameter corresponding to the i-th feature point. Represents the i-th feature point corresponding to , Represents the i-th feature point corresponding to .

8. The refractive self-calibration method for a binocular visual tactile sensor according to claim 7, characterized in that, The process of initializing d using collinearity constraints and pseudopoints includes: The pseudopoint P corresponding to the feature point v For the first camera light O l P0 and the second camera light O r P 0r The intersection point, the second camera is another camera in the binocular tactile sensor; according to the collinearity constraint, P v The relationship between P and P is: (10) in, Represented as pseudopoint P v In the first camera coordinate system; θ0 represents the angle between the camera ray v0 and the normal to the refraction interface; θ1 represents the angle between the refracted ray v1 and the normal to the refraction interface; θ2 represents the angle between the refracted ray v2 and the normal to the refraction interface. According to Snell's law, formula (10) is rewritten, and then d is derived by the least squares method. When the transparent elastomer is a plate with a thickness of d2, This simplifies to the initialization formula for d.

9. A refractive self-calibration method for a binocular visual tactile sensor according to any one of claims 5 to 8, characterized in that, Calculate pose parameters L R O and L t O The initialization process includes: Recover from E using the five-point method L R O ,according to recover L t O , The representation of a pseudopoint in the world coordinate system; satisfying L t O The correct result is obtained after the third component is greater than d. L R O and L t O Initial value.

10. The refractive self-calibration method for a binocular visual tactile sensor according to claim 9, characterized in that, according to recover L t O During the process It is obtained through the following steps: Point P v The line containing P is collinear with the normal vector n of the refraction interface. O =[x t y t , z t ] T and P v O =[x v y v , z v ] T The following relationship exists: x v =x t y v =y t ; combination Determine the relationship P v O .

Images