Calibration method of optical tracking and depth camera coordinates based on a new calibration model

Abstract

The application provides a kind of optical tracking and depth camera coordinate calibration method based on new type calibration model, the method includes providing a chessboard with optical positioning mark, the chessboard is placed in the effective range of optical tracking device, and the front of the chessboard is towards depth camera;The spatial position and posture of optical mark point are obtained, and the position of each corner point in the optical tracking coordinate system is calculated according to the position offset of each corner point on the chessboard to the optical mark point;Image information of the chessboard is obtained, corner point coordinates of the chessboard are extracted, and the corner point coordinates are converted to depth camera coordinate system;When more than three groups of corresponding corner point coordinates are obtained, the spatial transformation matrix of depth camera coordinate system to optical tracking coordinate system is calculated, and the calibration process is completed.The application can solve the problems of traditional calibration method, such as the need for multiple manual movement of optical positioning mark and the difficulty of automatic extraction of mark point position for depth camera, and can automatically and quickly realize accurate calibration.

Description

Technical Field

[0001] This invention relates to the field of machine vision calibration technology, and specifically to a calibration method for optical tracking and depth camera coordinates based on a novel calibration model. Background Technology

[0002] With the development of imaging and stereotactic technologies, surgical navigation systems are increasingly being used in orthopedic, neurosurgical, and otolaryngological surgeries. A crucial component of surgical navigation systems is instrument tracking technology. Commonly used tracking techniques include optical tracking, mechanical tracking, ultrasonic tracking, and electromagnetic tracking, with optical tracking currently being the mainstream method. Optical tracking uses an optical positioning marker as the target, with a reflective ball fixed to it. Infrared light sources are installed around the camera, and the infrared light reflected from the ball is captured by the camera. This allows the camera to calculate the relative spatial position of the optical positioning marker within the optical tracking system's coordinate system, providing real-time positional information for surgical guidance.

[0003] A depth camera is a technology for acquiring 3D surface point cloud data. Mainstream depth cameras primarily use three techniques: structured light method, which uses a laser to emit pre-defined images with specific structural features and calculates depth information based on changes in these features; time-of-flight (ToF) method, which continuously emits laser pulses to the target and calculates the round-trip time based on the reflected pulse information obtained by a sensor to determine the precise distance; and binocular method, which uses two cameras placed at a certain distance to capture two images of the object being measured and obtains the object's 3D geometric information by calculating the positional deviation of corresponding points. In medical imaging, the Kinect depth camera, developed by Microsoft, is commonly used. It uses the time-of-flight method and can simultaneously capture color images and point cloud data. One existing technology uses Kinect to track the movement of the needle during surgery; another uses Kinect to provide initial data for 2D / 3D registration; another uses Kinect to track head movement during positron emission tomography (PET) imaging of the brain; another integrates Kinect into an augmented reality system that displays forensic data; and yet another has developed a Kinect-based percutaneous puncture surgical navigation system. It can be said that depth cameras like Kinect are playing an increasingly important role in the medical field.

[0004] Compared to positional information obtained through optical tracking, point cloud data acquired by depth cameras provides geometric information about the surfaces of various objects in the scene and the positional relationships between them. Optical tracking, on the other hand, only provides the location of a single point to which an optical positioning marker is attached. Therefore, point cloud data from depth cameras plays an irreplaceable role in surgical navigation. However, since the two devices use different coordinate systems, additional processing is required in subsequent steps if they are not unified to the same coordinate system. To avoid this extra processing, a calibration method is needed to calculate the transformation relationship between the two coordinate systems, converting one coordinate system to the other.

[0005] P t =R·P s +T#(11)

[0006] Calculating the transformation matrix between two coordinate systems (assuming it's a rigid body transformation) actually requires that R and T in equation (1) satisfy equation (1) for any pair of P. t P s All are true, among which P t It is the position of a point in space within the target coordinate system, P. s This is the position of the point in the original coordinate system, where R represents the rotation transformation matrix and T represents the translation transformation matrix. The two can also be combined, and the above formula (11) can be written as formula (12):

[0007] P t =T s→t ·P s #(12)

[0008] Among them, T s→t Make any pair of P t P s Both are valid.

[0009]

[0010] For a particle that rotates α about the x-axis, β about the y-axis, and γ about the z-axis, its displacement in the x-axis direction is x. t The displacement in the y-axis direction is y t The displacement in the z-axis direction is z tThe transformation matrix is ​​of the form (13). The transformation matrix is ​​usually solved by Berthold's method or Arun's method. Both methods require at least three pairs of corresponding points. Therefore, how to obtain these three pairs of corresponding points is the key problem that needs to be solved. To obtain the coordinates of a point in the optical tracking system, existing positioning markers can be used to determine the coordinates of the position pointed to by the needle tip. There are several ways to obtain the coordinates of a specific point through a depth camera: obtain the coordinates of the visible light image based on the image features in the visible light image, and then transform them into the depth camera coordinate system. This transformation process usually depends on the software development kit (SDK) provided by the depth camera manufacturer; one existing method uses a calibration device with depth changes to extract the coordinates of points with obvious features on the calibration device based on the captured point cloud data. However, no matter which method is used, it is necessary to manually move the optical markers to obtain multiple positions in the optical tracking coordinate system. If the coordinates in the depth camera coordinate system are extracted using point cloud data, there are still problems such as large amount of computation and slow speed.

[0011] Therefore, the existing technology has the following drawbacks:

[0012] 1. It is necessary to manually move the positioning markers with optical markers multiple times to obtain points in multiple sets of optical tracking coordinate systems.

[0013] 2. It is difficult to easily obtain the corresponding point of the point in the optical tracking coordinate system in the depth camera coordinate system. Summary of the Invention

[0014] The purpose of this invention is to provide a calibration method for optical tracking and depth camera coordinates based on a novel calibration model. This method avoids the problems of traditional calibration methods, such as the need for multiple manual movements of optical positioning markers and the difficulty of automatically extracting marker point positions by the depth camera. It can achieve automatic, fast, and accurate calibration.

[0015] To solve the above problems, the technical solution adopted by the present invention is as follows:

[0016] A novel calibration model-based method for optical tracking and depth camera coordinate calibration includes the following steps:

[0017] A checkerboard pattern with optical positioning marks is provided, the checkerboard pattern is placed within the effective range of an optical tracking device, and the front of the checkerboard pattern faces the depth camera;

[0018] The spatial position and orientation of the optical markers are obtained through an optical tracking system, and the position of each corner point in the optical tracking coordinate system is calculated based on the positional offset of each corner point on the chessboard from the optical marker.

[0019] The image information of the chessboard is obtained by a depth camera, the coordinates of the corner points of the chessboard are extracted, and the coordinates of the corner points are transformed into the depth camera coordinate system.

[0020] When more than three sets of corresponding corner point coordinates are obtained, the spatial transformation matrix from the depth camera coordinate system to the optical tracking coordinate system is calculated to complete the calibration process.

[0021] According to the present invention, an optical tracking and depth camera coordinate calibration method based on a novel calibration model is provided, wherein the optical positioning markers are three infrared optically sensitive small balls fixed in an L-shape to each other on one side of the chessboard grid.

[0022] According to the present invention, an optical tracking and depth camera coordinate calibration method based on a novel calibration model is provided, wherein obtaining the spatial position and orientation of optical marker points through an optical tracking device includes:

[0023] The spatial positions of the three infrared optically sensitive spheres relative to the optical tracking system's own coordinate system {O} are obtained using an optical tracking system, and their distribution is represented as: p1=[x1,y1,z1] T p2 = [x2, y2, z2] T p3 = [x3, y3, z3] T ;

[0024] Based on the positions of points p1, p2, and p3, a local coordinate system {L} is constructed for the L-shaped calibration frame. The X-axis is X = (p2-p1) / ||p2-p1||, and the Y-axis is Y = (p3-p1) / ||p3-p1||. The Z-axis can then be expressed as the cross product of the X and Y axes, Z = X × Y. Therefore, the spatial transformation matrix from {L} to {O} is: T L→O =[X,Y,Z].

[0025] According to the present invention, an optical tracking and depth camera coordinate calibration method based on a novel calibration model is provided, which is based on the relative position p of the L-shaped marker frame and the checkerboard marker frame. i ={x1+iΔx,y1+jΔy,Δz} i＝1～7,j＝1～5 The coordinates of the K interior corner points on the chessboard grid in the optical tracking system coordinate system {O} are obtained as follows:

[0026] According to the present invention, an optical tracking and depth camera coordinate calibration method based on a novel calibration model is provided. The Harris algorithm is used to extract the corner points in the chessboard, and a fixed window is used to slide in any direction on the acquired image. The degree of change of pixel grayscale in the window before and after the sliding is compared.

[0027] If there is a significant change in grayscale when sliding in any direction, then the window is considered to have a corner point.

[0028] Let the grayscale transformation of image I at coordinates (x, y) be expressed as: R = ∑(I(x+u,y+v)-I(x,y)) 2 By performing a Taylor expansion on the image grayscale function I(x,y), we obtain formula (1):

[0029]

[0030] Transform formula (1) into formula (2):

[0031]

[0032] Among them, the middle matrix of formula (2) is the covariance matrix, which represents the rate of change of intensity in all directions;

[0033] By calculating the eigenvalues ​​of the local covariance matrix of the image, the coordinate positions (u,v) of the inner corner points on the chessboard grid are determined in the image. These coordinate positions (u,v) are two-dimensional pixel coordinates.

[0034] According to the present invention, a method for calibrating optical tracking and depth camera coordinates based on a novel calibration model, wherein the transformation of the corner point coordinates to the depth camera coordinate system includes:

[0035] Using the stereo imaging principle of binocular vision, the two-dimensional pixel coordinates (u,v) are converted into spatial homogeneous coordinates (X,Y,Z,W) under the depth camera, expressed as formula (3):

[0036]

[0037] Among them, f x f is the focal length expressed in pixels along the horizontal direction. y M1 is the focal length expressed in pixels in the vertical direction. The parameters that determine M1 depend entirely on the camera's internal parameters and are called camera internal parameters. M2 depends on the relative position of the camera with respect to the world coordinate system and is called camera external parameters. The matrix M formed by M1 and M2 is the projection matrix.

[0038] According to the optical tracking and depth camera coordinate calibration method based on a novel calibration model provided by the present invention, the three-dimensional coordinates of the point calculated by the reprojection matrix Q are defined as formula (4):

[0039]

[0040] Where b is the baseline distance, (c x ,c y ) represents the center coordinates of the image coordinate system, and f is the focal length.

[0041] According to the present invention, an optical tracking and depth camera coordinate calibration method based on a novel calibration model projects pixel points (u,v) into three-dimensional space using the disparity value d of the depth map and the reprojection matrix Q. In the middle, it is represented by formula (5):

[0042]

[0043] The projection matrix Q = T obtained through calculation L→D The extracted corner points p of the chessboard grid i ={x1+iΔx,y1+jΔy,Δz} i＝1～7,j＝1～5 Reprojected onto the 3D space in the depth camera coordinate system {D}, denoted as...

[0044] According to the present invention, a calibration method for optical tracking and depth camera coordinates based on a novel calibration model is provided to obtain the corner point p within a checkerboard pattern in an optical tracking system. i and spatial coordinates This set is denoted as

[0045] Obtain the corner point p within the same chessboard square i Spatial coordinates under depth camera This set is denoted as

[0046] The present invention provides a calibration method for optical tracking and depth camera coordinates based on a novel calibration model. This method uses K pairs of geometric points in three-dimensional space and the least squares method to solve for the spatial transformation matrix T from the depth camera coordinate system {D} to the optical tracking coordinate system {O}. D→O Represented as formula (6):

[0047] T D→O =(X T X) -1 X T Y (6).

[0048] Therefore, compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. This invention uses a checkerboard pattern to make it possible to automate the processing of corner points in visible light images. With the support of the camera manufacturer's SDK, the coordinates of the corner point in the depth camera coordinate system can be obtained indirectly, which simplifies the process of obtaining the corresponding point in the depth camera coordinate system, avoids the disadvantages of large data volume and slow speed in processing point cloud data, and does not require a specially designed depth camera calibration model.

[0050] 2. The present invention can provide position and attitude information in the optical tracking coordinate system by using positioning markers (such as L-shaped ones). By using position and attitude information, plus the position offset of each corner point in the checkerboard grid relative to the L-shaped optical marker point, the position of each corner point in the optical tracking coordinate system can be calculated.

[0051] Through the above two designs, the present invention solves the problem of obtaining the corresponding point of a point in the optical tracking coordinate system in the depth camera coordinate system, and also avoids manually moving the optical marker point.

[0052] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0053] Figure 1 This is a flowchart of an embodiment of an optical tracking and depth camera coordinate calibration method based on a novel calibration model according to the present invention.

[0054] Figure 2 This is a schematic diagram of an embodiment of an optical tracking and depth camera coordinate calibration method based on a novel calibration model according to the present invention.

[0055] Figure 3 This is a structural diagram of the novel calibration model in an embodiment of the optical tracking and depth camera coordinate calibration method based on a novel calibration model of the present invention.

[0056] Figure 4 This is a schematic diagram of the detection results of corner points within a checkerboard pattern in an image, as described in an embodiment of an optical tracking and depth camera coordinate calibration method based on a novel calibration model according to the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0058] Before starting the calibration method provided in this embodiment, it is first necessary to fix the relative positions of the optical tracking system and the depth camera. This relative position must remain unchanged during and after calibration. Additionally, it is necessary to measure the positional offsets from each corner point on the checkerboard grid to the optical marker points. The detailed calibration process is as follows:

[0059] See Figures 1 to 4 This invention provides a method for calibrating optical tracking and depth camera coordinates based on a novel calibration model. The method includes the following steps:

[0060] Step S1: Provide a checkerboard pattern with optical positioning marks, place the checkerboard pattern within the effective range of the optical tracking device, and face the depth camera with the checkerboard pattern facing the depth camera, so that the corner points in the visible light image captured by the depth camera are clearly identifiable.

[0061] Step S2: Since the optical positioning markers are three infrared optical sensitive balls fixed in an L-shape at the upper corner of one side of the chessboard, that is, three infrared optical sensitive balls fixed in an L-shape at the upper left corner of the chessboard, the spatial position and attitude of the optical markers can be obtained through the optical tracking system, and the position of each corner point in the optical tracking coordinate system can be calculated based on the position offset of each corner point on the chessboard from the optical markers.

[0062] Step S3: Acquire image information of the chessboard pattern using a depth camera, extract the coordinates of the corner points of the chessboard pattern, and transform these corner point coordinates to the depth camera coordinate system. This transformation can be performed using the SDK provided by the depth camera manufacturer.

[0063] Step S4: When more than three sets of corresponding corner point coordinates are obtained, calculate the spatial transformation matrix from the depth camera coordinate system to the optical tracking coordinate system to complete the calibration process.

[0064] In step S2 above, the spatial position and orientation of the optical marker point are obtained through an optical tracking device, including:

[0065] First, the spatial positions of the three infrared optically sensitive spheres relative to the optical tracking system's own coordinate system {O} are obtained using an optical tracking system, and their distribution is represented as: p1=[x1,y1,z1] T p2 = [x2, y2, z2] T p3 = [x3, y3, z3] T ;

[0066] Based on the positions of points p1, p2, and p3, a local coordinate system {L} is constructed for the L-shaped calibration frame. The X-axis is X = (p2-p1) / ||p2-p1||, and the Y-axis is Y = (p3-p1) / ||p3-p1||. The Z-axis can then be expressed as the cross product of the X and Y axes, Z = X × Y. Therefore, the spatial transformation matrix from {L} to {O} is: T L→O =[X,Y,Z].

[0067] Then, based on the relative positions p of the L-shaped marker frame and the checkerboard marker frame. i ={x1+iΔx,y1+jΔy,Δz} i＝1～7,j＝1～5 The coordinates of the K interior corner points on the chessboard in the optical tracking system coordinate system {O} are obtained as follows: In this embodiment, K is 35, but it is not limited to 35.

[0068] Therefore, this embodiment designs a novel calibration model, such as Figure 2 and 3 As shown, this calibration model combines the checkerboard pattern used in traditional camera calibration with an "L"-shaped optical positioning marker. It can obtain the coordinates of multiple sets of optical tracking coordinates without manually moving the positioning marker with optical marker points. It can quickly and accurately extract the corresponding points of optical tracking coordinates in the point cloud data captured by the depth camera, avoiding complex point cloud processing.

[0069] Of course, this embodiment uses a checkerboard pattern as the visible light calibration template, but it is not limited to this type of calibration template. Any pattern that allows for easy extraction of the coordinates of a specific point from a visible light image can be used as a calibration template. The optical positioning mark is not limited to the "L" shape; it can also be "+" or "X" shape, as long as the offset between the mark and the target point on the visible light calibration template can be calculated.

[0070] In step S3 above, the Harris algorithm is used to extract the corner points within the chessboard. The basic idea of ​​this corner detection algorithm is to use a fixed window (a neighborhood window of a certain pixel) to slide in any direction on the acquired image and compare the degree of change in pixel grayscale in the window before and after the sliding.

[0071] If there is a large change in grayscale when sliding in any direction, then it can be assumed that there are corner points in the window;

[0072] In mathematics, the grayscale transformation of image I at coordinates (x, y) is expressed as: R = ∑(I(x+u,y+v)-I(x,y)) 2 By performing a Taylor expansion on the image grayscale function I(x,y), we obtain formula (1):

[0073]

[0074] Transform formula (1) into formula (2):

[0075]

[0076] In formula (2), the intermediate matrix is ​​the covariance matrix, which represents the rate of change of intensity in all directions. The covariance matrix is ​​a real symmetric matrix, and its two eigenvalues ​​λ1 and λ2 characterize the local structural features of the image as follows: If the two eigenvalues ​​λ1 and λ2 are small, the point is in a relatively uniform region. If one eigenvalue λ1 is large and the other eigenvalue λ2 is small, the point must be at the edge of the image. If both eigenvalues ​​λ1 and λ2 are large, the point is at a corner position. Therefore, in this embodiment, the coordinates (u,v) of the inner corner point on the chessboard grid are determined by calculating the eigenvalues ​​of the local covariance matrix of the image. These coordinates (u,v) are two-dimensional pixel coordinates.

[0077] In this embodiment, the coordinates of the corner point are transformed to the depth camera coordinate system, including:

[0078] Using the stereo imaging principle of binocular vision, this embodiment can further convert the two-dimensional pixel coordinates (u,v) into spatial homogeneous coordinates (X,Y,Z,W) under the depth camera, expressed as formula (3):

[0079]

[0080] Among them, f x f is the focal length expressed in pixels along the horizontal direction. y M1 is the focal length expressed in pixels in the vertical direction. The parameters that determine M1 depend entirely on the camera's internal parameters and are called camera internal parameters. M2 depends on the relative position of the camera with respect to the world coordinate system and is called camera external parameters. The matrix M formed by M1 and M2 is the projection matrix, which can be obtained by calibration using the classic Zhang Zhengyou calibration method.

[0081] In binocular vision, target objects at different positions have different landing points on the image plane, and the values ​​of d = LR for image points at different positions are also different. d is inversely proportional to the z-axis coordinate of the image. Assuming the baseline distance between the left and right cameras is represented as b, that is, Z = f × b / (LR) = f × b / d. Therefore, the spatial coordinates of the image points can be calculated through the coordinate relationship between corresponding points. Therefore, the three-dimensional coordinates of the points calculated by the reprojection matrix Q are defined as formula (4):

[0082]

[0083] Where b is the baseline distance, (c x ,c y Let (u, v) be the center coordinates of the image coordinate system, and f be the focal length. The pixel (u, v) is projected into three-dimensional space using the disparity value d of the depth map and the reprojection matrix Q. In the middle, it is represented by formula (5):

[0084]

[0085] Thus, the projection matrix Q = T obtained through calculation is obtained. L→D The extracted corner points p of the chessboard grid i ={x1+iΔx,y1+jΔy,Δz} i＝1～7,j＝1～5 Reprojected onto the 3D space in the depth camera coordinate system {D}, denoted as...

[0086] In step S4 above, after step S3 above, the corner point p inside the checkerboard under the optical tracking system can be obtained. i and spatial coordinates This set is denoted as

[0087] At the same time, by following the above steps, the corner point p within the same chessboard square can be obtained. i Spatial coordinates under depth camera This set is denoted as

[0088] In this embodiment, based on 35 pairs of geometric points in three-dimensional space, the spatial transformation matrix T from the depth camera coordinate system {D} to the optical tracking coordinate system {O} is solved using the least squares method. D→O Represented as formula (6):

[0089] T D→O =(X T X) -1 X T Y (6).

[0090] In this embodiment, the transformation matrix calculated by this method has a very small error compared to the manually selected corresponding points; the error is also very small when the obtained transformation matrix is ​​used to verify on other known points.

[0091] In summary, this embodiment uses a checkerboard pattern to enable automated processing of corner points in visible light images. With the support of the camera manufacturer's SDK, the coordinates of these corner points in the depth camera coordinate system can be obtained indirectly, simplifying the process of obtaining the corresponding points in the depth camera coordinate system. This avoids the drawbacks of large data volume and slow processing speed in point cloud data processing and eliminates the need for a specially designed depth camera calibration model. This embodiment provides position and attitude information in the optical tracking coordinate system through positioning markers (such as L-shaped markers). By combining the position and attitude information with the position offset of each corner point in the checkerboard pattern relative to the L-shaped optical marker, the position of each corner point in the optical tracking coordinate system can be calculated.

[0092] Therefore, the newly designed calibration method in this embodiment utilizes both the checkerboard grid and optical positioning markers. By extracting the position information of each corner point in the checkerboard grid in the depth camera coordinate system and the position and orientation of the optical positioning markers in the optical tracking system coordinate system, the transformation relationship between the two coordinate systems can be automatically, accurately, and quickly calculated. This solves the problem of obtaining the corresponding points of points in the optical tracking coordinate system in the depth camera coordinate system, and avoids the problem of traditional calibration methods requiring multiple manual movements of the optical positioning markers and the difficulty of the depth camera in automatically extracting the position of the marker points.

[0093] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. A method for calibrating optical tracking and depth camera coordinates based on a novel calibration model, characterized in that, The method includes the following steps: A checkerboard pattern with optical positioning marks is provided, the checkerboard pattern is placed within the effective range of an optical tracking device, and the front of the checkerboard pattern faces the depth camera; The spatial position and orientation of the optical markers are obtained by an optical tracking device, and the position of each corner point in the optical tracking coordinate system is calculated based on the position offset of each corner point on the chessboard from the optical marker. The image information of the chessboard is obtained by a depth camera, the coordinates of the corner points of the chessboard are extracted, and the coordinates of the corner points are transformed into the depth camera coordinate system. When more than three sets of corresponding corner point coordinates are obtained, the spatial transformation matrix from the depth camera coordinate system to the optical tracking coordinate system is calculated to complete the calibration process. The optical positioning markers are three infrared optically sensitive small balls fixed to each other in an L-shape at the upper corner of one side of the chessboard grid; The process of obtaining the spatial position and orientation of the optical markers through an optical tracking device includes: The coordinates of the three infrared optically sensitive spheres relative to the optical tracking system are obtained using an optical tracking device. The various spatial locations are represented as follows: , , ; according to p 1. p 2. p 3. At these three points, construct the local coordinate system of the L-shaped calibration frame. Where the X-axis is The Y-axis is Then the Z-axis can be represented as the cross product of the X-axis and Y-axis. Thus, the coordinate system is derived from arrive The spatial transformation coordinate system transformation matrix is: ; Based on the relative positions of the L-shaped calibration frame and the checkerboard marking frame The coordinates of the K interior corner points on the chessboard grid in the optical tracking system were obtained. The coordinates below are: .

2. The method according to claim 1, characterized in that: The Harris algorithm is used to extract the corner points within the chessboard grid. A fixed window is used to slide in any direction on the acquired image, and the degree of change in pixel grayscale in the window is compared before and after the slide. If there is a significant change in grayscale when sliding in any direction, then the window is considered to have a corner point. Set Image In coordinates Grayscale transformation is expressed as: By adjusting the image grayscale function Performing a Taylor expansion, we obtain formula (1): （1） Transform formula (1) into formula (2): （2） Among them, the middle matrix of formula (2) is the covariance matrix, which represents the rate of change of intensity in all directions; By calculating the eigenvalues ​​of the local covariance matrix of the image, the coordinate positions of the interior corner points of the chessboard grid on the image are determined. The coordinate position These are two-dimensional pixel coordinates.

3. The method according to claim 2, characterized in that, The process of transforming the corner point coordinates to the depth camera coordinate system includes: Using the stereoscopic imaging principle of binocular vision, the coordinates of two-dimensional pixels are... Convert to homogeneous spatial coordinates under depth camera , expressed as formula (3): （3） in, The focal length is expressed in pixels along the horizontal direction. To determine the focal length, expressed in pixels along the vertical direction. The parameters depend entirely on the camera's internal parameters, and are called camera internal parameters. Dependent on the relative position of the camera to the world coordinate system, these are called camera extrinsic parameters. and The matrix formed together This is the projection matrix.

4. The method according to claim 3, characterized in that: Define the reprojection matrix The three-dimensional coordinates of the point are calculated using formula (4): （4） in, Baseline distance, The center coordinates of the image coordinate system It is the focal length.

5. The method according to claim 4, characterized in that: Disparity values ​​from depth map and reprojection matrix pixels Projected into three-dimensional space In the middle, it is represented by formula (5): （5） The projection matrix obtained through calculation The extracted corner points of the chessboard grid Reprojected to depth camera coordinate system The following three-dimensional space is denoted as .

6. The method according to claim 5, characterized in that: Obtain the inner corner points of the checkerboard under optical tracking device and spatial coordinates From spatial coordinates The set formed is denoted as ; Obtain the corner points of the same chessboard square Spatial coordinates under depth camera From spatial coordinates The set formed is denoted as .

7. The method according to claim 6, characterized in that: Based on K pairs of geometric points in 3D space, the solution from the depth camera coordinate system is obtained using the least squares method. To the optical tracking coordinate system spatial transformation matrix Represented as formula (6): （6）。

Images