Arthroscopic image calibration tool and method

Abstract

The present invention provides an arthroscopic image calibration tool and method. The tool comprises: an endoscope, an endoscope calibration point, a fixing seat and a calibration plate base, the endoscope is fixed on the fixing seat, the calibration plate base is arranged below the fixing seat, and a calibration plate is installed on the inner side of the bottom, the endoscope calibration point is also fixed on the fixing seat, and the spatial position relationship between the endoscope and the endoscope is fixed. The arthroscopic image calibration tool and method provided by the present invention solve the problems of large amount of calculation, complex calibration process and poor calibration accuracy in the arthroscopic image calibration process.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an arthroscopic image calibration tool and method. Background Art

[0002] In image-based computer-assisted surgery, especially arthroscopic surgery, camera pose calibration is essential. However, developing an effective calibration method for medical arthroscopy is a challenging task. There are three reasons: first, the calibration result must be accurate, because many surgeries require at least 5 mm positioning accuracy; second, the strong radial distortion of the camera must be taken into account; third, the final calibration steps must be simple, the calibration speed must be fast, and the calibration accuracy and robustness must be high, so that non-professionals can also operate it. The existing arthroscopic calibration methods are shown below:

[0003] Prior art solution 1 (Barreto JP, Roquette J, Sturm PF, et al. Automatic Camera Calibration Applied to Medical Endoscopy [C] / / British Machine Vision Conference. DBLP, 2009.): For the identification of the contour lines and sharp corners of the endoscopic image, the endoscopic image is converted into a polar coordinate image, and the optimal center position and image are found so that the contour line of the polar coordinate image is closest to the vertical straight line, and the point farthest from the center of the circle in the polar coordinate image is the sharp corner.

[0004] Existing technical solution 2 (Zhang Chunyan, Zou Wei. Design, detection and sorting method of fisheye lens calibration plate [J]. Computer Engineering and Applications, 2015, 51(015): 188-192.): For internal reference calibration, a wide-width multi-feature circle calibration plate suitable for fisheye calibration is designed based on the characteristics of fisheye imaging. Based on this dot calibration plate, the characteristic circles are calibrated and calculated using ellipse detection and parameter fitting technology.

[0005] Existing technical solution 3 (Melo R, Barreto JP, Falcao GA new solution for camera calibration and real-time image distortion correction in medical endoscopy-initial technical evaluation. [J]. IEEE Trans Biomed Eng, 2012, 59(3): 634-644.): For internal reference calibration, it is assumed that the lens distortion conforms to the parabolic model and a single image is used for calibration.

[0006] Prior art solution 4 (Sturm P, P.Barreto.General Imaging Geometry for Central Catadioptric Cameras[C] / / European Conference on Computer Vision.Springer,Berlin,Heidelberg,2008.): For intrinsic calibration, it is assumed that the lens distortion conforms to the spherical model and a single image is used for calibration.

[0007] Prior art solution 5 (Yamaguchi T, Nakamoto M, Sato Y, Konishi K, Hashizume M, Sugano N, Yoshikawa H, Tamura S, Development of a camera model and calibration procedure for oblique-viewing endoscopes, Computer Aided Surgery, 01 Jan 2004, 9(5): 203-214): For extrinsic parameter calibration, a drive motor is installed on the lens to drive the lens to rotate while recording the position of the lens, that is, the position of the calibration plate in the lens, to fit the extrinsic parameters of the lens.

[0008] The disadvantage of the prior art solution 1 is that the amount of calculation is huge and parallel computing must be used, otherwise the real-time requirement cannot be met.

[0009] The disadvantage of the existing technical solution 2 is that it needs to take multiple pictures at different positions and angles for calibration, which is time-consuming and labor-intensive.

[0010] The disadvantage of existing technical solutions 3 and 4 is that the model accuracy is poor, resulting in poor arthroscopic calibration accuracy.

[0011] The disadvantage of the prior art solution 5 is that the calibration process is complicated, and the motor needs to be installed and calibrated in combination with movement, which is time-consuming and labor-intensive, and is difficult to promote in actual use. Summary of the invention

[0012] The technical problem to be solved by the present invention is to provide an arthroscopic image calibration tool and method, which solves the difficult problems of large amount of calculation, complex calibration process and poor calibration accuracy in the arthroscopic image calibration process.

[0013] In order to solve the above technical problems, the present invention provides an arthroscopic image calibration tooling, which includes: an endoscope, an endoscope calibration point, a fixed seat and a calibration plate base. The endoscope is fixed on the fixed seat, the calibration plate base is arranged below the fixed seat, and a calibration plate is installed on the inner side of the bottom. The endoscope calibration point is also fixed on the fixed seat, and the spatial position relationship between the endoscope and the endoscope is fixed.

[0014] In some embodiments, there is a cavity structure between the calibration point body and the fixing seat that is consistent with the shape of the endoscope.

[0015] In some embodiments, the calibration point body and the fixing seat are connected by bolts.

[0016] In some embodiments, when the bolts are tightened, the calibration point body and the fixing seat fully embrace the endoscope.

[0017] In some embodiments, the calibration points are divided into two layers, upper and lower layers.

[0018] In some embodiments, two calibration point bodies are disposed on the upper layer, and three calibration point bodies are disposed on the lower layer.

[0019] In some embodiments, a side surface of the bottom of the calibration plate base is hollowed out.

[0020] In some embodiments, the calibration plate is mounted on the remaining three side surfaces and the bottom surface of the bottom of the calibration plate base.

[0021] In some embodiments, the calibration plate is adhered to the inner side of the bottom of the calibration plate base.

[0022] In addition, the present invention also provides an arthroscopic image calibration method, which is applied to the image calibration process of the arthroscopic image calibration tooling described above, and the method includes: identifying the outer contour of the arthroscopic image by an approximation method; identifying the outer contour gap and calculating the lens deflection angle; identifying the X corner point in the calibration plate; segmenting the calibration plate image according to the identified X corner point; sorting the X corner points in the calibration plate image according to the image segmentation result; calculating the internal parameters of the endoscope and the external parameters of the calibration plate relative to the endoscope according to the sorting result; and calculating the coordinate relationship between the endoscope and the calibration point according to the external parameters of the calibration plate relative to the endoscope.

[0023] After adopting such a design, the present invention has at least the following advantages:

[0024] First, compared with the prior art that installs a driving motor on the lens to drive the lens to rotate while recording the lens's posture, that is, the position of the calibration plate in the lens, to fit the lens's external parameters, this paper uses a specially designed arthroscopic calibration tooling. During calibration, the posture relationship between the calibration plate and the arthroscopic Marker is known, so that the coordinate relationship between the arthroscopic lens and the Marker can be directly calculated, which simplifies the calibration process and makes it easy for people to get started; second, compared with the prior art that calibrates through multiple pictures at different positions and angles, the present invention realizes the possibility of single picture calibration through a specially designed calibration plate, greatly shortening the calibration time; third, compared with the prior art method of converting endoscopic images into polar coordinate images, finding the best center of circle position and image, and then identifying the endoscopic image contour and sharp corners, this paper can quickly locate the arc contour and sharp corners through the approximation method, greatly reducing the amount of calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0026] Figure 1 It is a structural schematic diagram of an arthroscopic image calibration tool provided by an embodiment of the present invention;

[0027] Figure 2 is a schematic structural diagram of endoscope calibration points provided by an embodiment of the present invention;

[0028] Figure 3 is a schematic flow chart of an arthroscopic image calibration method provided by an embodiment of the present invention;

[0029] Figure 4 is a schematic diagram of the rotation of an arthroscopic lens provided by an embodiment of the present invention;

[0030] Figure 5 It is a schematic diagram of the principle of fitting 24 edge points using an approximation method provided by an embodiment of the present invention;

[0031] Figure 6 This is a diagram showing the effect of the minimum distance least squares method for fitting the contour provided by an embodiment of the present invention;

[0032] Figure 7 is a schematic diagram of the principle of calculating the rotation angle of a lens provided by an embodiment of the present invention;

[0033] Figure 8 This is a picture of a calibration plate under an arthroscopic view provided by an embodiment of the present invention;

[0034] Fig. 9 is a segmentation effect diagram of segmenting a calibration plate provided by an embodiment of the present invention;

[0035] Fig.10 It is a rendering of the calibration plate after sorting provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0037] See also Figure 1 , an endoscope rapid calibration device, comprising an endoscope calibration point part and a calibration plate part, the endoscope calibration point part comprising a calibration point body (2) and a fixing seat (3), the calibration plate part comprising a calibration plate base (1) and calibration plates (a, b, c, d);

[0038] See also Figure 2 The function of the endoscope calibration point is to facilitate the operator to obtain the position of the endoscope in real time during the operation through the calibration point on it. The endoscope calibration point part is fixed on the endoscope (4). There is a cavity structure consistent with the appearance of the endoscope (4) between the calibration point body (2) and the fixing seat (3). The calibration point body (2) and the fixing seat (3) are connected by bolts. When the bolts are tightened, the calibration point body (2) and the fixing seat (3) completely hold the endoscope (4) so ​​that the relative position between them is fixed.

[0039] The calibration plate is used to assist doctors to quickly determine the internal parameters of the endoscope and the relative position of the endoscope and the calibration point before surgery. The relative position of the endoscope and the calibration point is also called the external reference. The four positions a, b, c, and d of the calibration plate base (1) have four widths, thicknesses, and Figure 1 The grooves of the four calibration plates a, b, c, d shown are consistent, and the calibration plates (a, b, c, d) are fixed to the calibration plate base (1) by gluing.

[0040] The endoscope calibration point part and the calibration plate part are not fixed, but mechanically limited, that is, the connection method at their joints is unique and tight, ensuring that the relative relationship between the endoscope Marker part and the calibration plate part is consistent and conforms to the original design.

[0041] During the calibration process, you only need to take a picture to obtain the internal parameters of the endoscope and the relationship between the endoscope calibration points and the endoscope; in the subsequent surgical process, the position of the endoscope at that moment can be accurately obtained by tracking the calibration points on the endoscope.

[0042] See also Figure 3 The embodiment of the present invention also provides a method for rapid calibration of arthroscopy: that is, rapid calibration of arthroscopy images is achieved through specially designed arthroscopy calibration tooling and calibration plate, and the specific steps are as follows:

[0043] Step 1: Identify the outer contour of the arthroscopic image.

[0044] Since there is a lot of salt and pepper noise in the original arthroscopic image, it must be median filtered before use to reduce the noise scale of the image.

[0045] Intraoperative arthroscopic images, the effective image area is a circular area with sharp corners, and the pixel value outside the area is zero (after white balance). The arthroscopy is different from a general camera. The lens of the arthroscopy will rotate during use to expand the field of view. Figure 4 As shown, the image contour and sharp corners need to be positioned in real time to accurately obtain the relationship between the arthroscope and the camera, so that the calibration parameters will be effective. Through measurement, it is found that the contour position is different for different arthroscopes at different angles, so the contour of each image needs to be calculated quickly.

[0046] This paper uses the approximation method to quickly locate the arc contour, such as Figure 5 As shown in the figure, 24 straight lines with different slopes are used to approximate from the outside to the inside, the maximum gradient values ​​on the left and right sides of the image are taken as target points, the approximation range is set, and finally 24 points with the largest gradient values ​​are obtained.

[0047] After obtaining 24 edge points, we need to use these 24 edge points to fit the image contour. The usual method is to use the least squares method for fitting. However, due to the influence of light, noise and sharp corners, a small number of points will not be on the circle contour. At this time, directly using the least squares method for fitting will result in large errors. Therefore, this paper adopts a "minimum distance least squares method", the core idea of ​​which is to minimize the sum of the distances from the data points to the center of the circle, that is:

[0048]

[0049] x that makes f reach the minimum value c ,y c and R are the best fitting parameters. The fitting effect of the minimum distance least squares method is as follows Figure 6 shown.

[0050] Step 2: Identify the outer contour gap and calculate the lens deflection angle.

[0051] After identifying the contour of the internal image, it is necessary to find the sharp corner point from these 24 points. This paper calculates from each point to the center of the fitting circle, takes the point farthest from the center of the circle, and then combines threshold filtering and other means to accurately find the coordinates of the sharp corner vertex.

[0052] After calculating the vertex coordinates, combined with the coordinates of the center of the circle fitted in the first step, the rotation angle of the arthroscope is calculated. Similar to a clock, the zero point direction of the clock is zero degrees, and the clockwise angle is the rotation angle of the lens.

[0053] Step 3: Identify X corner points.

[0054] This paper uses the calibration plate style as follows Figure 8 As shown, the picture contains 4 calibration plates at the same time, so only one picture is needed to complete the calibration of the arthroscope.

[0055] In the identification of X corner points, morphological methods are first used for screening. To find dozens of candidate points from hundreds of thousands of image pixels, an efficient filtering operator is required. This paper adopts the corner point response value operator proposed by Bennett et al. The sum response value (SR), differential response (DR), mean response value (MR) and final response value R of the current pixel 20 are defined as follows:

[0056]

[0057]

[0058]

[0059] The weighted summation gives the total response value R of the point:

[0060] R=SR-DR-16MR

[0061] The higher the R, the closer it is to the X corner point. Then, by using threshold filtering, non-maximum suppression and other methods on the pixel points, the X corner point can be accurately screened out.

[0062] Step 4: Segment the calibration plate.

[0063] like Fig. 9 As shown in the figure, the X corner point in the figure contains a total of 4 points of the calibration plate, so the X corner point needs to be segmented.

[0064] When designing the calibration plate, for the convenience of segmentation, three colors of red, blue and green are used to divide the boundaries of the calibration plate. The purpose of segmentation is to accurately find the six points in the figure, so as to determine the five dividing lines, and then determine which calibration plate each X corner point belongs to through the relationship between the point and the line. The specific segmentation steps are as follows:

[0065] Separate the RGB channels of the image and perform threshold filtering on the pixel values ​​of each channel.

[0066] Specify a small radius r, find points that are within the circle with radius r and contain red, blue, and green colors, and quickly locate origin 1 and origin 2 by slope iteration. At this time, origin 1 and origin 2 cannot be directly associated with the calibration plate;

[0067] Specify a large radius R, and quickly determine the red and blue points corresponding to origin 1 and origin 2 with origin 1 and origin 2 as the center. Then, combine the positional relationship between the vectors to determine the relationship between the ①, ②, ③, ④ calibration plates and these six points.

[0068] Through the positional relationship between the X corner point and the five straight lines, determine which calibration plate each X corner point belongs to.

[0069] Step 5: Sort the corner points of the calibration plate.

[0070] Based on the principle of camera calibration, the points in the camera coordinate system need to correspond one-to-one with the points in the world coordinate system to achieve calibration. Therefore, it is necessary to find the origin of each calibration plate and then sort them one by one, such as Fig.10 shown.

[0071] The point sorting process for the four calibration plates is similar, so only the sorting process for the first calibration plate is introduced. The steps are as follows:

[0072] Traverse all points in partition 1 (hereinafter referred to as all points) and find the two points closest to origin 1 (the reason for finding two points is to avoid the nearest point being a noise point).

[0073] Taking the first point as the reference, find the four reference points 0, 1, 8, and 9 through angle, direction, and distance constraints. If the search fails, use the second point to search again. If it fails again, abandon the calibration plate.

[0074] From the found base point, use angle and distance constraints to locate the remaining points in turn, for example, use 0 and 1 to locate point 2, use 0 and 8 to locate point 16, use 0 and 9 to locate point 18, and so on, until all points are searched.

[0075] Step 6: Arthroscopic internal reference calibration.

[0076] The sorted calibration plate coordinates and their corresponding world coordinate system coordinates are calibrated using Zhang Zhengyou calibration method to obtain the camera's intrinsic parameters (intrinsic parameter matrix, distortion matrix) and the calibration plate's extrinsic parameters relative to the camera, such as Fig.10 shown.

[0077] Step 7: Calibration of arthroscopic external parameters.

[0078] like Fig.10 As shown in the figure, a special arthroscopic calibration fixture is designed in this paper. During calibration, the position relationship T1 from the calibration plate to the arthroscopic marker is known, so the coordinate relationship from the arthroscopic lens to the marker can be directly calculated, that is:

[0079] T0=T2*T1

[0080] In some embodiments, the internal reference calibration plate may be a circular calibration plate, a charucoboard, a two-dimensional code calibration plate, a spatial three-dimensional calibration plate with geometric constraints, etc.

[0081] In some embodiments, the spatial structure for installing the internal reference calibration plate may be a cube, a tetrahedron, an octahedron, or other three-dimensional structure in which multiple faces can be seen simultaneously, in addition to the currently used rectangular parallelepiped.

[0082] In some implementations, the number of internal reference calibration plates, in addition to the current 4 plates, may be 1, 2, 3 or even more plates.

[0083] In some embodiments, the shape of the camera marker can be replaced by an infrared reflective ball, a circular marking point, a QR code mark, etc.

[0084] In some embodiments, the endoscope distortion model may adopt a spherical model, a parabolic model, a segmented model, etc.

[0085] In some embodiments, when the linear approximation method is used to find the endoscope contour circle, different angle intervals can be set to set the approximation straight lines, so the number of approximation straight lines is not unique and can be any number in theory.

[0086] In some implementations, the processing technology and materials of the calibration tool are not unique, and mechanical processing can be used, and the material can be nylon, metal, etc.

[0087] In some embodiments, the segmentation between internal reference calibration plates may not adopt the color distinction method in this article, but may adopt different line shapes or different codes for segmentation.

[0088] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Those skilled in the art may make some simple modifications, equivalent changes or modifications using the technical contents disclosed above, which all fall within the protection scope of the present invention.

Claims

1. An arthroscopic image calibration tool, characterized in that: include: An endoscope, an endoscope calibration point, a fixing seat and a calibration plate base, wherein the endoscope is fixed on the fixing seat, the calibration plate base is arranged below the fixing seat, and a calibration plate is installed on the inner side of the bottom, and the endoscope calibration point is also fixed on the fixing seat, and the spatial position relationship between the endoscope and the calibration point is fixed; The calibration points are divided into two layers: upper and lower; There is a cavity structure between the calibration point body and the fixing seat that is consistent with the shape of the endoscope; One side surface of the bottom of the calibration plate base is hollowed out, and the calibration plate is installed on the remaining three side surfaces and the bottom surface of the bottom of the calibration plate base.

2. The arthroscopic image calibration tool according to claim 1, characterized in that: The calibration point body and the fixed seat are connected by bolts.

3. The arthroscopic image calibration tool according to claim 2, characterized in that: When the bolts are tightened, the calibration point body and the fixing base fully hold the endoscope.

4. The arthroscopic image calibration tool according to claim 1, characterized in that: The upper layer is provided with two calibration point bodies, and the lower layer is provided with three calibration point bodies.

5. The arthroscopic image calibration tool according to claim 1, characterized in that: The calibration plate is adhered to the inner side of the bottom of the calibration plate base.

6. An arthroscopic image calibration method, characterized in that: The image calibration process applied to the arthroscopic image calibration tool according to any one of claims 1 to 5 comprises: The outer contour of the arthroscopic image is identified by the approximation method; Identify the outer contour gap and calculate the lens deflection angle; Identify the X corner points in the calibration plate, including: Morphology is used to select candidate points from the pixels of the calibration plate image using the corner response value operator, and the X corner points are selected from the candidate points using the threshold filtering and non-maximum suppression method. According to the identified X corner points, the calibration plate image is segmented, including: To facilitate segmentation, the calibration plate uses red, blue, and green to divide the boundary of the calibration plate. The slope iteration method is used to locate the origin and the red and blue points corresponding to the origin. The relationship between the point and the line is used to determine which calibration plate each X corner point belongs to. According to the image segmentation results, sort the X corner points in the calibration plate image; According to the sorting results, the internal parameters of the endoscope and the external parameters of the calibration plate relative to the endoscope are calculated, including: The sorted calibration plate coordinates and their corresponding world coordinate system coordinates are calibrated using Zhang Zhengyou calibration method to obtain the camera's intrinsic parameter matrix and distortion matrix as well as the calibration plate's extrinsic parameters relative to the camera; According to the external parameters of the calibration plate relative to the endoscope, the coordinate relationship between the endoscope and the calibration point is calculated, including: The coordinate relationship between the endoscope and the calibration point is calculated using the following formula: T0=T2*T1 Among them, T0 is the pose relationship between the endoscope and the calibration point, T1 is the pose relationship between the calibration plate and the calibration point, and T2 is the pose relationship between the calibration plate and the endoscope, that is, the external parameters of the calibration plate relative to the endoscope.

Images