Endoscope calibration method, system and tool

Abstract

The invention relates to the technical field of image analysis and processing, and provides an endoscope calibration method, system and tool, and the method comprises the steps: obtaining an object distance initial value of a to-be-calibrated endoscope, determining a focal length initial value of the to-be-calibrated endoscope according to the object distance initial value, and image coordinates and world coordinates corresponding to a position index, and obtaining an internal reference initial value of the to-be-calibrated endoscope; according to the position index of the first marker and the position index of the second marker, the corresponding relation between image coordinates and world coordinates of the centers of all the markers is determined, and an internal reference model and an external reference model of the endoscope to be calibrated are established; through a single calibration image on which a first marker and a second marker are distributed, marker sampling on the to-be-calibrated image is closer to all areas of the image, and meanwhile, based on a prior real object distance Z value, an estimated value of a focal length initial value is more accurate; therefore, the optimal endoscope internal parameters and distortion parameters estimated through the minimum likelihood function are closer to real values.

Description

Technical Field

[0001] The present application relates to the technical field of image analysis and processing, and in particular to an endoscope calibration system, method, device and tooling structure. Background Art

[0002] At present, the virtual endoscope navigation system integrates the preoperative multimodal images with the intraoperative endoscopic 2D real images. Its main surgical process is to plan the cutting area in the preoperative 3D image, and establish a mapping relationship between the preoperative 3D image and the intraoperative endoscopic 2D image through the endoscope calibration technology, so as to achieve layered rendering of the preoperative planned image area and the intraoperative endoscopic image, thereby providing planning guidance for doctors to avoid important tissues and organs and reduce surgical risks.

[0003] However, the traditional endoscope calibration process requires capturing calibration images from multiple angles and performing multiple calibrations, which results in a complex calibration process and low calibration efficiency. Summary of the invention

[0004] The embodiments of the present application provide an endoscope calibration method, system and tooling to reduce the complexity of the endoscope calibration process and improve the endoscope calibration efficiency.

[0005] An endoscope calibration method provided in an embodiment of the present application includes:

[0006] Acquiring a calibration image, wherein the calibration image includes at least three first markers and a plurality of second markers arranged in an array;

[0007] The image coordinates of the centers of the first marker and the second marker are obtained by an image segmentation algorithm, a calibration coordinate system is established according to the topological relationship between the first marker and the second marker, and a corresponding relationship between the image coordinates and the position index of the centers of the first marker and the second marker on the calibration coordinate system is determined;

[0008] Establishing a world coordinate system, and determining a corresponding relationship between world coordinates and position indexes of the centers of the first marker and the second marker on the world coordinate system;

[0009] Obtain an initial value of the object distance of the endoscope to be calibrated, and determine an initial value of the focal length of the endoscope to be calibrated and an initial value of the projection coordinates of the optical center of the endoscope in the image coordinate system according to the initial value of the object distance, the image coordinates and the world coordinates corresponding to the position index, so as to obtain an initial value of the internal parameter of the endoscope to be calibrated;

[0010] According to the position index of the first marker and the second marker, the corresponding relationship between the image coordinates and the world coordinates of the center of each marker is determined, and the internal reference and external reference models of the endoscope to be calibrated are established;

[0011] The LM algorithm is used to minimize the likelihood function model to estimate the internal and external parameters of the endoscope to be calibrated.

[0012] In a possible implementation, obtaining the image coordinates of the centers of the first marker and the second marker by using an image segmentation algorithm includes:

[0013] Converting the calibration image into a grayscale image and binarizing it according to a predetermined threshold to obtain a first segmented image;

[0014] Traversing all pixel points in the grayscale image, calculating an adaptive threshold corresponding to the pixel point according to the pixel value of the neighboring point of the pixel point on the first segmented image, and segmenting the grayscale image by the adaptive threshold value to obtain a second segmented image;

[0015] The outer contours of the first marker and the second marker on the second segmented image are fitted to obtain the image coordinates of the centers of the markers.

[0016] In a possible implementation, binarizing it according to a predetermined threshold to obtain a first segmented image includes:

[0017] When the pixel value of a pixel in the grayscale image is greater than or equal to a preset threshold, the pixel value of the pixel is marked as 255; when the pixel value of a pixel in the grayscale image is less than the preset threshold, the pixel value of the pixel is marked as 0.

[0018] In a possible implementation, the calculating and obtaining the adaptive threshold corresponding to the pixel point includes:

[0019] A threshold set corresponding to the pixel point is established. If the pixel value of at least one of the neighborhood points of the pixel point in the first segmented image is equal to a preset value, the coordinates and grayscale values ​​of the pixel point and its neighborhood points in the grayscale image are put into the threshold set. The adaptive threshold corresponding to the pixel point is obtained by calculating the average grayscale values ​​of all pixels in the threshold set corresponding to the pixel point.

[0020] In a possible implementation, segmenting the grayscale image by using an adaptive threshold to obtain a second segmented image includes:

[0021] For a certain pixel in the grayscale image, if the pixel value in the grayscale image is greater than or equal to the adaptive threshold, the pixel value of the pixel is marked as 255; if the pixel value in the grayscale image is less than the adaptive threshold, the pixel value of the pixel is marked as 0.

[0022] In a possible implementation, establishing a calibration coordinate system according to the topological relationship between the first marker and the second marker, and determining a correspondence between image coordinates and position indexes of centers of the first marker and the second marker on the calibration coordinate system includes:

[0023] According to the topological structure between any two first markers, a calibration coordinate system is established to determine the position index of the first marker;

[0024] Determine the corresponding relationship between the image coordinates of the center of the first marker and its position index according to the distance between any two first markers;

[0025] Determine the correspondence between the image coordinates and the position index of the center of the second marker in the neighborhood of the first marker according to the relative direction and distance between the first marker and the second marker in the neighborhood of the first marker;

[0026] The corresponding relationship between the image coordinates and the position index of all the second marker centers is determined step by step through the neighborhood point confirmation method.

[0027] In a possible implementation, determining the correspondence between the image coordinates and the position index of the center of the second marker in the neighborhood of the first marker according to the relative direction and distance between the first marker and the second marker in the neighborhood of the first marker includes:

[0028] Setting a first condition: calculating a reference direction vector of the first marker in the calibration coordinate system according to the image coordinates of the center of the first marker;

[0029] Setting a second condition: calculating a neighborhood approximate distance between the first marker and the closest second marker;

[0030] Calculating the direction vectors of all second markers relative to the first marker and the straight-line distances between the second markers and the first marker;

[0031] The direction vector of each second marker is compared with the reference direction vector, and the straight-line distance of each second marker is compared with the neighborhood approximate distance, so as to find the second marker in the neighborhood of the first marker and determine the correspondence between the image coordinates of its center and the position index.

[0032] In a possible implementation, the obtaining of the initial value of the object distance of the endoscope to be calibrated, determining the initial value of the focal length of the endoscope to be calibrated and the initial value of the projection coordinates of the optical center of the endoscope in the image coordinate system according to the initial value of the object distance and the image coordinates and world coordinates corresponding to the marker position index, and obtaining the initial value of the internal parameter of the endoscope to be calibrated includes:

[0033] The vertical distance from the endoscope to be calibrated to the calibration plate estimated by using prior knowledge is set as the initial value of the object distance;

[0034] Determine the initial focal length value of the endoscope to be calibrated according to the initial object distance value, the image coordinates and the world coordinates corresponding to the marker position index;

[0035] Determine the initial value of the projection coordinate of the optical center of the endoscope to be calibrated in the image coordinate system according to the pixel width and pixel length of the calibration image;

[0036] According to the obtained initial value of the focal length and the initial value of the projection coordinates of the optical center in the image coordinate system, the initial internal reference model of the endoscope to be calibrated can be obtained.

[0037] In a possible implementation, the initial internal reference model of the endoscope to be calibrated can be obtained according to the obtained initial value of the focal length and the initial value of the projection coordinate of the optical center in the image coordinate system, including:

[0038]

[0039] Among them, f x is the initial focal length in the x direction, f y is the initial value of the focal length in the y direction; (u0, v0) is the initial value of the projection coordinates of the optical center of the endoscope to be calibrated in the image coordinate system.

[0040] In a possible implementation, determining the correspondence between the image coordinates and the world coordinates of the center of each marker according to the position index of the first marker and the second marker, and establishing the internal and external reference models of the endoscope to be calibrated include:

[0041]

[0042] Among them, α is the scale coefficient, x, y are the set of image coordinates of all centers, X, Y, Z are the set of world coordinates of all centers, and the two sets are arranged one by one according to the position index of the center; A is the internal parameter of the endoscope to be calibrated, R represents the 4 times 4 rotation matrix of the camera coordinate system of the endoscope relative to the world coordinate system, T is the translation matrix of the camera coordinate system of the endoscope relative to the world coordinate system, R and T constitute the external parameter of the endoscope; D is the distortion coefficient, which is composed of radial coefficients k1, k2, k3 and tangential coefficients p1, p2.

[0043] In a possible implementation, the likelihood function model includes:

[0044]

[0045] Among them, m j is the image coordinate of the center of the marker in the image coordinate system, M j is the world coordinate of the center of the marker in the world coordinate system, M jis the projection point in the image coordinate system.

[0046] In a possible implementation, the initial value of the object distance may be further optimized to reduce disturbance interference, including:

[0047] Reducing disturbance on an initial focal length value corresponding to an initial object distance value to obtain a first focal length, and calculating a corresponding first object distance according to the first focal length;

[0048] Adding a disturbance to an initial focal length value corresponding to an initial object distance value to obtain a second focal length, and calculating a corresponding second object distance according to the second focal length;

[0049] According to the first focal length, the first object distance, the second focal length, the second object distance and the initial value of the focal length, the difference is used to calculate the optimized value of the initial value of the object distance, and the value is substituted into the likelihood function model.

[0050] An endoscope calibration system provided in an embodiment of the present application includes:

[0051] An acquisition module, used for acquiring a calibration image, wherein the calibration image includes at least three first markers and a plurality of second markers arranged in an array;

[0052] An image coordinate establishment module, used to obtain the image coordinates of the centers of the first marker and the second marker by an image segmentation algorithm, establish a calibration coordinate system according to the topological relationship between the first marker and the second marker, and determine the corresponding relationship between the image coordinates and the position index of the centers of the first marker and the second marker on the calibration coordinate system;

[0053] A world coordinate establishment module, used to establish a world coordinate system and determine the correspondence between the world coordinates and position indexes of the centers of the first marker and the second marker in the world coordinate system;

[0054] An assignment module is used to obtain an initial value of the object distance of the endoscope to be calibrated, and determine an initial value of the focal length of the endoscope to be calibrated and an initial value of the projection coordinates of the optical center of the endoscope in the image coordinate system according to the initial value of the object distance, the image coordinates and the world coordinates corresponding to the position index, so as to obtain an initial value of the internal parameter of the endoscope to be calibrated;

[0055] A modeling module, used to determine the correspondence between the image coordinates and the world coordinates of the center of each marker according to the position index of the first marker and the second marker, and to establish an internal reference and external reference model of the endoscope to be calibrated;

[0056] The estimation module is used to use the LM algorithm to minimize the likelihood function model to estimate the internal parameters, external parameters, distortion coefficients of the endoscope to be calibrated, and the projection of the optical center of the endoscope to be calibrated in the image coordinate system.

[0057] An endoscope calibration tool provided in an embodiment of the present application includes:

[0058] A calibration plate, on which a plurality of markers arranged in an array are disposed, wherein the markers include at least three first markers and a plurality of second markers, and the first markers and the second markers have different diameters;

[0059] The endoscope to be calibrated is used to collect the image of the calibration plate to obtain a calibration image.

[0060] In a possible implementation, the endoscope to be calibrated adjusts the position of the endoscope relative to the calibration plate through a rotatable endoscope tube fixing frame.

[0061] In a possible implementation, at least three non-collinear optical tracking marks are provided on the mark plate.

[0062] The embodiments of the present application provide an endoscope calibration method, system and tooling. By calibrating a single calibration image with a first marker and a second marker distributed thereon, the marker sampling on the image to be calibrated is made closer to all areas of the image. At the same time, based on the a priori true object distance Z value, the initial focal length estimate is made more accurate, thereby making the optimal endoscope internal parameters and distortion parameters estimated by the minimum likelihood function closer to the true values. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0064] Figure 1 A schematic diagram of the structure of a calibration tool provided in an embodiment of the present application from a first perspective;

[0065] Figure 2 A schematic diagram of the structure of the calibration tooling provided in the embodiment of the present application from a second perspective;

[0066] Figure 3 A schematic diagram of a calibration method for an endoscope to be calibrated provided in an embodiment of the present application;

[0067] Figure 4 A schematic diagram of a first segmented image after adaptive threshold segmentation provided in an embodiment of the present application;

[0068] Figure 5 A schematic diagram of the fitting effect of the calibration circle provided in an embodiment of the present application;

[0069] Figure 6 A schematic diagram of an implementation of the image coordinate system provided in an embodiment of the present application;

[0070] Figure 7A schematic diagram of the functional relationship between the object distance and the focal length of the endoscope to be calibrated provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to reduce the complexity of the endoscope calibration process and improve the endoscope calibration efficiency, the embodiments of the present application provide an endoscope calibration system, method, device and tooling structure.

[0072] The present application provides an endoscope calibration tool, such as Figure 1 As shown, it includes: a calibration plate 12 and an endoscope to be calibrated. The calibration plate 12 is provided with markers required in the endoscope calibration process. The markers can generally be set to any regular shape. The endoscope is used to collect images of the calibration plate 12 to obtain a calibration image.

[0073] The marker in the embodiment of the present application can be set as a calibration circle, and the center of the marker is the center of the circle.

[0074] The calibration plate 12 is provided with a plurality of arrayed marking circles, including at least three first calibration circles and a plurality of second calibration circles. The sizes of the first calibration circles and the second calibration circles need to be different. In this embodiment, the diameter of the first calibration circle is larger than the diameter of the second calibration circle.

[0075] In order to fix the endoscope and obtain the image of the calibration plate 12, the endoscope calibration system also includes a base plate 11 and a tube fixing frame 13. The calibration plate 12 is vertically mounted on the base plate 11, and the tube of the endoscope is fixed by the tube fixing frame 13. Specifically, the tube fixing frame 13 is provided with a hole 131 for inserting the tube of the endoscope. When the tube of the endoscope is inserted and aligned to the outer edge of the hole 131, the tube of the endoscope is approximately perpendicular to the calibration plate 12, and the object distance of the calibration image captured by the endoscope to be calibrated, that is, the vertical distance from the end of the endoscope to the calibration plate 12, can be determined as the initial value of the object distance.

[0076] In order to further adjust the posture of the fixed frame 13, fix the position of the endoscope tube, and determine the initial value of the object distance between the endoscope and the calibration plate, the fixed frame 13 is connected to the calibration plate through a rotating mechanism 14. The posture of the endoscope relative to the calibration plate 12 can be adjusted by the rotating mechanism 14, so as to adjust the field of view of the image collected by the endoscope to be calibrated, so that the collected calibration image includes as many circular features as possible, and the circular features are located in the middle area of ​​the calibration image. In this way, the calibration circle can fill the entire calibration image, and the feature data of the calibration circle on the calibration image is rich and diverse, so that the internal parameters and distortion parameters of the endoscope calculated by calibration using a single calibration image are more accurate. Of course, in actual application, the rotating structure 14 can be omitted, and the fixed frame 13 can be set to a fixed posture.

[0077] In addition, if Figure 2As shown, at least three non-collinear optical tracking marks 15 are also provided on the other side of the marking plate 12. Among them, the optical tracking marks 15 can obtain a first conversion relationship between the calibration plate 12 and the optical tracking system under the tracking of the optical tracking system; since the calibration plate 12 and the endoscope are rigidly connected, the second conversion relationship between the endoscope and the optical tracking system can be obtained according to the installation parameters of the two combined with the first conversion relationship. The conversion relationship between the endoscope and the calibration image can be obtained through this scheme (i.e., the internal reference of the endoscope), and the conversion relationship between the optical tracking system and the calibration image can be obtained in combination with the second conversion relationship. This conversion relationship is used to position the surgical instrument through the optical tracking system during the navigation process of the surgical robot, so as to determine the corresponding posture of the instrument in the medical image in real time.

[0078] like Figure 3 As shown, the present application also provides an endoscope calibration method, including steps S310 to S350.

[0079] S310, obtaining a calibration image obtained by photographing the marking plate 12 with an endoscope, and obtaining the image coordinates of the centers of the first calibration circle and the second calibration circle through an image segmentation algorithm;

[0080] According to the topological relationship between the first calibration circle and the second calibration circle, a calibration coordinate system is established, and the corresponding relationship between the image coordinates and the position index of the centers of the first calibration circle and the second calibration circle on the calibration coordinate system is determined.

[0081] Before establishing the calibration coordinate system, the calibration circle on the calibration image needs to be segmented to obtain the image coordinates of the center of the calibration circle. Specifically, steps S410 to S430 are used to segment the marked circle on the calibration image and obtain the image coordinates of the center.

[0082] S410, converting the calibration image into a grayscale image, and binarizing it according to a predetermined threshold value to obtain a first segmented image.

[0083] First, the calibration image is converted into a grayscale image, and a fixed predetermined threshold is used to convert the grayscale image into a binary image to preliminarily segment the calibration circle. In actual application, the predetermined threshold can be selected according to the segmentation effect. The principle of the predetermined threshold is to be able to segment most of the marked circles in advance, that is, the predetermined threshold segmentation extracts the marked circle with less distortion in the middle part of the calibration image.

[0084] Specifically, when the pixel value of a pixel in the grayscale image is greater than or equal to a preset threshold, the pixel value of the pixel is marked as 255, that is, the color of the pixel is marked as white. When the pixel value of a pixel in the grayscale image is less than the preset threshold, the pixel value of the pixel is marked as 0, that is, the color of the pixel is marked as black, so as to obtain a first segmented image that preliminarily segments the calibration circle.

[0085] S420, traverse all pixel points in the grayscale image, calculate the adaptive threshold corresponding to the pixel point according to the pixel value of the neighboring points of the pixel point on the first segmented image, and segment the grayscale image by the adaptive threshold to obtain a second segmented image.

[0086] First, a threshold set corresponding to the pixel is established. If the pixel value of at least one of the neighboring points of the pixel in the first segmented image is equal to the preset value, the coordinates and grayscale values ​​of the pixel and its neighboring points in the grayscale image are placed in the threshold set, and the adaptive threshold corresponding to the pixel is obtained by calculating the average grayscale value of all pixels in the threshold set corresponding to the pixel. Then, traversal is performed starting from the first pixel in the grayscale image to obtain the adaptive thresholds corresponding to all pixels.

[0087] It should be noted that before judging the pixel value of the neighboring points of each pixel point and the preset value, the threshold set needs to be cleared. That is, each pixel point has a corresponding threshold set, and the elements in the threshold set only include its own coordinates and grayscale values ​​in the grayscale image, and the coordinates and grayscale values ​​of its neighboring points that meet the requirements in the grayscale image.

[0088] Exemplarily, the preset value is set to 255.

[0089] Then, for a certain pixel in the grayscale image, if the pixel value in the grayscale image is greater than or equal to the adaptive threshold, the pixel value of the pixel is marked as 255, and if the pixel value of the pixel in the grayscale image is less than the adaptive threshold, the pixel value of the pixel is marked as 0. Figure 4 As shown, a second segmented image can be obtained by segmenting the grayscale image using an adaptive threshold.

[0090] Since the calibration image of the endoscope has a large image distortion, the edge area and the middle area of ​​the calibration image have a large difference in brightness. It is difficult to segment all the calibration circles on the calibration image by using a fixed threshold to segment the calibration image. The embodiment of the present application determines an adaptive threshold based on a predetermined threshold, and uses the adaptive threshold to perform binarization on the calibration image. In this way, the calibration circle in the calibration image can be more easily identified and separated, thereby significantly improving the computational efficiency of image processing, and enhancing the contrast of the image, making the details of the calibration circle in the image clearer, and then using the outer contour fitting of the calibration circle.

[0091] It should be noted that the present application is not limited to only using the above method to determine the adaptive threshold. In the actual application process, other algorithms can also be used to determine the adaptive threshold. For example, the Otsu algorithm (Otsu algorithm) is used to determine the adaptive threshold. The Otsu algorithm calculates the threshold through statistics and probability knowledge, and then binarizes the threshold. The core idea is to find a threshold so that the inter-class variance between the foreground and the background is the largest, thereby achieving the best image segmentation effect. For another example, by Matlab adaptive threshold segmentation algorithm image segmentation, the Matlab adaptive threshold segmentation algorithm can automatically adjust the threshold according to the local characteristics of the image, thereby achieving more accurate image segmentation, usually using the maximum inter-class variance method, which is suitable for scenes with uneven illumination or complex backgrounds.

[0092] S430, fitting the outer contours of the first calibration circle and the second calibration circle on the second segmented image to obtain image coordinates of the circle centers.

[0093] On this basis, the outer contour of the calibration circle is detected and fitted. For example, the findContours library function of opencv (Open Source Computer Vision Library) is used to detect the outer contour of each calibration circle, and then the calibration circle is fitted to obtain the image position of the center of the calibration circle, such as Figure 5 shown.

[0094] The calibration image itself has an image coordinate system. Generally, the upper left corner of the calibration image is used as the origin of the image coordinate system. The horizontal and vertical axes of the calibration image are the coordinate axes. The image coordinates of the center of the circle obtained in this step are based on the coordinate values ​​on the image coordinate system.

[0095] It should be noted that the calibration coordinate system and the image coordinate system are essentially established on the plane where the calibration image is located, and the difference is that the center positions of the two coordinate systems are different. After the center coordinates of the calibration circle are extracted, the center coordinates are in disorder, and the calibration image is distorted, and the correspondence between the center coordinates of each calibration circle and each calibration circle cannot be known. Therefore, the calibration coordinate system can be quickly established based on the center position of the first calibration circle, and the correspondence between the center position of the second calibration circle on the calibration coordinate system can be found according to the topological relationship between the first calibration circle and the second calibration circle. Specifically, the step of establishing the calibration coordinate system is configured to execute S710 to S730.

[0096] S710: Establish a calibration coordinate system according to a topological structure between any two first calibration circles, and determine a position index of the first calibration circle.

[0097] In this embodiment, the number of first calibration circles is three. Since the positions of the first calibration circles on the calibration plate are known in advance, such as Figure 6As shown, any one of the first calibration circles is selected as the origin of the calibration coordinate system, its position index is set to (0, 0), and the directions of the other two first calibration circles are set to the horizontal axis and the vertical axis.

[0098] Specifically, the center of the first calibration circle with position index (0, 0) is used as the coordinate origin. According to the relative position relationship between the other two first calibration circles and the origin, it can be known that the position indexes of the other two first marking tables are (0, 2) and (-4, 0). That is, the center direction of the first calibration circle with position index (-4, 0) is used as the horizontal axis, and the center direction of the first calibration circle with position index (0, 2) is used as the vertical axis.

[0099] S720: Determine the correspondence between the image coordinates of the centers of the first calibration circles and their position indexes according to the distance between any two first calibration circles.

[0100] It should be noted that the distance between any two calibration circles can be calculated based on the image coordinates of the circle centers in the image coordinate system obtained in step S430.

[0101] It is known that the distance between the first calibration circle with position index (-4, 0) and the first calibration circle with position index (0, 2) is the farthest, and the distance between the first calibration circle with position index (0, 0) and the first calibration circle with position index (0, 2) is the shortest. Thus, according to the distance between any two calibration circles calculated in advance, the image coordinates of the center of the first calibration circle with position index (0, 2) are first determined; then, according to the farthest and shortest distances, the image coordinates of the center of the first calibration circle with position index (-4, 0) and position index (0, 0) are respectively determined.

[0102] S730: Determine a correspondence between the image coordinates and the position index of the center of the second calibration circle in the neighborhood according to the relative direction and distance between the first calibration circle and the second calibration circle in the neighborhood.

[0103] Due to the distortion of the calibration image, when the image coordinates of the center of the circle are mapped to the calibration coordinate system, they cannot be arranged strictly according to the array. Therefore, the image coordinates corresponding to the topological position are found by the following method:

[0104] First, condition 1 is set, and the reference direction vector of each first calibration circle on the horizontal axis and the vertical axis is calculated according to the image coordinates of the center of the first calibration circle.

[0105] Then, condition 2 is set, and according to the distance between the two first calibration circles and the number of second calibration circles between the two first calibration circles is known, the neighborhood approximate distance between the first calibration circle and the closest second calibration circle is calculated;

[0106] According to the above two setting conditions, for all first calibration circles, the direction vectors of all second calibration circles relative to the first calibration circles are calculated, and the straight-line distances between all second calibration circles and the first calibration circles are calculated.

[0107] In this way, the similarity between the direction vector of each second calibration circle and the reference direction vector is compared, and the similarity between the straight-line distance and the neighborhood approximate distance is compared to find the neighborhood second calibration circle that best meets setting conditions 1 and setting conditions 2, that is, find the second calibration circle that is on the same horizontal and vertical axes as the first calibration circle and is closest to it.

[0108] There are at most four neighborhood points for a first calibration circle. In this way, the position index corresponding to the image coordinates of the center of the second calibration circle around the first calibration circle can be determined.

[0109] S740, based on the image coordinates of the center of the second calibration circle of the position index determined in step S730, gradually determine the correspondence between the image coordinates of the center of all second calibration circles and the position index through a neighborhood point confirmation method.

[0110] For the four second calibration circles determined in step S730, the same method as step S720 is used to obtain the center image coordinates of the four second calibration circles in the neighborhood of the second calibration circle according to the direction vector of each second calibration circle neighborhood point and the neighborhood approximate distance.

[0111] The corresponding relationship between the image coordinates and the position index of the center of all second calibration circles is determined step by step based on the neighborhood relationship.

[0112] It should be noted that in the above process of determining the position attributes of the first calibration circle and the second calibration circle, the topological sorting method of the circle center point set based on regional update growth can effectively deal with the situation where the calibration image has large distortion, thereby simplifying the process of determining the position attributes of the second calibration circle.

[0113] S320, establishing a world coordinate system of the calibration plate 12, and determining a correspondence between world coordinates and position indexes of the centers of the first calibration circle and the second calibration circle in the world coordinate system.

[0114] According to the actual size of the calibration plate 12, a world coordinate system is established. Specifically, it is known that the centers of all calibration circles are on the same plane, so it is first assumed that the z axis in the world coordinate system is 0. Since the calibration circles are arranged in an array on the calibration plate 12,

[0115] Therefore, the center distance of each calibration circle is fixed.

[0116] After the world coordinate system is established, the position index corresponding to the calibration circle is determined, recorded as ptIndex; the world coordinates of the calibration circle in the world coordinate system can be calculated:

[0117] wordCircle.x=ptIndex.x×centerDis_x

[0118] wordCircle.y=ptIndex.y×centerDis_y

[0119] wordCircle.z = 0

[0120] Wherein, wordCircle.x is the x-axis coordinate of the calibration circle in the world coordinate system, wordCircle.y is the y-axis coordinate of the calibration circle in the world coordinate system, and wordCircle.z is the z-axis coordinate of the calibration circle in the world coordinate system; ptIndex.x is the position index of the center of the calibration circle in the x-direction of the horizontal coordinate, ptIndex.y is the position index of the center of the calibration circle in the y-direction of the horizontal coordinate, centerDis_x is the real physical distance between the centers of two adjacent circles in the x-direction of the world coordinate system, and centerDis_y is the real physical distance between the centers of two adjacent circles in the y-direction of the world coordinate system.

[0121] S330, obtain the initial value of the object distance of the endoscope to be calibrated, and determine the initial value of the focal length of the endoscope to be calibrated and the initial value of the projection coordinates of the optical center of the endoscope in the image coordinate system according to the initial value of the object distance, the image coordinates and the world coordinates corresponding to the position index, so as to obtain the initial value of the internal parameter of the endoscope to be calibrated.

[0122] In the embodiment of the present application, S330 may include the following steps:

[0123] S3301, the vertical distance from the endoscope to be calibrated to the calibration plate estimated using prior knowledge is set as the initial value of the object distance, denoted as Z.

[0124] S3302: Determine the initial focal length value of the endoscope to be calibrated according to the initial object distance Z, the image coordinates and the world coordinates corresponding to the calibration circle position index:

[0125] First, according to the image coordinates of the center of the calibration circle, the horizontal distance between the adjacent circle centers in the image coordinate system is calculated, recorded as imgLevelDis, and the vertical distance between the adjacent circle centers in the image coordinate system is calculated, recorded as imgVerDis.

[0126] It should be noted that the horizontal distance and longitudinal distance here are: first determine the center of the image according to the circular field of view of the endoscope, find the second calibration circle closest to the center, and use the distance between the second calibration circle and the second calibration circle adjacent to the horizontal direction as the initial horizontal distance, and use the distance between the second calibration circle and the second calibration circle adjacent to the vertical direction as the initial longitudinal distance.

[0127] Secondly, according to the world coordinates of the center of the calibration circle, the horizontal distance between the adjacent circle centers in the world coordinate system is calculated, recorded as wordLevelDis, and the vertical distance between the adjacent circle centers in the world coordinate system is calculated, recorded as wordVerDis.

[0128] Finally, when the imaging plane of the endoscope to be calibrated is parallel to the surface of the calibration plate of the calibration circle, the initial value of the focal length is determined by the following model:

[0129] f x =(Z×imgLevelDis) / wordLevelDis;

[0130] f y =(Z×imgVerDis) / wordVerDis;

[0131] Among them, f x is the initial focal length in the x direction, f y is the initial focal length in the y direction.

[0132] S3303, determining the initial value of the projection coordinate of the optical center of the endoscope to be calibrated in the image coordinate system according to the pixel width and pixel length of the calibration image.

[0133] Specifically, let (u0, v0) be the initial value of the projection coordinates of the endoscope optical center in the image coordinate system:

[0134] u0=imgWidth / 2;

[0135] v0 = imgHeight / 2;

[0136] Where imgWidth is the pixel width of the image, and imgHeight is the pixel length of the image.

[0137] S3304: Based on the above-obtained initial value of the focal length and the initial value of the projection coordinate of the optical center in the image coordinate system, an initial internal reference model of the endoscope to be calibrated can be obtained:

[0138]

[0139] In this way, the initial value of the internal parameter of the endoscope to be calibrated can be determined according to the initial value of the focal length and the initial value of the projection coordinates of the optical center of the endoscope to be calibrated in the image coordinate system.

[0140] S340, determining the correspondence between the image coordinates and the world coordinates of the center of each calibration circle according to the position indexes of the first calibration circle and the second calibration circle, and establishing an internal reference and external reference model of the endoscope to be calibrated.

[0141] Specifically, based on the image coordinates and world coordinates corresponding to the centers of the first calibration circle and the second calibration circle, combined with the internal and external parameters of the endoscope to be calibrated, the following model is satisfied:

[0142]

[0143] Among them, α is the scale factor, x, y are the set of image coordinates of all circle centers, X, Y, Z are the set of world coordinates of all circle centers, and the two sets are arranged one by one according to the position index of the circle center; A is the internal parameter of the endoscope to be calibrated, R represents the 4x4 rotation matrix of the endoscope's camera coordinate system relative to the world coordinate system, T is the translation matrix of the endoscope's camera coordinate system relative to the world coordinate system, and R and T constitute the external parameter of the endoscope. D is the distortion coefficient, which is composed of radial coefficients k1, k2, k3 and tangential coefficients p1, p2.

[0144] Specifically, T is composed of [x t ,y t ,z t ,1], where x t is the position of the origin of the camera coordinate system in the x direction of the world coordinate system, and y t is the coordinate of the origin of the camera coordinate system in the y direction of the world coordinate system, z t is the coordinate of the origin of the camera coordinate system in the z direction of the world coordinate system.

[0145]

[0146] Among them, x ′ ,y ′ is the image distortion point, x, y are the coordinates of the undistorted point in the image, and the two are related by distortion parameters k1, k2, k3 and tangent coefficients p1, p2.

[0147] S350, using the LM algorithm to minimize the likelihood function model to estimate the internal parameters, external parameters, distortion coefficients of the endoscope to be calibrated, and the projection of the optical center of the endoscope to be calibrated in the image coordinate system.

[0148] Among them, the likelihood function model is:

[0149]

[0150] m j is the image coordinate of the center of the calibration circle in the image coordinate system, M j is the world coordinate of the center of the circle in the world coordinate system, M j is the projection point in the image coordinate system.

[0151] The initial value of the object distance, the initial value of the focal length, the initial value of the projection coordinates of the optical center in the image coordinate system, and the internal and external parameter models obtained in S330 and S340 are substituted into the likelihood function to estimate various parameters.

[0152] The LM algorithm (Levenberg-Marquardt) is an iterative algorithm for solving nonlinear least squares problems. It can converge quickly when the parameters are close to the optimal solution, and can ensure the stability and convergence of the algorithm when the initial parameter estimate is inaccurate.

[0153] In the embodiment of the present application, the LM algorithm is used to minimize the reprojection error between the endoscope projection model and the actual observed image feature points. The core idea is to balance the contribution of the first-order derivative (gradient direction) and the second-order derivative (Hessian matrix approximation) in the search direction by adjusting the coefficient of the Marquardt parameter. When approaching the optimal solution, reducing the Marquardt parameter can speed up the convergence speed; and when the algorithm deviates from the optimal solution, increasing the Marquardt parameter can make the algorithm's behavior closer to the gradient descent method, ensuring that the algorithm can continue to move forward and eventually converge to the optimal solution.

[0154] The endoscope calibration method provided in the embodiment of the present application obtains a single calibration image arranged with a first calibration circle and a second calibration circle through an endoscope, quickly obtains the correspondence between the image coordinates and the world coordinates of the center of the circle on the calibration image, and establishes a model of internal and external parameters, so that the endoscope internal parameters, external parameters and distortion parameters estimated by the minimum likelihood function are closer to the true values.

[0155] In some embodiments, before the step of using the LM algorithm to minimize the likelihood function model to estimate the internal parameters, external parameters, distortion coefficients of the endoscope to be calibrated, and the projection of the optical center of the endoscope to be calibrated in the image coordinate system, the initial focal length value can be further optimized through S910 to S940, such as Figure 7 shown.

[0156] S910, reducing disturbance of the initial focal length value f corresponding to the initial object distance value Z to obtain a first focal length, and calculating a corresponding first object distance according to the first focal length;

[0157] Specifically, with respect to the initial value Z of the object distance and the initial value f of the focal length obtained in step S3302, the initial value f of the focal length is reduced by a disturbance Δf to obtain a first focal length f0:

[0158] f x0 =f x -Δf x ;

[0159] f y0 =f y -Δfy ;

[0160] Where, the subscripts x and y represent the focal length in the x-direction and y-direction, so the initial focal length f is expressed as (f x , f y ), the disturbance Δf is expressed as (Δf x , Δf y ), the first focal length f0 is expressed as (f x0 , f y0 ).

[0161] Based on the first focal length (f x0 , f y0 ), according to the model relationship between the object distance and the focal length in S3302, the first object distance Z0 corresponding to the first focal length is calculated.

[0162] S920, adding a disturbance to the initial focal length value f corresponding to the initial object distance value Z to obtain a second focal length, and calculating a corresponding second object distance according to the second focal length;

[0163] Specifically, with respect to the initial value Z of the object distance and the initial value f of the focal length obtained in step S3302, a disturbance Δf is added to the initial value f of the focal length to obtain a second focal length f1:

[0164] f x1 =f x +Δf x ;

[0165] f y1 =f y +Δf y ;

[0166] Among them, the second focal length f1 is expressed as (f x1 , f y1 ).

[0167] Based on the second focal length (f x1 , f y1 ), according to the model relationship between the object distance and the focal length in S3302, the second object distance Z1 corresponding to the second focal length is calculated.

[0168] S930, calculating an optimized value of the initial value of the object distance based on the difference between the first focal length, the first object distance, the second focal length, the second object distance and the initial value of the focal length, and substituting it into the likelihood function model of S350.

[0169] Specifically, a functional relationship between the object distance and the focal length of the endoscope to be calibrated can be established according to the first focal length, the first object distance, the second focal length, and the second object distance obtained in S910 and S920, such as Figure 7After the functional relationship is established, the optimized value Z′ of the initial value of the object distance is interpolated on the functional relationship according to the initial value of the focal length f; and on this basis, it is substituted into the likelihood function model of S350 to estimate the internal and external parameters of the endoscope to be calibrated, thereby reducing the focal length jump caused by the disturbance of the object distance, and further improving the calibration accuracy of the endoscope to be calibrated.

[0170] An endoscope calibration system provided in an embodiment of the present application includes:

[0171] An acquisition module, used to acquire a calibration image, and segment a calibration circle on the calibration image to obtain the center of the calibration circle, wherein the calibration image includes at least three first calibration circles and a plurality of second calibration circles, and the diameter of the first calibration circle is greater than the diameter of the second calibration circle;

[0172] An image coordinate system establishment module is used to establish a calibration coordinate system according to the center positions of the three first calibration circles in the calibration circle, determine the image center coordinates of all calibration circles in the image coordinate system, and calibrate the position indexes of all calibration circles;

[0173] A world coordinate system establishment module, used to establish a world coordinate system and determine the actual position coordinates of the calibration circle on the world coordinate system according to the position index of the calibration circle shown;

[0174] An assignment module is used to obtain an initial value of the object distance of the endoscope to be calibrated, and determine an initial value of the focal length of the endoscope to be calibrated and an initial value of the projection coordinates of the optical center of the endoscope in the image coordinate system according to the initial value of the object distance, the image coordinates and the world coordinates corresponding to the position index, so as to obtain an initial value of the internal parameter of the endoscope to be calibrated;

[0175] A modeling module, used to determine the correspondence between the image coordinates and the world coordinates of the center of each marker according to the position index of the first marker and the second marker, and to establish an internal reference and external reference model of the endoscope to be calibrated;

[0176] The estimation module is used to use the LM algorithm to minimize the likelihood function model to estimate the internal parameters, external parameters, distortion coefficients of the endoscope to be calibrated, and the projection of the optical center of the endoscope to be calibrated in the image coordinate system.

[0177] By applying the technical solution provided in the present application, a single calibration image with multiple calibration circles is distributed, so that the sampling of the calibration circles on the image to be calibrated is closer to all areas of the image. At the same time, based on the a priori true object distance Z value, the initial focal length estimation value is made more accurate, so that the optimal endoscope internal parameters and distortion parameters estimated by the minimum likelihood function are closer to the true values.

[0178] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above are only specific implementation methods of the present application and are not used to limit the protection scope of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. An endoscope calibration method, characterized in that: include: Acquiring a calibration image, wherein the calibration image includes at least three first markers and a plurality of second markers arranged in an array; The image coordinates of the centers of the first marker and the second marker are obtained by an image segmentation algorithm, a calibration coordinate system is established according to the topological relationship between the first marker and the second marker, and a corresponding relationship between the image coordinates and the position index of the centers of the first marker and the second marker on the calibration coordinate system is determined; Establishing a world coordinate system, and determining a corresponding relationship between world coordinates and position indexes of the centers of the first marker and the second marker on the world coordinate system; Obtain an initial value of the object distance of the endoscope to be calibrated, and determine an initial value of the focal length of the endoscope to be calibrated and an initial value of the projection coordinates of the optical center of the endoscope in the image coordinate system according to the initial value of the object distance, the image coordinates and the world coordinates corresponding to the position index, so as to obtain an initial value of the internal parameter of the endoscope to be calibrated; According to the position index of the first marker and the second marker, the corresponding relationship between the image coordinates and the world coordinates of the center of each marker is determined, and the internal reference and external reference models of the endoscope to be calibrated are established; The LM algorithm is used to minimize the likelihood function model to estimate the internal and external parameters of the endoscope to be calibrated.

2. The endoscope calibration method according to claim 1, characterized in that: The step of obtaining the image coordinates of the centers of the first marker and the second marker by using an image segmentation algorithm includes: Converting the calibration image into a grayscale image and binarizing it according to a predetermined threshold to obtain a first segmented image; Traversing all pixel points in the grayscale image, calculating an adaptive threshold corresponding to the pixel point according to the pixel value of the neighboring point of the pixel point on the first segmented image, and segmenting the grayscale image by the adaptive threshold value to obtain a second segmented image; The outer contours of the first marker and the second marker on the second segmented image are fitted to obtain the image coordinates of the centers of the markers.

3. The endoscope calibration method according to claim 2, characterized in that: The binarization is performed according to a predetermined threshold to obtain a first segmented image, comprising: When the pixel value of a pixel in the grayscale image is greater than or equal to a preset threshold, the pixel value of the pixel is marked as 255; when the pixel value of a pixel in the grayscale image is less than the preset threshold, the pixel value of the pixel is marked as 0.

4. The endoscope calibration method according to claim 2, characterized in that: The calculating and obtaining the adaptive threshold value corresponding to the pixel point includes: A threshold set corresponding to the pixel point is established. If the pixel value of at least one of the neighborhood points of the pixel point in the first segmented image is equal to a preset value, the coordinates and grayscale values ​​of the pixel point and its neighborhood points in the grayscale image are put into the threshold set. The adaptive threshold corresponding to the pixel point is obtained by calculating the average grayscale values ​​of all pixels in the threshold set corresponding to the pixel point.

5. The endoscope calibration method according to claim 2 or 4, characterized in that: The step of segmenting the grayscale image by using an adaptive threshold to obtain a second segmented image includes: For a certain pixel in the grayscale image, if the pixel value in the grayscale image is greater than or equal to the adaptive threshold, the pixel value of the pixel is marked as 255; if the pixel value in the grayscale image is less than the adaptive threshold, the pixel value of the pixel is marked as 0.

6. The endoscope calibration method according to claim 1, characterized in that: The method of establishing a calibration coordinate system according to the topological relationship between the first marker and the second marker, and determining the corresponding relationship between the image coordinates and the position index of the centers of the first marker and the second marker on the calibration coordinate system includes: According to the topological structure between any two first markers, a calibration coordinate system is established to determine the position index of the first marker; Determine the corresponding relationship between the image coordinates of the center of the first marker and its position index according to the distance between any two first markers; Determine the correspondence between the image coordinates and the position index of the center of the second marker in the neighborhood of the first marker according to the relative direction and distance between the first marker and the second marker in the neighborhood of the first marker; The corresponding relationship between the image coordinates and the position index of all the second marker centers is determined step by step through the neighborhood point confirmation method.

7. The endoscope calibration method according to claim 6, characterized in that: The determining, based on the relative direction and distance between the first marker and the second marker in its neighborhood, the corresponding relationship between the image coordinates and the position index of the center of the second marker in its neighborhood includes: Setting a first condition: calculating a reference direction vector of the first marker in the calibration coordinate system according to the image coordinates of the center of the first marker; Setting a second condition: calculating a neighborhood approximate distance between the first marker and the closest second marker; Calculating the direction vectors of all second markers relative to the first marker and the straight-line distances between the second markers and the first marker; The direction vector of each second marker is compared with the reference direction vector, and the straight-line distance of each second marker is compared with the neighborhood approximate distance, so as to find the second marker in the neighborhood of the first marker and determine the correspondence between the image coordinates of its center and the position index.

8. The endoscope calibration method according to claim 1, characterized in that: The method of obtaining the initial value of the object distance of the endoscope to be calibrated, determining the initial value of the focal length of the endoscope to be calibrated and the initial value of the projection coordinates of the optical center of the endoscope in the image coordinate system according to the initial value of the object distance and the image coordinates and world coordinates corresponding to the marker position index, and obtaining the initial value of the internal parameter of the endoscope to be calibrated includes: The vertical distance from the endoscope to be calibrated to the calibration plate estimated by using prior knowledge is set as the initial value of the object distance; Determine the initial focal length value of the endoscope to be calibrated according to the initial object distance value, the image coordinates and the world coordinates corresponding to the marker position index; Determine the initial value of the projection coordinate of the optical center of the endoscope to be calibrated in the image coordinate system according to the pixel width and pixel length of the calibration image; According to the obtained initial value of the focal length and the initial value of the projection coordinates of the optical center in the image coordinate system, the initial internal reference model of the endoscope to be calibrated can be obtained.

9. The endoscope calibration method according to claim 8, characterized in that: According to the obtained initial value of the focal length and the initial value of the projection coordinate of the optical center in the image coordinate system, an initial internal reference model of the endoscope to be calibrated can be obtained, including: Among them, f x is the initial focal length in the x direction, f y is the initial value of the focal length in the y direction; (u0, v0) is the initial value of the projection coordinates of the optical center of the endoscope to be calibrated in the image coordinate system.

10. The endoscope calibration method according to claim 1, characterized in that: The method of determining the correspondence between the image coordinates and the world coordinates of the center of each marker according to the position index of the first marker and the second marker, and establishing the internal reference and external reference models of the endoscope to be calibrated includes: Among them, α is the scale coefficient, x, y are the set of image coordinates of all centers, X, Y, Z are the set of world coordinates of all centers, and the two sets are arranged one by one according to the position index of the center; A is the internal parameter of the endoscope to be calibrated, R represents the 4 times 4 rotation matrix of the camera coordinate system of the endoscope relative to the world coordinate system, T is the translation matrix of the camera coordinate system of the endoscope relative to the world coordinate system, R and T constitute the external parameter of the endoscope; D is the distortion coefficient, which is composed of radial coefficients k1, k2, k3 and tangential coefficients p1, p2.

11. The endoscope calibration method according to claim 10, characterized in that: The likelihood function model includes: Among them, m j is the image coordinate of the center of the marker in the image coordinate system, M j is the world coordinate of the center of the marker in the world coordinate system, M j is the projection point in the image coordinate system.

12. The endoscope calibration method according to claim 1, characterized in that: The initial value of the object distance can also be further optimized to reduce disturbance interference, including: Reducing disturbance on an initial focal length value corresponding to an initial object distance value to obtain a first focal length, and calculating a corresponding first object distance according to the first focal length; Adding a disturbance to an initial focal length value corresponding to an initial object distance value to obtain a second focal length, and calculating a corresponding second object distance according to the second focal length; According to the first focal length, the first object distance, the second focal length, the second object distance and the initial value of the focal length, the difference is used to calculate the optimized value of the initial value of the object distance, and the value is substituted into the likelihood function model.

13. An endoscope calibration system, characterized in that: include: An acquisition module, used for acquiring a calibration image, wherein the calibration image includes at least three first markers and a plurality of second markers arranged in an array; An image coordinate establishment module, used to obtain the image coordinates of the centers of the first marker and the second marker by an image segmentation algorithm, establish a calibration coordinate system according to the topological relationship between the first marker and the second marker, and determine the corresponding relationship between the image coordinates and the position index of the centers of the first marker and the second marker on the calibration coordinate system; A world coordinate establishment module, used to establish a world coordinate system and determine the correspondence between the world coordinates and position indexes of the centers of the first marker and the second marker in the world coordinate system; An assignment module is used to obtain an initial value of the object distance of the endoscope to be calibrated, and determine an initial value of the focal length of the endoscope to be calibrated and an initial value of the projection coordinates of the optical center of the endoscope in the image coordinate system according to the initial value of the object distance, the image coordinates and the world coordinates corresponding to the position index, so as to obtain an initial value of the internal parameter of the endoscope to be calibrated; A modeling module, used to determine the correspondence between the image coordinates and the world coordinates of the center of each marker according to the position index of the first marker and the second marker, and to establish an internal reference and external reference model of the endoscope to be calibrated; The estimation module is used to use the LM algorithm to minimize the likelihood function model to estimate the internal parameters, external parameters, distortion coefficients of the endoscope to be calibrated, and the projection of the optical center of the endoscope to be calibrated in the image coordinate system.

14. An endoscope calibration tool, characterized in that: include: A calibration plate, on which a plurality of markers arranged in an array are disposed, wherein the markers include at least three first markers and a plurality of second markers, and the first markers and the second markers have different diameters; The endoscope to be calibrated is used to collect the image of the calibration plate to obtain a calibration image.

15. The endoscope calibration tool according to claim 14, characterized in that: The endoscope to be calibrated is used to adjust the position of the endoscope relative to the calibration plate through a rotatable endoscope tube fixing frame.

16. The endoscope calibration tool according to claim 14, characterized in that: At least three non-colinear optical tracking marks are arranged on the marking plate.

Images