Calibration method and system for a fluorescence endoscope

Abstract

The application provides a kind of calibration method and system of fluorescence endoscope, including control fluorescence endoscope to prefabricated calibration board image according to preset acquisition mode respectively visible light and fluorescence image acquisition, obtain visible light channel calibration image and fluorescence image calibration image;Chessboard corner coordinates in visible light channel calibration image are as input based on Zhang Zhengyou calibration method processing, based on the distortion parameter, chessboard corner coordinates calculation, obtain first feature circle center coordinates;Based on Zhang Zhengyou calibration method processing, obtain the distortion parameter of fluorescence lens of fluorescence endoscope, based on the distortion parameter, feature circle center coordinates calculation, obtain second feature circle center coordinates;According to the corresponding relationship of first feature circle center coordinates and second feature circle center coordinates, obtain homography matrix, record the calibration data of visible light lens and fluorescence lens according to the distortion parameter, homography matrix.

Description

TECHNICAL FIELD

[0001] The present application relates to endoscopy technology, in particular to a calibration method and system of a fluorescence endoscope. BACKGROUND

[0002] Endoscopes are commonly used surgical instruments in surgical operations. White light endoscopes are mainly used to observe the real images in the body cavity of a subject; fluorescence endoscopy technology has the advantage of wide spectrum imaging, which can improve the visibility of lesions and pre-lesion areas, and is mainly used to observe lesions or pre-lesion sites that cannot be effectively captured by real images. The main core components of a fluorescence endoscope include an endoscope tube, an endoscope light source module, an endoscope camera system, and adapter / lens core components. The fluorescence endoscope camera system has at least two light sensing channels, one of which is a fluorescence channel and the other of which is a visible light channel. For example, the optical design of an ICG fluorescence endoscope requires optical design and aberration optimization between 400-850nm.

[0003] Due to physical limitations in optics, there may be slight differences in image size between the fluorescence channel and the visible light channel. In addition, during the assembly of the fluorescence endoscope, the images of the fluorescence channel and the visible light channel need to be strictly aligned. The existing alignment method requires very high assembly precision and has low assembly efficiency, which limits the production capacity of fluorescence endoscopes.

[0004] Therefore, how to calibrate the endoscope, improve the alignment accuracy, and at the same time improve the assembly efficiency of the fluorescence endoscope, has become a problem to be solved. SUMMARY

[0005] The embodiments of the present application provide a calibration method and system of a fluorescence endoscope, which can calibrate the endoscope, improve the alignment accuracy, and at the same time improve the assembly efficiency of the fluorescence endoscope.

[0006] In a first aspect, the embodiments of the present application provide a calibration method of a fluorescence endoscope, comprising:

[0007] controlling the fluorescence endoscope to collect images of visible light and fluorescence respectively according to a preset collection mode on a pre-prepared calibration plate image, to obtain a visible light channel calibration image and a fluorescence image calibration image, wherein the calibration plate image comprises a square pattern with visible light color and a circular pattern with fluorescence color, and the square pattern and the circular pattern are distributed in the calibration plate image in the form of a checkerboard;

[0008] processing the checkerboard corner point coordinates in the visible light channel calibration image as input based on Zhang Zhengyou calibration method, to obtain distortion parameters of the visible light lens of the fluorescence endoscope, and calculating first feature circle center coordinates based on the distortion parameters and the checkerboard corner point coordinates;

[0009] The fluorescent image is processed based on the Zhang Zhengyou calibration method with the feature circle center coordinates in the calibration image as input, to obtain distortion parameters of the fluorescent endoscope fluorescent lens, and the second feature circle center coordinates are calculated based on the distortion parameters and the feature circle center coordinates;

[0010] A homography matrix is obtained according to the corresponding relationship between the first feature circle center coordinates and the second feature circle center coordinates, and calibration data of the visible light lens and the fluorescent lens are recorded based on the distortion parameters and the homography matrix.

[0011] Optionally, in a possible implementation manner of the first aspect, the calibration board image is generated by the following steps, specifically comprising:

[0012] A feature circle image is generated, which is an array including a plurality of circular patterns of the same specification, and adjacent circular patterns are distributed at a preset distance;

[0013] A checkerboard image is generated based on the feature circle image, which includes checkerboards that are alternately black and white and are uniformly distributed, and the circular patterns are located in the white grids in the checkerboard image;

[0014] The circular patterns in the feature circle image are printed by infrared ink, and the black grids in the checkerboard image are printed by white light ink, to obtain the calibration board image.

[0015] Optionally, in a possible implementation manner of the first aspect, the control of the fluorescent endoscope to pre-made calibration board images for image acquisition of visible light and fluorescence respectively in a preset acquisition mode to obtain visible light channel calibration images and fluorescent image calibration images comprises:

[0016] The calibration board image is placed under the imaging field of view of the fluorescent endoscope;

[0017] The relative angle of the calibration board and the endoscope is changed, and the calibration board is moved multiple times for image acquisition, to obtain visible light channel calibration images and fluorescent image calibration images at multiple angles.

[0018] Optionally, in a possible implementation manner of the first aspect, the checkerboard corner point coordinates in the visible light channel calibration image are processed based on the Zhang Zhengyou calibration method with the distortion parameters of the fluorescent endoscope visible light lens as output, and the first feature circle center coordinates are calculated based on the distortion parameters and the checkerboard corner point coordinates, comprising:

[0019] The calibration board checkerboard coordinates are extracted by processing the visible light calibration image, and the physical coordinate values of the checkerboard corner points in the calibration image are calculated according to the specification of the checkerboard and the origin of the world coordinate system;

[0020] The physical coordinate values of the chessboard corner points are input, and based on Zhang Zhengyou calibration method, distortion parameters of the visible light lens are calculated;

[0021] The distortion parameters are used to correct the distortion of the chessboard corner point coordinates to obtain processed chessboard corner point coordinates, and the first feature circle center coordinates are calculated according to the chessboard corner point coordinates and the offset of the corner point to the circle center.

[0022] Optionally, in a possible implementation manner of the first aspect, the processing of the visible light calibration image to extract the calibration board chessboard coordinates comprises:

[0023] The SURF feature point detection algorithm is used to detect the chessboard corner points, and non-maximum suppression is used to position the detected corner points to obtain the chessboard corner point coordinates;

[0024] The undetected image corner point positions are supplemented according to the detected chessboard corner point coordinates and the known corner point distribution, and the sub-pixel coordinates of each corner point in an 8-neighborhood range are calculated to obtain the physical coordinate values of the corresponding chessboard corner points;

[0025] The supplement of the undetected image corner point positions according to the detected chessboard corner point coordinates and the known corner point distribution comprises:

[0026] The number of the chessboard corner point coordinates of the same row with the same horizontal coordinate is counted to obtain a first corner point number, a first preset number corresponding to each same horizontal coordinate under the known corner point distribution is determined, and if the first corner point number corresponds to the first preset number, it is determined that the chessboard corner point coordinates of the corresponding row have all been detected;

[0027] If the first corner point number does not correspond to the first preset number, the chessboard corner point coordinates of the corresponding row are taken as first grid corner point coordinates, and the chessboard corner point coordinates of other rows are taken as second grid corner point coordinates;

[0028] The vertical coordinate values of the first grid corner point coordinates and the vertical coordinates of the second grid corner point coordinates are compared, and the vertical coordinates that exist in the second grid corner point coordinates and do not exist in the first grid corner point coordinates are determined as to-be-supplemented vertical coordinates;

[0029] The horizontal coordinates corresponding to the corresponding row are respectively supplemented to each to-be-supplemented vertical coordinate to obtain the coordinates corresponding to the undetected image corner point positions.

[0030] Optionally, in a possible implementation manner of the first aspect, the distortion correction on the checkerboard corner point coordinates based on the distortion parameter to obtain processed checkerboard corner point coordinates, and the calculation of the first feature circle center coordinates according to the checkerboard corner point coordinates and the offset of the corner point to the center of the circle comprises:

[0031] determining a circular pattern in a feature circle image corresponding to the corresponding checkerboard corner point coordinates, each checkerboard corner point having a corresponding circular pattern;

[0032] calculating the first feature circle center coordinates of the corresponding circular pattern by the following formula,

[0033] c = b + t

[0034] wherein c is the first feature circle center coordinates, b is the checkerboard corner point coordinates, and t is the offset of the corner point to the center of the circle.

[0035] Optionally, in a possible implementation manner of the first aspect, the feature circle center coordinates in the fluorescence image calibration image are input based on the Zhang Zhengyou calibration method to obtain the distortion parameter of the fluorescence endoscope fluorescence lens, and the second feature circle center coordinates are calculated based on the distortion parameter and the feature circle center coordinates, comprising:

[0036] identifying the center of the circular pattern in the fluorescence image calibration image based on the Hough circle detection algorithm, and obtaining the feature circle center coordinates;

[0037] determining the undetected feature circle according to the detected feature circle center coordinates and the preset circle distribution rule, and filling in the feature circle center coordinates of the undetected feature circle;

[0038] calculating the physical coordinate values of the feature circle centers in the calibration image according to the known feature circle distribution and the origin of the world coordinate system;

[0039] the determination of the undetected feature circle according to the detected feature circle center coordinates and the preset circle distribution rule, and the filling in of the feature circle center coordinates of the undetected feature circle, comprising:

[0040] counting the number of feature circle center coordinates of the same row with the same horizontal coordinate to obtain a first feature circle number, determining a second preset number corresponding to each same horizontal coordinate under the preset circle distribution rule, and if the first feature circle number and the second preset number correspond to each other, it is determined that the feature circle center coordinates corresponding to the corresponding row have been completely detected;

[0041] if the first feature circle number and the second preset number do not correspond to each other, the feature circle center coordinates corresponding to the corresponding row are taken as first circle center coordinates, and the feature circle center coordinates of other rows are taken as second circle center coordinates;

[0042] Compare the ordinate value of the first center coordinate with the ordinate value of the second center coordinate, and determine the ordinate values ​​that exist in the second center coordinate but not in the first center coordinate as the ordinate values ​​to be filled in.

[0043] The horizontal coordinates of the corresponding rows are filled in with the vertical coordinates of each undetected feature circle to obtain the coordinates of the feature circle center.

[0044] Optionally, in one possible implementation of the first aspect, obtaining the homography matrix based on the correspondence between the coordinates of the center of the first feature circle and the coordinates of the center of the second feature circle, and recording the distortion parameters and homography matrix to obtain calibration data for the visible light lens and the fluorescent lens, includes:

[0045] The visible light channel calibration image and the fluorescence image calibration image at the same angle are taken as a set of images. The coordinates of the center of the first feature circle and the center of the second feature circle corresponding to the same set of images are determined. The homography matrix of the center coordinates of the first feature circle and the center of the second feature circle is calculated using the following formula.

[0046] j = Hc

[0047] j (x1, y1, 1) represents the coordinates of the center of the second characteristic circle, and c (x2, y2, 1) represents the coordinates of the center of the circle. T Here are the coordinates of the center of the first characteristic circle, and H is a 3*3 homography matrix;

[0048] After determining the homography matrix of all groups of images, the mean is calculated to obtain the overall homography matrix. The distortion parameters and homography matrix are then recorded to obtain the calibration data for the visible light lens and the fluorescent lens.

[0049] Optionally, in one possible implementation of the first aspect, it also includes:

[0050] After acquiring the fluorescence image and the white light image to be processed by the fluorescence endoscope, the distortion of the fluorescence image to be processed is corrected based on the distortion parameters of the fluorescence channel. Then, the homography transformation is performed according to the homography matrix to obtain the calibrated fluorescence image.

[0051] After correcting the distortion of the fluorescence image to be processed based on the distortion parameters of the white light channel, a calibrated variable light image is obtained.

[0052] A second aspect of the present invention provides a calibration system for a fluorescence endoscope, comprising:

[0053] The control module is used to control the fluorescence endoscope to acquire visible light and fluorescence images of the pre-made calibration plate image according to the preset acquisition method, so as to obtain visible light channel calibration image and fluorescence image calibration image. The calibration plate image includes a square pattern for visible light color development and a circular pattern for fluorescence color development. The square pattern and the circular pattern are distributed in a checkerboard pattern within the calibration plate image.

[0054] The visible light processing module is used to take the coordinates of the checkerboard corner points in the visible light channel calibration image as input and process them based on the Zhang Zhengyou calibration method to obtain the distortion parameters of the visible light lens of the fluorescence endoscope. Based on the distortion parameters and the coordinates of the checkerboard corner points, the coordinates of the center of the first feature circle are calculated.

[0055] The fluorescence processing module is used to take the coordinates of the center of the feature circle in the fluorescence image calibration image as input and process it according to Zhang Zhengyou calibration method to obtain the distortion parameters of the fluorescence lens of the fluorescence endoscope. Based on the distortion parameters and the coordinates of the center of the feature circle, the coordinates of the center of the second feature circle are calculated.

[0056] The recording module is used to obtain the homography matrix based on the correspondence between the center coordinates of the first feature circle and the center coordinates of the second feature circle, and to record the distortion parameters and homography matrix to obtain the calibration data of the visible light lens and the fluorescent lens.

[0057] Beneficial effects:

[0058] This method ensures precise alignment between fluorescence and visible light images, improving the imaging accuracy of fluorescence endoscopes and significantly reducing the difficulty of assembling and adjusting imaging endoscopes. The method uses fluorescent ink for calibration plate printing, effectively solving the problem of visible light and fluorescent materials coexisting on the same calibration plate. Using a known calibration pattern, detection is more accurate and computational workload is greatly reduced. This simplifies calibration operations and improves calibration accuracy. Attached Figure Description

[0059] Figure 1 This is a schematic flowchart of a calibration method for a fluorescence endoscope provided in an embodiment of the present invention;

[0060] Figure 2 This is a schematic diagram of a calibration plate image provided in an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0063] It should be understood that in the various embodiments of the present invention, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0064] It should be understood that in this invention, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0065] It should be understood that in this invention, "multiple" refers to two or more. "And / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "Contains A, B, and C", "Contains A, B, and C" means that all three A, B, and C are contained; "Contains A, B, or C" means that one of A, B, and C is contained; "Contains A, B, and / or C" means that any one, two, or three of A, B, and C are contained.

[0066] It should be understood that in this invention, "B corresponding to A", "B corresponding to A", "A and B correspond", or "B and A correspond" means that B is associated with A, and B can be determined based on A. Determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information. Matching A and B is defined as a similarity between A and B that is greater than or equal to a preset threshold.

[0067] Depending on the context, "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection."

[0068] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0069] See Figure 1 This is a schematic flowchart of a calibration method for a fluorescence endoscope provided in an embodiment of the present invention. The method includes S101-S102, as follows:

[0070] S101, control the fluorescence endoscope to acquire visible light and fluorescence images of the pre-made calibration plate image according to the preset acquisition method, and obtain visible light channel calibration image and fluorescence image calibration image. The calibration plate image includes a square pattern for visible light color development and a circular pattern for fluorescence color development. The square pattern and the circular pattern are distributed in a checkerboard pattern within the calibration plate image.

[0071] During calibration, firstly, the fluorescence endoscope needs to be controlled to acquire visible light and fluorescence images of the pre-made calibration plate according to the preset acquisition method, so as to obtain visible light channel calibration images and fluorescence image calibration images.

[0072] See Figure 2 The aforementioned calibration plate image includes a square pattern that displays color under visible light and a circular pattern that displays color under fluorescence. The square and circular patterns are distributed in a checkerboard pattern within the calibration plate image.

[0073] The calibration board image is generated through the following steps:

[0074] Generate a feature circle image, which is an array of multiple circular patterns of the same size, with adjacent circular patterns distributed at a preset distance. See also Figure 2 The feature circle image has a series of feature circle array patterns, the circular patterns are of equal size and distributed at fixed distances.

[0075] A checkerboard image is generated based on the feature circle image. The checkerboard image comprises alternating black and white squares evenly distributed within the white squares of the checkerboard image, and the circular pattern is located within these white squares. (See checkerboard image for reference.) Figure 2 The circular pattern is located within the white squares of the chessboard grid image.

[0076] The circular pattern in the feature circle image is printed with infrared ink, and the black squares in the checkerboard image are printed with white ink to obtain the calibration plate image. It is understood that, in order to obtain the visible light channel calibration image and the fluorescence image calibration image, the circular pattern in the feature circle image of this scheme is printed with infrared ink, and the black squares in the checkerboard image are printed with white ink. It is worth mentioning that the infrared ink printing contains infrared upconversion materials, which can generate infrared light of 800-830nm when stimulated. This means that under infrared imaging, a fluorescence image calibration image can be obtained, in which only the circular pattern in the feature circle image can be seen. The white ink printing can be ordinary ink printing, meaning it cannot be stimulated to emit infrared light, only reflecting visible light. Under white light imaging, a visible light channel calibration image can be obtained, in which only the black squares in the checkerboard image can be seen.

[0077] In some embodiments, the controlled fluorescence endoscope acquires visible light and fluorescence images from a pre-fabricated calibration plate image according to a preset acquisition method, respectively, to obtain a visible light channel calibration image and a fluorescence image calibration image, including:

[0078] Place the calibration plate image under the imaging field of the fluorescence endoscope;

[0079] By changing the relative angle between the calibration plate and the endoscope and moving the calibration plate multiple times to acquire images, visible light channel calibration images and fluorescence image calibration images from multiple angles are obtained.

[0080] It is worth mentioning that, in order to improve the accuracy of calibration, this method requires changing the relative angle between the calibration plate and the endoscope and moving the calibration plate multiple times to acquire images, thereby obtaining visible light channel calibration images and fluorescence image calibration images from multiple angles.

[0081] S102, the coordinates of the checkerboard corner points in the visible light channel calibration image are used as input and processed according to Zhang Zhengyou calibration method to obtain the distortion parameters of the visible light lens of the fluorescence endoscope. Based on the distortion parameters and the coordinates of the checkerboard corner points, the coordinates of the center of the first feature circle are calculated.

[0082] After obtaining the visible light channel calibration image, this scheme uses the coordinates of the checkerboard corner points in the visible light channel calibration image as input and processes them based on the Zhang Zhengyou calibration method to obtain the distortion parameters of the visible light lens of the fluorescence endoscope. Then, using the distortion parameters and the coordinates of the checkerboard corner points, the coordinates of the center of the first feature circle are calculated.

[0083] Understandably, the above method calculates the coordinates of the center of the first feature circle by using the coordinates of the corner points of the checkerboard pattern of the black grid on the visible light channel calibration image. In subsequent steps, the circular coordinates of the circular pattern under the fluorescence image calibration image are obtained, and then the two circular coordinates are used for calibration.

[0084] S102 (taking the coordinates of the checkerboard corner points in the visible light channel calibration image as input and processing them based on the Zhang Zhengyou calibration method to obtain the distortion parameters of the visible light lens of the fluorescence endoscope, and calculating the coordinates of the center of the first feature circle based on the distortion parameters and the checkerboard corner point coordinates) includes S112-S132:

[0085] S112, process the visible calibration image to extract the coordinates of the calibration board checkerboard, and calculate the physical coordinate values ​​of the checkerboard corner points in the calibration image based on the checkerboard specifications and the origin of the world coordinate system.

[0086] First, this solution processes the visible cursor calibration image to extract the coordinates of the calibration board checkerboard, where the checkerboard coordinates are the coordinates of the corner points of the black squares.

[0087] Then, based on the dimensions of the checkerboard and the origin of the world coordinate system, the physical coordinates of the checkerboard corner points in the calibration image are calculated. The physical coordinates are related to the dimensions of the checkerboard. For example, if the checkerboard is 1cm x 1cm, the physical coordinates will be calculated using the 1cm x 1cm dimensions, with the horizontal and vertical coordinates represented by distance. The horizontal distance represents the horizontal coordinate, and the vertical distance represents the vertical coordinate. The method for obtaining these physical coordinates is existing technology and will not be elaborated further.

[0088] In some embodiments, the step of processing the visible calibration image to extract the coordinates of the calibration board checkerboard, and calculating the physical coordinate values ​​of the checkerboard corner points in the calibration image based on the checkerboard specifications and the world coordinate system origin, includes:

[0089] This method detects checkerboard corner points using the SURF feature point detection algorithm and then uses non-maximum suppression to perform primitive-level coordinate localization of the detected corner points, thus obtaining the checkerboard corner point coordinates. The approach first combines the existing SURF feature point detection algorithm to detect checkerboard corner points, and then uses non-maximum suppression to perform primitive-level coordinate localization of the detected corner points, thereby obtaining the checkerboard corner point coordinates.

[0090] Based on the detected checkerboard corner coordinates and the known corner distribution, undetected image corner positions are filled in. Subpixel coordinates are calculated by fitting within an 8-neighborhood of each corner point, serving as the physical coordinates of the corresponding checkerboard corner. It's worth noting that in some cases, due to lighting or detection issues, some checkerboard corner coordinates may not be detected. In such cases, filling in the missing coordinates and then calculating subpixel coordinates within an 8-neighborhood of each corner point is necessary to obtain the physical coordinates of the corresponding checkerboard corner.

[0091] The step of filling in the undetected image corner positions based on the detected checkerboard corner coordinates and known corner distribution includes A1-A4:

[0092] A1. Count the number of corner points of the same row of chessboard with the same horizontal coordinate to obtain the first number of corner points. Determine the first preset number corresponding to each same horizontal coordinate under the known corner point distribution. If the first number of corner points corresponds to the first preset number, it is determined that the corner point coordinates of the corresponding row of chessboard have been detected.

[0093] First, this scheme counts the number of chessboard corner coordinates with the same horizontal coordinate in the same row to obtain the first number of corners. Then, it determines the first preset number corresponding to each same horizontal coordinate under the known corner distribution. If the first number of corners corresponds to the first preset number, it is determined that the chessboard corner coordinates corresponding to the corresponding row have been detected.

[0094] For example, see Figure 2 The first preset quantity in the first row is 10. If the number of detected first corner points is 9, it means that one is missing. In this case, this solution will perform a supplementary operation.

[0095] A2, if the number of the first corner points does not correspond to the first preset number, then the coordinates of the corner points of the chessboard corresponding to the corresponding row are taken as the coordinates of the first corner point, and the coordinates of the corner points of the chessboard of other rows are taken as the coordinates of the second corner point.

[0096] When performing the completion process, this method first uses the coordinates of the corner points of the chessboard corresponding to the row as the first corner point coordinates. For example, the coordinates of the corner points of the chessboard corresponding to the first row are used as the first corner point coordinates. Then, the coordinates of the corner points of the chessboard in other rows are obtained as the second corner point coordinates. For example, the coordinates of the corner points of the chessboard in the third row are obtained as the second corner point coordinates.

[0097] A3. Compare the ordinate of the first corner point with the ordinate of the second corner point, and determine the ordinate that exists in the second corner point but does not exist in the first corner point as the ordinate to be filled.

[0098] Then, this scheme will compare the ordinate value of the first corner point with the ordinate value of the second corner point, and find the ordinate value that exists in the second corner point but does not exist in the first corner point as the ordinate value to be filled.

[0099] It is understandable that the ordinates that exist in the second cell corner coordinates but not in the first cell corner coordinates are missing in the current row but exist in other rows, and are used as the ordinates to be filled.

[0100] A4, fill in the corresponding horizontal coordinates of the corresponding rows with the corresponding vertical coordinates to obtain the coordinates of the undetected image corner points.

[0101] After obtaining the vertical coordinates to be filled, this method can fill in the horizontal coordinates of the corresponding rows with the vertical coordinates to be filled, thereby obtaining the coordinates corresponding to the undetected image corner points.

[0102] S122: Input the physical coordinates of the corner points of the chessboard grid, and calculate the distortion parameters of the visible light lens based on Zhang Zhengyou's calibration method.

[0103] After obtaining the physical coordinates of the corner points of the chessboard, these coordinates are used as input, and the distortion parameters of the visible light lens are calculated using the Zhang Zhengyou calibration method.

[0104] S132, based on the distortion parameters, the corner coordinates of the chessboard are distorted to obtain the processed corner coordinates of the chessboard. The coordinates of the center of the first feature circle are calculated based on the corner coordinates of the chessboard and the offset from the corner to the center of the circle.

[0105] After obtaining the distortion parameters, this scheme will perform distortion correction on the corner coordinates of the chessboard grid to obtain the processed corner coordinates of the chessboard grid. Then, the coordinates of the center of the first feature circle will be calculated by combining the corner coordinates of the chessboard grid and the offset of the corner point from the center of the circle.

[0106] In some embodiments, the process of correcting the distortion of the checkerboard corner coordinates based on distortion parameters to obtain the processed checkerboard corner coordinates, and calculating the coordinates of the center of the first feature circle based on the checkerboard corner coordinates and the offset from the corner to the center of the circle, includes:

[0107] This method determines the circular pattern within the feature circle image corresponding to the coordinates of each chessboard corner point. Each chessboard corner point has a corresponding circular pattern.

[0108] The coordinates of the center of the first feature circle of the corresponding circular pattern can be calculated using the following formula.

[0109] c = b + t

[0110] Where c is the coordinate of the center of the first feature circle, b is the coordinate of the corner point of the chessboard, and t is the offset of the corner point from the center of the circle.

[0111] It is worth mentioning that, where 'c' represents the coordinates of the center of the circle under the visible light channel, and 'b' represents the coordinates of a corner point in each square of the checkerboard, preferably the bottom right corner point of each square. 't' indicates that the offset from the corner point to the center of the circle in the visible light calibration plate is a known quantity, and the offset can be, for example, a horizontal offset or a vertical offset.

[0112] It is understandable that, by using the above method, the coordinates of the center of the first feature circle of the corresponding circular pattern can be calculated by combining the corner points of the chessboard.

[0113] S103, the coordinates of the center of the feature circle in the fluorescence image calibration image are used as input and processed according to Zhang Zhengyou calibration method to obtain the distortion parameters of the fluorescence lens of the fluorescence endoscope. Based on the distortion parameters and the coordinates of the center of the feature circle, the coordinates of the center of the second feature circle are calculated.

[0114] This scheme uses the coordinates of the center of the feature circle in the fluorescence image calibration image as input and processes it based on the Zhang Zhengyou calibration method to obtain the distortion parameters of the fluorescence lens of the fluorescence endoscope. Based on the distortion parameters and the coordinates of the center of the feature circle, the coordinates of the center of the second feature circle are calculated.

[0115] The process of using the coordinates of the center of the feature circle in the fluorescence image calibration image as input and processing it based on the Zhang Zhengyou calibration method to obtain the distortion parameters of the fluorescence lens of the fluorescence endoscope, and calculating the coordinates of the center of the second feature circle based on the distortion parameters and the coordinates of the center of the feature circle, includes:

[0116] The Hough circle detection algorithm is used to identify the center of circular patterns in the fluorescence image calibration image and obtain the coordinates of the feature circle center. This scheme identifies the calibration plate image of the obtained fluorescence channel to obtain the image coordinates of the center of the feature circle in the calibration plate. A preferred method for identifying the center of the feature circle is the Hough circle detection algorithm.

[0117] Based on the coordinates of the centers of the detected feature circles and the preset circle distribution pattern, the undetected feature circles are identified, and the coordinates of the center of the undetected feature circles are filled in. This scheme uses the known feature circle distribution pattern to filter out false identification points, and predicts and fills in the positions of the undetected feature circles based on the detected feature circles and the known feature circle distribution.

[0118] Based on the known distribution of feature circles and the origin of the world coordinate system, the physical coordinates of the centers of the feature circles in the calibration image are calculated.

[0119] The step of determining undetected feature circles based on the center coordinates of detected feature circles and a preset circle distribution pattern, and then filling in the center coordinates of the undetected feature circles, includes:

[0120] The number of feature circles with the same horizontal coordinate in the same row is counted to obtain the first number of feature circles. The second preset number corresponding to each same horizontal coordinate under the preset circle distribution pattern is determined. If the first number of feature circles and the second preset number correspond, it is determined that the center coordinates of all feature circles corresponding to the corresponding row have been detected.

[0121] For example, see Figure 2The second preset quantity corresponding to each identical horizontal coordinate in the first row is 4. If the number of first feature circles obtained by statistics is 3, it means that 1 is missing. At this time, this solution will perform a supplementary operation.

[0122] If the number of the first feature circles does not correspond to the second preset number, then the center coordinates of the feature circles corresponding to the corresponding row are used as the first center coordinates, and the center coordinates of the feature circles in other rows are used as the second center coordinates. For example, the center coordinates of the feature circles corresponding to the first row can be used as the first center coordinates, and the center coordinates of the feature circles corresponding to the third row can be used as the first center coordinates, where the third row is complete and has no missing data.

[0123] By comparing the ordinate values ​​of the first and second center coordinates, the ordinates that exist in the second center coordinates but not in the first center coordinates are identified as the ordinates to be filled in.

[0124] The horizontal coordinates of the corresponding rows are filled in with the vertical coordinates of each undetected feature circle to obtain the coordinates of the feature circle center.

[0125] After obtaining the vertical coordinates to be filled, this method will fill in the horizontal coordinates of the corresponding rows with the corresponding vertical coordinates to be filled, and then obtain the coordinates of the center of the feature circle corresponding to the center of the undetected feature circle.

[0126] S104. Based on the correspondence between the center coordinates of the first feature circle and the center coordinates of the second feature circle, the homography matrix is ​​obtained. The distortion parameters and the homography matrix are recorded to obtain the calibration data of the visible light lens and the fluorescent lens.

[0127] After obtaining the coordinates of the center of the first feature circle and the center of the second feature circle, this scheme combines the correspondence between the coordinates of the first feature circle and the center of the second feature circle to obtain the homography matrix. Then, the distortion parameters and the homography matrix are recorded to obtain the calibration data of the visible light lens and the fluorescent lens.

[0128] The process of obtaining the homography matrix based on the correspondence between the center coordinates of the first and second feature circles, and recording the distortion parameters and homography matrix to obtain calibration data for the visible light lens and the fluorescence lens, includes:

[0129] The visible light channel calibration image and the fluorescence image calibration image at the same angle are taken as a set of images. The coordinates of the center of the first feature circle and the center of the second feature circle corresponding to the same set of images are determined. The homography matrix of the center coordinates of the first feature circle and the center of the second feature circle is calculated using the following formula.

[0130] j = Hc

[0131] j (x1, y1, 1) represents the coordinates of the center of the second characteristic circle, and c (x2, y2, 1) represents the coordinates of the center of the circle. T The coordinates of the center of the first characteristic circle are given, and H is a 3*3 homography matrix.

[0132] After determining the homography matrix of all image groups, the average is calculated to obtain the overall homography matrix. The distortion parameters and homography matrix are then recorded to obtain calibration data for the visible light lens and the fluorescent lens. Once the calibration data is obtained, calibration can be performed by referring to it.

[0133] Understandably, a homography matrix value can be calculated for each calibration image, and the average of the homography matrices is used as the final homography matrix.

[0134] Based on the above embodiments, it also includes:

[0135] After acquiring the fluorescence image and the white light image to be processed by the fluorescence endoscope, the distortion of the fluorescence image to be processed is corrected based on the distortion parameters of the fluorescence channel. Then, homography transformation is performed according to the homography matrix to obtain the calibrated fluorescence image.

[0136] After correcting the distortion of the fluorescence image to be processed based on the distortion parameters of the white light channel, a calibrated variable light image is obtained.

[0137] The homography transformation method is as follows:

[0138] q = H - 1p

[0139] Where H is the homography matrix, p is the pixel coordinates of the original image of the fluorescence endoscope, and q is the pixel coordinates of each pixel in the corrected fluorescence image.

[0140] This invention provides a calibration system for a fluorescence endoscope, comprising:

[0141] The control module is used to control the fluorescence endoscope to acquire visible light and fluorescence images of the pre-made calibration plate image according to the preset acquisition method, so as to obtain visible light channel calibration image and fluorescence image calibration image. The calibration plate image includes a square pattern for visible light color development and a circular pattern for fluorescence color development. The square pattern and the circular pattern are distributed in a checkerboard pattern within the calibration plate image.

[0142] The visible light processing module is used to take the coordinates of the checkerboard corner points in the visible light channel calibration image as input and process them based on the Zhang Zhengyou calibration method to obtain the distortion parameters of the visible light lens of the fluorescence endoscope. Based on the distortion parameters and the coordinates of the checkerboard corner points, the coordinates of the center of the first feature circle are calculated.

[0143] The fluorescence processing module is used to take the coordinates of the center of the feature circle in the fluorescence image calibration image as input and process it according to Zhang Zhengyou calibration method to obtain the distortion parameters of the fluorescence lens of the fluorescence endoscope. Based on the distortion parameters and the coordinates of the center of the feature circle, the coordinates of the center of the second feature circle are calculated.

[0144] The recording module is used to obtain the homography matrix based on the correspondence between the center coordinates of the first feature circle and the center coordinates of the second feature circle, and to record the distortion parameters and homography matrix to obtain the calibration data of the visible light lens and the fluorescent lens.

[0145] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, is used to implement the methods provided in the various embodiments described above.

[0146] The storage medium can be a computer storage medium or a communication medium. A communication medium includes any medium that facilitates the transfer of computer programs from one location to another. A computer storage medium can be any available medium accessible to a general-purpose or special-purpose computer. For example, the storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be a component of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). This ASIC can also be located within a user device. Alternatively, the processor and storage medium can exist as discrete components in a communication device. Storage media can be read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices, etc.

[0147] The present invention also provides a program product including execution instructions stored in a storage medium. At least one processor of the device can read the execution instructions from the storage medium, and the execution instructions by the at least one processor cause the device to implement the methods provided in the various embodiments described above.

[0148] In the above-described terminal or server embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A calibration method for a fluorescence endoscope, characterized in that, include: The fluorescence endoscope is controlled to acquire visible light and fluorescence images of the pre-made calibration plate image according to the preset acquisition method, so as to obtain visible light channel calibration image and fluorescence image calibration image. The calibration plate image includes a square pattern for visible light color development and a circular pattern for fluorescence color development. The square pattern and the circular pattern are distributed in a checkerboard pattern within the calibration plate image. Using the checkerboard corner coordinates in the visible light channel calibration image as input, and processing them according to Zhang Zhengyou's calibration method, the distortion parameters of the visible light lens of the fluorescence endoscope are obtained. Based on the distortion parameters and the checkerboard corner coordinates, the coordinates of the center of the first feature circle are calculated, including: The visible calibration image is processed to extract the coordinates of the calibration board's checkerboard grid. Based on the checkerboard grid specifications and the world coordinate system origin, the physical coordinate values ​​of the checkerboard grid corner points in the calibration image are calculated, including: The corner points of the chessboard are detected based on the SURF feature point detection algorithm, and non-maximum suppression is used to locate the detected corner points at the primitive level to obtain the coordinates of the chessboard corner points. Based on the detected checkerboard corner coordinates and the known corner distribution, the undetected image corner positions are filled in, and the sub-pixel coordinates of each corner are obtained by fitting within the 8-neighborhood range as the physical coordinate values ​​of the corresponding checkerboard corner. The step of filling in the positions of undetected image corner points based on the detected checkerboard corner coordinates and known corner point distribution includes: The number of corner points in the same row of the chessboard with the same horizontal coordinate is counted to obtain the first number of corner points. The first preset number corresponding to each same horizontal coordinate under the known corner point distribution is determined. If the first number of corner points corresponds to the first preset number, it is determined that the corner point coordinates of the corresponding row of the chessboard have been detected. If the number of the first corner points does not correspond to the first preset number, the coordinates of the corner points of the chessboard corresponding to the corresponding row will be used as the coordinates of the first corner point, and the coordinates of the corner points of the chessboard in other rows will be used as the coordinates of the second corner point. Compare the ordinate of the first corner point with the ordinate of the second corner point, and determine the ordinates that exist in the second corner point but not in the first corner point as the ordinates to be filled. The horizontal coordinates of the corresponding rows are filled in with the vertical coordinates of each row to be filled in, so as to obtain the coordinates of the undetected image corner points. The coordinates of the center of the feature circle in the fluorescence image calibration image are used as input and processed according to Zhang Zhengyou calibration method to obtain the distortion parameters of the fluorescence lens of the fluorescence endoscope. Based on the distortion parameters and the coordinates of the center of the feature circle, the coordinates of the center of the second feature circle are calculated. The homography matrix is ​​obtained based on the correspondence between the center coordinates of the first and second feature circles. The calibration data of the visible light lens and the fluorescent lens are then recorded based on the distortion parameters and the homography matrix.

2. The calibration method for a fluorescence endoscope according to claim 1, characterized in that, The calibration board image is generated through the following steps: Generate a feature circle image, which is an array of multiple circular patterns of the same size, with adjacent circular patterns distributed at a preset distance; A checkerboard grid image is generated based on the feature circle image. The checkerboard grid image includes black and white checkerboard grids that are evenly distributed. The circular pattern is located within the white grids of the checkerboard grid image. The circular pattern in the feature circle image is printed with infrared ink, and the black squares in the checkerboard grid image are printed with white ink to obtain the calibration board image.

3. The calibration method for a fluorescence endoscope according to claim 2, characterized in that, The controlled fluorescence endoscope acquires visible light and fluorescence images from the pre-fabricated calibration plate image according to a preset acquisition method, obtaining visible light channel calibration images and fluorescence image calibration images, including: Place the calibration plate image under the imaging field of the fluorescence endoscope; By changing the relative angle between the calibration plate and the endoscope and moving the calibration plate multiple times to acquire images, visible light channel calibration images and fluorescence image calibration images from multiple angles are obtained.

4. The calibration method for a fluorescence endoscope according to claim 1, characterized in that, The process of taking the coordinates of the checkerboard corner points in the visible light channel calibration image as input and processing them based on the Zhang Zhengyou calibration method to obtain the distortion parameters of the visible light lens of the fluorescence endoscope, and calculating the coordinates of the center of the first feature circle based on the distortion parameters and the coordinates of the checkerboard corner points, includes: taking the physical coordinates of the checkerboard corner points as input and calculating the distortion parameters of the visible light lens based on the Zhang Zhengyou calibration method. The corner coordinates of the chessboard are corrected based on the distortion parameters to obtain the processed corner coordinates. The coordinates of the center of the first feature circle are calculated based on the corner coordinates and the offset from the corner to the center.

5. The calibration method for a fluorescence endoscope according to claim 4, characterized in that, The process of correcting the distortion of the chessboard corner coordinates based on distortion parameters to obtain the processed chessboard corner coordinates, and then calculating the coordinates of the center of the first feature circle based on the chessboard corner coordinates and the offset from the corner to the center of the circle, includes: Determine the circular pattern within the feature circle image corresponding to the coordinates of the corner point of the chessboard grid, with each corner point of the chessboard grid having a corresponding circular pattern; The coordinates of the center of the first feature circle of the corresponding circular pattern can be calculated using the following formula. in, The coordinates of the center of the first feature circle are: The coordinates of the corner points of the chessboard. This represents the offset from the corner point to the center of the circle.

6. The calibration method for a fluorescence endoscope according to claim 5, characterized in that, The process of using the coordinates of the center of the feature circle in the fluorescence image calibration image as input and processing it based on the Zhang Zhengyou calibration method to obtain the distortion parameters of the fluorescence lens of the fluorescence endoscope, and calculating the coordinates of the center of the second feature circle based on the distortion parameters and the coordinates of the center of the feature circle, includes: The Hough circle detection algorithm is used to identify the center of circular patterns in fluorescence image calibration images and obtain the coordinates of the center of the feature circle. Based on the center coordinates of the detected feature circles and the preset circle distribution pattern, the undetected feature circles are determined, and the center coordinates of the unseen feature circles are filled in. Based on the known distribution of feature circles and the origin of the world coordinate system, the physical coordinates of the centers of the feature circles in the calibration image are calculated. The step of determining undetected feature circles based on the center coordinates of detected feature circles and a preset circle distribution pattern, and then filling in the center coordinates of the undetected feature circles, includes: The number of feature circles with the same horizontal coordinate in the same row is counted to obtain the first number of feature circles. The second preset number corresponding to each same horizontal coordinate under the preset circle distribution pattern is determined. If the first number of feature circles and the second preset number correspond, it is determined that the center coordinates of all feature circles corresponding to the corresponding row have been detected. If the number of the first feature circles does not correspond to the number of the second preset number, the center coordinates of the feature circles corresponding to the corresponding row are used as the first center coordinates, and the center coordinates of the feature circles in other rows are used as the second center coordinates. Compare the ordinate value of the first center coordinate with the ordinate value of the second center coordinate, and determine the ordinate values ​​that exist in the second center coordinate but not in the first center coordinate as the ordinate values ​​to be filled in. The horizontal coordinates of the corresponding rows are filled in with the vertical coordinates of each undetected feature circle to obtain the coordinates of the feature circle center.

7. The calibration method for a fluorescence endoscope according to claim 6, characterized in that, The homography matrix is ​​obtained based on the correspondence between the coordinates of the center of the first feature circle and the coordinates of the center of the second feature circle. The calibration data for the visible light lens and the fluorescence lens are recorded using the distortion parameters and the homography matrix. This includes: The visible light channel calibration image and the fluorescence image calibration image at the same angle are taken as a set of images. The coordinates of the center of the first feature circle and the center of the second feature circle corresponding to the same set of images are determined. The homography matrix of the center coordinates of the first feature circle and the center of the second feature circle is calculated using the following formula. for These are the coordinates of the center of the second characteristic circle. The coordinates of the center of the first characteristic circle are... It is a 3x3 homography matrix; After determining the homography matrix of all groups of images, the mean is calculated to obtain the overall homography matrix. The distortion parameters and homography matrix are then recorded to obtain the calibration data for the visible light lens and the fluorescent lens.

8. The calibration method for a fluorescence endoscope according to claim 7, characterized in that, Also includes: After acquiring the fluorescence image and the white light image to be processed by the fluorescence endoscope, the distortion of the fluorescence image to be processed is corrected based on the distortion parameters of the fluorescence channel. Then, the homography transformation is performed according to the homography matrix to obtain the calibrated fluorescence image. After correcting the distortion of the fluorescence image to be processed based on the distortion parameters of the white light channel, a calibrated variable light image is obtained.

9. The calibration system for a fluorescence endoscope according to any one of claims 1-8, characterized in that, include: The control module is used to control the fluorescence endoscope to acquire visible light and fluorescence images of the pre-made calibration plate image according to the preset acquisition method, so as to obtain visible light channel calibration image and fluorescence image calibration image. The calibration plate image includes a square pattern for visible light color development and a circular pattern for fluorescence color development. The square pattern and the circular pattern are distributed in a checkerboard pattern within the calibration plate image. The visible light processing module is used to take the coordinates of the checkerboard corner points in the visible light channel calibration image as input and process them based on the Zhang Zhengyou calibration method to obtain the distortion parameters of the visible light lens of the fluorescence endoscope. Based on the distortion parameters and the coordinates of the checkerboard corner points, the coordinates of the center of the first feature circle are calculated. The fluorescence processing module is used to take the coordinates of the center of the feature circle in the fluorescence image calibration image as input and process it according to Zhang Zhengyou calibration method to obtain the distortion parameters of the fluorescence lens of the fluorescence endoscope. Based on the distortion parameters and the coordinates of the center of the feature circle, the coordinates of the center of the second feature circle are calculated. The recording module is used to obtain the homography matrix based on the correspondence between the center coordinates of the first feature circle and the center coordinates of the second feature circle, and to record the distortion parameters and homography matrix to obtain the calibration data of the visible light lens and the fluorescent lens.

Images