Calibration system and method based on monocular structured light three-dimensional medical endoscope

Abstract

The invention discloses a calibration system and method based on a monocular structured light three-dimensional medical endoscope. The calibration system comprises equipment for monocular structured light system calibration and a system calibration method. The calibration device comprises a diffuse reflection white calibration plate, a circular calibration plate and a calibration plate fixing device. The calibration method comprises a world coordinate algorithm for reversely resolving the structured light feature points through calibration data of a camera channel and a camera-projection channel joint calibration algorithm. Structured light is projected to the diffuse reflection white calibration plate and the circular calibration plate at fixed positions, and a calibration algorithm is matched to realize accurate calibration of the system, so that accurate three-dimensional reconstruction is realized.

Description

Technical Field

[0001] This invention belongs to the field of three-dimensional endoscopic imaging measurement, and specifically relates to a calibration system and method for a monocular structured light three-dimensional medical endoscope suitable for precise and rapid on-site calibration. Background Technology

[0002] Three-dimensional medical endoscopy based on monocular structured light is an important research direction in three-dimensional endoscopic imaging. It obtains accurate three-dimensional information of the object being measured by projecting structured light onto the target tissue. Its advantage lies in its ability to quickly and accurately obtain important parameters such as the morphological features of the tissue surface and the size of lesions, providing effective information for clinicians to make rapid diagnoses. Currently, the main methods used in three-dimensional medical endoscopy include: monocular structured light endoscopy, binocular endoscopy, and time-of-flight endoscopy. Monocular structured light endoscopy has a significant advantage in measuring tissue surfaces with sparse texture and difficult matching.

[0003] CN118000644A discloses a medical endoscope three-dimensional imaging system and method based on WeChat structured light projection. The system is designed with an endoscope lens end containing a structured light projection channel. It is a three-dimensional medical endoscope based on monocular structured light. Its micro structured light source channel is used to project structured light patterns to complete the encoding of the object under test. Its structured light projection function-related structure includes at least three parts: a high-power LED, a micro lithography mask containing structured light patterns, and a micro projection lens. After the system is assembled, the monocular structured light system needs to be calibrated to determine the three-dimensional measurement accuracy of the system.

[0004] Currently, the calibration methods for monocular structured light systems mainly utilize projectors to project standard checkerboard or circular calibration images, and use global or local homography for calibration. However, for monocular structured light 3D medical endoscopes, after assembly, the projected pattern is no longer adjustable, and calibration must be performed using the system's own pattern. Therefore, traditional methods are no longer applicable to this system. Summary of the Invention

[0005] The present invention aims to propose a calibration system and method for a three-dimensional medical endoscope based on monocular structured light, which can achieve rapid and accurate calibration results for monocular structured light endoscopes.

[0006] The technical solution adopted in this invention is as follows:

[0007] A three-dimensional medical endoscope calibration system based on monocular structured light is disclosed. The endoscope has a structured light projection channel and a camera channel. The structured light projection channel is equipped with a light source, a miniature photolithographic mask containing a structured light pattern, and a miniature projection lens for projecting the structured light pattern. The system includes: a diffuse white calibration plate, a circular calibration plate, a calibration plate fixing device, and an endoscope fixing device. The endoscope fixing device is fixed on a platform to fix the endoscope lens end and keep its position stationary. The calibration plate fixing device is installed on the platform to fix the diffuse white calibration plate or the circular calibration plate, making its surface perpendicular to the axis of the endoscope lens end, and enabling it to move along or rotate around the axis to form a central angle.

[0008] In the above technical solution, further, the surfaces of the diffuse white calibration plate and the circular calibration plate are both diffuse reflective surfaces, with consistent surface flatness that is less than the testing accuracy of the endoscope, and consistent calibration plate thickness that is within the range of 1-3mm; the circular calibration plate contains m rows and n columns of circular feature points, where m×n≥20, the side length of the circular calibration plate is between 10-15cm, and the diameter of the circular feature points is between 1-2mm.

[0009] The calibration method for the endoscope using the calibration system includes the following:

[0010] First, keep the endoscope in a fixed position and complete the image acquisition for a single position. Then, move the calibration plate fixing device within a range of ±15° with the endoscope at a central angle of 10-50mm from the endoscope tip to achieve the acquisition of ≥25 sets of images at different positions. Each set includes images of the circular calibration plate under white light, images of the circular calibration plate under structured light, and images of the diffuse white calibration plate under structured light.

[0011] The Zhang Zhengyou calibration method is used to solve the image set of the circular calibration plate under white light, obtain the intrinsic and extrinsic parameters and distortion parameters of the camera channel, and complete the steps of image set correction and back projection of structured light feature points onto the two calibration plates.

[0012] Valid image pairs are selected by image difference thresholding, back-projected feature points and discretized feature points are matched, and the intrinsic and extrinsic parameters of the projection channel and the positional relationship between the camera channel and the projection channel are calculated to complete the calibration.

[0013] In the above technical solution, the structured light feature point back projection specifically means that, in the same set of images at the same position, since the shooting environment and parameters are completely consistent, the feature point coordinates of the diffuse white calibration plate image under structured light are used as the structured light feature points on the circular calibration plate image under structured light.

[0014] Furthermore, the step of filtering valid image pairs through image difference thresholding specifically involves: performing difference processing on the circular calibration board image under structured light and the diffuse white calibration board image under structured light in each group; comparing the difference value of the area outside the feature points of the circular calibration board with the preset threshold to verify the positional consistency of the calibration board before and after switching; and filtering out calibration images that meet the requirements. Only when the difference value of the area outside the feature points of the circular calibration board is less than the preset threshold is the circular calibration board image under structured light and the diffuse white calibration board image under structured light considered as a valid image pair. Based on the back projection of the structured light feature points, the coordinates (u) of the structured light feature points on the circular calibration board image under structured light are obtained. s ,v s ).

[0015] Furthermore, extract structured light feature points (u s ,v s ) and the characteristic points of the circular calibration plate under white light (u c ,v c The world coordinates of the circular calibration plate are known to be (X). w ,Y w Based on the planar constraints of the calibration plate, the world coordinates of each structured light are obtained as follows:

[0016]

[0017] This yields the coordinates of the structured light feature points in the world coordinate system of the projection channel.

[0018] Furthermore, the micro-lithographic mask containing the structured light pattern in the endoscope is discretized, and its discretization degree is controlled to be consistent with the number of pixels on one side of the imaging channel, to obtain a coordinate-form mask image, and the coordinates (u) of each feature point in the discretized mask image are obtained. p ,v p This gives the coordinates of the feature points in the pixel coordinate system of the projection channel.

[0019] Furthermore, the matching of back-projected feature points and discretized feature points specifically involves: matching the coordinates of structured light feature points in the world coordinate system of the projection channel with the coordinates of feature points in the pixel coordinate system of the projection channel to establish the correspondence between the pixel coordinate system and the world coordinate system of the projection channel; calibrating the projection channel using the Zhang Zhengyou calibration method to obtain the intrinsic parameter matrix M of the projection channel. p Distortion coefficient D p And the rotation matrix R of the projection lens at each position p Translation vector t p .

[0020] Furthermore, by using the rotation matrices and translation vectors of the calibrated projection channel and camera channel, the positional relationship between the two channels is calculated, thereby completing the overall calibration of the system.

[0021] The beneficial effects of this invention are:

[0022] (1) In view of the limitation that the projection pattern of the endoscope based on monocular structured light cannot be changed, the present invention designs a simple, fast and accurate calibration system and method. The system applicable to the present invention is simple and easy to implement. Through the fixing device of the calibration plate, the positions of the two calibration plates with different patterns are the same before and after switching. By using planar constraints, the transformation of the feature points of the structured light itself from the pixel coordinate system of the camera to the world coordinate system is realized.

[0023] (2) This method solves the problem that the projection channel cannot observe feature points and realizes the self-calibration process of the system. Based on its own feature points, it completes the calibration work. The algorithm part is based on the existing mature Zhang Zhengyou calibration method and is improved. After establishing the correspondence of feature points, the calibration results can be obtained quickly and conveniently.

[0024] (3) The system structure of the present invention is simple and easy to use. It can be extended to all structured light system calibration methods based on feature points, and is a universal calibration method. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the system operation of the present invention, which includes a diffuse white calibration plate, a circular calibration plate, a calibration plate fixing device, a monocular structured light endoscope, and an endoscope fixing device.

[0026] Figure 2 This is a schematic diagram of the endoscope body of the present invention, which includes a monocular structured light endoscope body, a structured light projection channel, and an imaging channel (also referred to as a camera channel);

[0027] Figure 3 Schematic diagrams of the circular calibration plate and the diffuse reflection calibration plate;

[0028] Figure 4 The calibration process and algorithm flow are described below.

[0029] Explanation of reference numerals in the attached figures:

[0030] 101-Endoscope lens end; 102-Locking device; 103-Calibration plate limiting screw; 104-Fixing screw; 105-Fixing clamp; 106-Calibration plate; 201-System lens body; 202-Structured light projection channel; 203-Camera channel; 301-Circular calibration plate; 302-Diffuse white calibration plate. Detailed Implementation

[0031] The present invention will be further described in detail below with reference to embodiments, so that those skilled in the art can implement it based on the description.

[0032] It should be understood that terms such as “having,” “comprising,” and “including” used herein do not exclude the presence or addition of one or more other elements or combinations thereof.

[0033] This embodiment provides a calibration system and method for a monocular structured light three-dimensional endoscope system. The system includes an endoscope body, a circular calibration plate, a diffuse white calibration plate, a calibration plate fixing device, and an endoscope fixing device. The calibration plate fixing device is as follows: Figure 1 As shown in Figures 103-106. The calibration plate is fixed to the optical platform through the screw holes 104 at the bottom of the calibration plate fixing device, and then the fixing clamp 105 is pushed by the two limiting screws 103 on its side. The calibration plate is clamped by the two fixing clamps at the front and rear, thereby fixing the calibration plate and ensuring that the positions of the two calibration plates are consistent before and after switching.

[0034] In a preferred embodiment, the endoscope fixation device is as follows: Figure 1 As shown in Figures 101-102, the endoscope body 101 is placed in the semi-circular groove of the fixing device and locked by the locking device 102 above, thus fixing its position.

[0035] The calibration plate fixing device can move or rotate the fixed calibration plate relative to the endoscope fixing device to obtain images at different positions.

[0036] The working principle of the system is as follows Figure 2 As shown, the endoscope body 201 includes a structured light projection channel 202 and a camera channel 203 at its head end. The structured light projection channel includes at least three parts: a high-power LED, a micro-lithography mask containing a structured light pattern, and a micro-projection lens, which respectively capture images of a circular calibration plate 204 and a diffuse white calibration plate 205 when the structured light is turned on or off.

[0037] Both the diffuse white calibration plate and the circular calibration plate have diffuse reflective surfaces. Their surface flatness is designed to be consistent and less than the testing accuracy of the endoscope; for example, in this embodiment, it can be set to 0.25 mm. The calibration plate thickness is consistent and within the range of 1-3 mm. The circular calibration plate contains m rows and n columns of circular feature points, where m×n≥20. The side length of the circular calibration plate is between 10-15 cm, and the diameter of the circular feature points is between 1-2 mm. In a preferred embodiment, schematic diagrams of the circular calibration plate 301 and the diffuse white calibration plate 302 are shown below. Figure 3 As shown, there are a total of 10×10 circular feature points. The diameter of the feature points on the circular calibration plate is 2mm.

[0038] In a preferred embodiment, 30 sets of images for calibration are captured, according to... Figure 4 The calibration process shown completes image acquisition. The specific steps are as follows:

[0039] Step 1: Fix the endoscope onto the endoscope fixing device. Secure the endoscope's position using the top locking device. Turn on the white light. Fix the calibration plate fixing device onto the optical platform using the bottom positioning screws. Then, push the fixing clamps using the two side limit screws to clamp the circular calibration plate between the two fixing clamps. Take a picture of the circular calibration plate and obtain image img1.

[0040] Step 2: Turn off the white light, turn on the structured light, keep the circular calibration plate stationary, and capture an image of the structured light projected onto the circular calibration plate, obtaining image img2.

[0041] Step 3: Keep the calibration plate fixing device still, remove the circular calibration plate by rotating the two limit screws, replace the circular calibration plate with a diffuse reflection calibration plate, repeat the operation of fixing the calibration plate in Step 1, keep the structured light on, and acquire image img3.

[0042] Step 4: Repeat steps 1 to 3. Move the calibration plate fixing device within a range of ±15° from the endoscope tip and within 10-50mm from the center until a sufficient set of calibration images (≥25 sets) from different positions is obtained, thus completing the acquisition of calibration images.

[0043] After acquiring a sufficient number of images, according to a specific embodiment of the present invention, the calibration method includes the following steps:

[0044] The first step is to extract the centroids of the feature points on the circular calibration board from an image captured by the camera (i.e., image img1). Then, a relationship is established between the pixel coordinates of the centroids and their coordinates in the world coordinate system to ensure accurate matching. After this, the intrinsic parameter matrix of the camera channels is obtained using Zhang Zhengyou's calibration method. and distortion D c And obtain the rotation matrix R of the endoscope at each position. c Translation vector t c The obtained parameters are then used to perform distortion correction on the captured images img2 and img3, thereby completing the camera calibration.

[0045] The second step is to accurately identify and locate all structured light feature points after projecting the structured light onto the circular calibration plate. At the same time, the shooting environment and parameters of images img2 and img3 at the same location are the same. Therefore, the coordinates of the feature points projected onto the circular calibration plate by the structured light are obtained by using image img3, which projects the structured light onto the diffuse white calibration plate.

[0046] The third step is to set the difference threshold of each image pair formed by images img2 and img3 to α. This value can usually be set based on experience. In this example, the difference threshold can be set to 5. After the two sets of images are differentially analyzed, if the gray value threshold outside the feature points is higher than α, they are considered not completely overlapping and need to be removed. Only when the difference values ​​of the areas outside the feature points of the circular calibration plate are all less than the threshold are they valid image pairs. Finally, image pairs that can be used for calibration are selected.

[0047] Fourth, for the obtained structured light feature point coordinates, use the following formula to calculate the corresponding coordinates in the world coordinate system of the projection channel from the normalized pixel coordinates:

[0048]

[0049] The fifth step is to discretize the micro-photolithography mask in the endoscope. In this example, since the resolution of the imaging channel is 600×600, the mask is discretized into a 600×600 image, and the feature point coordinates of the discretized image of the mask are obtained as the feature point coordinates in the pixel coordinate system of the projection channel.

[0050] The sixth step is to establish the relationship between the coordinates in the world coordinate system of the projection channel and the coordinates of the feature points in the discretized image of the projection channel, complete the feature point matching, and obtain the intrinsic parameter matrix M of the projection lens by referring to Zhang Zhengyou's calibration method using the matched world coordinate system coordinates and the pixel coordinates of the feature points. p Distortion coefficient D p And the rotation matrix R of the projection lens at each position p Translation vector t p .

[0051] Step 7: Using the calibrated rotation matrices and translation vectors of the projection channel and camera channel, calculate the positional relationship between the two channels. The specific calculation is as follows:

[0052]

[0053] t = t p -Rt c

[0054] Thus, the individual calibration of the projection channel and camera channel, as well as the joint calibration of the entire system, were completed. After calibration, the projection channel and camera channel were corrected separately. After correction, the reprojection error of both the projection channel and camera channel was less than 1 pixel. Using the calibrated system for 3D measurement, under different targets, within a working distance of 20-40mm, the depth direction measurement error was less than 5% of the working distance, achieving the target accuracy.

[0055] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details.

Claims

1. A three-dimensional medical endoscope calibration system based on monocular structured light, characterized in that, The endoscope has a structured light projection channel and a camera channel. The structured light projection channel is equipped with a light source, a micro-photolithography mask containing a structured light pattern, and a micro-projection lens for projecting the structured light pattern. The system includes: a diffuse white calibration plate, a circular calibration plate, a calibration plate fixing device, and an endoscope fixing device. The endoscope fixing device is fixed on the platform to fix the endoscope lens end and keep its position stationary. The calibration plate fixing device is installed on the platform to fix the diffuse white calibration plate or the circular calibration plate, making its surface perpendicular to the axis of the endoscope lens end, and enabling it to move along or rotate around the axis to form a central angle.

2. The three-dimensional medical endoscope calibration system based on monocular structured light according to claim 1, characterized in that, Both the diffuse white calibration plate and the circular calibration plate have diffuse reflective surfaces. The surface flatness is designed to be consistent and less than the testing accuracy of the endoscope. The calibration plate thickness is consistent and within the range of 1-3mm. The circular calibration plate contains m rows and n columns of circular feature points, where m×n≥20. The side length of the circular calibration plate is between 10-15cm, and the diameter of the circular feature points is between 1-2mm.

3. A method for calibrating an endoscope based on the calibration system as described in claim 1, characterized in that, Including the following: First, keep the endoscope in a fixed position and complete the image acquisition for a single position. Then, move the calibration plate fixing device within a range of ±15° with the endoscope at a central angle of 10-50mm from the endoscope tip to achieve the acquisition of ≥25 sets of images at different positions. Each set includes images of the circular calibration plate under white light, images of the circular calibration plate under structured light, and images of the diffuse white calibration plate under structured light. The Zhang Zhengyou calibration method is used to solve the image set of the circular calibration plate under white light, obtain the intrinsic and extrinsic parameters and distortion parameters of the camera channel, and complete the steps of image set correction and back projection of structured light feature points onto the two calibration plates. Valid image pairs are selected by image difference thresholding, back-projected feature points and discretized feature points are matched, and the intrinsic and extrinsic parameters of the projection channel and the positional relationship between the camera channel and the projection channel are calculated to complete the calibration.

4. The calibration method according to claim 3, characterized in that, The reverse projection of structured light feature points specifically involves using the coordinates of feature points in the diffuse white calibration plate image under structured light as the structured light feature points on the circular calibration plate image under structured light, since the shooting environment and parameters are completely identical in the same set of images.

5. The calibration method according to claim 3, characterized in that, The method of selecting valid image pairs through image difference thresholding specifically involves: performing difference processing on the circular calibration board image under structured light and the diffuse white calibration board image under structured light in each group; comparing the difference value of the area outside the feature points of the circular calibration board with a preset threshold to verify the positional consistency of the calibration board before and after switching; and selecting calibration images that meet the requirements. Only when the difference value of the area outside the feature points of the circular calibration board is less than the preset threshold is the circular calibration board image and the diffuse white calibration board image under structured light considered a valid image pair. Based on the back projection of the structured light feature points, the coordinates (u) of the structured light feature points on the circular calibration board image under structured light are obtained. s ,v s ).

6. The calibration method according to claim 5, characterized in that, Extracting structured light feature points (u s ,v s ) and the characteristic points of the circular calibration plate under white light (u c ,v c The world coordinates of the circular calibration plate are known to be (X). w ,Y w Based on the planar constraints of the calibration plate, the world coordinates of each structured light are obtained as follows: This yields the coordinates of the structured light feature points in the world coordinate system of the projection channel.

7. The calibration method according to claim 3, characterized in that, The micro-lithographic mask containing the structured light pattern in the endoscope is discretized, and its discretization degree is controlled to maintain consistency with the number of pixels on one side of the imaging channel, resulting in a coordinate-form mask image. The coordinates (u) of each feature point in the discretized mask image are then obtained. p ,v p This gives the coordinates of the feature points in the pixel coordinate system of the projection channel.

8. The calibration method according to claim 3, characterized in that, The matching of back-projected feature points and discretized feature points specifically involves: matching the coordinates of structured light feature points in the world coordinate system of the projection channel with the coordinates of feature points in the pixel coordinate system of the projection channel to establish the correspondence between the pixel coordinate system and the world coordinate system of the projection channel; calibrating the projection channel using the Zhang Zhengyou calibration method to obtain the intrinsic parameter matrix M of the projection channel. p Distortion coefficient D p And the rotation matrix R of the projection lens at each position p Translation vector t p .

9. The calibration method according to claim 3, characterized in that, By using the rotation matrices and translation vectors of the calibrated projection and camera channels, the positional relationship between the two channels is calculated, thereby completing the overall calibration of the system.

Images