Method for compensating for switching error of microscope lens or camera and image analysis device
By using the combination of an imaging device, motion control mechanism and image processing module in the microscope imaging system, the mechanical error during switching of the microscope lens or camera is calculated and compensated, the imaging position offset problem is solved and the imaging quality is improved.
Patent Information
- Application Number
- CN202411297301.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-11-19
AI Technical Summary
In microscopic imaging systems, when switching objective lenses with different magnifications, the imaging position shifts due to mechanical errors, which affects the imaging quality. The switching error needs to be calculated and compensated to ensure the accuracy of the imaging system.
Using a system including an imaging device, a motion control mechanism and an image processing module, the samples are taken through the low-resolution imaging module, the target is determined, the motion mechanism is switched to the high-resolution imaging module, and the image processing module calculates the affine transformation matrix parameters to obtain the offset. Compensate the parameters and move the objective lens or slide through the motion control module to achieve the corresponding offset to complete the compensation of switching error.
Effectively analyze and compensate mechanical errors to ensure the precise alignment of the target position when switching between different magnifications, and improve imaging quality.
Smart Images

Figure CN118981105A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of morphological analysis, and in particular to a method for compensating for microscope lens or camera switching errors and an image analysis device. Background Art
[0002] In a microscopic imaging system, it is usually necessary to switch between objective lenses of different magnifications to improve shooting efficiency. Generally speaking, a low-resolution image is first taken to locate the target, and then the image is switched to a high-magnification objective lens to capture the details of the target.
[0003] This type of imaging system can be roughly divided into two categories. The first category: using the same camera, and achieving different magnification imaging by switching the camera's objective lens, for example, first taking a low-resolution image under a 10X objective lens to locate the target, and then switching to a 100X objective lens to take a high-resolution image to confirm the target details. The second category: using multiple cameras, each with a fixed objective lens, and achieving low-resolution positioning and high-resolution imaging by moving the slide to different cameras, that is, first moving to a low-resolution camera to shoot and locate the target, and then moving the slide to a high-resolution camera to shoot the details of the target.
[0004] Both of the above methods can achieve the switching of objective lenses of different magnifications, but during the switching process, a motion mechanism is required to move the objective lens or glass slide. The motion mechanism will inevitably have mechanical errors, so when switching from low-resolution to high-resolution shooting, the imaging position will often change, causing the position of the observed target to shift, affecting the final imaging quality. Therefore, in order to ensure that the position of the photographed target does not shift after the imaging system switches the camera or switches the lens, it is necessary to calculate the system's switching error and compensate for it. Summary of the invention
[0005] The purpose of the present invention is to provide a method and an image analysis device for compensating for microscope lens or camera switching errors, aiming to analyze mechanical errors and control the system to compensate for the errors to ensure the quality of the system.
[0006] To achieve the above-mentioned object, in a first aspect, the present invention provides a method and an image analysis device for compensating for switching errors of a microscope lens or a camera, comprising an imaging device, a motion control mechanism and an image processing module;
[0007] The imaging device is used to switch the lens to achieve high and low resolution imaging, and can also switch the shooting device to achieve high and low resolution switching;
[0008] The motion control mechanism is used to control the movement of the imaging device and the sample slide to switch to shooting targets with different resolutions, and at the same time, to move the objective lens or the slide by a corresponding offset to complete the compensation of the switching error;
[0009] The image processing module is used to match the positions of the imaging images, calculate the affine transformation matrix parameters between the images, and obtain the offset compensation parameters between the images.
[0010] Wherein, the imaging device includes a low-resolution imaging module and a high-resolution imaging module.
[0011] In a second aspect, the present invention further provides a method for compensating for a microscope lens or camera switching error, which is applied to the image analysis device for compensating for a microscope lens or camera switching error as described in the first aspect above, and is characterized in that it comprises the following steps:
[0012] The low-resolution imaging module of the imaging device takes a low-resolution picture of the sample slide, determines the target cells, and generates a first image;
[0013] The motion mechanism switches to a high-resolution imaging module to capture a high-resolution image of the sample slide to generate a second image;
[0014] The image processing module performs position matching on the first image and the second image, calculates affine transformation matrix parameters between the two images, and obtains offset compensation parameters;
[0015] The offset compensation parameter is converted into the moving distance of the objective lens or the glass slide, and the motion mechanism is controlled by the control module to move the objective lens or the glass slide by the corresponding offset, thereby completing the compensation of the switching error.
[0016] The sample slide is prepared from any sample of blood, body fluid, tissue fluid, urine, quality control material and microparticles.
[0017] The position matching adopts a feature point matching algorithm, such as a SIFT algorithm, or a method based on point set registration, such as a KC registration algorithm.
[0018] The image analysis device for compensating the switching error of a microscope lens or a camera of the present invention comprises the following steps: the imaging device is used for switching the lens to realize high-low resolution imaging, and the shooting device can also be switched to realize high-low resolution switching; the motion control mechanism controls the movement of the imaging device and the sample glass slide to switch to shooting targets with different resolutions, and at the same time, the objective lens or the glass slide is moved by a corresponding offset to complete the compensation of the switching error; the image processing module matches the positions of the imaging images, calculates the affine transformation matrix parameters between the images, and obtains the offset compensation parameters between the images; the low-resolution imaging module of the imaging device shoots a low-resolution picture of the sample glass slide, determines the target cells, and generates a first image; the motion mechanism switches to the high-resolution imaging module to shoot a high-resolution image of the sample glass slide to generate a second image; the image processing module performs position matching on the first image and the second image, calculates the affine transformation matrix parameters between the two images, and obtains the offset compensation parameters; the offset compensation parameters are converted into the moving distance of the objective lens or the glass slide, and the motion mechanism is controlled to operate by the control module to move the objective lens or the glass slide by a corresponding offset to complete the compensation of the switching error; the image analysis device can analyze the mechanical error, and the control system compensates for the error to ensure the quality of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0020] Figure 1 It is a schematic diagram of an image analysis device for compensating for microscope lens or camera switching errors provided by the present invention.
[0021] Figure 2 It is a schematic diagram of a type A imaging device provided by the present invention.
[0022] Figure 3 It is a schematic diagram of a Class B imaging device provided by the present invention.
[0023] Figure 4 It is a flow chart of the method for compensating microscope lens or camera switching error provided by the present invention.
[0024] Figure 5 It is a schematic diagram of capturing a low-resolution image and processing it to obtain a first image.
[0025] Figure 6 It is a P1 distribution diagram drawn with the key point as the center and radius 2.
[0026] Figure 7Schematic diagram of obtaining a second image by high-resolution image processing.
[0027] Figure 8 It is a P2 distribution diagram drawn with the key point as the center.
[0028] Fig. 9 It is a schematic diagram of the affine transformation matrix output by point set registration.
[0029] Fig.10 It is a schematic diagram of calculating the reprojection error by mapping P2 to the P1 coordinate system according to the affine transformation matrix.
[0030] Fig.11 It is a schematic diagram of calculating the switching error based on the centers of the first and second images.
[0031] Fig.12 This is a schematic diagram showing that the white blood cells in the next round of 100X images captured after compensation are corrected to the center of the image.
[0032] Fig.13 This is a schematic diagram of the SIFT algorithm matching results.
[0033] Fig.14 It is a step diagram of the method for compensating microscope lens or camera switching error provided by the present invention.
[0034] In the figure: 1-imaging device, 2-motion control mechanism, 3-image processing module, 4-low-resolution imaging module, 5-high-resolution imaging module. DETAILED DESCRIPTION
[0035] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0036] See also Figures 1 to 3 , In a first aspect, the present invention provides an image analysis device for compensating for microscope lens or camera switching errors, comprising an imaging device 1, a motion control mechanism 2 and an image processing module 3;
[0037] The imaging device 1 is used to switch the lens to achieve high and low resolution imaging, and can also switch the shooting device to achieve high and low resolution switching;
[0038] The motion control mechanism 2 is used to control the movement of the imaging device 1 and the sample slide to switch to shooting targets with different resolutions, and at the same time, to move the objective lens or the slide by a corresponding offset to complete the compensation of the switching error;
[0039] The image processing module 3 is used to match the positions of the imaging images, calculate the affine transformation matrix parameters between the images, and obtain the offset compensation parameters between the images.
[0040] In the embodiment of the present invention, the imaging device 1 is used to switch the lens to realize high- and low-resolution imaging, and can also switch the shooting device to realize high- and low-resolution switching; the motion control mechanism 2 controls the movement of the imaging device 1 and the sample slide to switch to different resolution shooting targets, and at the same time, the objective lens or the slide moves the corresponding offset to complete the compensation of the switching error; the image processing module 3 matches the imaging image positions, calculates the affine transformation matrix parameters between the images, and obtains the offset compensation parameters between the images. The low-resolution imaging module 4 of the imaging device 1 takes a low-resolution picture of the sample slide, determines the target cells, and generates a first image; the motion mechanism switches to the high-resolution imaging module 5 to take a high-resolution image of the sample slide to generate a second image; the image processing module 3 performs position matching on the first image and the second image, calculates the affine transformation matrix parameters between the two images, and obtains the offset compensation parameters; the offset compensation parameters are converted into the moving distance of the objective lens or the slide, and the motion mechanism is controlled by the control module to operate, so that the objective lens or the slide moves the corresponding offset to complete the compensation of the switching error. The image analysis device can analyze the mechanical error, and the control system compensates for the error to ensure the quality of the system.
[0041] Furthermore, the imaging device 1 includes a low-resolution imaging module 4 and a high-resolution imaging module 5 .
[0042] In the embodiment of the present invention, the imaging device 1 includes two types, namely type A and type B.
[0043] Category A: The low-resolution imaging module 4 is a camera and a low-magnification objective lens, and the high-resolution imaging module 5 is a camera and a high-magnification objective lens. During operation, the low-magnification objective lens is first used to capture and locate the target, and the first image is obtained by processing. Then the motion mechanism moves the target into the field of view, switches to the high-magnification objective lens, captures and processes the second image. The structure of the device is as follows: Figure 2 shown.
[0044] Category B: The low-resolution imaging module 4 is a first camera and a low-magnification objective lens, and the high-resolution imaging module 5 is a second camera and a high-magnification objective lens. When imaging, first move to the low-resolution imaging module 4 to shoot and locate the target, and process to obtain the first picture. Then, the target is moved to the high-resolution imaging module 5 through the motion mechanism to shoot and process to obtain the second picture. Its structure is as follows: Figure 3 shown.
[0045] See also Figures 4 to 14In a second aspect, the present invention further provides a method for compensating for a microscope lens or camera switching error, which is applied to the image analysis device for compensating for a microscope lens or camera switching error as described in the first aspect above, and is characterized in that it comprises the following steps:
[0046] S1: The low-resolution imaging module 4 of the imaging device 1 takes a low-resolution picture of the sample slide, determines the target cells, and generates a first image;
[0047] In the embodiment of the present invention, the low-resolution imaging module 4 of the imaging device 1 captures and acquires the first image, the image processing module 3 analyzes the first image, extracts the target position in the image, and takes a ROI according to the target position. The key points in the ROI are calculated, which are called the key points of the first image.
[0048] The size of the ROI can be set according to the actual situation. Generally speaking, the maximum switching error of the imaging system and the field of view size of the second image in the first image can be estimated in advance. The size of the ROI should be set larger than the sum of the field of view size and the maximum switching error to ensure that the extracted key points can cover the target position.
[0049] The sample slide is prepared from any sample of blood, body fluid, tissue fluid, urine, quality control material and microparticles.
[0050] S2 The motion mechanism switches to the high-resolution imaging module 5 to capture a high-resolution image of the sample slide to generate a second image;
[0051] In the embodiment of the present invention, according to the target position extracted from the first image, the motion mechanism is controlled to switch to the high-resolution imaging module 5, the second image of the sample is shot at the target, and the key points in the second image are calculated, which are called the key points of the second image.
[0052] The key points of the second image are extracted using the same method as the first image without limitation. When extracting the key points, the high-resolution second image can be scaled to the same magnification as the first image based on the objective lens magnification relationship between the two shots to reduce the amount of calculation. The key points can be a set of target center points, a set of target contour points, a set of SIFT corner points, etc.
[0053] S3 image processing module 3 performs position matching on the first image and the second image, calculates the affine transformation matrix parameters between the two images, and obtains the offset compensation parameters;
[0054] In an embodiment of the present invention, in the image processing module 3, image matching is performed through the key points of the first image and the key points of the second image, the affine transformation matrix between the two images is calculated, the key points in the second image are mapped to the coordinate system of the first image, the distance error between the mapping point and the corresponding point in the first image is calculated as the reprojection error, and whether the matrix parameters are calculated correctly is determined according to the set threshold.
[0055] If the reprojection error is less than the threshold, the center point of the second image is mapped to the coordinate system of the first image, and the offset between the mapped center point and the center point of the first image is calculated, which is the switching error of the system.
[0056] Image key point matching can use feature point matching algorithms such as SIFT algorithm, or point set registration-based methods such as KC registration algorithm. The reprojection error is the error between the position of the key point in the second image that is successfully matched and the actual position after transformation in the first image. The reprojection error can be calculated to verify whether the affine transformation matrix is calculated correctly. Other parameter thresholds can also be set according to the characteristics of the system hardware. For example, if the rotation angle between the first image and the second image is known to be no more than 2 degrees, it can be verified whether the rotation angle of the rotation matrix is less than 2 degrees. The key points used to calculate the affine transformation matrix and the key points used to calculate the reprojection error can be the same group of key points or different key points without restriction. For example, the red blood cell center point can be used as the key point for registration, and the white blood cell center point can be used as the key point to calculate the reprojection error. Alternatively, 80% of the key points can be extracted for registration, and the remaining 20% of the key points can be used to calculate the reprojection error.
[0057] S4 converts the offset compensation parameter into a moving distance of the objective lens or the glass slide, controls the motion mechanism to operate through the control module, and moves the objective lens or the glass slide by a corresponding offset to complete the compensation of the switching error.
[0058] In order to better understand the present technical solution, the following embodiments are provided for further explanation:
[0059] Example 1
[0060] Take the A-type imaging device 1 as an example
[0061] The imaging device 1 is of Class A, the low-resolution imaging module 4 is a 10X objective lens, the high-resolution imaging module 5 is a 100X objective lens, the camera resolution is 1920 x 1200 pixels, the motion control mechanism 2 is a plurality of stepper motors for driving lens switching, up and down movement of the lens barrel, and movement of the slide platform, the image processing module 3 is a PC, and the sample slide is a venous blood slide.
[0062] In this example, the maximum error caused by the motion mechanism when the system switches resolution is 28um, which is converted to an error of 36 pixels in the first image. To ensure reliability, the error is multiplied by 1.5 in actual use. Since the objective lens magnifications of the first and second images differ by 10 times, the field of view size of the second image in the first image is 192x 120 pixels. Therefore, in this implementation, the size of the ROI is set to (192+36x 1.5)x(120+36x 1.5) pixels, that is, 288x 180 pixels, which is used to capture the first image from the low-resolution image.
[0063] In this embodiment, there is no rotation transformation between the first and second images.
[0064] In the first image, cells are detected by an artificial intelligence algorithm, and the center points of red blood cells with a confidence level greater than 0.9 are taken as the key point set P1 of the first image, and the center coordinates of white blood cells are taken for later use in reprojection error calculation.
[0065] In the implementation process, firstly, a positioning image of the blood sample is taken through a 10X objective lens, and the white blood cells in the 10x image are detected through the artificial intelligence algorithm in the image processing module 3 in the PC, the target white blood cells are determined, and a 288x 180 pixel ROI is taken on the 10x image with the center point of the white blood cells as the center as the first image.
[0066] The motion control mechanism 2 controls the objective lens to rotate to the 100X objective lens, and the stepper motor moves the slide to aim the lens at the target leukocyte, and takes a high-definition image of the target leukocyte. It can be seen from the image that the leukocyte is not located in the center of the image at this time, indicating that a switching error exists.
[0067] The high-definition image is reduced by 10 times to obtain a second image of 192 x 120 pixels. All cells in the second image are detected by an artificial intelligence algorithm, and the center points of the red blood cells with a confidence level greater than 0.9 are taken as the key point set P2 of the second image.
[0068] The KCREG point set matching algorithm is used to align P1 and P2, and the affine transformation matrix between the two images is calculated.
[0069] Apply the affine transformation matrix to the coordinates of the center point of the white blood cells in the second image to obtain the coordinates of the white blood cells after being mapped to the first image. Calculate the Euclidean distance between the coordinates and the center point of the white blood cells in the first image, which is the reprojection error of this calculation. In this example, the reprojection error is 1.813 pixels, and the reprojection error threshold is set to 3 pixels. Therefore, it is considered that this calculation is successful and the switching error is acceptable. If the reprojection error exceeds the threshold, the calculation fails and the system does not accept the switching error calculated this time. Use the affine transformation matrix to map the image center of the second image to the first image to obtain coordinate C2, and calculate the center coordinate C1 of the first image. Calculate the offset from C2 to C1, which is the switching error of the system. In this embodiment, the coordinates of C2 are (182.24335, 149.47089), and C1 are (159, 124), so the switching error is (-23.24, -25.47) pixels, which is converted to (-18.07, -19.81) um in the physical coordinate system of the motion mechanism, indicating that when switching from the first image to the second image, in order to compensate for the switching error, the objective lens needs to move 18.07 um to the left and 19.81 um upward.
[0070] Fig.11 As shown in the figure, the arrow is C2 pointing to C1. Finally, the compensation parameters are converted into the number of steps of the stepper motor. In the next round of shooting, the stepper motor is automatically controlled to move the objective lens 18.07um to the left and 19.81um upward, so that the target cell being photographed is located in the center of the 100X image, completing the compensation of the switching error. Fig.12 After compensation, the cells obtained in the next round of shooting are located in the center of the image.
[0071] Example 2
[0072] Take the B type imaging device 1 as an example
[0073] In another example, the imaging device 1 is Class B, the low-resolution imaging module 4 is a 10X objective lens and a 1.3 million camera, the high-resolution imaging module 5 is a 60X objective lens and a 5 million camera, the motion control mechanism 2 is a plurality of stepper motors used to drive the movement of the slide platform to achieve focusing and shooting switching, the image processing module 3 is a PC, and the sample slide is a urine sample.
[0074] In this embodiment, since the maximum error when the system switches resolution is not accurately measured and there is a ±1° rotation error between the first image and the second image due to camera installation accuracy issues, the ROI size of the first image is set slightly larger, 320 x 320 pixels.
[0075] During the implementation process, the glass slide is first moved to the first camera and an image of the urine sample is captured through a 10X objective lens. Formed elements are identified on the image to determine the target cells. A 320x 320 pixel ROI is taken with the target cells as the center as the first image.
[0076] The motion control mechanism 2 controls the objective lens to rotate to the 60X objective lens, and at the same time the stepper motor moves the glass slide to the second camera so that the second camera lens is aimed at the target cell, and a high-definition image of the target cell is taken, which is used as the second image.
[0077] The SIFT algorithm is used to extract the key points P1 of the first image and the key points P2 of the second image and complete the key point matching, and then calculate the affine transformation matrix between the two images.
[0078] Use this matrix to map all successfully paired key points in the second image to the first image, calculate the reprojection error, and determine whether it is less than the threshold. In this example, the reprojection error is 2.513 pixels, and the reprojection error threshold is set to 3 pixels, so this calculation is successful.
[0079] The subsequent steps are similar to the previous implementation, that is, the specific switching error is calculated according to the affine transformation matrix. The difference from the previous implementation is that in addition to the translation error, the rotation error value in the affine transformation matrix needs to be processed in this implementation, but the principle is basically the same and will not be repeated here. After the calculated pixel error is converted into the moving distance of the objective lens, the stepper motor can be controlled to move the objective lens by the corresponding distance to complete the compensation of the switching error.
[0080] The above disclosure is only a preferred embodiment of the method for compensating microscope lens or camera switching errors and the image analysis device of the present invention. Of course, this cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.
Claims
1. An image analysis device for compensating for microscope lens or camera switching errors, characterized in that: It includes an imaging device, a motion control mechanism and an image processing module; The imaging device is used to switch the lens to achieve high and low resolution imaging, and can also switch the shooting device to achieve high and low resolution switching; The motion control mechanism is used to control the movement of the imaging device and the sample slide to switch to shooting targets with different resolutions, and at the same time, to move the objective lens or the slide by a corresponding offset to complete the compensation of the switching error; The image processing module is used to match the positions of the imaging images, calculate the affine transformation matrix parameters between the images, and obtain the offset compensation parameters between the images.
2. The image analysis device for compensating for microscope lens or camera switching errors according to claim 1, characterized in that: The imaging device comprises a low-resolution imaging module and a high-resolution imaging module.
3. A method for compensating for microscope lens or camera switching errors, applied to the image analysis device for compensating for microscope lens or camera switching errors as claimed in any one of claims 1 to 2, characterized in that: The following steps are involved: The low-resolution imaging module of the imaging device takes a low-resolution picture of the sample slide, determines the target cells, and generates a first image; The motion mechanism switches to a high-resolution imaging module to capture a high-resolution image of the sample slide to generate a second image; The image processing module performs position matching on the first image and the second image, calculates affine transformation matrix parameters between the two images, and obtains offset compensation parameters; The offset compensation parameter is converted into the moving distance of the objective lens or the glass slide, and the motion mechanism is controlled by the control module to move the objective lens or the glass slide by the corresponding offset, thereby completing the compensation of the switching error.
4. The method for compensating microscope lens or camera switching error according to claim 3, characterized in that: The sample slide is prepared from any sample of blood, body fluid, tissue fluid, urine, quality control material and microparticles.
5. The method and image analysis device for compensating microscope lens or camera switching errors according to claim 3, characterized in that: The position matching adopts a feature point matching algorithm, such as a SIFT algorithm, or a method based on point set registration, such as a KC registration algorithm.
