Lensless imaging-based full-slide digital image automatic focusing device and method

Abstract

The invention discloses a full-slide digital image automatic focusing device and method based on lensless imaging, and the method comprises the steps: combining a lensless imaging system with a microscopic imaging system, and calculating a linear relation between an axial position and a pixel offset distance in advance through the lensless imaging system; according to the full-slide digital image automatic focusing method, the full-slide digital image is subjected to linear relation obtaining, then the microscopic imaging system completes automatic focusing of the full-slide digital image according to the obtained linear relation, the full-region focusing position of the sample can be obtained only through two times of exposure, and compared with a traditional full-slide automatic focusing method, z-stack scanning does not need to be carried out in each region, the automatic focusing speed is greatly increased, and the automatic focusing accuracy is improved. And the time cost is saved.

Description

Technical Field

[0001] The present invention belongs to the technical field of automatic focusing of microscopes, and particularly relates to an automatic focusing device and method for whole-slide digital images based on lensless imaging. Background Art

[0002] The whole-slide microscopy imaging system is an integrated software and hardware system that converts samples into digital images, and can realize functions such as large-capacity image storage and transmission, navigation and zooming of virtual pathological images, and remote intelligent analysis and diagnosis of diseases. It uses an electric displacement platform to mechanically scan all regions of tissue sections and forms a whole digitalized section through image stitching. It is the hardware foundation of modern digital pathology and provides important support and assistance for pathological diagnosis and medical research. Traditional whole-slide systems acquire images through high-magnification objective lenses, which results in a small depth of field, causing image defocus and blurring. In order to obtain clear and high-quality images, fine focusing needs to be repeated. The automatic focusing technology is an important factor affecting the scanning efficiency and focusing accuracy of microscopes, and can directly affect the section scanning speed and the clarity of section images. Therefore, the automatic focusing technology has become the core technology of the whole-slide microscopy imaging system.

[0003] As the most widely used automatic focusing technology at present, the focus map measurement method obtains a z-axis image stack by scanning a sub-region of the sample along the z-axis, and estimates the in-focus position based on the contrast maximization of the stack images or other image quality evaluation methods. However, this method requires z-stack scanning for each sequentially scanned sub-region, and a large number of images need to be acquired and repeated z-axis measurements are required, resulting in a significant slowdown in the imaging speed. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an automatic focusing device and method for whole-slide digital images based on lensless imaging that can greatly improve the imaging speed.

[0005] The technical solution adopted by the present invention to solve the above technical problems is as follows: An automatic focusing device for whole-slide digital images based on lensless imaging is composed of a lensless imaging system and a microscopy imaging system. The lensless imaging system is used to obtain the linear relationship between the axial position of the standard sample and the pixel offset distance, divide the sample to be measured into multiple sub-regions with equal sizes according to the field of view of the microscopy imaging system, and then obtain the focus map of the sample to be measured according to the obtained linear relationship. The microscopy imaging system is used to find the in-focus position of the sub-region of the sample to be measured observed for the first time, and complete the automatic focusing of the whole-slide digital image in combination with the focus map of the sample to be measured.

[0006] Compared with the prior art, the advantages of the present invention lie in combining a lensless imaging system with a microscopic imaging system. The linear relationship between the axial position and the pixel offset distance pre-calculated by the lensless imaging system is then used by the microscopic imaging system to complete the autofocus of the whole-slide digital image. Only two exposures are required to obtain the in-focus positions of the entire sample area. Compared with the traditional whole-slide autofocus method, the method of the present invention does not require z-stack scanning in each area, greatly improving the autofocus speed and saving time costs.

[0007] Preferably, the lensless imaging system includes a first image sensor, a first sample stage for placing a sample, a first coherent light source, and a second coherent light source arranged in sequence. The first image sensor is located above the first sample stage and can move up and down in the vertical direction to perform axial scanning on the sample. The first coherent light source and the second coherent light source are symmetrically arranged with respect to the target surface center of the first image sensor below the first sample stage. By using the illumination of the symmetrically arranged first coherent light source and second coherent light source and the axial scanning of the image sensor, the pixel offset distance is calculated through image correlation analysis to obtain the corresponding relationship between the standard sample axial position and the pixel offset distance. The first coherent light source and the second coherent light source are used to irradiate the sample to be measured in sequence, and a large-field-of-view lensless diffraction image is collected. The axial position difference of different sub-regions is calculated according to the linear relationship to obtain the focus map of the sample to be measured.

[0008] Preferably, a first collimating lens is provided between the first coherent light source and the first sample stage, and a second collimating lens is provided between the second coherent light source and the first sample stage. The first collimating lens and the second collimating lens are symmetrically placed at the same height and are respectively perpendicular to the optical axes of the first coherent light source and the second coherent light source.

[0009] Preferably, the microscopic imaging system includes a second image sensor, a tube lens, an objective lens, a second sample stage for placing a sample, a third collimating lens, and a microscope illumination light source. The optical axes of the tube lens, the objective lens, and the third collimating lens coincide. The microscope illumination light source is located below the second sample stage, the third collimating lens is located between the second sample stage and the microscope illumination light source, the objective lens is located above the second sample stage, the tube lens is located above the objective lens, and the second image sensor is located above the tube lens.

[0010] Preferably, the first image sensor is a monochromatic sensor, and the second image sensor is a color sensor. The lensless imaging system uses symmetric coherent light source illumination to determine the axial position. Since the wavelength is single, using a monochromatic sensor can improve the accuracy; while the microscopic imaging system needs to collect color microscopic images and requires white light illumination, so a color sensor is used to collect images.

[0011] Preferably, the first coherent light source and the second coherent light source are narrowband LED light sources with the same central wavelength.

[0012] The focusing method of the automatic focusing device for whole-slide digital images based on lensless imaging described above includes a calibration step, a focus map acquisition step, and an automatic focusing step. The calibration step is used to obtain the linear relationship between the axial position of the standard sample and the pixel offset distance. The focus map acquisition step divides the sample to be measured into multiple sub-regions with equal sizes according to the field of view of the microscopic imaging system, and then calculates the axial position differences between the sub-regions of the sample to be measured according to the linear relationship obtained in the calibration step to obtain the focus map of the sample to be measured. The automatic focusing step is used to find the quasi-focus position of the first sub-region of the sample to be measured and complete the automatic focusing of the whole-slide digital image in combination with the focus map of the sample to be measured.

[0013] Preferably, the focusing method specifically includes the following steps: Step 1: Place the standard sample on the first sample stage, and move the first image sensor perpendicular to the target surface of the first image sensor to a position 1 - 5 mm away from the standard sample. Step 2: Light the first coherent light source and the second coherent light source in sequence, and collect two diffraction images. Step 3: Move the first image sensor away from the standard sample step by step at equal intervals perpendicular to the target surface of the first image sensor by N times with a preset step length. After each movement, repeat Step 2 to construct an image stack pair containing N pairs of images. Step 4: Take two images at the same distance in the image stack pair for cross-correlation analysis, and calculate the pixel offset distances of all N points. Step 5: Use linear fitting of the axial position of the standard sample and the pixel offset distance of the image at this position to obtain the linear relationship between the axial position of the standard sample and the pixel offset distance. Step 6: Place the sample to be measured on the first sample stage, light the first coherent light source and the second coherent light source in sequence, and collect two diffraction images. Step 7: Divide the two diffraction images collected in Step 6 into multiple sub-regions respectively. The physical size of the sub-regions is equal to the physical size of a single field of view of the high-power objective lens in the microscopic imaging system, and use cross-correlation analysis to calculate the pixel offset distance difference between each sub-region; Step 8: Calculate the axial position difference between each sub-region according to the linear relationship obtained in Step 5 to obtain the focus map of the sample to be measured; Step 9: Transfer the sample to be measured from the first sample stage to the second sample stage. Take the first observed field of view area as the first sub-region, and manually focus to obtain the quasi-focus position of the first sub-region of the sample to be measured; Step 10: Scan all sub-regions in the x-y direction and adjust each sub-region to the quasi-focus position according to the focus map of the sample to be measured to complete the automatic focusing of the full glass slide digital image.

[0014] Preferably, the image stack is composed of a first image sequence and a second image sequence. The first image sequence is obtained by illuminating with the first coherent light source, and the second image sequence is obtained by illuminating with the second coherent light source. The specific method of cross-correlation analysis in Step 4 is as follows: Step a: Use the Fourier transform convolution theorem and the fast Fourier transform to calculate the first image sequence and the cross-correlation function between the two images, and the expression is: where represents the cross-correlation function, represents the first image sequence, represents the Fourier transform, represents the inverse Fourier transform;

[0015] Step b: Maximize the cross-correlation function to determine the pixel offset distance between the two images. Description of the Drawings

[0016] Figure 1 is a schematic structural diagram of an automatic focusing device for full glass slide digital imaging based on lensless imaging provided by an embodiment of the present invention; Figure 2 is a technical principle diagram of a lensless imaging system provided by an embodiment of the present invention; Figure 3 is a linear relationship diagram between the first image sensor at different axial positions and the pixel offset distance provided by an embodiment of the present invention.

[0017] Description of the Reference Numerals: 100 - Lensless imaging system; 200 - Microscopic imaging system; 11 - First sample stage; 12 - Second sample stage; 21 - First coherent light source; 22 - Second coherent light source; 3 - First collimating lens; 4 - Second collimating lens; 5 - First image sensor; 6 - Third collimating lens; 7 - Objective lens; 8 - Tube lens; 9 - Second image sensor; 10 - Microscope illumination light source. Detailed implementation manners

[0018] The specific embodiments of the method of the present invention will be further described below in conjunction with the accompanying drawings.

[0019] The automatic focusing device for whole slide digital images based on lensless imaging provided by the embodiments of the present invention, as Figure 1 shown, is composed of a lensless imaging system 100 and a microscopic imaging system 200. The lensless imaging system 100 includes a first image sensor 5 arranged in sequence, a first sample stage 11 for placing a sample, a first coherent light source 21 and a second coherent light source 22. The first image sensor 5 is a monochromatic sensor. The first image sensor 5 is located above the first sample stage 11 and can move up and down in the vertical direction to perform axial scanning on the sample. The first coherent light source 21 and the second coherent light source 22 are narrow-band LED light sources with the same central wavelength. Exemplarily, the narrow-band LED light source is a green light source. The first coherent light source 21 and the second coherent light source 22 are symmetrically arranged with respect to the target surface center of the first image sensor 5 below the first sample stage 11. A first collimating lens 3 is arranged between the first coherent light source 21 and the first sample stage 11, and a second collimating lens 4 is arranged between the second coherent light source 22 and the first sample stage 11. The first collimating lens 3 and the second collimating lens 4 are symmetrically placed at the same height and are respectively perpendicular to the optical axes of the first coherent light source 21 and the second coherent light source 22. By using the illumination of the symmetrically arranged first coherent light source 21 and the second coherent light source 22 and the axial scanning of the first image sensor 5, the pixel offset distance can be calculated through image correlation analysis, and the corresponding relationship between the axial position of the standard sample and the pixel offset distance can be obtained; and by using the first coherent light source 21 and the second coherent light source 22 to irradiate the sample to be measured in sequence and collecting the large-field-of-view lensless diffraction image, the axial position difference of different sub-regions can be calculated according to the linear relationship, and the focus map of the sample to be measured can be obtained.

[0020] The microscopic imaging system 200 includes a second image sensor 9, a tube lens 8, an objective lens 7, a second sample stage 12 for placing a sample, a third collimating lens 6, and a microscope illumination light source 10, which are arranged in sequence. The second image sensor 9 is a color sensor. The optical axes of the tube lens 8, the objective lens 7, and the third collimating lens 6 coincide. The microscope illumination light source 10 is located below the second sample stage 12. The third collimating lens 6 is located between the second sample stage 12 and the microscope illumination light source 10. The objective lens 7 is located above the second sample stage 12. The tube lens 8 is located above the objective lens 7. The second image sensor 9 is located above the tube lens 8.

[0021] The automatic focusing method for the whole slide digital image based on lensless imaging provided by the embodiment of the present invention includes a calibration step, a focus map acquisition step, and an automatic focusing step: The calibration step is used to obtain the linear relationship between the axial position of the standard sample and the pixel offset distance, and includes: Step 1: Place the standard sample on the first sample stage 11, and move the first image sensor 5 perpendicular to the target surface of the first image sensor 5 to a position 1-5 mm away from the standard sample; Step 2: Light the first coherent light source 21 and the second coherent light source 22 in sequence, and collect two diffraction images; Step 3: Move the first image sensor 5 perpendicular to the target surface of the first image sensor 5 at equal intervals N times step by step away from the standard sample, and repeat Step 2 after each movement to construct an image stack pair containing N pairs of images; In this embodiment, the image stack pair includes two image sequences, each image sequence has N images, where the first image sequence is collected after being illuminated by the first coherent light source 21, and the second image sequence is collected after being illuminated by the second coherent light source 22.

[0022] First image sequence The expression is: , Second image sequence The expression is: , Among them, represents the propagation process, represents the distance from the image sensor to the sample.

[0023] Step 4: Take two images at the same distance in the image stack pair for cross-correlation analysis, and calculate the pixel offset distance of all N points; In this embodiment, the digital image correlation analysis includes the following steps: First, use the Fourier transform convolution theorem and the fast Fourier transform to quickly calculate the cross-correlation function between the two images. The expression is: , wherein represents the cross-correlation function, represents the first image sequence, represents the second image sequence, represents the Fourier transform, represents the inverse Fourier transform, Then, maximize the cross-correlation function to determine the pixel offset distance between the two images. The expression is: ; Step Five: Use the axial position of the standard sample and the pixel offset distance of the image at this position for linear fitting to obtain the linear relationship between the axial position and the pixel offset distance of the standard sample; Exemplarily, place the first image sensor 5 at an arbitrary position above the first sample stage 11 as the starting position, sequentially turn on the first coherent light source 21 and the second coherent light source 22 to collect images, and then move the first image sensor 5 at a step interval of 10 µm. For each 10-µm movement, collect the images at the corresponding positions.

[0024] Exemplarily, use for linear fitting to obtain the linear parameters a and b , wherein y represents the distance between the image sensor and the sample, x represents the pixel offset distance of the standard sample image. The fitting curve graph of the pixel offset distance of the standard sample image and the axial position is as shown in Figure 3 .

[0025] The focus map acquisition step obtains the focus map of the sample to be measured according to the linear relationship of the calibration step, mainly including: Step Six: Place the sample to be measured on the first sample stage 11, sequentially turn on the first coherent light source 21 and the second coherent light source 22, and collect two diffraction images; Step Seven: Divide the two diffraction images collected in Step Six into multiple sub-regions respectively. The physical size of the sub-regions is equal to the physical size of a single field of view of the high-magnification objective lens in the microscopic imaging system 200, and use cross-correlation analysis to calculate the pixel offset distance difference between each sub-region; Exemplarily, the physical size of each sub-region image after image division is equal to the field of view size collected by the high-magnification objective lens in the microscopic imaging system 200, and a total of 14 * 14 sub-region images of the same size can be divided.

[0026] Step Eight: Calculate the axial position difference between each sub-region according to the linear relationship obtained in Step Five to obtain the focus map of the sample to be measured; The autofocus procedure is used to find the in-focus position of the first sub-region of the sample to be measured and complete the autofocus of the whole-slide digital image in combination with the focus map of the sample to be measured, mainly including: Step Nine: Transfer the sample to be measured from the first sample stage 11 to the second sample stage 12. Take the field-of-view area first observed by the microscopic imaging system 200 as the first sub-region, and manually focus to obtain the in-focus position of the first sub-region of the sample to be measured; In this embodiment, move the second image sensor 9 up and down relative to the second sample stage 12 to collect a stack of images of the first region of the sample; from the obtained stack of images, extract the image quality feature evaluation function of each image to calculate the focusing quality at this position; select the image corresponding to the position with the best focusing quality as the in-focus image; Step Ten: Scan all sub-regions in the plane direction, and adjust each sub-region to the in-focus position according to the focus map of the sample to be measured to complete the autofocus of the whole-slide digital image.

[0027] The technical principle of the lensless imaging system of the present invention is as Figure 2 shown. Through the lensless imaging system 100, use a symmetric coherent light source to irradiate the standard sample in sequence. Move the first image sensor 5 perpendicular to the target surface of the first image sensor 5 at a preset step size N times and gradually move away from the standard sample. Repeat Step Two after each movement to construct an image stack pair containing 2N images; perform digital image correlation analysis on two images at the same position in the stack to calculate the pixel offset distance, so as to obtain the linear relationship between the axial position and the pixel offset distance; use a symmetric coherent light source to irradiate the sample in sequence, collect a large-field-of-view lensless diffraction image, and calculate the focal plane positions of different sub-regions according to the linear relationship in the calibration step to obtain the focus map of the whole-slide sample. Subsequently, find the best focal plane position of the first sub-region through the microscopic imaging system 200, and the in-focus position of the whole field of view can be quickly obtained in combination with the focus map.

[0028] The above content is a further detailed description of the present invention in combination with specific preferred embodiments. It cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, several simple deductions or substitutions can still be made, which should all be regarded as belonging to the protection scope of the present invention.

Claims

1. An automatic focusing device for whole-slide digital images based on lensless imaging, characterized in that, It consists of a lensless imaging system and a microscopic imaging system. The lensless imaging system is used to obtain the linear relationship between the axial position of a standard sample and the pixel offset distance, divide the sample to be measured into multiple sub-regions of equal size according to the field of view of the microscopic imaging system, and then obtain the focus map of the sample to be measured based on the obtained linear relationship. The microscopic imaging system is used to find the quasi-focus position of the sub-region of the sample to be measured observed for the first time and complete the autofocus of the full-slide digital image in combination with the focus map of the sample to be measured.

2. The automatic focusing device for whole slide digital images based on lensless imaging according to claim 1, wherein, The lensless imaging system includes a first image sensor, a first sample stage for placing the sample, a first coherent light source, and a second coherent light source arranged in sequence. The first image sensor is located above the first sample stage and can move up and down in the vertical direction to perform axial scanning on the sample. The first coherent light source and the second coherent light source are symmetrically arranged with respect to the target surface center of the first image sensor below the first sample stage. By using the illumination of the symmetrically arranged first coherent light source and second coherent light source and the axial scanning of the first image sensor, the pixel offset distance is calculated through image correlation analysis, and the corresponding relationship between the axial position of the standard sample and the pixel offset distance is obtained. The first coherent light source and the second coherent light source are used to irradiate the sample to be measured in sequence, collect large-field lensless diffraction images, calculate the axial position difference of different sub-regions according to the linear relationship, and obtain the focus map of the sample to be measured.

3. The automatic focusing device for whole slide digital images based on lensless imaging according to claim 2, wherein A first collimating lens is arranged between the first coherent light source and the first sample stage, and a second collimating lens is arranged between the second coherent light source and the first sample stage. The first collimating lens and the second collimating lens are symmetrically placed at the same height and are respectively perpendicular to the optical axes of the first coherent light source and the second coherent light source.

4. The automatic focusing device for whole-slide digital images based on lensless imaging according to claim 2, wherein, The microscopic imaging system includes a second image sensor, a tube lens, an objective lens, a second sample stage for placing the sample, a third collimating lens, and a microscope illumination light source. The optical axes of the tube lens, the objective lens, and the third collimating lens coincide. The microscope illumination light source is located below the second sample stage, the third collimating lens is located between the second sample stage and the microscope illumination light source, the objective lens is located above the second sample stage, the tube lens is located above the objective lens, and the second image sensor is located above the tube lens.

5. The automatic focusing device for whole slide digital images based on lensless imaging according to claim 4, wherein The first image sensor is a monochromatic sensor, and the second image sensor is a color sensor.

6. The automatic focusing device for whole-slide digital images based on lensless imaging according to any one of claims 2 to 5, characterized in that, The first coherent light source and the second coherent light source are narrowband LED light sources with the same central wavelength.

7. The focusing method of the autofocus device for whole-slide digital images based on lensless imaging according to claim 4, characterized in that It includes a calibration step, a focal map acquisition step, and an autofocus step. The calibration step is used to obtain the linear relationship between the axial position of the standard sample and the pixel offset distance. The focal map acquisition step divides the sample to be measured into multiple sub-regions of equal size according to the field of view of the microscopic imaging system, and then calculates the axial position difference between the sub-regions of the sample to be measured based on the linear relationship obtained in the calibration step to obtain the focal map of the sample to be measured. The autofocus step is used to find the quasi-focus position of the first sub-region of the sample to be measured and complete the autofocus of the full-slide digital image in combination with the focal map of the sample to be measured.

8. The focusing method according to claim 7, wherein, Specifically, it includes the following steps: Step 1: Place the standard sample on the first sample stage, and move the first image sensor perpendicular to the target surface of the first image sensor to a position 1-5 mm away from the standard sample. Step 2: Turn on the first coherent light source and the second coherent light source in sequence, and collect two diffraction images. Step 3: Move the first image sensor perpendicular to the target surface of the first image sensor at equal intervals N times step by step away from the standard sample according to a preset step size. After each movement, repeat Step 2 to construct an image stack pair containing N pairs of images. Step 4: Take two images at the same distance in the image stack pair for cross-correlation analysis, and calculate the pixel offset distance of all N points. Step 5: Perform linear fitting using the axial position of the standard sample and the pixel offset distance of the image at this position to obtain the linear relationship between the axial position of the standard sample and the pixel offset distance. Step 6: Place the sample to be measured on the first sample stage, turn on the first coherent light source and the second coherent light source in sequence, and collect two diffraction images. Step 7: Divide the two diffraction images collected in Step 6 into multiple sub-regions respectively. The physical size of the sub-regions is equal to the physical size of a single field of view of the high-magnification objective lens in the microscopic imaging system, and use cross-correlation analysis to calculate the pixel offset distance difference between the sub-regions. Step 8: Calculate the axial position difference between the sub-regions according to the linear relationship obtained in Step 5 to obtain the focal map of the sample to be measured. Step 9: Transfer the sample to be measured from the first sample stage to the second sample stage, take the first observed field of view area as the first sub-region, and manually focus to obtain the quasi-focus position of the first sub-region of the sample to be measured. Step 10: Scan all sub-regions in the x-y direction, and adjust each sub-region to the quasi-focus position according to the focal map of the sample to be measured to complete the autofocus of the full-slide digital image.

9. The focusing method according to claim 8, wherein The image stack is composed of a first image sequence and a second image sequence. The first image sequence is obtained by illuminating with the first coherent light source, and the second image sequence is obtained by illuminating with the second coherent light source. The specific method of cross-correlation analysis in Step 4 is as follows: Step a: Calculate the first image sequence using the Fourier transform convolution theorem and the fast Fourier transform and the second image sequence The cross-correlation function between two images , the expression is: ; where represents the cross-correlation function, represents the first image sequence, represents the second image sequence, represents the Fourier transform, represents the inverse Fourier transform; Step b: Maximize the cross-correlation function to determine the pixel offset distance between the two images .

Images