A real-time autofocus system for microscopes

Abstract

The present invention discloses a real-time autofocus system for a microscope. The system includes an infrared light source module, an offset lens optical path module, a detection module and a focus control module, wherein the infrared light source module is used to emit a parallel laser beam; the offset lens optical path module expands the parallel laser beam emitted by the infrared light source module into a divergent light beam, and irradiates the sample interface through a microscope objective lens, so that the laser diffraction spot irradiated at the sample reflection interface forms a comet-shaped aberration; the detection module modulates the coma-shaped diffraction spot formed by the reflection of the sample interface into a linear spot and obtains the defocus information of the microscope system in the current field of view by analyzing the linear spot; the focus control module iteratively controls the axial movement of the microscope objective lens based on the obtained defocus information until the defocus meets the set target. The system of the present invention can achieve high-precision real-time focusing of the microscope system.

Description

Technical Field

[0001] The present invention relates to the technical field of automatic focusing, and more particularly to a real-time automatic focusing system for a microscope. Background Art

[0002] In recent years, microscopic imaging technology has been widely used in the rapidly developing biomedical field, which is divided into research-level fully automatic microscopes and digital pathology slide scanners used for pathological diagnosis. Fully automatic microscopes usually require a long time (lasting about several hours to dozens of hours) to automatically observe biological samples, while digital pathology slide scanners use microscope systems to quickly scan and collect pathology slides to obtain digital images of the entire slide, providing a basis for consultation and diagnosis.

[0003] Autofocus technology is a key core technology for stable imaging of microscopic systems. It can compensate for focus drift caused by ambient temperature changes, mechanical vibrations, motion vibrations, etc. during long-term image data acquisition of the microscopic system, ensuring that clear microscopic images are obtained. Traditional microscope operation requires users to frequently adjust the distance between the objective lens and the slide to obtain the appropriate focus, which is time-consuming and laborious. Automated microscopic imaging technology requires stable and reliable autofocus technology to ensure the quality of collected image data.

[0004] At present, there are two main types of autofocus methods used in microscopes. One is based on software image algorithms, that is, scanning different imaging planes along the optical axis of the objective lens to obtain multiple groups of sample images, and determining the defocus amount based on the image quality evaluation function; the other is based on hardware, using additional auxiliary light sources (such as lasers, LEDs, etc.) to illuminate the surface of the sample slide through the objective lens, and determining the defocus amount of the system based on the shape of the reflected light spot.

[0005] In the prior art, the hardware-based autofocus method usually uses an auxiliary light source and calculates the shape of the reflected light spot to determine the defocus amount of the microscope system. The obvious disadvantages of this method are: first, the focus reference plane is not the actual sample imaging focal plane. Since the auxiliary light source spot can only be reflected on the interface between the sample and the glass slide, the focus reference plane is this reflection interface, which is different from the imaging focal plane, resulting in the need to move the objective lens a fixed distance after the focusing is completed to reach the imaging focal plane, which makes the focusing accuracy worse; second, since the one-time compensation of the focus drift depends on the accurate calculation of the defocus amount and the accuracy of the moving mechanism, it will bring a considerable deviation under the objective lens with a high numerical aperture, thereby further reducing the focusing accuracy; third, the introduction of a complex beam method leads to a complex system and high cost. The software-based autofocus method takes a long time and cannot focus in real time. Moreover, when the sample has a three-dimensional structure, the software algorithm cannot find the clearest imaging plane position due to the deep range of the clear imaging focal plane, resulting in focusing failure. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the prior art and provide a real-time automatic focusing system for a microscope, which dynamically controls the objective lens to adjust to the optimal focus point by detecting and analyzing the instant focus state of the microscope system.

[0007] The present invention provides a real-time autofocus system for a microscope. The system includes an infrared light source module, an offset lens optical path module, a detection module and a focus control module, wherein the infrared light source module is used to emit a parallel laser beam; the offset lens optical path module expands the parallel laser beam emitted by the infrared light source module into a divergent light beam, and irradiates the sample interface through a microscope objective lens, so that the laser diffraction spot irradiated at the sample reflection interface forms a comet-shaped aberration; the detection module modulates the coma-shaped diffraction spot formed by the sample interface reflection into a linear spot, and obtains the defocus amount related information of the microscope system in the current field of view by analyzing the linear spot; the focus control module iteratively controls the axial movement of the microscope objective lens based on the obtained defocus amount related information until the defocus amount meets the set target.

[0008] In one embodiment, the infrared light source module includes a laser and a spectrometer. The laser is used to generate a parallel laser beam. The parallel laser beam passes through a collimator and an attenuation plate in sequence and then is incident on the offset lens optical path module through the spectrometer.

[0009] In one embodiment, the offset lens optical path module includes a first reflector, an offset lens fixed on a one-dimensional electric translation stage, and a second reflector. The first reflector receives the parallel laser beam emitted by the infrared light source module and reflects it to the offset lens. The beam is incident on the second reflector through the offset lens and is reflected into the objective lens through the internal filter of the microscope to irradiate the sample interface.

[0010] In one embodiment, the detection module includes a focusing lens, a cylindrical mirror, a camera and a control system. The laser diffraction spot reflected by the irradiated sample interface passes through the spectrometer of the infrared light source module and is focused on the photosensitive chip of the camera by the focusing lens and the cylindrical mirror. The cylindrical mirror transforms the circular spot into an elliptical spot in space to form a linear spot. The control system communicates with the camera and obtains information related to the defocus amount of the microscope system under the current field of view by analyzing the spatial position change of the linear spot.

[0011] In one embodiment, the defocus amount of the microscope system is calculated according to the following steps:

[0012] Collecting the original image containing the line-shaped spot;

[0013] The pixels of the original image are accumulated along the direction perpendicular to the linear spot to obtain the image intensity curve;

[0014] Detect the peak position of the linear laser spot with the highest signal value from the image intensity curve;

[0015] Cutting an image sub-window containing a plurality of pixels based on the peak position;

[0016] Calculate the binary segmentation threshold for the original image;

[0017] Use the segmentation threshold to binarize the image sub-window to obtain a binary image;

[0018] Extract the linear spot at the center of the binary image, and calculate the center coordinates of the spot with sub-pixel accuracy by combining the image sub-window;

[0019] The light spot movement amount dx is obtained using the obtained light spot center coordinates;

[0020] The defocus amount δ is obtained through the corresponding relationship between the focus drift amount and the light spot movement distance δ=K·dx, where K is the conversion coefficient.

[0021] In one embodiment, the focus control module is respectively connected to the control system and the microscope objective, and is used to iteratively control the axial movement of the microscope objective according to the correspondence between the obtained defocus related information and the microscope objective adjustment feedback amount until the defocus amount meets the set target.

[0022] In one embodiment, the focus control module is a motor controller, which determines the adjustment feedback amount of the microscope objective lens through a proportional-integral-differential controller.

[0023] In one embodiment, the adjustment feedback amount of the microscope objective lens is calculated by the following formula:

[0024] F(t)=K P ·δ(t)+K I ∫δ(t)+K D δ′(t), where K P , K I , K D represents the coefficient of the corresponding term, and δ(t) is the defocus amount of the microscope system under the current field of view calculated in real time.

[0025] In one embodiment, the infrared light source module uses a near-infrared or infrared wavelength laser with a wavelength range of 650 nm to 1100 nm and a power less than 5 mW.

[0026] In one embodiment, the beam splitter is a beam splitter, a D-shaped reflector, or a reflector with a knife edge.

[0027] Compared with the prior art, the present invention has the advantages that the operation process of the proposed autofocus system is independent of the selection of the imaging focal plane by the microscopic imaging system, and the instantaneous focus state of the microscopic system is detected with sub-pixel precision, the defocus amount of focus drift is calculated, and the objective lens is dynamically feedback-controlled to adjust to the optimal focus point. The present invention has the advantages of real-time performance, high stability, high precision and low cost.

[0028] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0030] Figure 1 is a schematic diagram of a real-time autofocus system for a microscope according to one embodiment of the present invention;

[0031] Figure 2 is a schematic diagram of the real-time auto-focusing principle according to an embodiment of the present invention;

[0032] Figure 3 is a flow chart of a method for detecting defocus amount according to an embodiment of the present invention;

[0033] Figure 4 It is a data diagram of an autofocus effect test experiment according to an embodiment of the present invention.

[0034] In the attached figure, 1-laser; 2-beam splitter; 6-first reflector; 7-offset lens; 8-second reflector; 3-focusing lens; 4-cylindrical mirror; 5-imaging camera; 9-short-pass filter in microscope; 10-electrically movable objective lens; 11-sample reflection interface; 12-control system; 13-motor controller. DETAILED DESCRIPTION

[0035] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0036] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0037] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0038] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0039] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0040] The present invention provides a real-time high-precision autofocus system for a microscope. In short, the system uses a near-infrared laser beam to irradiate the interface of a biological sample slide through the objective lens of the microscope to form a laser spot. The laser spot is reflected by the interface and collected by the same objective lens to form an image on a photosensitive chip of the autofocus system. The drift of the focus will bring about a corresponding movement of the position of the reflected light spot on the photosensitive chip. By analyzing the light spot movement in real time and calculating the focus drift of the system, the objective lens of the microscope is controlled to correct the drift, thereby completing the real-time autofocus process.

[0041] See also Figure 1 As shown, the system involves multiple modular subsystems, including an infrared light source module, an offset lens optical path module, a detection module and a focus control module.

[0042] The infrared light source module includes a laser 1 and a spectroscope 2. For example, the laser 1 generates a laser with a near-infrared or infrared wavelength. The output end of the laser 1 is a single-mode optical fiber. The laser output is coupled to a collimator using a single-mode optical fiber to form a fundamental mode Gaussian beam and collimate the output parallel laser beam with a divergence angle less than 0.5 mrad. The laser beam then reduces the power through an attenuation plate and is incident on the offset lens optical path module through a spectroscope 2. In an embodiment of the present invention, a low-power (less than 5 mW) laser with a near-infrared or infrared wavelength (such as 650 nm-1100 nm) can be used.

[0043] The offset lens optical path module includes a first reflector 6, an offset lens 7, a one-dimensional electric translation stage (not shown), and a second reflector 8. The laser beam emitted by the infrared light source module is incident on the first reflector 6 and reflected to the offset lens 7. The offset lens 7 is mounted and fixed on the one-dimensional electric translation stage, and its position can be adjusted along the optical axis. The laser beam passes through the offset lens 7 and is incident on the second reflector 8, and is reflected into the objective lens 10 through the short-pass filter 9 inside the microscope, and finally irradiates the sample interface 11. In the system of the present invention, the laser beam after the spatial transformation of the offset lens is a divergent beam with a certain divergence, and a light spot (such as 3mm to 4mm) close to the aperture size of the objective lens is formed at the rear focal plane of the objective lens 10 and enters the objective lens 10. Its incident position is adjusted by the first reflector 6 and the second reflector 8 to the edge of the aperture of the objective lens, so that the laser beam converged by the objective lens 10 forms a certain inclination angle with the optical axis of the objective lens and irradiates the sample interface 11; the incident laser and the optical axis of the objective lens are adjusted by the first reflector 6 and the second reflector 8 to form a certain inclination angle, so that the laser light spot converged at the sample interface 11 forms a relatively obvious comet-shaped aberration. The reflected laser light spot returns via the same path, and then enters the detection module after being reflected by the beam splitter 2 in the infrared light source module.

[0044] In summary, the offset lens optical path module provided by the present invention has the following characteristics: the offset lens is fixed on a one-dimensional electric translation stage, and its position can be adjusted along the optical axis, so that the transformation of the laser beam can meet the selection of different objective lenses; the offset lens transforms the parallel laser beam into a divergent beam with a certain divergence, so that the focus of the beam passing through the objective lens is far away from the focal plane of the objective lens imaging; the divergent beam forms a laser beam with a diameter close to the size of the objective lens aperture at the focal plane behind the objective lens, so that the laser spot irradiated at the sample reflection interface forms a relatively obvious diffraction spot; the optical axis of the laser beam incident on the objective lens does not coincide with the optical axis of the objective lens, and is close to the edge of the objective lens aperture, so that the beam converged through the objective lens forms a certain inclination angle with the optical axis of the objective lens, and is obliquely irradiated at the sample reflection interface; the optical axis of the laser beam incident on the objective lens has a certain angle with the optical axis of the objective lens, so that the laser spot irradiated at the sample reflection interface forms a relatively obvious comet-shaped aberration.

[0045] From the perspective of optical path configuration, the detection module includes a focusing lens 3, a cylindrical mirror 4, and an imaging camera 5. The reflected laser spot passes through the beam splitter 2 in the optical path system and is focused on the photosensitive chip of the imaging camera 5 by the focusing lens 3 and the cylindrical mirror 4. The cylindrical mirror 4 transforms the circular spot into an elliptical spot in space, and after properly selecting the focal length, a spot approximately in line can be formed. The imaging spot is adjusted to the center of the photosensitive chip of the camera 5 by the beam splitter 2.

[0046] The detection module also includes a control system 12. In this article, a computer control system is used as an example for description. Specifically, the imaging camera 5 communicates with the computer, and the laser spot image collected by its photosensitive chip is detected and analyzed by the computer control system 12 through an algorithm, and the sub-pixel precision coordinates of the linear spot are calculated, and the defocus amount of the focus drift is obtained by coefficient transformation.

[0047] The focus control module (taking the motor controller 13 as an example) is used to receive feedback from the control system 12 on the defocus analysis and control the movement of the objective lens. For example, focus drift correction can be achieved through a proportional-integral-differential controller (PID).

[0048] Combination Figure 1 As shown, the automatic focusing process of the system provided by the present invention includes the following steps:

[0049] Step S210, the infrared light source module emits a parallel laser beam, which is expanded into a divergent light by the offset lens optical path module, and irradiates the sample interface through the microscope objective lens. The coma-shaped diffraction spot formed by the interface reflection is further spatially modulated into an approximately parallel linear spot by the detection module, and image acquisition is performed to obtain a spot image;

[0050] Step S220, the computer control system runs an image detection algorithm on the linear spot image to calculate the defocus amount of the microscope system in the current field of view;

[0051] Step S230, the motor controller compensates for the defocus amount;

[0052] Step S240, repeating steps S210 to S230 until the defocus amount in the current field of view is less than the set threshold, and the autofocus is completed.

[0053] For the sake of clarity, the following will be combined with Figure 1 and Figure 2 The principle of the real-time focusing system provided by the present invention is explained.

[0054] See also Figure 2 As shown in (A), the present invention uses the reflected light spot of near-infrared or infrared laser at the glass slide interface to analyze and determine the defocus state of the microscope system. In the application of high numerical aperture objective lenses, the immersion medium between the objective lens and the glass slide is usually oil or water to improve the resolution. At this time, due to the similar refractive index, the interface with stronger laser reflection is the interface with a large refractive index difference between the glass slide and the biological sample. The focal plane of the objective lens (i.e., the object plane) is in the sample area far away from this reflection interface, and its distance from the reflection interface is the focus offset distance. Adjusting this focus offset distance value can realize imaging observation of different depths of biological samples with three-dimensional structures. In the imaging process of the microscope, when a specific imaging focal plane is selected, the focus offset distance is determined.

[0055] A beam of laser light with a very small divergence angle can be regarded as parallel light, which will be focused on the focal plane of the objective lens through the converging effect of the objective lens. The laser spot focused on the focal plane of the objective lens has a very high energy density due to its extremely small size, which will form a local thermal effect on the biological tissue sample. The present invention adds an offset lens in the optical path so that the parallel laser light passes through the offset lens to form a divergent light beam, which is incident on the objective lens with a certain divergence. The laser light with a certain divergence passes through the converging effect of the objective lens and forms a laser focusing surface with a certain distance from the focal plane of the objective lens at a distance away from the objective lens. The distance between this laser focusing surface and the focal plane of the objective lens depends on the divergence of the incident laser. At this time, the laser spot with high energy density on the laser focusing surface will no longer affect the biological sample because it is far away from the biological sample, thereby realizing the protection of the biological sample in long-term microbiological experiments.

[0056] Under the configuration of the real-time focusing system provided by the present invention, a larger laser spot will be formed at the reflective interface where the glass slide and the sample meet. The reflective interface has a strong reflective effect on the laser spot, so a laser with lower energy (such as less than 1mW) can realize the technical solution. During the focusing process, the spatial position state of the non-focused laser spot at the current reflective interface will be used as the basis for the focus drift of the microscopic imaging system, and will not depend on the change in the distance between the imaging plane and the reflective interface (i.e., the focus offset distance). Furthermore, by adjusting the distance of the offset lens relative to the objective lens in the optical path, the laser beam incident on the objective lens reaches a size similar to the aperture of the objective lens, forming a more significant diffraction effect on the focused laser spot. Under the influence of the diffraction effect, the larger laser spot at the reflective interface has a circular hole diffraction structure, i.e., a concentric ring-shaped spot. As the distance between the objective lens and the reflective interface changes, the size of this laser spot will change slightly. The offset lens is fixed on a one-dimensional electric platform, and the corresponding distance adjustment can be made according to the specifications of the objective lens (such as magnification, numerical aperture, etc.).

[0057] See also Figure 2 As shown in (B), the laser optical axis incident on the objective lens deviates from the objective lens optical axis by a certain distance, and forms an angle θ with the objective lens optical axis through the converging effect of the objective lens and is incident on the reflective interface. At the same time, the laser optical axis forms a certain angle with the objective lens optical axis, and the laser beam forms an oblique incidence state, so that the laser spot at the reflective interface is away from the center of the imaging field of view, avoiding direct irradiation of the biological sample. On the one hand, oblique incidence can avoid the ghost image of the light spot caused by multiple reflections between optical elements in the optical path of the microscope system under vertical incidence, thereby improving the focusing effect; on the other hand, the laser spot on the reflective interface forms a comet-shaped aberration, forming an eccentric annular diffraction spot with an asymmetric structure, and the energy is concentrated on one side of the diffraction spot. The two sides of the center of gravity of the spot are the annular structure and the background noise respectively. The present invention uses the asymmetry of the diffraction spot to determine the defocus direction of the microscope system.

[0058] For the optical path configuration provided by the embodiment of the present invention, within a certain range, the size of the laser spot located on the reflective interface at different distances from the objective lens will linearly decrease as the objective lens moves away from the reflective interface, and the position of its diffraction structure on the plane will also move monotonically along a one-dimensional direction. When the focus of the microscope system drifts due to various reasons, the distance between the objective lens and the reflective interface will change. When the objective lens is away from the reflective interface and the distance change is +δ, the center of the laser spot will be horizontally translated by +dx; when the objective lens is close to the reflective interface and the distance change is -δ, the center of the laser spot will be horizontally translated by -dx.

[0059] The laser spot with a comet-shaped phase difference diffraction structure at the reflection interface passes through the objective lens and is focused on the photosensitive chip of the imaging camera. In the detection module, the present invention uses a cylindrical mirror to stretch the light spot in one dimension. Specifically, the cylindrical mirror stretches in a direction perpendicular to the direction of movement of the light spot on the plane, so that the laser spot of the diffraction structure forms a series of parallel linear spots on the imaging photosensitive chip. The comet-shaped phase difference effect makes the linear spot with the most concentrated energy located on one side of the parallel line. The defocus amount of focus drift is reflected in the movement of the parallel line along its orthogonal direction, and the direction of drift reflects the movement direction of the linear spot with the most concentrated energy. The corresponding relationship between the defocus amount and the moving distance of the linear spot is δ=K·dx, where the transformation coefficient K is obtained by fitting the system axial scanning correction curve.

[0060] Furthermore, the imaging camera communicates with the computer, transmits image data in real time and applies a sub-pixel precision displacement detection algorithm to the parallel linear laser spot. Figure 3 As shown, in one embodiment, the displacement-based defocus detection method includes the following steps:

[0061] Step S310, collecting an original image containing parallel linear light spots, for example, an image of M×N pixel size, represented as I, containing parallel linear light spots in the vertical direction, wherein M and N can select appropriate values ​​according to the resolution and image size of the original image;

[0062] Step S320, accumulating the original image along the vertical line spot to obtain a 1XN image intensity curve, which is expressed as Profile;

[0063] Step S330, detecting the peak position of the linear laser spot with the highest signal value from the curve Profile, expressed as PeakLoc;

[0064] Step S340, intercepting an image sub-window of 64×64 pixels with [PeakLoc, M / 2] as the center, denoted as ISub;

[0065] Step S350, using an OTSU (maximum between-class variance method) threshold algorithm to calculate a binary segmentation threshold T for the original image;

[0066] Step S360, binarizing the image sub-window using the threshold T to obtain a binarized image, denoted as BSub;

[0067] Step S370, extracting the linear spot at the center of BSub through morphological operation, and combining ISub to calculate the centroid coordinates with sub-pixel accuracy, expressed as LocX;

[0068] Step S380, comparing the light spot coordinate LocX with the focal plane setting value coordinate ZeroX calculated by steps S310 to S370, to obtain the light spot movement amount, expressed as dx;

[0069] Step S390, obtaining the system defocus amount δ by transforming the coefficient of the correspondence relationship δ=K·dx between the focus drift amount and the light spot movement distance.

[0070] When the microscope system starts imaging, the position of the objective lens imaging focal plane is first determined (for example, manually determined). At this time, the position of the laser spot in the imaging camera is automatically set to the focal position (Z=0) by the control system. The control system uses the detection algorithm to analyze the spot image at this time in the above steps S310 to S370 to obtain the microscope system focal position reference value ZeroX. When focus drift occurs, the control system uses a complete detection algorithm to detect the displacement dx of the laser spot, and calculates the defocus amount δ. The objective lens adjustment feedback amount F(t)=K is calculated by the PID algorithm. P ·δ(t)+K I ∫δ(t)+K D ·δ′(t)(K P , K I , K D is the coefficient of the corresponding term) and fed back to the objective lens axial motion control motor to compensate for focus drift. Iterative feedback is performed before the defocus amount of the microscope system falls below a preset threshold.

[0071] It should be noted that the above embodiments are only illustrative. Without violating the spirit of the present invention, those skilled in the art may make appropriate changes or modifications. For example, the beam splitter may be replaced by a D-shaped reflector or a reflector with a knife edge. Under this alternative, the power of the laser can be appropriately reduced; the offset lens may be replaced by a variable focal length lens, such as a flexible zoom lens driven by electro-deformation; the imaging camera may be replaced by a linear array camera; the feedback PID algorithm may be replaced by other algorithms with fast adjustment and stability.

[0072] In order to verify the feasibility and effect of the present invention, the following experiments were conducted.

[0073] (1) Experiment on defocus amount and linear spot moving distance

[0074] The experiment determines the transformation coefficient K in the corresponding relationship between the defocus amount and the moving distance of the linear spot δ=K·dx. In the experiment, the objective lens is scanned along the optical axis of the objective lens with a fixed step length of 10nm accuracy, and the linear spot collected by the imaging camera is detected by algorithm at each position point to obtain its spatial position (in pixel coordinates). The scanning range is ±4μm, which is far beyond the depth of field of the objective lens. The data of the relationship between pixel coordinates and objective lens axial coordinates are obtained and linear fitting is performed. The R square factor in the fitting result reaches 99.5%, and the transformation coefficient K of the autofocus system is 240 nanometers / pixel, that is, for every displacement of 1 pixel detected, the linear corresponding microscopic imaging system has a defocus amount of about 240nm.

[0075] (2) Experiments to test the autofocus effect

[0076] In the experiment, a 60X oil-immersion objective lens was used, with a numerical aperture of 1.40 and a depth of field of approximately 542nm. The sample slide was placed in a constant temperature box, and both the constant temperature box and the objective lens were continuously heated to 37°C and maintained for 4 hours to achieve normal experimental conditions for observation of living organisms. The experimental test time was 30 minutes, and the focus drift of the microscope imaging system was tested with and without autofocus turned off. Without autofocus turned on, the imaging system lost focus within a few minutes and drifted out of the depth of field of the objective lens. During the entire experiment, the standard deviation (STD) of the drift reached 578.2nm; with autofocus turned on, the imaging system continuously maintained the focus state, with a standard deviation of 4.7nm and a distribution range of ±14.1nm. The focus repeatability accuracy can reach 5.2% of the depth of field of the objective lens, achieving a high-precision autofocus effect. For experimental results and experimental data, see Figure 4 As shown, the ordinate represents the axial position, the abscissa represents the test time, AutofocusOFF represents autofocus off (corresponding to the upper curve), and Autofocus ON represents autofocus on (corresponding to the lower curve).

[0077] In summary, in view of the existing hardware-based autofocus method, the laser focusing plane is the sample reflection plane, which will cause local high-energy density irradiation to the biological sample. The present invention uses an offset lens to make the laser focusing plane away from the sample imaging area to avoid local high-energy density light irradiation; in view of the existing hardware-based autofocus method, the laser focusing plane is the objective lens focal plane, which has a certain distance from the sample reflection interface (focusing reference plane), resulting in the need to move the objective lens again for a fixed distance after focusing, thereby reducing the focusing accuracy. The present invention uses the spatial position state of the diffraction spot in the non-focused state as the basis for the focus drift of the microscopic imaging system, does not rely on the change in the distance between the objective lens focal plane and the reflection interface, and improves the stability and focusing accuracy; the present invention uses the diffraction spot with comet-shaped aberration to analyze its real-time displacement at sub-pixel accuracy, and iteratively adjusts the real-time defocus amount of the microscopic imaging system through the PID algorithm to achieve high-precision autofocus.

[0078] The defocus detection process of the above-mentioned embodiment of the present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for enabling a processor to implement various aspects of the present invention.

[0079] A computer-readable storage medium may be a tangible device that holds and stores instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0080] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0081] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0082] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0083] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0084] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0085] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0086] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A real-time autofocus system for a microscope, comprising an infrared light source module, an offset lens optical path module, a detection module and a focus control module, wherein: The infrared light source module is used to emit a parallel laser beam; The offset lens optical path module expands the parallel laser beam emitted by the infrared light source module into divergent light, and irradiates the sample interface through the microscope objective lens, so that the laser diffraction spot irradiated at the sample reflection interface forms a comet-shaped aberration, wherein the laser optical axis forms a set angle with the optical axis of the microscope objective lens, so that the laser beam forms an inclined incident state; The detection module modulates the coma-shaped diffraction spot formed by the sample interface reflection into a linear spot, and obtains information related to the defocus amount of the microscope system in the current field of view by analyzing the linear spot; The focus control module iteratively controls the axial movement of the microscope objective lens based on the obtained defocus amount related information until the defocus amount meets the set target; The infrared light source module includes a laser and a spectrometer. The laser is used to generate a parallel laser beam. The parallel laser beam passes through a collimator and an attenuation plate in sequence and then is incident on the offset lens optical path module through the spectrometer. The offset lens optical path module includes a first reflector, an offset lens fixed on a one-dimensional electric translation stage, and a second reflector. The first reflector receives the parallel laser beam emitted by the infrared light source module and reflects it to the offset lens. The beam is incident on the second reflector through the offset lens and is reflected into the objective lens through the internal filter of the microscope to irradiate the sample interface. Among them, the detection module includes a focusing lens, a cylindrical mirror, a camera and a control system. The laser spot reflected by the irradiated sample interface passes through the spectrometer of the infrared light source module and is focused on the photosensitive chip of the camera by the focusing lens and the cylindrical mirror. The cylindrical mirror transforms the circular spot space into an elliptical spot to form a linear spot. The control system communicates with the camera and obtains information related to the defocus amount of the microscope system under the current field of view by analyzing the spatial position change of the linear spot.

2. The real-time autofocus system for a microscope according to claim 1, wherein: Calculate the defocus of the microscope system according to the following steps: Collecting the original image containing the line-shaped spot; The pixels of the original image are accumulated along the direction perpendicular to the linear spot to obtain the image intensity curve; Detect the peak position of the linear laser spot with the highest signal value from the image intensity curve; Cutting an image sub-window containing a plurality of pixels based on the peak position; Calculate the binary segmentation threshold for the original image; Use the segmentation threshold to binarize the image sub-window to obtain a binary image; Extract the linear spot at the center of the binary image, and calculate the center coordinates of the spot with sub-pixel accuracy by combining the image sub-window; The light spot movement amount dx is obtained using the obtained light spot center coordinates; The defocus amount δ is obtained through the corresponding relationship between the focus drift amount and the light spot movement distance δ=K·dx, where K is the conversion coefficient.

3. The real-time autofocus system for a microscope according to claim 1, wherein: The focus control module is connected to the control system and the microscope objective lens respectively, and is used to iteratively control the axial movement of the microscope objective lens according to the corresponding relationship between the obtained defocus amount related information and the microscope objective lens adjustment feedback amount until the defocus amount meets the set target.

4. The real-time autofocus system for a microscope according to claim 3, wherein: The focus control module is a motor controller, which determines the adjustment feedback amount of the microscope objective lens through a proportional-integral-differential controller.

5. The real-time autofocus system for a microscope according to claim 4, wherein: The adjustment feedback amount of the microscope objective is calculated by the following formula: F(t)=K P ·δ(t)+K I ∫δ(t)+K D δ′(t), where K P , K I , K D represents the coefficient of the corresponding term, and δ(t) is the defocus amount of the microscope system under the current field of view calculated in real time.

6. The real-time autofocus system for a microscope according to claim 1, wherein: The infrared light source module uses a near-infrared or infrared wavelength laser with a wavelength range of 650nm to 1100nm and a power less than 5mW.

7. The real-time autofocus system for a microscope according to claim 1, wherein: The beam splitter is a beam splitter, a D-shaped reflector or a reflector with a knife edge.

Images