Microscope automatic focusing method and system based on image processing

Abstract

The invention discloses a microscope automatic focusing method and system based on image processing, an image is regarded as integration of local areas, the local areas are called windows or image blocks, the size of each local area is 5 pixels * 5 pixels, an evaluation function calculation formula independently acts on each window, a central pixel point is taken as a reference, and the central pixel point is taken as a reference point. Gray gradient difference values after absolute values are obtained through calculation in the X-axis direction and the Y-axis direction respectively, gray gradient difference values after absolute values are obtained through calculation in the main diagonal direction and the secondary diagonal direction respectively and serve as weight factors, and then the gray gradient difference values obtained through calculation are subjected to weighting operation; and finally, accumulating the calculated values of the windows to obtain a definition quantized value of the image. The problems that a traditional method is time-consuming in focusing and insufficient in accuracy are solved.

Description

Technical Field

[0001] The invention belongs to the technical field of microscope focusing, and in particular relates to an image processing-based microscope automatic focusing method and a system thereof. Background Art

[0002] As an effective tool for observing the microscopic world, optical microscopes are widely used in biomedicine and scientific research. With the development of automation technology, the development of automatic detection systems can significantly reduce the workload of manual operations, improve detection efficiency and accuracy, and save a lot of time. In computer vision automatic detection systems, realizing automatic focusing of microscopes is one of the most important key technologies.

[0003] For optical microscopes, autofocus refers to the process by which the system adjusts the mechanical structure (such as image distance or object distance) to make the collected image the clearest. Compared with ordinary light microscopes, fluorescence microscopes use more powerful light sources and filter light through two sets of filters: one set is used to control the light reaching the sample, and the other set is used to filter the fluorescent signal from the sample. Through specially designed dichroic mirrors, objective lenses and filters, the light beam reflected from the surface is collected into the CMOS camera for fluorescence detection.

[0004] Since fluorescence microscopy requires scanning multiple images within the sample volume, this process is time-consuming and increases the risk of sample photobleaching, photodamage, or phototoxicity. Therefore, fluorescence microscopy autofocus must be able to quickly determine the optimal focus plane for each field of view (FOV) in "real time" during image acquisition.

[0005] Autofocus based on image processing is to process the sequence of images collected by the image sensor in real time to obtain an evaluation value that can represent the image clarity. The clarity of the image is determined based on the evaluation value, and a corresponding feedback signal is given to drive the motor to control the movement of the stage until the clearest image is obtained, completing the autofocus.

[0006] Image analysis-based passive focusing relies on image analysis algorithms. This approach is often directly adaptable to existing microscope systems and is relatively low cost, making it more popular. Passive focusing can be divided into three main categories:

[0007] (1) Autofocus based on phase detection; the phase detection method is only applicable to a small number of wide-field microscopes.

[0008] (2) Depth estimation based on defocus (DFD): Depth-based defocus methods require prior knowledge of a clear image of the focused object, compare the current image with the clear image, and estimate the defocus parameters. Another DFD method used in microscopy extracts defocus information through deep learning methods, which has low versatility and is only applicable to specific targets.

[0009] (3) Depth-from-focus (DFF). The depth estimation method based on focusing, Depth-from-focus, determines whether the focus is completed by analyzing the clarity of the image. When the maximum value of the image clarity is reached, the focus is considered to be completed. Therefore, the clarity evaluation function (CEF) is the focus of research in the autofocus technology based on image processing.

[0010] Selection of clarity evaluation function: There is a dependency between image clarity and depth of field change or focal length change. This relationship is called clarity function, which is characterized by the presence of a maximum value at the image focus point. Existing clarity evaluation functions include statistical, frequency domain, and spatial domain types:

[0011] Statistical functions include contrast statistics, image information entropy, and Vollath cross-correlation function. These functions all obey Shannon's theorem. When the function value is larger, it indicates that the image has more content information, which can be used as an indicator to judge the clarity of the image. This type of function has good unbiasedness, but the calculation process is complicated, the sensitivity is low, and the trend of the function curve is too flat, the extreme value is not obvious enough, and it is easy to skip the best position in the subsequent focal plane search process.

[0012] Frequency domain functions mainly include discrete Fourier transform (DFT) and discrete wavelet transform. The image is converted from the time domain to the frequency domain through DFT transform or wavelet transform. The more high-frequency information the image has in the frequency domain, the richer the edge information of the ROI area of ​​the image, and the higher the clarity of the image. This type of function has high sensitivity and can well separate high-frequency information from low-frequency information. Although it is feasible to determine the clarity of the image by the richness of high-frequency information, some impulse noise is also high-frequency, which will disturb the original real image high-frequency information. In addition, the calculation is relatively complex and the real-time performance is poor, so it is rarely used in practice.

[0013] The spatial domain class function evaluation criteria focuses on using the first and second order differentials of the image and extracting edge gradient information using differential operators such as Roberts, Sobel, and Laplacian operators. Compared with blurry images, clear images usually have larger grayscale differences, resulting in higher variance, so the variance (VAR) can be used as CEF. Images with sharper edges are considered to be clearer. Several CEFs have been proposed, including summed gradient (GRA), summed Laplace gradient (LAP), squared gradient (SG), Brenner gradient, and Tenengrad gradient. However, traditional CEFs are not ideal solutions for processing various types of sample images, because the spatial or frequency domain selection of all CEFs is fixed, and their properties cannot be changed, and the accuracy will be reduced.

[0014] Another key technology of the autofocus system is the focus search strategy. In the autofocus system based on digital image processing, the focus search is the process of finding the peak on the autofocus evaluation function curve. The rapid and accurate determination of the focus position determines the accuracy and real-time performance of the autofocus. However, the existing methods have a slow execution speed due to the large number of searches and are easily affected by local extreme points. Summary of the invention

[0015] One of the purposes of the present invention is to provide a microscope autofocus method based on image processing, which improves and optimizes the image clarity evaluation function and the autofocus extreme value search algorithm, aiming to solve the problem that the traditional method is time-consuming and inaccurate in focusing.

[0016] Firstly, a focusing evaluation function that is more suitable for fluorescence microscopy images is proposed. Compared with other spatial domain functions, the proposed function has higher sensitivity and stability, better unimodality and unbiasedness, and can process images faster.

[0017] Secondly, in order to avoid the interference of local extrema on the focusing curve on autofocus, we proposed an improved hill climbing algorithm to achieve accurate search for microscope autofocus. The simulation results show that this method has good effect in the field of microscope autofocus and can meet the needs of microscope autofocus.

[0018] The specific technical solutions adopted by the present invention to achieve the above-mentioned purpose are as follows:

[0019] An image processing-based microscope autofocus method regards an image as an integration of local areas, which are called windows or tiles. The size of each local area is 5 pixels × 5 pixels. The evaluation function calculation formula will act independently on each window. Taking the central pixel as the reference, the grayscale gradient difference after taking the absolute value is calculated in the X-axis and Y-axis directions respectively. At the same time, the grayscale gradient difference after taking the absolute value is calculated in the main diagonal and sub-diagonal directions and used as a weight factor. Then, the grayscale gradient differences obtained by these calculations are weighted. Finally, the calculated values ​​of each window are accumulated to obtain the image clarity quantization value.

[0020] The grayscale gradient evaluation function applies the specified convolution template to the image grayscale to characterize the clarity of the image.

[0021] Assume that the size of image I is M×N, the gray value of a pixel (x, y) in image I is f(x, y), the quantitative value of image clarity evaluation is F, and the focus evaluation function expression is as follows:

[0022] At x = 5k-2, k∈N * , y=5k-2,k∈N * Under the premise, where N* is a set of positive integers, k is a set N * An element value in

[0023]

[0024] In the evaluation function expression: F proposed That is, the evaluation function. The subscript proposed is used to indicate a new evaluation function that is different from other existing evaluation functions. M is the number of pixels in the horizontal direction of the image, N is the number of pixels in the vertical direction of the image, x is the horizontal coordinate of the center pixel of the window, and y is the vertical coordinate of the center pixel of the window. and is the basic value of grayscale gradient statistics, which is the grayscale value of the new pixel after convolution of the central grayscale pixel and its surrounding pixels at 1 (INT) or 2 (EXT) positions on the X-axis or Y-axis with the corresponding differential operator. The subscripts X and Y respectively indicate that the window is currently operating on the X-axis and Y-axis. The superscript INT indicates that the current central pixel of the window is operating with the pixel at 1 adjacent position. The superscript EXT indicates that the current central pixel of the window is operating with the pixel at 2 adjacent positions.

[0025]

[0026] In the above formula, f(x, y) represents the grayscale value of the pixel at the coordinate (x, y), and the symbol is the convolution operator, and the differential operator matrix in the formula is as follows:

[0027]

[0028] Among them, m, n=1, 2, 3, 4, 5, δ ij It is a binary function used to simplify the differential operator matrix. Its independent variable, i.e., input value, is generally two integers. If the two are equal, its output value is 1, otherwise it is 0.

[0029]

[0030] and is the weight factor of grayscale gradient statistics, which is the grayscale value of the new pixel after convolution of the original grayscale pixel and the adjacent 1 (INT) or 2 (EXT) pixels on the main diagonal (main-diag) or anti-diag with the corresponding differential operator. The subscripts main-diag and anti-diag indicate that the window is currently operating on the main diagonal and anti-diag, respectively.

[0031]

[0032]

[0033] The differential operator matrix in the formula is as follows:

[0034]

[0035] The core of the evaluation function proposed by the present invention is:

[0036] The image is regarded as an integration of local areas. The local areas can be called windows or tiles. The size of each local area is 5 pixels × 5 pixels. The function calculation formula will act independently on each window. Taking the central pixel as the reference, the grayscale gradient difference S after taking the absolute value is calculated in the X-axis and Y-axis directions respectively. At the same time, the grayscale gradient difference after taking the absolute value is calculated in the main diagonal and sub-diagonal directions and used as the weight factor W. Then these calculated values ​​are weighted. Finally, the calculated values ​​of each window are accumulated to obtain the image clarity quantization value F.

[0037] It is not difficult to find that in the evaluation function of the present invention, the 16 pixels adjacent to the central pixel are all involved in the calculation process of the function. Therefore, the proposed function fully considers the anisotropy of the edge direction of the ROI region in the fluorescent image, weakens the sensitivity of the function to noise, and thus improves the sensitivity of the function. In addition, the concept of image windowing abandons the lengthy calculation method in the traditional method, so that the real-time performance of the function is further enhanced.

[0038] In the present invention, the basic value of grayscale gradient statistics and the calculation formula of weight factor are all processed by taking absolute values, which is to avoid the negative term offsetting the accumulated quantized value of the positive term in the process of gradient statistics of the whole image. The negative term itself is also part of the grayscale gradient, and not taking the absolute value will weaken the sensitivity of the whole function. This point is not fully considered in many traditional focus evaluation functions.

[0039] The present invention also proposes an improved focus search strategy, which divides the focus search process into two search stages: coarse adjustment and fine adjustment. The purpose of the coarse adjustment search stage is to quickly find the peak point range, and this stage still uses the traditional hill climbing search algorithm. In the fine adjustment search stage, an improved hill climbing algorithm is used, and the stepper motor drives the stage to move slightly. At this time, the image grayscale changes little, and the evaluation function is easily disturbed by external environmental noise and local extreme values ​​appear. Therefore, this stage starts to shoot three images continuously to obtain their evaluation values ​​to determine the search direction. The peak point range can be quickly determined through the initial large step search, and then in the small step search stage, three images are selected to be shot continuously to calculate their clarity evaluation values ​​to determine the search direction, which can effectively avoid falling into the local extreme value situation.

[0040] The basic steps for focus search are as follows:

[0041] S1: Determine the moving step of the Z-axis stage by calculating the depth of field D at the current objective magnification, initialize the clarity value at the current position, and initialize the moving direction.

[0042] S2: Enter the coarse adjustment search stage and move the stage in steps of 2D.

[0043] S3: Obtain the current image and evaluate the clarity.

[0044] S4: Update the old clarity value with the current position and clarity.

[0045] S5: Determine whether the clarity value at the current position is less than the clarity value corresponding to the previous position.

[0046] If the clarity value at the current position is less than the clarity value corresponding to the previous position, it indicates that the possible best focus position has been exceeded. The stage needs to continue to move in steps of 2D to collect the clarity value corresponding to the next position. If the clarity value corresponding to the next position is less than the clarity value at the current position, go to step S6; otherwise, continue to execute steps S2 to S5 in sequence.

[0047] S6: The stage retreats and enters the fine search stage.

[0048] S7: Move the stage in smaller steps of D for fine search, and continuously capture three images to obtain their clarity evaluation values F.

[0049] S9: Compare the clarity evaluation values F1, F2, and F3 at three consecutive positions to determine the next moving direction.

[0050] If F1 < F2 < F3, it indicates that the current is in the stage of increasing clarity value, and continue to move the stage in the current direction and enter step S10; if F1 > F2 and F2 < F3, it indicates that the middle value is a local minimum and can be ignored, and continue to move in the current direction and enter step S10; if F1 < F2 and F2 > F3, it indicates that the middle value is a possible local maximum, but it cannot be determined as the global maximum, and further exploration is needed. Move in the current direction and enter step S10; if F1 > F2 > F3, it indicates that the maximum point has been passed, and the direction needs to be changed and fine-tuned to accurately find the maximum point, and enter step S11.

[0051] S10: Move the stage in the original direction in steps of D, obtain the current new image, evaluate the clarity, and update the clarity evaluation values F1, F2, and F3, and enter step S9 for comparison.

[0052] S11: moving the stage in the opposite direction with a step length of 0.5D, continuously taking three images and obtaining updated clarity evaluation values ​​Fb1, Fb2 and Fb3;

[0053] S12: comparing the clarity evaluation value Fb1, the clarity evaluation value Fb2 and the clarity evaluation value Fb3 of three consecutive positions;

[0054] If Fb1>Fb2>Fb3, it is considered that the global maximum point Fb1 is found, and step S13 is executed; otherwise, step S11 and step S12 are continued in sequence;

[0055] S13: Move the stage to the currently considered global maximum point Fb1, successfully find the focal plane and exit.

[0056] The present invention adopts an optimized search strategy: a phased coarse-adjustment and fine-adjustment search algorithm is used to quickly locate the peak point range, and the search direction is accurately determined by continuously shooting three images, which can effectively avoid the influence of local extreme values.

[0057] The second object of the present invention is to provide a microscope focusing system, which adopts the above-mentioned focusing method. The system is mainly composed of the following four modules: image acquisition, image analysis, stage control and result analysis. Among them, the image acquisition module acquires images through a USB camera connected to an intelligent fluorescent microscope; the image analysis module receives and processes the images and calculates the clarity; the stage control module controls the movement of the Z-axis stage through serial communication to adjust its position according to the results of the image analysis; the result analysis module determines whether the image has reached the clearest state, and gives a feedback signal according to the search strategy until the focus is successful.

[0058] The focusing system of the present invention is developed using C++ language, QT framework and OpenCV image processing software library, which can realize automated focusing process, integrate complex steps, greatly reduce the time and effort of manual operation, and make the experimental tool more efficient and reliable.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The present invention introduces a focus evaluation function designed for microscopic images, which is particularly suitable for fluorescence microscopy and has higher sensitivity, stability and unimodality, ensuring fast and accurate image processing and enhancing the adaptability of the microscope.

[0061] 2. The present invention optimizes the search strategy, adopts a phased coarse-adjustment and fine-adjustment search algorithm, quickly locates the peak point range, and accurately determines the search direction by continuously shooting three images, effectively avoiding the influence of local extreme values.

[0062] 3. The focusing system of the present invention can realize an automated focusing process, integrate complex steps, significantly reduce the time and effort of manual operation, and make the experimental tool more efficient and reliable. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The present invention is a workflow diagram of a microscope auto-focusing method based on image processing;

[0064] Figure 2 The present invention is a framework diagram of an image processing-based microscope autofocus system. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0066] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0067] It should be understood that the present invention can be implemented in different forms and should not be interpreted as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to make the disclosure thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0068] An image processing-based microscope autofocus method regards an image as an integration of local areas, which are called windows or tiles. The size of each local area is 5 pixels × 5 pixels. The evaluation function calculation formula will act independently on each window. Taking the central pixel as the reference, the grayscale gradient difference after taking the absolute value is calculated in the X-axis and Y-axis directions respectively. At the same time, the grayscale gradient difference after taking the absolute value is calculated in the main diagonal and sub-diagonal directions and used as a weight factor. Then, the grayscale gradient differences obtained by these calculations are weighted. Finally, the calculated values ​​of each window are accumulated to obtain the image clarity quantization value.

[0069] The grayscale gradient evaluation function applies the specified convolution template to the image grayscale to characterize the clarity of the image.

[0070] Assume that the size of image I is M×N, the gray value of a pixel (x, y) in image I is f(x, y), the quantitative value of image clarity evaluation is F, and the focus evaluation function expression is as follows:

[0071] At x = 5k-2, k∈N * , y=5k-2,k∈N * Under the premise, where N * is a set of positive integers, k is a set N * An element value in

[0072]

[0073] In the evaluation function expression: F proposed That is, the evaluation function. The subscript proposed is used to indicate a new evaluation function that is different from other existing evaluation functions. M is the number of pixels in the horizontal direction of the image, N is the number of pixels in the vertical direction of the image, x is the horizontal coordinate of the center pixel of the window, and y is the vertical coordinate of the center pixel of the window. and is the basic value of grayscale gradient statistics, which is the grayscale value of the new pixel after convolution of the central grayscale pixel and its surrounding pixels at 1 (INT) or 2 (EXT) positions on the X-axis or Y-axis with the corresponding differential operator. The subscripts X and Y respectively indicate that the window is currently operating on the X-axis and Y-axis. The superscript INT indicates that the current central pixel of the window is operating with the pixel at 1 adjacent position. The superscript EXT indicates that the current central pixel of the window is operating with the pixel at 2 adjacent positions.

[0074]

[0075] In the above formula, f(x, y) represents the grayscale value of the pixel at the coordinate (x, y), and the symbol is the convolution operator, and the differential operator matrix in the formula is as follows:

[0076]

[0077] Among them, m, n=1, 2, 3, 4, 5, δ ij It is a binary function used to simplify the differential operator matrix. Its independent variable, i.e., input value, is generally two integers. If the two are equal, its output value is 1, otherwise it is 0.

[0078]

[0079] and is the weight factor of grayscale gradient statistics, which is the grayscale value of the new pixel after convolution of the original grayscale pixel and the adjacent 1 (INT) or 2 (EXT) pixels on the main diagonal (main-diag) or anti-diag with the corresponding differential operator. The subscripts main-diag and anti-diag indicate that the window is currently operating on the main diagonal and anti-diag, respectively.

[0080]

[0081]

[0082] The differential operator matrix in the formula is as follows:

[0083]

[0084] As a preferred embodiment, the focus search process is divided into two search stages: coarse adjustment and fine adjustment. The purpose of the coarse adjustment search stage is to quickly find the peak point range, and this stage still uses the traditional hill climbing search algorithm. In the fine adjustment search stage, an improved hill climbing algorithm is used, and the stepper motor drives the stage to move slightly. At this time, the image grayscale changes little, and the evaluation function is easily disturbed by external environmental noise and has local extreme values. Therefore, this stage starts with taking three images continuously to obtain their evaluation values ​​to determine the search direction. The peak point range can be quickly determined through the initial large step search, and then in the small step search stage, three images are taken continuously to calculate their clarity evaluation values ​​to determine the search direction, which can effectively avoid falling into the local extreme value situation.

[0085] like Figure 1 As shown, the basic steps of focus search are as follows:

[0086] S1: Determine the Z-axis stage moving step length by obtaining the depth of field D of the current objective lens magnification, initialize the clarity value of the current position, and initialize the moving direction;

[0087] S2: Enter the coarse search phase and move the stage with a step size of 2D;

[0088] S3: Get the current image and evaluate its clarity;

[0089] S4: Update the old definition value with the current position and definition;

[0090] S5: Determine whether the clarity value of the current position is less than the clarity value corresponding to the previous position;

[0091] If the clarity value at the current position is less than the clarity value corresponding to the previous position, it indicates that the possible best focus position has been exceeded. The stage needs to continue moving the stage in steps of 2D to collect the clarity value corresponding to the next position. If the clarity value corresponding to the next position is less than the clarity value at the current position, go to step S6; otherwise, continue to execute steps S2 to S5 in sequence;

[0092] S6: The stage retreats and enters the fine search stage;

[0093] S7: Move the stage in steps of a smaller step size D for fine search, and continuously capture three images to obtain their clarity evaluation values F;

[0094] S9: Compare the clarity evaluation values F1, F2, and F3 at three consecutive positions to determine the next moving direction;

[0095] If F1 < F2 < F3, it indicates that the current is in the stage of increasing clarity value, and continue to move the stage in the current direction and go to step S10; if F1 > F2 and F2 < F3, it indicates that the middle value is a local minimum and can be ignored, and continue to move in the current direction and go to step S10; if F1 < F2 and F2 > F3, it indicates that the middle value is a possible local maximum, but it cannot be determined as the global maximum, and further exploration is needed. Move in the current direction and go to step S10; if F1 > F2 > F3, it indicates that the maximum point has been passed, and the direction needs to be changed and fine-tuned to accurately find the maximum point, and go to step S11;

[0096] S10: Move the stage in the original direction in steps of D, obtain the current new image, evaluate the clarity, and update the clarity evaluation values F1, F2, and F3, and enter step S9 for comparison;

[0097] S11: Move the stage in the reverse direction in steps of 0.5D, continuously capture three images, and obtain the updated clarity evaluation values Fb1, Fb2, and Fb3;

[0098] S12: Compare the clarity evaluation values Fb1, Fb2, and Fb3 at three consecutive positions;

[0099] If Fb1 > Fb2 > Fb3, it is considered that the global maximum point Fb1 has been found, and execute step S13; otherwise, continue to execute steps S11 and S12 in sequence;

[0100] S13: Move the stage to the position of the currently considered global maximum point Fb1, successfully find the focal plane, and exit.

[0101] Next, a systematic test plan is designed and implemented to verify the technical effect of the microscope autofocus method.

[0102] Experimental Example 1 compares the performance of the improved function and the traditional function in the focus evaluation of the fluorescent image, so as to evaluate whether the newly proposed focus evaluation function (proposed function) can provide higher clarity, sensitivity, unimodality and processing speed. The Brenner function, Roberts function, SMD (Sum of Modified Differences) function, EOG (Energy of Gradient) function, Sobel function, Brenner2d function, improved-Tenengrad function, Bre2d_Rob function, SMD_MRob function and the proposed function of the present invention are specially selected for experimental testing. Through a series of quantitative and qualitative analyses, it is proved that the improved function can significantly improve the accuracy and efficiency of the autofocus system.

[0103] Experimental Example 2 verifies the application effect of the improved hill climbing algorithm in microscope autofocus, especially the algorithm's ability to quickly locate the peak point range through a staged coarse and fine adjustment search strategy, and detects whether the algorithm can effectively avoid the influence of local extreme values, thereby improving the accuracy and efficiency of focusing.

[0104] Experimental preparation: image dataset construction.

[0105] In order to ensure the reliability and wide applicability of the experimental results, a diverse sample image library with different step length conditions was constructed. The dataset was taken by an intelligent fluorescence microscope and covers the following settings:

[0106] Step size: Three different stage movement step sizes of 2D, D and 0.5D are used.

[0107] Number of samples: 12 independent sets of sample data were taken under each step length condition.

[0108] Content of each sample group: Each sample group includes 21 consecutive fluorescence images, which gradually change from a defocused state to the clearest state (the 11th image is in focus), and then gradually return to a defocused state.

[0109] Experimental instrument: intelligent fluorescence microscope.

[0110] The microscope used in this experiment is equipped with a motorized stage, which allows the sample to be moved accurately in set steps, thereby simulating various focusing requirements that may be encountered in actual application scenarios. The motorized objective turret allows the host computer to switch between different objective lenses to calculate the depth of field D and thus determine the Z-axis stage movement step. At the same time, the microscope has a built-in high-performance camera to ensure the acquisition of high-quality fluorescence images.

[0111] Experimental Example 1:

[0112] The performance of different functions is tested using different image data sets, including the clarity ratio R, sensitivity S, number of local peaks α, focus search interval β, and algorithm time t.

[0113] 1. Preprocessing stage

[0114] Calibrate the microscope to ensure it is in optimal working condition. Set the experimental parameters, including step length, exposure time, etc.

[0115] 2. Image acquisition

[0116] For each sample, the stage is moved sequentially at a specified step length and a fluorescence image is taken at each position. This ensures consistent lighting conditions before and after each shot to reduce the impact of external factors on the experimental results.

[0117] 3. Data processing

[0118] The evaluation value of each image is calculated using the focus evaluation function mentioned above. For each set of samples, the following five key performance indicators are recorded and analyzed:

[0119] Clarity ratio R: The higher the clarity ratio, the easier it is for the evaluation function to distinguish between in-focus images and out-of-focus images.

[0120] Sensitivity S: reflects the sensitivity of a function to small changes.

[0121] Number of local peaks α: Count the number of local extreme points.

[0122] Focus search interval β: Determines the effective search range.

[0123] Algorithm time t: measures the time required to calculate the evaluation value.

[0124] 4. Data Analysis

[0125] All data is organized into a table to facilitate intuitive comparison of the performance of different functions.

[0126] Calculate the average value of each indicator under each step condition to evaluate the consistency and stability of the function. Conclusion of Experimental Example 1:

[0127] By quantifying it according to the performance evaluation indicators, we can get the following table:

[0128]

[0129] Through a detailed analysis of a large amount of experimental data under different step length conditions, the following conclusion can be drawn: the focus evaluation function of the present invention shows better performance in most cases, especially in terms of clarity ratio, sensitivity and processing speed.

[0130] The evaluation function of the present invention can effectively reduce the number of local peaks and improve the accuracy of focus search. The function has strong anti-interference ability and high stability.

[0131] In general, the evaluation function of the present invention not only improves the speed and accuracy of auto-focusing, but also enhances the stability and robustness of the system.

[0132] Experimental results show that the evaluation function of the present invention is particularly suitable for fluorescence microscopy applications requiring high resolution and fast processing.

[0133] Experimental Example 2:

[0134] Three samples A, B, and C corresponding to a certain image are selected to represent light, medium, and severe defocus states, respectively, to facilitate the simulation algorithm's logic of changing the step size during the focusing process.

[0135] Parameter setting: The initial step size is set to 2D, representing a severe defocus state.

[0136] 1. Coarse search stage: two images are continuously acquired after each movement, and their clarity evaluation values ​​are calculated.

[0137] 2. Fine-tuning search stage: When the specific conditions of the invention are met, the search direction is changed and the fine-tuning search stage is entered. The moving step size is changed to D, three images are continuously collected and the clarity evaluation value is calculated.

[0138] 3. Termination condition: If the condition is further met, the step size is reduced to 0.5D and the search direction is adjusted until the clearest image is reached and the focusing process is stopped.

[0139] Data recording and analysis: For this sample, the clarity evaluation values ​​of all images and their corresponding position information and moving steps were recorded in detail throughout the process, and the focusing speed, accuracy and stability were analyzed and compared.

[0140] The initial step size is set to 2D and the coarse adjustment search stage is directly entered. In this stage, two images are continuously collected and the clarity evaluation value is calculated. When a local maximum point is encountered when moving the third step size, the algorithm of the present invention effectively avoids the local maximum point. The fourth step size continues to collect and compare, and after determining the local maximum point, step S2 is entered;

[0141] When moving the 11th step length, a maximum point appears in the comparison, indicating that the possible best focus position has been exceeded. At this time, to avoid collecting the next image for the local extreme point and continuing the comparison, see step S5;

[0142] At this time, if the specific conditions of the present invention are met, that is, after comparison in step S5, F2 < F1 indicates that the extreme point position has been crossed. When the 13th step length is retracted and moved to the 14th step length, the search direction is changed and the fine-tuning search stage step S7 is entered. The moving step length is changed to D, and three images are continuously collected and the sharpness evaluation value is calculated;

[0143] Compare the sharpness evaluation values F1, sharpness evaluation value F2, and sharpness evaluation value F3 at three consecutive positions to determine the next moving direction. F1 > F2 > F3 indicates that the maximum point has been passed. When moving the 16th step length, change the direction and reduce the step length for fine-tuning to accurately find the maximum point, and enter step S11;

[0144] Move the stage in the reverse direction with a step length of 0.5D, continuously take three images and obtain the updated sharpness evaluation values Fb1, sharpness evaluation value Fb2, and sharpness evaluation value Fb3 until moving the 21st step length. If Fb1 > Fb2 > Fb3, it is considered that the global maximum point Fb1 has been found, and execute step S13: move the stage to the position of the currently considered global maximum point Fb1, successfully find the focal plane and exit.

[0145] The number of moving steps and the sharpness evaluation value are shown in the following table:

[0146]

[0147]

[0148]

[0149] The traditional hill climbing algorithm finds the best focus by setting the initial scanning direction and search step length, gradually acquiring images and calculating the focus evaluation function value. However, when encountering a situation where the focus evaluation value decreases, it simply reverses the direction and reduces the step length, which makes it easy to stagnate at local extreme positions such as "-14D" and misjudge it as the optimal solution, resulting in autofocus failure.

[0150] There is also a focus search method in the prior art that uses a global search with the minimum step distance. Although it can ensure finding the global optimal solution, its efficiency is low. It takes 41 steps to complete the search within 20D. In contrast, the method of the present invention improves the retrieval efficiency by about twice.

[0151] The above application experiments show that the improved hill climbing algorithm can complete focusing in a shorter time, especially under severe defocus conditions, showing significant advantages. Regardless of the initial defocus state, the algorithm can effectively avoid local extreme value interference, stably locate the global maximum clarity point, and ensure the accuracy of focusing. Compared with the traditional method, the algorithm of the present invention reduces unnecessary search times and improves overall efficiency by alternating between reasonably designed large and small step sizes.

[0152] The improved hill climbing algorithm of the present invention not only improves the speed and accuracy of microscope autofocus, but also enhances the reliability and adaptability of the system. The algorithm is an effective solution and can be applied to actual microscope autofocus systems.

[0153] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A microscope autofocus method based on image processing, characterized in that: The image is regarded as an integration of local regions, which are called windows or tiles, and each local region has a size of 5 pixels × 5 pixels. The evaluation function calculation formula will act independently on each window. Based on the central pixel point, the absolute value of the gray-scale gradient difference is calculated in the X-axis and Y-axis directions respectively. At the same time, the absolute value of the gray-scale gradient difference is calculated in the main diagonal and secondary diagonal directions respectively and used as a weight factor. Then, the weighted operation is performed on the gray-scale gradient differences obtained from these calculations. Finally, the calculated values of each window are accumulated to obtain the quantization value of the image sharpness. The gray-scale gradient evaluation function applies a specified convolution template to the image gray-scale to characterize the sharpness of the image. Assume that the size of image I is M×N, the gray-scale value of a certain pixel point (x, y) in image I is f(x, y), and the quantization value of the image sharpness evaluation is F. The expression of the focus evaluation function is as follows: At x = 5k-2, k∈N * , y=5k-2,k∈N * Under the premise, where N * is a set of positive integers, k is a set N * An element value in In the evaluation function expression: F proposed That is, the evaluation function. The subscript proposed is used to indicate a new evaluation function that is different from other existing evaluation functions. M is the number of pixels in the horizontal direction of the image, N is the number of pixels in the vertical direction of the image, x is the horizontal coordinate of the pixel in the center of the window, and y is the vertical coordinate of the pixel in the center of the window. and is the basic value of grayscale gradient statistics, which is the grayscale value of the new pixel after convolution of the central grayscale pixel and its surrounding pixels at 1 (INT) or 2 (EXT) positions on the X-axis or Y-axis with the corresponding differential operator. The subscripts X and Y respectively indicate that the window is currently operating on the X-axis and Y-axis. The superscript INT indicates that the current central pixel of the window is operating with the pixel at 1 adjacent position. The superscript EXT indicates that the current central pixel of the window is operating with the pixel at 2 adjacent positions. In the above formula, f(x, y) represents the grayscale value of the pixel at the coordinate (x, y), and the symbol is the convolution operator, and the differential operator matrix in the formula is as follows: Among them, m, n=1, 2, 3, 4, 5, δ ij It is a binary function used to simplify the differential operator matrix. Its independent variable, i.e., input value, is generally two integers. If the two are equal, its output value is 1, otherwise it is 0. and is the weight factor of grayscale gradient statistics, which is the grayscale value of the new pixel after convolution of the original grayscale pixel and the adjacent 1 (INT) or 2 (EXT) pixels on the main diagonal (main-diag) or anti-diag with the corresponding differential operator. The subscripts main-diag and anti-diag indicate that the window is currently operating on the main diagonal and anti-diag, respectively. The differential operator matrix in the formula is as follows:

2. The microscope autofocus method based on image processing according to claim 1, characterized in that: The focus search process is divided into two search stages: coarse adjustment and fine adjustment. The purpose of the coarse adjustment search stage is to quickly find the peak point range, and the traditional hill-climbing search algorithm is used in this stage; in the fine adjustment search stage, an improved hill-climbing algorithm is used. The steps of the focus search are as follows: S1: Calculate and determine the moving step of the Z-axis stage by obtaining the depth of field D of the current objective magnification, initialize the sharpness value of the current position, and initialize the moving direction. S2: Enter the coarse adjustment search stage and move the stage in steps of 2D. S3: Obtain the current image and evaluate the sharpness. S4: Update the old sharpness value with the current position and sharpness. S5: Determine whether the sharpness value of the current position is less than the sharpness value corresponding to the previous position. If the sharpness value of the current position is less than the sharpness value corresponding to the previous position, it indicates that the possible best focus position has been exceeded, and it is necessary to continue moving the stage in steps of 2D to collect the sharpness value corresponding to the next position. If the sharpness value corresponding to the next position is less than the sharpness value of the current position, go to step S6; otherwise, continue to execute steps S2 to S5 in sequence. S6: The stage retreats and enters the fine search stage. S7: Move the stage with a smaller step D for fine search, and continuously capture three images to obtain their sharpness evaluation values F. S9: Compare the sharpness evaluation values F1, F2, and F3 of three consecutive positions to determine the next moving direction. If F1 < F2 < F3, it indicates that the current is in the stage of increasing sharpness value, and continue to move the stage in the current direction and enter step S10; if F1 > F2 and F2 < F3, it indicates that the middle value is a local minimum, ignore it and continue to move in the current direction and enter step S10; if F1 < F2 and F2 > F3, it indicates that the middle value is a possible local maximum, but it cannot be determined as the global maximum, and further exploration is needed. Move in the current direction and enter step S10; if F1 > F2 > F3, it indicates that the maximum point has been passed, and it is necessary to change the direction and perform fine adjustment to accurately find the maximum point, and enter step S11. S10: Move the stage in the original direction with a step length D, obtain the current new image, evaluate the clarity and update the clarity evaluation value F1, the clarity evaluation value F2 and the clarity evaluation value F3, and proceed to step S9 for comparison; S11: moving the stage in the opposite direction with a step length of 0.5D, continuously taking three images and obtaining updated clarity evaluation values ​​Fb1, Fb2 and Fb3; S12: comparing the clarity evaluation value Fb1, the clarity evaluation value Fb2 and the clarity evaluation value Fb3 of three consecutive positions; If Fb1>Fb2>Fb3, it is considered that the global maximum point Fb1 is found, and step S13 is executed; otherwise, step S11 and step S12 are continued in sequence; S13: Move the stage to the currently considered global maximum point Fb1, successfully find the focal plane and exit.

3. A focusing system using the microscope automatic focusing method based on image processing according to claim 1 or 2, characterized in that: It includes an image acquisition module, an image analysis module, a stage control module and a result analysis module.

4. The focusing system according to claim 3, characterized in that: The image acquisition module acquires images through a USB camera connected to the intelligent fluorescent microscope; the image analysis module receives and processes the images and calculates the clarity; the stage control module controls the movement of the Z-axis stage to adjust its position through serial communication according to the results of the image analysis; The result analysis module determines whether the image has reached the clearest state, and gives a feedback signal according to the search strategy until the focus is successful.

5. The focusing system according to claim 4, characterized in that: The focusing system uses C++ language, QT framework and OpenCV image processing software library.

Images