Target detection-based microscope dynamic automatic focusing method and system

Abstract

The invention discloses a microscope dynamic automatic focusing method and system based on target detection, and relates to the field of microscopic imaging and automatic control. Comprising the following steps: acquiring a real-time view field image, and acquiring a view field image of a microscope in real time; and detecting by using a target detection model, and processing the real-time view field image by using a target detection model finely adjusted by a microscopic image data set so as to identify at least one target object in real time and output bounding box and confidence information of the target object. The system has the advantages that microscopic images are acquired in real time through the image acquisition module; a real-time image is analyzed by using a YOLOv8 target detection model subjected to fine adjustment of a specific microscopic image data set so as to dynamically identify and position a target sample in a visual field, and one or more dynamic regions of interest (ROI) are generated according to a preset rule, so that the intellectualization and real-time performance of a focusing process are realized, and the focusing efficiency is improved. And the focusing efficiency, the focusing precision and the adaptability to a dynamic sample are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of microscopic imaging and automatic control, in particular to a microscope dynamic automatic focusing method and system based on target detection. BACKGROUND

[0002] Microscope automatic focusing technology is the key in the field of microscopic imaging automation. In the prior art, there are several mainstream automatic focusing methods, but each has its limitations.

[0003] A common method is the sharpness evaluation based on image processing. For example, Chinese patent CN2023113837079 discloses an evaluation function based on gradient method, but such a single evaluation function lacks robustness when facing different samples and noise environments. More importantly, such methods face serious challenges when dealing with scenes containing moving targets. For example, when observing living cells in a culture dish, cells will constantly drift and move. If the traditional strategy of focusing on the whole image or the fixed area in the center of the image is adopted, as soon as the target cell moves out of this area, the focusing system will "lose the target" and focus on the background or other irrelevant objects, resulting in focusing failure. This makes long-term, unattended dynamic observation of living cells almost impossible.

[0004] Another method is proposed for large field scanning (WSI). For example, Chinese patent CN202410117395 proposes a static field locking method based on microsphere coordinate calibration, which is essentially still based on historical data or preset path scanning. Such methods are completely unsuitable for observing dynamic samples such as living cells, because the path planned based on past data or fixed markers cannot adapt to the real-time movement and changes of the sample, and its application scenarios are extremely limited.

[0005] In addition, there are also active ranging schemes, such as the grating projection-based ranging method disclosed in US20140168404A1. Although this method has high accuracy, it requires invasive modification of the microscope and integration of a dedicated optical module, which not only significantly increases hardware costs, but also damages the original structure of the microscope, making it less versatile.

[0006] In summary, the prior art either lacks robustness, or relies on static data, or has strong hardware invasiveness, especially in the face of real-time, accurate, non-invasive, continuous tracking focusing of moving targets, there is still a large technical gap and challenge. Therefore, we propose a microscope dynamic automatic focusing method and system based on target detection. SUMMARY

[0007] The main objective of this invention is to overcome the shortcomings of the prior art and provide a systematic and intelligent method and system for dynamic autofocusing of microscopes. This method does not rely on historical data, has high computational efficiency, and can track and focus on dynamic targets in real time.

[0008] A target detection-based dynamic autofocusing method for microscopes includes the following steps:

[0009] Acquire real-time field-of-view images; capture a single field-of-view image from a microscope in real time.

[0010] The real-time field-of-view image is processed using a target detection model finely tuned to a microscopic image dataset to identify at least one target object in real time and output its bounding box and confidence information. The acquired image is then fed into the YOLOv8 model for target detection. If no target is detected, the process returns to step two (acquiring the image and performing target detection) and reports an error; if a target is detected, the process continues to the next step.

[0011] The confidence score here is a value between 0 and 1, representing the model's degree of certainty that a target (such as a cell or lesion area) exists within the detection box, and a comprehensive assessment of the accuracy of the bounding box position. It is used to filter valid detection results and assist in focusing decisions. Essentially, confidence score is a quantitative indicator of the model's reliability of the prediction results, containing two meanings: 1) Target existence probability: Whether a target (such as a cell structure) actually exists within the box; 1 if it exists, 0 otherwise. 2) Position accuracy: The intersection-over-union (IoU) ratio between the predicted box and the true target box, measuring the accuracy of the bounding box positioning. In autofocus, high confidence scores can serve as a basis for judging the sharp focus area, assisting the system in locking the optimal focal length. In this program, the predicted box is generated from the prior box using offset calculations. The predicted box is used to predict the position and size of the target (e.g., a cell or cell cluster). The true target box is the actual location (the actual location of the cell or cell cluster) in the labeled data.

[0012] Determine a dynamic ROI by selecting a target from the at least one target object according to a preset rule, and using its bounding box as a dynamic region of interest (ROI).

[0013] Calculate the composite sharpness value, and calculate the sharpness value of the real-time field of view image only within the dynamic region of interest (ROI);

[0014] A three-stage search algorithm is executed. Based on the sharpness value, a focusing actuator is driven by a preset three-stage search algorithm to adjust the focus of the microscope until the sharpness value reaches the optimal state. When executing the three-stage search algorithm, if it is in the coarse search stage, a large step search is performed, data points are recorded, and the fine search interval is determined; if it is in the fine search stage (entered after the interval is determined by the coarse search), a small step search is performed, and five data points are recorded.

[0015] Preferably, the target detection model is a target detection model from the You Only Look Once (YOLO) series, capable of quickly detecting targets in an image and outputting information such as the target's bounding box and confidence score. Specifically, the YOLOv8 model is a target detection model from the You Only Look Once (YOLO) series, capable of quickly detecting targets in an image and outputting information such as the target's bounding box and confidence score. ROI (Region of Interest) is a specific region defined in an image for subsequent analysis (such as calculating sharpness values), which can reduce computational load and improve targeting. Composite sharpness value: a comprehensive index used to measure the sharpness of a specific region (ROI) in an image, calculated through a certain algorithm. During focusing, the optimal focus position is determined by comparing the composite sharpness values ​​at different locations.

[0016] Preferably, the preset rule is: selecting the target with the highest confidence score among the at least one target object as the target. This rule is: selecting the target with the highest confidence score among all candidate targets and using its bounding box as the dynamic ROI of the current frame. In another alternative embodiment, this rule could also be selecting the target closest to the image center, or selecting the target with the largest area.

[0017] Preferably, the dynamic region of interest (ROI) is a specific region defined in the image for subsequent analysis (such as calculating sharpness values), which can reduce the amount of computation and improve the targeting.

[0018] Preferably, the composite sharpness value is a comprehensive index used to measure the sharpness of a specific region (ROI) in an image, calculated by a certain algorithm, and the optimal focus position is determined by comparing the composite sharpness values ​​at different positions during the focusing process.

[0019] Preferably, in the calculation of the composite sharpness value, the sharpness value is calculated by a composite sharpness evaluation function that combines the Tenengrad gradient function and the improved Laplacian function (SML), and the weight factor α of the composite sharpness evaluation function is experimentally calibrated and its value ranges from 0.5 to 0.7.

[0020] The calculation formula for the composite sharpness evaluation function is as follows:

[0021] F composite =α*F Tenengrad +(1-α)*F SML

[0022] Where α is the optimal weighting factor.

[0023] This invention conducts the following calibration experiment: Ten typical cell image sequences were selected, each sequence containing images of the entire process from blurry to clear and then back to blurry. α values ​​were set to 0.1, 0.2, ..., 0.9, and the sharpness curves for each sequence at different α values ​​were calculated. By evaluating the curves' unimodality, noise-free properties, peak sensitivity, and other indicators, it was found that the function exhibits the best overall performance when α is within the range of 0.5 to 0.7.

[0024] Preferably, in the execution of the three-stage search algorithm, the preset search algorithm is a three-stage search process, which includes:

[0025] A1: Drive the focusing actuator with a large step size to perform a rapid coarse search;

[0026] A2: Within the peak range determined by the coarse search, the focusing actuator is driven with small steps to perform a fine search to collect multiple discrete position and sharpness data points;

[0027] A3: Perform a quadratic polynomial function fitting on the discrete position and sharpness data points to calculate the theoretical optimal focus position, and drive the focusing actuator to move to that position.

[0028] A target detection-based dynamic autofocus system for microscopes includes: an image acquisition module, a focusing execution module, and a control processing module.

[0029] Image acquisition module: used to acquire field-of-view images of the microscope in real time;

[0030] Focusing execution module: used to drive the focusing knob of the microscope;

[0031] Control processing module: The control processing module is electrically connected to the image acquisition module and the focus execution module.

[0032] Preferably, the image acquisition module consists of a CMOS camera or an image acquisition device with equivalent performance, which is installed on the observation tube of the XSP-200D biological microscope.

[0033] Preferably, the control processing module consists of a host computer and a slave computer working together to form a two-layer control architecture. The host computer is a personal computer responsible for running the core algorithm; the slave computer is an STM32F103C8T6 microcontroller minimum system board responsible for executing hardware control. The two communicate via a USB-to-TTL serial port module. The host computer software platform is developed based on Python 3.8 and core libraries such as PyQt5, OpenCV, PyTorch, PySerial, and Matplotlib (see reference). Figure 3 The YOLOv8 model used in this invention was fine-tuned based on the official pre-trained model using a dataset of approximately 5,000 microscopic images of different cell types (such as HeLa cells, stem cells, etc.) acquired under different lighting and magnification levels. The fine-tuning process used a learning rate of 1e-4 and trained for 50 epochs to enable the model to more accurately identify cellular targets in microscopic images.

[0034] Preferably, the focusing execution module consists of a 42-series two-phase hybrid stepper motor, a TB6600 stepper motor driver, and a three-jaw self-centering linkage fixture. The GPIO pins of the lower-level machine (such as PA4, PA5, PA6) are respectively connected to the pulse, direction, and enable terminals of the TB6600 stepper motor driver.

[0035] The beneficial effects of this invention are reflected in:

[0036] 1. Acquire a real-time field-of-view image: Acquire a microscope field-of-view image in real time; Detect an object using a target detection model: Process the real-time field-of-view image using a target detection model finely tuned to a microscopic image dataset to identify at least one target object and output its bounding box and confidence information in real time; Determine a dynamic Region of Interest (ROI): Select a target from at least one target object according to a preset rule, and use its bounding box as a dynamic ROI; Calculate a composite sharpness value: Calculate the sharpness value of the real-time field-of-view image only within the dynamic ROI; Execute a three-stage search algorithm: Based on the sharpness value, drive a focusing actuator to adjust the microscope's focus using a preset search algorithm until the sharpness value reaches the optimal state. When executing the three-stage search algorithm, if in the coarse search stage, perform a large step search, record data points, and determine the fine search interval; if in the fine search stage (entered after the coarse search determines the interval), perform a small step search and record five data points. Attached Figure Description

[0037] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. The elements or parts in the drawings are not necessarily drawn to scale.

[0038] Figure 1 System structure block diagram. This diagram illustrates the connection relationships between the image acquisition module, control processing module, and focus adjustment execution module of the system of this invention.

[0039] Figure 2 : Flowchart of the autofocus method. It details the core algorithm process from image acquisition to final focusing.

[0040] Figure 3 : A schematic diagram of the host computer software user interface. This illustrates the human-computer interaction interface of the present invention.

[0041] Figure 4 : Schematic diagram of the sharpness value-motor position change curve. The X-axis of this graph represents the stepper motor rotation angle (unit: degrees), and the Y-axis represents the normalized sharpness evaluation value. The graph distinguishes the three-stage search process with different visual markers and labels the key parameters. Detailed Implementation

[0042] The embodiments of the technical solution of the present invention will now be described in detail with reference to the accompanying drawings. These embodiments are merely illustrative of the technical solution of the present invention and are therefore intended to limit the scope of protection of the present invention.

[0043] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application should have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0044] like Figure 2 As shown, a target detection-based dynamic autofocus method for microscopes is characterized by the following steps:

[0045] Acquire real-time field-of-view images; capture a single field-of-view image from a microscope in real time.

[0046] The system utilizes an object detection model finely tuned to a microscopic image dataset to process real-time field-of-view images. This model identifies at least one target object in real-time and outputs its bounding box and confidence information. The acquired image is then fed into the YOLOv8 model for object detection. If no target is detected, the system returns to step two (image acquisition and object detection) and reports an error; if a target is detected, the process continues to the next step.

[0047] The confidence score here is a value between 0 and 1, representing the model's degree of certainty that a target (such as a cell or lesion area) exists within the detection box, and a comprehensive assessment of the accuracy of the bounding box position. It is used to filter valid detection results and assist in focusing decisions. Essentially, confidence score is a quantitative indicator of the model's reliability of the prediction results, containing two meanings: 1) Target existence probability: Whether a target (such as a cell structure) actually exists within the box; 1 if it exists, 0 otherwise. 2) Position accuracy: The intersection-over-union (IoU) ratio between the predicted box and the true target box, measuring the accuracy of the bounding box positioning. In autofocus, high confidence scores can serve as a basis for judging the sharp focus area, assisting the system in locking the optimal focal length. In this program, the predicted box is generated from the prior box using offset calculations. The predicted box is used to predict the position and size of the target (e.g., a cell or cell cluster). The true target box is the actual location (the actual location of the cell or cell cluster) in the labeled data.

[0048] Determine the dynamic ROI by selecting a target from at least one target object according to a preset rule, and using its bounding box as a dynamic region of interest (ROI).

[0049] Calculate composite sharpness values, and calculate the sharpness values ​​of the real-time field of view image only within the region of interest (ROI);

[0050] A three-stage search algorithm is executed. Based on the sharpness value, a focusing actuator is driven by a preset three-stage search algorithm to adjust the focus of the microscope until the sharpness value reaches the optimal state. When executing the three-stage search algorithm, if it is in the coarse search stage, a large step search is performed, data points are recorded, and the fine search interval is determined; if it is in the fine search stage (entered after the interval is determined by the coarse search), a small step search is performed, and five data points are recorded.

[0051] Furthermore, one of the target detection models is a target detection model from the You Only Look Once (YOLO) series, which can quickly detect targets in images and output information such as the target's bounding box and confidence score.

[0052] Specifically, the YOLOv8 model is an object detection model in the You Only Look Once (YOLO) series, capable of quickly detecting objects in images and outputting information such as the object's bounding box and confidence score. ROI (Region of Interest) is a specific region defined in an image for subsequent analysis (such as calculating sharpness values), reducing computational load and improving targeting. Composite sharpness value is a comprehensive metric used to measure the sharpness of a specific ROI in an image. It is calculated using a specific algorithm, and the optimal focus position is determined by comparing the composite sharpness values ​​at different locations during focusing.

[0053] Furthermore, the preset rule is: select the target with the highest confidence among at least one target object as the target.

[0054] Specifically, the rule is to select the candidate target with the highest confidence score and use its bounding box as the dynamic ROI for the current frame. In another alternative embodiment, the rule could also be to select the target closest to the image center or the target with the largest area.

[0055] Furthermore, a dynamic region of interest (ROI) is a specific region defined in an image for subsequent analysis (such as calculating sharpness values), which can reduce computational load and improve targeting.

[0056] Furthermore, composite sharpness value: a comprehensive index used to measure the sharpness of a specific region (ROI) in an image. It is calculated by a certain algorithm and the optimal focus position is determined by comparing composite sharpness values ​​at different positions during the focusing process.

[0057] Furthermore, in calculating the composite sharpness value, a composite sharpness evaluation function combining the Tenengrad gradient function and the improved Laplacian function (SML) is used to calculate the sharpness value, and the weighting factor α of the composite sharpness evaluation function is experimentally calibrated, with a value range of 0.5 to 0.7.

[0058] The formula for calculating a composite sharpness evaluation function is as follows:

[0059] F composite =α*F Tenengrad +(1-α)*F SML

[0060] Where α is the optimal weighting factor.

[0061] Specifically, the present invention conducts the following calibration experiment: Ten typical cell image sequences are selected, each sequence containing images of the entire process from blurry to clear and then back to blurry. α values ​​are set to 0.1, 0.2, ..., 0.9, and the sharpness curves for each sequence at different α values ​​are calculated. By evaluating the curves' unimodality, noise-free properties, peak sensitivity, and other indicators, it is found that the function exhibits the best overall performance when α is within the range of 0.5 to 0.7.

[0062] Furthermore, in executing the three-stage search algorithm, the preset search algorithm is a three-stage search process, which includes:

[0063] A1: Drive a focusing actuator with a large step length to perform a rapid coarse search;

[0064] A2: Within the peak range determined by the coarse search, a focusing actuator is driven with small steps to perform a fine search to collect multiple discrete position and sharpness data points;

[0065] A3: Perform a quadratic polynomial function fitting on discrete position and sharpness data points to calculate the theoretical optimal focus position, and drive a focusing actuator to move to that position.

[0066] A target detection-based dynamic autofocus system for microscopes includes: an image acquisition module S1, a focusing execution module S3, and a control processing module S2, characterized in that:

[0067] Image acquisition module S1: Used for real-time acquisition of field-of-view images from the microscope;

[0068] Focusing execution module S3: Used to drive the focusing knob of the microscope;

[0069] Control processing module S2: The control processing module S2 is electrically connected to the image acquisition module S1 and the focus execution module S3.

[0070] Furthermore, the image acquisition module S1 consists of a CMOS camera 11 or an image acquisition device with equivalent performance, which is mounted on the observation tube of the XSP-200D biological microscope 40.

[0071] Furthermore, the control processing module S2 consists of a host computer 21 and a slave computer 22 working together to form a two-layer control architecture. The host computer 21 is a personal computer responsible for running the core algorithm; the slave computer 22 is an STM32F103C8T6 microcontroller minimum system board responsible for executing hardware control. The two communicate with each other through a USB-to-TTL serial port module.

[0072] Specifically, the host computer software platform is developed based on Python 3.8 and core libraries such as PyQt5, OpenCV, PyTorch, PySerial, and Matplotlib (see reference). Figure 3 The YOLOv8 model used in this invention was fine-tuned based on the official pre-trained model using a dataset of approximately 5,000 microscopic images of different cell types (such as HeLa cells, stem cells, etc.) acquired under different lighting and magnification levels. The fine-tuning process used a learning rate of 1e-4 and trained for 50 epochs to enable the model to more accurately identify cellular targets in microscopic images.

[0073] Furthermore, the focusing execution module S3 consists of a 42-series two-phase hybrid stepper motor 31, a TB6600 stepper motor driver 33, and a three-jaw self-centering linkage fixture 32. The GPIO pins of the lower-level machine 22 (such as PA4, PA5, PA6) are respectively connected to the pulse, direction, and enable terminals of the TB6600 stepper motor driver 33.

[0074] refer to Figure 3 The host computer software platform is developed based on Python 3.8 and core libraries such as PyQt5, OpenCV, PyTorch, PySerial, and Matplotlib.

[0075] The YOLOv8 model used in this invention was fine-tuned based on the official pre-trained model using a dataset of approximately 5,000 microscopic images of different cell types (such as HeLa cells and stem cells) acquired under different lighting and magnification levels. The fine-tuning process involved setting the learning rate to 1e-4 and training for 50 epochs to enable the model to more accurately identify cellular targets in the microscopic images.

[0076] It should be noted that during use, the process involves: acquiring a real-time field-of-view image by continuously capturing a single image of the microscope's field of view; using a target detection model finely tuned to a microscopic image dataset to process the real-time field-of-view image to identify at least one target object and output its bounding box and confidence information; determining a dynamic Region of Interest (ROI) by selecting one target from at least one target object according to a preset rule and using its bounding box as a dynamic ROI; calculating a composite sharpness value by calculating the sharpness value of the real-time field-of-view image only within the dynamic ROI; and executing a three-stage search algorithm by driving a focusing actuator to adjust the microscope's focus based on the sharpness value using a preset search algorithm until the sharpness value reaches its optimal state. During the three-stage search algorithm, if in the coarse search stage, a large step search is performed, data points are recorded, and the fine search interval is determined; if in the fine search stage (entered after the coarse search determines the interval), a small step search is performed, and five data points are recorded.

Claims

1. A method for dynamic autofocusing of a microscope based on target detection, characterized in that, Includes the following steps: Acquire real-time field-of-view images; capture a single field-of-view image from a microscope in real time. The real-time field of view image is processed using a target detection model finely tuned to a microscopic image dataset to identify at least one target object in real time and output its bounding box and confidence information. Determine a dynamic ROI by selecting a target from the at least one target object according to a preset rule, and using its bounding box as a dynamic region of interest (ROI). Calculate the composite sharpness value, and calculate the sharpness value of the real-time field of view image only within the dynamic region of interest (ROI); A three-stage search algorithm is executed. Based on the sharpness value, a focusing actuator is driven by a preset three-stage search algorithm to adjust the focus of the microscope until the sharpness value reaches the optimal state. When executing the three-stage search algorithm, if it is in the coarse search stage, a large step search is performed, data points are recorded, and the fine search interval is determined; if it is in the fine search stage (entered after the interval is determined by the coarse search), a small step search is performed, and five data points are recorded.

2. The microscope dynamic autofocus method based on target detection according to claim 1, characterized in that: The target detection model is a type of target detection model in the You Only Look Once (YOLO) series, which can quickly detect targets in images and output information such as the target's bounding box and confidence score.

3. The microscope dynamic autofocus method based on target detection according to claim 1, characterized in that: The preset rule is: select the target with the highest confidence level among the at least one target object.

4. The microscope dynamic autofocus method based on target detection according to claim 1, characterized in that: The dynamic region of interest (ROI) is a specific area defined in an image for subsequent analysis (such as calculating sharpness values), which can reduce computational load and improve targeting.

5. The microscope dynamic autofocus method based on target detection according to claim 1, characterized in that: Composite sharpness value: A comprehensive index used to measure the sharpness of a specific region (ROI) in an image. It is calculated by a certain algorithm and the optimal focus position is determined by comparing the composite sharpness values ​​at different positions during the focusing process.

6. The microscope dynamic autofocus method based on target detection according to claim 1, characterized in that: In the calculation of the composite sharpness value, the sharpness value is calculated by a composite sharpness evaluation function that combines the Tenengrad gradient function and the improved Laplace function (SML), and the weight factor α of the composite sharpness evaluation function is experimentally calibrated, with a value range of 0.5 to 0.

7. The calculation formula for the composite sharpness evaluation function is as follows: ； Where α is the optimal weighting factor.

7. The microscope dynamic autofocus method based on target detection according to claim 1, characterized in that: In the execution of the three-stage search algorithm, the preset three-stage search algorithm is a three-stage search process, which includes: A1: Drive the focusing actuator with a large step size to perform a rapid coarse search; A2: Within the peak range determined by the coarse search, the focusing actuator (10) is driven with small steps to perform a fine search to collect multiple discrete position and sharpness data points; A3: Perform a quadratic polynomial function fitting on the discrete position and sharpness data points to calculate the theoretical optimal focus position, and drive the focusing actuator to move to that position.

8. A target detection-based dynamic autofocus system for microscopes, comprising: The image acquisition module (S1), the focus adjustment execution module (S3), and the control processing module (S2) are characterized by: Image acquisition module (S1): Used for real-time acquisition of field-of-view images from the microscope; Focusing execution module (S3): Used to drive the focusing knob of the microscope; Control processing module (S2): The control processing module (S2) is electrically connected to the image acquisition module (S1) and the focus execution module (S3).

9. A microscope dynamic autofocus system based on target detection according to claim 7, characterized in that: The image acquisition module (S1) consists of a COMS camera (11) or an image acquisition device with equivalent performance, which is installed on the observation tube of the XSP-200D biological microscope (40).

10. A microscope dynamic autofocus system based on target detection according to claim 7, characterized in that: The control processing module (S2) consists of a host computer (21) and a slave computer (22) working together to form a two-layer control architecture. The host computer (21) is a personal computer responsible for running the core algorithm; the slave computer (22) is an STM32F103C8T6 microcontroller minimum system board responsible for executing hardware control. The two communicate with each other through a USB to TTL serial port module. The focusing execution module (S3) consists of a 42 series two-phase hybrid stepper motor (31), a TB6600 stepper motor driver (33), and a three-jaw self-centering linkage fixture (32). The GPIO pins of the slave computer (such as PA4, PA5, PA6) are respectively connected to the pulse, direction, and enable terminals of the TB6600 stepper motor driver (33).

Images