Microscope rapid focusing method and device, storage medium and program product

Abstract

The invention discloses a microscope rapid focusing method and device, a storage medium and a program product. Aiming at the industrial pain point of microscope automatic focusing, a brand-new focusing scheme based on traditional contrast focusing and double AI model complementary verification is provided, and the problems that in the prior art, the contradiction between the high frame rate and the computing power is solved, the traditional algorithm does not have the unified measurement, the cross-scene adaptability is poor, and the speed and the precision cannot be considered at the same time are solved. And a high-speed, high-precision and high-stability solution is provided for an automatic focusing scene of the microscope.

Description

Technical Field

[0001] This invention belongs to the field of microscope imaging, specifically involving automatic focusing technology for precision optical imaging equipment such as biological microscopes, metallurgical microscopes, and industrial inspection microscopes. It is particularly suitable for automated scenarios requiring micron-level precision and high frame rate fast focusing under high magnification. Background Technology

[0002] As microscopic imaging technology develops towards automation, high speed, and high precision, autofocus (AF) has become a core standard feature of modern automated microscopes. In high-magnification scenarios, the microscope depth of field is only 1-3 μm, requiring focusing accuracy to reach the sub-micron level; at the same time, automated batch inspection scenarios place extremely high demands on focusing speed.

[0003] The mainstream autofocus solution in the current field of microscopy is contrast-detection autofocus (CDAF), which determines the focus state by detecting the peak contrast of the image, offering advantages such as high accuracy and strong interpretability. However, since it requires the acquisition of sequential images, the focusing speed is strongly correlated with the camera's frame rate; high frame rate acquisition is a crucial prerequisite for achieving fast focusing. Although traditional CDAF can adapt to high frame rate frame-by-frame processing, its output is a relative contrast value, which has two inherent drawbacks that cannot be resolved: (1) No unified measurement: its numerical range and peak amplitude are greatly affected by light source brightness, sample material, lens optical parameters and magnification. The sharpness curves in different scenes are very different, and it is impossible to determine whether the current frame is in true focus by a fixed threshold. (2) Severe false peak misjudgment: Under high magnification, it is easily affected by sample impurities, noise, local texture, and uneven lighting, resulting in a large number of local false peaks. The relative contrast alone cannot distinguish between the real focus peak and the false peak, leading to misjudgment of the focusing system and back-and-forth "hunting". For low texture samples such as transparent cell slices and polished metal, the focusing success rate of traditional CDAF is low.

[0004] With the rapid development of AI technology, introducing AI into microscopic focusing technology has become one of the main trends. This includes pure AI focusing solutions, multi-scale AI focusing solutions, and AI-assisted focusing solutions. Existing pure AI focusing solutions require AI inference for each frame of the acquired image, which is time-consuming and cannot adapt to high frame rate acquisition scenarios. This results in the focusing speed being limited by the AI ​​inference speed, failing to leverage the speed advantage of high frame rate cameras. Furthermore, pure AI solutions have relatively poor focusing accuracy and interpretability, failing to meet the compliance requirements of medical and industrial quality inspection scenarios. Existing multi-scale AI focusing solutions sometimes have insufficient receptive fields, causing image features to spread outwards after defocusing under high magnification, easily overflowing the small focusing area and leading to model misjudgment. Others have excessively large receptive fields, making it impossible to lock onto the target focusing area when multiple focal planes appear within the field of view. These solutions are completely unsuitable for the specific characteristics of extremely shallow depth of field and defocus diffusion under high magnification microscopes. Summary of the Invention

[0005] To address the technical problems mentioned in the background section, this invention proposes a method, apparatus, storage medium, and program product for rapid focusing of a microscope.

[0006] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows: A method for rapid focusing in a microscope involves constructing and training a two-branch sharpness regression model for large and small regions. The input to this model is the RGB three-channel image of the central small region and the RGB three-channel image of the global large region of the microscope image. The central small region branch and the global large region branch independently extract multi-scale features, and then the features from the two branches are concatenated and fused to output a sharpness effectiveness score. A sharpness temporal trend classification model is also constructed and trained. The input to this model is the RGB three-channel image of the current frame and the N frames before and after it. After feature extraction and feature fusion, the model outputs various sharpness trends. The probability distribution, the sharpness trend includes continuously increasing sharpness, sharpness at the true peak, and continuously decreasing sharpness; for the acquired microscope images, the contrast algorithm is used to calculate the sharpness value of the current frame in real time, and the Z-axis motor coding position corresponding to the current frame is recorded to generate a sharpness change curve within the focusing stroke. The sharpness change curve is fitted with a local trend. Dual model verification is triggered only when a suspected peak point is detected. The acquired microscope images are respectively input into the large and small area dual-branch sharpness regression model and the sharpness time series trend classification model. Only when the outputs of the two models simultaneously meet the verification pass condition is the current Z-axis motor position locked as the focusing position, and automatic focusing is completed.

[0007] Furthermore, each image in the training set of the large and small region bi-branch sharpness regression model is labeled with a sharpness tag of 0-1, where 0 indicates that the image is completely out of focus and 1 indicates that the image is in perfect focus; the large and small region bi-branch sharpness regression model outputs a sharpness effectiveness score of 0-1.

[0008] Furthermore, the training strategy for the large and small region bi-branch sharpness regression model is as follows: The training set is sampled evenly for different magnifications, light sources, and sample types; the L1 loss function is used; the Adam optimizer is used with a learning rate decay strategy; only horizontal / vertical flipping is used for data augmentation.

[0009] Furthermore, the training strategy for the clarity time-series trend classification model is as follows: We collected continuous image sequences of the focusing process under different lenses, magnifications, and samples, which need to fully cover the entire focusing journey from "blurred → sharp → blurry". We also oversampled the trend categories where the sharpness is at the true peak. We used a cross-entropy loss function with class weights, employed the Adam optimizer, and used a learning rate decay strategy. We only used horizontal / vertical flipping for data augmentation.

[0010] Furthermore, the Sigmoid activation function is applied to the large and small region bi-branch sharpness regression model to constrain the output range; the Softmax activation function is applied to the sharpness time-series trend classification model.

[0011] Furthermore, a sliding window method is used to fit the local trend of the sharpness curve: the sharpness values ​​of the current frame and the M frames before and after it, i.e., the continuous 2M+1 frames, are taken and fitted to a local second-order curve; when the local second-order curve satisfies that the current frame is the local maximum value within the continuous 2M+1 frame window, the current frame is determined to be a suspected peak point, and dual-model verification is triggered.

[0012] Furthermore, a sharpness validity threshold is set. When the sharpness validity score output by the dual-branch sharpness regression model of the large and small regions is greater than or equal to the sharpness validity threshold, and the maximum probability distribution value output by the sharpness time-series trend classification model corresponds to the sharpness being in the true peak trend category, it indicates that the outputs of the two models simultaneously meet the verification conditions.

[0013] Secondly, the present invention also provides a computer device, which includes a processor and a memory, the memory storing a computer program, and the processor executing the computer program to implement the above-described microscope rapid focusing method.

[0014] Furthermore, the present invention also provides a computer storage medium storing a computer program that, when executed on a processor, implements the aforementioned rapid focusing method for microscopes.

[0015] Finally, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the above-described microscope fast focusing method.

[0016] The beneficial effects of adopting the above technical solution are as follows: 1. Adapted to high frame rate scenarios: Traditional frame-by-frame scanning + dual AI-verified focus closed-loop mechanism To address the contradiction between high frame rate acquisition and AI computing power bottlenecks, as well as the pain point of false peak misjudgment in traditional algorithms, this invention designs a layered and progressive focusing logic that balances speed, accuracy, and interpretability.

[0017] Underlying technology: Using traditional contrast-detection autofocus algorithms, real-time frame-by-frame calculations are performed in high frame rate acquisition streams to generate sharpness change curves within the focus travel range. This ensures that the focus speed is strongly tied to the camera frame rate throughout the process, eliminating computational bottlenecks while retaining the interpretability and accuracy advantages of traditional algorithms.

[0018] Triggering logic: The AI ​​verification process is triggered only when the traditional algorithm detects a suspected peak point. There is no need to run the AI ​​model throughout the process, reducing computing power consumption by more than 90% and adapting to high frame rate acquisition scenarios.

[0019] Dual complementary verification: Two AI models are used to solve the two inherent defects of traditional algorithms respectively: the sharpness model solves the problem of no unified measurement of relative sharpness, and the trend model solves the problem of large differences in sharpness curves of different lenses and the inability to distinguish false peaks.

[0020] Stop rule: The motor stops focusing only when all three conditions are met: traditional peak screening, AI clarity validity verification, and AI trend peak verification. This solves the problem of false peak misjudgment from the root, eliminates the need for back-and-forth scanning, and completely avoids the "hunting" phenomenon.

[0021] 2. A dual-branch sharpness regression model for large and small regions enables consistent sharpness validity assessment across different scenes. To address the shortcomings of traditional algorithms, such as the lack of a unified metric for relative sharpness, high-magnification defocusing diffusion, and interference from multiple focal planes, this invention designs a microscope-specific dual-branch multi-scale architecture.

[0022] Core design logic: It adopts a 6-channel input of a central small region cropping image and a global large region cropping image. After extracting independent features from two branches, it fuses and outputs a sharpness effectiveness score of 0-1, which perfectly balances the contradiction between receptive field and focal plane locking.

[0023] Small area branch: Take the focal plane responsible for locking the target. Even if there are multiple focal planes in a large area, the model can focus on the sharpness of the central target area and avoid interference from multiple focal planes.

[0024] Large area branch: responsible for capturing global out-of-focus features, solving the problem of features spreading outward and overflowing into small areas after defocusing under high magnification, so that the model can accurately determine the global out-of-focus / focus state.

[0025] Core value: It transforms the relative sharpness value of traditional algorithms, which lacks universal boundaries, into a universal 0-1 validity score that is applicable across light sources, materials, lenses, and magnification scenarios, thus solving the problem that traditional algorithms cannot use a unified threshold to determine the focus state.

[0026] 3. The time-series trend classification model complements the clarity model. To address the pain points of large differences in sharpness curves under different lenses and magnifications, and the inability of traditional algorithms to distinguish between true peaks and false peaks, this invention designs a trend classification model that forms a complementary closed loop with the sharpness model.

[0027] Core input design: The model input uses a forward image + current image + backward image to highlight the trend of sharpness changes during focusing, rather than the content of the sample itself. This makes the model completely unaffected by lens, magnification, and sample material, alleviating the problem of large differences in sharpness curves between different lenses.

[0028] Complementary division of labor logic: The sharpness model is responsible for judging whether the current frame is sharp enough, solving the problem of absolute value measurement; the trend model is responsible for judging whether the current frame is the true peak of the sharpness curve, solving the problem of relative trend. The two complement each other, confirming the effectiveness of focus and locking the position of the focus peak, forming a complete decision-making closed loop. Attached Figure Description

[0029] Figure 1 This is a flowchart of the real-time focusing execution stage in the embodiment. Detailed Implementation

[0030] This embodiment includes two core parts: the pre-preparation stage and the real-time focus execution stage.

[0031] I. Preparatory Stage (Pre-trained Model and System Initialization) 1. Two-branch clarity regression model for large and small regions (for absolute value validity verification): 1) Dataset Construction Microscopic images of different brands of lenses, different magnifications, different light source brightness, and different sample types (metal materials, semiconductor wafers, etc.) are collected. Each image is labeled with a sharpness tag of 0-1 (0=completely out of focus, 1=perfectly in focus) according to the gradient of "completely blurred - semi-sharp - fully in focus". The central small area cropping image and the global large area cropping image are saved simultaneously for each sample to cover the out-of-focus diffusion and multi-focal interference under different scenarios.

[0032] 2) Model Architecture Employing a lightweight U2Net-Light encoder-decoder architecture, balancing inference speed and feature extraction capabilities: Input layer: 6-channel input, corresponding to 3 RGB channels of the small area image + 3 RGB channels of the large area image; Dual-branch feature extraction: Small region branch and large region branch are extracted with multi-scale features by independent RSU-Light modules to avoid mutual interference between information of different scales; Feature fusion layer: The features output from the two branches are concatenated and fused, and then optimized by dimensionality reduction through convolutional layers to balance local details and global out-of-focus features; The regression output head, after adaptive average pooling and fully connected layers, finally outputs a sharpness validity score of 0-1. The output range is constrained by the Sigmoid activation function to achieve a sharpness measurement that is universal across scenes.

[0033] 3) Training strategy Loss function: The L1 loss function is used to reduce the interference of outliers on the model; Optimizer: Adam optimizer, paired with a learning rate decay strategy; Data augmentation: Only horizontal / vertical flipping is used to avoid rotation from damaging the texture features of microscope samples; Sampling strategy: Perform balanced sampling for different magnifications, different light sources, and different sample types to improve the model's cross-scene generalization ability.

[0034] 2. Differential-driven time-series trend classification model (for relative peak verification, complementary to the clarity model): 1) Dataset Construction Collect continuous image sequences of the focusing process under different lenses, magnifications, and sample sizes, fully covering the entire focusing journey from "blurred → sharp → blurry"; each sample consists of 3 consecutive frames (previous frame, current frame, and next frame), labeled with 4 types of trend tags: 0 = Up (Clarity continues to increase, but has not yet reached its peak). 1 = Peak (Current frame is the true peak sharpness, in focus position); 2 = Down (Clarity continues to decline, peak value has been exceeded); 3 = Unknown (Unable to determine).

[0035] 2) Model Architecture Employing a lightweight U2Net-Light classification architecture, it is adapted to real-time requirements triggered on demand. Input layer: RGB 3-channel image input; Encoder: Extracts multi-scale temporal variation features; Decoder: Performs upsampling and feature fusion on the corresponding encoder to enhance trend features; Classification output head: After adaptive average pooling and fully connected layers, the final output is the probability distribution of four trends, using the Softmax activation function.

[0036] 3) Training strategy Loss function: Cross-entropy loss function with class weights, which solves the imbalance problem of insufficient sample size for Peak class and improves the accuracy of peak recognition; Optimizer: Adam optimizer, paired with a learning rate decay strategy; Data augmentation: Only horizontal / vertical flipping is used to ensure the integrity of temporal features; Sampling strategy: The Peak category has been sampled to ensure the model's sensitivity in recognizing real peak values.

[0037] 3. Microscope system initialization Hardware parameter configuration: Based on the microscope lens magnification, preset the scanning step size, moving speed, and focusing travel limit of the Z-axis motor; configure the ROI acquisition area and exposure parameters of the image sensor; Inference environment initialization: Load the pre-trained dual AI model into the embedded controller or host computer of the microscope, initialize the inference engine, and optimize the inference speed; System self-test: Completes Z-axis motor reset and image sensor calibration, confirms normal hardware status, and ensures hardware synchronization between traditional algorithm and image acquisition.

[0038] II. Real-time focusing execution phase (e.g.) Figure 1 (As shown) 1. High frame rate frame-by-frame scanning and real-time resolution calculation The Z-axis motor is controlled to move at a constant speed along the focusing direction at a preset scanning step size, simultaneously acquiring images of the Region of Interest (ROI) within the microscope's field of view via an image sensor, achieving hard synchronization between acquisition and processing. For each acquired image frame, a traditional contrast algorithm is used for real-time frame-by-frame processing, outputting the relative sharpness value of the current frame. The corresponding Z-axis motor coding position is recorded synchronously, generating a real-time sharpness-position change curve within the focusing stroke. The images and sharpness values ​​of the current frame and the five consecutive frames before and after it are simultaneously cached for subsequent peak screening.

[0039] 2. Initial screening of suspected peak points and AI triggering A sliding window method is used to fit the local trend of the sharpness curve. AI dual-model verification is triggered only when a suspected peak point is detected; otherwise, a coarse scan continues frame by frame. Specifically, the sharpness values ​​of the current frame and the two frames before and after it (a total of 5 frames) are taken, and a local second-order curve is fitted. If this curve satisfies the condition that the current frame is the local maximum within the 5-frame window, the current frame is determined to be a suspected peak point, and the AI ​​dual-model verification process is triggered. If no suspected peak point is detected, the Z-axis motor continues to move, and step 1 is repeated.

[0040] 3. AI Dual-Model Complementary Verification The two verification branches infer in parallel to maximize the verification speed, and the two models form a complementary closed loop. The specific logic is as follows: Verification Branch 1: Sharpness Validity Verification (Solving the problem of the lack of a unified metric for traditional relative sharpness) For the current frame image, crop the small central region image and the large global region image respectively, and perform normalization preprocessing consistent with the training phase; The preprocessed 6-channel tensor is input into the sharpness regression model to infer the sharpness validity score S (0-1, applicable across scenes) of the current frame. Validity judgment: A preset general validity threshold (e.g., 0.7) is set. If S ≥ the threshold, the absolute sharpness verification is deemed to have passed, and the current frame is confirmed to meet the focus standard. If it does not meet the standard, it is deemed to be a false peak and the process returns to step 1 to continue scanning.

[0041] Verification Branch 2: Trend Peak Verification (Solve the problem of large differences in sharpness curves between different lenses and misjudgment of false peaks) Extract the current frame, its previous frame, and its next frame from the cache, for a total of 3 consecutive frames. Perform differential preprocessing consistent with the training phase to generate a 3-channel input tensor. The input tensor is fed into the temporal trend classification model, and the probability distribution of the three trends is obtained through inference. The category with the highest probability is taken as the trend state T of the current frame. Trend judgment logic: If T = Peak, then the relative trend verification is passed, confirming that the current frame is the true focus peak.

[0042] 4. Complete autofocus If both the absolute sharpness check and the relative trend check pass simultaneously, the current Z-axis motor position is immediately locked as the focus position, the motor stops moving, and autofocus is completed. If neither condition is met simultaneously, the Z-axis motor resumes moving at a preset step size, and steps 1-4 are repeated until focus is achieved.

[0043] This embodiment also relates to a computer device, which includes a processor and a memory. The memory stores a computer program, and the processor executes the computer program to implement the above-described contents. This embodiment also relates to a computer storage medium storing a computer program, which, when executed on a processor, implements the above-described contents. This embodiment also relates to a computer program product, including a computer program or instructions, which, when executed by a processor, implements the above-described contents.

[0044] The embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for rapid focusing of a microscope, characterized in that: A two-branch sharpness regression model for large and small regions is constructed and trained. The input of the two-branch sharpness regression model for large and small regions is the RGB three channels of the central small region image and the global large region image of the microscope image. After the central small region branch and the global large region branch extract multi-scale features independently, the features of the two branches are spliced ​​and fused to finally output the sharpness effectiveness score. A sharpness temporal trend classification model is constructed and trained. The input of the sharpness temporal trend classification model is the RGB three channels of the current frame of the microscope image and the N frames before and after it. After feature extraction and feature fusion, the model finally outputs the probability distribution of various sharpness trends. The sharpness trends include sharpness continuously increasing, sharpness at the true peak, and sharpness continuously decreasing. For the acquired microscope images, a contrast algorithm is used to perform real-time frame-by-frame calculations, outputting the sharpness value of the current frame, recording the Z-axis motor coding position corresponding to the current frame, generating a sharpness variation curve within the focusing stroke, and performing local trend fitting on the sharpness variation curve. Dual-model verification is triggered only when a suspected peak point is detected. The acquired microscope images are respectively input into the large-area dual-branch sharpness regression model and the sharpness temporal trend classification model. Only when the outputs of the two models simultaneously meet the verification pass conditions is the current Z-axis motor position locked as the focusing position, and automatic focusing is completed.

2. The microscope rapid focusing method according to claim 1, characterized in that: Each image in the training set of the large and small region bi-branch sharpness regression model is labeled with a sharpness label of 0-1, where 0 indicates that the image is completely out of focus and 1 indicates that the image is in sharp focus; the large and small region bi-branch sharpness regression model outputs a sharpness validity score of 0-1.

3. The microscope rapid focusing method according to claim 1, characterized in that: The training strategy for the large and small region bi-branch sharpness regression model is as follows: The training set is sampled evenly for different magnifications, light sources, and sample types; the L1 loss function is used; the Adam optimizer is used with a learning rate decay strategy; only horizontal / vertical flipping is used for data augmentation.

4. The microscope rapid focusing method according to claim 1, characterized in that: The training strategy for the clarity time-series trend classification model is as follows: We collected continuous image sequences of the focusing process under different lenses, magnifications, and samples, which need to fully cover the entire focusing journey from "blurred → sharp → blurry". We also oversampled the trend categories where the sharpness is at the true peak. We used a cross-entropy loss function with class weights, employed the Adam optimizer, and used a learning rate decay strategy. We only used horizontal / vertical flipping for data augmentation.

5. The microscope rapid focusing method according to claim 1, characterized in that: The Sigmoid activation function is used to constrain the output range for the two-branch sharpness regression model of the large and small regions; the Softmax activation function is used for the sharpness time-series trend classification model.

6. The microscope rapid focusing method according to claim 1, characterized in that: The sliding window method is used to fit the local trend of the sharpness curve: take the sharpness value of the current frame and the M frames before and after it, that is, the continuous 2M+1 frames, and fit a local second-order curve; when the local second-order curve satisfies that the current frame is the local maximum value within the continuous 2M+1 frame window, the current frame is determined to be a suspected peak point and the dual model verification is triggered.

7. The microscope rapid focusing method according to claim 1, characterized in that: A sharpness validity threshold is set. When the sharpness validity score output by the dual-branch sharpness regression model of the large and small regions is greater than or equal to the sharpness validity threshold, and the maximum probability distribution value output by the sharpness time series trend classification model corresponds to the sharpness being in the true peak trend category, it indicates that the outputs of the two models simultaneously meet the verification conditions.

8. A computer device comprising a processor and a memory, the memory storing a computer program, the processor being configured to execute the computer program to implement the microscope rapid focusing method as described in any one of claims 1-7.

9. A computer storage medium storing a computer program, which, when executed on a processor, implements the microscope rapid focusing method as described in any one of claims 1-7.

10. A computer program product comprising a computer program or instructions that, when executed by a processor, implement the microscope rapid focusing method as described in any one of claims 1-7.

Images