Microscope automatic focusing method based on end-to-end model

Abstract

The invention provides a microscope automatic focusing method based on an end-to-end model, and relates to the technical field of optical instruments. The method comprises the following steps: shooting an image on a microscope glass slide through an image acquisition module, and carrying out image preprocessing through a gray histogram equalization algorithm; deploying an end-to-end neural network model at the cloud server, receiving the preprocessed image and predicting a first focusing position value; the first focusing position value is sent to an electric focusing module controlled by a stepping motor, and focusing is completed; and acquiring a second image, performing image preprocessing, predicting a second focusing position value by using the end-to-end neural network model, and sending a relative position between the second focusing position value and the first focusing position value to the electric focusing module to complete final automatic focusing. According to the method, the optimal focusing position is directly predicted from the microscopic image through the deep learning model, a traditional focusing scoring function or an image sequence analysis process is simplified, end-to-end efficient prediction is realized, and the focusing speed and precision are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of optical instruments, and particularly relates to a microscope automatic focusing method based on an end-to-end model. BACKGROUND

[0002] Microscopes are indispensable tools in the fields of biology, medicine and environmental science. Traditional microscope focusing methods rely on manual operation or image processing-based automatic focusing techniques such as edge detection or contrast analysis, which determine the best focusing position by taking multiple images at different focal lengths. These methods are often inefficient when dealing with complex samples of movable microcells due to low contrast or impurities, and the focusing speed is slow, making it difficult to meet the high-throughput imaging demand.

[0003] With the development of artificial intelligence technology, deep learning, especially convolutional neural networks (CNN), has shown great advantages in image recognition and processing. However, there is currently a lack of an efficient implementation scheme that can directly predict the focal position based on a single frame of input image and drive the microscope to automatically focus end-to-end. SUMMARY

[0004] The present application aims to solve the problem of microscope automatic focusing by providing a microscope automatic focusing method based on an end-to-end model.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows: The present application provides a microscope automatic focusing method based on an end-to-end model, which comprises: An image on a microscope slide is taken by an image acquisition module, and image preprocessing is performed by a gray level histogram equalization algorithm; An end-to-end neural network model based on MobileNetV3 is deployed on a cloud server, which receives the preprocessed image and predicts a first focus position value; The first focus position value is sent to a stepper motor controlled electric focusing module to complete the focusing; A second image is collected at the focused position and preprocessed, the second focus position value is predicted using the end-to-end neural network model, and the relative position between the second focus position value and the first focus position value is sent to the electric focusing module to complete the final automatic focusing.

[0006] Optionally, the loss function of the end-to-end neural network model is Smooth L1, defined as: , Wherein, represents the difference between the predicted value and the true focal position.

[0007] Optionally, the end-to-end neural network model based on MobileNetV3 comprises a channel attention module, which is used to perform channel attention enhancement after convolution operation, and improve the feature expression ability of the model through feature re-labeling in the channel dimension.

[0008] Optionally, the channel attention module comprises: a global average pooling layer for calculating global spatial information for each channel; a bottleneck structure composed of two fully connected layers, wherein the first layer is used for dimension reduction, and the second layer is used for dimension increase, The channel attention module generates channel attention weights using a sigmoid activation function; and the channel attention module realizes feature enhancement through point-by-point multiplication with the original feature map channel.

[0009] Optionally, the end-to-end neural network model adopts an h-swish activation function, which is defined as: , wherein .

[0010] Optionally, the image acquisition module acquires an image with a resolution of 672*672 pixels.

[0011] Optionally, the end-to-end neural network model is trained using an Adam optimizer.

[0012] Optionally, the precision control range of the stepper motor of the electric focusing module is microns.

[0013] Optionally, a batch normalization layer is added to the output end of the end-to-end neural network model to stabilize the training process and improve the convergence speed and generalization ability of the model.

[0014] The beneficial effects of the present application include: The microscope automatic focusing method based on an end-to-end model provided by the present application comprises: capturing an image on a microscope slide through an image acquisition module, and performing image preprocessing through a gray level histogram equalization algorithm; deploying an end-to-end neural network model based on MobileNetV3 on a cloud server, receiving the preprocessed image and predicting a first focusing position value; sending the first focusing position value to an electric focusing module controlled by a stepper motor to complete focusing; capturing a second image at the focused position and performing image preprocessing, predicting a second focusing position value using the end-to-end neural network model, and sending the relative position between the second focusing position value and the first focusing position value to the electric focusing module to complete the final automatic focusing. The present application directly predicts the best focusing position from the microscopic image through a deep learning model, simplifies the traditional focusing score function or image sequence analysis process, realizes efficient prediction in an end-to-end manner, and significantly improves the focusing speed and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 A structural block diagram of a microscope autofocus system based on an end-to-end neural network model provided by an embodiment of the present invention is shown; Figure 2 A schematic diagram of a process flow of a microscope autofocus method using an end-to-end neural network model provided by an embodiment of the present invention is shown; Figure 3 A hierarchical schematic diagram of an end-to-end neural network model provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0018] The purpose of the present invention is to overcome the problems of poor robustness, weak adaptability to complex scenes, high computational overhead and poor real-time performance of existing microscope focusing methods that rely on manual operation or image processing-based autofocus technology. The present invention provides a microscope autofocus method based on an end-to-end deep neural network, which can achieve high-precision and high-efficiency autofocus in complex backgrounds and is particularly suitable for autofocus and classification and recognition scenarios of microcell images.

[0019] Figure 1 FIG. 4 shows a structural block diagram of a microscope autofocus system based on an end-to-end neural network model provided by an embodiment of the present invention. Figure 1 As shown in the figure, the system consists of three modules, including: microscope system: equipped with a motorized Z-axis stage or piezoelectric brake for precise adjustment of the objective lens height, and a high-resolution digital camera to capture images; neural network model: deployed on a cloud server using a modified MobileNetV3 network, with an input of 672x672 pixel image and an output of regression prediction value of defocus distance to control the motor movement position; control module: integrating image acquisition, neural network inference and motor control functions to ensure real-time focusing.

[0020] Figure 2A flowchart of a microscope automatic focusing method provided by an embodiment of the present application is shown.

[0021] As shown in Figure 2 , the microscope automatic focusing method based on an end-to-end model provided by the present application comprises: An image on a microscope slide is photographed by an image acquisition module, and image preprocessing is performed by a gray level histogram equalization algorithm. Specifically, an image on the current slide is acquired using an electric Z-axis platform and a high-resolution camera provided with the microscope system. The acquired image is first subjected to gray level histogram equalization processing to enhance contrast.

[0022] An end-to-end neural network model based on MobileNetV3 is deployed on a cloud server, which receives the preprocessed image and predicts a first focus position value. The precision control range of the stepper motor of the electric focusing module is microns. The end-to-end neural network model is constructed based on an improved MobileNetV3, the image acquisition module acquires an image with a resolution of 672x672 pixels, and the output is a focus position prediction value in microns. Depth separable convolution, channel attention mechanism, h-swish activation function, global average pooling layer and batch normalization layer are introduced into the network to improve feature expression ability and training stability.

[0023] The first focus position value is sent to the electric focusing module controlled by the stepper motor to drive the microscope objective lens to move along the Z-axis, perform preliminary focusing, and thus complete focusing.

[0024] A second image is acquired at the focused position (i.e., the first focus position) and subjected to image preprocessing, an end-to-end neural network model is used to predict a second focus position value, and the relative position between the second focus position value and the first focus position value is sent to the electric focusing module to complete the final automatic focusing, realize closed-loop feedback automatic focusing, and further improve the final imaging clarity.

[0025] The method proposed in the present application directly predicts the best focus position from the microscopic image through a deep learning model, simplifies the traditional focus scoring function or image sequence analysis process, realizes end-to-end efficient prediction, and significantly improves the focus speed and accuracy. The method directly predicts the best focus position from the microscopic image through a deep learning model, simplifies the traditional focus scoring function or image sequence analysis process, realizes end-to-end efficient prediction, and significantly improves the focus speed and accuracy.

[0026] The loss function of the end-to-end neural network model is Smooth L1, which is defined as: , wherein, The difference between the predicted value and the true focus position is represented. The end-to-end neural network model is trained using an Adam optimizer to improve convergence speed and stability.

[0027] The end-to-end neural network model based on MobileNetV3 includes a channel attention module for performing channel attention enhancement after convolution operation, improving the feature expression capability of the model through feature re-labeling in the channel dimension.

[0028] The channel attention module includes a global average pooling layer for calculating global spatial information for each channel, and a bottleneck structure composed of two fully connected layers, where the first layer is used for dimension reduction and the second layer is used for dimension increase. The channel attention module uses a sigmoid activation function to generate channel attention weights, and realizes feature enhancement through point-by-point multiplication with the original feature map channel.

[0029] The end-to-end neural network model uses a h-swish activation function, which is defined as: , where The activation function is used to replace the traditional ReLU to improve the non-linear expression capability of the network and maintain the calculation efficiency.

[0030] The end-to-end neural network model introduces a lightweight depth separable convolution operation in its bottleneck structure to reduce the model parameter quantity and computational load, thereby speeding up the focus prediction speed.

[0031] The end-to-end neural network model adds a batch normalization layer at the output end to stabilize the training process and improve the model convergence speed and generalization capability.

[0032] Figure 3 The end-to-end neural network model layer hierarchy provided by the embodiment of the application is shown. As shown in Figure 3 The model hierarchy of the end-to-end deep neural network model is as follows: 1) Feature extraction layer MobileNetV3 in the convolutional neural network (CNN) is used as the basic backbone network, and lightweight depth separable convolution and bottleneck structure are introduced to reduce the parameter quantity and computational load, thereby speeding up the inference speed. MobileNetV3 itself has excellent performance on mobile terminals and is suitable for real-time prediction requirements of microscope images. Among them, for cell feature extraction: lightweight design: a large number of lightweight depth separable convolutions (Depthwise Separable Convolution) are used in the bottleneck structure of the network, which effectively reduces the model parameters and computational load, and speeds up the prediction speed.

[0033] 2) Channel Attention Layer To enhance the network's ability to perceive important feature channels, a Squeeze-and-Excitation (SE) module (i.e., a channel attention module) is incorporated into each bottleneck block to perform channel attention reweighting on the feature maps of the convolutional output. Specifically, the SE module first performs global average pooling on the input feature maps in the spatial dimension to generate global descriptors for each channel. Then, a two-layer fully connected network with a bottleneck structure (the first layer reduces the dimension, and the second layer increases the dimension) is used to calculate the attention weights for each channel using a Sigmoid activation function . Finally, the weight vector is multiplied with the original feature map channel by channel to enhance important feature channels, i.e. , where is the original feature map of the feature extraction layer, is the enhanced feature map, and c is the number of channels. This mechanism enables the model to adaptively highlight channel features related to focus prediction, improving feature expression capability.

[0034] 3) Global Average Pooling Layer This layer performs mean operation on each channel's two-dimensional spatial region to achieve dimension reduction of the features, where the output of the cth channel of this layer is: , where H and W are the height and width of the feature map, respectively, C is the number of channels, represents the activation value of the cth channel at position (i, j), is the average value of the corresponding channel, and the final output of this layer is a vector of length C , which collects the global features of all channels as input for the subsequent fully connected regression layer.

[0035] 4) Fully Connected Layer This layer is responsible for integrating all the extracted and weighted features described above and making a decision on the focus position value of the current image through a deep neural network, where FC is the fully connected network used for decision-making, with an output dimension of (1, C). This layer bears the main computational responsibility for the regression task.

[0036] 5) Batch Normalization Layer To further improve the training stability and prediction accuracy of the model, a batch normalization layer is introduced at the last output stage of the neural network. This layer is executed after the second-to-last layer of the model and is used to standardize each channel of the feature tensor, making the feature value distribution have a uniform mean and variance. By standardizing the features in each layer of the network, it solves problems such as gradient vanishing, gradient explosion, and internal covariate shift, enhancing the network's convergence and generalization ability.

[0037] To sum up, the application discloses a microscope automatic focusing method based on an end-to-end neural network model, which is suitable for various cells or microorganisms and other scenes with high requirements for image clarity. The method comprises the steps of image acquisition and preprocessing, deep neural network prediction, focusing control and closed-loop feedback. The neural network is based on the MobileNetV3 structure, adopts deep separable convolution, channel attention module, h-swish activation function and batch normalization layer for construction, and is trained through the SmoothL1 loss function and the Adam optimizer, so that the accurate regression prediction of the focusing position can be realized. The output is used to drive the stepping motor to control the objective lens position, so that high-precision automatic focusing is realized. The application has the advantages of lightweight structure, fast response speed, strong robustness and wide adaptability, and is particularly suitable for fast focusing tasks in the field of microscopic imaging of small cells.

[0038] The method proposed in the application does not depend on specific image sequences or traditional focusing functions, has good universality and scalability, and is suitable for various cell or microorganism image processing tasks. Through the closed-loop control mechanism of image acquisition, prediction focusing, focusing execution, image reacquisition and prediction, micron-level focusing control is realized, and the clarity and focus accuracy of the final image are further ensured, which has significant engineering application value and promotion potential.

[0039] The above examples are only for illustrating the technical concept and characteristics of the application, and the purpose is to enable those skilled in the art to understand the content of the application and implement it, and cannot limit the protection scope of the application. Any equivalent changes or modifications made according to the spirit and essence of the application shall be covered within the protection scope of the application.

Claims

1. A microscope autofocus method based on an end-to-end model, characterized in that: The method comprises: The image on the microscope slide is captured by the image acquisition module and the image is preprocessed by the grayscale histogram equalization algorithm; Deploy an end-to-end neural network model based on MobileNetV3 on the cloud server, receive the preprocessed image and predict the first focus position value; The first focus position value is sent to the electric focus module controlled by the stepper motor to complete the focus adjustment; A second image is captured at the focused position and image preprocessing is performed. The second focus position value is predicted using an end-to-end neural network model, and the relative position between the second focus position value and the first focus position value is sent to the electric focus module to complete the final automatic focus.

2. The microscope autofocus method based on the end-to-end model according to claim 1, characterized in that: The loss function of the end-to-end neural network model is Smooth L1, defined as: ， Where 𝑥 represents the difference between the predicted value and the true focus position.

3. The microscope autofocus method based on the end-to-end model according to claim 2, characterized in that: The end-to-end neural network model based on MobileNetV3 includes a channel attention module, which is used to perform channel attention enhancement after the convolution operation and improve the feature expression ability of the model by recalibrating features in the channel dimension.

4. The microscope autofocus method based on the end-to-end model according to claim 3, characterized in that: The channel attention module includes: a global average pooling layer for calculating global spatial information for each channel; a bottleneck structure consisting of two fully connected layers, where the first layer is used for dimensionality reduction and the second layer is used for dimensionality increase. The channel attention module generates channel attention weights using a sigmoid activation function; the channel attention module achieves feature enhancement by performing point-by-point multiplication with the original feature map channel.

5. The microscope autofocus method based on the end-to-end model according to claim 4, characterized in that: The end-to-end neural network model adopts the h-swish activation function, which is defined as: ， in .

6. The microscope autofocus method based on the end-to-end model according to claim 1, characterized in that: The image acquisition module acquires images with a resolution of 672×672 pixels.

7. The microscope autofocus method based on the end-to-end model according to claim 1, characterized in that: The end-to-end neural network model is trained using the Adam optimizer.

8. The microscope autofocus method based on the end-to-end model according to claim 1, characterized in that: The precision control range of the stepper motor of the electric focusing module is micrometer.

9. The microscope autofocus method based on the end-to-end model according to claim 1, characterized in that: A batch normalization layer is added to the output of the end-to-end neural network model to stabilize the training process and improve the model's convergence speed and generalization ability.

Images