Real-time microscope focusing method, device and electronic device
The Brenner gradient function and Lorentz function fit the curve, combined with the modified MobileNetV3 network, solve the time-consuming and accurate problems of the microscope focus method in high-resolution image processing, and achieve a fast and accurate focus effect.
Patent Information
- Application Number
- CN202411147340.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2044-08-21
AI Technical Summary
Existing microscope focus methods are slow in high-resolution image processing and large-scale acquisition, lack of accuracy, and deep learning model training is time-consuming and complex, making it difficult to quickly process in clinical environments.
The image clarity is calculated by using the Brenner gradient function, the Cauchy curve is fitted through the Lorentz function, and the modified MobileNetV3 network is used to predict the quasi-focus distance and the Z-axis movement direction, reducing the number of axial photos, and improving the focus speed and accuracy.
The quasi-focus position can be determined by a single shot, which reduces the sensitivity to outliers, improves the accuracy and efficiency of focus results, reduces scanning time, and is suitable for rapid clinical analysis.
Smart Images
Figure CN118655691B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of glass slide focusing technology, and more specifically, to a real-time microscope focusing method, device and electronic equipment. Background Art
[0002] With the rapid development of medical imaging technology, automatic film reading technology that automatically scans and captures cell images through advanced microscopes has become an important branch in the field of medical diagnosis. In automatic film reading technology, the focusing process of the microscope is crucial. At present, traditional focusing methods mainly include active focusing and passive focusing. Active focusing relies on measuring the distance between the lens and the object being photographed, and adjusting the lens to the appropriate focus position accordingly. Passive focusing adjusts the focus by analyzing the clarity of the image signal.
[0003] Active focusing methods usually rely on complex ranging technology, which is limited in their application in high-resolution microscopy. It not only increases the cost, but is also difficult to implement in practical applications. Passive focusing technology has been widely used in automatic microscopy due to its simplicity, but it still has obvious limitations when processing high-resolution images and collecting a large number of images. It is necessary to scan and compare the clarity of the image at multiple axial positions to determine the optimal focus position. This process is not only time-consuming, but also particularly slow when processing high-resolution images. In addition, due to the limitations of the focus evaluation function, this method also has certain deficiencies in accuracy. For example, when there is a lack of sufficient texture or contrast in the image, the evaluation function may not accurately reflect the clarity of the image, resulting in inaccurate focusing.
[0004] In addition, the curve fitting method often uses interpolation to obtain the maximum value, but the number of interpolation times and outliers can cause the predicted results to be far from the actual results; and the focusing algorithm is inefficient when processing a large number of images, especially in clinical environments where a large number of samples need to be processed and analyzed quickly. This not only affects the efficiency of diagnosis, but may also lead to delays in important information. Therefore, attempts are made to use deep learning and artificial intelligence algorithms to optimize the focusing process, but deep learning model training takes a very long time, the model complexity is high and requires more expensive hardware equipment support, and deep learning algorithms usually predict the defocus distance, but the defocus distance usually cannot determine the direction.
[0005] In view of this, the present invention provides a real-time microscope focusing method, device and electronic device. Summary of the invention
[0006] In order to overcome the problems in the prior art, the present invention proposes a real-time microscope focusing method, device and electronic device, which calculates the image clarity by using the Brenner gradient function and performs Cauchy fitting of the clarity-focus position curve with the Lorentz function, so that the focal distance and Z-axis movement direction can be calculated in a single shot.
[0007] The present invention provides a real-time microscope focusing method, comprising the following steps:
[0008] S1, sending the microscopic image and the Z coordinate position corresponding to the microscopic image to the modified MobileNetV3 network;
[0009] S2, the modified MobileNetV3 network uses the prediction results as parameters in the Lorentz function;
[0010] S3, calculating the real-time focus position based on the curve fitted by the Lorentz function;
[0011] S4. Move the microscope platform to the focus position calculated in step S3.
[0012] In a preferred embodiment, the microscopic image is acquired by a single focused axial scan of the microscope.
[0013] In a preferred embodiment, modifying the MobileNetV3 network in step S2 includes the following steps:
[0014] S201, adaptively adjust the fully connected layer based on the original MobileNetV3 network architecture;
[0015] S202, adding a global average pooling layer before adjusting the fully connected layer;
[0016] S203, changing the loss function;
[0017] S204, adaptively adjust the optimizer to obtain a modified MobileNetV3 network.
[0018] In a preferred embodiment, in step S201, the activation function is deleted in the fully connected layer and the number of neurons in the fully connected layer is modified.
[0019] In a preferred embodiment, the predicted result corresponds to a continuous value, the predicted result is compared and analyzed with the actual result to obtain the difference, and the output parameter of the bottleneck layer is updated inversely using the L1 loss function based on the difference.
[0020] In a preferred embodiment, the Lorentz function includes:
[0021] ;
[0022] Where: represents the Brenner gradient value, which can be calculated , Indicates the current position, thereby inferring The exact value of Indicates the focus position; and It indicates that the prediction result in step S1 corresponds to the parameters in the Lorentz function.
[0023] In a preferred embodiment, the learning and training of the modified MobileNetV3 network includes the following steps:
[0024] Step A1: Collect training data, the training data includes the Z coordinate value and the clarity evaluation value corresponding to the current microscopic image:
[0025] Step A2: Learn and train the modified MobileNetV3 network, use the training data as the input of the MobileNetV3 network, and output the prediction results as continuous values, which are used as parameters in the Lorentz function. and .
[0026] In a preferred embodiment, the training data is collected by adopting multi-interval sampling to reduce the collection space, the collection space includes a quasi-focal plane, and the quasi-focal plane is taken as the 0 plane.
[0027] In a second aspect, the present invention provides a real-time microscope focusing device, based on the implementation of the first aspect, comprising:
[0028] The image acquisition module sends the microscopic image and the Z coordinate position corresponding to the microscopic image to the modified MobileNetV3 network;
[0029] Image processing module, the modified MobileNetV3 network uses the prediction results as parameters in the Lorentz function;
[0030] Focus analysis module, which calculates the real-time focus position based on the curve fitted by the Lorentz function;
[0031] Focus control module, moves the microscope platform to the focusing position.
[0032] In a third aspect, the present invention provides an electronic device comprising:
[0033] at least one processor; and,
[0034] A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in the first aspect.
[0035] The technical effects and advantages of a real-time microscope focusing method, device and electronic device of the present invention are as follows:
[0036] 1. The present invention only needs to take an axial picture once to determine the focal position, and the curve shape makes the movement directional.
[0037] 2. The model is lightweight and the modified deep learning network is less sensitive to outliers, reducing interference from impurities on the surface of the slide.
[0038] 3. The training data collection method can make the data in the focus interval more continuous, thereby increasing the accuracy of the results.
[0039] 4. Effectively reduce the number of axial photos taken during the scanning process, saving a lot of time. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 This is a schematic diagram of the structure of the original MobileNetV3 network;
[0041] Figure 2 This is the network structure diagram of the modified MobileNetV3 network;
[0042] Figure 3 is the Lorenz curve distribution diagram;
[0043] Figure 4 is a flow chart of the focusing method of the present invention;
[0044] Figure 5 This is a flow chart of the MobileNetV3 network modification method of the present invention;
[0045] Figure 6 The learning and training flow chart of the MobileNetV3 network modified by the present invention;
[0046] Figure 7 This is a schematic diagram of multi-interval sampling of the present invention;
[0047] Figure 8 It is a schematic diagram of the structure of the focusing device of the present invention;
[0048] Fig. 9 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] Example 1: Please refer to Figure 4 As shown, the real-time microscope focusing method described in this embodiment includes the following steps:
[0051] S1, sending the microscopic image and the Z coordinate position corresponding to the microscopic image to the modified MobileNetV3 network;
[0052] Specifically, the microscopic image is acquired by focusing the microscope and scanning the axial direction once. When the microscope is focusing and scanning the axial direction, the Z coordinate position corresponding to the current microscopic image is fixed, so the microscopic image and the Z coordinate position corresponding to the microscopic image are used as input. The parameters in the Lorentz function are generated by the MobileNetV3 network. Based on the automatic curve fitting of the Lorentz function, only one axial photo is needed to determine the quasi-focal plane, and the quasi-focal plane is taken as the 0 plane. The traditional microscope focuses and scans the axial direction at least three times to determine the focal plane, thus effectively reducing the number of axial photos during the scanning process and saving a lot of time.
[0053] S2, the modified MobileNetV3 network uses the prediction results as parameters in the Lorentz function;
[0054] Specifically, the modified MobileNetV3 network includes modifying the original MobileNetV3 model from a classification task to a regression task, and modifying the prediction result of the fully connected layer based on the regression task, wherein the prediction result is a continuous value;
[0055] Among them: The original MobileNetV3 network is a lightweight deep learning model, mainly used for image classification tasks, such as Figure 1 As shown in the figure, the original MobileNetV3 network includes multiple convolutional layers and depth-separable convolutions to reduce the amount of computation and number of parameters of the model. It is mainly used for classification tasks, and the prediction results output by its fully connected layer are category probabilities. The modified MobileNetV3 network is shown in the figure. Figure 2 As shown in the figure, in the regression task, the fully connected layer corresponding to the classification output is modified into a fully connected layer of two regression results, and the regression result is a predicted continuous value. In this way, the fully connected layer originally used for classification can be modified into a fully connected layer for predicting continuous values; the modified MobileNetV3 network is adapted to the regression task, and the regression result is used as the parameter in the Lorentz function and .
[0056] The structural changes between the modified MobileNetV3 network and the original MobileNetV3 network are mainly concentrated at the end of the network: that is, the adjustment of the fully connected layer. Other deep feature extraction parts mostly remain unchanged. The overall framework and training order will not change, but appropriate adjustments need to be made in data processing, loss calculation and evaluation indicators.
[0057] More specifically, the modified MobileNetV3 network starts from convolution, enters the bottleneck layer, and then enters the global average pooling layer for global average pooling. The global average pooling data enters the fully connected layer through 1x1 convolution to output the prediction result. The prediction result corresponds to a continuous value. The prediction result is compared with the actual result to obtain the difference. Based on the difference, the loss function is used to update the output parameters of the bottleneck layer in reverse, so as to ensure the accuracy of the prediction result. During the processing, the model is trained and the model training data is collected, and two values are output, corresponding to and , the L1 loss function is used as the loss function to optimize the model. The regression task needs to quantify the difference between the predicted value and the actual value. The characteristics of the training data collection method are adopted, that is, the collection interval is expanded without making the calculation too large. In addition, its coverage reduces the sensitivity to outliers.
[0058] like Figure 5 As shown, the specific steps to modify the MobileNetV3 network are as follows:
[0059] S201. Adaptively adjust the fully connected layer based on the original MobileNetV3 network architecture, delete the activation function in the fully connected layer and modify the number of neurons in the fully connected layer:
[0060] Specifically, delete the softmax layer: the original MobileNetV3 network is designed for classification tasks, and includes an activation function (such as softmax) in the fully connected layer to normalize the output into a probability distribution. In the regression model, the regression task does not require a softmax function, so this type of activation function needs to be removed, so that the prediction result can be a continuous value, thereby replacing the probability distribution of the original MobileNetV3 network. Modify the number of neurons in the fully connected layer: modify the number of neurons in the fully connected layer to match the target regression task. For example, in this embodiment, the number of parameters in the Lorentz function for subsequent analysis is 2, so the fully connected layer is set with two neurons.
[0061] It should be noted that: in actual applications, the modified MobileNetV3 network predicts 1 and 2 networks differently when designing the fully connected layer and loss function. When predicting 1 neuron, assuming the output is X, the loss function uses L1(x). If there are 2 neurons, assuming the output is X, Y, then the loss function uses L1(X+αY), and α needs to be set in advance. In this embodiment, α is determined by calculating the cross-correlation between A and γ in the collected training data.
[0062] S202, adding a global average pooling layer before adjusting the fully connected layer;
[0063] Specifically, the global average pooling used in this embodiment helps to reduce model parameters, prevent overfitting, and maintain spatial information, while the original MobileNetV3 network uses Max pooling. In practical applications, the choice of pooling method depends on the specific task requirements and network design. For example, Max Pooling is mainly used in scenarios that emphasize the most significant changes in feature maps, such as object recognition; global average pooling is used to replace the traditional fully connected layer before converting the feature map to the fully connected layer to reduce the number of parameters and the amount of calculation.
[0064] S203, changing the loss function;
[0065] Specifically, the loss functions commonly used in classification tasks, such as cross entropy, are not suitable for regression tasks. Common loss functions for regression tasks include mean square error (MSE), mean absolute error (MAE), and smooth L1 loss. These loss functions can measure the difference between the predicted value and the true value, so the appropriate loss function is adaptively selected.
[0066] More specifically, this embodiment uses smooth L1 loss, namely Smooth L1 Loss: it is a loss function that combines the characteristics of L1 loss (absolute value loss) and L2 loss (square loss). The L1 loss function is used as the loss function to optimize the model. The regression task needs to quantify the difference between the predicted value and the actual value. The reason for using the L1 loss function is that it is insensitive to outliers. Due to the influence of impurities on the surface of the slide, the clarity calculation may be interfered by outliers.
[0067] S204, adaptively adjust the optimizer to obtain a modified MobileNetV3 network.
[0068] Specifically, the Adam optimizer is used in the model training process. The Adam optimizer combines momentum and adaptive learning rate, and usually performs well in a variety of deep learning tasks. According to the characteristics of the regression task, the learning rate or other optimizer parameters of the optimizer are adaptively adjusted to obtain better performance.
[0069] The modified MobileNetV3 network takes the real-time microscopic image as input and the parameters required by the Lorentz function as output, which corresponds to and , using L1 loss function as loss function to optimize the model. Regression tasks need to quantify the difference between predicted values and actual values. The characteristics of the training data collection method are adopted, that is, the collection interval is expanded without excessive calculation. In addition, its coverage reduces the sensitivity to outliers. The model is retrained using the modified MobileNetV3 network and the new loss function. While retaining the weights of the feature extraction part of the original model, only the modified fully connected layer is trained. The fully connected layer does not use an activation function. The regression task does not require a nonlinear activation function because this solution predicts continuous values.
[0070] S3, using the prediction results in step S2 as the parameters of the Lorentz function and , calculate the real-time focus position based on the curve fitted by the Lorentz function;
[0071] The Lorenz curve distribution is as follows Figure 3 As shown, the Lorentz function includes:
[0072] ;
[0073] Where: represents the Brenner gradient value, which can be calculated , Indicates the current position, thereby inferring The exact value of represents the peak position of the curve, and the peak position of the curve is the focus position; and It indicates that the prediction result in step S1 corresponds to the parameters in the Lorentz function.
[0074] S4. Move the microscope platform to the focus position calculated in step S3.
[0075] Specifically, this embodiment collects microscopic images and uses the parameters generated by the MobileNetV3 network for automatic fitting, which effectively reduces the number of axial photos taken during the scanning process, saves a lot of time, and has a better focusing effect.
[0076] Example 2: Please refer to Figure 6 As shown, the part not described in detail in this embodiment is as shown in Example 1. This embodiment also provides learning and training of the modified MobileNetV3 network, including the following steps:
[0077] Step A1: Collect the training data: use multi-interval sampling to reduce the collection space, the collection space includes the quasi-focal plane, and the quasi-focal plane is the 0 plane.
[0078] Specifically, 900 sets of training data are collected based on the method of step A1, and the training data include the Z coordinate value and the clarity evaluation value corresponding to the current microscopic image, the coordinate value corresponding to the current microscopic image is 0, the Z coordinate value represents the defocus distance value between the current microscopic image and the in-focus position, and the clarity evaluation value corresponding to the current microscopic image is the Brenner gradient value calculated by the Lorentz function.
[0079] More specifically, taking the quasi-focal plane as plane 0, multiple interval sampling is performed as follows Figure 7 As shown, from the three defocus intervals of (-30, 30), (-29, 31), and (-31, 29), samples are collected every 3μm, and each set of training data contains three sets of samples. This method compensates for the incoherence of the intervals. Specifically, it includes:
[0080] The first group of samples is from -30μm to 30μm, with an interval of 3μm. Sampling includes: collecting images at corresponding positions and calculating the corresponding clarity evaluation value, L(z)=Brenner(z).
[0081] The second set of samples is from -29μm to 31μm, with an interval of 3μm. Sampling includes: collecting images at corresponding positions and calculating their corresponding clarity evaluation values, L(z)=Brenner(z).
[0082] The third group of samples is from -31μm to 29μm, with an interval of 3μm. Sampling includes: collecting images at corresponding positions and calculating their corresponding clarity evaluation values, L(z)=Brenner(z).
[0083] Multi-interval sampling not only reduces the sampling interval, but also expands the sampling interval, while making each set of training data not too large, reducing the computational cost.
[0084] Step A2: Learn and train the modified MobileNetV3 network:
[0085] Specifically, the training data is used as the input of the MobileNetV3 network, and the output prediction result is a continuous value, which is used as the parameter in the Lorentz function. and .
[0086] The Adam optimizer is used during model training. The Adam optimizer combines momentum and adaptive learning rate and generally performs well in a variety of deep learning tasks. The real-time focus position is calculated using the curve fitted by the Lorentz function.
[0087] More specifically, during the training data collection process, it is necessary to ensure the quality and consistency of the image data and perform necessary preprocessing steps, such as scaling and normalization. Although the modified MobileNetV3 network uses the L1 loss function to have a certain degree of robustness to outliers, the identification and processing strategies of outliers can also be considered in the data preprocessing stage.
[0088] Example 3: Figure 8 As shown, the parts not described in detail in this embodiment are detailed in Example 1. This embodiment provides a real-time microscope focusing device, including an image acquisition module, an image processing module, a focusing analysis module and a focusing control module, and each module is connected by wire or wirelessly:
[0089] The image acquisition module sends the microscopic image and the Z coordinate position corresponding to the microscopic image to the modified MobileNetV3 network;
[0090] Image processing module, the modified MobileNetV3 network uses the prediction results as parameters in the Lorentz function;
[0091] Focus analysis module, which calculates the real-time focus position based on the curve fitted by the Lorentz function;
[0092] Focus control module, moves the microscope platform to the focusing position.
[0093] Example 4: Fig. 9 As shown, the parts not described in detail in this embodiment are detailed in Embodiment 1. This embodiment provides an electronic device including:
[0094] At least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform a real-time microscope focusing method as described above.
[0095] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.
[0096] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.
[0097] Example 5: For details of the parts not described in this example, please refer to Example 1. This example provides a computer-readable storage medium storing instructions. When the instructions are executed on a computer, the computer executes the above-mentioned real-time microscope focusing method.
[0098] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.
[0099] In summary, the present embodiment can quickly determine the focal position after a single shot, thereby effectively reducing the number of axial shots required during the scanning process. In data acquisition, the multi-interval sampling method is adopted to make the data acquisition of the focus interval more continuous, improve the accuracy of the focus result, and more carefully evaluate the image clarity at different focal lengths, so as to more accurately determine the optimal focal length.
[0100] By modifying the deep learning network, reducing its sensitivity to outliers and becoming more lightweight, the interference caused by factors such as impurities on the slide surface can be reduced, and the accuracy and reliability of focusing can be improved. While improving the focusing efficiency and accuracy of the microscope, it also reduces the complexity and time cost of the operation, which is of great significance for improving the level of automation of medical image analysis.
[0101] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
[0102] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A real-time microscope focusing method, characterized in that: The following steps are involved: S1, sending the microscopic image and the Z coordinate position corresponding to the microscopic image to the modified MobileNetV3 network; The microscopic image is obtained by focusing the microscope and scanning the axial direction once; S2, the modified MobileNetV3 network uses the prediction results as parameters in the Lorentz function; Modifying the MobileNetV3 network includes the following steps: S201. Adaptively adjust the fully connected layer based on the original MobileNetV3 network architecture, delete the activation function in the fully connected layer and modify the number of neurons in the fully connected layer; S202, adding a global average pooling layer before adjusting the fully connected layer; S203, changing the loss function; S204, adaptively adjusting the optimizer to obtain a modified MobileNetV3 network; The predicted result corresponds to a continuous value. The predicted result is compared with the actual result to obtain the difference, and the L1 loss function is used to update the output parameters of the bottleneck layer based on the difference. The Lorentz function includes: ； Where: Represents the Brenner gradient value, calculated , Indicates the current position, thereby inferring The exact value of Indicates the focus position; and Indicates that the prediction result in step S1 corresponds to the parameter in the Lorentz function; S3, calculating the real-time focus position based on the curve fitted by the Lorentz function; S4. Move the microscope platform to the focus position calculated in step S3.
2. A real-time microscope focusing method according to claim 1, characterized in that: The learning and training of the modified MobileNetV3 network includes the following steps: Step A1: Collect training data, the training data includes the Z coordinate value and the clarity evaluation value corresponding to the current microscopic image: Step A2: Learn and train the modified MobileNetV3 network, use the training data as the input of the MobileNetV3 network, and output the prediction results as continuous values, which are used as parameters in the Lorentz function. and .
3. A real-time microscope focusing method according to claim 2, characterized in that: The training data is collected by adopting multi-interval sampling to reduce the collection space, and the collection space includes the quasi-focal plane, and the quasi-focal plane is taken as the 0 plane.
4. A real-time microscope focusing device, which is an implementation of a real-time microscope focusing method as claimed in any one of claims 1 to 3, characterized in that: include: The image acquisition module sends the microscopic image and the Z coordinate position corresponding to the microscopic image to the modified MobileNetV3 network; Image processing module, the modified MobileNetV3 network uses the prediction results as parameters in the Lorentz function; Focus analysis module, which calculates the real-time focus position based on the curve fitted by the Lorentz function; Focus control module, moves the microscope platform to the focusing position.
5. An electronic device, characterized in that: include: at least one processor; as well as, A memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute a real-time microscope focusing method as described in any one of claims 1-3.