A microscope image definition evaluation method based on an improved MobileViT network

Abstract

The application discloses a kind of definition evaluation methods of microscope image, the method is: different focusing distance is obtained by microscope microscope image, and data enhancement such as rotation and flip is carried out to image;Adjust the number of MV2 module and MobileViT block module in model to improve the expression ability and accuracy of model;Adjust model optimizer;Add cosine annealing learning rate adjustment strategy to dynamically adjust learning rate;Adjust the node number of model fully connected layer, change the model from a classification model to a regression model;And introduce mean absolute error and mean square error to calculate the error between the predicted distance of output and the real distance of image.The application proposes a microscope image definition evaluation algorithm based on MobileViT network, which can ensure the focusing distance of microscope image is predicted.

Description

TECHNICAL FIELD

[0001] The present application relates to a method for evaluating the sharpness of an image, in particular to a method for evaluating the sharpness of a microscope image. BACKGROUND

[0002] About 78% of human information is obtained from the eyes. The human eye is considered as a scaled-down optical imaging system. The quality of the imaging image determines the efficiency of information acquisition. With the progress of science, various imaging principles have been gradually understood by humans. In order to better understand the world, people have planned and manufactured many imaging systems, such as cameras, microscopes, and telescopes. Regardless of the type of optical imaging system, the most important thing is how to get a clear image.

[0003] An unclear image is often caused by two factors. One is the reason for the object being photographed itself: the brightness of the environment in which the object is located at the time and the activity state; the second is that the optical imaging system causes the image to be unclear due to its own problems: the position of the imaging surface changes, the lens parameters of the imaging are set incorrectly, etc. According to the above reasons, in order to make the imaging image clearer, people generally choose to adjust the position of the imaging surface, which is the focus we often mentioned.

[0004] In the early development of optical imaging systems, the focusing method of the image still adopts the simplest manual focusing mode. This manual focusing method requires our operators to have strong professional ability and proficiency, and the method also has strong subjective nature, takes a long time and has low accuracy. In contrast, the emergence of automatic focusing technology that replaces the human eye to complete the focusing monitoring and control can well solve the problems of time-consuming and low accuracy caused by manual focusing. The focusing efficiency is much higher than manual focusing. Therefore, since the late 19th century, the increasingly perfect theory system of optical imaging systems and the continuous development of industrial manufacturing level have laid a foundation for the research of many researchers on automatic focusing technology.

[0005] As an important part of automatic focusing technology, the sharpness evaluation method plays a crucial role in microscopic imaging. Traditional automatic focusing methods usually rely on manually designed features and rules, which are difficult to adapt to various complex scenes and condition changes. However, the introduction of deep learning technology can help us automatically learn the feature representation of microscopic image sharpness from data, improving the robustness and accuracy of automatic focusing.

[0006] The current deep learning sharpness evaluation method mainly uses convolutional neural network (CNN) as the basic architecture, which has achieved remarkable results in image processing field. However, due to the relatively weak ability of CNN to capture global context information, it cannot handle some sequence data with long-range dependencies well, so there are certain limitations in dealing with some specific tasks. In contrast, the transformer architecture has very good application effect in natural language processing field, and is widely used because of its excellent ability to capture global context information. However, for the sharpness evaluation task in image processing field, the application research of transformer architecture is very limited, even can be said to be blank. Therefore, this paper proposes a sharpness evaluation method based on improved MobileViT lightweight network model, which uses transformer architecture and has high flexibility and scalability, can greatly reduce the model complexity and computational amount while ensuring the model accuracy.

[0007] In summary, the contribution of the present application is to apply the transformer architecture to the image sharpness evaluation task, and propose a sharpness evaluation method based on improved MobileViT lightweight network model, which has high flexibility and scalability, can greatly reduce the model complexity and computational amount while ensuring the model accuracy. SUMMARY

[0008] The purpose of the present application is to propose a MobileViT network-based sharpness evaluation method for microscope images, which can ensure that the model can make correct prediction on the focusing distance of microscope images after training.

[0009] A microscope image sharpness evaluation method based on MobileViT network, the focusing process can include the following steps:

[0010] 1. Data acquisition and augmentation, the dataset was captured using a Nikon Eclipse motorized microscope, 0.75 NA, 20x objective lens. The samples used for training were 35 research grade human pathology slides stained with hematoxylin and eosin (Omano OMSK-HP50). The images were captured using a 5 million pixel color camera (Pointgrey BFS-U3-51S5C-C) with a 3.45 pm pixel size. During the acquisition, microscope cell images at different focus states were captured by moving the sample to 41 different defocus positions (ranging from -10 pm to +10 pm with a step size of 0.5 pm). In most cases, a range of -10 pm to +10 pm was sufficient to cover most of the images at different focus states. In addition, two types of samples were used as the prediction set, the first type of samples were 697 stained tissue slides from hematoxylin and eosin stained research grade human pathology slides (Omano OMSK-HP50), which were the same as the slides used in the training dataset (these slides were not used in the training process). The second type of samples were 1312 de-identified H&E skin tissue slides prepared by an independent clinical laboratory (Department of Dermatology, University of Connecticut Health Center). The first type of samples were referred to as "same tissue slides" and the second type of samples were referred to as "different tissue slides".

[0011] In addition, to address the problem of insufficient data image samples, the data augmentation strategy adopted by the present application includes two types of flipping and rotation. There are two ways to flip an image, one is vertical flipping and the other is horizontal flipping. Vertical flipping is to exchange the upper half and the lower half of the image according to the horizontal middle axis. Similarly, horizontal flipping is to exchange the left half and the right half of the image according to the vertical middle axis of the image; the rotation operation is to rotate the image by 90°, 180° and 270° with the upper left corner coordinate origin as the vertex and the x-axis positive direction as 0° angle. Finally, through a series of amplification processing, the original 13552 pictures are amplified to 72848 pictures, of which 65592 are used as training data and 7256 are used as verification data.

[0012] 2. Adjustment of module number, the MobileViT network is mainly composed of MV2 modules and MobileViT block modules, the number of these two modules will affect the computational load and expression ability of the final model, by adjusting the module number ratio of the two parts to change their relative proportion. Specifically, the number of MV2 modules in the model layer2 layer is reduced from 2 to 1, and the number of MobileViT block modules in layer3 layer is increased from 2 to 4, by increasing the number of MobileViT block modules in the model to increase the expression ability and accuracy of the model.

[0013] 3、Optimizer test adjustment, in order to further improve the performance of the model, the present application tries to use different optimizers, including SGD, Adam and AdamW. These optimizers are commonly used optimizers, and their differences mainly lie in the optimization algorithm and parameter update strategy. By comparing the effects of different optimizers, we hope to find the optimal training strategy to improve the training efficiency and generalization ability of the model. Finally, the model trained by the present application is used to predict new microscope images, and the average absolute error and mean square error are calculated to test the generalization ability and practicality of the model. The present application compares the performance of different models and optimizers in predicting new images to determine the best model and training strategy.

[0014] 4、Add cosine annealing learning rate adjustment strategy, in order to further improve the training effect of the model, this paper also introduces the cosine annealing learning rate adjustment strategy. This strategy can help the model converge quickly in the early stage of training, and gradually reduce the learning rate in the later stage, so that the model can learn the features in the data more carefully. Specifically, the present application sets the initial learning rate to a larger value, and then gradually reduces the learning rate within a certain period, while using a cosine function to adjust the change of learning rate. This adjustment strategy can make the model learn data features more efficiently and avoid overfitting in the later stage of training. The present application applies the cosine annealing learning rate adjustment strategy to the clarity evaluation method based on MobileViT, and through the comparison of experimental results, it is found that this strategy can help the model achieve better clarity evaluation results under the same training rounds, showing good effect. Therefore, the cosine annealing learning rate adjustment strategy plays an important role in the method of the present application. Its mathematical formula is expressed as follows:

[0015] wherein, is the index value of the learning rate adjustment period, and represent the maximum and minimum values of the learning rate, records how many iterations the learning rate has undergone since the last restart, is the total number of iterations in each adjustment period.

[0016] 5、Network output adjustment, in order to better adapt to the microscopic image definition evaluation task, the original classification network is adjusted to a regression network. In the classification network, the model output is the probability value of each category, and the focusing distance of the image needs to be determined according to the probability value. In the regression network, the model output is a continuous real value, which directly represents the specific focusing distance of the image, which is more in line with the actual needs of the definition of the application. Specifically, when adjusting the network structure, the last layer of the network is replaced by two fully connected layers containing the RELU activation function. On this basis, a fully connected layer with one node is added, and the output of this layer is the focusing distance of the image. At the same time, the mean square error (MSE) is used as the loss function to measure the gap between the model output focusing distance and the actual focusing distance, and the network parameters are updated through the back propagation method.

[0017] By changing the classification network to a regression network, not only the adaptability of the model is improved, but also the calculation amount and parameter quantity are reduced, further improving the efficiency and practicability of the model. In the experiment, the application found that this adjustment strategy can improve the performance of the definition evaluation method to a certain extent, and provide more accurate and reliable results for the definition evaluation of the microscopic image.

[0018] 6、Model prediction, the data set used in the prediction process is 2448x2048, and the data set used in the training and verification is 224x224, so in the prediction process, each predicted image needs to be cropped into 90 images with a size of 224x224, and the remaining part of the image is discarded. 90 images are predicted by regression, and the mean absolute error and mean square error of each image are calculated. In addition, in the microscope image, some areas may be empty, that is, there is no effective information to extract, and the focusing distance of these areas may have abnormal values. If these abnormal values are not processed, it will cause a large error to the entire prediction result. In addition, the contrast of some areas may be small or have no contrast, and these areas are also prone to abnormal values. Therefore, taking the median of all predicted focusing distances of an image as the predicted distance of the image can effectively avoid these problems. The mathematical expressions of the mean absolute error and the mean square error are as follows:

[0019] , wherein, represents the true focusing distance, represents the model predicted focusing distance.

[0020] 7、Model comparison, the application uses other different deep learning models and the improved MobileViT network model in the application to perform comparison experiments, compares the model parameter amount and other data of these networks, and similarly, the two types of prediction samples of "same tissue sections" and "different tissue sections" used in the experiments are respectively predicted and calculated. In addition, the predicted distance values of each image in the prediction set are collected, and finally the predicted distance of all prediction set images and its true distance are taken as the vertical coordinate and the horizontal coordinate respectively, and the error scatter plot of different models is drawn.

[0021] The application has the following characteristics:

[0022] 1、The application proposes a definition method based on the improved MobileViT network, which can ensure that the model trained can make correct prediction on the focusing distance of microscope images.

[0023] 2、The system is simple to implement, and the core method part can be completed only with one computer, and after the system starts running, it only needs to load the trained weight to predict the focusing distance of microscope images. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 The method flowchart proposed by the application

[0025] Figure 2 The improved MobileViT network structure diagram of the application

[0026] Figure 3 MV2 module structure diagram

[0027] Figure 4 MobileViT block module structure diagram

[0028] Figure 5 Training image data augmentation example diagram

[0029] Figure 6 Prediction image example diagram

[0030] Figure 7 Prediction image improved MobileViT network error scatter plot

[0031] Figure 8 Prediction image comparison model "same tissue section" error scatter plot

[0032] Figure 9 Prediction image comparison model "different tissue section" error scatter plot DETAILED DESCRIPTION

[0033] In the following, the application will be further described in combination with the drawings and specific implementation cases.

[0034] The present application provides a definition evaluation method based on an improved MobileViT network, which combines Figures one to nine The details are as follows:

[0035] As Figure one The flow chart of the method proposed by the present application is shown. First, in view of the problem of insufficient image data samples, the data enhancement strategy adopted by the present application includes two kinds of flipping and rotating. There are two ways of flipping an image, one is vertical flipping, and the other is horizontal flipping. Vertical flipping is to exchange the upper half and the lower half of the image according to the horizontal center axis. Similarly, horizontal flipping is to exchange the left half and the right half of the image according to the vertical center axis of the image. Through a series of expansion processing, the original 13552 pictures are expanded to 72848 pictures, of which 65592 are used as training data and 7256 are used as verification data. The images are input into the improved MobileViT network in batches for regression training, and the size of the input image in the network is fixed at 224x224. In this experiment, the input microscope image set is trained by the regression model, and the training results of each image are summed and averaged to finally calculate the average focusing distance of all training images. In the prediction process, the mean absolute error (MAE) and the mean squared error (MSE) are introduced as evaluation indexes of the model, and the smaller the value, the closer the prediction result is to the actual result. The mathematical formula is as follows:

[0036] , , wherein represents the true focusing distance, represents the focusing distance predicted by the model.

[0037] It should be noted that the size of the data set used in the prediction process is 2448x2048, while the size of the data set used in the training and verification is 224x224, so in the prediction process, each predicted image needs to be cropped into 90 images of size 224x224, the remaining part of the image is discarded, and the average absolute error and mean square error of each image are calculated. In addition, in the microscope image, some areas may be empty, that is, there is no effective information to extract, and the focusing distance of these areas is likely to have outliers. If these outliers are not processed, it will cause a large error in the entire prediction result. In addition, the contrast of some areas may be small or have no contrast, and these areas are also prone to outliers. Therefore, taking the median of all predicted focusing distances of an image as the predicted distance of the image can effectively avoid these problems. Finally, the error scatter plot is drawn.

[0038] As Figure two The structure diagram of the improved MobileViT network of the application is shown, and the MobileViT network is mainly composed of MV2 modules and MobileViT block modules. The number of the two modules will affect the calculation amount and expression ability of the final model. By adjusting the module number ratio of the two parts to change their relative proportion. Specifically, the number of MV2 modules in the model layer2 layer is reduced to 1, and the number of MobileViT block modules in layer3 layer is increased from 2 to 4. By increasing the number of MobileViT block modules in the model, the expression ability and accuracy of the model are increased.

[0039] As Figure three The structure diagram of the MV2 module is shown. In the MV2 structure diagram, first, a 1x1 convolution kernel is used to perform dimensionality increasing operation on the image to make the channel deeper, and then a 3x3 convolution kernel is used for depth separable convolution, and finally a 1x1 convolution kernel is used for dimensionality reduction operation. The activation function is set to ReLU6, which sets the input value less than 0 to 0, does not change the input value when the input value is in the interval of 0-6, and assigns the input value greater than 6 to 6. The specific expression formula is as follows:

[0040] .

[0041] As Figure fourThe diagram shows the structure of the MobileViT block. In the MobileViT block, the initial input is a feature map with height (H), width (W), and number of channels (C). This is convolved by a 3×3 kernel, followed by a 1×1 kernel to adjust the number of channels for local representations. Then, global representations are performed by expanding L Transformer Blocks, folding back the feature map, and finally using a 1×1 kernel to adjust the number of channels again to restore the original feature map to its original state. Figure one The two feature maps are then concatenated using a shortcut branch and fused using a 3×3 convolution kernel.

[0042] like Figure five and Figure six The images shown are examples of augmented training and predicted image data. To improve the robustness and generalization ability of the model, various data augmentation methods were applied to the training data, including rotation, horizontal and vertical flipping, etc. These methods increase the diversity and quantity of the dataset, effectively avoiding overfitting. In addition, two different types of samples were used as the prediction set. The first type consisted of 697 stained tissue slides (Omano OMSK-HP50) from research-grade human pathology sections stained with hematoxylin and eosin, the same slides used in the training dataset (these slides were not used during training). The second type consisted of 1312 de-identified H&E skin tissue slides prepared by an independent clinical laboratory (Department of Dermatology, University of Connecticut Health). The first type of samples was referred to as "identical tissue slides," and the second type as "different tissue slides."

[0043] like Figure seven , Figure eight and Figure nine The table shows the scatter plots of errors from the improved MobileViT network for predicting images, the scatter plots of errors from the "same tissue slice" comparison model for predicting images, and the scatter plots of errors from the "different tissue slices" comparison model for predicting images, in an experiment evaluating the sharpness of microscope images. First, to verify the training efficiency and generalization ability of different optimizers on the MobileViT network model used in this experiment, regression predictions were performed on the prediction set using the network models trained under different optimizers. The mean absolute error and mean squared error under different optimizers were obtained. Prediction calculations were performed separately for the two types of prediction samples used in the experiment: "same tissue slice" and "different tissue slice". Specific experimental results are shown in Table 1, with all values ​​rounded to four decimal places.

[0044] According to Table 1, it is found through experimental tests that the AdamW optimizer achieves the best effect on the model used in the present application when different optimizers are used to optimize the network model, indicating that the AdamW optimizer is more suitable than SGD and Adam for this task. The AdamW optimizer adds a weight decay term based on the Adam optimizer, which can effectively control the overfitting problem of the model and performs better in handling L2 regularization. Therefore, the AdamW optimizer is used as the optimizer for training the model in the present application.

[0045] Table 1 MobileViT network prediction results under different optimizers

[0046] Optimizer Same tissue slice Same tissue slice Different tissue slice Different tissue slice MAE MSE MAE MSE SGD 1.3894 2.1445 2.0206 6.4182 Adam 0.6643 0.5255 1.2169 1.8251 AdamW 0.4481 0.3118 0.8443 1.3299

[0047] Secondly, after selecting the model optimizer, the present application uses deep learning models AlexNet, VGG16, ResNet50, MobilenetV2, MobilenetV3 and ConvNeXt to conduct comparative experiments with the improved MobileViT network model used in the present application, and compares the computational amount, model parameter amount and other data of these networks; similarly, the two types of prediction samples "same tissue slices" and "different tissue slices" used in the experiment are respectively predicted and calculated. The specific experimental results are shown in Tables 2 and 3. In addition, the predicted distance values of each image in the prediction set are collected, and finally the predicted distance of all prediction set images and its true distance are taken as the vertical coordinate and horizontal coordinate respectively, and the error scatter plot of different models is drawn.

[0048] Table 2 AdamW optimizer "same tissue slice" result comparison

[0049] Model Flops Params MAE MSE AlexNet 0.71B 61.1M 2.0755 4.4333 VGG16 154B 0.1G 1.9089 3.8790 ResNet50 4.14B 25.6M 0.5119 0.3430 MobilenetV2 0.33B 3.5M 1.3175 1.9603 MobilenetV3 0.66B 4.1M 1.1490 1.5785 ConvNeXt 0.38B 15.4M 1.1635 1.3871 Improved MobileViT 0.24B 3.6M 0.4481 0.3118

[0050] According to Table 2, under the AdamW optimizer, because the prediction image "same tissue slice" used is also a stained tissue slide from the hematoxylin and eosin-stained research-grade human pathological section (Omano OMSK-HP50), the mean absolute error and mean square error are smaller than those using "different tissue slices" prediction images in Table 1.

[0051] In addition, compared with ResNet50, the improved MobileViT network adopts the Transformer architecture, not only has higher representation ability and better generalization ability, but also can use the attention mechanism of Transformer to improve the feature extraction ability and position perception ability of the network, thereby further improving the performance of the model. In addition, MobileViT further reduces the computational complexity and storage space occupation of the model through the careful design of deep separable convolution, attention mechanism and modular structure, and uses some skills such as data augmentation and optimizer adjustment in the training process to optimize the performance of the network. These optimization measures make the improved MobileViT network of the present application can surpass the traditional ResNet50 network in both training speed and model performance. Therefore, in Table 2, the improved MobileViT network model used in this paper achieves the smallest mean absolute error and mean square error, which is about 14.2% and 10% smaller than the second smallest ResNet50 network.

[0052] Table 3 Comparison of results of "different tissue slices" under AdamW optimizer

[0053] Model Flops Params MAE MSE AlexNet 0.71B 61.1M 3.6133 17.949 VGG16 154B 0.1G 2.9609 9.3219 ResNet50 4.14B 25.6M 0.7650 0.9912 MobilenetV2 0.33B 3.5M 2.0234 4.3142 MobilenetV3 0.66B 4.1M 1.9044 3.9488 ConvNeXt 0.38B 15.4M 1.2581 1.6388 Improved MobileViT 0.24B 3.6M 0.8443 1.3299

[0054] However, as shown in Table 3, the improved MobileViT network model proposed in this application only achieved the second lowest mean absolute error and mean square error on the predicted images of "different tissue slices", and the reason may be that the similarity between the predicted images and the training images is too low, and MobileViT as a lightweight network model still has some shortcomings compared with ResNet50, so that the mean absolute error and mean square error did not achieve the minimum value. However, although the mean absolute error and mean square error did not take the minimum value, they were only about 9.4% and 25.5% larger than the smallest ResNet50 network.

[0055] As shown in Figure seven and Figure eight , the predicted distance of all prediction set images and its true distance are taken as the vertical coordinate and horizontal coordinate respectively, and the focus error scatter plot of the comparison model is drawn. Since the prediction results of AlexNet and VGG16 network for the prediction images in the experiment deviate greatly from the actual results, they are not shown in the focus error scatter plot. The focus error scatter plot of the improved MobileViT network is as shown in Figure sixAs shown in the figure, it can be seen from the figure that the error of the predicted distance of the improved MobileViT network used by the application and the real distance is basically concentrated in the range of-0.5 to 0.5 mu m, and the predicted distances of the other several network models are slightly worse than the predicted result of the improved MobileViT network, which also shows that some optimization and fine-tuning of the application on the MobileViT network improves the performance of the network to a certain extent.

Claims

1. A microscope image sharpness evaluation method based on an improved MobileViT deep neural network, characterized in that, The method comprises the following steps: (1) image preprocessing step: size normalization and data augmentation of the microscope image input into the network; (2) network model construction step: improved MobileViT network is adopted, wherein the number of MV2 modules in the second layer is reduced from 2 to 1, and the number of MobileViTblock modules in the third layer is increased from 2 to 4; (3) optimizer selection step: by comparing the experiments of different optimizers, the AdamW optimizer is selected as the optimizer; (4) learning rate adjustment step: the cosine annealing learning rate adjustment strategy is adopted, and the learning rate is dynamically adjusted according to the preset formula; (5) regression output step: the last layer of the MobileViT network is replaced by a full connection layer containing two full connection layers and a full connection layer with an output node of 1, so that the network output is a regression value representing the focusing distance of the image; (6) error calculation step: the mean absolute error and the mean squared error between the predicted focusing distance of the model and the real focusing distance are calculated as evaluation indexes.

2. The method of claim 1, wherein, The image preprocessing step comprises: performing size normalization on the image input into the network to scale the image size to 224*224 pixels; and the image augmentation method is to perform vertical mirroring, horizontal mirroring and rotation operation at angles of 90°, 180° and 270° on the divided image for augmentation.

3. The method of claim 1, wherein, The optimizer selection step comprises: by comparing the experiments of SGD, Adam and AdamW optimizers, based on the mean absolute error and mean squared error results, the AdamW optimizer is selected.

4. The method of claim 1, wherein, The learning rate adjustment step adopts the cosine annealing learning rate adjustment strategy, and the learning rate is adjusted according to the following formula: where i is the index value of the learning rate adjustment period, and min and max represent the minimum and maximum values of the learning rate, respectively, T cur records how many iterations the learning rate has gone through since the last restart up to the current position, T i is the total number of iterations in each adjustment period.

5. The method of claim 1, wherein, The error calculation step comprises: two errors of mean absolute error (MAE) and mean squared error (MSE) are introduced as evaluation indexes, the error between the focusing distance predicted by the model for the microscope image and the real focusing distance of the microscope image is calculated as the evaluation index of the model, and the smaller the error value, the closer the predicted result is to the actual result, and the mathematical formula is as follows where y i denotes the true focus distance, denotes the model-predicted focus distance, m denotes the number of samples.

Images