Microstructure image segmentation method based on semi-supervised learning and calculation device

Abstract

The invention discloses a microstructure image segmentation method based on semi-supervised learning. The microstructure image segmentation method comprises the following steps: providing a training sample comprising a label data set and a label-free data set; an image segmentation network Net A and an image segmentation network Net B based on the DenseUNet are established; a random disturbance input Net A is applied to a sample image and a label in a training sample, the same disturbance is applied after the training sample is input into a Net B, a total loss function based on a training round number weight for constraining an output result of the Net A and an output result of the Net B is minimum so as to update a network parameter theta of the Net A, and the Net B is further updated by using an index moving average of the theta, so that the network parameter theta of the Net A is updated. And when the difference value of the output results of the Net A and the Net B is smaller than a given threshold value, training is ended, and the Net B is utilized to execute microstructure image segmentation. According to the method, the non-label data is fully utilized for network training, the training efficiency of the image segmentation network is improved, and the accuracy of microstructure image segmentation is improved. The invention further provides a computing device.

Description

Technical Field

[0001] The present invention belongs to the field of material characterization, and particularly relates to a microscopic tissue image segmentation method and a computing device based on semi-supervised learning. Background Art

[0002] Microscopic tissue analysis is one of the bases for metal material characterization. Identifying different phases in the alloy microscopic tissue and determining the size and distribution of the matrix or precipitated phase are of great significance for clarifying the tissue evolution process of the alloy and guiding the optimization of alloy composition, process, and properties. Analyzing microscopic tissue images relying on manual identification is very inefficient. With the development of technologies in the field of computer vision, using image segmentation networks to batch process microscopic tissue images is obtaining more and more applications. However, existing segmentation networks such as PCN, CNN, or Deeplab, etc. adopt supervised networks, and their training process has a large dependence on labeled data, while labeled data often still needs to be manually annotated, resulting in the restriction of training efficiency and image segmentation efficiency. Therefore, providing a microscopic tissue image segmentation method based on semi-supervised learning to improve the utilization level of unlabeled data has high value for improving image segmentation efficiency. Summary of the Invention

[0003] The purpose of the present invention is to provide a microscopic tissue image segmentation method based on semi-supervised learning to improve the training efficiency of the image segmentation network using unlabeled data. The present invention also provides a computing device.

[0004] According to an embodiment of one aspect of the present invention, there is provided a microscopic tissue image segmentation method based on semi-supervised learning, and the method includes the following steps:

[0005] Providing a training sample including a plurality of microscopic tissue images, the training sample including a labeled data set and an unlabeled data set, the microscopic tissue images in the labeled data set being labeled images, the microscopic tissue images in the unlabeled data set being unlabeled images, the number of the unlabeled images being not less than that of the labeled images, and the label being the segmentation result of a given sample image;

[0006] Constructing an image segmentation network based on DenseUNet, the image segmentation network including Net A and Net B, and Net A and Net B having the same structure;

[0007] Providing a random perturbation u i to the training sample and the label in the labeled data set, and inputting the result into Net A to obtain the output result of Net A; inputting the training sample into Net B for processing and then applying the random perturbation u iObtain the output result of NetB. The output results of Net A and Net B respectively include and output labels;

[0008] Establish the cross-entropy loss function for the labeled dataset:

[0009] where x i is the input sample image, y i is the label, θ is the parameter of each layer of neurons in Net A, f(x i , θ) is the image segmentation network, and B is the number of input images;

[0010] Establish the consistency loss function for the output results of Net A and Net B:

[0011]

[0012] where, μ i ’ is the output label in the output result of Net A, and μ i is the output label in the output result of Net B;

[0013] Perform multiple rounds of training on Net A and Net B. Set the overall loss function loss = a(T)L + b(T)J, where T is the number of training rounds, a(T) and b(T) are weight functions, and the value of b(T) / a(T) is monotonically increasing with respect to T;

[0014] Constrain the overall loss function to take the minimum value to update the parameter θ in Net A, and use the exponential moving average θ’ of the parameter θ to update the parameter in Net B;

[0015] When the difference between the output results of Net A and Net B is less than the given consistency threshold, stop training, and use Net B to perform microscopic tissue image segmentation to obtain the output image.

[0016] Through this method, the unlabeled image data can be effectively used to train the image segmentation network, improve the utilization efficiency of data, reduce the workload of manual labeling, and improve the segmentation efficiency of microscopic tissue images.

[0017] Further, in some embodiments, the image segmentation network includes multiple layers of networks, each layer of the network includes an encoder and a decoder, the encoder includes a convolution module, after the sample image is input into the image segmentation network, the sample image is converted into a digital matrix and the feature image of the sample image is extracted through the convolution module; each convolution module in the encoder downsamples the feature image through one max pooling, and the decoder restores the feature image through upsampling; the encoder and the decoder are connected by skip connection layers corresponding to the highest feature image sampling; the end of the decoder uses a 1*1 convolution to restore the original size of the sample image and outputs the corresponding output label.

[0018] Further, in some embodiments, the random perturbation u i includes rotation or mirror flipping.

[0019] Further, in some embodiments, a(T) = 1, where k is a weight parameter.

[0020] Further, in some embodiments, θ t ’ = αθ t-1 ’ + (1 - α)θ t , where α is a balance coefficient and t is a sequence number.

[0021] Further, in some embodiments, after performing microstructural image segmentation using the Net B, a morphological processing step is further included, and the morphological processing step fills the voids in the image output by the Net B.

[0022] Further, in some embodiments, the morphological processing step includes forming a probability image of the output image, setting the threshold to 0.5 to generate a binary segmentation result to fill the voids.

[0023] According to an embodiment of another aspect of the present invention, a computing device is provided, the computing device includes a memory and a processor, the memory stores a computing program, and when the computing program is executed by the processor, it can implement the method for microstructural image segmentation based on semi-supervised learning provided in any of the foregoing embodiments. Description of the Drawings

[0024] Figure 1 is a flowchart of a method for microstructural image segmentation based on semi-supervised learning in an embodiment;

[0025] Figure 2a is an alloy microstructural image to be processed in an embodiment;

[0026] Figure 2b is a label of the alloy microstructural image in an embodiment;

[0027] Figure 2c The segmentation result of Net B in an embodiment;

[0028] Figure 2d The segmentation result after morphological processing in an embodiment.

[0029] The purpose of the above-mentioned drawings is to make a detailed description of the present invention so that those skilled in the art can understand the technical concept of the present invention, rather than aiming to limit the present invention. Detailed implementation manners

[0030] The present invention will be further described in detail below through specific embodiments in conjunction with the drawings.

[0031] The mention of "embodiment" in this article means that the specific features, structures or characteristics described in connection with the embodiment may be included in at least one embodiment of this article. The phrase appearing in various positions in the specification does not necessarily refer to the same embodiment, nor is it limited to mutually exclusive independent or alternative embodiments. Those skilled in the art should be able to understand that the embodiments in this article can be combined with other embodiments without structural conflicts. In the description of this article, the meaning of "a plurality" is at least two.

[0032] Using computer vision recognition to replace manual segmentation of the microstructure of alloy materials and quantitatively identify and analyze the microstructure can effectively improve the efficiency of material characterization. At present, some image segmentation networks based on machine learning, such as FCN, CNN or Deeplab, etc., have achieved good applications in the field of microstructure image segmentation. However, these image segmentation networks rely on the training method of supervised learning and need to be supported by sufficient labeled data to fully exert their effects. For microstructure analysis, the process of obtaining labeled data (in microstructure image segmentation, the label is the segmentation result of a given image) itself is relatively difficult. Relying on manual labeling of microstructure images is costly and inefficient. Providing a large number of labeled images for the purpose of training an image segmentation network is contrary to the purpose of realizing automated microstructure images.

[0033] To solve the above problems, an embodiment of one aspect of the present invention provides a microstructure image segmentation method based on semi-supervised learning, which can use a small amount of labeled data and a large amount of unlabeled data to train an image segmentation network and improve the efficiency of microstructure image segmentation and analysis. This method is as Figure 1 shown and includes the following steps:

[0034] First, perform Step 1. Provide training samples including multiple microstructural images. The training samples include a labeled dataset with fewer sample images and an unlabeled dataset with more sample images. Limited by the field of view of electron microscope photography, certain preprocessing needs to be carried out on the photos obtained by electron microscope photography. Take photos of small areas separately and then splice them into a large image. Since the large-size image has a higher resolution, considering the processing power of the computer and the computational time cost, the image needs to be cropped to obtain segmented small images, and the bicubic interpolation algorithm is used to adjust the image size to obtain the final sample image as shown in Figure 2a . The sample image uses the superalloy microstructure image obtained at 800 °C. The training samples include 155 labeled sample images with a resolution of 512×512 and 1000 unlabeled sample images with the same resolution. For the sample image as shown in Figure 2a , its label is as shown in Figure 2b , which is a grayscale image with pixel values between 0 and 255. The area to be recognized is 255, and the background area is 0. The label is obtained by manual annotation. Denote the labeled dataset as D L : {x i , y i}, and the unlabeled dataset as D U : {x i}, where x i is the image sample and y i is the label.

[0035] Next, perform Step 2. Build network models NetA and NetB as image segmentation networks with DenseUNet as the basic structure. Since there are many tiny voids in the alloy fiber structure, traditional image segmentation networks such as FCN, ResNet, DenseNet, etc. have limited reception of image information from different depth networks. UNet uses skip connections to obtain semantic features in the encoder structure, and DenseUNet directly connects each layer in the multi-layer neural network to the previous layers, which is beneficial to the repeated use of features, reduces unnecessary calculations, and can effectively reduce the risk of gradient descent or disappearance caused by the increase in network depth. Among them, Net A and Net B have the same structure.

[0036] Specifically, in the preferred embodiment, the image segmentation networks (Net A and Net B) include a multi-layer neural network. Each layer of the network includes an encoder and a decoder, and the encoder has a convolutional module. After the sample image is input into the image segmentation network, the sample image is converted into a digital matrix, and the convolutional module extracts the feature image of the sample image. In the encoder, the feature image output by each convolutional module is downsampled through a max pooling operation, and in the decoder, the feature image information is restored through an upsampling operation. The network encoder and decoder corresponding to the highest feature images are connected using the same skip connection layer. At the end of the decoder, a 1*1 convolution is used to restore the original input image size, and at the same time, the output label predicted by the image segmentation network is output.

[0037] Subsequently, step 3 is performed. A random perturbation u is applied to the sample images in the training samples i , where the sample image X i in the labeled dataset i is applied with the same random perturbation u i . In different embodiments, the random perturbation u i includes rotation or mirror flipping. The training samples with the applied random perturbation are input into Net A to obtain the output result of Net A. Since the convolutional network is insensitive to the transformed (randomly perturbed) image, the same transformation is performed on the sample image before and after convolution, and different results are obtained after convolution. After the training samples are input into Net B, the same random perturbation u i is applied to it to obtain the output result of Net B.

[0038] Next, step 4 is executed. The loss functions of the two types of labels are calculated to minimize the overall loss function.

[0039] The cross-entropy loss function of the labeled data is established: where θ is the parameter of the neurons in each layer of Net A, f(x i , θ) is the image segmentation network, which is a non-linear function, and B is the number of images input into Net A in each batch.

[0040] The consistency loss function between the output result of Net A and the output result of Net B is established: where, μ i ’ is the pixel-level output label in the output result of Net A, and μ i is the pixel-level output label in the output result of Net B. In the initial stage of training, the training accuracy of the labeled dataset is much higher than that of the unlabeled dataset. After multiple rounds of training, the training effect of the unlabeled dataset will improve. Therefore, it is necessary to set a weight coefficient that changes with the training round T to gradually increase the importance of the unlabeled dataset in the overall loss function. The overall loss function is denoted as loss =

[0041] a(T)L + b(T)J, where a(T) and b(T) are weight functions, and the value of b(T) / a(T) is monotonically increasing with respect to T to achieve a gradual increase in the weight of J. In a preferred embodiment, a(T) = 1 (i.e., a constant), where k is a given weight parameter used to determine the weight of the unlabeled dataset in the final image segmentation network, and the value of k can be taken as 1. As T increases, b(T) monotonically increases and gradually approaches k.

[0042] Next, perform step 5. Update the network parameters of Net A and update the network parameters of Net B by moving average.

[0043] After establishing the overall loss function loss, the network parameters θ in Net A can be updated by minimizing the overall loss, and then the network parameters in Net B can be updated using the exponential moving average θ' of θ. In a preferred embodiment, θ t ’ = αθ t-1 ’ + (1 - α)θ t , where α is a balance coefficient, t is the serial number, θ is the parameter obtained after error backpropagation in the Net A network, and θ' is the parameter of Net B.

[0044] Finally, perform step 6. At the initial stage of network training, the predicted label result of Net B for the unlabeled sample image has a large error and mainly relies on Net A for prediction. As the number of training times increases, the accuracy of Net B gradually improves. After the output results of Net A and Net B are less than a given consistency threshold, it is considered that the output results of Net A and Net B are already similar. Taking the intersection over union IoU (the ratio of the intersection of the part predicted as true in the prediction result and the part that is true in the label to the union of the two) as an index to measure the accuracy of the network for segmenting images, since the network parameters θ' of Net B are obtained by moving average of the network parameters θ of Net A, it has better stability for image segmentation. Therefore, Net B is used to perform microscopic tissue image segmentation to obtain the output image.

[0045] For a sample image as Figure 2a shown, the result expected to be obtained by processing with the image segmentation network is the same as the "label" of the manual segmentation as Figure 2b shown. After multiple rounds of training, the "label" predicted by Net B for the sample image, that is, the segmentation result, is as Figure 2c shown. It can be seen that in the prediction result of Net B, for Figure 2a the area 7 with a smaller area in, the segmentation result is accurate, and the segmentation result 7' in Figure 2c is accurately obtained, but there is an error in the prediction of the area 8 with a larger area. InFigure 2c An error region 9 is formed, indicating that there are defects in the segmentation results of the image segmentation network for relatively large voids. In a preferred embodiment, after Net B performs microscopic tissue image segmentation on the image, morphological processing is required to fill the voids. In a further preferred embodiment, the following method is adopted: a probability image is formed from the output image of Net B, the threshold is set to 0.5 to generate a binary segmentation result, and the error region 2c is filled to obtain Figure 2d the corrected region 8' as shown. The segmentation result after morphological processing Figure 2d has good consistency with the manual label Figure 2b indicating that the efficient automatic segmentation of microscopic tissue images can be effectively achieved through the above method.

[0046] An embodiment of another aspect of the present invention further provides a computing device. The computing device includes a memory and a processor. A computing program is stored in the memory. When the computing program is executed by the processor, the microscopic tissue image segmentation method based on semi-supervised learning provided in the above embodiments can be implemented. The computing device can be a general-purpose computer or a dedicated computing device specifically built for image segmentation.

[0047] The purpose of the above embodiments is to further elaborate on the present invention in conjunction with the accompanying drawings so that those skilled in the art can understand the technical concept of the present invention. Within the scope disclosed by the present invention, optimizing or equivalently replacing the method steps involved, and combining the implementation manners in different embodiments without conflict in principles, all fall within the protection scope of the present invention.

Claims

1. A method for microscopic tissue image segmentation based on semi - supervised learning, characterized in that, it includes the following steps: Provide a training sample including multiple microscopic tissue images, the training sample includes a labeled data set and an unlabeled data set. The microscopic tissue images in the labeled data set are labeled images, and the microscopic tissue images in the unlabeled data set are unlabeled images. The number of unlabeled images is not less than that of labeled images. The label is the segmentation result of a given sample image; Build an image segmentation network based on DenseUNet. The image segmentation network includes Net A and Net B, and Net A and Net B have the same structure; Provide a random perturbation u to the labels in the training samples and the labeled dataset i , and input it into the Net A to obtain the output result of Net A; Input the training sample into Net B, and then apply the random perturbation u i Obtain the output result of Net B. The output results of Net A and Net B respectively include output labels; Establish the cross - entropy loss function of the labeled data set: where x i is the input sample image, y i is the label, θ is the parameter of each layer of neurons in the Net A, f(x i , θ) is the image segmentation network, and B is the number of input images; Establish the consistency loss function of the output results of Net A and Net B: where μ i ’ is the output label in the output result of the Net A, and μ i is the output label in the output result of the Net B; Perform multiple rounds of training on Net A and Net B, and set the overall loss function loss = a(T)L + b(T)J, where T is the number of training rounds, a(T) and b(T) are weight functions, and the value of b(T) / a(T) is monotonically increasing with respect to T; Update the parameters θ in Net A by constraining the overall loss function to take the minimum value, and update the parameters in Net B using the exponential moving average θ’ of the parameters θ; When the difference between the output results of Net A and Net B is less than a given consistency threshold, stop training, and use Net B to perform microscopic tissue image segmentation to obtain an output image.

2. The method for microscopic tissue image segmentation based on semi - supervised learning according to claim 1, characterized in that, The image segmentation network includes multiple layers of networks. Each layer of the network includes an encoder and a decoder. The encoder includes a convolution module. After the sample image is input into the image segmentation network, the sample image is converted into a digital matrix, and the convolution module extracts the feature image of the sample image; each convolution module in the encoder performs downsampling on the feature image through one max - pooling. The decoder restores the feature image through upsampling; the encoder and the decoder are connected by skip connection layers that sample the same highest - level feature image; the end of the decoder uses a 1*1 convolution to restore the original size of the sample image and outputs the corresponding output label.

3. The method for microscopic tissue image segmentation based on semi - supervised learning according to claim 1 or 2, characterized in that, The random perturbation u i includes rotation or mirror flipping.

4. The method for microscopic tissue image segmentation based on semi - supervised learning according to claim 1 or 2, characterized in that, a(T) = 1, where k is a weight parameter.

5. The method for microscopic tissue image segmentation based on semi - supervised learning according to claim 1 or 2, characterized in that, θ t ’ = αθ t-1 ’ + (1 - α)θ t , where α is the balance coefficient and t is the serial number.

6. The method for microscopic tissue image segmentation based on semi - supervised learning according to claim 1 or 2, characterized in that, After performing microscopic tissue image segmentation using Net B, it further includes a morphological processing step, and the morphological processing step fills the voids in the output image of Net B.

7. The method for microscopic tissue image segmentation based on semi-supervised learning according to claim 6, characterized in that, the morphological processing step includes forming a probability image of the output image, setting a threshold to 0.5 to generate a binary segmentation result, and filling the voids.

8. A computing device, comprising a memory and a processor, characterized in that, the memory stores a computing program, and when the computing program is executed by the processor, it can implement the method for microscopic tissue image segmentation based on semi-supervised learning according to any one of claims 1 to 7.

Images