A microstructure image segmentation method and computing device based on semi-supervised learning

Abstract

A microstructure image segmentation method based on semi-supervised learning comprises the following steps: providing training samples including a labeled data set and an unlabeled data set; establishing image segmentation networks Net A and Net B based on DenseUNet; inputting sample images and labels in the training samples into Net A after applying random disturbance, inputting the training samples into Net B and then applying the same disturbance, constraining the overall loss function based on the training round weight of the output results of Net A and Net B to be minimum to update the network parameters θ of Net A, and further updating Net B by using the exponential moving average of θ until the output result difference between Net A and Net B is less than a given threshold, ending the training and performing microstructure image segmentation by using Net B. The method fully utilizes the unlabeled data for network training, improves the training efficiency of the image segmentation network, and improves the accuracy of the microstructure image segmentation. The application also provides a computing device.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of material characterization, and particularly relates to a microstructure image segmentation method based on semi-supervised learning and a computing device. BACKGROUND

[0002] Microstructure analysis is one of the foundations of metal material characterization. Identifying different phases in alloy microstructure, determining the size and distribution of matrix or precipitated phase, and understanding the microstructure evolution process of the alloy are of great significance for optimizing alloy composition, process and performance. Manual identification for microstructure image analysis is very inefficient. With the development of computer vision technology, image segmentation networks are increasingly used for batch processing of microstructure images. However, existing segmentation networks such as PCN, CNN or Deeplab use supervised networks, which rely heavily on labeled data. However, labeled data often still requires manual annotation, which limits the training efficiency and image segmentation efficiency. Therefore, a microstructure image segmentation method based on semi-supervised learning is provided to improve the utilization level of unlabeled data and improve the image segmentation efficiency. SUMMARY

[0003] The purpose of the present application is to provide a microstructure image segmentation method based on semi-supervised learning, which uses unlabeled data to improve the training efficiency of the image segmentation network. The present application also provides a computing device.

[0004] According to an embodiment of one aspect of the present application, a microstructure image segmentation method based on semi-supervised learning is provided, which comprises the following steps:

[0005] A training sample comprising a plurality of microstructure images is provided, the training sample comprising a labeled data set and an unlabeled data set, the microstructure images in the labeled data set being labeled images, the microstructure images in the unlabeled data set being unlabeled images, the number of unlabeled images being not less than that of labeled images, and the label being the segmentation result of a given sample image;

[0006] An image segmentation network based on DenseUNet is built, the image segmentation network comprising Net A and Net B, the Net A and the Net B having the same structure;

[0007] Random disturbance u is provided to the training sample and the label in the labeled data set i , and the Net A output result is obtained by inputting the Net A; the training sample is input into the Net B for processing, and then the random disturbance u iobtaining a Net B output result, the Net A output result and the Net B output result each comprising and output label;

[0008] establishing a cross-entropy loss function of the labeled data set:

[0009] wherein x i is an input sample image, y i is a label, θ is a parameter of each layer of neurons in the Net A, f(x i , θ) is the image segmentation network, and B is the number of input images;

[0010] establishing a consistency loss function of the Net A output result and the Net B output result:

[0011]

[0012] wherein μ i ' is the output label in the Net A output result, and μ i is the output label in the Net B output result;

[0013] performing multi-round training on the Net A and the Net B, setting a total loss function loss = a(T) L + b(T) J, wherein T is the number of training rounds, a(T) and b(T) are weight functions, and the value of b(T) / a(T) monotonically increases with respect to T;

[0014] constraining the total loss function to take a minimum value, updating the parameter θ in the Net A, and updating the parameter in the Net B by using an exponential moving average θ' of the parameter θ;

[0015] stopping training when the difference between the Net A output result and the Net B output result is less than a given consistency threshold, and performing microscopic tissue image segmentation by using the Net B to obtain an output image.

[0016] By this method, the image segmentation network can be effectively trained by using unlabeled image data, the utilization efficiency of data is improved, the workload of manual marking is reduced, and the segmentation efficiency of microscopic tissue images is improved.

[0017] Further, in some embodiments, the image segmentation network comprises a multi-layer network, each layer of the network comprising an encoder and a decoder, the encoder comprising a convolution module, after the sample image is input into the image segmentation network, the sample image is converted into a digital matrix and the feature image of the sample image is extracted through the convolution module; each convolution module in the encoder performs down-sampling on the feature image through one maximum pooling; the decoder restores the feature image through up-sampling; the encoder and the decoder are connected through a same skip connection layer corresponding to the highest feature image; a 1*1 convolution is used at the end of the decoder to restore the original size of the sample image, and the corresponding output label is output.

[0018] Further, in some embodiments, the random disturbance u i including rotation or mirror flip.

[0019] Further, in some embodiments, a(T) = 1, wherein k is a weight parameter.

[0020] Further, in some embodiments, θ t ’ = aθ t-1 ’ + (1-a)θ t , wherein a is a balance coefficient and t is a serial number.

[0021] Further, in some embodiments, after the microstructure image segmentation is performed by using the Net B, a morphological processing step is further included, the morphological processing step fills the gaps in the image output by the Net B.

[0022] Further, in some embodiments, the morphological processing step comprises forming a probability image for the output image, setting a threshold value to 0.5 to generate a binary segmentation result, and filling the gaps.

[0023] According to an embodiment of another aspect of the present application, a computing device is provided, comprising a memory and a processor, the memory storing a computing program, when the computing program is executed by the processor, the method for microstructure image segmentation based on semi-supervised learning provided in any of the preceding embodiments can be implemented. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is a flow chart of the method for microstructure image segmentation based on semi-supervised learning in an embodiment;

[0025] Figure 2a is an alloy microstructure image to be processed in an embodiment;

[0026] Figure 2b is a label of an alloy microstructure image in an embodiment;

[0027] Figure 2c Segmentation result of Net B in an embodiment;

[0028] Figure 2d Segmentation result after morphological processing in an embodiment.

[0029] The purpose of the above drawings is to make a detailed description of the present application so that those skilled in the art can understand the technical concept of the present application, and is not intended to limit the present application. DETAILED DESCRIPTION

[0030] The present application will be further described in detail below with specific embodiments in conjunction with the drawings.

[0031] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment. The appearances of the phrase that the phrase in various places in the specification are not necessarily referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of one another. Those skilled in the art will appreciate from the present description that various embodiments of the present application can comprise combinations of features from the described embodiments. The description herein of any particular feature is meant to apply to each and every embodiment of the present application where such feature is present. The description herein of "a plurality" means at least two.

[0032] Using computer vision recognition instead of manual segmentation of alloy microstructure can effectively improve the efficiency of material characterization. At present, some image segmentation networks based on machine learning such as FCN, CNN or Deeplab have achieved good application in microstructure image segmentation field, but these image segmentation networks rely on supervised learning training method and need sufficient labeled data to fully play their effect. However, for microstructure analysis, the process of obtaining labeled data (in microstructure image segmentation, the label is the segmentation result of a given image) is difficult itself, and it is costly and inefficient to rely on manual labeling of microstructure images to train image segmentation networks. A large number of labeled images for the purpose of training image segmentation networks themselves contradict the purpose of automatic microstructure image.

[0033] In order to solve the above problems, an embodiment of one aspect of the present application provides a microstructure image segmentation method based on semi-supervised learning, which can use a small amount of labeled data and a large amount of unlabeled data to train the image segmentation network and improve the efficiency of microstructure image segmentation and analysis. As shown in the method Figure 1 The method comprises the following steps:

[0034] First, proceed to step 1. Provide training samples consisting of multiple microscopic tissue images. These training samples include a labeled dataset with fewer images and an unlabeled dataset with more images. Due to the limited field of view of electron microscope images, some preprocessing is required. Small-area images are taken separately and then stitched together to create a larger image. However, large images have higher resolution. Considering computer processing power and computation time costs, the images need to be cropped to obtain segmented smaller images. The quadratic cubic interpolation algorithm is used to adjust the image size to obtain the desired result. Figure 2a The final sample image shown is a microscopic image of the superalloy obtained at 800℃. The training sample includes 155 labeled sample images with a resolution of 512×512 and 1000 unlabeled sample images of the same resolution. For example... Figure 2a The sample images shown have labels as follows: Figure 2b The image shown is a grayscale image with pixel values ​​between 0 and 255. The region to be identified is 255, and the background region is 0. The labels were manually assigned. The labeled dataset is D. L :{x i ,y i The unlabeled dataset is D. U :{x i}, where x i For image samples, y i For tags.

[0035] Next, proceed to step 2. Using DenseUNet as the basic structure, build NetA and NetB network models as image segmentation networks. Due to the numerous tiny gaps in the alloy fiber structure, traditional image segmentation networks such as FCN, ResNet, and DenseNet have limited reception of image information from networks of different depths. UNet, however, uses skip connections to acquire semantic features from the encoder structure. DenseUNet directly connects each layer in the multi-layer neural network to the preceding layers, which facilitates feature reuse, reduces unnecessary computation, and effectively mitigates the risk of gradient descent or vanishing as network depth increases. NetA and NetB have the same structure.

[0036] Specifically, in the preferred embodiment, the image segmentation network (Net A and Net B) comprises a multi-layer neural network, each layer of which comprises an encoder and a decoder, and the encoder has a convolution module. After the sample image is input into the image segmentation network, the sample image is converted into a digital matrix, and the feature image of the sample image is extracted by the convolution module. In the encoder, the feature image output by each convolution module is down-sampled by one maximum pooling operation, and in the decoder, the feature image information is recovered by an up-sampling operation. The network encoder and the decoder corresponding to the highest feature image are connected by using the same skip connection layer. A 1*1 convolution is used at the end of the decoder to restore the original input image size, while the output label predicted by the image segmentation network is output.

[0037] Subsequently, step 3 is performed. Random perturbation u is applied to the sample image in the training sample i , wherein the sample image X in the labeled data set i and the label Y i are the same. i The same random perturbation u is applied i . In different embodiments, the random perturbation u includes rotation or mirror flip. The training sample after the random perturbation is input into Net A to obtain the output result of Net A. Since the convolution network is not sensitive to the image after transformation (random perturbation), the same transformation is performed on the sample image before and after convolution, and the result obtained after convolution is different. The training sample is input into Net B, and the same random perturbation u is applied to obtain the output result of Net B. i .

[0038] Next, step 4 is performed. The loss function of the two types of labels is calculated, and the overall loss function is minimized.

[0039] The cross-entropy loss function of the labeled data is established as follows: where θ is the parameter of the neurons in each layer of Net A, f(x i , θ) is the image segmentation network, which is a nonlinear function, and B is the number of images input into Net A in each batch.

[0040] The consistency loss function of the output results of Net A and Net B is established as follows: where μ i ’ is the pixel-level output label in the output result of Net A, and μ i is the pixel-level output label in the output result of Net B. In the early stage of training, the training accuracy of the labeled data set is much higher than that of the unlabeled data set, and after multiple rounds of training, the training effect of the unlabeled data set improves, so it is necessary to set a weight coefficient that changes with the training round T to gradually increase the importance of the unlabeled data set in the overall loss function. The overall loss function is denoted as loss =

[0041] a(T)L+b(T)J, where a(T) and b(T) are weight functions, and the value of b(T) / a(T) increases monotonically with respect to T, so as to gradually increase the weight of J. In a preferred embodiment, a(T)=1 (i.e., a constant), Here, k is a given weight parameter used to determine the weights of the unlabeled dataset in the final image segmentation network; the value of k can be 1. As T increases, b(T) monotonically increases and gradually approaches k.

[0042] Next, proceed to step 5. Update the network parameters of Net A, and update the network parameters of Net B using a moving average.

[0043] After establishing the overall loss function, the network parameters θ in Net A can be updated by constraining the minimization of the overall loss, and then the network parameters in Net B can be updated using the exponential moving average θ' of θ. In a preferred embodiment, θ t '=αθ t-1 '+(1-α)θ t , where α is the balance coefficient, t is the index, θ is the parameter obtained by propagating the error direction in Net A, and θ' is the parameter of Net B.

[0044] Finally, proceed to step 6. In the early stages of network training, Net B has a large error in predicting the labels of unlabeled sample images, primarily relying on Net A for prediction. As the number of training iterations increases, the accuracy of Net B gradually improves. Once the outputs of Net A and Net B are below a given consistency threshold, they are considered similar. The Intersection over Union (IoU) (the ratio of the intersection of the predicted true parts and the true parts in the labels to their union) is used as a metric to measure the network's image segmentation accuracy. Since Net B's network parameters θ' are obtained by moving average of Net A's network parameters θ, it exhibits better stability for image segmentation. Therefore, Net B is used to perform microstructural image segmentation to obtain the output image.

[0045] For example Figure 2a The sample image shown is processed using an image segmentation network, and the result is expected to be similar to that shown. Figure 2b The results shown are the same as the manually segmented "labels". After multiple rounds of training, the "labels" predicted by Net B for the sample images, i.e., the segmentation results, are as follows. Figure 2c As shown. It can be seen that in Net B's prediction results, for Figure 2a The segmentation of the smaller region into 7 segments was accurate, and the results were obtained precisely. Figure 2c The segmentation result 7' is correct, but errors occurred in the prediction of the larger region 8.Figure 2c The error region 9 is formed, indicating that the image segmentation network has defects in the segmentation result of the larger gap. In a preferred embodiment, the Net B needs to be further morphologically processed after performing the microstructure image segmentation on the image, and the gap is filled. In a further preferred embodiment, the following method is used: a probability image is formed for the output image of the Net B, a threshold value of 0.5 is set to generate a binary segmentation result, the error region 2c is filled, and a corrected region 8' as shown in Figure 2d is obtained. The segmentation result after morphological processing Figure 2d has good consistency compared with the manual label Figure 2b , indicating that the above method can effectively realize the efficient automatic segmentation of the microstructure image.

[0046] An embodiment of another aspect of the present application also provides a computing device. The computing device includes a memory and a processor, and the memory stores a computing program which, when executed by the processor, can implement the semi-supervised learning-based microstructure image segmentation method provided in the above embodiments. The computing device can be a general-purpose computer or a special-purpose computing device specially built for image segmentation.

[0047] The above embodiments are intended to further illustrate the present application in conjunction with the accompanying drawings, so that those skilled in the art can understand the technical concept of the present application. Within the scope of the present application, optimization or equivalent replacement of the methods involved, and combination of the embodiments in different embodiments without principle conflict, all fall within the protection scope of the present application.

Claims

1. A method for microstructure image segmentation based on semi-supervised learning, characterized in that, The method comprises the following steps: providing a training sample comprising a plurality of microscopic tissue images, the training sample comprising a labeled data set and an unlabeled data set, the microscopic tissue images in the labeled data set being labeled images, the microscopic tissue images in the unlabeled data set being unlabeled images, the number of the unlabeled images not being less than that of the labeled images, the label being a segmentation result of a given sample image; building an image segmentation network based on DenseUNet, the image segmentation network comprising Net A and Net B, the Net A and the Net B having the same structure; providing a random perturbation u to the training sample and labels in the labeled data set i and inputting the Net A to obtain a Net A output result; applying the random perturbation u to the output of the Net B after inputting the training sample into the Net B i obtaining a Net B output result, the Net A output result and the Net B output result each comprising an output label establishing a cross-entropy loss function of the labeled data set; wherein x i is an input sample image, y i is a label, θ is a parameter of each neuron in the Net A, f(x i , θ) is the image segmentation network, and B is the number of input images; establishing a consistency loss function of the output results of the Net A and the Net B; wherein μ i is the output label in the output result of the Net A, μ i is the output label in the output result of the Net B; training the Net A and the Net B for multiple rounds, setting a total loss function loss=a(T)L+b(T)J, wherein T is a training round number, a(T) and b(T) are weight functions, and the value of b(T) / a(T) monotonically increases with respect to T; updating parameters θ in the Net A by constraining the total loss function to take a minimum value, and updating parameters in the Net B by using an exponential moving average θ' of the parameters θ; stopping training when the difference between the output results of the Net A and the Net B is less than a given consistency threshold, and performing microscopic tissue image segmentation by using the Net B to obtain an output image.

2. The microstructure image segmentation method based on semi-supervised learning according to claim 1, characterized in that, The image segmentation network comprises a plurality of networks, each layer of the network comprising an encoder and a decoder, the encoder comprising a convolution module, after the sample image is input into the image segmentation network, the sample image is converted into a digital matrix and the feature image of the sample image is extracted through the convolution module; each convolution module in the encoder performs down-sampling on the feature image through one maximum pooling; the decoder restores the feature image through up-sampling; the encoder and the decoder are connected through a corresponding highest feature image sampling same skip connection layer; a 1*1 convolution is used at the end of the decoder to restore the original size of the sample image, and the corresponding output label is output.

3. The microstructure image segmentation method based on semi-supervised learning according to claim 1 or 2, characterized in that, The random perturbation u i includes rotation or mirror flipping.

4. The microstructure image segmentation method based on semi-supervised learning according to claim 1 or 2, characterized in that, a(T) = 1, where k is a weight parameter.

5. The microstructure image segmentation method based on semi-supervised learning according to claim 1 or 2, characterized in that, θ t ’ = a θ t-1 ’ + (1 - a) θ t where a is a balancing factor and t is the sequence number.

6. The microstructure image segmentation method based on semi-supervised learning according to claim 1 or 2, characterized in that, After performing microscopic tissue image segmentation by using the Net B, a morphological processing step is further included, which fills the gaps in the output image of the Net B.

7. The method of claim 6, wherein the semi-supervised learning-based microstructure image segmentation method is characterized by, The morphological processing step comprises forming a probability image from the output image, setting a threshold value to 0.5 to generate a binary segmentation result, and filling the gaps.

8. A computing device comprising a memory and a processor, wherein: The memory stores a computing program, which can implement the microscopic tissue image segmentation method based on semi-supervised learning according to any one of claims 1 to 7 when executed by the processor.

Images