A method and device for image segmentation processing using deep transfer learning

Abstract

The present application discloses a method and device for image segmentation processing using deep transfer learning, the method comprising: obtaining a metal microstructure image at a first temperature; using each image of a first training set to train a segmentation model, wherein after each round of training using each image, the training is verified using a validation set to obtain evaluation indicators of the segmentation model after this round of training; selecting the optimal model according to the evaluation indicators of each round, and obtaining the model parameters corresponding to the optimal model; using the model parameters to construct a new segmentation model; using the metal microstructure image at a second temperature as a second training set to adjust the new segmentation model; using the adjusted new segmentation model to predict and obtain a prediction result. The present application solves the problem of poor model prediction effect caused by the small number of training set images in the prior art, thereby improving the prediction effect of the model.

Description

Technical Field

[0001] The present application relates to the field of image processing, and specifically, to a method and device for image segmentation processing using deep transfer learning. Background Art

[0002] Microstructure is the key to the relationship between material composition, process, organization and performance. It can reveal the synergy between material composition and process, and then adjust the performance of the material. Therefore, how to use deep learning technology to improve the accuracy of microstructure recognition and effectively extract microstructure image features is of great significance for further understanding the relationship between microstructure and material performance.

[0003] In the field of metallic structural materials, the microstructural image is characterized by the presence of different interlaced precipitates within a small area. Figure 2 This is a comparison chart of the model prediction results and the true labels obtained by training under the small sample conditions of the existing technology, such as Figure 2 As shown in the figure, there are many holes in the image. When there are few images in the training set, the model prediction effect is very poor. This also brings difficulties to the labeling work, which requires a lot of manpower and material resources, and the time cost is also very high. Summary of the invention

[0004] The embodiments of the present application provide a method and device for image segmentation processing using deep transfer learning, so as to at least solve the problem in the prior art that the small number of images in the training set leads to poor model prediction effect.

[0005] According to one aspect of the present application, a method for image segmentation processing using deep transfer learning is provided, including: obtaining metal microstructure images at a first temperature, wherein the metal microstructure images are all labeled, and the metal microstructure images are at least divided into a first training set and a validation set; using each image of the first training set to train a segmentation model, wherein after each round of training using each image, the validation set is used to verify the training to obtain evaluation indicators of the segmentation model after this round of training; selecting an optimal model according to the evaluation indicators of each round, and obtaining model parameters corresponding to the optimal model; using the model parameters to construct a new segmentation model; using the metal microstructure images at a second temperature as a second training set to adjust the new segmentation model; using the adjusted new segmentation model to perform predictions to obtain prediction results.

[0006] Furthermore, the number of images in the second training set is smaller than the number of images in the first training set.

[0007] Furthermore, obtaining the metal microstructure image at the first temperature includes: cropping the image taken by the electron microscope to obtain the metal microstructure image; and expanding the metal microstructure image using image enhancement.

[0008] Furthermore, the image enhancement includes at least one of the following: flipping, rotating, and adding Gaussian noise.

[0009] Furthermore, the method further includes: performing morphological processing on the image in the prediction result to obtain a predicted segmented image.

[0010] According to another aspect of the present application, there is also provided an image segmentation processing device using deep transfer learning, comprising: an acquisition module, used to acquire metal microstructure images at a first temperature, wherein the metal microstructure images are all labeled, and the metal microstructure images are at least divided into a first training set and a validation set; a segmentation module, used to train a segmentation model using each image of the first training set, wherein the validation set is used to verify the training after each round of training using each image, and an evaluation index of the segmentation model after this round of training is obtained; a selection module, used to select an optimal model according to the evaluation index of each round, and obtain model parameters corresponding to the optimal model; a construction module, used to construct a new segmentation model using the model parameters; an adjustment module, used to adjust the new segmentation model using the metal microstructure images at a second temperature as the second training set; and a processing module, used to perform prediction using the adjusted new segmentation model to obtain a prediction result.

[0011] Furthermore, the number of images in the second training set is smaller than the number of images in the first training set.

[0012] Furthermore, the acquisition module is used to: obtain the metal microstructure image by clipping the image taken by the electron microscope; and expand the metal microstructure image using image enhancement.

[0013] Furthermore, the image enhancement includes at least one of the following: flipping, rotating, and adding Gaussian noise.

[0014] Furthermore, the processing module is also used to: perform morphological processing on the image in the prediction result to obtain a predicted segmented image.

[0015] In an embodiment of the present application, a method is adopted to obtain metal microstructure images at a first temperature, wherein the metal microstructure images are all labeled, and the metal microstructure images are at least divided into a first training set and a validation set; each image of the first training set is used to train a segmentation model, wherein the validation set is used to validate the training after each round of training with each image, and the evaluation index of the segmentation model after this round of training is obtained; the optimal model is selected according to the evaluation index of each round, and the model parameters corresponding to the optimal model are obtained; a new segmentation model is constructed using the model parameters; the metal microstructure images at a second temperature are used as the second training set to adjust the new segmentation model; and the adjusted new segmentation model is used to predict and obtain the prediction result. This application solves the problem of poor model prediction effect caused by the small number of training set images in the prior art, thereby improving the prediction effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 It is a flowchart of using a deep transfer learning method proposed in an embodiment of the present application to achieve good microscopic tissue image segmentation accuracy with a small number of samples;

[0018] Figure 2 This is a comparison chart of the model prediction results and the true labels obtained by training with a small sample size based on the existing technology;

[0019] Figure 3 This is a comparison chart of prediction results when there are only three training atlases according to an embodiment of the present application;

[0020] Figure 4 This is a comparison chart of prediction effects when there are only 4 training atlases according to an embodiment of the present application;

[0021] Figure 5 This is a comparison chart of prediction results when there are only 5 training atlases according to the embodiment of the present application;

[0022] Figure 6 This is a comparison chart of prediction results when there are only 6 training atlases according to the embodiment of the present application;

[0023] Figure 7 is a flowchart of a method for image segmentation processing using deep transfer learning according to an embodiment of the present application;

[0024] Figure 8 It is a U-net network structure diagram for training image segmentation tasks according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0026] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0027] In this embodiment, a method for image segmentation processing using deep transfer learning is provided. Figure 7 is a flowchart of a method for image segmentation processing using deep transfer learning according to an embodiment of the present application. Figure 7 As shown, the process includes the following steps:

[0028] Step S702, acquiring metal microstructure images at a first temperature, wherein the metal microstructure images are all labeled, and the metal microstructure images are at least divided into a first training set and a validation set;

[0029] There are many ways to obtain an image, for example, cropping an image taken by an electron microscope to obtain the metal microstructure image; expanding the metal microstructure image using image enhancement. Optionally, the image enhancement may include at least one of the following: flipping, rotating, and adding Gaussian noise.

[0030] Step S704, using each image in the first training set to train a segmentation model, wherein after each round of training using each image, the training is verified using a validation set to obtain an evaluation index of the segmentation model after this round of training;

[0031] Step S706, selecting the optimal model according to the evaluation index of each round, and obtaining the model parameters corresponding to the optimal model;

[0032] Step S708, constructing a new segmentation model using the model parameters;

[0033] Step S710, using the metal microstructure image at a second temperature as a second training set to adjust the new segmentation model;

[0034] In this step, since adjustment is performed, the required number may be relatively small. At this time, the number of images in the second training set is smaller than the number of images in the first training set.

[0035] Step S712: Use the adjusted new segmentation model to perform prediction to obtain a prediction result.

[0036] Optionally, after this step, morphological processing may be performed on the image in the prediction result to obtain a predicted segmented image.

[0037] The present application solves the problem in the prior art that the small number of training set images leads to poor model prediction effect, thereby improving the prediction effect of the model.

[0038] The following is an explanation in conjunction with an optional embodiment. In this embodiment, a method for achieving good microscopic tissue image segmentation accuracy with a small number of samples using a deep transfer learning method is provided. In this method, a preprocessing operation is performed on each image; then an improved U-net network is constructed for the image segmentation task; the constructed network is used to train images under temperature conditions with a large amount of labeled data using cross entropy loss; the model obtained in the previous step is fine-tuned using a small amount of labeled data under another temperature condition; the optimal model is obtained through the validation set; finally, the optimal model is used to predict the test set, the predicted image is post-processed, and the segmentation accuracy is calculated.

[0039] Figure 1 is a flowchart of using a deep transfer learning method to achieve good microscopic tissue image segmentation accuracy with a small number of samples according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0040] Step 1: preprocess each image;

[0041] Since the resolution of images taken by electron microscopes is too high and the computer's ability to process images is limited, considering the GPU cost and time cost, this embodiment crops the images to obtain small images, and uses the quadratic cubic interpolation algorithm to adjust the image size to 512*512, and finally obtains 12 labeled images of the W-1000 data set and 67 labeled images of the W-900 data set. W-1000 and W-900 refer to the microstructure images of nickel-based high-temperature alloys at 1000℃ and 900℃, respectively. This embodiment can also perform standardization on the adjusted images, which helps to improve the convergence speed and accuracy of the model.

[0042] Step 2: Use image enhancement methods to expand the training data set;

[0043] Considering that the number of 67 images of W-900 is still small, in order to improve the generalization performance and segmentation accuracy of the model, this embodiment adopts the method of image enhancement to expand the training data set. Image enhancement is a method of data expansion. This embodiment mainly adopts the method of flipping, rotating and adding Gaussian noise to achieve data enhancement. For example, the following operations can be performed:

[0044] Step 2.1, randomly flip and rotate the image by 90° to increase the number of training samples;

[0045] In step 2.2, data augmentation is performed randomly on the transformed images, and Gaussian noise is added to increase the generalization and stability of the model.

[0046] Step 3, construct a segmentation network structure based on U-net;

[0047] Since the image size is 512*512, this embodiment selects the classic U-net segmentation network. The specific structure of the network is as follows: Figure 8 As shown in Figure 1, it is a typical fully convolutional neural network with a typical encoder and decoder structure. The encoder extracts deep features from the image, the pooling layer downsamples the feature map, and then the decoder upsamples the feature map to reconstruct the segmentation result. U-net also adds a skip connection between the encoder and decoder to help restore the micro details lost in the downsampling operation.

[0048] Each encoder layer contains two convolutional layers and a maximum pooling layer, and each decoder layer contains an upsampling layer and two convolutional layers. At the same time, the output of the second convolutional layer of each encoder is connected to the upsampled output of the decoder of the same layer through a skip connection. Finally, the output of the last decoder is mapped to 0-1 as the segmentation output result through the 'Sigmoid' activation function.

[0049] The purpose of this embodiment can also be achieved by using other convolutional segmentation networks. In an optional implementation, the U-net segmentation network is improved as follows to achieve better segmentation accuracy:

[0050] Step 3.1, because the pixels of the image are too large, a convolutional neural network is needed to extract deeper features, so a downsampling, convolution and upsampling operation is added on the basis of the U-net network; that is, a maximum pooling layer, two convolution layers and an upsampling layer structure are added to the smallest layer of the U-net network feature map. For an image of size 512*512, this can further extract the deep features of the image, which is conducive to the segmentation of the upsampling layer.

[0051] In step 3.2, the feature map obtained by each upsampling is convolved with 2*2 and then connected with the feature map of the same size obtained by the previous downsampling to fully utilize the context information. That is, in order to improve the utilization of features and segmentation accuracy, the features extracted by the downsampling layer are passed to the upsampling layer of the same level, and the feature map of the downsampling layer is convolved with 3*3 and then connected with the feature map of the same size of the upsampling layer. In this way, the network utilizes both low-level features and high-level features, and combines the above information to make the network's pixel positioning more accurate and the segmentation accuracy higher.

[0052] Step 4: Use the data from the large dataset to train the segmentation network.

[0053] In order to obtain a transferable model, this embodiment needs to first use the data set after the above data expansion to train the segmentation network. In the experiment, this embodiment selected the W-900 data set, which is a microstructure image of a nickel-based high-temperature alloy taken at 900°C. There are 67 labeled images with a resolution of 512*512. After data enhancement, the number of samples is sufficient to train a stable and generalizable model.

[0054] Step 5: Select the optimal model based on the evaluation index of the validation set;

[0055] In order to compare and obtain the optimal model, this embodiment will use the current model to verify on the validation set after each epoch training, obtain the size of the evaluation index based on the predicted value and the true value of the label, and compare it with the model of the previous epoch, and only save the model parameters with the best index.

[0056] Epoch: During training, all training data sets are trained once as one epoch.

[0057] Step 6: Use deep transfer learning to transfer the model and fine-tune it on a small dataset.

[0058] After obtaining the optimal network model trained on the large data set, this embodiment directly migrates the parameters of all layers of the network to form a new network, and reduces the learning rate of the training to half of that of the training on the large data set. This is the model-based deep transfer learning method. Finally, this embodiment fine-tunes the model with data from the small data set and selects the optimal model based on the evaluation index.

[0059] Step 7: Perform morphological processing on the segmentation result predicted by the fine-tuned optimal model to obtain a predicted segmentation image.

[0060] This embodiment uses the model obtained in step 6 to predict the images of the test set. The value range of each pixel in the image output by the model is 0-1, which is inconsistent with the actual binary label image. This embodiment performs morphological processing on the image predicted by the model and sets the threshold to 0.5 to generate a binary image. The image processing function can also be used to remove small connected domains that may be noise to obtain the final segmentation result image.

[0061] In step 4 and step 6, since the network trained on the large dataset already has a certain segmentation accuracy, a smaller learning rate needs to be selected when fine-tuning on the small dataset so that the mature model can gradually learn the characteristics of the small dataset data. Therefore, the learning rate for training on the small dataset is set to half of the learning rate for training on the large dataset;

[0062] In step 4, step 5 and step 6, due to the imbalance of positive and negative samples of the label, three different evaluation indicators can be used to represent the segmentation accuracy and select the optimal model. The three evaluation indicators are: Dice coefficient, HD95 distance and IoU (intersection over union).

[0063] The Dice coefficient is a measure of the overlap between the true value label and the predicted segmentation result. It is used to calculate the similarity between the true value image and the predicted image. The value range is between 0 and 1. The specific formula of the Dice coefficient is as follows:

[0064]

[0065] Where T represents the true value label and P represents the predicted segmentation result;

[0066] HD95 distance is also known as the maximum surface distance. It is a measure of the similarity between two sets of points. It is the definition of the distance between all points on a set P and the nearest points on another set T. Its specific formula is as follows:

[0067]

[0068] IoU (Intersection over Union) is a standard metric for image segmentation problems. It gives the similarity between the predicted segmented region and the ground truth region present in the image. The specific formula of IoU is as follows:

[0069]

[0070] Table 1 is a quantitative comparison table of the prediction accuracy of the deep transfer learning method and random initialization when the number of training set images is different. The trained model is tested with evaluation indicators to obtain the result scores. Table 1 is as follows:

[0071] W-1000 W-900 Dice-coef HD95(mm) IoU 3 0 49.48％ 1.570 36.46％ 67 80.68％ 1.458 69.05％ 4 0 60.79％ 1.520 45.56％ 67 80.29％ 1.435 68.64％ 5 0 72.96％ 1.720 61.02％ 67 80.68％ 1.273 69.24％ 6 0 73.88％ 1.537 60.84％ 67 81.37％ 1.306 70.18％ 7 0 73.28％ 1.577 60.28％ 67 81.37％ 1.525 70.26％ 8 0 79.69％ 1.556 68.61％ 67 81.98％ 1.424 70.78％ 12 0 82.60％ 1.412 72.20％ 67 82.31％ 1.481 71.42％

[0072] Figure 3 , Figure 4 , Figure 5 , Figure 6 The following are comparisons of the prediction effects of the models trained using the deep transfer learning method and the random initialization method on the test set images when there are only 3, 4, 5, and 6 images in the training set. From left to right are: SE original image, random initialization prediction result, deep transfer learning prediction result, and true label.

[0073] This embodiment can achieve high segmentation accuracy using a small number of labeled images, greatly saving labeling time and cost.

[0074] In the above embodiment, only a small number of labeled samples are used to achieve a good segmentation effect. First, all images are preprocessed and adjusted to the same size; then the image enhancement method is used to expand the data set; then this embodiment builds a segmentation network based on U-net for subsequent training; this embodiment first uses a data set under a certain temperature condition with a large amount of data to train the segmentation network; then the model with the best effect on the validation set is selected according to the evaluation index, and its model parameters are saved; then this embodiment transfers the model parameters to the new model, and fine-tunes it with a data set under another temperature condition with only a small amount of labeled data; finally, this embodiment uses the fine-tuned optimal model to predict the test set, and performs morphological processing on the predicted results to obtain the final segmented image.

[0075] In this embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the method in the above embodiment.

[0076] The above program can be run in the processor, or it can be stored in the memory (or computer-readable medium), which includes permanent and non-permanent, removable and non-removable media. Information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0077] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0078] Such a device or system is provided in this embodiment. The device is called an image segmentation processing device using deep transfer learning, and includes: an acquisition module, used to acquire metal microstructure images at a first temperature, wherein the metal microstructure images are all labeled, and the metal microstructure images are at least divided into a first training set and a verification set; a segmentation module, used to train a segmentation model using each image of the first training set, wherein the training is verified using the verification set after each round of training with each image, and the evaluation index of the segmentation model after this round of training is obtained; a selection module, used to select the optimal model according to the evaluation index of each round, and obtain the model parameters corresponding to the optimal model; a construction module, used to construct a new segmentation model using the model parameters; an adjustment module, used to use the metal microstructure image at a second temperature as the second training set to adjust the new segmentation model; a processing module, used to use the adjusted new segmentation model to predict and obtain a prediction result.

[0079] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0080] For example, the acquisition module is used to: obtain the metal microstructure image by clipping the image taken by the electron microscope; and expand the metal microstructure image using image enhancement. Optionally, the image enhancement includes at least one of the following: flipping, rotating, and adding Gaussian noise.

[0081] For another example, the processing module is further used to: perform morphological processing on the image in the prediction result to obtain a predicted segmented image.

[0082] The above embodiments solve the problem in the prior art that the small number of training set images leads to poor model prediction effect, thereby improving the prediction effect of the model.

[0083] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for image segmentation processing using deep transfer learning, characterized in that: include: Acquire metal microstructure images at a first temperature, wherein the metal microstructure images are all labeled, and the metal microstructure images are at least divided into a first training set and a validation set; Using each image of the first training set to train the segmentation model, wherein after each round of training using each image, the validation set is used to validate the training to obtain an evaluation index of the segmentation model after this round of training; Select the optimal model according to the evaluation indicators of each round, and obtain the model parameters corresponding to the optimal model; A new segmentation model is constructed using the model parameters. The construction of the segmentation model is based on the segmentation network structure of U-net, which is a typical full convolutional neural network with a typical encoder and decoder structure. The encoder extracts deep features from the image, the pooling layer downsamples the feature map, and then the decoder upsamples the feature map to reconstruct the segmentation result. U-net also adds a jump connection between the encoder and the decoder. Each encoder layer includes two convolutional layers and a maximum pooling layer, and each decoder layer includes an upsampling layer and two convolutional layers. At the same time, the output of the second convolutional layer of each encoder is connected to the upsampled output of the decoder of the same layer through a jump connection. Finally, the output of the last decoder is mapped to between 0 and 1 as the segmentation output result through the 'Sigmoid' activation function; Using the metal microstructure image at a second temperature as a second training set to adjust the new segmentation model; The adjusted new segmentation model is used to perform predictions to obtain prediction results.

2. The method according to claim 1, characterized in that: The number of images in the second training set is smaller than the number of images in the first training set.

3. The method according to claim 1, characterized in that Acquiring the metal microstructure image at the first temperature includes: The metal microstructure image is obtained by clipping the image taken by the electron microscope; The metal microstructure image is expanded using image enhancement.

4. The method according to claim 3, characterized in that The image enhancement includes at least one of the following: flipping, rotating, and adding Gaussian noise.

5. The method according to any one of claims 1 to 4, characterized in that Also includes: Morphological processing is performed on the image in the prediction result to obtain a predicted segmented image.

6. A device for image segmentation processing using deep transfer learning, characterized in that: include: An acquisition module, used for acquiring metal microstructure images at a first temperature, wherein the metal microstructure images are all labeled, and the metal microstructure images are at least divided into a first training set and a validation set; A segmentation module, used to train a segmentation model using each image of the first training set, wherein after each round of training using each image, the training is verified using a validation set to obtain an evaluation index of the segmentation model after this round of training; The selection module is used to select the optimal model according to the evaluation indicators of each round and obtain the model parameters corresponding to the optimal model; A construction module is used to construct a new segmentation model using the model parameters. The construction of the segmentation model is based on the segmentation network structure of U-net, which is a typical full convolutional neural network with a typical encoder and decoder structure. The encoder extracts deep features from the image, the pooling layer downsamples the feature map, and then the decoder upsamples the feature map to reconstruct the segmentation result. U-net also adds a jump connection between the encoder and the decoder. Each encoder layer contains two convolutional layers and a maximum pooling layer, and each decoder layer contains an upsampling layer and two convolutional layers. At the same time, the output of the second convolutional layer of each encoder is connected to the upsampled output of the decoder of the same layer through a jump connection. Finally, the output of the last decoder is mapped to between 0 and 1 as the segmentation output result through the 'Sigmoid' activation function; An adjustment module, used to adjust the new segmentation model using the metal microstructure image at a second temperature as a second training set; The processing module is used to use the adjusted new segmentation model to perform prediction to obtain a prediction result.

7. The device according to claim 6, characterized in that The number of images in the second training set is smaller than the number of images in the first training set.

8. The device according to claim 6, characterized in that The acquisition module is used for: The metal microstructure image is obtained by clipping the image taken by the electron microscope; The metal microstructure image is expanded using image enhancement.

9. The device according to claim 8, characterized in that The image enhancement includes at least one of the following: flipping, rotating, and adding Gaussian noise.

10. The device according to any one of claims 6 to 9, characterized in that The processing module is also used for: Morphological processing is performed on the image in the prediction result to obtain a predicted segmented image.

Images