Image segmentation method of nanometer material based on HRU2-Netmodel

Abstract

The invention provides an image segmentation method of a nanometer material based on an HRU2-Netmodel, and relates to data processing in the field of materials. The method comprises the following steps: acquiring nano-particle image data, marking the nano-particle image data, and separating nano-particles from a background area; dividing the labeled nano-particle image data into a training set and a test set according to a preset proportion; adding a multi-level semantic information fusion module into an uppermost layer encoder and decoder of a U-shaped structure nested in the U2-Netmodel, and constructing an HRU2-Netmodel; inputting the training set into an HRU2-Netmodel to carry out model training; inputting the test set into the trained HRU2-Net model, and evaluating the segmentation performance of the HRU2-Net model; and inputting an image of a nano material to be segmented into the trained HRU2-Netmodel to realize image segmentation. According to the method, the titanium dioxide nanoparticle image obtained from the scanning electron microscope is segmented, and the experimental result shows that the HRU2-Netmodel provided by the invention can process the nanoparticle image in a complex environment and can accurately segment the nanoparticles.

Description

Technical Field

[0001] The present invention relates to a material field data processing technology field, specifically to a method based on Image segmentation method for nanomaterials based on models. Background Art

[0002] Nanoparticles have extremely small sizes and unique electrical, optical, magnetic and thermal properties. They have great application potential in agricultural production, biomedicine, new energy and aerospace, and provide a material science basis for many advanced technologies. Due to the complex nature and extremely small size of nanoparticles, their research requires the use of high-precision equipment such as optical microscopes, atomic force microscopes and electron microscopes. Acquiring nanoparticle data and extracting effective information from large equipment is labor-intensive and inefficient, and it is difficult to extract the complex information present in nanoparticles using manual methods, resulting in slow nanoparticle research and application.

[0003] In recent years, deep learning technology has developed rapidly, providing strong support for research in the field of materials science. At the same time, deep learning technology has been widely used in the fields of new material prediction, material structure and scale determination. Some reports have proposed a general framework based on deep learning and used lightweight deep learning networks to achieve segmentation, shape extraction and statistical analysis of nanoparticles. Other reports have used deep learning and multi-target tracking technology to achieve the process of automatic tracking, scanning and observation of nanoparticles, or by designing a transformer-enhanced segmentation network, using a hybrid CNN transformer architecture based on transformers and mask R-CNN to accurately segment and measure nanoparticles. Or by solving the problem of nanoparticle data volume and labeling through synthetic images, the most advanced deep neural networks can be trained.

[0004] Most of the existing nanoparticle segmentation methods use data statistics, machine learning and a small amount of deep learning methods, but the shapes of nanoparticles vary greatly, and most of them are in an aggregated state. Using more classic network models such as U-Net, SegNet, ResNet, etc. to complete the task of nanoparticle segmentation lacks universality and advancement, which is not conducive to the segmentation of nanoparticles and particle segmentation is unfavorable. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides a The image segmentation method of the nanomaterial of the model includes:

[0006] Acquiring nanoparticle image data and annotating the nanoparticle image data to distinguish the nanoparticle from the background;

[0007] The annotated nanoparticle image data is divided into a training set and a test set according to a preset ratio;

[0008] exist A multi-level semantic information fusion module is added to the top encoder and decoder of the nested U-shaped structure of the model to construct Model;

[0009] Input the training set into the Model training is performed in the model;

[0010] The test set is input into the trained In the model, the evaluation The segmentation performance of the model;

[0011] The image of the nanomaterial to be segmented is input into the trained In the model, image segmentation is implemented.

[0012] As a further illustration of the present invention, the nanoparticle image data is generated using a scanning electron microscope and a GAN network.

[0013] As a further illustration of the present invention, during labeling, the nanoparticle image is labeled at the pixel level, wherein the nanoparticles are labeled as white and the background is labeled as black.

[0014] As a further illustration of the present invention, before dividing the labeled nanoparticle image data into a training set and a test set according to a preset ratio, the method further includes:

[0015] The annotated nanoparticle image data was cropped to a preset pixel size, and the image data was expanded using data expansion and enhancement tools.

[0016] As a further illustration of the present invention, the overall number of layers of the multi-level semantic information fusion module and The model remains consistent, and the structure adopts the form of fusion of upper and lower level semantic information.

[0017] As a further illustration of the present invention, during the model training process, the number of training iterations was set to 100 times, the cross entropy loss function was used to represent the degree of difference between the predicted data and the actual data, the initial learning rate was set to 0.01, and the learning rate decay method was used for gradual decay, the optimization method used SGD, the batch size was 16, and random initialization was used for initialization.

[0018] As a further illustration of the present invention, the formula of the cross entropy loss function is expressed as:

[0019]

[0020] In the formula, P ic is the predicted probability that the observed sample i belongs to category c, Y icis a symbolic function, if the true category of sample i is equal to c, it takes 1, otherwise it takes 0; M is the number of label types, and N is the total number of pixels.

[0021] As a further illustration of the present invention, the test set is used to evaluate the When evaluating the segmentation performance of the model, the evaluation indicators used include average intersection over union (MIoU), accuracy, Kappa coefficient, Dice coefficient, intersection over union (IoU), precision, and recall.

[0022] Compared with the prior art, the present invention has the following beneficial technical effects:

[0023] The present invention proposes a new model—— model, which is in Based on the model, the encoder part of the nested U module is designed with multiple branches and multi-scale fusion, focusing on improving the segmentation precision and the recognition ability of details. The experimental results show that the image of titanium dioxide nanoparticles obtained from a scanning electron microscope can be segmented. The model is able to process images of nanoparticles in complex environments and can accurately segment nanoparticles.

[0024] Other features and advantages of the technical solution will be described in the subsequent description, and partly become apparent from the description, or understood by implementing the technical solution. The purpose and other advantages of the technical solution can be achieved and obtained through the structures specifically pointed out in the written description and the drawings.

[0025] The technical solution of the present technical solution is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to provide a further understanding of the technical solution and constitute a part of the specification. Together with the embodiments of the technical solution, they are used to explain the technical solution and do not constitute a limitation of the technical solution. In the accompanying drawings:

[0027] Figure 1 The method constructed in the embodiment of the present invention is The overall structure of the model.

[0028] Figure 2 In the embodiment of the present invention U-shaped multi-level semantic information fusion structure diagram in the model, where (a) is The En_1 and De_1 structures of the model, (b) is the structure diagram of the U-shaped multi-level semantic information fusion module.

[0029] Figure 31 is an image of titanium dioxide nanoparticles in an embodiment of the present invention, wherein (a) is a data image and (b) is a labeled image.

[0030] Figure 4 It is a loss function curve diagram during the model training process in an embodiment of the present invention.

[0031] Figure 5 This is the evaluation result of the image segmentation method in the embodiment of the present invention on the nanoparticle dataset. DETAILED DESCRIPTION

[0032] The preferred embodiments of the present technical solution are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present technical solution, and are not used to limit the present technical solution.

[0033] The embodiment of the present invention provides a The image segmentation method of the nanomaterial of the model comprises the following steps:

[0034] S100, obtaining nanoparticle image data and annotating the data to distinguish the nanoparticles from the background.

[0035] The nanoparticle dataset in the embodiment of the present invention is obtained from the open source project of Bastian Rühle et al., and its image data is generated and annotated using the GAN network, which can be used for the training of convolutional neural networks. Specifically, a total of 40 titanium dioxide nanoparticle images with a size of 1024x762 pixels are obtained using a scanning electron microscope and a GAN network and annotated at the pixel level, where titanium dioxide nanoparticles are marked as white and the background is marked as black, such as Figure 3 As shown, the image data is cropped into 512x512 pixel size for a total of 200 images as needed, and the image data is expanded using data expansion and enhancement tools. Finally, the processed image data totals 2500 images.

[0036] S200, dividing the annotated nanoparticle image data into a training set and a test set according to a preset ratio.

[0037] Specifically, the 2500 image data obtained by the above enhancement processing are randomly divided into a training data set and a test data set in a ratio of 8:2, including 2000 training data sets and 500 test data sets.

[0038] S300, A multi-level semantic information fusion module is added to the top encoder and decoder of the nested U-shaped structure of the model to construct Model.

[0039] The embodiment of the present invention is constructed The model is Based on the model, the multi-level semantic information fusion design and the idea of ​​improving the feature map resolution are combined, and a multi-level semantic information fusion module is added to the top-level encoder and decoder of its nested U-shaped structure, which effectively improves the feature map resolution segmentation details and segmentation speed.

[0040] The overall structure of the model is as follows Figure 1 As shown in the figure, it mainly includes the following three parts: (1) a six-level encoder located on the left and bottom of the U-shaped junction; (2) a five-level decoder located on the right side of the U-shaped junction; (3) a multi-layer joint output module.

[0041] The six-level encoding path consists of En_1-En_6. For encoder levels En_5 and En_6, the input feature maps of these two levels have lower resolutions. In this case, downsampling them will result in the loss of useful context information. Therefore, at encoder levels En_5 and En_6, dilated convolutions are used instead of pooling and upsampling operations in the residual U-shaped module. Table 1 lists the Detailed configuration at each encoder level of the network.

[0042] Table 1

[0043] Tier Name En_1 En_2 En_3 En_4 En_5 En_6 Modules RSU-7 RSU-6 RSU-5 RSU-4 RSU-4F RSU-4F <![CDATA[C in ]]> 3 64 64 64 64 64 M 16 16 16 16 16 16 <![CDATA[C out ]]> 64 64 64 64 64 64

[0044] The five-level decoding path consists of De_5-De_1, and the decoder level has a similar structure as the encoder level symmetric with respect to En_1-En_5. In addition, similar to the configuration in encoder levels En_5 and En_6, the residual U-shaped module using dilated convolution is also applied to the decoder level De_5. Figure 1 It can be seen that The input of each decoder layer in the network is the concatenation of the upsampled feature maps from the previous layer and from the symmetric encoding layer. Table 2 lists the Detailed configuration at each decoder level of the network.

[0045] Table 2

[0046] Tier Name De_5 De_4 De_3 De_2 De_1 Modules RSU-4F RSU-4 RSU-5 RSU-6 RSU-7 <![CDATA[C in ]]> 128 128 128 128 128 M 16 16 16 16 16 <![CDATA[C out ]]> 64 64 64 64 64

[0047] Multi-layer joint output module is used to generate the final segmentation result map. The network first uses 3×3 convolution and Sigmoid function to generate 6 corresponding side output segmentation result maps in 6 layers respectively, then upsamples these side output segmentation result maps to the size of the input image, and then performs cascade operation to achieve channel dimension combination, and finally generates the final segmentation result map after 1×1 convolution layer and Sigmoid function operation.

[0048] Above In the model, En_1 and De_1 are U-shaped multi-level semantic information fusion modules designed in this paper, such as Figure 2 As shown in the figure, the multi-level semantic information fusion module can connect the semantic information between feature maps of different resolutions, which enables the semantic information to flow better between different resolutions, enhances the model's image segmentation capability, reduces the model's computational complexity, and improves the resolution of the feature map. At the same time, high-resolution structural design is only performed on the encoding part, which makes the model's size and operating efficiency higher.

[0049] Specifically, Figure 2 (a) The En_1, De_1 structure of the model, Figure 2 (b) A multi-level semantic information fusion module designed by the present invention, which is applied in The En_1 and De_1 parts of the model. The multi-level semantic information fusion module has the same overall number of layers as The model remains consistent, but the structure adopts the form of upper and lower semantic information fusion. This improvement can be applied to En_2, En_3, De_2, and De_3 as needed. While ensuring that the resolution of the feature map remains unchanged, in the Stage 1 stage of the multi-level semantic information fusion module, the initial input feature map is passed through four convolution blocks, which is beneficial for the next step of feature extraction. The second convolution block is downsampled as the input of the Stage 2 stage, which helps to preserve more semantic information. The fourth convolution block is fused with the feature map of the decoder stage, which helps to remove useless information in the image, refine the segmentation results, and ensure the high resolution of the output feature map. The same method is used in stages 1-5. In the Stage 2 stage, four convolution blocks are also used. The first and second convolution blocks of this stage realize the semantic information exchange between Stage 1 and Stage 2 through multi-level semantic information connection. Similarly, Stage 2, Stage 3, Stage 4 and Stage 5 also exchange semantic information with each other in this way, which is conducive to the transmission of the above information, optimizes the segmentation results, reduces the loss of image information, and can more accurately distinguish the boundaries of nanoparticles, reducing the impact of category imbalance and the small gap between image background and foreground.

[0050] S400, input the training set Model training is performed in the model.

[0051] During the training process, the size of the nanomaterial image was adjusted to 512×512 pixels. When pre-training the network model, we found that the loss function of the network model converged in about 50 iterations, so the number of training iterations was set to 100. Figure 4 As shown. We use the cross entropy loss function to represent the gap between the predicted data and the actual data. The initial learning rate is set to 0.01 and the learning rate decay method is used for gradual decay. The optimization method uses SGD, the batch size is 16, and the random initialization method is used for initialization. The whole experiment takes about 400 hours. The experimental equipment uses a 64-bit Windows 11 operating system and builds training and test networks based on PaddleSeg. The detailed configuration is as follows: Anaconda3, PaddlePaddle2.4.0, Paddleseg2.7.0, OpenCV4.6.0, Cuda11.2 and Cudnn8.2.

[0052] The data set used in the experiment consists of two categories of nanoparticles, metal composites, and three categories of two-dimensional materials. According to the size and category of the data set, this paper uses the cross entropy loss function to represent the gap between the predicted data and the actual data. The formula is:

[0053]

[0054] In the formula, P ic is the predicted probability that the observed sample i belongs to category c, Y ic is a symbolic function, if the true category of sample i is equal to c, it takes 1, otherwise it takes 0; M is the number of label types, and N is the total number of pixels.

[0055] S500, input the test set into the trained In the model, evaluation Segmentation performance of the model.

[0056] Nanomaterials are small in size, diverse in variety and complex in structure, which brings many inconveniences to research and preparation. At the same time, the physical and chemical properties of various nanomaterials vary greatly, and some nanomaterials even have toxic side effects. Therefore, it is crucial to select a semantic segmentation model with reliable performance to complete the image segmentation task of nanomaterials. However, a network model used in practice should not only consider its accuracy, but also its robustness, scalability and dependence on resources. In order to comprehensively evaluate the network model provided by the embodiment of the present invention, this paper adopts 7 evaluation indicators: (1) Mean intersection over union (MIoU), (2) Accuracy, (3) Kappa coefficient (Kappa), (4) Dice coefficient (Dice), (5) Intersection over union (IoU), (6) Precision, (7) Recall. The specific analysis is as follows:

[0057] (1) MIoU is one of the metrics for semantic segmentation, which calculates the average ratio of the intersection and union of all categories. Its public representation is as follows:

[0058]

[0059] According to the confusion matrix, Ncls represents the total number of categories, Nxx represents true positives, Nxy represents false positives, Nyx represents false negatives, and Nyy represents true negatives.

[0060] (2) Accuracy is an indicator used to evaluate classification models, that is, the proportion of the total number of correct predictions by the model. The formula is as follows:

[0061]

[0062] According to the confusion matrix, TP is true positive, TN is true negative, FP is false positive, and FN is false negative.

[0063] (3) Kappa coefficient is an indicator of consistency test. Consistency refers to whether the model prediction results are consistent with the actual classification results. It can be used to measure the classification effect. Its formula is as follows:

[0064]

[0065] Among them, Po is the sum of the number of correctly classified samples in each category divided by the total number of samples, that is, the overall classification accuracy. Pe is the sum of the products of the actual and predicted numbers corresponding to all categories, divided by the square of the total number of samples.

[0066] (4) Dice is a set similarity measurement function used to calculate the similarity between two samples and is often used to evaluate the quality of segmentation algorithms. Its public representation is as follows:

[0067]

[0068] Among them, |AB| is the intersection of A and B, |A| and |B| respectively represent the number of elements in the sum, and the coefficient of the numerator is 2 because the denominator repeats the calculation of the common elements between A and B.

[0069] (5) IoU represents the result of dividing the overlapping part of two regions by the combined part of the two regions. It is one of the metrics for semantic segmentation. Its formula is as follows:

[0070]

[0071] It is defined as the intersection area between the predicted segmentation image A and the truth image B, divided by the union area between the two images, ranging between 0 and 1.

[0072] (6) Precision represents the probability that the samples predicted as positive are actually positive. Its formula is as follows:

[0073]

[0074] According to the confusion matrix, TP is a true positive and FP is a false positive.

[0075] (7) Recall rate indicates the probability of being predicted as a positive sample among samples that are actually positive. Its formula is as follows:

[0076]

[0077] According to the confusion matrix, TP is a true positive and FN is a false negative.

[0078] Table 3 lists the evaluation data of the above-mentioned semantic segmentation methods based on deep learning on the nanoparticle dataset. The results show that the MIoU of all models is above 83%, the accuracy is around 97%, the Kappa coefficient is above 83%, and the Dice coefficient is above 91%, indicating that the semantic segmentation model based on deep learning has better segmentation effect and higher segmentation accuracy. The MIoU of classic segmentation models (such as U-Net and PSPNet) is about 86%. This is because they use deeper convolutional neural networks, which improves the accuracy to a certain extent, but the model is large in size, which is not conducive to deployment, as well as the disadvantages in efficiency. DDRNet, BiseNetV2, PPLiteSeg, and RTFormer are lightweight network models that are small for deployment and conversion in segmentation environments, with MIoU between 83% and 87%. HRU-Net uses HRNe as the backbone network, which has a large model size and runs slowly, with an MIoU of 87.21%. The model,has the characteristics of small size and fast segmentation speed. Its MIoU is 87.37%, the highest among all models,,with the best overall performance and is very competitive.

[0079] Table 3

[0080]

[0081] like Figure 5 As shown in the figure, the actual segmentation effect image of each model is shown, where the yellow box indicates the comparison in the segmentation details, and the pink box indicates the existence of unsegmented tiny areas. As can be seen from the figure, When performing nanoparticle segmentation, the model can accurately segment tiny nanoparticles in the image, and the model does not have any segmentation error during segmentation. The model can finely segment nanoparticles of different morphologies. 2 -Net model outperforms DDRNet, BiseNetV2, PP-LiteSeg, and RTFormer models. In terms of boundary clarity, DDRNet and PP-LiteSeg models have poor segmentation effects, while BiseNetV2 and RTFormer models lack segmentation details. 2 -Net model has some unsegmented areas, while DDRNet and PP-LiteSeg models have a large number of unsegmented areas, and BiseNetV2 and RTFormer models have a large part of unsegmented areas. 2 -Net model has smaller segmentation failures in the image boundary area, while DDRNet, BiseNetV2 and PP-LiteSeg models have fewer segmentation failures. These results show that the proposed The model has better adaptability and segmentation ability for nanoparticles, which is of greater significance for expanding the morphological analysis research of nanoparticles and achieving a more efficient working method.

[0082] S600: Input the image of the nanomaterial to be segmented into the trained In the model, image segmentation is implemented.

[0083] In summary, the present invention proposes a new model: Model. The model is Based on the model, the encoder part of the nested U module is designed with multiple branches and multi-scale fusion, focusing on improving the segmentation precision and the ability to recognize details. By segmenting the titanium dioxide nanoparticle images obtained from a scanning electron microscope, the experimental results show that the #imgpt52# model is able to process nanoparticle images in complex environments and accurately segment nanoparticles. The average intersection rate (MIoU) of the model is 87.37% and the accuracy is 97.31%, providing better segmentation results and higher accuracy.

[0084] Obviously, those skilled in the art can make various changes and modifications to the technical solution without departing from the spirit and scope of the technical solution. Thus, if these modifications and variations of the technical solution fall within the scope of the claims of the technical solution and their equivalents, the technical solution is also intended to include these modifications and variations.

Claims

1. A HRU-based 2 -Net † Image segmentation method of nanomaterials based on models, It is characterized in that include: Acquiring nanoparticle image data and annotating the nanoparticle image data to distinguish the nanoparticle from the background; The annotated nanoparticle image data is divided into a training set and a test set according to a preset ratio; in U 2 -Net † A multi-level semantic information fusion module is added to the top encoder and decoder of the nested U-shaped structure of the model to construct HRU 2 -Net † Model; Input the training set into the HRU 2 -Net † Model training is performed in the model; Input the test set into the trained HRU 2 -Net † In the model, the HRU is evaluated 2 -Net † The segmentation performance of the model; Input the image of the nanomaterial to be segmented into the trained HRU 2 -Net † In the model, image segmentation is implemented.

2. The HRU-based method according to claim 1 2 -Net † Image segmentation method of nanomaterials based on models, It is characterized in that The nanoparticle image data is generated using a scanning electron microscope and a GAN network.

3. The HRU-based method according to claim 1 2 -Net † Image segmentation method of nanomaterials based on models, It is characterized in that During labeling, the nanoparticle image is labeled at the pixel level, wherein the nanoparticles are labeled as white and the background is labeled as black.

4. The HRU-based method according to claim 1 2 -Net † Image segmentation method of nanomaterials based on models, It is characterized in that Before dividing the labeled nanoparticle image data into a training set and a test set according to a preset ratio, the method further includes: The annotated nanoparticle image data was cropped to a preset pixel size, and the image data was expanded using data expansion and enhancement tools.

5. The HRU-based method according to claim 1 2 -Net † Image segmentation method of nanomaterials based on models, It is characterized in that The overall number of layers and U of the multi-level semantic information fusion module 2 -Net † The model remains consistent, and the structure adopts the form of fusion of upper and lower level semantic information.

6. The HRU-based method according to claim 1 2 -Net † Image segmentation method of nanomaterials based on models, It is characterized in that During the model training process, the number of training iterations was set to 100 times, and the cross entropy loss function was used to represent the degree of difference between the predicted data and the actual data. The initial learning rate was set to 0.01, and the learning rate decay method was used for gradual decay. The optimization method used was SGD, the batch size was 16, and random initialization was used for initialization.

7. The HRU-based method according to claim 6. 2 -Net † Image segmentation method of nanomaterials based on models, It is characterized in that The formula of the cross entropy loss function is expressed as: ； In the formula, P ic is the predicted probability that the observed sample i belongs to category c, Y ic is a symbolic function, if the true category of sample i is equal to c, it takes 1, otherwise it takes 0; M is the number of label types, and N is the total number of pixels.

8. The HRU-based method according to claim 1 2 -Net † Image segmentation method of nanomaterials based on models, It is characterized in that The HRU is evaluated using the test set. 2 -Net † When evaluating the segmentation performance of the model, the evaluation indicators used include average intersection over union (MIoU), accuracy, Kappa coefficient, Dice coefficient, intersection over union (IoU), precision, and recall.

Images