A multi-scale martensitic microstructure aging and damage classification method
Through deep learning technology, the ResNet-FPN-MC model is constructed using the combination of deep residual network and feature pyramid network, and the automation problems of microstructure aging and damage detection of high-chromium martensite heat-resistant steel are solved, achieving high precision and Efficient multi-scale recognition reduces the dependence of manual judgment.
Patent Information
- Application Number
- CN202011053202.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2040-09-29
AI Technical Summary
The prior art has high dependence and large errors in detecting the aging and damage degree of microstructure of high-chromium martensite heat-resistant steel, and it is difficult to adapt to metallographic image data of different scales, and lacks effective automated analysis methods.
Using deep learning-based methods, the ResNet-FPN-MC model is built using deep residual networks and feature pyramid networks, and the microstructure of steel is automatically classified. Multi-scale data sets are constructed through preprocessing and data enhancement methods to improve the model's differentiation. Resolution image recognition capability.
It improves the recognition accuracy and efficiency of steel microstructure aging and damage assessment, can realize automatic grading on pictures with different resolutions, reduces the dependence of manual judgment, and improves the accuracy and scalability of recognition.
Smart Images

Figure CN112132086B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the field of aging and damage identification of high-chromium martensitic heat-resistant steel microstructures, and in particular to a method for automatic metallographic structure rating. Background Art
[0002] The microstructure characteristics of steel materials are one of the important factors that determine the performance of materials. Due to the influence of factors such as environment, temperature, and pressure, the microstructure of steel materials often ages and is damaged to varying degrees during use, which poses a great threat to production safety. Therefore, how to scientifically and efficiently detect the aging and damage of steel materials has become one of the problems that need to be solved in theory and practice. In recent years, in the field of thermal power generation, high-chromium martensitic heat-resistant steels represented by P91 steel and P92 steel have been widely used in key pressure-bearing components such as the main steam pipes and reheating hot section pipes of ultra-supercritical units, and the aging and damage of their material structures have received increasing attention. At present, the research on this issue mainly focuses on on-site filming and photography or observation and analysis under a metallographic microscope in the laboratory after sampling by cutting the pipe. However, such methods are often highly dependent on the professional technical level and practical experience of researchers and are highly subjective. At the same time, due to the high cost and low reusability of manual observation and judgment, the experimental results often have relatively large errors.
[0003] With the advent of the era of artificial intelligence, machine learning algorithms represented by deep learning have made great progress in the field of image analysis and recognition. More and more scholars have begun to pay attention to the application of deep learning in the field of material research, which provides the possibility of exploring scientific and efficient methods for automatic identification of material microstructures. Among them, some scholars have explored the application of machine learning methods in material microstructures. For example, Azimi et al. used deep learning methods to transform the problem of material defect detection through images into the problem of semantic segmentation of images, and used semantic segmentation algorithms to detect defects in metal materials.
[0004] At present, the exploration of materials research using deep learning methods is still in its infancy. There are few existing studies on the application of material organization state classification, and there is even less research on the microstructure aging characteristics and damage degree of metallographic images of different scales. In the actual production environment, due to the differences in the conditions of metallographic data acquisition equipment, the scales of metallographic image data often vary greatly. At present, the study of microstructure aging and damage characteristics of metallographic image data with a fixed ratio in the experimental environment is difficult to meet the effective analysis of such data. For this reason, the use of deep learning methods to study the automatic classification of high chromium martensite microstructure aging characteristics and damage degree in metallographic images of different scales has important academic value and practical urgency. Summary of the invention
[0005] The object of the present invention is to solve the problems existing in the prior art, and to propose an automatic grading method for the microstructure damage and aging of high chromium martensitic heat-resistant steel based on a deep residual network, so as to improve the recognition accuracy and efficiency of the aging and damage assessment of the steel microstructure.
[0006] The object of the present invention is achieved by the following technical solutions:
[0007] A method for grading the aging and damage of martensite microstructure based on deep learning, comprising the following steps:
[0008] Step 1: Determine the standard resolution, that is, the magnification factor a of the steel microstructure to be recognized, where 50 < a < 1000. Collect the microstructure pictures of steel materials of the same specification size obtained by a metallurgical microscope at this magnification factor, as well as the microstructure pictures of steel materials with a smaller magnification factor and different resolutions. After the expert's grading assessment of the aging damage, bind the assessment level with the pictures to construct a data set.
[0009] Step 2: Perform the same preprocessing on all the pictures collected in Step 1. The preprocessing method is as follows:
[0010] 1)剔除金相显微镜获取的微观组织图中含有的文字说明部分,得到只含有微观组织图本体的初始数据集,将采集到的标准分辨率数据作为初始训练数据集T0,将采集到的非标准多种分辨率部分的数据作为验证数据集V0; 1) Eliminate the text description part contained in the microstructure pictures obtained by the metallurgical microscope to obtain an initial data set containing only the microstructure picture body. Use the collected standard resolution data as the initial training data set T0, and use the collected data of non-standard multiple resolutions as the validation data set V0;
[0011] 2)将初始训练数据集T0中三通道的灰度图转化为单通道的灰度图,得到训练数据集T1; 2) Convert the three-channel grayscale pictures in the initial training data set T0 into single-channel grayscale pictures to obtain the training data set T1;
[0012] 3)使用双线性插值法,对T1中的图像数据进行不同倍率的缩小,得到训练数据集T2; 3) Use the bilinear interpolation method to reduce the image data in T1 by different magnification factors to obtain the training data set T2;
[0013] 4)对训练数据集T2中的每张图片,随机取大小为nxnx1图像,其中1<m<100,100<n<800,如原始图片小于目标分辨率,则以补全的方式将原始图片补全到nxn大小。得到新的训练数据集T3。 4) For each picture in the training data set T2, randomly select an image of size n x n x 1, where 1 < m < 100 and 100 < n < 800. If the original picture is smaller than the target resolution, complement the original picture to the size of n x n in a complementary manner. Obtain the new training data set T3.
[0014] Step 3: Use the deep learning framework PyTorch to build the ResNet-FPN-MC model, where ResNet represents a convolutional neural network architecture: residual neural network, FPN stands for Feature Pyramid Networks, and MC (Martensite classification) is the abbreviation of Martensite classification. The network input image size is nxnx1, and the output is a 5-dimensional vector, where each dimension represents the probability that the material belongs to that category.
[0015] Step 4: Use the training data set T3, adopt cross entropy as the loss function (without adding regularization terms), and apply the stochastic gradient descent method to train the model (the number of samples in each batch of the stochastic gradient descent algorithm is set to 24, the training rounds are pre-set to 30, and the model parameters of each iteration are retained);
[0016] The specific training steps are as follows:
[0017] 1) Use the pre-trained ResNet to initialize the parameters in the current model backbone network.
[0018] 2) Randomly shuffle the order of images in the training set.
[0019] 3) Each time a batch of images is input into the neural network (the number of images in a batch can be 16, 32, or 64 images), the output vector of the neural network is recorded, the cross entropy value is calculated through the network output vector and the category label of the data, and then back propagation is performed according to the cross entropy value to update the model parameters; the loss function is calculated for each image in the training data set T3, and after the back propagation is completed, it is recorded as a round of training (a round of training means that all images have been forward calculated by the network. Due to the large number of images, the computer cannot calculate them at one time, so it can only be divided into batches. Therefore, one iterative calculation is divided into multiple batches, and a certain number of images are input for calculation in each batch).
[0020] 4) Model fine-tuning (a technique in which a pre-trained model uses a smaller learning rate to update model parameters)
[0021] 5) Record the model parameters in each round of training, and determine whether the maximum number of training rounds has been reached. If the maximum number of training rounds has been reached, execute step 6; if not, return to step 3 to continue training.
[0022] 6) Take out the model with the highest accuracy on the validation set during the training process.
[0023] Step 5: Determine whether the resolution of the microstructure image of the high-chromium martensitic heat-resistant steel to be identified is within the range that can be determined. If it is within the range that can be determined, pre-process it according to 1) and 2) in step 2, and then use the model trained in step 4 for automatic classification.
[0024] At present, the martensitic aging and damage grading task is to have experts manually observe the sample images and judge their damage levels. Manual judgment is more dependent on the professional knowledge level and practical experience of the technicians, and due to the different professional levels of the technicians, it will produce more ambiguous results in the judgment of samples between different levels of aging and damage, making manual classification inefficient and with larger errors. The neural network grading method is often only trained for image data of a specific resolution, and cannot be conveniently extended to image data of different resolutions in practical applications. In view of the above-mentioned problem of martensitic aging and damage grading, the present invention combines the advantages of deep learning and proposes a method for automatic identification of the microstructure of high chromium martensitic heat-resistant steel based on residual neural network and feature pyramid network, which can not only improve the recognition accuracy, but also facilitate the expansion of images with different resolutions. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0026] Figure 1 The standard architecture of the 18-layer ResNet disclosed in the embodiment of this application;
[0027] Figure 2 Examples of training data of different scales disclosed in the embodiments of this application. DETAILED DESCRIPTION
[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0029] This application uses the metallographic structure images of P91 and P92 materials with different degrees of aging collected in the laboratory as training sets, and attempts to use data enhancement methods to construct a multi-scale data set. On this basis, the deep residual network method is used to classify the aging and damage degree of the material microstructure, and this method is used to enhance the model's cognition of images of different scales. The experimental results show that this method enhances the model's recognition of different scales, and the recognition accuracy of the multi-scale test set can reach up to 60%; it further promotes the study of the aging and damage characteristics of material microstructures by deep learning methods in complex real-world production environments. This provides the possibility of constructing a metallographic image dataset of steel materials for deep learning methods to solve the problem of microstructure feature analysis, and lays the data foundation for the automated analysis of material microstructures.
[0030] Deep Residual Networks
[0031] The deep residual network (ResNet) is a convolutional neural network architecture that can effectively improve the training speed of deep models and the performance of models by introducing a residual connection structure
[11] . Based on this, this paper will use the ResNet50 model as the basic architecture to carry out image classification research. In this paper, the ResNet50 model will be used as the baseline classification model and the backbone model of FPN.
[0032] Feature Pyramid Network
[0033] In the problem of target detection in the field of computer vision, there are often cases where the targets in the data are of different sizes. To solve this problem, Tsung-Yi Lin et al. proposed Feature Pyramid Networks (FPN) in 2017. This method extracts multi-scale feature information and fuses it to improve the accuracy of target detection, especially in the detection of small objects. FPN is an additional component of general feature extraction networks such as ResNet model or DenseNet model, which can be combined with classic networks to improve the effect of the original network. Its biggest role is that FPN connects the high-level features with low resolution and high semantic information with the low-level features with high resolution and low semantic information from top to bottom, so that the features at all scales have rich semantic information. Its architecture is as follows: Figure 1 shown.
[0034] Example 1
[0035] The present application is an automatic grading method for martensitic damage and aging based on a deep residual network, which uses a deep learning framework PyTorch convolutional neural network model and an image preprocessing library torchvision to preprocess images.
[0036] Taking 315 microscopic images of steel materials collected in the laboratory at 500× (i.e., magnified 10,000 times) as an example, deep learning is used for automatic recognition of the microscopic structure of steel materials.
[0037] This application discloses an automatic grading method for martensite damage and aging based on deep learning, which is specifically carried out according to the following steps:
[0038] Step 1: Determine the standard resolution, that is, the magnification factor a of the microscopic structure of the steel to be recognized, where 50 < a < 1000. Collect microscopic images of steel materials with the same specification size obtained by a metallurgical microscope at this magnification factor, as well as microscopic images of steel materials with a smaller magnification factor and different resolutions. The images are as Figure 2 shown. After the expert's grading and evaluation of aging damage, bind the evaluation level to the image to construct a data set.
[0039] Step 2: Perform the same preprocessing on all the images collected in Step 1. The preprocessing method is as follows:
[0040] 1) Eliminate the text description part contained in the microscopic structure diagram obtained by the metallurgical microscope to obtain an initial data set containing only the microscopic structure diagram itself. Take the collected standard resolution data as the initial training data set T0, and take the collected data with non-standard multiple resolutions as the validation data set V0;
[0041] 2) Convert the three-channel grayscale image in the initial training data set T0 into a single-channel grayscale image to obtain the training data set T1;
[0042] 3) Use the bilinear interpolation method to reduce the image data in T1 by different magnification factors to obtain the training data set T2;
[0043] 4) For each image in the training data set T2, randomly take an image of size n×n×1, where 1 < m < 100 and 100 < n < 800. If the original image is smaller than the target resolution, fill in the original image to the size of n×n in a complementary way. Obtain the new training data set T3.
[0044] Step 3: Use the deep learning framework PyTorch to construct a ResNet-FPN-MC model, where ResNet represents a convolutional neural network architecture: Residual Neural Network, FPN represents Feature Pyramid Networks, and MC is Martensite classification. The input image size of the network is n×n×1, and the output is a 5-dimensional vector, and each dimension represents the probability that the material belongs to this category.
[0045] Step 4: Use the training data set T3, adopt cross entropy as the loss function (without adding regularization terms), and apply the stochastic gradient descent method to train the model (the number of samples in each batch of the stochastic gradient descent algorithm is set to 24, the training rounds are pre-set to 30, and the model parameters of each iteration are retained);
[0046] The specific training steps are as follows:
[0047] 1) Use the pre-trained ResNet to initialize the parameters in the current model backbone network.
[0048] 2) Randomly shuffle the order of images in the training set.
[0049] 3) Each time a batch of images is input into the neural network (the number of images in a batch can be 16, 32 or 64 images), the output vector of the neural network is recorded, the cross entropy value is calculated through the network output vector and the category label of the data, and then back propagation is performed according to the cross entropy value to update the model parameters; the loss function has been calculated for each image in the training data set T3, and after the back propagation is completed, it is recorded as a round of training (a round of training means that all images have been forward calculated by the network. Due to the large number of images, the computer cannot calculate them at one time, so it can only be divided into batches. Therefore, one iterative calculation is divided into multiple batches, and a certain number of images are input for calculation in each batch).
[0050] 4) Model fine-tuning (a technique in which a pre-trained model uses a smaller learning rate to update model parameters)
[0051] 5) Record the model parameters in each round of training, and determine whether the maximum number of training rounds has been reached. If the maximum number of training rounds has been reached, execute step 6; if not, return to step 3 to continue training.
[0052] 6) Take out the model with the highest accuracy on the validation set during the training process.
[0053] The precision and recall of various types of steel microstructure diagrams in the validation data set under the Best_Model are statistically calculated. As can be seen from Table 1, the precision and recall of all types of steel microstructure diagrams reach 100%.
[0054] Table 1 Accuracy of martensite microstructure diagrams of different scales under Best_Model
[0055]
[0056] It can be seen that the automatic classification method of martensitic damage and aging based on deep residual network of the present invention can automatically classify martensitic microstructure diagrams, and achieve a certain accuracy rate at multiple expansion scales based on the original data set with a single resolution.
[0057] Step 5: Determine whether the resolution of the microstructure image of the high-chromium martensitic heat-resistant steel to be identified is within the range that can be determined. If it is within the range that can be determined, pre-process it according to 1) and 2) in step 2, and then use the model trained in step 4 for automatic classification.
[0058] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for aging and damage grading of multi-scale martensite microstructure, characterized in that: It includes the following steps: Step 1: Determine the standard resolution, that is, the magnification factor a of the microstructure of the steel material to be identified, where 50 < a < 1000. Collect the microstructure pictures of the steel material with the same specification size obtained by the metallographic microscope at this magnification factor, as well as the microstructure pictures of the steel material with a smaller magnification factor and different resolutions. After the expert's grading and evaluation of the aging damage, bind the evaluation level to the picture to construct a data set; Step 2: Perform the same preprocessing on all the pictures collected in Step 1. The preprocessing method is as follows: 1)剔除金相显微镜获取的微观组织图中含有的文字说明部分,得到只含有微观组织图本体的初始数据集,将采集到的标准分辨率数据作为初始训练数据集T0,将采集到的非标准多种分辨率部分的数据作为验证数据集V0; 1)剔除金相显微镜获取的微观组织图中含有的文字说明部分,得到只含有微观组织图本体的初始数据集,将采集到的标准分辨率数据作为初始训练数据集T0,将采集到的非标准多种分辨率部分的数据作为验证数据集V0; Remove the text description part contained in the microstructure picture obtained by the metallographic microscope to obtain an initial data set containing only the microstructure picture itself. Use the collected standard resolution data as the initial training data set T0, and use the collected data with non-standard multiple resolutions as the validation data set V0; 2) Convert the three-channel grayscale image in the initial training data set T0 into a single-channel grayscale image to obtain the training data set T1; 3) Use the bilinear interpolation method to reduce the image data in T1 by different magnification factors to obtain the training data set T2; 4) For each picture in the training data set T2, randomly take an image of size mxnx1, where 1 < m < 100 and 100 < n < 800. If the original picture is smaller than the target resolution, fill in the original picture to the size of nxn in a complementary way to obtain a new training data set T3; Step 3: Use the deep learning framework PyTorch to build a ResNet-FPN-MC model, where ResNet represents a convolutional neural network architecture: Residual Neural Network, FPN represents Feature Pyramid Network, and MC is martensite grading. The input image size of the network is nxnx1, and the output is a 5-dimensional vector, and each dimension represents the probability that the material belongs to a certain category; Step 4: Use the training data set T3, adopt cross-entropy as the loss function, and apply the stochastic gradient descent method to train the model; The specific training steps are as follows: 1) Initialize the parameters in the backbone network of the current model using the pre-trained ResNet; 2) Randomly shuffle the order of the pictures in the training set; 3) Input a batch of pictures into the neural network each time, record the output vector of the neural network, calculate the value of the cross-entropy through the output vector of the network and the class label of the data, and then perform backpropagation according to the value of the cross-entropy to update the model parameters; After calculating the loss function for each picture in the training data set T3 and completing the backpropagation, it is recorded as one round of training; 4) Model fine-tuning; 5) Record the model parameters in each round of training, and at the same time judge whether the maximum number of training rounds is reached. If the maximum number of training rounds is reached, execute Step 6. If not, return to Step 3 to continue training; 6) Take out the model with the highest accuracy on the validation set during the training process; Step 5: determine whether the resolution of the microstructure image of the high-chromium martensitic heat-resistant steel to be identified is within the range that can be determined. If it is within the range that can be determined, perform preprocessing according to 1) and 2) in step 2, and then use the model trained in step 4 for automatic classification; The structure of the ResNet-FPN-MC model includes an input layer, four convolutional modules, an FPN network, and an output layer; The input and output channels of the four convolutional modules are (64, 256), (128, 512), (256, 1024), and (512, 2048), respectively. The four convolutional modules are stacked with 3, 4, 6, and 3 residual blocks respectively; The FPN network has four 1x1 convolution kernels, whose input and output channels are (256, 256), (512, 256), (1024, 256), and (2048, 256). Each 1x1 convolution kernel is followed by a 3x3 convolution kernel with input and output of (256, 256). The output layer consists of two parts. The first part has two multi-layer perceptron layers with input and output of (12544, 1024) and (1024, 1024) respectively. The second part has only one multi-layer perceptron base layer with input and output of (1024, 5).
2. The method according to claim 1, characterized in that In step 3) of the above step 2, all images in the training data set T1 are reduced with interpolation to construct a multi-scale training set with data of different resolutions.
3. The method according to claim 1, characterized in that In the above step 4, the number of iterations is pre-set to 10 to 100 times; the cross entropy is used as the loss function, and no regularization term is added; when the stochastic gradient descent method is applied to train the model, the model parameters of each iteration are retained.
4. The method according to claim 1, characterized in that In step 3) of step 4 above, 16, 32 or 64 images are sequentially input to ResNet-FPN-MC each time.
