Iron-carbon alloy microscopic structure identification method based on deep learning

Abstract

The invention relates to the technical field of material science and image processing, in particular to an iron-carbon alloy microstructure identification method based on deep learning, which comprises the following steps of: processing and shooting all metallographic samples; the method comprises the following steps of: calculating a gray histogram of a metallographic image, then calculating a cumulative distribution function, then equalizing the image, and finally cutting the preprocessed image to a proper size; inputting the preprocessed picture into a Swin Transform model to carry out deep feature extraction learning on the picture; and inputting the extracted image features of each class of iron-carbon alloy microstructures into a full-connection layer through a LayerNorm layer and an Average Pool layer to train a multi-classifier, thereby realizing classification and identification of the iron-carbon alloy microstructures. According to the method, various microstructures of the iron-carbon alloy can be efficiently classified and recognized, and the accuracy and efficiency of recognizing the microstructures of the iron-carbon alloy are improved.

Description

Technical Field

[0001] The present invention relates to the fields of material science and image processing technology, and in particular to an iron-carbon alloy microstructure recognition method based on deep learning. Background Art

[0002] Iron-carbon alloys are binary alloys with iron and carbon as components, which are widely used in industry and daily life. According to the carbon content, iron-carbon alloys can be divided into two main types. 1. Steel: The carbon content is generally between 0.02% and 2.11%. Steel plays a vital role in the construction of modern society and is widely used in automobiles, machinery manufacturing and national military industries. 2. Cast iron: The carbon content is generally between 2.11% and 6.67%. Cast iron has the main advantages: it is inexpensive and has excellent casting properties. Therefore, cast iron is widely used in large mechanical parts such as wheels, gears and crankshafts of automobiles, as well as wind power generation components. As an important engineering material, the performance and application of iron-carbon alloys are widely affected by the microstructure. Accurate classification and identification of the microstructure of iron-carbon alloys is crucial to improving the performance of materials, which is of great practical significance in both the steel industry and the foundry industry.

[0003] Metallographic inspection is one of the most widely used research and testing methods in the field of materials science and engineering. Due to the large number of metal microstructure classifications and complex features, the identification process is difficult and professional. Traditional metallographic identification is often affected by subjective factors of professionals, low efficiency, poor repeatability and other factors, resulting in poor accuracy. Most of the early metallographic identification methods used a combination of image processing and machine learning, that is, the metallographic images were first grayed and filtered, and then machine learning methods were used to extract features from the metallographic images. However, there is still a problem of low accuracy in microstructure classification and recognition. In recent years, with the advancement of computer vision and deep learning technology, especially the breakthrough progress of convolutional neural networks (CNN), various convolutional neural networks have been proven to be able to extract complex features from various types of images through learning and training, and have performed well in tasks such as classification, segmentation, and recognition. Some convolutional neural networks have been applied to the classification and recognition of metal microstructures. However, previous studies have mainly focused on the recognition of a single category of microstructures, such as low-carbon steel, cast iron, heat-resistant steel, and aluminum, and relatively little research has been done on iron-carbon alloys.

[0004] In order to solve the above problems, the present invention designs a new iron-carbon alloy microstructure identification method, which can effectively classify and identify various types of microstructures, including pure iron, hypoeutectoid steel, eutectoid steel, hypereutectoid steel, hypoeutectic white cast iron, eutectic white cast iron and hypereutectic white cast iron. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides an iron-carbon alloy microstructure identification method based on deep learning, which adopts an end-to-end deep neural network classification model and has been verified to show superior performance in the identification of the microstructure of iron-carbon alloys. It can accurately distinguish different types of microstructures and provide strong technical support for the research of materials science.

[0006] A method for identifying iron-carbon alloy microstructure based on deep learning, comprising the following steps:

[0007] S1. Data collection:

[0008] After the metallographic samples were processed by grinding, polishing and etching in sequence, the metallographic samples were photographed by IE200 and RX50M optical microscopes to obtain metallographic images;

[0009] S2. Data preprocessing:

[0010] First, the original metallographic image is grayed out. The graying process converts the color image into a single-channel image, which is achieved by formula (1):

[0011] Y=0.2989R+0.5870G+0.1140B (1)

[0012] Among them, Y represents the grayscale value, while R, G and B are the pixel values ​​of the red, green and blue channels respectively;

[0013] The generated grayscale image is subjected to histogram equalization. First, the grayscale histogram h(K) of the image is calculated by formula (2):

[0014]

[0015] Where n(k) is the number of pixels at gray level k, and N is the total number of pixels in the image;

[0016] Then calculate the cumulative distribution function CDF(K), as shown in formula (3):

[0017]

[0018] Where h(i) usually represents the number of pixels with grayscale value i in the image, that is, the frequency of the corresponding grayscale level i in the grayscale histogram; then the equalized grayscale value is calculated by the following formula (4):

[0019] G(k)=T·CDF(k) (4)

[0020] Where G(k) is the gray value after equalization, T is the maximum gray level of the image, and the cumulative distribution function CDF(k) represents the total number of all pixels whose gray value is less than or equal to k;

[0021] S3. Deep feature extraction learning:

[0022] Adopting Swin Transformer network as the backbone;

[0023] In the Swin Transformer model, there are two main modules: the window-based multi-head attention W-MHSA module and the moving window-based multi-head self-attention SW-MHSA module; the W-MHSA module is calculated within a local window of fixed size; the computational complexity of the standard global attention mechanism is O(n 2 ), while W-MHSA only calculates attention within a local window, so the computational complexity is reduced to O(m 2 ), where m is the size of the window, which is much smaller than the size of the entire input n;

[0024] W-MHSA focuses on the relationship within the local area and can effectively capture local features. In addition, the SW-MHSA module uses window shifting to enable information exchange between pixels originally in different windows through the attention mechanism.

[0025] Equations (5) to (8) describe the detailed procedures of the W-MHSA and SW-MHSA modules:

[0026]

[0027] Where MLP(.) represents multi-layer perceptron, LN(.) represents LayerNorm layer; represents the output of the Lth layer after being processed by the window multi-head self-attention mechanism (W-MHSA); y l represents the output of the Lth layer after MLP processing; represents the output of the L+1th layer after being processed by the moving window multi-head self-attention mechanism (SW-MHSA); y l+1 represents the output of the L+1th layer after MLP processing;

[0028] S4. Microstructure classification:

[0029] The high-level features extracted from the microstructure are input into the trained classifier for classification and identification of the microstructure;

[0030] In order to improve the performance of the model and stabilize the training process, a LayerNorm layer and an Average Pool layer are added at the end of the model; in addition, the classification layer of the original Swin Transformer is replaced with a fully connected layer that receives d-dimensional features and outputs a 7-dimensional vector;

[0031] For multi-class classification tasks, the cross entropy loss function was used;

[0032] Formula (9) defines the multi-classification cross entropy loss formula:

[0033]

[0034] Where N represents the number of samples, C represents the number of categories; y * Represents a one-hot encoding vector corresponding to the true label of the i-th sample, p * A vector representing the predicted probability distribution of the model.

[0035] Preferably, in step S1, the etchant is 4% nitric acid alcohol solution.

[0036] Preferably, at the end of step S2, the preprocessed image is cropped to a suitable size.

[0037] Preferably, the Swin Transformer model in step S3 includes four main steps, each of which extracts features of a specific scale; Patch Partition first divides the input image into multiple small blocks, then flattens the small blocks and passes them to the subsequent linear embedding layer; each small block is mapped to a higher-dimensional embedding vector; the LinearEmbedding step maps the divided small blocks to a higher-dimensional feature space through a linear transformation; SwinTransformer Block uses a sliding window mechanism to process image blocks, which can retain the features of local areas; Patch Merging Layer performs downsampling at the beginning of each step, expanding the number of channels of the feature map while reducing its overall size.

[0038] The beneficial effects of the present invention are:

[0039] The present invention discloses an iron-carbon alloy microstructure identification method based on deep learning, which can efficiently classify and identify various microstructures of iron-carbon alloys. It adopts an end-to-end deep neural network classification model, which has been verified to show excellent recognition performance and can accurately distinguish different types of microstructures. It not only improves the accuracy and efficiency of iron-carbon alloy microstructure identification, but also can realize comprehensive classification and identification of various types of microstructures, further promoting the progress in the field of materials engineering. Through accurate classification and identification, it can provide important reference for subsequent material performance optimization and application development. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 This is a flow chart of a method for identifying the microstructure of an iron-carbon alloy based on deep learning according to the present invention;

[0042] Figure 2 This is the data preprocessing operation workflow in step 2 of the present invention;

[0043] Figure 3 It is the iron-carbon alloy microstructure identification model based on Swin Transformer network in step 3;

[0044] Figure 4 It is the W-MSA module and SM-MSA module structure in step 3. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0046] The present invention proposes a method for identifying the microstructure of an iron-carbon alloy based on deep learning. The method is divided into the following four steps: first, seven processed metallographic samples are placed under an optical microscope for photographing; second, the collected metallographic images are first grayed and then the grayscale histogram of the image is calculated, and then the cumulative distribution function is calculated, and then the image is equalized, and finally the preprocessed image is cropped to an appropriate size; third, the preprocessed image is input into a deep neural network classification model for feature extraction, and the high-level features or structures (such as edges, textures, shapes, etc.) representing the metallographic image are extracted; fourth, the extracted features representing each type of metallographic microstructure are input into a classifier, and then the classifier is trained to classify different types of microstructures. The specific implementation method of each step is as follows:

[0047] S1. Data collection:

[0048] In this step, all metallographic samples were processed through three steps: grinding, polishing, and etching. The etchant was 4% nitric acid alcohol solution, and the metallographic samples were photographed using IE200 and RX50M optical microscopes.

[0049] S2. Data preprocessing:

[0050] In this step, in order to reduce the computational complexity, the original metallographic image is first grayscaled. The grayscale process converts the color image into a single-channel image, which is achieved through formula (1):

[0051] Y=0.2989R+0.5870G+0.1140B (1)

[0052] In this formula, Y represents the grayscale value, while R, G, and B are the pixel values ​​of the red, green, and blue channels, respectively. Next, the generated grayscale image is histogram equalized. First, the grayscale histogram h(K) of the image is calculated by formula (2):

[0053]

[0054] Where n(k) is the number of pixels at gray level k, and N is the total number of pixels in the image. The cumulative distribution function CDF(K) is then calculated as shown in formula (3):

[0055]

[0056] In formula (3), h(i) usually represents the number of pixels with grayscale value i in the image, that is, the frequency of the corresponding grayscale level i in the grayscale histogram. Then the equalized grayscale value is calculated by the following formula (4):

[0057] G(k)=T·CDF(k) (4)

[0058] Among them, G(k) is the gray value after equalization, T is the maximum gray level of the image, and the cumulative distribution function CDF(k) represents the total number of all pixels with gray values ​​less than or equal to k.

[0059] Finally, in order to reduce the amount of data and improve computational efficiency, the preprocessed image is cropped to a suitable size. This step ensures that the size of the input image meets the requirements of subsequent processing.

[0060] S3. Deep feature extraction learning:

[0061] In this step, we designed a microstructure recognition model for iron-carbon alloys, which uses the Swin Transformer network as the backbone. The Swin Transformer is an advanced Transformer architecture optimized for computer vision tasks. The computational complexity of the classic Transformer self-attention on N tokens is O(N 2), which is impractical for large-scale visual data. Swin Transformer reduces this complexity to O(N) using a divide-and-conquer strategy, thereby improving computational efficiency.

[0062] The model consists of four main steps, each of which extracts features of a specific scale. Patch Partition first divides the input image into multiple small patches, then flattens the patches and passes them to the subsequent linear embedding layer. Each patch is mapped to a higher-dimensional embedding vector. The Linear Embedding step maps the divided patches to a higher-dimensional feature space through a linear transformation. The uniqueness of the Swin Transformer Block is that it uses a sliding window mechanism to process image patches, which can preserve the features of local areas. The Patch Merging Layer performs downsampling at the beginning of each step, expanding the number of channels of the feature map while reducing its overall size. This operation not only reduces the amount of computation, but also improves the model's perception of large-scale structures.

[0063] In the Swin Transformer model, there are two main modules: the window-based multi-head attention (W-MHSA) module and the moving window-based multi-head self-attention (SW-MHSA) module. The W-MHSA module is calculated within a local window of fixed size. This means that the attention calculation is no longer performed on all pixels of the entire image, but within the local window into which the image is divided, which greatly reduces the computational cost. The computational complexity of the standard global attention mechanism is O(N 2 ), while W-MHSA only calculates attention within a local window, so the computational complexity is reduced to O(m 2 ), where m is the size of the window, which is much smaller than the size of the entire input n. W-MHSA focuses on the relationship within the local area and can effectively capture local features, which is particularly important for local patterns in image tasks such as textures and edges. In addition, the SW-MHSA module overcomes the limitation of W-MHSA being limited to local areas. The use of window shifting allows pixels in different windows to communicate information through the attention mechanism, expanding the receptive field of the model and improving the ability to capture global features. Formulas (5) to (8) illustrate the detailed procedures of the W-MHSA and SW-MHSA modules:

[0064]

[0065] Where MLP(.) represents multi-layer perceptron, LN(.) represents LayerNorm layer; represents the output of the Lth layer after being processed by the window multi-head self-attention mechanism (W-MHSA); yl Represents the output of the Lth layer after MLP processing; represents the output of the L+1th layer after being processed by the moving window multi-head self-attention mechanism (SW-MHSA); y l+1 represents the output of the L+1th layer after MLP processing;

[0066] S4. Microstructure classification:

[0067] In this step, the high-level features extracted from the microstructure are input into the trained classifier for classification and identification of the microstructure.

[0068] In order to improve the performance of the model and stabilize the training process, this solution adds a LayerNorm layer and an Average Pool layer at the end of the model. In addition, the classification layer of the original Swin Transformer is replaced with a fully connected layer that receives d-dimensional features and outputs a 7-dimensional vector.

[0069] For multi-class classification tasks, this solution uses the cross entropy loss function. Formula (9) defines the multi-classification cross entropy loss formula:

[0070]

[0071] Where N represents the number of samples, C represents the number of categories; y * Represents a one-hot encoding vector corresponding to the true label of the i-th sample, p * A vector representing the predicted probability distribution of the model.

[0072] As attached Figure 1 As shown:

[0073] Step 1: Grind, polish and etch all metallographic samples, and use an optical microscope to photograph the metallographic samples;

[0074] Step 2: As attached Figure 2 As shown, the original metallographic image is first grayed out, the grayscale histogram of the image is calculated, and then the cumulative distribution function is calculated. Then the image is equalized, and finally the preprocessed image is cropped to a suitable size. Examples of the microstructures of various types of metallographic samples after preprocessing are as follows:

[0075] Table 1 Microstructure of metallographic samples of various categories after pretreatment

[0076]

[0077] Step 3: As attached Figure 3-4 As shown, the preprocessed image is input into the Swin Transformer model to perform deep feature extraction and learning on the image;

[0078] Step 4: The image features extracted from each type of iron-carbon alloy microstructure are input into the fully connected layer through the LayerNorm layer and an Average Pool layer to train multiple classifiers, and finally the classification and recognition of the iron-carbon alloy microstructure is realized.

[0079] The present invention discloses an iron-carbon alloy microstructure identification method based on deep learning, which can efficiently classify and identify various microstructures of iron-carbon alloys, including pure iron, hypoeutectoid steel, eutectoid steel, hypereutectoid steel, hypoeutectic white cast iron, eutectic white cast iron and hypereutectic white cast iron. The method adopts an end-to-end deep neural network classification model, which has been verified to show excellent recognition performance and can accurately distinguish different types of microstructures. This not only improves the accuracy and efficiency of iron-carbon alloy microstructure identification, reduces dependence on manual inspection, but also provides a reference for metallographic analysis, promotes the automation process of metallographic testing, and provides strong technical support for further research in the field of materials science and engineering. Through accurate classification and identification, it lays an important foundation for subsequent material performance optimization and application development.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for identifying the microstructure of an iron-carbon alloy based on deep learning, characterized in that: The following steps are involved: S1. Data collection: After the metallographic samples were processed by grinding, polishing and etching in sequence, the metallographic samples were photographed by IE200 and RX50M optical microscopes to obtain metallographic images; S2. Data preprocessing: First, the original metallographic image is grayed out. The graying process converts the color image into a single-channel image, which is achieved by formula (1): Y=0.2989R+0.5870G+0.1140B (1) Among them, Y represents the grayscale value, while R, G and B are the pixel values ​​of the red, green and blue channels respectively; The generated grayscale image is subjected to histogram equalization. First, the grayscale histogram h(K) of the image is calculated by formula (2): Where n(k) is the number of pixels at gray level k, and N is the total number of pixels in the image; Then calculate the cumulative distribution function CDF(K), as shown in formula (3): Where h(i) usually represents the number of pixels with grayscale value i in the image, that is, the frequency of the corresponding grayscale level i in the grayscale histogram; then the equalized grayscale value is calculated by the following formula (4): G(k)=T·CDF(k) (4) Where G(k) is the gray value after equalization, T is the maximum gray level of the image, and the cumulative distribution function CDF(k) represents the total number of all pixels whose gray value is less than or equal to k; S3. Deep feature extraction learning: Adopting Swin Transformer network as the backbone; In the Swin Transformer model, there are two main modules: the window-based multi-head attention W-MHSA module and the moving window-based multi-head self-attention SW-MHSA module; the W-MHSA module is calculated within a local window of fixed size; the computational complexity of the standard global attention mechanism is O(n 2 ), while W-MHSA only calculates attention within a local window, so the computational complexity is reduced to O(m 2 ), where m is the size of the window, which is much smaller than the size of the entire input n; W-MHSA focuses on the relationship within the local area and can effectively capture local features. In addition, the SW-MHSA module uses window shifting to enable information exchange between pixels originally in different windows through the attention mechanism. Equations (5) to (8) describe the detailed procedures of the W-MHSA and SW-MHSA modules: Where MLP(.) represents multi-layer perceptron, LN(.) represents LayerNorm layer; represents the output of the Lth layer after being processed by the window multi-head self-attention mechanism (W-MHSA); yl represents the output of the Lth layer after being processed by MLP; represents the output of the L+1th layer after being processed by the moving window multi-head self-attention mechanism (SW-MHSA); y l+1 represents the output of the L+1th layer after MLP processing; S4. Microstructure classification: The high-level features extracted from the microstructure are input into the trained classifier for classification and identification of the microstructure; In order to improve the performance of the model and stabilize the training process, a LayerNorm layer and an Average Pool layer are added at the end of the model; in addition, the classification layer of the original Swin Transformer is replaced with a fully connected layer that receives d-dimensional features and outputs a 7-dimensional vector; For multi-class classification tasks, the cross entropy loss function was used; Formula (9) defines the multi-classification cross entropy loss formula: Where N represents the number of samples, C represents the number of categories; y * Represents a one-hot encoding vector corresponding to the true label of the i-th sample, p * A vector representing the predicted probability distribution of the model.

2. The method for identifying the microstructure of an iron-carbon alloy based on deep learning according to claim 1, characterized in that: In step S1, the etchant is 4% nitric acid alcohol solution.

3. The method for identifying the microstructure of an iron-carbon alloy based on deep learning according to claim 1, characterized in that: Finally, step S2 cuts the preprocessed image to a suitable size.

4. The method for identifying the microstructure of an iron-carbon alloy based on deep learning according to claim 1, characterized in that: The Swin Transformer model in step S3 includes four main steps, each of which extracts features of a specific scale; Patch Partition first divides the input image into multiple small blocks, then flattens the small blocks and passes them to the subsequent linear embedding layer; each small block is mapped to a higher-dimensional embedding vector; the Linear Embedding step maps the divided small blocks to a higher-dimensional feature space through linear transformation; Swin Transformer Block uses a sliding window mechanism to process image blocks, which can retain the features of local areas; Patch Merging Layer performs downsampling at the beginning of each step, expanding the number of channels of the feature map while reducing its overall size.

Images