Steel metallographic grain size grade determination method and system, electronic equipment and medium

Abstract

The invention discloses a steel metallographic grain size grade determination method and system, electronic equipment and a medium, and relates to the technical field of material metallographic image grain size grade evaluation. The method comprises the following steps: cutting an original metallographic image set to obtain an expanded metallographic image set; cutting the metallographic grain boundary image set to obtain an extended tag set; training and verifying the convolutional neural network according to the extended metallographic image set and the extended label set to obtain a grain boundary segmentation convolutional neural network model; erasing the grain boundary of the extended tag set to obtain a missing tag set; training the convolutional neural network according to the missing tag set and the extended tag set to obtain a missing grain boundary automatic closing convolutional neural network model; inputting a to-be-predicted image into the two models to obtain a to-be-predicted closed grain boundary image; and performing boundary closing processing on the to-be-predicted closed grain boundary image to obtain a steel metallographic grain size grade. The method can accurately and efficiently determine the metallographic grain size grade of the steel.

Description

Technical Field

[0001] This invention relates to the field of metallographic image grain size grade assessment technology, and in particular to a method, system, electronic device and medium for determining the metallographic grain size grade of steel. Background Technology

[0002] Steel is a widely used structural material. Its internal structure is a reflection of its mechanical and chemical properties. Under the influence of different processing techniques and service conditions, its internal structure exhibits significant changes.

[0003] Metallographic structure mainly refers to the specific microscopic morphology of various metals or alloys under an optical microscope. Under a metallographic microscope, the surface of most steel materials exhibits a granular structure. The size of the metallographic grains is expressed as grain size, which basically reflects the size of the internal grains of the steel part under actual heat treatment or hot working conditions. Usually, the grains in the metal matrix cannot be completely the same size, so "average grain size" and "average grain size grade" are often used to characterize the dimensional characteristics of the internal grains of steel.

[0004] In automated assessment of metallographic grain size, the accuracy of grain boundary identification directly impacts the accuracy of grain size assessment. Traditional grain size calculation methods typically involve manual extraction of grain boundaries. However, manual grain boundary extraction is subject to subjectivity and randomness. Simple features such as color and brightness differences between grain boundaries and grains, as well as shallow gradient information between pixels, can all affect the actual detection results due to human subjectivity, leading to poor accuracy. Furthermore, manually extracted grain size assessment methods are extremely time-consuming. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, electronic device, and medium for determining the metallographic grain size grade of steel, which can accurately and efficiently determine the metallographic grain size grade of steel.

[0006] To achieve the above objectives, the present invention provides the following solution: a method for determining the metallographic grain size grade of steel, comprising: acquiring an original metallographic image set and cropping each metallographic image of steel in the original metallographic image set according to a set pixel interval and a set size to obtain an expanded metallographic image set; the original metallographic image set includes multiple metallographic images of steel samples that have been prepared.

[0007] The grain boundaries in the metallographic images of each steel material in the original metallographic image set are drawn to obtain the metallographic grain boundary image set.

[0008] An expanded tag set is obtained by cropping the metallographic grain boundary images of each steel material in the metallographic grain boundary image set according to the expanded metallographic image set; the expanded tag set includes the metallographic grain boundary images of each steel material corresponding to the metallographic images in the expanded metallographic image set.

[0009] Using the expanded metallographic image set as input and the expanded label set as output, the convolutional neural network is trained and validated to obtain a grain boundary segmentation convolutional neural network model.

[0010] The grain boundaries on the metallographic grain boundary images of each steel material in the expanded tag set are erased according to the masking coefficient to obtain the missing tag set.

[0011] Using the missing label set as input and the expanded label set as output, the convolutional neural network is trained to obtain a convolutional neural network model for automatically closing missing grain boundaries.

[0012] The image to be predicted is obtained, and the image to be predicted is input into the grain boundary segmentation convolutional neural network model to obtain the grain boundary image to be predicted.

[0013] The image of the grain boundary to be predicted is input into the automatic closing convolutional neural network model for missing grain boundaries to obtain the image of the closed grain boundary to be predicted.

[0014] The image of the closed grain boundary to be predicted is processed by boundary closure to obtain a metallographic image with completely closed grain boundaries.

[0015] The metallographic grain size grade of the steel in the image to be predicted is obtained from the metallographic image of the grain boundary being completely closed.

[0016] Optionally, the step of cropping each metallographic image of steel in the original metallographic image set according to a set pixel interval and a set size to obtain an expanded metallographic image set specifically includes: converting each metallographic image of steel in the original metallographic image set into a grayscale image to obtain a grayscale image set.

[0017] Histogram equalization is performed on each image in the grayscale image set to obtain an equalized image set.

[0018] Each image in the equalized image set is filtered to obtain a filtered image set.

[0019] Each image in the filtered image set is cropped according to a set pixel interval and a set size to obtain an expanded metallographic image set.

[0020] Optionally, the convolutional neural network includes: a first max pooling layer, a second max pooling layer, a first convolutional activation module, a second convolutional activation module, a first copy stacking unit, a second copy stacking unit, a first upsampling convolutional unit, and a second upsampling convolutional unit; the first convolutional activation module includes a first convolutional activation unit and a second convolutional activation unit connected in sequence; the second convolutional activation module includes a third convolutional activation unit and a fourth convolutional activation unit connected in sequence.

[0021] The input of the first max pooling layer is connected to the input of the first copy stacking unit and the input of the first convolutional activation unit, respectively. The output of the first convolutional activation unit is connected to the input of the second convolutional activation unit. The output of the second convolutional activation unit is connected to the input of the second copy stacking unit and the input of the second max pooling layer, respectively. The output of the second max pooling layer is connected to the input of the first upsampling convolutional unit. The outputs of the first copy stacking unit, the second copy stacking unit, and the first upsampling convolutional unit are all connected to the input of the third convolutional activation unit. The output of the third convolutional activation unit is connected to the input of the fourth convolutional activation unit. The output of the fourth convolutional activation unit is connected to the input of the second upsampling convolutional unit.

[0022] Optionally, the step of performing boundary closure processing on the image of the closed grain boundary to be predicted to obtain a metallographic image with completely closed grain boundaries specifically includes: performing adaptive threshold binarization processing on the image of the closed grain boundary to be predicted to obtain a binary image of the closed grain boundary to be predicted.

[0023] A morphological opening operation is performed on the binary image of the closed grain boundary to be predicted to obtain a binarized image after the opening operation.

[0024] The image after the opening operation is obtained by removing regions with closed holes smaller than a set threshold from the binarized image.

[0025] The skeleton of the removed image is thinned to obtain a single-pixel grain binarized boundary image.

[0026] A metallographic image with completely closed grain boundaries is obtained from the binarized grain boundary image of the single pixel.

[0027] Optionally, obtaining the metallographic grain size grade of the steel in the image to be predicted from the metallographic image with completely closed grain boundaries specifically includes: equally dividing and cropping the metallographic image with completely closed grain boundaries to obtain multiple equally divided and cropped metallographic images.

[0028] For any metallographic image after equal division and cropping, the intersection of the equal-divided metallographic image with the standard measurement grid is obtained to obtain the set of intercept points corresponding to the equal-divided metallographic image; the set of intercept points includes the points where the boundary of the equal-divided metallographic image intersects with the standard measurement grid.

[0029] The length between every two points in the cutoff point set is subjected to extremum removal processing, and the average cutoff length of the metallographic image after equal division is calculated based on the length between every two points after extremum removal processing; the length between every two points is the length of the boundary of the metallographic image after equal division corresponding to the two points.

[0030] Calculate the average of the average cross-section lengths of all metallographic images after equal cropping.

[0031] The average cross-sectional length of all equally divided metallographic images is compared with the equiaxed grain micrograin size relationship table to obtain the grain size grade of the metallographic image to be predicted.

[0032] A system for determining the metallographic grain size grade of steel includes: a metallographic image expansion module, used to acquire an original metallographic image set and crop each metallographic image of steel in the original metallographic image set according to a set pixel interval and a set size to obtain an expanded metallographic image set; the original metallographic image set includes multiple metallographic images of steel samples that have been prepared.

[0033] The grain boundary drawing module is used to draw the grain boundaries in the metallographic images of each steel in the original metallographic image set to obtain a metallographic grain boundary image set.

[0034] The tag expansion module is used to crop the metallographic grain boundary images of each steel material in the metallographic grain boundary image set according to the expanded metallographic image set to obtain an expanded tag set; the expanded tag set includes the metallographic grain boundary images of each steel material corresponding to the metallographic images in the expanded metallographic image set.

[0035] The grain boundary segmentation convolutional neural network model determination module is used to train and verify the convolutional neural network with the expanded metallographic image set as input and the expanded label set as output to obtain the grain boundary segmentation convolutional neural network model.

[0036] The grain boundary erasure module is used to erase the grain boundaries on the metallographic grain boundary images of each steel material in the expanded tag set according to the mask coefficient, so as to obtain the missing tag set.

[0037] The module for determining the automatic closing convolutional neural network model of missing grain boundaries is used to train the convolutional neural network with the missing label set as input and the expanded label set as output to obtain the automatic closing convolutional neural network model of missing grain boundaries.

[0038] The image acquisition module is used to acquire the image to be predicted and input the image to be predicted into the grain boundary segmentation convolutional neural network model to obtain the grain boundary image to be predicted.

[0039] The module for determining the closed grain boundary image to be predicted is used to input the image of the grain boundary to be predicted into the automatic closing convolutional neural network model of the missing grain boundary to obtain the image of the closed grain boundary to be predicted.

[0040] The boundary closure processing module is used to perform boundary closure processing on the image of the grain boundary to be predicted to obtain a metallographic image with completely closed grain boundaries.

[0041] The steel metallographic grain size determination module is used to obtain the steel metallographic grain size of the image to be predicted based on the metallographic image with completely closed grain boundaries.

[0042] Optionally, the metallographic image augmentation module specifically includes: a grayscale unit, used to convert each metallographic image of steel in the original metallographic image set into a grayscale image to obtain a grayscale image set.

[0043] The equalization unit is used to perform histogram equalization on each image in the grayscale image set to obtain an equalized image set.

[0044] The filtering unit is used to filter each image in the equalized image set to obtain a filtered image set.

[0045] The cropping unit is used to crop each image in the filtered image set according to a set pixel interval and a set size to obtain an expanded metallographic image set.

[0046] Optionally, the closure processing module specifically includes: a binarization processing unit, used to perform adaptive threshold binarization processing on the image of the closed grain boundary to be predicted, to obtain a binary image of the closed grain boundary to be predicted.

[0047] The morphological opening unit is used to perform a morphological opening operation on the binary image of the closed grain boundary to be predicted to obtain a binarized image after the opening operation.

[0048] The removal unit is used to remove regions with closed holes smaller than a set threshold from the binarized image after the opening operation to obtain the removed image.

[0049] A skeleton refinement unit is used to perform skeleton refinement operations on the removed image to obtain a single-pixel grain binarized boundary image.

[0050] The grain boundary fully closed image determination unit is used to obtain a metallographic image with fully closed grain boundaries based on the single-pixel grain binarized boundary image.

[0051] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor running the computer program to cause the electronic device to perform the above-described method for determining the metallographic grain size grade of steel.

[0052] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for determining the metallographic grain size grade of steel.

[0053] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: The present invention trains a convolutional neural network to obtain two models, uses the two models to obtain the image of the closed grain boundary to be predicted, maximizes the extraction of effective feature information of metallographic image to achieve the optimal grain boundary, and then performs boundary closure processing on the image of the closed grain boundary to be predicted to obtain a completely closed image to optimize the grain boundary. This not only overcomes the subjectivity and inefficiency of human intervention, but also greatly improves the accuracy and speed of grain size prediction of material metallographic image. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart illustrating the method for determining the metallographic grain size grade of steel provided in this embodiment of the invention.

[0056] Figure 2 This is a comparison of the grayscale histogram images before and after histogram equalization.

[0057] Figure 3 This is the original metallographic image of the present invention.

[0058] Figure 4 To Figure 3 The result is obtained by plotting the grain boundaries.

[0059] Figure 5 This is an example image of the austenitic metallographic structure predicted by the present invention.

[0060] Figure 6 This is an example metallographic image of ferrite + pearlite predicted by the present invention.

[0061] Figure 7 This is a schematic diagram of the convolutional neural network of the present invention.

[0062] Figure 8 This is an image of the grain boundaries.

[0063] Figure 9 To Figure 8 The result image obtained after erasing.

[0064] Figure 10 The image to be predicted.

[0065] Figure 11 To be Figure 10 The result is shown in the image obtained from the input grain boundary segmentation convolutional neural network model.

[0066] Figure 12 A visualization image of the microcrystalline grain size relationship table provided by this invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0068] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] This invention provides a method for determining the metallographic grain size grade of steel, comprising: acquiring an original metallographic image set and cropping each metallographic image of steel in the original metallographic image set according to a set pixel interval and a set size to obtain an expanded metallographic image set; the original metallographic image set includes multiple metallographic images of steel samples that have been prepared.

[0070] The grain boundaries in the metallographic images of each steel in the original metallographic image set are drawn to obtain the metallographic grain boundary image set. Specifically, the corresponding metallographic image grain boundaries can be manually drawn according to the original metallographic image set to obtain the metallographic grain boundary image set.

[0071] An expanded tag set is obtained by cropping the metallographic grain boundary images of each steel material in the metallographic grain boundary image set according to the expanded metallographic image set. The expanded tag set includes the metallographic grain boundary images of each steel material in the expanded metallographic image set. Specifically, the metallographic grain boundary image set is cropped according to the above-described method for cropping the original metallographic image set to obtain the expanded tag set.

[0072] To ensure strict consistency and independence between the training and validation sets, both the training and validation sets were created according to the above-described process, with a ratio of N:1 (N>3). The expanded metallographic image set was used as input and the expanded label set was used as output to train and validate the convolutional neural network, resulting in a grain boundary segmentation convolutional neural network model.

[0073] The grain boundaries on the metallographic grain boundary images of each steel material in the expanded tag set are erased according to the masking coefficient to obtain the missing tag set.

[0074] Using the missing label set as input and the expanded label set as output, a convolutional neural network is trained to obtain a convolutional neural network model for automatically closing missing grain boundaries. This model is used to automatically close and connect unclosed grain boundaries in the image obtained by the grain boundary segmentation convolutional neural network model.

[0075] The image to be predicted is obtained, and the image to be predicted is input into the grain boundary segmentation convolutional neural network model to obtain the grain boundary image to be predicted.

[0076] The image of the grain boundary to be predicted is input into the convolutional neural network model for automatic closure of missing grain boundaries to obtain the image of the closed grain boundary to be predicted. Specifically, after the image of the grain boundary to be predicted is grayscaled and binarized, it is fed into the convolutional neural network model for automatic closure of missing grain boundaries to complete the missing grain boundaries and obtain the image of the closed grain boundary to be predicted.

[0077] The image of the closed grain boundary to be predicted is processed to obtain a metallographic image with completely closed grain boundaries. Specifically, a series of digital image processing operations (such as opening operation, removal of small areas, removal of burrs and isolated points, and skeletonization) are performed on the image of the closed grain boundary to be predicted to obtain a metallographic image with completely closed grain boundaries.

[0078] The metallographic grain size grade of the steel in the image to be predicted is obtained from the metallographic image of the grain boundary being completely closed.

[0079] In practical applications, the process of cropping the metallographic images of steel in the original metallographic image set according to a set pixel interval and a set size to obtain an expanded metallographic image set specifically includes: converting all metallographic images of steel in the original metallographic image set into grayscale images to obtain a grayscale image set.

[0080] Histogram equalization is performed on each image in the grayscale image set to obtain an equalized image set, thereby enhancing the contrast of the images.

[0081] Each image in the equalized image set is filtered to obtain a filtered image set, which removes noise from the original metallographic image set.

[0082] Each image in the filtered image set is cropped according to a set pixel interval and a set size to obtain an expanded metallographic image set.

[0083] In practical applications, the convolutional neural network includes: a first convolutional module (comprising two 3D convolutional modules). The first convolutional module (including two 3D convolutional modules) and the second convolutional module (including two 3D convolutional modules) (convolution of the first convolution), the third convolution module (including two 3) (convolution of the first convolution), the fourth convolutional module (including two 3) The fifth convolutional module (including two 3) The sixth convolutional module (including two 3) The seventh convolutional module (including two 3) The convolution and a 1 1 convolution), the first downsampling module (for 2) Max pooling), second downsampling module (for 2) Max pooling), the third downsampling module (for 2) Max pooling), first upsampling module (for 2) The deconvolutional layer), the second upsampling module (for 2) The deconvolutional layer), the third upsampling module (for 2) (deconvolutional layer), first replication stacking unit, second replication stacking unit, and third replication stacking unit, Figure 7 The solid arrow pointing horizontally to the right represents the copying process, and stacking the three layers by channel represents the splicing process.

[0084] The first convolution module performs two convolution operations on the input image; the first downsampling module performs max pooling downsampling on the output of the first convolution module; the second convolution module performs two convolution operations on the output of the first downsampling module; the second downsampling module performs max pooling downsampling on the output of the second convolution module; the third convolution module performs two convolution operations on the output of the second downsampling module; the third downsampling module performs max pooling downsampling on the output of the third convolution module; the fourth convolution module performs two convolution operations on the output of the third downsampling module; the first upsampling module performs deconvolution on the output of the fourth convolution module; the first copy and overlay unit performs deconvolution on the third convolution module. The outputs of the first convolution module, the first deconvolution module, and the second downsampling module are concatenated through channels; the fifth convolution module performs two convolution operations on the output of the first copy-overlay unit; the second upsampling module performs deconvolution on the output of the fifth convolution module; the second copy-overlay unit performs channel concatenation on the outputs of the second convolution module, the second deconvolution module, and the first downsampling module; the sixth convolution module performs two convolution operations on the output of the second copy-overlay unit; the third upsampling module performs deconvolution on the output of the sixth convolution module; the third copy-overlay unit performs channel concatenation on the outputs of the first convolution module and the third deconvolution module; and the seventh convolution module performs a convolution operation on the output of the third copy-overlay unit.

[0085] In practical applications, a convolutional neural network (CNN) model for grain boundary segmentation is obtained by training and validating the extended metallographic image set as input and the extended label set as output. Specifically, this includes: inputting the training set into a CNN (such as U-Net, FCN, etc.) and training it using conventional CNN training methods to obtain the grain boundary segmentation CNN model. This model is used for image segmentation of grain boundaries and grains in the image to be predicted. Simultaneously, after each training round, the network parameters are fixed, and the training effect of this round is verified using a validation set.

[0086] In practical applications, the step of performing boundary closure processing on the image of the closed grain boundary to be predicted to obtain a metallographic image with completely closed grain boundaries specifically includes: performing adaptive threshold binarization processing on the image of the closed grain boundary to be predicted to obtain a binary image of the closed grain boundary to be predicted.

[0087] A morphological opening operation is performed on the binary image of the closed grain boundary to be predicted to obtain a binarized image after the opening operation.

[0088] The image after the opening operation is obtained by removing regions with closed holes smaller than a set threshold from the binarized image.

[0089] The skeleton of the removed image is thinned to obtain a single-pixel grain binarized boundary image.

[0090] A metallographic image with completely closed grain boundaries is obtained from the binarized grain boundary image of the single pixel.

[0091] In practical applications, obtaining the metallographic grain size grade of the steel in the image to be predicted from the metallographic image with completely closed grain boundaries specifically includes: equally dividing and cropping the metallographic image with completely closed grain boundaries to obtain multiple equally divided and cropped metallographic images.

[0092] For any metallographic image after equal division and cropping, the intersection of the equal-divided metallographic image with the standard measurement grid is obtained to obtain the set of intercept points corresponding to the equal-divided metallographic image; the set of intercept points includes the points where the boundary of the equal-divided metallographic image intersects with the standard measurement grid.

[0093] The length between every two points in the cutoff point set is subjected to extremum removal processing, and the average cutoff length of the metallographic image after equal division is calculated based on the length between every two points after extremum removal processing; the length between every two points is the length of the boundary of the metallographic image after equal division corresponding to the two points.

[0094] Calculate the average of the average cross-section lengths of all metallographic images after equal cropping.

[0095] The average cross-sectional length of all equally divided metallographic images is compared with the equiaxed grain micrograin size relationship table to obtain the grain size grade of the metallographic image to be predicted.

[0096] This invention provides a more specific embodiment to describe the above method in detail, such as... Figure 1 As shown, the general steps include (1) collecting metallographic images of steel, preprocessing and expanding them, and making grain boundary images; (2) making independent training sets and validation sets respectively using the same method, and using the training set to train the grain boundary segmentation convolutional neural network model; (3) using the grain boundary images and randomly erased grain boundary images to train the convolutional neural network model that can automatically close grain boundaries and the missing grain boundary automatic closure convolutional neural network model; (4) sending the test images to the grain boundary segmentation convolutional neural network model and the missing grain boundary automatic closure convolutional neural network model in sequence to perform grain boundary segmentation and closure operations. Based on the digital image processing method, a certain threshold is set to perform morphological operations on small grains and burrs in the closed grain boundary image to obtain a relatively complete closed grain boundary image. Finally, the average grain size level is calculated. The specific steps are as follows: 1. Use a metallographic microscope to collect the metallographic images of the steel that have been prepared as the original metallographic image set.

[0097] 2. Preprocess the original metallographic image set to obtain an expanded metallographic image set. The specific steps are as follows: 2.1. Load the original images from the original metallographic image set and apply the formula... Convert the original image to a grayscale image.

[0098] 2.2. A contrast-limited histogram equalization method is used to perform gray-level histogram equalization on the image. This mainly involves calculating the local gray-level histogram of the image and redistributing brightness to change the image contrast. This algorithm is more suitable for improving the local contrast of an image and obtaining more image details. The algorithm's core idea is that the template W moves row by row across the image, and the center of template W... Points on the corresponding image The histogram equalization variation relationship of the template W region is calculated as follows: The relationship between the equalized pixel values ​​of the template center point c(x0,y0) is calculated as follows: ,in, This involves a series of equalization operations performed on the pixel grayscale values ​​within the W region corresponding to the template, including calculating the sub-block histogram of the W region, calculating the clipping threshold, and redistributing pixels. This represents the equalized pixel grayscale value within the W region corresponding to the template.

[0099] This method (2.1 and 2.2) is used to perform batch grayscale histogram equalization on the original image set, changing the image contrast and making its brightness distribution more even. A comparison of the image grayscale histograms before and after the equalization operation is shown below. Figure 2 As shown, where Figure 2 In the image, (a) is the grayscale histogram of the original image. Figure 2 (b) in the image is the grayscale histogram after histogram equalization.

[0100] 2.3. Bilateral filtering is used to filter the image obtained in 2.2. Bilateral filtering is an improved nonlinear filtering method based on Gaussian filtering. It combines spatial proximity and pixel value similarity in a trade-off, considering both spatial information and grayscale similarity. Its key feature is that it incorporates the image's brightness information into the Gaussian filtering function, which is then convolved with the original image to achieve edge-preserving noise reduction, effectively retaining edge details.

[0101] Its corresponding spatial kernel function is: .

[0102] Its corresponding range kernel function is: .

[0103] The bilateral filter weight function obtained by multiplying the above two equations is: .

[0104] The data formula for a bilateral filter can be expressed as: ,in, The center coordinates of the template window; The coordinates of the other coefficients in the template window (that is, the coordinates of other points in the template window excluding the center point); The standard deviation of the Gaussian function (i.e., the sigma value of the coordinate space filter) corresponds to the pixel value. , It is the standard deviation of the Gaussian function (i.e., the sigma value of the color space filter).

[0105] 2.4. Crop the image obtained in 2.3 by pixels. One image Crop into one image every 128px The images are expanded to obtain a metallographic image set. This operation can make the cropped images more suitable for training convolutional neural networks, and at the same time achieve the purpose of expanding the metallographic images, which can reduce the pressure of the scarcity of metallographic image sets to a certain extent.

[0106] 3. Expand the tag set based on the original metallographic image set. The specific steps are as follows: 3.1. Manually draw the corresponding grain boundary images based on the original metallographic images using professional image processing software. The original metallographic images are as follows: Figure 3 As shown, the hand-drawn grain boundary image is as follows: Figure 4 As shown.

[0107] 3.2. Convert the manually drawn grain boundary images into grayscale images and perform binarization. Crop the corresponding manually drawn grain boundary images according to the method of cropping the original metallographic image set to obtain grain boundary images that correspond one-to-one with the metallographic images in the metallographic image set. Use the metallographic images obtained in step 2.4 and their corresponding grain boundary images as the training dataset for the convolutional neural network.

[0108] 4. To ensure the consistency and independence of the training and validation sets, the validation dataset for the convolutional neural network is created using the same method as the training set. The total dataset contains 1404 training images and 114 validation images, with a training to validation ratio of 12:1.

[0109] 5. The training set is input into the improved U-net convolutional neural network model. This model combines the U-net network structure and first prunes the original U-net network, reducing the number of network layers (pruning the last layer of the network), which significantly reduces the network's parameter computation. Simultaneously, the metallographic image feature layers that did not pass through convolutional blocks during downsampling are channel-stacked using a copy-stack strategy (first copy-stack unit and second copy-stack unit). This involves concatenating the pre-convolutional, post-convolutional, and upsampled feature layers of the same layer. Then, convolution and upsampling operations are performed on the stitched feature layers, thereby maximizing the utilization of effective feature information from each image feature layer during downsampling. The feature layer stitching strategy is as follows: Figure 7 As shown. The network is trained using the conventional training method for convolutional neural networks to obtain a convolutional neural network model for grain boundary segmentation. This model can be used for grain boundary and grain segmentation of the image to be predicted. During training, the training effect is verified using a validation set after each training epoch, and the consistency parameter between the predicted grain boundaries and manually labeled grain boundaries is output. This parameter is described as follows: 5.1. Pixel accuracy is the number of correctly predicted samples divided by the total number of samples. Pixel accuracy represents the proportion of pixels with correctly predicted categories out of the total number of pixels. The formula is... ,in, This represents the probability that the actual grain boundary is i but is predicted as j. This represents the probability that the actual grain boundary value is i but is predicted as i. Pixel accuracy can also be simplified as: the sum of the diagonal elements in the confusion matrix divided by the sum of all elements in the matrix, i.e.: .

[0110] 5.2. Dice coefficient.

[0111] The Dice coefficient is a set similarity metric, typically used to calculate the similarity between two samples. Its value ranges from 0 to 1, with a value of 1 representing the best segmentation result and 0 representing the worst. The formula is: .

[0112] 5.3. Class Pixel Accuracy (Cpa).

[0113] This represents the number of correctly predicted pixels in each category divided by the total number of pixels in that category. This invention primarily focuses on the grain boundary pixel accuracy in CPA, and its calculation formula is: ,in, This indicates that the prediction was correct, the prediction result was positive, and the actual result was positive. This indicates a prediction error; the prediction result is positive, and the result is negative. This indicates that the prediction was correct, the predicted result was negative, and the actual result was negative. This indicates a prediction error; the predicted result is negative, while the actual result is positive.

[0114] 6. The grain boundary image described in step 3.2 is randomly erased according to the mask coefficient P to create a randomly missing grain boundary mask image. Then, the grain boundary image described in step 3.2 is used as the original dataset, and the randomly missing grain boundary mask image is used as the image label input into the improved U-Net convolutional neural network. Training is performed using the conventional training method to obtain a convolutional neural network model that can automatically close missing grain boundaries. The coefficient of the randomly missing grain boundary mask, i.e., the mask coefficient P, is defined as: ,in P In the program, this means randomly filling all pixels in the entire binary image with 0, and this loop is executed P times. m represents the length of the entire binary image, n represents the width of the entire image, and C is a user-defined scaling factor. Specifically, C is set to 2, 3, and 4 respectively. Figure 8 The grain boundary image shown is erased, and the resulting images are as follows: Figure 9 As shown in (a), (b), and (c), the grain boundary image and the mask image with randomly missing grain boundaries of different random coefficients are respectively input into the convolutional neural network for training.

[0115] 7. The image to be predicted Crop into 12 images by pixel. One of the images is like Figure 10 As shown, the image is input into a convolutional neural network model for grain boundary segmentation to obtain the grain boundary image to be predicted, as shown below. Figure 11 As shown; Figure 5 The image to be predicted shown can be an austenitic structure, such as... Figure 6 The image to be predicted shown can also be a ferrite + pearlite structure.

[0116] 8. After the image of the grain boundary to be predicted is grayscaled, it is fed into the convolutional neural network model for automatic closure of missing grain boundaries to close the missing grain boundaries, thus obtaining a metallographic grain boundary image with basically closed grain boundaries.

[0117] 9. A series of digital image processing operations (such as binarization, opening operation, removing small area regions, removing burrs and isolated points, etc.) are performed on the metallographic grain boundary image with basically closed grain boundaries to obtain a metallographic image with completely closed grain boundaries. The specific operation steps are as follows: 9.1. Perform adaptive threshold binarization (Otsu) processing on the obtained metallographic grain boundary image with basically closed grain boundaries. This method is also called the maximum inter-class variance method because after image binarization segmentation according to the threshold obtained by the Otsu method, the inter-class variance between the foreground and background images is the largest. The assumption of the OTSU algorithm is that there exists a threshold TH that divides all pixels of the image into two categories C1 (less than TH) and C2 (greater than TH), then the respective means of these two categories of pixels are m1 and m2, and the global mean of the image is mG. At the same time, the probabilities of pixels being divided into C1 and C2 categories are p1 and p2 respectively, and k is the gray level that maximizes the inter-class variance. It is the best algorithm for threshold selection in image segmentation, with simple calculation, not affected by image brightness and contrast, and the segmentation that maximizes the inter-class variance means the minimum misclassification probability. Among them, the formula for the maximum inter-class variance of the gray values of the foreground and background pixels in a gray image is expressed as: , where, , , ; represents the maximum inter-class variance, represents the probability of the i-th gray level, i represents the gray level, L represents the total gray level, and m represents the sum of the products of the gray levels corresponding to the first k gray levels and their probabilities.

[0118] 9.2. Perform a morphological opening operation on the binarized predicted grain boundary image, that is, perform dilation and erosion operations on the binary image respectively. Among them, dilation is an operation of "lengthening" or "thickening" in the binary image, and erosion is an operation of "shrinking" or "refining" the object in the binary image; The morphological opening operation of A by B can be denoted as , and this operation is the result of eroding A by B and then dilating it with B: .

[0119] 9.3. Set a certain threshold S = 60px for the binarized image after the above opening operation, and remove the closed regions in the image where the area si of the closed hole region is < S to obtain a binarized image after removing small area holes, that is, the image after removal.

[0120] 9.4. Perform a thinning skeleton operation on the image after removal to obtain a single-pixel binarized grain boundary image.

[0121] 9.5. Set a certain traversal coefficient M=8, and scan the single-pixel grain binarized boundary image obtained in 9.4 horizontally and vertically by traversing M times respectively. Calculate the sum of the pixel values ​​of the eight neighboring points of each pixel, remove some burrs that are not connected at the grain boundary and isolated points in the pixel to obtain a metallographic image with a completely closed grain boundary. The traversal coefficient M determines the maximum pixel length that can be removed from the burrs.

[0122] 10. For metallographic images with completely closed grain boundaries, the scikit-image library function in Python is used to automatically label the grain regions. This library function can automatically label each closed region in the image and extract the region parameters of each closed region (such as region number, pixel area, centroid coordinates, circumscribed rectangle coordinates, etc.). The center point is calculated based on the circumscribed rectangle, and the region is marked with a serial number on the image. At the same time, the pixel area values ​​of each region are written into a list for statistics. This invention was trained for 20 rounds, and a validation set was performed after each round of training. The optimal index parameters of the predicted grain boundary images and hand-drawn labels obtained during the 20 rounds of training are shown in Table 1.

[0123] Table 1. Results of the optimal evaluation metrics on the validation set.

[0124] 11. Input the metallographic image into the program and cut the metallographic image into equal parts; intersect the cut metallographic image with the standard measurement grid, and automatically count the intersection points of the cut metallographic image and the standard grid and the length of the intercept line between each intersection point; the average number of intersection points of the image to be measured is 33.5. The magnification of the metallographic image used in this invention is uniformly 200 times, and its pixel scale is 0.29 mm / pixel. The formula for calculating the circumference C of the standard circular grid is: Based on the radius and average number of intercepts of the standard circular grid, the circumference C of the standard circle is calculated to be 364.24 μm, and the average intercept length is 10.87 μm. The grain size relationship of uniform, isotropic, equiaxed microcrystalline grains in the national standard is shown in Table 2. This result is compared with... Figure 12 The equiaxed grain microcrystalline grain size relationship table is automatically compared to obtain the grain size level of the metallographic image to be predicted.

[0125] Table 2. Grain size relationship of uniform, isotropic equiaxed microcrystals in national standards.

[0126] This invention uses deep learning to maximize the extraction of effective feature information from metallographic images to achieve optimal segmentation results. It also integrates the advantages of digital image processing methods and morphological operations to optimize segmentation results. This not only overcomes the subjectivity and inefficiency of human intervention, but also greatly improves the accuracy and speed of grain size prediction from metallographic images of materials.

[0127] To address the aforementioned method, this invention provides a system for determining the metallographic grain size grade of steel, comprising: a metallographic image expansion module, used to acquire an original metallographic image set and crop each metallographic image of steel in the original metallographic image set according to a set pixel interval and a set size to obtain an expanded metallographic image set; the original metallographic image set includes multiple metallographic images of steel samples that have been prepared.

[0128] The grain boundary drawing module is used to draw the grain boundaries in the metallographic images of each steel in the original metallographic image set to obtain a metallographic grain boundary image set.

[0129] The tag expansion module is used to crop the metallographic grain boundary images of each steel material in the metallographic grain boundary image set according to the expanded metallographic image set to obtain an expanded tag set; the expanded tag set includes the metallographic grain boundary images of each steel material corresponding to the metallographic images in the expanded metallographic image set.

[0130] The grain boundary segmentation convolutional neural network model determination module is used to train and verify the convolutional neural network with the expanded metallographic image set as input and the expanded label set as output to obtain the grain boundary segmentation convolutional neural network model.

[0131] The grain boundary erasure module is used to erase the grain boundaries on the metallographic grain boundary images of each steel material in the expanded tag set according to the mask coefficient, so as to obtain the missing tag set.

[0132] The module for determining the automatic closing convolutional neural network model of missing grain boundaries is used to train the convolutional neural network with the missing label set as input and the expanded label set as output to obtain the automatic closing convolutional neural network model of missing grain boundaries.

[0133] The image acquisition module is used to acquire the image to be predicted and input the image to be predicted into the grain boundary segmentation convolutional neural network model to obtain the grain boundary image to be predicted.

[0134] The module for determining the closed grain boundary image to be predicted is used to input the image of the grain boundary to be predicted into the automatic closing convolutional neural network model of the missing grain boundary to obtain the image of the closed grain boundary to be predicted.

[0135] The boundary closure processing module is used to perform boundary closure processing on the image of the grain boundary to be predicted to obtain a metallographic image with completely closed grain boundaries.

[0136] The steel metallographic grain size determination module is used to obtain the steel metallographic grain size of the image to be predicted based on the metallographic image with completely closed grain boundaries.

[0137] In practical applications, the metallographic image augmentation module specifically includes a grayscale unit, used to convert each metallographic image of steel in the original metallographic image set into a grayscale image set.

[0138] The equalization unit is used to perform histogram equalization on each image in the grayscale image set to obtain an equalized image set.

[0139] The filtering unit is used to filter each image in the equalized image set to obtain a filtered image set.

[0140] The cropping unit is used to crop each image in the filtered image set according to a set pixel interval and a set size to obtain an expanded metallographic image set.

[0141] In practical applications, the closure processing module specifically includes: a binarization processing unit, used to perform adaptive threshold binarization processing on the image of the closed grain boundary to be predicted, to obtain a binary image of the closed grain boundary to be predicted.

[0142] The morphological opening unit is used to perform a morphological opening operation on the binary image of the closed grain boundary to be predicted to obtain a binarized image after the opening operation.

[0143] The removal unit is used to remove regions with closed holes smaller than a set threshold from the binarized image after the opening operation to obtain the removed image.

[0144] A skeleton refinement unit is used to perform skeleton refinement operations on the removed image to obtain a single-pixel grain binarized boundary image.

[0145] The grain boundary fully closed image determination unit is used to obtain a metallographic image with fully closed grain boundaries based on the single-pixel grain binarized boundary image.

[0146] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor runs the computer program to enable the electronic device to perform the steel metallographic grain size determination method described in the above embodiments.

[0147] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for determining the metallographic grain size grade of steel as described in the above embodiments.

[0148] This invention trains two convolutional neural network models to perform grain boundary recognition and grain boundary closure processing respectively, and then uses morphological methods for feature extraction. It combines the advantages of lightweight convolutional neural networks and digital image processing to solve the problems of low accuracy, high human subjectivity and poor repeatability of traditional methods and existing intelligent grain size rating methods, which greatly improves the accuracy and speed of grain size level prediction in metallographic images.

[0149] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0150] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for determining the metallographic grain size grade of steel, characterized in that, include: An expanded metallographic image set is obtained by acquiring an original metallographic image set and cropping each metallographic image of steel in the original metallographic image set according to a set pixel interval and a set size; the original metallographic image set includes multiple metallographic images of steel samples that have been prepared. The grain boundaries in the metallographic images of each steel in the original metallographic image set are plotted to obtain a metallographic grain boundary image set; An expanded tag set is obtained by cropping the metallographic grain boundary images of each steel material in the metallographic grain boundary image set according to the expanded metallographic image set; the expanded tag set includes the metallographic grain boundary images of each steel material corresponding to the metallographic images in the expanded metallographic image set. Using the expanded metallographic image set as input and the expanded label set as output, the convolutional neural network is trained and validated to obtain a grain boundary segmentation convolutional neural network model; The grain boundaries on the metallographic grain boundary images of each steel material in the expanded tag set are erased according to the masking coefficient to obtain the missing tag set; Using the missing label set as input and the expanded label set as output, the convolutional neural network is trained to obtain a convolutional neural network model for automatically closing missing grain boundaries; Obtain the image to be predicted, and input the image to be predicted into the grain boundary segmentation convolutional neural network model to obtain the grain boundary image to be predicted; The image of the grain boundary to be predicted is input into the automatic closing convolutional neural network model of the missing grain boundary to obtain the image of the closed grain boundary to be predicted. The image of the closed grain boundary to be predicted is processed to obtain a metallographic image with completely closed grain boundaries; The metallographic grain size grade of the steel in the image to be predicted is obtained from the metallographic image of the grain boundary being completely closed.

2. The method for determining the metallographic grain size grade of steel according to claim 1, characterized in that, The process of cropping the metallographic images of each steel material in the original metallographic image set according to a set pixel interval and a set size to obtain the expanded metallographic image set specifically includes: The metallographic images of each steel material in the original metallographic image set are converted into grayscale images to obtain a grayscale image set; Histogram equalization is performed on each image in the grayscale image set to obtain an equalized image set; Each image in the equalized image set is filtered to obtain a filtered image set; Each image in the filtered image set is cropped according to a set pixel interval and a set size to obtain an expanded metallographic image set.

3. The method for determining the metallographic grain size grade of steel according to claim 1, characterized in that, The convolutional neural network includes: The system comprises a first max pooling layer, a second max pooling layer, a first convolutional activation module, a second convolutional activation module, a first copy stacking unit, a second copy stacking unit, a first upsampling convolutional unit, and a second upsampling convolutional unit; the first convolutional activation module includes a first convolutional activation unit and a second convolutional activation unit connected in sequence; the second convolutional activation module includes a third convolutional activation unit and a fourth convolutional activation unit connected in sequence. The input of the first max pooling layer is connected to the input of the first copy stacking unit and the input of the first convolutional activation unit, respectively. The output of the first convolutional activation unit is connected to the input of the second convolutional activation unit. The output of the second convolutional activation unit is connected to the input of the second copy stacking unit and the input of the second max pooling layer, respectively. The output of the second max pooling layer is connected to the input of the first upsampling convolutional unit. The outputs of the first copy stacking unit, the second copy stacking unit, and the first upsampling convolutional unit are all connected to the input of the third convolutional activation unit. The output of the third convolutional activation unit is connected to the input of the fourth convolutional activation unit. The output of the fourth convolutional activation unit is connected to the input of the second upsampling convolutional unit.

4. The method for determining the metallographic grain size grade of steel according to claim 1, characterized in that, The step of performing boundary closure processing on the image of the closed grain boundary to be predicted to obtain a metallographic image with completely closed grain boundaries specifically includes: Adaptive threshold binarization is performed on the image of the closed grain boundary to be predicted to obtain a binary image of the closed grain boundary to be predicted. A morphological opening operation is performed on the binary image of the closed grain boundary to be predicted to obtain a binarized image after the opening operation. The image after the opening operation is obtained by removing regions whose closed hole area is less than a set threshold from the binarized image. The skeleton of the removed image is thinned to obtain a single-pixel grain binarized boundary image. A metallographic image with completely closed grain boundaries is obtained from the binarized grain boundary image of the single pixel.

5. The method for determining the metallographic grain size grade of steel according to claim 1, characterized in that, The metallographic grain size grade of the steel to be predicted is obtained from the metallographic image of the grain boundary with complete closure, specifically including: The metallographic image with completely closed grain boundaries is divided into equal parts and cropped to obtain multiple equally cropped metallographic images; For any metallographic image after equal division and cropping, the metallographic image after equal division and cropping is intersected with the standard measurement grid to obtain the set of intercept points corresponding to the metallographic image after equal division and cropping; the set of intercept points includes the points where the boundary of the metallographic image after equal division and cropping intersects with the standard measurement grid; The length between every two points in the cutoff point set is subjected to extremum removal processing, and the average cutoff length of the metallographic image after equal division and cropping is calculated based on the length between every two points after extremum removal processing; the length between every two points is the length of the boundary of the metallographic image after equal division and cropping corresponding to the two points. Calculate the average of the average cut lengths of all metallographic images after equal cropping; The average cross-sectional length of all equally divided metallographic images is compared with the equiaxed grain micrograin size relationship table to obtain the grain size grade of the metallographic image to be predicted.

6. A system for determining the metallographic grain size grade of steel, characterized in that, include: The metallographic image expansion module is used to acquire an original metallographic image set and crop each metallographic image of steel in the original metallographic image set according to a set pixel interval and a set size to obtain an expanded metallographic image set; the original metallographic image set includes multiple metallographic images of steel samples that have been prepared. The grain boundary drawing module is used to draw the grain boundaries in the metallographic images of each steel in the original metallographic image set to obtain a metallographic grain boundary image set. The tag expansion module is used to crop the metallographic grain boundary images of each steel in the metallographic grain boundary image set according to the expanded metallographic image set to obtain an expanded tag set; the expanded tag set includes the metallographic grain boundary images of each steel in the expanded metallographic image set. The grain boundary segmentation convolutional neural network model determination module is used to train and verify the convolutional neural network with the expanded metallographic image set as input and the expanded label set as output to obtain the grain boundary segmentation convolutional neural network model. The grain boundary erasure module is used to erase the grain boundaries on the metallographic grain boundary images of each steel material in the expanded tag set according to the mask coefficient, so as to obtain the missing tag set; The module for determining the automatic closing convolutional neural network model of missing grain boundaries is used to train the convolutional neural network with the missing label set as input and the expanded label set as output to obtain the automatic closing convolutional neural network model of missing grain boundaries. The image acquisition module is used to acquire the image to be predicted and input the image to be predicted into the grain boundary segmentation convolutional neural network model to obtain the grain boundary image to be predicted. The module for determining the image of the closed grain boundary to be predicted is used to input the image of the grain boundary to be predicted into the automatic closing convolutional neural network model of the missing grain boundary to obtain the image of the closed grain boundary to be predicted. A boundary closure processing module is used to perform boundary closure processing on the image of the grain boundary to be predicted to obtain a metallographic image with completely closed grain boundaries; The steel metallographic grain size determination module is used to obtain the steel metallographic grain size of the image to be predicted based on the metallographic image with completely closed grain boundaries.

7. The steel metallographic grain size determination system according to claim 6, characterized in that, The metallographic image augmentation module specifically includes: The grayscale unit is used to convert each metallographic image of steel in the original metallographic image set into a grayscale image set. An equalization unit is used to perform histogram equalization on each image in the grayscale image set to obtain an equalized image set. A filtering unit is used to filter each image in the equalized image set to obtain a filtered image set; The cropping unit is used to crop each image in the filtered image set according to a set pixel interval and a set size to obtain an expanded metallographic image set.

8. The steel metallographic grain size determination system according to claim 6, characterized in that, The closing processing module specifically includes: The binarization processing unit is used to perform adaptive threshold binarization processing on the image of the closed grain boundary to be predicted, so as to obtain the binary image of the closed grain boundary to be predicted. The morphological opening unit is used to perform a morphological opening operation on the binary image of the closed grain boundary to be predicted to obtain a binarized image after the opening operation. The removal unit is used to remove regions in the binarized image after the opening operation whose closed hole area is less than a set threshold to obtain the removed image. A skeleton refinement unit is used to perform a skeleton refinement operation on the removed image to obtain a single-pixel grain binarized boundary image. The grain boundary fully closed image determination unit is used to obtain a metallographic image with fully closed grain boundaries based on the single-pixel grain binarized boundary image.

9. An electronic device, characterized in that, include: A memory and a processor, the memory being used to store a computer program, the processor running the computer program to cause the electronic device to perform the method for determining the metallographic grain size grade of steel according to any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for determining the metallographic grain size grade of steel as described in any one of claims 1 to 5.

Images