Intelligent crystal grain three-dimensional characterization method based on deep learning

Abstract

The application provides an intelligent grain three-dimensional characterization method based on deep learning, which comprises the following steps: constructing a grain instance segmentation model and an image quality classification model, collecting series of angle grain images, comprehensively evaluating and intelligently screening grain image quality, jointly registering images by using multiple algorithms, and reconstructing grains in three dimensions and analyzing crystallography. The grain instance segmentation model and the image quality classification model are trained by stage data enhancement and iteration; high-quality grain images are intelligently screened based on multi-index comprehensive quantitative evaluation and intelligent image classification auxiliary evaluation; high-precision image registration is completed by using multiple algorithms; and the three-dimensional morphology and grain boundary orientation distribution characteristics of grains are accurately analyzed by combining three-dimensional reconstruction and crystallographic analysis. The method constructs an automatic, integrated and intelligent grain three-dimensional characterization technology system, realizes three-dimensional quantitative characterization of grain morphology and crystallographic characteristic parameters, and provides important technical support for material science related basic scientific problem research.

Description

Technical Field

[0001] This invention relates to the field of materials crystal analysis and characterization technology, specifically to an intelligent three-dimensional grain characterization method based on deep learning. Background Technology

[0002] Grains are crucial structural units in crystalline materials, and their three-dimensional morphology, crystallographic orientation, and interface characteristics significantly influence the microstructure and mechanical properties of these materials. Establishing accurate structure-property relationships requires comprehensive and accurate acquisition of grain three-dimensional spatial information. However, traditional two-dimensional characterization techniques, due to inherent dimensional deficiencies, cannot fully reveal the true morphology and crystallographic parameters of grains in three-dimensional space. To overcome the limitations of two-dimensional grain characterization, researchers have developed a three-dimensional grain characterization technique based on transmission electron microscopy. This technique achieves high-precision three-dimensional reconstruction by continuously tilting the sample rod at large angles under specific crystallographic conditions and acquiring images. However, it suffers from cumbersome procedures, heavy reliance on manual operations, and long overall time consumption.

[0003] In recent years, with the popularization of deep learning-based image recognition technology, researchers have attempted to use deep learning models to identify and extract grains in images, replacing traditional manual extraction and significantly improving data processing efficiency. However, this method inevitably suffers from certain accuracy defects. Low-quality identification results in batches of images require manual screening, and the process still fails to address issues such as rotation and scaling correction in the image registration stage. Specifically, existing schemes using deep learning models for grain identification and extraction only improve the efficiency of the single-stage grain extraction, without overcoming the systemic bottleneck of error propagation throughout the entire process of three-dimensional grain characterization in transmission electron microscopy. This type of method lacks an automated quality control mechanism for identification results; erroneous identification results in scenarios with weak contrast and blurred boundaries directly introduce initial spurious signals, making it impossible to completely eliminate manual intervention. Furthermore, it fails to address the difficulty of using traditional grayscale registration methods to adapt to the drastic contrast fluctuations in tilted images from transmission electron microscopy and the rotation and scaling issues present in some images, further amplifying data defects due to registration errors. The accumulated errors throughout the entire process become concentrated during 3D reconstruction using the weighted back projection algorithm, leading to data distortion and artifacts in the reconstructed volume and directly reducing the accuracy of quantitative analysis of grain microstructure parameters. These issues indicate that optimization of a single step can only improve local performance; there is still significant room for improvement in the entire 3D characterization process. A comprehensive technical system covering the entire process is urgently needed to drive the automation and intelligentization of grain 3D characterization. Summary of the Invention

[0004] The purpose of this invention is to provide an intelligent three-dimensional grain characterization method based on deep learning, which automates and automates several key steps, thereby improving the processing efficiency and analysis accuracy of three-dimensional grain characterization. To achieve the above objective, this invention provides the following technical solution:

[0005] A deep learning-based intelligent three-dimensional grain characterization method includes:

[0006] Acquire grain images and perform grain annotation. Construct an original grain segmentation dataset based on the grain images and corresponding annotation information. Use a multi-stage data augmentation strategy to enhance the original grain segmentation dataset. Iteratively optimize the initial image instance segmentation network in stages based on the grain segmentation datasets before and after enhancement to construct a grain instance segmentation model.

[0007] The grain instance segmentation model is used to segment the grain image and generate a binary image. The binary image is classified and labeled as a high-quality image or a low-quality image according to the image quality. A grain image quality classification dataset is constructed based on the binary image and the corresponding labeling information. An initial image classification network is trained based on the grain image quality classification dataset to construct a grain image quality classification model.

[0008] A series of grain images of the sample at multiple tilt angles were obtained by transmission electron microscopy, a sample coordinate system was constructed, and the initial orientation matrix from the sample coordinate system to the crystal coordinate system was obtained.

[0009] The grain instance segmentation model is used to segment the series of grain images and generate a series of grain binarized images. The quality of the series of grain binarized images is comprehensively quantized and scored using a multi-index scoring rule. The grain image quality classification model is used to obtain the confidence level that the series of grain binarized images are high-quality images. Based on the comprehensive image quantization score and confidence level, high-quality grain binarized images are selected and retained.

[0010] A reference image is selected from a series of high-quality binarized grain images. The scaling factor, rotation angle, and translation parameters of each image in the series of high-quality binarized grain images relative to the reference image are calculated. The scaling factor, rotation angle, and translation parameters are then applied to the series of high-quality binarized grain images to obtain the registered series of grain images.

[0011] The registered series of grain images are reconstructed using a three-dimensional reconstruction algorithm to obtain the grain surface coordinates. Combined with the initial orientation matrix, a three-dimensional quantitative integrated characterization of the grain surface morphology and crystallographic orientation is achieved.

[0012] Optionally, the enhancement of the original grain segmentation dataset using a multi-stage data augmentation strategy includes:

[0013] The original grain segmentation dataset is randomly divided into a training set, a validation set, and a test set according to a preset ratio;

[0014] A two-stage data augmentation strategy is performed on the grain images of the training set. The first stage employs basic geometric and photometric augmentation, including at least one of random translation, random rotation, random scaling, random contrast adjustment, and random saturation adjustment. The second stage employs advanced semantic augmentation, including at least one of image erasure, shearing transformation, and feature enhancement. All augmentations simultaneously transform the annotation information corresponding to the grain images of the training set.

[0015] Optionally, the step of iteratively optimizing the initial image instance segmentation network in stages based on the grain segmentation datasets before and after enhancement to construct a grain instance segmentation model includes:

[0016] The initial image instance segmentation network uses a YOLOv8 network. In the first stage, the initial image instance segmentation network is trained using the original grain segmentation dataset to obtain baseline weights. In the second stage, the image instance segmentation network obtained in the first stage is trained using the grain segmentation dataset after basic geometric and photometric enhancement to improve generalization ability. In the third stage, the image instance segmentation network obtained in the second stage is trained using the grain segmentation dataset after advanced semantic enhancement to strengthen noise and occlusion resistance. During the training process, a unified training configuration and strategy are set, including using the AdamW optimizer, setting a cosine annealing learning rate strategy, setting key hyperparameters, and enabling an early stopping mechanism. GPU is used for accelerated computation. After the entire training process is completed, the grain instance segmentation model is constructed.

[0017] Optionally, training an initial image classification network based on the grain image quality classification dataset to construct a grain image quality classification model includes:

[0018] The initial image classification network uses the YOLOv8-Cls network. The AdamW optimizer is used during training. A cosine annealing learning rate strategy is set, key hyperparameters are defined, and an early stopping mechanism is enabled. GPU is used for accelerated computation. After training, the grain image quality classification model is constructed. The grain image quality classification model receives the input binarized grain image and outputs the confidence that the binarized grain image is a high-quality image.

[0019] Optionally, the step of acquiring a series of grain images of the sample at multiple tilt angles using a transmission electron microscope, constructing a sample coordinate system, and obtaining an initial orientation matrix from the sample coordinate system to the crystal coordinate system includes:

[0020] A sample is mounted in a transmission electron microscope (TEM), and the target grains are imaged using dual-beam imaging to acquire a series of grain images at multiple tilt angles. A sample coordinate system is constructed using the electron beam incident direction of the TEM and the sample tilt axis as coordinate axes. The sample is tilted so that the electron beam incident direction is parallel to the three non-collinear zone axes of the sample, and the orientation matrix in the crystal coordinate system is calculated accordingly. And based on the tilt angles corresponding to the three non-collinear zone axes, the following was calculated: Finally, the initial orientation matrix G from the sample coordinate system to the crystal coordinate system is calculated. .

[0021] Optionally, the step of performing a comprehensive image quantization score on the quality of the series of grain binarized images by combining multi-index scoring rules includes:

[0022] The area ratio score is obtained by calculating the proportion of pixels in the grain region to the total pixels in each binarized image.

[0023] in, This represents the number of pixels in the grain region. This represents the total number of pixels in the image.

[0024] Extract the outer contours of each binarized image and count their number. Combine this with the average contour count of the series of binarized grain images to calculate the contour count score.

[0025]

[0026] in, The number of contours in the current binarized image. The average number of contours in a series of binarized grain images;

[0027] Edge contours are extracted from each binarized image using the Canny operator. A two-dimensional discrete Fourier transform is performed on the edge images, and the sum of the power spectra in the high-frequency regions is calculated as the high-frequency energy. The edge complexity score is obtained through sequence normalization.

[0028]

[0029] in, This represents the sum of the power spectrum in the high-frequency region. This is the total power spectrum;

[0030] The scale-invariant feature transform algorithm is used to extract feature points from adjacent binarized images and perform matching. The random sampling consensus algorithm is used to fit homography features, and the ratio of inliers between the current frame and the previous and next frames is calculated to obtain a sequence similarity score.

[0031]

[0032] in, This represents the ratio of the inlier points of the current frame to those of the previous frame. This represents the ratio of the inlier points of the current frame to the inlier point of the next frame.

[0033] Optionally, the step of filtering based on the comprehensive quantitative score and confidence level of the image to retain high-quality grain binarized images includes:

[0034] Scoring based on area proportion Outline quantity score Marginal complexity score Sequence similarity score With good confidence in the image The final total score is calculated by weighting the scores according to preset weights.

[0035]

[0036] in, These are the weighting coefficients, and ;

[0037] Set threshold Compare the total score of each binarized image. With threshold ,like Then retain the high-quality binarized image, if Then discard the low-quality binarized image.

[0038] Optionally, the step of selecting a reference image from a series of high-quality binarized grain images and calculating the scaling factor of each image in the series of high-quality binarized grain images relative to the reference image includes:

[0039] For the high-quality series of grain binarized images, the Hu invariant moments, the pixel area of ​​the target region, the major axis, minor axis, and principal axis orientation angle of the minimum circumscribed ellipse of the target region are extracted for each frame. The Hu moment similarity is then calculated for adjacent frames. Area similarity and ellipse similarity The three factors are weighted according to preset weights to obtain the overall similarity:

[0040]

[0041] in, These are the weighting coefficients, and ;

[0042] Within a preset search range, a scaling factor that maximizes the overall similarity is obtained through a similarity matching algorithm. Then, using the intermediate frame with a selected tilt angle as a reference image, the scaling factor of each frame relative to the previous frame is calculated, and the scaling factor of each frame relative to the reference image is accumulated.

[0043] Optionally, calculating the rotation angle and translation parameters includes:

[0044] A two-dimensional rigid body transformation including rotation angle and translation is defined. Zero-mean normalized cross-correlation is used as the similarity measure. Iterative optimization is performed by inverse combination Gauss-Newton method to find the rotation angle and translation parameters that maximize the similarity measure. The iteration continues until the parameter increment norm is less than a preset threshold or the maximum number of iterations is reached to obtain the optimal rotation angle and translation parameters for each frame relative to the reference image.

[0045] Optionally, the step of using a three-dimensional reconstruction algorithm to perform three-dimensional reconstruction on the registered series of grain images and obtain grain surface coordinates, combined with the initial orientation matrix, to achieve a three-dimensional quantitative integrated characterization of grain surface morphology and crystallographic orientation, including:

[0046] Based on the registered series of grain images, the three-dimensional volume data is reconstructed by the weighted back projection algorithm, and the target grain structure is extracted by normalization and binarization.

[0047] Isosurface extraction algorithm is used to extract isosurfaces and construct a 3D mesh of grain surface. Independent grain meshes are separated by adjacency graph and depth-first search. After parallel vertex merging simplification, a refined 3D surface mesh of grain is obtained, with the basic unit being a triangular facet.

[0048] In the sample coordinate system, the coordinates of the three vertices of each triangular facet are obtained, and the normal direction of each triangular facet is calculated using the cross product of vectors. Combining the initial orientation matrix, the normal direction is... Transform to the crystal coordinate system to obtain the crystallographic orientation.

[0049] By assigning RGB colors to each triangular facet, the three-dimensional morphology and crystallographic orientation of the target grain can be visualized.

[0050] This invention provides an intelligent 3D grain characterization method based on deep learning, comprising: constructing a grain instance segmentation model and an image quality classification model; acquiring grain images from a series of angles; comprehensive evaluation and intelligent screening of grain image quality; image registration using multiple algorithms; and 3D grain reconstruction and crystallographic analysis. The method involves staged data augmentation and iterative training of the grain instance segmentation model and the image quality classification model; intelligent screening of high-quality grain images based on multi-index comprehensive quantitative evaluation and intelligent image classification-assisted evaluation; high-precision image registration using multiple algorithms; and accurate analysis of grain 3D morphology and grain boundary orientation distribution characteristics achieved through 3D reconstruction and crystallographic analysis. This method constructs an automated, integrated, and intelligent 3D grain characterization technology system, realizing 3D quantitative characterization of grain morphology and crystallographic characteristic parameters. It effectively reduces errors caused by manual intervention and the cumulative error effect throughout the process, while also improving characterization efficiency, providing important technical support for research on fundamental scientific issues related to grains in materials science. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 A flowchart illustrating an intelligent three-dimensional grain characterization method based on deep learning, provided in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram illustrating grain boundary labeling using Labelme software according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram illustrating the training effect of a grain instance segmentation model based on a systematic data augmentation method according to an embodiment of the present invention;

[0055] Figure 4 This is a diagram showing the effect of grain segmentation using a grain instance segmentation model according to an embodiment of the present invention.

[0056] Figure 5 This is a schematic diagram of the grain image acquisition process according to an embodiment of the present invention;

[0057] Figure 6 This is an example of an original image of a grain and a segmented image of a corresponding instance of the present invention;

[0058] Figure 7A comparison of the binarization screening effect of grain images in an embodiment of the present invention: (a) low-quality binarized image; (b) high-quality binarized image;

[0059] Figure 8 The following is a three-dimensional reconstruction result of a grain according to an embodiment of the present invention: (a) voxel data after three-dimensional reconstruction and threshold segmentation; (b) grain surface mesh after coordinate extraction and optimization; (c) magnified display of local mesh on the grain surface;

[0060] Figure 9 This is a crystal orientation distribution diagram on the grain surface of an embodiment of the present invention (observed from the perspective of the sample coordinate system). Detailed Implementation

[0061] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0062] Figure 1 A flowchart illustrating an intelligent three-dimensional grain characterization method based on deep learning, as provided in an embodiment of the present invention, is shown below. Figure 1 As shown, the intelligent three-dimensional characterization method for grains based on deep learning includes the following steps:

[0063] S1: Acquire grain images and perform grain annotation. Construct an original grain segmentation dataset based on the grain images and corresponding annotation information. Use a multi-stage data augmentation strategy to enhance the original grain segmentation dataset. Iteratively optimize the initial image instance segmentation network in stages based on the grain segmentation datasets before and after enhancement, and construct a grain instance segmentation model.

[0064] S1-1: Acquire grain images and perform grain annotation, and construct the original grain segmentation dataset based on the grain images and corresponding annotation information.

[0065] For example, batch grain images are obtained through independent experimental acquisition and literature data collection, such as... Figure 2As shown, the Labelme dataset annotation tool was used to annotate 669 dark-field transmission electron microscope grain images, including but not limited to grain boundary annotation. Polygons were drawn along the grain boundaries, with polygon vertices arranged along the center line of the contrast transition zone between the grain and the background. The spacing between adjacent vertices was ≤20 pixels to ensure boundary fitting accuracy. During grain annotation, independent annotation by two people and cross-checking quality control were employed. The Dice coefficient of 20% of the randomly selected samples was >0.95 to ensure the reliability of the annotation results. After completing the grain image annotation, the annotation files were converted to a standard format compatible with the YOLO series framework using a script. The original grain segmentation dataset was then constructed based on the grain images and corresponding annotation information.

[0066] S1-2: The original grain segmentation dataset is enhanced using a multi-stage data augmentation strategy.

[0067] The entire original grain segmentation dataset was randomly divided into training, validation, and test sets in a ratio of 8:1:1, which were used for model training, hyperparameter tuning, and final independent performance evaluation, respectively.

[0068] A two-stage data augmentation strategy was strictly executed sequentially on the grain images in the training set. The first stage was basic geometric and photometric augmentation, randomly applying translation (±10% image size), rotation (±15°), isotropic scaling (0.85-1.15x), and contrast and saturation adjustment (0.80-1.20x) with a 50% independent probability. Parameters were randomly sampled within a preset range to improve the robustness of the grain instance segmentation model to changes in grain position, scale, and imaging conditions. The second stage was advanced semantic augmentation, selecting up to two combinations from image erasure, affine shearing, and noisy guided filtering edge enhancement with a 40% single-method selection probability to simulate artifacts and local occlusion. All augmentations used bilinear interpolation and simultaneously transformed the corresponding annotation information of the grain images, such as annotation masks, to ensure geometric consistency between the image and the annotations. After the two-stage data augmentation, the number of images in the grain segmentation dataset reached 4000.

[0069] S1-3: Based on the grain segmentation dataset before and after enhancement, the initial image instance segmentation network is iteratively optimized in stages to construct the grain instance segmentation model.

[0070] YOLOv8 was selected as the initial image instance segmentation network for grain recognition and segmentation training, with a three-stage iterative optimization: The first stage used the original grain segmentation dataset without data augmentation to obtain baseline weights; the second stage used a grain segmentation dataset with basic geometric and photometric augmentations to improve generalization ability; the third stage used a grain segmentation dataset with advanced semantic augmentation to train the deep learning neural network obtained in the second stage, enhancing noise and occlusion resistance. A unified configuration and strategy were used throughout the training process, including optimizing the optimizer, setting the learning rate strategy, and defining key hyperparameters. The optimizer used was AdamW, and the learning rate strategy employed a cosine annealing decay mechanism with corresponding weight decay and early stopping mechanisms. The initial learning rate was set to 0.001, the final learning rate decay factor to 0.01, the number of warm-up epochs to 3, the number of training epochs to 100, the batch size to 16, and the weight decay to 0.0005. An early stopping mechanism was also enabled, automatically terminating training when the classification accuracy on the validation set showed no improvement for five consecutive epochs. The training was accelerated using NVIDIA GPUs. Multi-dimensional loss and performance metrics were monitored during training, and evaluation was performed based on an independent test set. Figure 3 The validation set determined the optimal confidence level to be 0.942. The final grain instance segmentation model achieved an average precision of 0.995 and an average intersection-over-union (IoU) exceeding 0.90 on the test set. The segmentation results are shown in [link to test set]. Figure 4 The grain instance segmentation model, which has completed the entire training process, accepts grain images, performs grain segmentation, and outputs binarized grain images.

[0071] S2: The grain instance segmentation model is used to segment the grain image and generate a binary image. The binary image is classified and labeled as a high-quality image or a low-quality image according to the image quality. A grain image quality classification dataset is constructed based on the binary image and the corresponding labeling information. An initial image classification network is trained based on the grain image quality classification dataset to construct a grain image quality classification model.

[0072] S2-1: Specifically, the trained grain instance segmentation model is used to identify 800 newly collected raw transmission electron microscope images of grains, generating corresponding binarized images. Manual binary quality annotations are applied to these 800 binarized images, categorizing them as either high-quality or low-quality images. A grain image quality classification dataset is constructed based on the binarized images and their corresponding annotations, and randomly divided into training, validation, and test sets in an 8:1:1 ratio for model training, hyperparameter tuning, and independent performance evaluation, respectively.

[0073] S2-2: The initial image classification network uses the YOLOv8-Cls network. The classification model is trained based on a grain image quality classification dataset. During training, the same configuration and strategy as the grain instance segmentation model are adopted, including using the AdamW optimizer, setting a cosine annealing learning rate strategy, defining key hyperparameters, enabling early stopping, and utilizing GPUs for accelerated computation. The trained image quality classification model receives the input binarized grain image and outputs the confidence score that the binarized grain image is a high-quality image.

[0074] S3: Obtain a series of grain images of the sample at multiple tilt angles using a transmission electron microscope, construct a sample coordinate system, and obtain the initial orientation matrix from the sample coordinate system to the crystal coordinate system.

[0075] The sample is loaded into a transmission electron microscope (TEM), and the target grains are imaged using dual-beam imaging to obtain a series of grain images at multiple tilt angles. For example... Figure 5 As shown, a sample coordinate system is constructed using the electron beam incident direction of the transmission electron microscope and the sample tilt axis as coordinate axes. Tilting the sample causes the electron beam incident direction to be aligned with the three non-collinear zone axes of the sample. , , Parallelism allows for the calculation of the orientation matrix in the crystal coordinate system. :

[0076]

[0077] The asterisk (*) indicates normalization.

[0078] In addition, the zone axis of the crystal The corresponding tilt angle is , The corresponding tilt matrix is zone axis The corresponding tilt angle is , The corresponding tilt matrix is zone axis The corresponding tilt angle is , The corresponding tilt matrix is .

[0079]

[0080]

[0081] Finally, the initial orientation matrix from the sample coordinate system to the crystal coordinate system is:

[0082]

[0083] in, The initial orientation matrix, Let T be the orientation matrix in the crystal coordinate system, and T be the three-axis tilt matrix.

[0084] In a specific embodiment of the present invention, an ultrafine-grained Al-Cu alloy sample (average grain size 400 nm) is mounted on a high-tilt uniaxial sample holder, a target grain is selected, a series of tilt images of the grain are obtained under dual-beam imaging conditions, and the initial crystallographic orientation of the sample is determined. The specific steps include:

[0085] like Figure 5 As shown, under a transmission electron microscope, the diffraction vector... A series of tilt-angle dark-field images of the target grain were performed under dual-beam conditions, with tilt angles ranging from -72° to +72° and a step size of 1°. A total of 145 two-dimensional dark-field images of the grain were obtained. Images at some angles are shown below. Figure 6 As shown.

[0086] The sample was tilted to three non-collinear zone axes, when... , When, the corresponding zone axis index is ;when , When, the corresponding zone axis index is ;when , When, the corresponding zone axis index is The initial orientation matrix from the sample coordinate system to the crystal coordinate system is calculated according to formula (4). .

[0087] S4: The grain instance segmentation model is used to segment the series of grain images and generate a series of grain binarized images. The quality of the series of grain binarized images is comprehensively quantized and scored using a multi-index scoring rule. The grain image quality classification model is used to obtain the confidence level that the series of grain binarized images are high-quality images. Based on the comprehensive image quantization score and confidence level, high-quality grain binarized images are selected and retained.

[0088] By conducting comprehensive evaluation and intelligent screening of grain image quality, the following steps are completed in sequence: image quality quantification scoring, inter-frame sequence continuity verification, basic comprehensive score fusion, image quality classification model-assisted judgment, and quality grading and automatic removal. This filters out low-quality images with segmentation anomalies, boundary distortions, and sequence discontinuities, ensuring the reliability and consistency of subsequent 3D reconstruction input data.

[0089] S4-1: Using the grain instance segmentation model trained in step S1, perform grain segmentation on the 145 grain images obtained at various tilt angles, automatically identify and segment the grains, and generate corresponding binarized images. Binarized images for some angles are shown below. Figure 6 As shown.

[0090] S4-2: For each binarized image in the series of grain binarized images, calculate the area percentage score. Outline quantity score Marginal complexity score and sequence similarity score The image good confidence score of the grain image quality classification model trained in step S2 is combined with the image good confidence score of the model output. The overall image quality score is calculated by combining the results. Specifically, it includes:

[0091] (1) Calculate the proportion of pixels in the grain region to the total pixels in each binarized image to obtain the area ratio score:

[0092]

[0093] in, This represents the number of pixels in the grain region. This represents the total number of pixels in the image.

[0094] (2) Extract the outer contour of each binarized image and count its number. Combine the average contour count of the series of grain binarized images to obtain the contour count score:

[0095]

[0096] in, The number of contours in the current binarized image. This represents the average number of contours in a series of binarized grain images.

[0097] (3) Use the Canny operator to extract edge contours from each binarized image, perform a two-dimensional discrete Fourier transform on the edge images, calculate the sum of the power spectra of the high-frequency regions as the high-frequency energy, and obtain the edge complexity score through sequence normalization:

[0098]

[0099] in, This represents the sum of the power spectrum in the high-frequency region. This is the total power spectrum.

[0100] (4) The scale-invariant feature transformation (SIFT) algorithm is used to extract feature points of adjacent binarized images and match them. The homography features are fitted using the random sampling consensus algorithm. The ratio of inliers between the current frame and the previous and next frames is calculated to obtain the sequence similarity score.

[0101]

[0102] in, This represents the ratio of the inlier points of the current frame to those of the previous frame. This represents the ratio of the inlier points of the current frame to the inlier point of the next frame.

[0103] (5) Load the grain image quality classification model, input the binarized image into the model to obtain the confidence score that it belongs to the good image category (high-quality image). The area ratio will be scored. Outline quantity score Marginal complexity score Sequence similarity score With good confidence in the image The final total score is calculated by weighting the scores according to preset weights.

[0104]

[0105] in, To obtain the weighting coefficients based on experimental experience, and .

[0106] Set threshold Compare the total score of each binarized image. With threshold ,like Then retain the image, if Then remove the image.

[0107] Based on the validation set prepared in S1-2, binarized images were obtained through the grain instance segmentation model, and the total score distribution was statistically analyzed. Combined with the qualitative effect evaluation of the 3D reconstruction experiment, and through precision-recall tradeoff analysis, the optimal balance point was selected to ensure the retention rate of high-quality binarized images while maximizing the elimination of low-quality binarized images. Finally, the quality threshold was determined. .like Figure 7 As shown, if the total image score is greater than or equal to the threshold T, it is determined to be a high-quality binarized image, which is retained and sent to the subsequent multi-algorithm joint image registration process; if the total image score is less than the threshold T, it is determined to be a low-quality binarized image and is automatically removed.

[0108] S5: Select a reference image in the high-quality series of binarized grain images, calculate the scaling factor, rotation angle and translation parameters of each image in the high-quality series of binarized grain images relative to the reference image, and apply the scaling factor, rotation angle and translation parameters to the high-quality series of binarized grain images to obtain the registered series of grain images.

[0109] Image registration using a multi-algorithm approach was conducted. For a series of high-quality, binarized grain images selected from a pool, the Hu invariant moments, target region pixel area, and major, minor, and principal axis orientation angles of the minimum bounding ellipse of the target region were extracted for each frame using the built-in Matlab function `regionprops`. Subsequently, the Hu moment similarity between adjacent frames was calculated. Area similarity and ellipse similarity The similarity calculations were all implemented using built-in Matlab functions. Finally, the three factors were weighted according to preset weights to obtain the comprehensive similarity score.

[0110]

[0111] in, These are the weighting coefficients, and .

[0112] In a specific embodiment of the present invention, the three factors are weighted according to preset weights to obtain a comprehensive similarity: Within a preset search range (0.95-1.05), a scaling factor that maximizes the overall similarity is obtained using a similarity matching algorithm (Brent algorithm). Then, using an intermediate frame with a tilt angle of 0° as a reference frame (reference image), the scaling factor of each frame relative to the previous frame is calculated, and the scaling factor of each frame relative to the reference frame is accumulated. A two-dimensional rigid body transformation including rotation angle and translation is defined, and zero-mean normalized cross-correlation is used as the similarity measure. Iterative optimization is performed using an inverse combined Gauss-Newton method to find the rotation angle and translation parameters that maximize the similarity measure. This iteration continues until the parameter increment norm is less than a preset threshold or the maximum number of iterations is reached, obtaining the optimal rotation angle and translation parameters of each frame relative to the reference frame. The scaling factor, rotation angle, and translation parameters obtained in the above steps are applied to a series of grain binarized images to generate a series of registered grain images.

[0113] S6: The registered series of grain images are reconstructed using a three-dimensional reconstruction algorithm to obtain the grain surface coordinates. Combined with the initial orientation matrix, a three-dimensional quantitative integrated characterization of the grain surface morphology and crystallographic orientation is achieved.

[0114] We carried out three-dimensional reconstruction of the grains and extraction and optimization of three-dimensional coordinates of the grain surface. We successively completed weighted back projection reconstruction, voxel data threshold segmentation, grain three-dimensional coordinate extraction and coordinate optimization operations. The registered two-dimensional tilt series images were transformed into high-precision three-dimensional volume data of the grains containing accurate spatial coordinates, and the spatial morphology of the grains was completely restored.

[0115] S6-1: A weighted back projection algorithm is used to reconstruct the 3D structure of the grains. A mathematical model is constructed based on the projection equation, and a parallel acceleration framework is designed through task decomposition to improve reconstruction efficiency. After normalizing the original volume data in the [0,1] interval, an adaptive threshold is determined based on the intensity histogram using the method of maximizing inter-class variance to complete the binarization of the volume data. The voxel rendering effect after processing is as follows: Figure 8 As shown in a.

[0116] S6-2: An isosurface extraction algorithm is used to obtain the surface morphology of the grains. After setting an isosurface threshold, the state encoding of the binary voxels is completed. Combined with a pre-calculated lookup table and a triangulation table, triangular patches are generated by linear interpolation to construct a three-dimensional mesh of the grain surface. The surface mesh is further converted into an adjacency graph. Connected components are separated by depth-first search, and mesh reconstruction and reindexing are completed. A parallel simplified framework is designed based on independent components. Mesh optimization is completed by vertex merging, deduplication, and removal of redundant elements. Figure 8 b) Finally, a finely refined three-dimensional surface mesh containing approximately 12,000 triangular facets is obtained, with its basic unit being triangular facets. A magnified view of the local area is shown below. Figure 8 As shown in c.

[0117] S6-3: Conduct three-dimensional crystallographic feature analysis of grains to achieve three-dimensional quantitative integrated characterization of grain surface (grain boundary) morphology and crystallographic orientation, including:

[0118] Extract the spatial coordinates of the three vertices of each triangular facet in the optimized mesh, and calculate the normal vector of each facet in the sample coordinate system using the cross product. :

[0119]

[0120] In the sample coordinate system, the coordinates of the three vertices of any triangle are denoted as follows: , ;

[0121] Then, the initial orientation matrix from the sample coordinate system to the crystal coordinate system is used. Direction of normal Transform to the crystal coordinate system to obtain the crystallographic orientation. :

[0122]

[0123] in, Let be the initial orientation matrix from the sample coordinate system to the crystal coordinate system. Let be the normalized normal direction of the triangular plane in the crystal coordinate system.

[0124] Finally, the inverse pole figure color coding method is used to determine the direction of the crystal plane normal. Mapped to the RGB color space.

[0125] For example, the normal vector of a triangular facet in the sample coordinate system is... After conversion, we get Corresponding crystal plane index Based on the symmetry of the cubic crystal system, all crystal plane indices are transformed to those of the cubic crystal system. , , The defined equivalent orientation space is satisfied. Equivalent orientation ,Will , , The three base orientations are assigned RGB values ​​respectively. , , A basis matrix is ​​constructed, and the color weight vector is solved using linear interpolation. This completes the RGB color assignment for each triangular facet, ultimately enabling visualization of the three-dimensional morphology and crystallographic orientation of the target grain. The interface colors can intuitively reflect the crystallographic orientation distribution in different regions of the grain surface, such as... Figure 9 As shown.

Claims

1. A deep learning-based intelligent three-dimensional grain characterization method, characterized in that, include: Acquire grain images and perform grain annotation. Construct an original grain segmentation dataset based on the grain images and corresponding annotation information. Use a multi-stage data augmentation strategy to enhance the original grain segmentation dataset. Iteratively optimize the initial image instance segmentation network in stages based on the grain segmentation datasets before and after enhancement to construct a grain instance segmentation model. The grain instance segmentation model is used to segment the grain image and generate a binary image. The binary image is classified and labeled as a high-quality image or a low-quality image according to the image quality. A grain image quality classification dataset is constructed based on the binary image and the corresponding labeling information. An initial image classification network is trained based on the grain image quality classification dataset to construct a grain image quality classification model. A series of grain images of the sample at multiple tilt angles were obtained by transmission electron microscopy, a sample coordinate system was constructed, and the initial orientation matrix from the sample coordinate system to the crystal coordinate system was obtained. The grain instance segmentation model is used to segment the series of grain images and generate a series of grain binarized images. The quality of the series of grain binarized images is comprehensively quantized and scored using a multi-index scoring rule. The grain image quality classification model is used to obtain the confidence level that the series of grain binarized images are high-quality images. Based on the comprehensive image quantization score and confidence level, high-quality grain binarized images are selected and retained. A reference image is selected from a series of high-quality binarized grain images. The scaling factor, rotation angle, and translation parameters of each image in the series of high-quality binarized grain images relative to the reference image are calculated. The scaling factor, rotation angle, and translation parameters are then applied to the series of high-quality binarized grain images to obtain the registered series of grain images. The registered series of grain images are reconstructed using a three-dimensional reconstruction algorithm to obtain the grain surface coordinates. Combined with the initial orientation matrix, a three-dimensional quantitative integrated characterization of the grain surface morphology and crystallographic orientation is achieved.

2. The intelligent three-dimensional grain characterization method based on deep learning according to claim 1, characterized in that, The multi-stage data augmentation strategy employed to augment the original grain segmentation dataset includes: The original grain segmentation dataset is randomly divided into a training set, a validation set, and a test set according to a preset ratio; A two-stage data augmentation strategy is performed on the grain images of the training set. The first stage employs basic geometric and photometric augmentation, including at least one of random translation, random rotation, random scaling, random contrast adjustment, and random saturation adjustment. The second stage employs advanced semantic augmentation, including at least one of image erasure, shearing transformation, and feature enhancement. All augmentations simultaneously transform the annotation information corresponding to the grain images of the training set.

3. The intelligent three-dimensional grain characterization method based on deep learning according to claim 1 or 2, characterized in that, The initial image instance segmentation network is iteratively optimized in stages based on the pre- and post-enhancement grain segmentation datasets to construct a grain instance segmentation model, including: The initial image instance segmentation network uses a YOLOv8 network. In the first stage, the initial image instance segmentation network is trained using the original grain segmentation dataset to obtain baseline weights. In the second stage, the image instance segmentation network obtained in the first stage is trained using the grain segmentation dataset after basic geometric and photometric enhancement to improve generalization ability. In the third stage, the image instance segmentation network obtained in the second stage is trained using the grain segmentation dataset after advanced semantic enhancement to strengthen noise and occlusion resistance. During the training process, a unified training configuration and strategy are set, including using the AdamW optimizer, setting a cosine annealing learning rate strategy, setting key hyperparameters, and enabling an early stopping mechanism. GPU is used for accelerated computation. After the entire training process is completed, the grain instance segmentation model is constructed.

4. The intelligent three-dimensional grain characterization method based on deep learning according to claim 1 or 2, characterized in that, The step of training an initial image classification network based on the grain image quality classification dataset and constructing a grain image quality classification model includes: The initial image classification network uses the YOLOv8-Cls network. The AdamW optimizer is used during training. A cosine annealing learning rate strategy is set, key hyperparameters are defined, and an early stopping mechanism is enabled. GPU is used for accelerated computation. After training, the grain image quality classification model is constructed. The grain image quality classification model receives the input binarized grain image and outputs the confidence that the binarized grain image is a high-quality image.

5. The intelligent three-dimensional grain characterization method based on deep learning according to claim 1 or 2, characterized in that, The process of acquiring a series of grain images of the sample at multiple tilt angles using a transmission electron microscope, constructing a sample coordinate system, and obtaining an initial orientation matrix from the sample coordinate system to the crystal coordinate system includes: A sample is mounted in a transmission electron microscope (TEM), and the target grains are imaged using dual-beam imaging to acquire a series of grain images at multiple tilt angles. A sample coordinate system is constructed using the electron beam incident direction of the TEM and the sample tilt axis as coordinate axes. The sample is tilted so that the electron beam incident direction is parallel to the three non-collinear zone axes of the sample, and the orientation matrix in the crystal coordinate system is calculated accordingly. And based on the tilt angles corresponding to the three non-collinear zone axes, the following was calculated: Finally, the initial orientation matrix G from the sample coordinate system to the crystal coordinate system is calculated. .

6. The intelligent three-dimensional grain characterization method based on deep learning according to claim 1 or 2, characterized in that, The method of combining multi-index scoring rules to perform comprehensive image quantification scoring on the quality of the series of grain binarized images includes: The area ratio score is obtained by calculating the proportion of pixels in the grain region to the total pixels in each binarized image. in, This represents the number of pixels in the grain region. This represents the total number of pixels in the image. Extract the outer contours of each binarized image and count their number. Combine this with the average contour count of the series of binarized grain images to calculate the contour count score. in, The number of contours in the current binarized image. The average number of contours in a series of binarized grain images; Edge contours are extracted from each binarized image using the Canny operator. A two-dimensional discrete Fourier transform is performed on the edge images, and the sum of the power spectra in the high-frequency regions is calculated as the high-frequency energy. The edge complexity score is obtained through sequence normalization. in, This represents the sum of the power spectrum in the high-frequency region. This is the total power spectrum; The scale-invariant feature transform algorithm is used to extract feature points from adjacent binarized images and perform matching. The random sampling consensus algorithm is used to fit homography features, and the ratio of inliers between the current frame and the previous and next frames is calculated to obtain a sequence similarity score. in, This represents the ratio of the inlier points of the current frame to those of the previous frame. This represents the ratio of the inlier points of the current frame to the inlier point of the next frame.

7. The intelligent three-dimensional grain characterization method based on deep learning according to claim 1 or 2, characterized in that, The process of filtering based on the comprehensive quantitative score and confidence level of the image to retain high-quality grain binarized images includes: Scoring based on area proportion Outline quantity score Marginal complexity score Sequence similarity score With good confidence in the image The final total score is calculated by weighting the scores according to preset weights. in, These are the weighting coefficients, and ; Set threshold Compare the total score of each binarized image. With threshold ,like Then retain the high-quality binarized image, if Then discard the low-quality binarized image.

8. The intelligent three-dimensional grain characterization method based on deep learning according to claim 1 or 2, characterized in that, The selected high-quality series of binarized grain images includes a reference image. The scaling factor of each image in the high-quality series of binarized grain images relative to the reference image is calculated, including: For the high-quality series of grain binarized images, the Hu invariant moments, the pixel area of ​​the target region, the major axis, minor axis, and principal axis orientation angle of the minimum circumscribed ellipse of the target region are extracted for each frame. The Hu moment similarity is then calculated for adjacent frames. Area similarity and ellipse similarity The three factors are weighted according to preset weights to obtain the overall similarity: in, These are the weighting coefficients, and ; Within a preset search range, a scaling factor that maximizes the overall similarity is obtained through a similarity matching algorithm. Then, using the intermediate frame with a selected tilt angle as a reference image, the scaling factor of each frame relative to the previous frame is calculated, and the scaling factor of each frame relative to the reference image is accumulated.

9. The intelligent three-dimensional grain characterization method based on deep learning according to claim 1 or 2, characterized in that, Calculating the rotation angle and translation parameters includes: A two-dimensional rigid body transformation including rotation angle and translation is defined. Zero-mean normalized cross-correlation is used as the similarity measure. Iterative optimization is performed by inverse combination Gauss-Newton method to find the rotation angle and translation parameters that maximize the similarity measure. The iteration continues until the parameter increment norm is less than a preset threshold or the maximum number of iterations is reached to obtain the optimal rotation angle and translation parameters for each frame relative to the reference image.

10. The intelligent three-dimensional grain characterization method based on deep learning according to claim 1 or 2, characterized in that, The process involves using a 3D reconstruction algorithm to reconstruct the registered series of grain images in 3D, obtaining grain surface coordinates, and combining this with the initial orientation matrix to achieve a 3D quantitative integrated characterization of grain surface morphology and crystallographic orientation, including: Based on the registered series of grain images, the three-dimensional volume data is reconstructed by the weighted back projection algorithm, and the target grain structure is extracted by normalization and binarization. Isosurface extraction algorithm is used to extract isosurfaces and construct a 3D mesh of grain surface. Independent grain meshes are separated by adjacency graph and depth-first search. After parallel vertex merging simplification, a refined 3D surface mesh of grain is obtained, with the basic unit being a triangular facet. In the sample coordinate system, the coordinates of the three vertices of each triangular facet are obtained, and the normal direction of each triangular facet is calculated using the cross product of vectors. Combining the initial orientation matrix, the normal direction is... Transform to the crystal coordinate system to obtain the crystallographic orientation. By assigning RGB colors to each triangular facet, the three-dimensional morphology and crystallographic orientation of the target grain can be visualized.

Images