A deep learning-based in-situ microscopic image characterization method for nanometer thin film materials

Abstract

The application discloses a kind of in-situ microscopic image characterization methods of nanometer thin film material based on deep learning, comprising: real-time acquisition of in-situ microscopic platform under the growth video data of nanometer thin film material and synchronous picture correction processing;Using swin-TRANSEM neural network loads pre-training weight, the corrected growth video image is identified, and identification frame is generated;According to the size of identification frame, the early growth frame position of crystal domain is determined, and the center position information of crystal domain growth is obtained;Using improved image segmentation algorithm, the edge information of single crystal domain discretization is extracted;Combined with the center position information of crystal domain growth and the edge information of crystal domain discretization, the gradient of each edge representative point in the direction of coordinate axis is calculated, and the growth speed information in each direction is obtained by taking module;Draw growth heat map, and establish growth model.The application can significantly improve the efficiency and accuracy of nanometer thin film material growth analysis.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of nanometer thin film material growth tracking, and particularly relates to a nanometer thin film material in-situ microscopic image characterization method based on deep learning. BACKGROUND

[0002] Nanometer thin film materials have attracted extensive attention due to their unique size effects and interface characteristics. Such materials, such as indium tin oxide (ITO), zinc sulfide (ZnS) thin films, etc., exhibit excellent electrical, optical and mechanical properties. For example, ITO thin films have important applications in touch screens and optoelectronic devices due to their high transparency and good conductivity; while ZnS thin films perform well in optical coatings and display technology due to their excellent optical properties and good chemical stability. As the research on nanometer thin film materials gradually deepens, understanding their growth mechanism and interface characteristics becomes particularly important for optimizing their performance and expanding their application fields.

[0003] However, traditional microscopic techniques face many challenges in monitoring the growth process of these materials in-situ. For example, the resolution limit and image distortion problems of microscopes make it difficult to accurately capture complex growth morphologies. Although existing microscopic techniques can provide certain structural information, in the face of microstructure changes of nanometer thin film materials under different growth conditions, traditional methods often appear insufficient in positioning accuracy and data analysis. Therefore, developing an advanced method that can accurately and real-time characterize the growth process of nanometer thin film materials is crucial for optimizing material performance and understanding their growth mechanism. SUMMARY

[0004] The technical problem solved by the present application is that the resolution of optical microscope imaging is low and the reaction temperature causes different degrees of field distortion and distortion. The present application provides a nanometer thin film material in-situ microscopic image characterization method based on deep learning, which uses optical microscope to observe the growth of nanometer thin film materials in real time, and uses swin-TRANSEM algorithm based on deep learning to make statistical analysis to help guide the controllable growth of nanometer thin film materials, which can significantly improve the efficiency and accuracy of nanometer thin film material growth analysis.

[0005] Technical solution

[0006] A nanometer thin film material in-situ microscopic image characterization method based on deep learning, the nanometer thin film material in-situ microscopic image characterization method based on deep learning comprises the following steps:

[0007] S1, real-time acquisition of growth video data of nanometer thin film materials under in-situ microscopic platform and synchronous picture correction processing;

[0008] S2, using swin-TRANSEM neural network to load pre-trained weights, identifying the corrected growth video image to generate an identification frame;

[0009] S3, determining the early growth frame position of the crystal domain according to the size of the identification frame and obtaining the center position information of the crystal domain growth;

[0010] S4, using an improved image segmentation algorithm, discretizing the edge information and discretizing the growth center angle into N points to extract the discretized edge information of a single crystal domain;

[0011] S5, combining the center position information of the crystal domain growth and the discretized edge information of the crystal domain, calculating the gradient of each edge representative point in the coordinate axis direction and taking the modulus to obtain the growth speed information in each direction;

[0012] S6, drawing a growth heat map and establishing a growth model.

[0013] Step S1 further comprises:

[0014] Introducing inert impurities in the center part of the reaction substrate and reducing them to be stably placed in the reaction furnace;

[0015] Fine-tuning the lens to make the inert impurities as the correction reference in the non-edge of the field of view, and marking the area O worth noting during correction;

[0016] Warming up the reaction system and starting real-time image recording, and binarizing the picture;

[0017] In the binarized real-time image, edge detection is performed on the area O to find multiple connected regions, and the multiple connected regions are sorted from large to small according to the area; the center coordinates Oti and the area Sti of each connected region Bi at time t during the growth process are recorded, and (Sti+1-Sti) / Sti>5% is used as the judgment condition to filter out part of the connected regions, and the average position of the center points of the remaining connected regions is used to calculate the lens drift matrix Mdrift.

[0018] Further, in step S2, 2, 2, 6 and 2 Swin-Transformer modules are used in layers to replace the convolution layers of the original YOLOv8 backbone module for feature extraction, and shape-IoU is introduced as a regression loss function to optimize the detection head, and a swin-TRANSEM neural network is constructed, which includes a backbone module, a feature pyramid and a detection head.

[0019] Further, the backbone module splits the input shape [H, W, 3] image, i.e. a three-channel RGB image with height H and width W, into non-overlapping sub-images with height and width both being P, shape [N0, P, P, 3], The token sequence is flattened into [N0, P2*3].

[0020] The token sequence is projected into a new token tensor with shape [N1, C] by a linear layer, N1=N0, and is input into a Swin-Transformer module for twice feature extraction, and the output shape remains unchanged, completing the first level feature extraction.

[0021] A new sub-image is formed by merging adjacent 2*2 sub-images, and the token tensor becomes [N2, 2C],

[0022] The token tensor becomes [N3, 4C] and [N4, 8C] through 6 times and 2 times feature extraction of the Swin-Transformer module, respectively, N0, N2, N3 and N4 are the number of sub-images at each stage.

[0023] The detection head introduces shape-IoU as a regression loss function, and optimizes the detection accuracy of early-stage small domains by considering the influence of the shape and angle of the bounding box itself on the detection accuracy, and the specific calculation method is:

[0024] L shape-IoU =1-IoU+distance shape +0.5*Omega shape

[0025] Wherein, theta=4,x c and y c are the center coordinates of the predicted box, and the subscript represents the center point, and are the center coordinates of the real box; hh and ww are the weights for adjusting the width and height difference; omega t measures the shape difference of width or height, and theta is the adjustment parameter of the shape difference; IoU is the intersection over union, i.e. the ratio of the intersection area to the union area between the detection box and the real box (IoU=S 重叠部分 / S 合并面积 ).

[0026] Step S4 further comprises the following steps:

[0027] For a single domain, track n frames of its growth, and select m points with a relative center angle interval of theta * as representative points, theta *θm=360±2°, get the position matrix A of all edge representative points; wherein θ * ∝(ρ×α), ρ=total number of frame pixels / average maximum number of possible domains, α is the accuracy requirement.

[0028] Step S5 further includes the following steps:

[0029] The gradient of the point in each direction transformed in the x-axis and y-axis is calculated using the second-order difference method, and the specific calculation formula is:

[0030]

[0031] For the boundary, the second-order one-sided difference method is adopted:

[0032]

[0033] f is x or y, used to calculate the gradient of the two axes, t represents time, and the subscript represents the time scale (starting from 0).

[0034] The modulus of the gradient in the x and y directions at each angle is calculated to obtain a growth rate matrix V of the shape [n, m-1], and the value of the ith row and jth column represents the growth rate of the selected ith frame of the domain at the (j×θ * ) angle;

[0035] According to the specific situation of material growth and data acquisition equipment, filtering operation is performed in time dimension.

[0036] Step S6 further includes the following steps:

[0037] Draw a pixelated 2D growth heat map, draw a continuous 3D growth heat map after linear interpolation, and establish a growth model.

[0038] Further, the process of drawing a pixelated 2D growth heat map includes the following steps:

[0039] Reconstruct the coordinate system with the growth center as the origin and grid the entire frame with the origin as the center of a square, the grid length is L, and the position matrix A is converted to a new position matrix A';

[0040] Traverse all points in the position matrix A' and determine their corresponding grid, and distribute the data in the position matrix V to the corresponding grid, and take the average value of the speed value in each grid as the representative value of the grid;

[0041] Draw a color bar according to the distribution of the representative value, and fill the voids with box filtering.

[0042] Further, the process of drawing a continuous 3D growth heat map includes the following steps:

[0043] Select some frame positions in n frames, use their time series as the Z-axis, and relocate the edge feature points in the XY plane according to the information in the position matrix A. Smooth the discretized edge information to restore it to continuous edges. At the same time, the velocity matrix V is also linearly interpolated.

[0044] A 3D interactive growth rate heatmap is drawn based on the correspondence between edge position, edge growth rate, and color bar.

[0045] Beneficial effects:

[0046] First, the in-situ microscopic image characterization method for nanofilm materials based on deep learning of the present invention can accurately capture the growth kinetics information of nanofilm materials, which is convenient for combining microscopic molecular theory to make a better explanation of the material growth mechanism. At the same time, this method can solve the problem that traditional methods may misjudge edges and have difficulty in handling drift errors.

[0047] Secondly, the in-situ microscopic image characterization method for nanofilm materials based on deep learning in this invention not only helps optimize the growth conditions of nanofilm materials but also enables a deeper understanding of their growth mechanisms and their relationship with physicochemical properties. Furthermore, advancements in computer vision technology have made high-throughput data processing and large-scale sample analysis possible, providing a new approach for rapidly screening optimal growth conditions and customizing nanofilm materials. Attached Figure Description

[0048] Figure 1 This is a flowchart of the in-situ microscopic image characterization method for nanofilm materials based on deep learning according to the present invention.

[0049] Figure 2 This is a diagram of the swin-TRANSEM algorithm architecture in this invention;

[0050] Figure 3 This is a schematic diagram of a smoothed edge image and a 2-D heatmap of edge pixelation drawn according to an embodiment of the present invention;

[0051] Figure 4 This is a schematic diagram of the continuity of a 3-D thermal image drawn from different viewing angles in an embodiment of the present invention. Detailed Implementation

[0052] The following embodiments are provided to enable those skilled in the art to more fully understand the present invention, but do not limit the invention in any way.

[0053] This invention discloses an in-situ microscopic image characterization method for nanofilm materials based on deep learning. Taking MoS2 as an example, the in-situ microscopic image characterization method for nanofilm materials includes the following steps:

[0054] S1: Real-time acquisition of MoS2 growth video data on the in-situ microscopy platform and simultaneous image correction processing;

[0055] S2, using the swin-TRANSEM neural network to load pre-trained weights, recognizes the corrected grown video image and generates a recognition box;

[0056] S3, determine the early growth frame position of the crystal domain according to the size of the recognition box and obtain the center position information of the crystal domain growth;

[0057] S4. Using an improved image segmentation algorithm, the edge information is discretized into N points according to the angle of the growth center, and the edge information of a single crystal domain is extracted.

[0058] S5. Combining the center position information of the crystal domain growth and the edge information of the crystal domain discretization, the gradient of each edge representative point in the coordinate axis direction is calculated and the modulus is taken to obtain the growth rate information in each direction.

[0059] S6. Draw a growth heatmap and establish a growth model.

[0060] This invention, based on an improved image segmentation and detection algorithm and integrating machine vision technology, achieves higher throughput and accuracy in the automatic localization and analysis of MoS2 domain growth dynamics. The method first collects MoS2 growth video data using an in-situ optical microscopy imaging platform; then, it addresses camera drift errors during filming using an impurity point-assisted correction method; finally, it uses the swin-TRANSEM algorithm to locate the domains, obtain their center points, and perform quantity statistics; next, it uses the improved U-net algorithm to acquire MoS2 domain shape (edge) information; and finally, it integrates the center point coordinates and edge information to create a morphologically reconstructed MoS2 growth rate analysis heatmap.

[0061] See Figure 1 The in-situ microscopic image characterization method for MoS2 specifically includes the following steps:

[0062] S1. Place the reaction substrate containing calibration impurities into the reaction system, and obtain the growth effect in real time through a microscope to correct lens shake. The specific steps are as follows:

[0063] S11. Inert impurities are introduced into the central part of the reaction substrate and then placed stably inside the reactor.

[0064] S12. Fine-tuning the lens ensures that inert impurities, which can also serve as a calibration reference, are not located at the edge of the field of view to prevent them from exceeding the field of view due to shake, and delineates a noteworthy area O during calibration (considering that the size and distribution of calibration impurities are relatively random, delineating a noteworthy area can eliminate a large amount of interference and thus improve accuracy).

[0065] S13. Begin heating the reaction system and start real-time video recording, simultaneously binarizing the image with a threshold of 125. In the binarized real-time image, perform edge detection on noteworthy areas to find existing connected regions B1, B2… and sort them by area from largest to smallest. Record the center coordinates Oti and area Sti of region Bi at time t during the growth process. Based on the criterion (Sti+1-Sti) / Sti>5%, some non-compliant connected regions are no longer considered. Finally, calculate the lens offset matrix M from the average position of the center points of the remaining compliant connected regions. drift .

[0066] S2. Each frame is fed into a deep learning network based on the swin-TRANSEM neural network, which has been loaded with pre-trained weights, in real time for recognition.

[0067] See Figure 2 The Swin-TRANSEM neural network, based on the YOLO architecture, mainly consists of three parts: the backbone, the feature pyramid, and the detector head.

[0068] The backbone uses the Swin-Transformer module instead of traditional convolution for feature extraction and improves efficiency through a window attention mechanism. First, the input RGB image ([H,W,3]) is split into several non-overlapping sub-images of size P×P. The token sequence is flattened into a [N0, P2×3] sequence. A linear layer projects this sequence onto a new token tensor of shape [N1, C] (N1 = N0). This tensor is then input into a Swin-Transformer module for two feature extractions, with the output shape remaining unchanged, completing the first level of feature extraction. In the second stage, adjacent 2×2 subgraphs are merged to form a new subgraph, and the token tensor becomes [N2, 2C]. In the third and fourth stages, feature extraction is performed by the Swin-Transformer module 6 times and 2 times respectively, resulting in token tensors of [N3, 4C] and [N4, 8C]. Where N0, N2, N3, and N4 represent the number of sub-images in each stage. These four stages achieve hierarchical extraction of image features while maintaining computational efficiency.

[0069] The feature pyramid part adopts the YOLOv8 design, using modules such as C2f and convolution to complete the feature hierarchy;

[0070] Considering that the initial crystal domains are relatively small compared to the image size, shape-IoU is introduced as the regression loss function in the detection head. The detection accuracy is optimized for the early stage when the crystal domains are small by taking into account the influence of the bounding box's shape and angle on the detection accuracy. The specific calculation method is as follows: L shape-Io =1-IoU+distance shape +0.5×Ω shape ,in θ = 4, x c and y c These are the center coordinates of the prediction box. and These are the center coordinates of the actual bounding box. hh and ww are weights used to adjust the difference between width and height; θ=4, ω t The shape difference in width or height is measured by θ, which is the adjustment parameter for the shape difference.

[0071] S3. Store the position of the first detected bounding box (if the overlap between detected bounding boxes is less than 5%, it is considered a new detected bounding box). Coordinate center matrix M center (M center =M0+M drift M0 is the center of the directly obtained detection frame and the detection frame size information matrix M size Matrix M center M size The data corresponds to the corresponding dimension and the corresponding position.

[0072] S4, via M center M size Each detection box is located, and a pre-trained U-net segmentation network is used to segment each detection box and obtain the edge information of the crystal domains.

[0073] Specifically, a segmentation algorithm is used to obtain the edge information of the crystal domains in the recognition box, and the edge information is discretized by combining it with the corresponding center coordinates. For a single crystal domain, n frames are tracked for its growth, and m points (θ×m=360±2°) with a relative center angle interval of θ are selected as representative points. Finally, the position matrix A of all edge representative points is obtained, where θ∝(ρ×α) (ρ=total number of pixels in the frame / average maximum number of possible crystal domains, α is the accuracy requirement).

[0074] S5. Examine the growth performance of a single domain in n frames (in this example, n = 59, the starting frame is frame 990, the ending frame is frame 1860, and the interval is 15 frames). Traverse all frames, starting from M... centerThe center point coordinates are obtained from S4. The m (m=720) directions of the crystal domain in the i-th frame are obtained with a selection angle interval of θ (θ=0.5°). The edge coordinates Ai corresponding to each direction are stored and corrected to obtain Ai' (Ai'=Ai+Mdrift). Thus, the position matrix A is obtained (A=[A0,A1,..,A58], with shape [59,720,2]).

[0075] The gradient of a point along the x and y axes in each direction is calculated using the second-order finite difference method. The specific calculation formula is as follows:

[0076]

[0077] For the boundary, the second-order one-sided difference method is used:

[0078] f = x, y

[0079] Subsequently, the magnitude of the gradient in the x and y directions at each angle is calculated to obtain a growth rate matrix V of shape [n, m-1]. The value in the i-th row and j-th column represents the growth rate of the crystal domain at the (j×θ) angle at the selected i-th frame. Then, depending on the specific material growth and data acquisition equipment, a certain degree of filtering operation can be performed along the time dimension to eliminate or mitigate the interference of some outliers. In the example of this invention, the shape of V is [58, 720], and Gaussian filtering is applied to the second dimension of matrix V.

[0080] S6. There are two ways to draw the growth heatmap, as follows:

[0081] 1. Divide the image frame into a pixel grid with a side length of 2.2, and reconstruct the coordinate system by selecting any one of the grids as the center. Using this origin as the growth center, calculate the grid corresponding to the edge points in E and assign its corresponding value in the growth rate matrix V to the corresponding pixel. If multiple points belong to the same pixel, take the average value as the representative value of that grid, and draw a color bar according to the representative value. For example, when n>4 and m>180, A' 13 and A' 23 Within the same grid, then the grid's Color bars are drawn based on the distribution of representative values. Some holes are filled using box filtering. The final result is a 2D pixelated heatmap, as shown below. Figure 3 As shown.

[0082] 2. Select a subset of frames from n frames, using their time series as the Z-axis. Relocate edge feature points in the XY plane according to the information in position matrix A. Smooth the discretized edge information to restore continuous edges. Simultaneously, perform linear interpolation on the velocity matrix V. Draw a 3D interactive growth rate heatmap based on the correspondence between edge position, edge growth rate, and color bar. Allow interactive observation of the crystal domain growth from different angles; the observation direction can be manually changed to obtain more time-related growth information. Specifically, as shown... Figure 4 As shown.

[0083] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A deep learning-based in-situ microscopic image characterization method for nanofilm materials, characterized in that, The in-situ microscopic image characterization method for nanofilm materials includes the following steps: S1, real-time acquisition of video data of nano-thin film material growth under in-situ microscopy platform and simultaneous image correction processing; S2, using 2, 2, 6 and 2 Swin-Transformer modules to replace the convolutional layers of the original YOLOv8 backbone module for feature extraction, and introducing shape-IoU as a regression loss function to optimize the detection head, thus constructing a swin-TRANSEM neural network, which includes a backbone module, a feature pyramid and a detection head; The swin-TRANSEM neural network is used to load pre-trained weights to recognize the corrected grown video images and generate recognition boxes. S3, determine the early growth frame position of the crystal domain according to the size of the recognition box and obtain the center position information of the crystal domain growth; S4. Using an improved image segmentation algorithm, the edge information is discretized into N points according to the angle of the growth center, and the edge information of a single crystal domain is extracted. S5. Combining the center position information of the crystal domain growth and the edge information of the crystal domain discretization, the gradient of each edge representative point in the coordinate axis direction is calculated and the modulus is taken to obtain the growth rate information in each direction. S6, Draw a growth heatmap and establish a growth model; Step S4 further includes the following steps: For a single crystal domain, n1 frames are tracked during its growth, and those with a relative center angle interval of [missing information] are selected. m1 points are used as representative points. ×m1=360±2°, thus obtaining the position matrix A of all edge representative points; where ∝(ρ×α), ρ=total number of pixels in the frame / average maximum number of crystal domains, α is the accuracy requirement; Step S5 further includes the following steps: The gradient of a point along each direction on the x-axis and y-axis is calculated using the second-order finite difference method. The specific calculation formula is as follows: ； For the boundary, the second-order one-sided difference method is used: Calculating the magnitude of the gradient in the x and y directions at each angle yields a growth rate matrix V of shape [n, m-1]. The value in the i-th row and j-th column represents the value of the selected domain in the j×j-th frame. Growth rate at an angle; f is x or y, used to calculate the two axial gradients; t represents time, and the subscripts all represent time scales. Filtering is performed according to the time dimension, based on the specific conditions of material growth and data acquisition equipment.

2. The in-situ microscopic image characterization method for nanofilm materials based on deep learning according to claim 1, characterized in that, Step S1 further includes: Inert impurities are introduced into the central portion of the reaction substrate and allowed to settle and stabilize within the reactor. The fine-tuning lens places inert impurities, which serve as a correction reference, at the non-edge of the field of view and delineates the area O that is worth noting during correction; The reaction system is heated and real-time video recording is started, and the image is binarized. In the binarized real-time image, edge detection is performed on region O to find multiple connected regions. These connected regions are then sorted by area from largest to smallest. During the growth process, the center coordinates Oti and area Sti of each connected region Bi at time t are recorded. Some connected regions are filtered out using (Sti+1-Sti) / Sti > 5% as a criterion. The lens offset matrix Mdrift is calculated based on the average position of the center points of the remaining connected regions that meet the criteria.

3. The in-situ microscopic image characterization method for nanofilm materials based on deep learning according to claim 1, characterized in that, The backbone module splits the input three-channel RGB image with height H and width W into several non-overlapping sub-images with height and width P and shape [N0, P, P, 3]. Flatten it into a token sequence of [N0, P²×3]; The token sequence is projected onto a new token tensor of shape [N1, C] through a linear layer, where N1=N0. The input to the Swin-Transformer module is used for two feature extractions, and the output shape remains unchanged, thus completing the first level of feature extraction. By merging adjacent 2×2 subgraphs to form a new subgraph, the token tensor becomes [N2, 2C]. ; After feature extraction by the Swin-Transformer module in 6 and 2 iterations respectively, the token tensor becomes [N3, 4C] and [N4, 8C]. , N0, N2, N3, and N4 represent the number of subgraphs in each stage.

4. The in-situ microscopic image characterization method for nanofilm materials based on deep learning according to claim 1, characterized in that, The detection head introduces shape-IoU as a regression loss function, optimizing the detection accuracy for early-stage crystal domains by considering the influence of the bounding box's shape and angle on the detection accuracy. The specific calculation method is as follows: ； in, , , and These are the center coordinates of the prediction box; the subscript indicates that they are the coordinates of the center point. and These are the center coordinates of the actual bounding box; hh and ww are weights used to adjust the difference between width and height. The shape difference in width or height is measured by θ, which is the adjustment parameter for the shape difference; IoU is the intersection-union ratio, which is the ratio of the intersection area between the detection box and the ground truth box to the union area. A pixelated 2D growth heatmap is drawn, and a continuous 3D growth heatmap is drawn after linear interpolation to establish a growth model.

5. The in-situ microscopic image characterization method for nanofilm materials based on deep learning according to claim 4, characterized in that, The process of creating a pixelated 2D growth heatmap includes the following steps: Reconstruct the coordinate system with the growth center as the origin and grid the entire image frame with the origin as the center of a square. The grid side length is L. At the same time, transform the position matrix A to the new coordinate system to obtain the new position matrix A'. Traverse all points in the position matrix A' and determine the grid to which it belongs. At the same time, assign the data in the velocity matrix V to the corresponding grid and take the average of the velocity values ​​in each grid as the representative value of that grid. Color bars are drawn based on the distribution of representative values, and holes are filled using box filtering.

6. The in-situ microscopic image characterization method for nanofilm materials based on deep learning according to claim 4, characterized in that, The process of creating a continuous 3D growth heatmap includes the following steps: Select some frame positions in n frames, use their time series as the Z-axis, and relocate the edge feature points in the XY plane according to the information in the position matrix A. Smooth the discretized edge information to restore it to continuous edges. At the same time, the velocity matrix V is also linearly interpolated. A 3D interactive growth rate heatmap is drawn based on the correspondence between edge position, edge growth rate, and color bar.

Images