Method and device for aligning serial slice images of scanning transmission electron microscopy, and electronic device

Abstract

The application discloses a scanning transmission electron microscope continuous slice image alignment method and device and electronic equipment, and gradually eliminates the offset error in the slice preparation and imaging process through the alignment from the initial coordinate system to the grid unit level alignment matrix calculation, effectively improving the alignment accuracy between the continuous slices. Moreover, the spliced image after alignment can not only be used for defect detection and output of defect prompt information, but also can be used for targeted replacement by using the simulation image for the target spliced image with defects, effectively solving the problems of complex and low-efficiency traditional manual processing and the inability to repair defects, and providing reliable image data support for the subsequent image segmentation, three-dimensional reconstruction, tissue cell structure analysis and other researches of the continuous slice image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a scanning transmission electron microscope continuous slice image alignment method and device and electronic equipment. BACKGROUND

[0002] Scanning transmission electron microscope can clearly image ultra-fine structures such as cells and tissues with nanoscale resolution, and provide high-quality cell tomographic images for biological process research. However, during sample preparation, the sample may be offset, wrinkled, contaminated, broken or missing, resulting in poor alignment of continuous slice images. Moreover, manual screening of the same information area is mainly used, which is not only complex and inefficient, but also cannot repair the defect area, resulting in low image content utilization and serious loss of interlayer continuous information. SUMMARY

[0003] The present application provides a scanning transmission electron microscope continuous slice image alignment method, device and electronic equipment to at least solve the above technical problems in the prior art.

[0004] In one aspect, the present application provides a scanning transmission electron microscope continuous slice image alignment method, which comprises:

[0005] aligning the coordinate systems of a plurality of continuous slice images collected by a scanning transmission electron microscope;

[0006] block collecting a region of interest of the slice image after the coordinate system alignment, obtaining a plurality of sub-images of the slice image, and splicing the plurality of sub-images to obtain a spliced image of the region of interest of the slice image;

[0007] preprocessing the spliced image, grid processing the preprocessed spliced image, selecting a reference spliced image from all spliced images, determining the alignment matrix of each corresponding grid unit of the reference spliced image and each to-be-aligned spliced image, and aligning the to-be-aligned spliced image and the reference spliced image according to the alignment matrix;

[0008] detecting defects in each aligned spliced image, generating a corresponding simulation image for a target spliced image with defects, determining the similarity between the simulation image and the target spliced image, returning a corresponding defect prompt if the similarity is lower than a threshold, returning a corresponding defect prompt if the similarity is higher than or equal to the threshold, and extracting image content at the same position in the simulation image based on defect position information therein to replace image content at the defect position in the target spliced image.

[0009] In an embodiment, the aligning the coordinate systems of the plurality of continuous slice images collected by the scanning transmission electron microscope comprises:

[0010] determining a minimum bounding rectangle of the slice image, and determining a coordinate system of the slice image according to the minimum bounding rectangle;

[0011] selecting a reference slice image from the plurality of continuous slice images, each slice image other than the reference slice image being a slice image to be aligned, determining at least two key regions in the reference slice image, and determining a corresponding position of the key region in a coordinate system of the slice image to be aligned as a region to be matched according to a position of the key region in the coordinate system of the reference slice image;

[0012] obtaining an image corresponding to the key region in the reference slice image as a template image, and obtaining an image corresponding to the region to be matched in the slice image to be aligned as a matching image;

[0013] obtaining a central region image of the template image as a sub-template image, matching the sub-template image and the matching image based on a normalized correlation coefficient, and obtaining an offset of the slice image to be aligned in a horizontal axis direction and a vertical axis direction of the coordinate system and a relative rotation angle of the slice image to be aligned compared with the reference slice image when the matching is successful;

[0014] moving the coordinate system of the slice image to be aligned according to the offset and the relative rotation angle, so as to align the coordinate system of the reference slice image and the coordinate system of the slice image to be aligned.

[0015] In an embodiment, the aligning the coordinate systems of the plurality of continuous slice images collected by the scanning transmission electron microscope comprises:

[0016] obtaining an offset between two sub-images to be spliced, moving the sub-images according to the offset, and splicing the two sub-images.

[0017] In an embodiment, the determining, for each slice image to be aligned, an alignment matrix of each pair of corresponding grid cells of the slice image to be aligned and a reference spliced image comprises:

[0018] for each grid cell to be aligned of the slice image to be aligned, determining a corresponding reference grid cell of the reference spliced image, to obtain a plurality of grid cell pairs;

[0019] For each grid cell pair, determine the feature point pair similarity between each feature point of the grid cell to be aligned in the grid cell pair and each feature point of the reference grid cell, select a target feature point pair that meets the similarity condition, and obtain a target feature point pair set corresponding to the grid cell pair; and determine the alignment matrix of the grid cell to be aligned relative to the reference grid cell in the grid cell pair according to the coordinates of the feature points in the target feature point pair set.

[0020] In an implementation manner, the aligning the stitching image to be aligned and the reference stitching image according to the alignment matrix comprises:

[0021] For each coordinate point of the stitching image to be aligned, the coordinates of the coordinate point are corrected according to the alignment matrix corresponding to the grid cell pair to which the coordinate point belongs, so that the stitching image to be aligned and the reference stitching image are aligned.

[0022] In an implementation manner, the defect detection on each aligned stitching image, and the generation of a corresponding simulation image for a target stitching image with defects comprises:

[0023] The first deep learning model is used to detect defects in each aligned stitching image, and identification information, a defect type, and a defect position of a target stitching image with defects are obtained;

[0024] The second deep learning model is used to generate a simulation image of the target stitching image.

[0025] In an implementation manner, the second deep learning model is a GAN;

[0026] The second deep learning model is used to generate a simulation image of the target stitching image, which comprises:

[0027] For the target stitching image, a simulation image of the target stitching image is generated according to a previous stitching image and a next stitching image.

[0028] In an implementation manner, the determination of the similarity between the simulation image and the target stitching image comprises:

[0029] A pixel difference index between the simulation image and the target stitching image is determined, and the pixel difference index represents the degree of gray difference between the simulation image and the target stitching image at each position;

[0030] A content similarity index between the simulation image and the target stitching image is determined, and the content similarity index represents the degree of similarity between the simulation image and the target stitching image in image content;

[0031] Determine the similarity of the simulation image and the target splicing image according to the pixel difference index and the content similarity index.

[0032] Another aspect of the present application provides a scanning transmission electron microscope continuous slice image alignment device, comprising:

[0033] A coordinate system alignment module is configured to perform coordinate system alignment on a plurality of continuous slice images collected by a scanning transmission electron microscope.

[0034] A coarse alignment module is configured to perform block collection on a region of interest of the slice image after coordinate system alignment, to obtain a plurality of sub-images of the slice image; and perform splicing on the plurality of sub-images, to obtain a splicing image of the region of interest of the slice image.

[0035] A fine alignment module is configured to perform preprocessing on the splicing image, perform meshing processing on the preprocessed splicing image, select a reference splicing image from all splicing images of slice images, determine an alignment matrix of each corresponding mesh unit of the reference splicing image for each to-be-aligned splicing image, and align the to-be-aligned splicing image and the reference splicing image according to the alignment matrix.

[0036] An image repair module is configured to perform defect detection on each aligned splicing image, generate a corresponding simulation image for a target splicing image with defects, determine the similarity of the simulation image and the target splicing image, return a corresponding defect prompt information if the similarity is lower than a threshold value, return a corresponding defect prompt information if the similarity is higher than or equal to the threshold value, and extract image content at the same position in the simulation image based on defect position information therein, to replace image content at the defect position in the target splicing image.

[0037] Another aspect of the present application provides an electronic device, comprising:

[0038] A processor;

[0039] A memory for storing executable instructions of the processor;

[0040] The processor is configured to read the executable instructions from the memory and execute the instructions to implement the scanning transmission electron microscope continuous slice image alignment method of the present application.

[0041] The alignment method of the continuous slice image of the scanning transmission electron microscope of the present disclosure gradually eliminates the offset error in the slice preparation and imaging process from the initial coordinate system alignment to the grid cell level alignment matrix calculation, effectively improving the alignment accuracy between the continuous slices. Moreover, for the aligned spliced image, not only can it be used for defect detection and output of defect prompt information, but also can be used for targeted replacement using simulation images for the detected target spliced image with defects, effectively solving the problems of complex and inefficient traditional manual processing and inability to repair defects, providing reliable image data support for subsequent image segmentation, three-dimensional reconstruction, tissue cell structure analysis and other researches of continuous slice images.

[0042] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 The implementation flowchart of the alignment method of the continuous slice image of the scanning transmission electron microscope of the present embodiment of the present disclosure is shown;

[0044] Figure 2 The structure diagram of the multiple slice images in the alignment method of the continuous slice image of the scanning transmission electron microscope of the present embodiment of the present disclosure is shown;

[0045] Figure 3 The structure diagram of the slice defect image in the alignment method of the continuous slice image of the scanning transmission electron microscope of the present embodiment of the present disclosure is shown;

[0046] Figure 4 The initial coordinate system diagram in the alignment method of the continuous slice image of the scanning transmission electron microscope of the present embodiment of the present disclosure is shown;

[0047] Figure 5 The schematic diagram of the spliced image to be aligned and the reference spliced image in the alignment method of the continuous slice image of the scanning transmission electron microscope of the present embodiment of the present disclosure is shown;

[0048] Figure 6 The module diagram of the alignment device of the continuous slice image of the scanning transmission electron microscope of the present embodiment of the present disclosure is shown;

[0049] Figure 7 The composition structure diagram of an electronic device is shown. DETAILED DESCRIPTION

[0050] In order to make the purposes, characteristics and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0051] The method for aligning the continuous slice images of the scanning transmission electron microscope provided by the embodiments of the present application is as shown in Figure 1 The method comprises the following steps.

[0052] S101, coordinate system alignment is performed on a plurality of continuous slice images collected by a scanning transmission electron microscope.

[0053] In this step, the slice image is an image after slicing of ultra-microstructure such as cells, tissues, viruses or biological macromolecules. Due to the above-mentioned biological sample in the preparation process, the processes such as sampling, block repairing, slicing and staining will all cause the sample to deviate from the expected reference position, so that the plurality of continuous slice images collected by the scanning transmission electron microscope have poor alignment, which affects the subsequent image information utilization rate. In addition, when the scanning transmission electron microscope (STEM) is shooting, the structure of the motion platform will have nonlinear movement error, and the axial gap is easy to have movement deviation during reciprocating motion, so that the alignment effect of the continuous slice images shot by the STEM is poor. Therefore, the plurality of continuous slice images are subjected to coordinate alignment in this step, which provides a basis for the subsequent utilization of image information.

[0054] As shown in Figure 2 , the method comprises the following steps. Figure 2Five slice images are shown: slice image 100, slice image 200, slice image 300, slice image 400, and slice image 500, each with its own initial coordinate system. Since STEM imaging achieves nanometer-level precision, the initial coordinate system exhibits significant deviation at this precision. Therefore, this step selects one slice image as the reference slice image and aligns the initial coordinate systems of the remaining slice images with those of the reference slice image. For example, if slice image 100 is the reference slice image, at least two key regions are selected from it. Based on normalized correlation coefficient module matching, template matching is sequentially performed on slice images 200, 300, 400, and 500 to calculate the offset and relative rotation angle of each subsequent slice image relative to the reference slice image along the horizontal and vertical axes of the coordinate system, thus achieving coordinate system alignment between consecutive slice images. It should be noted that the selected key areas should have unique characteristics, such as specific shapes, textures, or other identifiable attributes, so that they can be accurately identified during the template matching process.

[0055] S102. The region of interest of the sliced ​​image after coordinate system alignment is segmented and acquired to obtain multiple sub-images of the sliced ​​image. The multiple sub-images are then stitched together to obtain a stitched image of the region of interest of the sliced ​​image.

[0056] In this step, for the coordinate-aligned slice images, a Region of Interest (ROI) is determined within the reference slice image. The ROI is a specific region defined in the reference slice image. For example, in a STEM slice of a material, certain crystal structure regions are beneficial for analyzing material properties, and these corresponding crystal structure regions can be designated as ROIs. By mapping the coordinates of the ROI in the reference slice image to the aligned coordinate system of the remaining slice images, the location of the ROI in each of the remaining slice images is obtained.

[0057] The region of interest of each slice image is block collected to obtain a plurality of sub-images of the slice image, and the plurality of sub-images of each slice image are spliced using a phase correlation method to obtain a spliced image (coarse alignment image) of the region of interest of the slice image. It should be noted that the division of the region of interest of each slice image into a plurality of small blocks is usually based on the spatial position of the pixels, so that a larger image region can be divided into relatively smaller and easier-to-process sub-images. Further, when splicing using the phase correlation method, the relative positions between the sub-images are found more accurately based on the frequency domain characteristics of each sub-image, and high-precision image splicing is realized.

[0058] S103, pre-processing the spliced image, grid processing the pre-processed spliced image, selecting a reference spliced image from all spliced images of the slice images, determining, for each remaining to-be-aligned spliced image, an alignment matrix of each pair of corresponding grid units of the to-be-aligned spliced image and the reference spliced image, and aligning the to-be-aligned spliced image and the reference spliced image according to the alignment matrix;

[0059] In this step, all spliced images are pre-processed, and the sizes of all spliced images are uniformly processed in sequence. The contrast of all pre-processed spliced images is equalized, and the image noise of all pre-processed spliced images is reduced to obtain the pre-processed spliced image. It should be noted that the size of the reference spliced image can be used as a standard to adjust the size of the remaining spliced images to unify the size of all spliced images. Further, the thickness of different slices may be different when the slice images are taken, and the slice thickness is the dominant factor that determines the brightness of the pixels. Different slice thicknesses make it difficult for image pixels to reflect the true structural characteristics, so the contrast of the spliced image of each slice image needs to be adjusted. The contrast of all spliced images can be adjusted using a contrast equalization algorithm to keep the contrast of all spliced images uniform after the size is unified. The contrast equalization algorithm can be selected according to actual application requirements, which is not limited here. Further, the noise pollution of all pre-processed spliced images also needs to be reduced to improve the image quality of the spliced image.

[0060] After obtaining the preprocessed stitching image, grid processing is performed on the preprocessed stitching image, and each preprocessed stitching image is divided into a plurality of grid units. In addition, a reference stitching image needs to be selected from all preprocessed stitching images, and the remaining is a to-be-aligned stitching image. For each to-be-aligned stitching image, an alignment matrix of a grid unit pair composed of each grid unit of the to-be-aligned stitching image and the corresponding grid unit of the reference stitching image is determined, and the to-be-aligned image is aligned with the reference stitching image based on the alignment matrix, further improving the alignment degree of the continuous slice image and ensuring the utilization rate of subsequent image information.

[0061] S104, defect detection is performed on each aligned stitching image, a corresponding simulation image is generated for a target stitching image with defects, the similarity between the simulation image and the target stitching image is determined, if the similarity is lower than a threshold, a corresponding defect prompt information is returned, if the similarity is higher than or equal to the threshold, a corresponding defect prompt information is returned, and based on the defect position information therein, image content at the same position in the simulation image is extracted to replace image content at the defect position in the target stitching image.

[0062] In this step, for each aligned stitching image (fine alignment image), a trained deep learning model is used for defect detection, and the detected defect-free image is used as a fine alignment image as the image data basis for subsequent analysis of cell structure, and the detected stitching image with defects is used as a target stitching image, which needs to be further processed. For example, as shown in the figure, the black area 301 in the figure is a slice contamination area, indicating that the slice has defects, and the target stitching image corresponding to the defect image needs to be processed. Figure 3

[0063] ​For the target splicing image, it is necessary to determine the defect degree of the target splicing image, and the defect content on the target splicing image with small defect degree needs to be replaced, and the corresponding defect prompt information is returned, and for the target splicing image with large defect degree (serious slice damage), the defect prompt information is directly returned without replacement. Among them, by determining the similarity between the simulation image corresponding to the target splicing image and the target splicing image, and comparing the obtained similarity with the pre-set threshold, the defect degree of the target splicing image is determined. If the similarity is less than the threshold, it means that the defect degree of the target splicing image is large, that is, the slice is seriously damaged, and the serial number, defect type and defect position of the target splicing image are directly returned as the defect prompt information without replacement; if the similarity is higher than or equal to the threshold, it means that the defect degree of the target splicing image is small, in addition to returning the above defect prompt information, the corresponding image content is also extracted from the same position of the simulation image according to the information related to the defect position in the defect prompt information, and the simulation image content is used to replace the defect content at the same position of the target splicing image, so as to repair the defect of the target splicing image and improve the image quality of the target splicing image.

[0064] The alignment method for the continuous slice images of the scanning transmission electron microscope provided by the embodiments of the present disclosure can gradually eliminate the offset errors in the slice preparation and imaging process by calculating the alignment matrix from the initial coordinate system to the grid unit level for a plurality of continuous slice images, thereby effectively improving the alignment accuracy between the continuous slices. Moreover, not only can the defect detection and defect prompt information output be performed for the splicing image after alignment, but also the simulation image can be used for targeted replacement for the target splicing image with detected defects, thereby effectively solving the problems of complex and inefficient manual processing and inability to repair defects, and providing reliable image data support for subsequent image segmentation, three-dimensional reconstruction, and tissue cell structure analysis of the continuous slice images.

[0065] In an implementable manner, the coordinate system alignment is performed on a plurality of continuous slice images collected by a scanning transmission electron microscope, including:

[0066] determining the minimum circumscribed rectangle of the slice image, and determining the coordinate system of the slice image according to the minimum circumscribed rectangle;

[0067] selecting a reference slice image from the plurality of continuous slice images, and each remaining slice image being a slice image to be aligned, determining at least two key regions in the reference slice image, and determining the corresponding positions of the key regions in the coordinate system of the slice image to be aligned as the matching regions according to the positions of the key regions in the coordinate system of the reference slice image;

[0068] The image corresponding to the key region in the reference slice image is obtained as a template image, and the image corresponding to the to-be-aligned region in the to-be-aligned slice image is obtained as a to-be-matched image;

[0069] The center region image of the template image is obtained as a sub-template image, the sub-template image and the to-be-matched image are matched based on a normalized correlation coefficient, and when the matching is successful, the offset of the to-be-aligned slice image compared with the reference slice image in the horizontal axis direction and the vertical axis direction of the coordinate system and the relative rotation angle are obtained;

[0070] The coordinate system of the to-be-aligned slice image is moved according to the offset and the relative rotation angle, so that the coordinate system of the reference slice image and the coordinate system of the to-be-aligned slice image are aligned.

[0071] In this embodiment, first, the initial coordinate system corresponding to each slice image needs to be determined, as shown in Figure 4 Since the slice is trapezoidal, for the slice image, a minimum circumscribed rectangle needs to be generated according to the slice endpoints, and the point at the top left corner of the rectangle is set as the origin to establish the initial coordinate system.

[0072] The establishment of the initial coordinate system of the slice image will be described in detail below:

[0073] S1011, the outermost contour point set of the slice is set as The point set calculated after the convex hull is ;

[0074] S1012, the vector of each edge of the convex hull is calculated, and each edge on the convex hull is composed of adjacent points ; , ;

[0075] S1013, the direction vector is calculated and normalized to a unit vector , , and the normal vector ;

[0076] S1014, project all points of the convex hull to the , direction to calculate the projection coordinates, and the coordinates of the point projected to , are respectively: , ;

[0077] S1015, extreme value calculation, the width , height and area corresponding to the edge are obtained;

[0078] S1016, select the minimum area, calculate the minimum area, height, width, direction vector and point coordinates corresponding to the edge of all convex hulls, and obtain the minimum area after comparison and the corresponding , and the four point coordinates of the rectangle , , , ;

[0079] S1017, specify the point of the upper left corner as the coordinate origin, the direction from point to , the direction from point to , in the initial coordinate system, , , , the expression of any point , in the coordinate system is , .

[0080] That is, according to the outermost contour point set of each slice, the area, length, width and four vertex coordinates of the minimum circumscribed rectangle are obtained through convex hull calculation, wherein the minimum circumscribed rectangle calculated according to the shape of the slice is specified, the bottom edge of the trapezoid pointing up is the initial coordinate system origin of a single slice, and the four vertex coordinates are determined clockwise as , , , , , , , , the coordinate origin , points to , is the direction, the coordinate origin , points to , is the direction, and the initial coordinate system is determined in this way.

[0081] The first slice image obtains an initial coordinate system according to steps S1011-S1017, which is set as coordinate system A. Similarly, for the second slice image, the minimum bounding rectangle is generated in the same way as the first slice image, and an initial coordinate system is established with its upper left corner as the origin, set as coordinate system B. And so on, the initial coordinate system of the third slice image is set as coordinate system C, the initial coordinate system of the fourth slice image is set as coordinate system D, and the initial coordinate system of the fifth slice image is set as coordinate system E.

[0082] Because STEM imaging achieves nanometer-level precision, each initial coordinate system may exhibit significant deviations at this precision. For example, coordinate systems A and B are difficult to directly use to uniformly describe the positional relationship between the first and second slice images. Therefore, this embodiment selects one of the first and second slice images as the reference slice image. If the first slice image is the reference slice image, the corresponding second slice image is the slice image to be aligned. At least two key regions are selected from the reference slice image, and the positions of the key regions in coordinate system B are determined based on their positions in coordinate system A, serving as the matching regions. The image corresponding to the key regions of the STEM-captured reference slice image is the template image, and the image of the matching region of the same size as the template image, captured by the STEM, is the image to be matched. After determining the template image and the image to be matched, the central region image of the template image is selected as the sub-template image. The sub-template image and the image to be matched are matched using a template matching method based on a normalized correlation coefficient. The formula for the normalized correlation coefficient is as follows:

[0083]

[0084] In the formula, The sub-template image is in position pixel values, It is the average pixel value of the sub-template image. The image to be matched is in pixel values, It is the position of the top-left corner of the sub-template image in the image to be matched during the current NCC calculation. It is the image to be matched that contains This represents the average pixel value of the top-left sub-region, which is the same size as the sub-template image. Iterate through all pixel positions of the sub-template image.

[0085] The normalized correlation coefficient NCC determined by the above formula is used for template matching, and when the sub-template image and the image to be matched are successfully matched, the offset of the second slice image relative to the first slice image in the x-axis and y-axis directions and the relative rotation angle are obtained. The x-axis offset represents the horizontal position difference of the coordinate system B of the second slice image relative to the coordinate system A of the first slice image, the y-axis offset represents the vertical position difference of the coordinate system B of the second slice image relative to the coordinate system A of the first slice image, and the relative rotation angle represents the angular misalignment of the coordinate system B of the second slice image relative to the coordinate system A of the first slice image in the rotation direction. The coordinate system B of the second slice image is translated and / or rotated according to the calculated offset and relative rotation angle, so that the coordinate system B of the second slice image is aligned with the coordinate system A of the first slice image (reference slice image) in position, providing a unified spatial reference for subsequent steps such as splicing and defect detection.

[0086] For example, the second slice image moves 5 units to the right in the horizontal direction (x-axis direction) and 3 units downward in the vertical direction (y-axis direction) compared with the first slice image, and the x-axis offset is 3 and the y-axis offset is 5. The unit here can be a pixel, i.e. the second slice image moves 5 pixels in the horizontal direction and 3 pixels in the vertical direction. Correspondingly, if it moves to the left, the x-axis offset is negative, and if it moves upward, the y-axis offset is negative, for example, if it moves 2 pixels to the left and 1 pixel upward, the x-axis offset is -2 and the y-axis offset is -1.

[0087] In addition, when determining the relative rotation angle of the second slice image relative to the first slice image, a point a1(xa1,ya1) and a point a2(xa2,ya2) are determined in the two key regions of the first slice image respectively, the corresponding point of the point a1 in the second slice image is b1(xb1,yb1), and the corresponding point of the point a2 in the second slice image is b2(xb2,yb2). The line segment of the point a1 and the point a2 is La12, and the angle between the line segment La12 and the x-axis is The line segment of the points b1 and b2 is Lb12, and the angle between the line segment Lb12 and the x-axis is Therefore, the relative rotation angle of the second slice image relative to the first slice image is: If is positive, the second slice image rotates counterclockwise by the corresponding angle with the point b1 as the center of rotation, and if If the value is negative, the second slice image is rotated clockwise by the corresponding angle with b1 as the rotation center, so that the second slice image is aligned with the first slice image in the coordinate system. That is, the coordinate systems of the two slice images can be described in a unified coordinate system, facilitating further analysis, measurement and data processing of the two slice images, and avoiding errors and inconvenience caused by inconsistent coordinate systems. The alignment methods of the remaining slice images, i.e., the third slice image, the fourth slice image and the fifth slice image, are the same as that of the second slice image, and the repeated parts are not described again.

[0088] It should be noted that if the sub-template image and the to-be-matched image fail to match, the eight-connected region of the to-be-matched image is collected, a 3x3 image is obtained, and image stitching is performed to obtain a new to-be-matched image. The new to-be-matched image is matched again using the above template matching method. If the matching is successful, the corresponding offset and relative rotation angle are obtained. If the matching fails, the matching process is ended.

[0089] In an implementation, the plurality of sub-images are stitched, including:

[0090] An offset between the two sub-images to be stitched is obtained, and the sub-images are moved according to the offset to stitch the two sub-images.

[0091] In this embodiment, for the slice images aligned in the coordinate system, the region of interest in the reference slice image is determined, and the region of interest in the remaining slice images is determined based on the region of interest in the reference slice image. The STEM automatically captures images at the ROI positions of all continuous slices including the reference slice image. The ROI of each slice image is block collected to obtain MxN sub-images. Taking the second slice image as an example, the ROI of the second slice image is evenly divided into M parts in the horizontal direction and N parts in the vertical direction to obtain MxN sub-images. Similarly, the ROI of other slice images is block collected in the same way as the second slice image, and the repeated parts are not described again.

[0092] After obtaining the plurality of sub-images of the slice image, the plurality of sub-images are stitched to obtain a stitched image of the region of interest of the slice image. When stitching the plurality of sub-images, the offset between the two sub-images to be stitched is calculated using the phase correlation method, i.e., the Fourier transform of the two sub-images obtained based on the pixel-based spatial position division is performed to the frequency domain, and the phase information in the frequency domain is used to calculate the translation relationship between the two sub-images, i.e., the offset between the two sub-images is determined, and the sub-images are moved according to the offset to stitch the two sub-images. The offset between the two sub-images I1 and I2 is calculated by the following formula:

[0093]

[0094] wherein, is the cross power spectrum, is is the Fourier transform of is is the Fourier transform conjugate of and respectively represent the amplitude of , is the impulse function, is the impulse function of the offset; when the value of the impulse function is maximum, the offset is determined.

[0095] In an implementation, for each to-be-aligned stitching image, the alignment matrix of each corresponding grid cell pair between the to-be-aligned stitching image and the reference stitching image is determined, comprising:

[0096] for each to-be-aligned grid cell of the to-be-aligned stitching image, a corresponding reference grid cell in the reference stitching image is determined respectively to obtain a plurality of grid cell pairs;

[0097] for each grid cell pair, a feature point pair similarity between each feature point of the to-be-aligned grid cell and each feature point of the reference grid cell in the grid cell pair is determined, a target feature point pair satisfying a condition is selected, and a target feature point pair set corresponding to the grid cell pair is obtained; and an alignment matrix of the to-be-aligned grid cell relative to the reference grid cell in the grid cell pair is determined according to the coordinates of the feature points in the target feature point pair set.

[0098] In the embodiment, the reference stitching image and the to-be-aligned stitching image are subjected to the same grid processing, as shown in FIG. 10, for each to-be-aligned stitching image, each to-be-aligned grid cell thereon can find a corresponding reference grid cell in the reference stitching image, as shown in FIG. 11, the first grid cell 1001 of the to-be-aligned stitching image 1000 and the first grid cell 2001 of the reference stitching image 2000 are a grid cell pair. Figure 5 Figure 5 Further, for each grid cell pair, a feature point pair similarity between each feature point of the to-be-aligned grid cell and each feature point of the reference grid cell in the grid cell pair is determined. Wherein, the to-be-aligned grid cell and the reference grid cell in each grid cell pair are subjected to feature retrieval matching, and the corresponding grid feature retrieval matching formula is as follows:

[0099]

[0100] ​​​​

[0101] wherein, represents the first m elements selected from S, represents arranging T in descending order, is a point and a point is a correlation coefficient of the two points, represents the first m matching point set after sorting the correlation coefficients from large to small, the point is a feature point in a reference grid cell, is a feature point in a grid cell to be aligned.

[0102] The first m matching point set of each grid cell pair is determined by the above formula , and then the alignment matrix h of the grid cell pair to be aligned relative to the reference grid cell can be calculated according to the coordinates of the feature points in .

[0103] In an implementation manner, the to-be-aligned stitched image and the reference stitched image are aligned according to the alignment matrix, including:

[0104] For each coordinate point of the to-be-aligned stitched image, the coordinates of the coordinate point are corrected according to the alignment matrix corresponding to the grid cell pair to which the coordinate point belongs, so that the to-be-aligned stitched image and the reference stitched image are aligned.

[0105] In this embodiment, after obtaining the alignment matrix h of each grid cell pair, the global alignment matrix is calculated based on the Meshflow method, wherein the formula of the global alignment matrix is as follows:

[0106]

[0107] wherein, M and N represent that the reference stitched image and the to-be-aligned stitched image are divided into grid cells, k is a grid row index, l is a grid column index, rows is a total number of image rows, cols is a total number of image columns, and each grid cell (k, l) corresponds to a transformation matrix , is a coordinate of the to-be-aligned stitched image, is a coordinate of the to-be-aligned stitched image after coordinate alignment transformation.

[0108] The global alignment matrix includes the alignment matrix corresponding to the grid cell pair to which each coordinate point belongs, that is, for each to-be-aligned stitched image, the alignment matrix corresponding to the grid cell pair to which all coordinate points in the to-be-aligned stitched image belong is determined through the global alignment matrix, so that the coordinates of all coordinate points in the to-be-aligned stitched image are corrected, the to-be-aligned stitched image and the reference stitched image are aligned, and after alignment, the image gap is completed by using the interpolation method to obtain the aligned stitched image.

[0109] In an implementable manner, defect detection is performed on each aligned spliced image, and a corresponding simulation image is generated for a target spliced image with defects, including:

[0110] Defect detection is performed on each aligned spliced image by a first deep learning model, and identification information, a defect type, and a defect position of a target spliced image with defects are obtained.

[0111] A simulation image of the target spliced image is generated by a second deep learning model.

[0112] In this embodiment, defect detection is performed on each aligned spliced image by using a first deep learning model, and the first deep learning model can be a unique visual feature model capable of deep learning defects such as YOLOv8. Taking the YOLOv8 model as an example, in the training stage, the aligned spliced images of normal slice images and the aligned spliced images of defect images such as wrinkles, pollution, breakage, and loss are used as a training set, and the aligned spliced images with defects are labeled, each defect position and defect type are framed for each image, and the YOLOv8 model is trained to enable the trained YOLOv8 model to perform defect detection on each aligned spliced image. The aligned spliced image with defects detected by the YOLOv8 model is a target spliced image, and the identification information, the defect type, and the defect position of the target spliced image can be obtained through the detection of the YOLOv8 model. Further, a second deep learning model is used to generate a simulation image corresponding to the target spliced image.

[0113] In an implementable manner, the second deep learning model is a GAN.

[0114] The simulation image of the target spliced image is generated by the second deep learning model, including:

[0115] For the target spliced image, a simulation image of the target spliced image is generated according to the previous spliced image and the next spliced image.

[0116] In this embodiment, the GAN model is used as the second deep learning model for defect repair. The GAN model includes a generator and a discriminator. In the model training process, the input data of the generator is the aligned images adjacent to the real reference image, that is, the corresponding simulation image is generated by the previous spliced image and the next spliced image of the real reference image; the discriminator is used to distinguish the authenticity of the real image and the simulation image until it cannot be distinguished.

[0117] In actual application, the simulation image of the target spliced image is generated by using the trained GAN model, i.e., the simulation image of the target spliced image is generated according to the previous spliced image and the next spliced image of the target spliced image. It should be noted that if the previous spliced image or the next spliced image of the target spliced image is also a spliced image with defects, if the previous spliced image of the current target spliced image is a spliced image with defects, the simulation image of the current target spliced image is further generated by using the previous spliced image of the previous spliced image and the next spliced image of the current target spliced image. Similarly, if the previous two spliced images of the current target spliced image are both spliced images with defects, continue to find forward until a spliced image without defects is determined.

[0118] In an implementation manner, the similarity between the simulation image and the target spliced image is determined, comprising:

[0119] The pixel difference index between the simulation image and the target spliced image is determined, and the pixel difference index represents the degree of gray difference between the simulation image and the target spliced image at each position;

[0120] The content similarity index between the simulation image and the target spliced image is determined, and the content similarity index represents the similarity between the simulation image and the target spliced image in image content;

[0121] The similarity between the simulation image and the target spliced image is determined according to the pixel difference index and the content similarity index.

[0122] In this embodiment, after the simulation image of the target spliced image is generated, the similarity between the simulation image and the target spliced image is calculated. The similarity calculation formula is as follows:

[0123]

[0124] In the formula, and are coefficients, , is the simulation image, is the target spliced image, is the pixel mean value of the simulation image, is the pixel mean value of the target spliced image, M is the number of image rows, N is the number of image columns, L is the maximum possible value of image pixel value, i is the row index, and j is the column index.

[0125] In the above similarity calculation formula, is the sum of normalized absolute differences, which is used to measure the gray difference of the same position of two images, If the contents of the two images are similar, such as different shooting angles of the same object, slight noise interference, the Pearson correlation coefficient r is close to 1, if the contents are irrelevant, the Pearson correlation coefficient r is close to 0, and if it is a negative film relationship (black to white, white to black), the Pearson correlation coefficient is close to -1. After the similarity JS calculated by the above similarity formula is compared with the pre-set threshold T, the defect degree of the target splicing image is determined. If JS < T, it means that the defect degree of the target splicing image is large, and the corresponding defect prompt information is directly returned, that is, the serial number, defect type and defect position of the target splicing image are returned, and no replacement processing is performed; if JS >= T, it means that the defect degree of the target splicing image is small, in addition to returning the defect prompt information, the corresponding image content is extracted from the same position of the simulation image according to the information related to the defect position in the defect prompt information, and the defect content of the same position of the target splicing image is replaced with the simulation image content, so as to repair the defects of the target splicing image and improve the image quality of the target splicing image.

[0126] In order to realize the above method, an example of the present application further provides an alignment device 600 for continuous slice images of a scanning transmission electron microscope, as shown in the figure, the device comprises: Figure 6

[0127] A coordinate system alignment module 601 is configured to align the coordinate systems of a plurality of continuous slice images collected by the scanning transmission electron microscope.

[0128] A coarse alignment module 602 is configured to block collect a region of interest of the slice image after the coordinate system alignment to obtain a plurality of sub-images of the slice image, and splice the plurality of sub-images to obtain a spliced image of the region of interest of the slice image.

[0129] A fine alignment module 603 is configured to pre-process the spliced image, grid process the pre-processed spliced image, select a reference spliced image from all spliced images of the slice image, determine an alignment matrix of each corresponding grid unit of the reference spliced image for each to-be-aligned spliced image, and align the to-be-aligned spliced image and the reference spliced image according to the alignment matrix.

[0130] An image repair module 604 is configured to detect defects of each spliced image after alignment, generate a corresponding simulation image for a target spliced image with defects, determine the similarity between the simulation image and the target spliced image, return corresponding defect prompt information if the similarity is lower than a threshold, and return corresponding defect prompt information and extract image content at the same position in the simulation image based on defect position information in the defect prompt information to replace image content at the defect position in the target spliced image.

[0131] ​In an embodiment, the coordinate system alignment module 601 is configured to determine a minimum bounding rectangle of the slice image, and determine a coordinate system of the slice image according to the minimum bounding rectangle;

[0132] select a reference slice image from the plurality of continuous slice images, each slice image other than the reference slice image being a slice image to be aligned, determine at least two key regions in the reference slice image, and determine a position of the key region in the coordinate system of the slice image to be aligned as a matching region according to a position of the key region in the coordinate system of the reference slice image;

[0133] obtain an image corresponding to the key region in the reference slice image as a template image, and obtain an image corresponding to the matching region in the slice image to be aligned as a matching image;

[0134] obtain a center region image of the template image as a sub-template image, match the sub-template image and the matching image based on a normalized correlation coefficient, and obtain an offset of the slice image to be aligned in the coordinate system in a horizontal direction and a vertical direction and a relative rotation angle with respect to the reference slice image when the matching is successful;

[0135] move the coordinate system of the slice image to be aligned according to the offset and the relative rotation angle, so that the coordinate system of the reference slice image and the coordinate system of the slice image to be aligned are aligned.

[0136] In an embodiment, the coarse alignment module 602 is configured to obtain an offset between two sub-images to be stitched, and move the sub-images according to the offset to stitch the two sub-images.

[0137] In an embodiment, the fine alignment module 603 is configured to, for each alignment grid cell of the slice image to be aligned, determine a reference grid cell corresponding to the alignment grid cell in the reference slice image, to obtain a plurality of grid cell pairs.

[0138] For each grid cell pair, determine a feature point pair similarity between each feature point of the alignment grid cell and each feature point of the reference grid cell in the grid cell pair, select a target feature point pair satisfying a condition, and obtain a target feature point pair set corresponding to the grid cell pair; and determine an alignment matrix of the alignment grid cell with respect to the reference grid cell in the grid cell pair according to coordinates of the feature points in the target feature point pair set.

[0139] In an embodiment, the fine alignment module 603 is configured to, for each coordinate point of the slice image to be aligned, correct a coordinate of the coordinate point according to an alignment matrix corresponding to a grid cell pair to which the coordinate point belongs, so that the slice image to be aligned and the reference slice image are aligned.

[0140] In an implementation, the image repairing module 604 is configured to perform defect detection on each aligned spliced image by the first deep learning model to obtain identification information, defect type and defect position of the target spliced image with defects.

[0141] The second deep learning model is configured to generate a simulation image of the target spliced image.

[0142] In an implementation, the second deep learning model is a GAN, and the image repairing module 604 is configured to generate the simulation image of the target spliced image by the second deep learning model, including:

[0143] For the target spliced image, the simulation image of the target spliced image is generated according to the previous spliced image and the next spliced image.

[0144] In an implementation, the image repairing module 604 is configured to determine a pixel difference index between the simulation image and the target spliced image, and the pixel difference index represents a degree of gray difference between the simulation image and the target spliced image at each position.

[0145] The image repairing module 604 is configured to determine a content similarity index between the simulation image and the target spliced image, and the content similarity index represents a degree of similarity between the simulation image and the target spliced image in image content.

[0146] The image repairing module 604 is configured to determine a similarity between the simulation image and the target spliced image according to the pixel difference index and the content similarity index.

[0147] Exemplarily, the present application also provides an electronic device, including:

[0148] a processor;

[0149] a memory for storing processor-executable instructions;

[0150] the processor is configured to read the executable instructions from the memory and execute the instructions to implement the above-mentioned alignment method of the continuous slice images of the scanning transmission electron microscope.

[0151] Exemplarily, the present application also provides a computer-readable storage medium, which stores a computer program for executing the above-mentioned alignment method of the continuous slice images of the scanning transmission electron microscope.

[0152] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present disclosure described and / or claimed in this document.

[0153] As shown in Figure 7 The device 700 includes a computing unit 701 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 702 or a computer program loaded into a random access memory (RAM) 703 from a storage unit 708. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0154] Various components in the device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, a magneto-optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0155] The computing unit 701 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, and the like. The computing unit 701 performs various methods and processes described above, such as a method of alignment of serial section images of a scanning transmission electron microscope. For example, in some embodiments, a method of alignment of serial section images of a scanning transmission electron microscope can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of a method of alignment of serial section images of a scanning transmission electron microscope described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform a method of alignment of serial section images of a scanning transmission electron microscope by any other appropriate means, such as by means of firmware.

[0156] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0157] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0158] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0159] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input. The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0160] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, in series, or in a different order, as long as the desired results of the technology disclosed in the present disclosure are achieved, and the present disclosure is not limited herein. In addition, the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise specifically limited. The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present disclosure, which should be covered within the protection scope of the present disclosure.

Claims

1. A method of aligning serial section images of a scanning transmission electron microscope, characterized by, The method comprises: aligning coordinate systems of a plurality of continuous slice images collected by a scanning transmission electron microscope; block collecting a region of interest of the slice images after the coordinate system alignment, obtaining a plurality of sub-images of the slice images, and splicing the plurality of sub-images to obtain a spliced image of the region of interest of the slice images; the splicing of the plurality of sub-images comprises: obtaining an offset between two sub-images to be spliced, moving the sub-images according to the offset, and splicing the two sub-images; preprocessing the spliced image, grid processing the preprocessed spliced image, selecting a reference spliced image from the spliced images of all slice images, determining, for each remaining to-be-aligned spliced image, an alignment matrix of each pair of corresponding grid cells of the to-be-aligned spliced image and the reference spliced image, aligning the to-be-aligned spliced image and the reference spliced image according to the alignment matrix; detecting defects in each aligned spliced image, generating a corresponding simulation image for a target spliced image with defects, determining a similarity between the simulation image and the target spliced image, returning a corresponding defect prompt information if the similarity is lower than a threshold, returning a corresponding defect prompt information if the similarity is higher than or equal to the threshold, and extracting image content at the same position in the simulation image based on defect position information therein to replace image content at a defect position in the target spliced image.

2. The method of claim 1, wherein, The aligning of coordinate systems of a plurality of continuous slice images collected by a scanning transmission electron microscope comprises: determining a minimum circumscribed rectangle of the slice images, and determining a coordinate system of the slice images according to the minimum circumscribed rectangle; selecting a reference slice image from the plurality of continuous slice images, each remaining slice image being a to-be-aligned slice image, determining at least two key regions in the reference slice image, and determining corresponding positions of the key regions in the coordinate system of the to-be-aligned slice image as to-be-matched regions according to positions of the key regions in the coordinate system of the reference slice image; obtaining an image corresponding to the key regions in the reference slice image as a template image, and obtaining an image corresponding to the to-be-matched regions in the to-be-aligned slice image as a to-be-matched image; obtaining a center region image of the template image as a sub-template image, matching the sub-template image and the to-be-matched image based on a normalized correlation coefficient, and obtaining, when the matching is successful, an offset of the to-be-aligned slice image compared with the reference slice image in a horizontal axis direction and a vertical axis direction of the coordinate system and a relative rotation angle; moving the coordinate system of the to-be-aligned slice image according to the offset and the relative rotation angle to align the coordinate system of the reference slice image and the coordinate system of the to-be-aligned slice image.

3. The method of claim 1, wherein, The determination of an alignment matrix of each pair of corresponding grid cells of a to-be-aligned spliced image and a reference spliced image comprises: For each to-be-aligned grid cell of the to-be-aligned stitched image, a corresponding reference grid cell in the reference stitched image is determined respectively, to obtain a plurality of grid cell pairs; For each grid cell pair, a feature point pair similarity between each feature point of the to-be-aligned grid cell and each feature point of the reference grid cell in the grid cell pair is determined, a target feature point pair satisfying a condition is selected, and a target feature point pair set corresponding to the grid cell pair is obtained; and an alignment matrix of the to-be-aligned grid cell relative to the reference grid cell in the grid cell pair is determined according to coordinates of feature points in the target feature point pair set.

4. The method of claim 3, wherein, The aligning the to-be-aligned stitched image and the reference stitched image according to the alignment matrix comprises: For each coordinate point of the to-be-aligned stitched image, the coordinates of the coordinate point are corrected according to the alignment matrix corresponding to the grid cell pair to which the coordinate point belongs, so that the to-be-aligned stitched image and the reference stitched image are aligned.

5. The method of claim 1, wherein, The defect detection on each stitched image after the aligning, and the generation of a corresponding simulation image for a target stitched image with defects, comprises: Defect detection on each stitched image after the aligning is performed through a first deep learning model, to obtain identification information, a defect type, and a defect position of a target stitched image with defects; A simulation image of the target stitched image is generated through a second deep learning model.

6. The method of claim 5, wherein, The second deep learning model is a GAN; The generation of the simulation image of the target stitched image through the second deep learning model comprises: For the target stitched image, a simulation image of the target stitched image is generated according to a previous stitched image and a next stitched image.

7. The method of claim 5, wherein, The determination of the similarity between the simulation image and the target stitched image comprises: A pixel difference index between the simulation image and the target stitched image is determined, the pixel difference index representing a gray difference degree of the simulation image and the target stitched image at each position; A content similarity index between the simulation image and the target stitched image is determined, the content similarity index representing a similarity degree of the simulation image and the target stitched image in image content; The similarity between the simulation image and the target stitched image is determined according to the pixel difference index and the content similarity index.

8. An apparatus for aligning serial section images of a scanning transmission electron microscope, characterized by, It comprises: A coordinate system alignment module is configured to align a plurality of continuous slice images collected by a scanning transmission electron microscope in a coordinate system; A coarse alignment module is configured to block collect a region of interest of the slice image after the coordinate system alignment, to obtain a plurality of sub-images of the slice image; The plurality of sub-images are stitched to obtain a stitched image of the region of interest of the slice image; The stitching of the plurality of sub-images comprises: An offset between two to-be-stitched sub-images is obtained, and the sub-images are moved according to the offset to stitch the two sub-images; a fine alignment module, configured to preprocess the stitched images, perform grid processing on the preprocessed stitched images, select a reference stitched image from the stitched images of all the slice images, determine, for each to-be-aligned stitched image, an alignment matrix of each pair of corresponding grid cells of the to-be-aligned stitched image and the reference stitched image, and align the to-be-aligned stitched image and the reference stitched image according to the alignment matrix; an image repairing module, configured to perform defect detection on each aligned stitched image, generate a corresponding simulation image for a target stitched image with defects, determine a similarity between the simulation image and the target stitched image, return a corresponding defect prompt information if the similarity is lower than a threshold, return a corresponding defect prompt information if the similarity is higher than or equal to the threshold, and extract image content at a same position in the simulation image based on defect position information in the defect prompt information to replace image content at a defect position in the target stitched image.

9. An electronic device, comprising: comprising: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for aligning continuous slice images of a scanning transmission electron microscope according to any one of claims 1-7.

Images