Image registration method and device

Abstract

The invention discloses an image registration method and device, and relates to the technical field of image processing. The method comprises the following steps: converting a plurality of spatial domain images in an image series into corresponding first frequency domain amplitude diagrams through Fourier transform; superposing the first frequency domain amplitude images to construct a frequency domain amplitude summation graph; correcting each spatial domain image in the image series based on the set drift curve; all the corrected spatial domain images are superposed, and a spatial domain addition image is constructed; converting the spatial domain addition graph into a second frequency domain amplitude graph through Fourier transform; comparing the frequency domain amplitude summation graph with the second frequency domain amplitude graph; taking the adjusted drift curve as a set drift curve, and outputting a target drift curve; and performing image registration processing on the image series by using the target drift curve. According to the scheme, the image registration efficiency can be effectively improved, and consumption of computing resources by image registration is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image registration method and device. Background Art

[0002] Image registration is to align multiple images acquired at different times, different sensors or different viewing angles in space so that they can achieve spatial consistency for subsequent image analysis and processing. This image registration has important applications in various fields involving image analysis, such as medical imaging (such as multimodal image fusion), remote sensing image analysis (such as satellite image stitching), computer vision (such as target tracking and stereo vision), and electron microscope image processing. In addition, image registration also plays an important role in electron microscope image processing.

[0003] At present, the technologies of image registration mainly involve feature registration, intensity registration, deformation model registration and deep learning registration. These existing image registration technologies are relatively complex and have relatively high requirements on image features and computing resources, resulting in high demand for image registration computing resources and low efficiency. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides an image registration method and device. The solution provided by the embodiment of the present invention can effectively improve the efficiency of image registration and reduce the consumption of computing resources for image registration.

[0005] To achieve the above object, according to a first aspect of an embodiment of the present invention, there is provided an image registration method, comprising:

[0006] Converting a plurality of spatial domain images in the image series into corresponding first frequency domain amplitude images through Fourier transformation;

[0007] superimposing the first frequency domain amplitude images to construct a frequency domain amplitude summation graph;

[0008] The following steps N1 to N5 are executed repeatedly until the comparison result meets the convergence condition:

[0009] N1. Correcting each spatial domain image in the image series based on the set drift curve;

[0010] N2, superimposing the corrected spatial domain images to construct a spatial domain summation map;

[0011] N3. Convert the spatial domain summation graph into a second frequency domain amplitude graph through Fourier transform;

[0012] N4, comparing the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, executing step N5; if the comparison result meets the convergence condition, executing step N6;

[0013] N5: Adjust the parameters of the preset drift curve, and use the adjusted drift curve as the set drift curve, and execute step N1;

[0014] N6. Determine that the set drift curve is a target drift curve, and output the target drift curve;

[0015] The target drift curve is used to perform image registration processing on the image series.

[0016] Optionally, determining the position of the theoretical acquisition time in the image set includes:

[0017] According to the acquisition time of each of the images, the theoretical acquisition time, and the acquisition sequence of the multiple images, the position of the theoretical acquisition time in the image set is determined.

[0018] Optionally, the image registration method is characterized by further comprising:

[0019] An initial drift curve is constructed according to the displacement difference between the first space domain image and the last space domain image in the image series.

[0020] Optionally, the above image registration method further includes: Step N5 includes:

[0021] For each of the spatial domain images, perform the following operations:

[0022] The parameters of the drift curve are adjusted for the space domain image, and the adjusted drift curve is set as the drift curve corresponding to the space domain image.

[0023] Optionally, step N1 includes:

[0024] In the case where each of the spatial domain images corresponds to a different drift curve,

[0025] The drift curve is used to correct the corresponding spatial domain image.

[0026] Optionally, step N2 includes:

[0027] The spatial domain images corrected for the current cycle period of each spatial domain image are superimposed to construct a spatial domain summation graph;

[0028] or,

[0029] For each of the spatial domain images, perform the following operations:

[0030] The space domain image corrected in the current cycle of the space domain image is superimposed with the space domain images corrected in the previous cycle of other space domain images to construct a plurality of space domain summation images.

[0031] Optionally, step N4 includes:

[0032] Calculating the variance or mean square error between the frequency domain amplitude sum graph and the second frequency domain amplitude graph;

[0033] When the variance is less than or equal to the first preset threshold or the mean square error is less than or equal to the second preset threshold, it is determined that the comparison result meets the convergence condition.

[0034] Optionally, performing image registration processing on the image series includes:

[0035] Correcting each spatial domain image in the image series using the target drift curve;

[0036] Each spatial domain image corrected by the target drift curve is superimposed.

[0037] In a second aspect, an embodiment of the present invention provides an image registration device, comprising: an image conversion unit, a curve construction unit, and a registration processing unit, wherein:

[0038] The image conversion unit is used to convert the multiple spatial domain images in the image series into corresponding first frequency domain amplitude diagrams through Fourier transform, and after receiving the spatial domain sum diagram sent by the curve construction unit, convert the spatial domain sum diagram into a second frequency domain amplitude diagram;

[0039] The curve construction unit is used to superimpose the first frequency domain amplitude images to construct a frequency domain amplitude summation graph; and to loop through the following steps M1 to M5 until the comparison result meets the convergence condition:

[0040] M1. Correcting each spatial domain image in the image series based on a set drift curve;

[0041] M2, superimposing the corrected spatial domain images to construct a spatial domain summation map;

[0042] M3. Send the spatial domain summation graph to the image conversion unit, and receive the second frequency domain amplitude graph sent by the image conversion unit;

[0043] M4, comparing the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, executing step M5; if the comparison result meets the convergence condition, executing step M6;

[0044] M5: adjusting the parameters of the preset drift curve, and using the adjusted drift curve as the set drift curve, and executing step M1;

[0045] M6. Determine the set drift curve as the target drift curve, and output the target drift curve to the registration processing unit;

[0046] The registration processing unit is used to perform image registration processing on the image series using the target drift curve.

[0047] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0048] one or more processors;

[0049] a storage device for storing one or more programs,

[0050] When the one or more programs are executed by the one or more processors, the one or more processors implement the image registration method provided in the above-mentioned first aspect embodiment.

[0051] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the image registration method provided in the embodiment of the first aspect is implemented.

[0052] An embodiment of the above invention has the following advantages or beneficial effects: a frequency domain amplitude summation diagram is obtained by superimposing each first frequency domain amplitude diagram in an image sequence, and the frequency domain amplitude summation diagram is used as a reference diagram, and a spatial domain summation diagram is constructed from each spatial domain image in the image series that has been corrected using a drift curve, and the frequency domain amplitude diagram corresponding to the spatial domain summation diagram (i.e., the second frequency domain amplitude diagram) is compared with the frequency domain amplitude summation diagram used as a reference diagram. If the comparison result does not meet the convergence condition, the drift curve is continued to be adjusted. If the convergence condition is met, image registration processing is performed on the image series based on the drift curve that meets the convergence condition. Each step of the technical solution provided by the embodiment of the present invention is relatively simple, does not involve a complex calculation process, can complete image registration processing relatively quickly, can effectively improve image registration efficiency, and improve image calibration accuracy. Moreover, relatively small computing resources can meet the requirements of each step, effectively reducing the consumption of computing resources for image registration.

[0053] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0055] Figure 1 is a schematic diagram of the main process of the image registration method according to an embodiment of the present invention;

[0056] Figure 2 is a spatial domain image collected by a transmission electron microscope (TEM) in STEM mode according to an embodiment of the present invention;

[0057] Figure 3 According to an embodiment of the present invention Figure 2 A first frequency domain amplitude map obtained after the spatial domain image of is processed in step S101;

[0058] Figure 4 It is a single-frame STEM spatial domain image of SrTiO3 crystal;

[0059] Figure 5 The embodiment of the present invention provides Figure 4 The drift curve of a single frame STEM spatial domain image is given;

[0060] Figure 6 is a registered image obtained by performing registration processing on a series of SrTiO3 crystal STEM spatial domain images according to an embodiment of the present invention;

[0061] Figure 7 It is an image obtained by directly superimposing the STEM spatial domain images of the SrTiO3 crystal in the image sequence without correction according to an embodiment of the present invention;

[0062] Figure 8 is the drift curve obtained by applying the traditional cross-correlation;

[0063] Fig. 9 is the registered image obtained based on the existing cross-correlation;

[0064] Fig.10 It is a single-frame SEM spatial domain image of amorphous Sn sphere;

[0065] Fig.11 is a drift curve of the SEM spatial domain image of a non-crystalline Sn ball provided by an embodiment of the present invention;

[0066] Fig.12 is a registered image obtained by performing registration processing on a series of SEM spatial domain images of amorphous Sn spheres according to an embodiment of the present invention;

[0067] Fig.13 An image obtained by directly superimposing the SEM spatial domain images of the amorphous Sn sphere in the image sequence without correction according to an embodiment of the present invention;

[0068] Fig.14 is a schematic structural diagram of an image registration device according to an embodiment of the present invention;

[0069] Fig.15 is a schematic diagram of an exemplary system architecture according to an embodiment of the present invention;

[0070] Fig.16 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0072] In addition, the terms "first", "second", etc. included in the embodiments of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific number or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing objects with the same attributes when describing them in the embodiments of the present invention.

[0073] In order to solve the problem that the existing image alignment process requires a relatively complex calculation method or relatively large computing resources, the embodiment of the present invention provides an image registration method and device. Figure 1 As shown, the image registration method may include the following steps:

[0074] Step S101: converting a plurality of spatial domain images in the image series into corresponding first frequency domain amplitude images through Fourier transform;

[0075] Among them, the image series is generally composed of images collected continuously for the same sample in a continuous time period. For example, multiple TEM images obtained by TEM scanning the same sample in a continuous time period can be collected at the same sample rod tilt angle or at different sample rod rotation angles. Another example is the images collected by medical imaging equipment in a continuous time period. Another example is multiple SEM images obtained by SEM scanning the same sample in a continuous time period.

[0076] The Fourier transform used in this step can be directly implemented using existing technologies and will not be described in detail here.

[0077] For example, Figure 2 The spatial domain image of a crystalline material collected by a transmission electron microscope (TEM) in STEM mode is shown. Figure 2 The first frequency domain amplitude diagram obtained by performing Fourier transform of the spatial domain image shown in this step is as follows Figure 3 shown.

[0078] Step S102: superimposing the first frequency domain amplitude graphs to construct a frequency domain amplitude sum graph;

[0079] This step is specifically implemented by adding up the amplitudes corresponding to each pixel in each of the first frequency domain amplitude maps for each pixel, and the positions of each pixel and the superimposed amplitudes corresponding to each pixel constitute a frequency domain amplitude summation map.

[0080] In addition, the sampling time points of each pixel point on the spatial domain image are different (for example, for a STEM image, a 256×256 image needs to be scanned, that is, there are 256 pixels in each row and 256 pixels in each column, and each pixel point is obtained by sequential scanning. Accordingly, each pixel point has a different sampling time point). For each image, if the first pixel point collected is taken as the sampling 0 point, the sampling time points of other pixel points are all timed relative to the sampling 0 point, then the positions of the above-mentioned pixel points are converted to corresponding time points, that is, the above-mentioned frequency domain amplitude summation diagram can also be the sum of the amplitudes of the same sampling time points in each spatial domain image, and the sum of the amplitudes corresponding to each sampling time point constitutes the frequency domain amplitude summation diagram.

[0081] The following steps S103 to S107 are executed cyclically until the comparison result meets the convergence condition:

[0082] Step S103: correcting each spatial domain image in the image series based on the set drift curve;

[0083] The drift curve may be set by the user based on his / her own experience, or may be set based on the displacement difference between the first spatial domain image and the last spatial domain image in the image series.

[0084] For the case of setting according to the displacement difference, the image registration method may further include: constructing an initial drift curve according to the displacement difference between the first spatial domain image and the last spatial domain image in the image series. For example, the initial drift curve y=kx+A, where y represents the corrected pixel point, k is obtained by dividing the displacement difference between the first spatial domain image and the last spatial domain image by (the number of spatial domain images between the first spatial domain image and the last spatial domain image); A represents a setting constant; x represents a pixel point in the spatial domain image to be corrected. It is worth noting that the drift curve can be a linear function, a quadratic function, etc., which can be set by the user. In addition, the drift curve can also be obtained through machine learning.

[0085] In addition, in this step, each spatial domain image in the image sequence can be corrected using the same drift curve, and each spatial domain image can be set to correspond to a drift curve, that is, different spatial domain images correspond to different drift curves, so as to better correct the spatial domain image.

[0086] In the case where each space domain image corresponds to a different drift curve, the drift curve is used to correct the corresponding space domain image. For example, space domain image A corresponds to drift curve 1, space domain image B corresponds to drift curve 2, and space domain image C corresponds to drift curve 3. In this step, drift curve 1 is used to correct space domain image A, drift curve 2 is used to correct space domain image B, and drift curve 3 is used to correct space domain image C.

[0087] It is worth noting that, in the process of initially establishing the drift curve, an initial drift curve may be established for an image sequence, and subsequently, based on the initial drift curve, a drift curve may be established for each spatial domain image.

[0088] Step S104: superimposing the corrected spatial domain images to construct a spatial domain summation map;

[0089] There are two specific implementation methods for this step. Specifically, the first implementation method is to superimpose the spatial domain images corrected for the current cycle period of each spatial domain image to construct a spatial domain summation graph. The second implementation method is to perform the following operation for each of the spatial domain images: superimpose the spatial domain image corrected for the current cycle period of the spatial domain image with the spatial domain images corrected for the previous cycle period of other spatial domain images to construct multiple spatial domain summation graphs.

[0090] Here, superimposing different spatial domain images means superimposing pixels at the same position. For example, if the positions of some pixels contained in a spatial domain image are (1, 1), (1, 2), (1, 3), (1, 4), etc., then superimposing different spatial domain images means superimposing pixels at position (1, 1) in different spatial domain images, superimposing pixels at position (1, 2) in different spatial domain images, superimposing pixels at position (1, 3) in different spatial domain images, superimposing pixels at position (1, 4) in different spatial domain images, etc.

[0091] Step S105: converting the spatial domain summation graph into a second frequency domain amplitude graph through Fourier transform;

[0092] Step S106: comparing the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, executing step S107; if the comparison result meets the convergence condition, executing step S108;

[0093] The specific implementation method of this step can be to calculate the error between the frequency domain amplitude sum graph and the second frequency domain amplitude graph; when the error is less than or equal to a preset threshold, determine that the comparison result meets the convergence condition. The error between the frequency domain amplitude sum graph and the second frequency domain amplitude graph can be generally expressed as a variance or a mean square error. The preset threshold is generally set by the user according to the needs, and after the preset threshold is set, it can be further adjusted according to the output result.

[0094] Step S107: adjusting the parameters of the preset drift curve, and using the adjusted drift curve as the set drift curve, and executing step S103;

[0095] The adjustment of the parameters of the preset drift curve can be achieved by a preset adjustment strategy.

[0096] For each space domain image, the following operations are performed: the parameters of the drift curve are adjusted for the space domain image, and the adjusted drift curve is set as the drift curve corresponding to the space domain image. The space domain image is corrected by the drift curve corresponding to each space domain image, so that each space domain image can be corrected in a personalized and differentiated manner, which effectively improves the accuracy of the image registration processing, and effectively improves the clarity of the image after the registration processing.

[0097] Step S108: determining that the set drift curve is a target drift curve, and outputting the target drift curve;

[0098] Step S109: using the target drift curve, perform image registration processing on the image series.

[0099] Each spatial domain image in the image series is corrected using the target drift curve; each spatial domain image corrected by the target drift curve is superimposed. It can be understood that, in the case where an image sequence corresponds to one target drift curve, all spatial domain images in the image sequence use the target drift curve to correct drift; in the case where each spatial domain image corresponds to a different target drift curve, the target drift curve is used to correct the corresponding spatial domain image.

[0100] It is worth noting that the spatial domain image targeted by the above processing process can be a spatial domain image scanned by any electron microscope, for example, a spatial domain image obtained by a crystalline sample or a non-crystalline sample obtained by a scanning electron microscope (SEM) or TEM.

[0101] In addition, the above-mentioned process of iterating and obtaining the drift curve is performed for each image sequence during the image registration process. That is to say, each image registration has a process of iterating and obtaining the drift curve to better calibrate the image and improve the image calibration accuracy.

[0102] A frequency domain amplitude summation diagram is obtained by superimposing each first frequency domain amplitude diagram in an image sequence, and the frequency domain amplitude summation diagram is used as a reference diagram. A spatial domain summation diagram is constructed from each spatial domain image in the image series that has been corrected using a drift curve, and the frequency domain amplitude diagram (i.e., the second frequency domain amplitude diagram) corresponding to the spatial domain summation diagram is compared with the frequency domain amplitude summation diagram used as a reference diagram. If the comparison result does not meet the convergence condition, the drift curve is continued to be adjusted. If the convergence condition is met, image registration processing is performed on the image series based on the drift curve that meets the convergence condition. Each step of the technical solution provided in the embodiment of the present invention is relatively simple, does not involve a complex calculation process, can complete image registration processing relatively quickly, can effectively improve image registration efficiency, and improve image calibration accuracy. Moreover, relatively small computing resources can meet the requirements of each step, and effectively reduce the consumption of computing resources for image registration.

[0103] The following uses the STEM image sequence of SrTiO3 crystal and the SEM image sequence of amorphous Sn sphere as examples to explain in detail the image registration process completed by the technical solution provided by the embodiment of the present invention.

[0104] Figure 4 The single-frame STEM spatial domain image of SrTiO3 crystal in image sequence 1 is shown as an example. Specifically, for the registration processing of image sequence 1: each frame of SrTiO3 crystal STEM spatial domain image in image sequence 1 is replaced with the corresponding first frequency domain amplitude diagram through the above step S101; then through the above step S102 and the loop execution of steps S103 to S107, the drift curve for the single-frame image is obtained as shown in FIG. Figure 5 Finally, the SrTiO3 crystal STEM spatial domain image series in image sequence 1 is registered through the drift curve corresponding to each frame image, and the registered image is as shown in Figure 6 shown.

[0105] In order to clearly illustrate the processing effect of the technical solution provided by the embodiment of the present invention, Figure 4 The image sequence 1 of the image shown in the figure is directly superimposed without correction. Figure 7 The drift curve of a single frame image obtained by using the existing cross-correlation processing method (the cross-correlation processing method is a frequency domain calculation method of cross-correlation, which determines the relative displacement between images by using the phase information of the image. Specifically, this method converts two images into phase maps in the frequency domain and calculates the product of the complex conjugate of one image and the other image to evaluate the phase consistency) is shown in Figure 8 As shown, the image after registration using the existing cross-correlation processing method is as follows Fig. 9Compared with the existing cross-correlation processing method, the technical solution provided by the embodiment of the present invention requires fewer computing resources and has higher processing efficiency, can obtain processing results more quickly, and can improve image calibration accuracy.

[0106] from Figure 4 It can be seen that the signal-to-noise ratio of a single frame image is poor and cannot provide clear image details. Since there is displacement between the images, they are directly superimposed, such as Figure 7 As shown in Figure 1, the image will be obviously blurred, resulting in almost complete loss of image details. The drift curve obtained by applying traditional cross-correlation ( Figure 8 ) and the drift curve obtained by the technical solution proposed by the present invention ( Figure 5 ), it can be seen that the drift curve calculated by the technical solution proposed by the present invention is almost consistent with the drift curve obtained by cross-correlation. Further, compared with the existing cross-correlation obtained registration image ( Fig. 9 ) and the registered image obtained by the technical solution provided by the embodiment of the present invention ( Figure 6 ) It can be seen that after the image is registered by the technical solution provided by the embodiment of the present invention, the image quality is significantly improved, the signal-to-noise ratio is increased, the details are increased, and the image calibration accuracy can be improved. It illustrates the effectiveness of the image registration method proposed in the present invention for crystal sequence image registration. In addition, the frequency domain calculation method of cross-correlation is very dependent on the phase information of the image, and thus faces some challenges. For example, the phase information is periodic and is usually limited to between -𝜋 and 𝜋. When the calculated phase difference exceeds this range, phase wrapping occurs, that is, the phase suddenly jumps to the opposite extreme value. This nonlinear jump may cause errors in interpreting and processing data, especially when the phase difference needs to be accurately measured to calculate the precise displacement. Phase wrapping needs to be corrected by phase unwrapping, which increases the computational complexity and implementation difficulty. In addition, the phase is particularly sensitive to noise, especially the noise in the high-frequency part, which may distort the phase information and cause inaccurate registration. In practical applications, the frequency domain calculation method of cross-correlation must adopt various noise suppression techniques, such as filtering, to ensure the accuracy of phase estimation. When dealing with noise, it is necessary to find a balance between retaining important information and eliminating interference. Therefore, the existing frequency domain calculation method of cross-correlation has relatively large technical challenges. The technical solution provided by the embodiment of the present invention can simply and efficiently align the sequence images. The technical solution provided by the embodiment of the present invention only requires the amplitude part of the frequency domain and does not rely on the phase, thereby overcoming the shortcomings of inaccurate alignment caused by phase wrapping and phase sensitivity to noise.

[0107] Fig.10The single-frame SEM spatial domain image of amorphous Sn ball in image sequence 2 is exemplarily shown. Specifically, for the registration processing of image sequence 2: each frame of the SEM spatial domain image of amorphous Sn ball in image sequence 1 is replaced with the corresponding first frequency domain amplitude diagram through the above step S101; then through the above step S102 and the loop execution of steps S103 to S107, the drift curve for the SEM spatial domain image of amorphous Sn ball is obtained as shown in FIG. Fig.11 Finally, the SEM spatial domain image series of amorphous Sn spheres in image sequence 2 is registered through the drift curves corresponding to each frame of the image, and the registered image is shown in Fig.12 shown.

[0108] In order to clearly illustrate the processing effect of the technical solution provided by the embodiment of the present invention, Fig.10 The image sequence 2 of the image shown in the figure is directly superimposed without correction. Fig.13 shown.

[0109] from Fig.10 It can be seen that the signal-to-noise ratio of a single frame image is poor and cannot provide clear image details. Since there is displacement between the images, they are directly superimposed, such as Fig.13 As shown in FIG. 1 , the image will be obviously blurred, resulting in almost complete loss of image details. The drift curve ( Fig.11 ) The corrected image ( Fig.12 ) It can be seen that after the image is registered by the technical solution provided by the embodiment of the present invention, the image quality is significantly improved, the signal-to-noise ratio is increased, and the details are increased. This shows that the image registration method proposed by the present invention is also applicable to the effectiveness of non-crystal sequence image registration and can improve the image calibration accuracy.

[0110] Further, Fig.14 FIG. 1 is a schematic diagram of the structure of an image registration device provided by an embodiment of the present invention. Fig.14 As shown, the image registration device 1400 may include: an image conversion unit 1401, a curve construction unit 1402 and a registration processing unit 1403, wherein:

[0111] The image conversion unit 1401 is used to convert multiple spatial domain images in the image series into corresponding first frequency domain amplitude diagrams through Fourier transform, and after receiving the spatial domain sum diagram sent by the curve construction unit 1402, convert the spatial domain sum diagram into a second frequency domain amplitude diagram;

[0112] The curve construction unit 1402 is used to superimpose the first frequency domain amplitude images to construct a frequency domain amplitude summation graph; and cyclically execute the following steps M1 to M5 until the comparison result meets the convergence condition:

[0113] M1. Correct each spatial domain image in the image series based on the set drift curve;

[0114] M2, superimposing the corrected spatial domain images to construct a spatial domain summation map;

[0115] M3, sending the spatial domain summation image to the image conversion unit 1403, and receiving the second frequency domain amplitude image sent by the image conversion unit;

[0116] M4, comparing the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, executing step M5; if the comparison result meets the convergence condition, executing step M6;

[0117] M5: Adjust the parameters of the preset drift curve, and use the adjusted drift curve as the set drift curve, and execute step M1;

[0118] M6. Determine the set drift curve as the target drift curve, and output the target drift curve to the registration processing unit 1403;

[0119] The registration processing unit 1403 is used to perform image registration processing on the image series using the target drift curve.

[0120] In the embodiment of the present invention, the curve construction unit 1402 is further configured to construct an initial drift curve according to the displacement difference between the first space domain image and the last space domain image in the image series.

[0121] In the embodiment of the present invention, M5 performed by the curve construction unit 1402 includes: for each spatial domain image, performing the following operations: adjusting the parameters of the drift curve for the spatial domain image, and setting the adjusted drift curve as the drift curve corresponding to the spatial domain image.

[0122] In the embodiment of the present invention, M1 executed by the curve construction unit 1402 includes: in the case where each space domain image corresponds to a different drift curve, using the drift curve to correct the corresponding space domain image.

[0123] In an embodiment of the present invention, M2 executed by the curve construction unit 1402 includes: superimposing the spatial domain images of each spatial domain image after current cycle correction to construct a spatial domain sum graph; or, for each spatial domain image, performing the following operation: superimposing the spatial domain image of the current cycle correction of the spatial domain image with the spatial domain images of the previous cycle correction of other spatial domain images to construct multiple spatial domain sum graphs.

[0124] In an embodiment of the present invention, M4 executed by the curve construction unit 1402 includes: calculating the variance or mean square error between the frequency domain amplitude sum graph and the second frequency domain amplitude graph; when the variance is less than or equal to the first preset threshold or the mean square error is less than or equal to the second preset threshold, determining that the comparison result meets the convergence condition.

[0125] In the embodiment of the present invention, the registration processing unit 1403 is further configured to correct each space domain image in the image series by using the target drift curve; and to superimpose each space domain image corrected by the target drift curve.

[0126] It is worth noting that the above-mentioned image registration device can be installed in the form of a plug-in in existing image processing software or image processing software used for electron microscopes such as TEM software, SEM software, etc., or can be installed as an application in a terminal device or server.

[0127] Taking the processing of images scanned by a transmission electron microscope as an example, Fig.15 An exemplary system architecture 1500 to which the image registration method or image registration device according to the embodiment of the present invention can be applied is shown.

[0128] As shown in Figure 1500, the system architecture 1500 may include a first terminal device 1501 directly connected to a TEM device, a TEM device 15202, a network 1503, and a second terminal device 1504 or a server 1505 for image data processing. The network 1503 is used to provide a medium for a communication link between the first terminal device 1501 and the TEM device 15202, between the TEM device 15202 and the second terminal device 1503, between the first terminal device 1501 and the second terminal device 1504, or between the first terminal device 1501 and the server 1505. The network 1503 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0129] The user can use the first terminal device 1501 to interact with the TEM device 15202 through the network 1503 to obtain scanning information of the TEM device 15202 or send control information of the TEM device 15202. The first terminal device 1501 may be installed with software for controlling the TEM device or for displaying TEM scanning results (for example only).

[0130] The second terminal device 1504 or the server 1505 is used to obtain the image series scanned by the TEM device from the first terminal device 1501 and the TEM device 15202 respectively, and to perform image registration on the images in the image series and to reconstruct the registered images, and to store the reconstructed three-dimensional structure and provide it to the user.

[0131] The first terminal device 1501 and the second terminal device 1504 may be various electronic devices having a display screen and supporting web browsing, including but not limited to a laptop computer and a desktop computer, and the like.

[0132] It should be noted that the image registration method provided in the embodiment of the present invention can be executed by the first terminal device 1501 or the second terminal device 1504 , and accordingly, the image registration device is generally disposed in the first terminal device 1501 or the second terminal device 1504 .

[0133] In addition, the above-mentioned first terminal device 1501 and second terminal device 1504 may also belong to the same terminal device. Accordingly, the image registration device is installed in the first terminal device 1501 in the form of a plug-in to realize image registration of transmission electron microscopy image data through the image registration method provided by an embodiment of the present invention.

[0134] It should be understood that Fig.15 The number of the second terminal devices or servers in the embodiment is only for illustration. Any number of second terminal devices or servers may be provided according to implementation requirements.

[0135] Reference below Fig.16 , which shows a schematic diagram of the structure of a computer system 1600 of a terminal device or a server suitable for implementing an embodiment of the present invention. Fig.16 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0136] like Fig.16 As shown, the computer system 1600 includes a central processing unit (CPU) 1601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1602 or a program loaded from a storage part 1608 into a random access memory (RAM) 1603. In the RAM 1603, various programs and data required for the operation of the system 1600 are also stored. The CPU 1601, the ROM 1602, and the RAM 1603 are connected to each other via a bus 1604. An input / output (I / O) interface 1605 is also connected to the bus 1604.

[0137] The following components are connected to the I / O interface 1605: an input section 1606 including a keyboard, a mouse, etc.; an output section 1607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN card, a modem, etc. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the I / O interface 1605 as needed. A removable medium 1611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1610 as needed, so that a computer program read therefrom is installed into the storage section 1608 as needed.

[0138] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1609, and / or installed from the removable medium 1611. When the computer program is executed by the central processing unit (CPU) 1601, the above-mentioned functions defined in the system of the present invention are executed.

[0139] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0140] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The modules involved in the embodiments of the present invention may be implemented by software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes an image conversion unit, a curve construction unit, and a registration processing unit. The names of these modules or units do not constitute a limitation on the modules themselves in some cases. For example, the image conversion unit may also be described as "a unit or module that converts a spatial domain image into a corresponding frequency domain amplitude map".

[0142] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: converting multiple spatial domain images in the image series into corresponding first frequency domain amplitude maps through Fourier transform; superimposing each first frequency domain amplitude map to construct a frequency domain amplitude sum map; looping the following steps N1 to N5 until the comparison result meets the convergence condition: N1. Correct each spatial domain image in the image series based on the set drift curve; N2. Superimpose each corrected spatial domain image to construct a spatial domain sum Figure; N3, through Fourier transform, convert the spatial domain summation diagram into a second frequency domain amplitude diagram; N4, compare the frequency domain amplitude summation diagram and the second frequency domain amplitude diagram; when the comparison result does not meet the convergence condition, execute step N5; when the comparison result meets the convergence condition, execute step N6; N5: adjust the parameters of the preset drift curve, and use the adjusted drift curve as the set drift curve, and execute step N1; N6, determine that the set drift curve is the target drift curve, and output the target drift curve; use the target drift curve to perform image registration processing on the image series.

[0143] According to the technical solution of the embodiment of the present invention, a frequency domain amplitude summation diagram is obtained by superimposing each first frequency domain amplitude diagram in the image sequence, and the frequency domain amplitude summation diagram is used as a reference diagram. A spatial domain summation diagram is constructed from each spatial domain image in the image series that has been corrected using a drift curve, and the frequency domain amplitude diagram (i.e., the second frequency domain amplitude diagram) corresponding to the spatial domain summation diagram is compared with the frequency domain amplitude summation diagram used as the reference diagram. If the comparison result does not meet the convergence condition, the drift curve is further adjusted. If the convergence condition is met, image registration processing is performed on the image series based on the drift curve that meets the convergence condition. Each step of the technical solution provided by the embodiment of the present invention is relatively simple, does not involve a complex calculation process, can complete image registration processing relatively quickly, can effectively improve image registration efficiency, can improve image calibration accuracy, and relatively small computing resources can meet the requirements of each step, effectively reducing the consumption of computing resources for image registration.

[0144] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image registration method, characterized in that: include: Converting a plurality of spatial domain images in the image series into corresponding first frequency domain amplitude images through Fourier transformation; Superimposing the first frequency domain amplitude graphs to construct a frequency domain amplitude sum graph; The following steps N1 to N5 are executed repeatedly until the comparison result meets the convergence condition: N1. Correcting each spatial domain image in the image series based on the set drift curve; N2, superimposing the corrected spatial domain images to construct a spatial domain summation map; N3. Convert the spatial domain summation graph into a second frequency domain amplitude graph through Fourier transform; N4, comparing the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, executing step N5; if the comparison result meets the convergence condition, executing step N6; N5: Adjust the parameters of the preset drift curve, and use the adjusted drift curve as the set drift curve, and execute step N1; N6. Determine that the set drift curve is a target drift curve, and output the target drift curve; The target drift curve is used to perform image registration processing on the image series.

2. The image registration method according to claim 1, characterized in that: Also includes: An initial drift curve is constructed according to the displacement difference between the first space domain image and the last space domain image in the image series.

3. The image registration method according to claim 1 or 2, characterized in that: Step N5 includes: For each of the spatial domain images, perform the following operations: The parameters of the drift curve are adjusted for the space domain image, and the adjusted drift curve is set as the drift curve corresponding to the space domain image.

4. The image registration method according to claim 3, characterized in that: Step N1 includes: In the case where each of the spatial domain images corresponds to a different drift curve, The drift curve is used to correct the corresponding spatial domain image.

5. The image registration method according to claim 4, characterized in that: Step N2 includes: The spatial domain images corrected for the current cycle period of each spatial domain image are superimposed to construct a spatial domain summation graph; or, For each of the spatial domain images, perform the following operations: The space domain image corrected in the current cycle of the space domain image is superimposed with the space domain image corrected in the previous cycle of other space domain images to construct a space domain summation graph.

6. The image registration method according to any one of claims 1 to 5, characterized in that: Step N4 includes: Calculating an error between the frequency domain amplitude sum graph and the second frequency domain amplitude graph; When the error is less than or equal to a preset threshold, it is determined that the comparison result meets the convergence condition.

7. The image registration method according to claim 1, characterized in that: The performing image registration processing on the image series comprises: Correcting each spatial domain image in the image series using the target drift curve; Each spatial domain image corrected by the target drift curve is superimposed.

8. An image registration device, characterized in that: include: An image conversion unit, a curve construction unit and a registration processing unit, wherein: The image conversion unit is used to convert the multiple spatial domain images in the image series into corresponding first frequency domain amplitude diagrams through Fourier transform, and after receiving the spatial domain sum diagram sent by the curve construction unit, convert the spatial domain sum diagram into a second frequency domain amplitude diagram; The curve construction unit is used to superimpose the first frequency domain amplitude images to construct a frequency domain amplitude summation graph; and to loop through the following steps M1 to M5 until the comparison result meets the convergence condition: M1. Correcting each spatial domain image in the image series based on a set drift curve; M2, superimposing the corrected spatial domain images to construct a spatial domain summation map; M3. Send the spatial domain summation graph to the image conversion unit, and receive the second frequency domain amplitude graph sent by the image conversion unit; M4, comparing the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, executing step M5; if the comparison result meets the convergence condition, executing step M6; M5: adjusting the parameters of the preset drift curve, and using the adjusted drift curve as the set drift curve, and executing step M1; M6. Determine the set drift curve as the target drift curve, and output the target drift curve to the registration processing unit; The registration processing unit is used to perform image registration processing on the image series using the target drift curve.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images