Image registration method and device

Abstract

The present invention discloses an image registration method and device, which relates to the field of image processing technology. The method includes: converting multiple spatial domain images in an image series into corresponding first frequency domain amplitude maps through Fourier transform; superimposing each first frequency domain amplitude image to construct a frequency domain amplitude sum map; correcting each spatial domain image in the image series based on a set drift curve; superimposing each corrected spatial domain image to construct a spatial domain sum map; converting the spatial domain sum map into a second frequency domain amplitude map through Fourier transform; comparing the frequency domain amplitude sum map and the second frequency domain amplitude map; and using the adjusted drift curve as the set drift curve to output a target drift curve; and performing image registration processing on the image series using the target drift curve. This solution can effectively improve the efficiency of image registration and reduce the consumption of computing resources for image registration.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image registration method and device. Background Art

[0002] Image registration is the process of spatially aligning multiple images acquired at different times, using different sensors, or from different perspectives to achieve spatial consistency for subsequent image analysis and processing. This type of image registration has important applications in various fields involving image analysis, such as medical imaging (such as multimodal image fusion), remote sensing image analysis (such as satellite image stitching), computer vision (such as target tracking and stereo vision), and electron microscopy image processing. Image registration also plays a crucial role in electron microscopy image processing.

[0003] At present, the technologies of image registration mainly involve feature registration, intensity registration, deformation model registration and deep learning registration. These existing image registration technologies are relatively complex and have relatively high requirements on image features and computing resources, resulting in high demand for computing resources and low efficiency for image registration. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides an image registration method and apparatus. The solution provided by the embodiment of the present invention can effectively improve the efficiency of image registration and reduce the consumption of computing resources for image registration.

[0005] To achieve the above object, according to a first aspect of an embodiment of the present invention, there is provided an image registration method, comprising:

[0006] Converting the plurality of spatial domain images in the image series into corresponding first frequency domain amplitude images through Fourier transform;

[0007] superimposing the first frequency domain amplitude graphs to construct a frequency domain amplitude sum graph;

[0008] Repeat steps N1 to N5 until the comparison result meets the convergence condition:

[0009] N1. Correcting each spatial domain image in the image series based on a set drift curve;

[0010] N2. Superimpose the corrected spatial domain images to construct a spatial domain summation map;

[0011] N3. Convert the spatial domain summation graph into a second frequency domain amplitude graph through Fourier transform;

[0012] N4. Compare the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, execute step N5; if the comparison result meets the convergence condition, execute step N6;

[0013] N5: Adjust the parameters of the preset drift curve, and use the adjusted drift curve as the set drift curve, and execute step N1;

[0014] N6. Determine that the set drift curve is a target drift curve, and output the target drift curve;

[0015] The target drift curve is used to perform image registration processing on the image series.

[0016] Optionally, determining the position of the theoretical acquisition time in the image set includes:

[0017] The position of the theoretical acquisition time in the image set is determined according to the acquisition duration of each image, the theoretical acquisition time, and the acquisition order of the plurality of images.

[0018] Optionally, the image registration method is characterized by further comprising:

[0019] An initial drift curve is constructed according to the displacement difference between the first spatial domain image and the last spatial domain image in the image series.

[0020] Optionally, the above image registration method further includes: Step N5 includes:

[0021] For each of the spatial domain images, perform the following operations:

[0022] Parameters of the drift curve are adjusted for the spatial domain image, and the adjusted drift curve is set as the drift curve corresponding to the spatial domain image.

[0023] Optionally, step N1 includes:

[0024] For the case where each of the spatial domain images corresponds to a different drift curve,

[0025] The drift curve is used to correct the corresponding spatial domain image.

[0026] Optionally, step N2 includes:

[0027] The spatial domain images corrected for the current cycle period of each spatial domain image are superimposed to construct a spatial domain summation graph;

[0028] or,

[0029] For each of the spatial domain images, perform the following operations:

[0030] The spatial domain image corrected in the current cycle of the spatial domain image is superimposed with the spatial domain images corrected in the previous cycle of other spatial domain images to construct a plurality of spatial domain summation images.

[0031] Optionally, step N4 includes:

[0032] Calculating the variance or mean square error between the frequency domain amplitude sum graph and the second frequency domain amplitude graph;

[0033] When the variance is less than or equal to the first preset threshold or the mean square error is less than or equal to the second preset threshold, it is determined that the comparison result meets the convergence condition.

[0034] Optionally, performing image registration processing on the image series includes:

[0035] Correcting each spatial domain image in the image series using the target drift curve;

[0036] Each spatial domain image corrected by the target drift curve is superimposed.

[0037] In a second aspect, an embodiment of the present invention provides an image registration device, comprising: an image conversion unit, a curve construction unit, and a registration processing unit, wherein:

[0038] The image conversion unit is configured to convert the plurality of spatial domain images in the image series into corresponding first frequency domain amplitude maps through Fourier transform, and after receiving the spatial domain summation map sent by the curve construction unit, convert the spatial domain summation map into a second frequency domain amplitude map;

[0039] The curve construction unit is configured to superimpose the first frequency domain amplitude graphs to construct a frequency domain amplitude sum graph; and cyclically execute the following steps M1 to M5 until the comparison result meets the convergence condition:

[0040] M1. Correcting each spatial domain image in the image series based on a set drift curve;

[0041] M2, superimpose the corrected spatial domain images to construct a spatial domain summation map;

[0042] M3. Send the spatial domain sum image to the image conversion unit, and receive the second frequency domain amplitude image sent by the image conversion unit;

[0043] M4. Compare the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, execute step M5; if the comparison result meets the convergence condition, execute step M6;

[0044] M5: Adjust the parameters of the preset drift curve, and use the adjusted drift curve as the set drift curve, and execute step M1;

[0045] M6. Determine the set drift curve as a target drift curve, and output the target drift curve to the registration processing unit;

[0046] The registration processing unit is used to perform image registration processing on the image series using the target drift curve.

[0047] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0048] one or more processors;

[0049] a storage device for storing one or more programs,

[0050] When the one or more programs are executed by the one or more processors, the one or more processors implement the image registration method provided in the above-mentioned first aspect embodiment.

[0051] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the image registration method provided in the embodiment of the first aspect is implemented.

[0052] An embodiment of the above invention has the following advantages or beneficial effects: a frequency domain amplitude summation map is obtained by superimposing each first frequency domain amplitude map in an image sequence, and the frequency domain amplitude summation map is used as a reference map, and a spatial domain summation map is constructed from each spatial domain image in the image series that has been corrected using a drift curve, and the frequency domain amplitude map corresponding to the spatial domain summation map (i.e., the second frequency domain amplitude map) is compared with the frequency domain amplitude summation map used as a reference map. If the comparison result does not meet the convergence condition, the drift curve is continued to be adjusted. If the convergence condition is met, the image series is subjected to image registration processing based on the drift curve that meets the convergence condition. Each step of the technical solution provided by the embodiment of the present invention is relatively simple, does not involve complex calculation processes, can complete image registration processing relatively quickly, can effectively improve image registration efficiency, and improve image calibration accuracy. Moreover, relatively small computing resources can meet the requirements of each step, effectively reducing the consumption of computing resources for image registration.

[0053] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0055] Figure 1is a schematic diagram of the main process of the image registration method according to an embodiment of the present invention;

[0056] Figure 2 is a spatial domain image acquired by a transmission electron microscope (TEM) in STEM mode according to an embodiment of the present invention;

[0057] Figure 3 According to an embodiment of the present invention Figure 2 A first frequency domain amplitude map obtained after the spatial domain image of is processed in step S101;

[0058] Figure 4 This is a single-frame STEM spatial domain image of a SrTiO3 crystal;

[0059] Figure 5 The embodiment of the present invention provides Figure 4 The drift curve of a single-frame STEM spatial domain image is given;

[0060] Figure 6 is a registered image obtained by performing registration processing on a series of STEM spatial domain images of a SrTiO3 crystal according to an embodiment of the present invention;

[0061] Figure 7 This is an image obtained by directly superimposing the STEM spatial domain images of the SrTiO3 crystal in the image sequence without correction according to an embodiment of the present invention;

[0062] Figure 8 is the drift curve obtained by applying traditional cross-correlation;

[0063] Figure 9 is the registered image obtained based on the existing cross-correlation;

[0064] Figure 10 This is a single-frame SEM spatial domain image of amorphous Sn sphere;

[0065] Figure 11 is a drift curve of the SEM spatial domain image of an amorphous Sn ball provided by an embodiment of the present invention;

[0066] Figure 12 is a registered image obtained by performing registration processing on a series of SEM spatial domain images of amorphous Sn spheres according to an embodiment of the present invention;

[0067] Figure 13 This is an image obtained by directly superimposing the SEM spatial domain images of the amorphous Sn sphere in the image sequence without correction according to an embodiment of the present invention;

[0068] Figure 14 is a structural diagram of an image registration device according to an embodiment of the present invention;

[0069] Figure 15 is a schematic structural diagram of an exemplary system architecture according to an embodiment of the present invention;

[0070] Figure 16 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0072] Furthermore, the terms "first," "second," and so on, used in the embodiments of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific number or order. It should be understood that the terms used in this manner are interchangeable where appropriate. This is merely a way of distinguishing objects with the same attributes in the embodiments of the present invention.

[0073] In order to solve the problems that the existing image alignment processing requires a relatively complex calculation method or relatively large computing resources, the embodiment of the present invention provides an image registration method and device. Figure 1 As shown, the image registration method may include the following steps:

[0074] Step S101: converting multiple spatial domain images in the image series into corresponding first frequency domain amplitude images through Fourier transform;

[0075] An image series generally consists of images acquired continuously over a continuous period of time for the same sample. For example, multiple TEM images acquired over a continuous period of time by a TEM scan of the same sample can be acquired at the same or different sample holder tilt angles. Another example is images acquired over a continuous period of time by a medical imaging device. Another example is multiple SEM images acquired over a continuous period of time by a SEM scan of the same sample.

[0076] The Fourier transform used in this step can be directly implemented using existing technologies and will not be described in detail here.

[0077] For example, Figure 2 The spatial domain image of a crystalline material collected by a transmission electron microscope (TEM) in STEM mode is shown. Figure 2The first frequency domain amplitude diagram obtained by Fourier transforming the spatial domain image shown in this step is as follows Figure 3 shown.

[0078] Step S102: superimposing the first frequency domain amplitude graphs to construct a frequency domain amplitude sum graph;

[0079] This step is specifically implemented by adding up the amplitudes corresponding to each pixel in each of the first frequency domain amplitude maps for each pixel, and the positions of each pixel and the superimposed amplitudes corresponding to each pixel constitute a frequency domain amplitude summation map.

[0080] In addition, the sampling time points of each pixel point on the spatial domain image are different (for example, for STEM images, a 256×256 image needs to be scanned, that is, there are 256 pixels in each row and 256 pixels in each column. The pixels are obtained by sequential scanning, and accordingly, the pixels have different sampling time points). For each image, if the first pixel point collected is used as the sampling 0 point, the sampling time points of the other pixels are all timed relative to the sampling 0 point, then the positions of the above-mentioned pixels are converted into corresponding time points, that is, the above-mentioned frequency domain amplitude summation graph can also be the sum of the amplitudes at the same sampling time points in each spatial domain image, and the sum of the amplitudes corresponding to each sampling time point constitutes the frequency domain amplitude summation graph.

[0081] The following steps S103 to S107 are executed cyclically until the comparison result meets the convergence condition:

[0082] Step S103: correcting each spatial domain image in the image series based on the set drift curve;

[0083] The drift curve can be set by the user based on his or her own experience, or can be set based on the displacement difference between the first spatial domain image and the last spatial domain image in the image series.

[0084] For settings based on displacement differences, the image registration method can further include constructing an initial drift curve based on the displacement difference between the first and last spatial domain images in the image series. For example, the initial drift curve y = kx + A, where y represents the corrected pixel, k is the displacement difference between the first and last spatial domain images divided by the number of spatial domain images between the first and last spatial domain images, A represents a set constant, and x represents the pixel in the spatial domain image to be corrected. It is worth noting that the drift curve can be a linear function, a quadratic function, or other functions, and can be set by the user. Alternatively, the drift curve can be derived through machine learning.

[0085] In addition, in this step, each spatial domain image in the image sequence can be corrected using the same drift curve, and each spatial domain image can be set to correspond to a drift curve, that is, different spatial domain images correspond to different drift curves, so as to better correct the spatial domain image.

[0086] If each spatial domain image corresponds to a different drift curve, the drift curve is used to correct the corresponding spatial domain image. For example, if spatial domain image A corresponds to drift curve 1, spatial domain image B corresponds to drift curve 2, and spatial domain image C corresponds to drift curve 3, then in this step, drift curve 1 is used to correct spatial domain image A, drift curve 2 is used to correct spatial domain image B, and drift curve 3 is used to correct spatial domain image C.

[0087] It is worth noting that, in the process of initially establishing the drift curve, an initial drift curve may be established for an image sequence, and subsequently, based on the initial drift curve, a drift curve may be established for each spatial domain image.

[0088] Step S104: superimposing the corrected spatial domain images to construct a spatial domain summation image;

[0089] There are two specific implementations for this step. Specifically, the first implementation is to superimpose the spatial domain images corrected for the current cycle period of each spatial domain image to construct a spatial domain summation map. The second implementation is to perform the following operation for each spatial domain image: superimpose the spatial domain image corrected for the current cycle period of the spatial domain image with the spatial domain images corrected for the previous cycle period of other spatial domain images to construct multiple spatial domain summation maps.

[0090] Here, superimposing different spatial domain images means superimposing pixels at the same position. For example, if a spatial domain image contains some pixels at positions (1, 1), (1, 2), (1, 3), (1, 4), etc., then superimposing different spatial domain images means superimposing the pixel at position (1, 1) in different spatial domain images, superimposing the pixel at position (1, 2) in different spatial domain images, superimposing the pixel at position (1, 3) in different spatial domain images, superimposing the pixel at position (1, 4) in different spatial domain images, and so on.

[0091] Step S105: converting the spatial domain summation graph into a second frequency domain amplitude graph through Fourier transform;

[0092] Step S106: comparing the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, executing step S107; if the comparison result meets the convergence condition, executing step S108;

[0093] A specific implementation of this step may include calculating the error between the frequency domain amplitude sum plot and the second frequency domain amplitude plot; and determining that the comparison result meets the convergence condition when the error is less than or equal to a preset threshold. The error between the frequency domain amplitude sum plot and the second frequency domain amplitude plot can be expressed as a variance or a mean square error. The preset threshold is generally set by the user based on their needs and can be further adjusted based on the output results.

[0094] Step S107: adjusting the parameters of the preset drift curve, and using the adjusted drift curve as the set drift curve, and executing step S103;

[0095] The parameters of the preset drift curve can be adjusted by a preset adjustment strategy.

[0096] For each spatial domain image, the following operations are performed: the parameters of the drift curve for the spatial domain image are adjusted, and the adjusted drift curve is set as the drift curve corresponding to the spatial domain image. The spatial domain image is corrected using the drift curve corresponding to each spatial domain image, allowing for personalized and differentiated correction of each spatial domain image, effectively improving the accuracy of the image registration process and the clarity of the registered image.

[0097] Step S108: Determine that the set drift curve is the target drift curve, and output the target drift curve;

[0098] Step S109: performing image registration processing on the image series using the target drift curve.

[0099] Each spatial domain image in the image series is corrected using the target drift curve; each spatial domain image corrected by the target drift curve is superimposed. It can be understood that if an image sequence corresponds to a single target drift curve, all spatial domain images in the sequence are drift corrected using that target drift curve; if different spatial domain images correspond to different target drift curves, the corresponding spatial domain images are corrected using that target drift curve.

[0100] It is worth noting that the spatial domain image targeted by the above processing process can be a spatial domain image scanned by any electron microscope, for example, a spatial domain image obtained by a crystalline sample or a non-crystalline sample obtained by a scanning electron microscope (SEM) or TEM.

[0101] In addition, the above-mentioned process of iteratively obtaining the drift curve will be performed for each image sequence during the image registration process. That is to say, each image registration will have an iterative process of obtaining the drift curve to better calibrate the image and improve the image calibration accuracy.

[0102] A frequency domain amplitude summation diagram is obtained by superimposing the first frequency domain amplitude maps in the image sequence, and the frequency domain amplitude summation diagram is used as a reference diagram. A spatial domain summation diagram is constructed from the spatial domain images in the image series that have been corrected using the drift curve, and the frequency domain amplitude map corresponding to the spatial domain summation diagram (i.e., the second frequency domain amplitude map) is compared with the frequency domain amplitude summation diagram serving as the reference diagram. If the comparison result does not meet the convergence condition, the drift curve is continued to be adjusted. If the convergence condition is met, the image series is subjected to image registration processing based on the drift curve that meets the convergence condition. Each step of the technical solution provided by the embodiment of the present invention is relatively simple, does not involve complex calculation processes, can complete image registration processing relatively quickly, can effectively improve image registration efficiency, and improve image calibration accuracy. Moreover, relatively small computing resources can meet the requirements of each step, effectively reducing the consumption of computing resources for image registration.

[0103] The following takes the STEM image sequence of SrTiO3 crystal and the SEM image sequence of amorphous Sn sphere as examples to explain in detail the image registration process accomplished by the technical solution provided by the embodiment of the present invention.

[0104] Figure 4 The STEM spatial domain image of a single frame of SrTiO3 crystal in image sequence 1 is shown as an example. Specifically, for the registration processing of image sequence 1: each frame of SrTiO3 crystal STEM spatial domain image in image sequence 1 is replaced with the corresponding first frequency domain amplitude diagram through the above step S101; then, through the above step S102 and the loop execution of steps S103 to S107, the drift curve for the single frame image is obtained as shown below. Figure 5 Finally, the SrTiO3 crystal STEM spatial domain image series in image sequence 1 is registered using the drift curve corresponding to each frame of the image. The registered image is shown in Figure 6 shown.

[0105] In order to clearly illustrate the processing effect of the technical solution provided by the embodiment of the present invention, Figure 4 The image sequence 1 of the image shown in the figure is directly superimposed without correction. Figure 7 The drift curve of a single frame image obtained by using the existing cross-correlation processing method (the cross-correlation processing method is a frequency domain calculation method of cross-correlation, which determines the relative displacement between images by using the phase information of the image. Specifically, this method converts the two images into phase maps in the frequency domain and calculates the product of the complex conjugate of one image and the other image to evaluate the phase consistency) is shown as follows Figure 8 As shown, the image after registration using the existing cross-correlation processing method is as follows Figure 9Compared with the existing cross-correlation processing method, the technical solution provided by the embodiment of the present invention requires fewer computing resources and has higher processing efficiency, can obtain processing results more quickly, and can improve image calibration accuracy.

[0106] from Figure 4 It can be seen that the signal-to-noise ratio of a single frame image is poor and cannot provide clear image details. Due to the displacement between the images, they are directly superimposed, such as Figure 7 As shown in Figure 2, the image will be obviously blurred, resulting in almost complete loss of image details. The drift curve obtained by applying traditional cross-correlation ( Figure 8 ) and the drift curve obtained by the technical solution proposed by the present invention ( Figure 5 ), it can be seen that the drift curve calculated by the technical solution proposed in the present invention is almost consistent with the drift curve obtained by cross-correlation. Further, compared with the existing cross-correlation obtained registration image ( Figure 9 ) and the registered image obtained by the technical solution provided by the embodiment of the present invention ( Figure 6 ) As can be seen, after the images are registered using the technical solution provided by the embodiments of the present invention, image quality is significantly improved, the signal-to-noise ratio is increased, and details are increased, thereby enhancing image calibration accuracy. This demonstrates the effectiveness of the image registration method proposed in the present invention for crystal sequence image registration. Furthermore, frequency-domain calculation methods for cross-correlation rely heavily on the phase information of the image, which presents some challenges. For example, phase information is periodic and typically limited to between -𝜋 and 𝜋. When the calculated phase difference exceeds this range, phase wrapping occurs, where the phase suddenly jumps to the opposite extreme value. This nonlinear jump can lead to errors in data interpretation and processing, especially when precise phase difference measurement is required to calculate precise displacement. Phase wrapping requires correction through phase unwrapping, increasing computational complexity and implementation difficulty. Furthermore, phase is particularly sensitive to noise, especially high-frequency noise, which can distort phase information and lead to inaccurate registration. In practical applications, frequency-domain calculation methods for cross-correlation must employ various noise suppression techniques, such as filtering, to ensure accurate phase estimation. When dealing with noise, a balance must be struck between preserving important information and eliminating interference. Therefore, existing frequency-domain calculation methods for cross-correlation present significant technical challenges. The technical solution provided by the embodiment of the present invention can simply and efficiently align the sequence images. The technical solution provided by the embodiment of the present invention only requires the amplitude part of the frequency domain and does not rely on the phase, thereby overcoming the shortcomings of inaccurate alignment caused by phase wrapping and phase sensitivity to noise.

[0107] Figure 10The single-frame SEM spatial domain image of amorphous Sn sphere in image sequence 2 is shown as an example. Specifically, the registration process for image sequence 2 is as follows: each frame of the SEM spatial domain image of amorphous Sn sphere in image sequence 1 is replaced with the corresponding first frequency domain amplitude diagram through the above step S101; then, through the above step S102 and the loop execution of steps S103 to S107, the drift curve of the SEM spatial domain image of amorphous Sn sphere is obtained as shown below. Figure 11 Finally, the SEM spatial domain image series of amorphous Sn spheres in image sequence 2 is registered using the drift curves corresponding to each frame of the image. The registered image is shown in Figure 12 shown.

[0108] In order to clearly illustrate the processing effect of the technical solution provided by the embodiment of the present invention, Figure 10 The image sequence 2 of the image shown in the figure is directly superimposed without correction. Figure 13 shown.

[0109] from Figure 10 It can be seen that the signal-to-noise ratio of a single frame image is poor and cannot provide clear image details. Due to the displacement between the images, they are directly superimposed, such as Figure 13 As shown in the figure, the image will be obviously blurred, resulting in almost complete loss of image details. The drift curve ( Figure 11 ) The corrected image ( Figure 12 ) As can be seen, after the images are registered using the technical solution provided by the embodiments of the present invention, image quality is significantly improved, with an increased signal-to-noise ratio and more detail. This demonstrates the effectiveness of the image registration method proposed in this invention for the registration of amorphous sequential images and can improve image calibration accuracy.

[0110] Furthermore, Figure 14 FIG. 1 is a structural diagram of an image registration device provided by an embodiment of the present invention. Figure 14 As shown, the image registration device 1400 may include: an image conversion unit 1401, a curve construction unit 1402 and a registration processing unit 1403, wherein:

[0111] The image conversion unit 1401 is configured to convert the plurality of spatial domain images in the image series into corresponding first frequency domain amplitude maps through Fourier transform, and after receiving the spatial domain summation map sent by the curve construction unit 1402, convert the spatial domain summation map into a second frequency domain amplitude map;

[0112] The curve construction unit 1402 is configured to superimpose the first frequency domain amplitude graphs to construct a frequency domain amplitude sum graph; and to loop through steps M1 to M5 until the comparison result meets a convergence condition.

[0113] M1. Correct each spatial domain image in the image series based on the set drift curve;

[0114] M2, superimpose the corrected spatial domain images to construct a spatial domain summation map;

[0115] M3. Send the spatial domain sum image to the image conversion unit 1401, and receive the second frequency domain amplitude image sent by the image conversion unit;

[0116] M4. Compare the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, execute step M5; if the comparison result meets the convergence condition, execute step M6;

[0117] M5: Adjust the parameters of the preset drift curve, and use the adjusted drift curve as the set drift curve, and execute step M1;

[0118] M6. Determine the set drift curve as the target drift curve, and output the target drift curve to the registration processing unit 1403;

[0119] The registration processing unit 1403 is configured to perform image registration processing on the image series using the target drift curve.

[0120] In this embodiment of the present invention, the curve construction unit 1402 is further configured to construct an initial drift curve according to the displacement difference between the first spatial domain image and the last spatial domain image in the image series.

[0121] In an embodiment of the present invention, M5 performed by the curve construction unit 1402 includes: for each spatial domain image, performing the following operations: adjusting the parameters of the drift curve for the spatial domain image, and setting the adjusted drift curve as the drift curve corresponding to the spatial domain image.

[0122] In the embodiment of the present invention, the M1 executed by the curve construction unit 1402 includes: in the case where each spatial domain image corresponds to a different drift curve, correcting the corresponding spatial domain image using the drift curve.

[0123] In an embodiment of the present invention, M2 executed by the curve construction unit 1402 includes: superimposing the spatial domain images of each spatial domain image after current cycle correction to construct a spatial domain sum graph; or, for each spatial domain image, performing the following operation: superimposing the spatial domain image of the spatial domain image after current cycle correction with the spatial domain images of other spatial domain images after previous cycle correction to construct multiple spatial domain sum graphs.

[0124] In an embodiment of the present invention, M4 executed by the curve construction unit 1402 includes: calculating the variance or mean square error between the frequency domain amplitude sum graph and the second frequency domain amplitude graph; when the variance is less than or equal to the first preset threshold or the mean square error is less than or equal to the second preset threshold, determining that the comparison result meets the convergence condition.

[0125] In the embodiment of the present invention, the registration processing unit 1403 is further configured to correct each spatial domain image in the image series using the target drift curve; and superimpose each spatial domain image corrected by the target drift curve.

[0126] It is worth noting that the above-mentioned image registration device can be installed as a plug-in in existing image processing software or image processing software used for electron microscopes such as TEM software, SEM software, etc., or can be installed as an application on a terminal device or server.

[0127] Taking the processing of transmission electron microscope scanned images as an example, Figure 15 An exemplary system architecture 1500 is shown to which the image registration method or image registration apparatus according to an embodiment of the present invention may be applied.

[0128] like Figure 15 As shown, system architecture 1500 may include a first terminal device 1501 directly connected to a TEM device, a TEM device 1502, a network 1503, and a second terminal device 1504 or a server 1505 for image data processing. Network 1503 is used to provide a medium for a communication link between the first terminal device 1501 and the TEM device 1502, between the TEM device 1502 and the second terminal device 1504, between the first terminal device 1501 and the second terminal device 1504, or between the first terminal device 1501 and the server 1505. Network 1503 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0129] A user can use a first terminal device 1501 to interact with a TEM device 1502 via a network 1503 to obtain scanning information from the TEM device 1502 or send control information to the TEM device 1502. The first terminal device 1501 may be installed with software for controlling the TEM device or displaying TEM scanning results (for example only).

[0130] The second terminal device 1504 or the server 1505 is used to obtain the image series scanned by the TEM device from the first terminal device 1501 and the TEM device 1502 respectively, perform image registration on the images in the image series and reconstruct the registered images, store the reconstructed three-dimensional structure and provide it to the user.

[0131] The first terminal device 1501 and the second terminal device 1504 can be various electronic devices with display screens and supporting web browsing, including but not limited to laptop computers and desktop computers.

[0132] It should be noted that the image registration method provided in the embodiment of the present invention can be executed by the first terminal device 1501 or the second terminal device 1504. Accordingly, the image registration device is generally provided in the first terminal device 1501 or the second terminal device 1504.

[0133] In addition, the above-mentioned first terminal device 1501 and second terminal device 1504 may also belong to the same terminal device. Accordingly, the image registration device is installed in the first terminal device 1501 in the form of a plug-in to realize image registration of transmission electron microscopy image data through the image registration method provided by the embodiment of the present invention.

[0134] It should be understood that Figure 15 The number of second terminal devices or servers in the embodiment is merely illustrative. Any number of second terminal devices or servers may be provided as required.

[0135] Reference below Figure 16 , which shows a structural diagram of a computer system 1600 of a terminal device or server suitable for implementing an embodiment of the present invention. Figure 16 The terminal device or server shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0136] like Figure 16 As shown, computer system 1600 includes a central processing unit (CPU) 1601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1602 or a program loaded from a storage unit 1608 into a random access memory (RAM) 1603. Various programs and data required for the operation of system 1600 are also stored in RAM 1603. CPU 1601, ROM 1602, and RAM 1603 are connected to each other via a bus 1604. An input / output (I / O) interface 1605 is also connected to bus 1604.

[0137] The following components are connected to the I / O interface 1605: an input section 1606 including a keyboard, mouse, and the like; an output section 1607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 1608 including a hard disk; and a communication section 1609 including a network interface card such as a LAN card or a modem. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the I / O interface 1605 as needed. Removable media 1611, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1610 as needed, so that computer programs read from the removable media can be installed in the storage section 1608 as needed.

[0138] In particular, according to embodiments disclosed herein, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed herein include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program containing program code for executing the methods illustrated in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1609 and / or installed from removable media 1611. When executed by central processing unit (CPU) 1601, the computer program performs the aforementioned functions defined in the system of the present invention.

[0139] It should be noted that the computer-readable medium described in the present invention may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present invention, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0141] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be provided in a processor. For example, they may be described as: a processor including an image conversion unit, a curve construction unit, and a registration processing unit. The names of these modules or units do not, in some cases, limit the modules themselves. For example, the image conversion unit may also be described as a "unit or module that converts a spatial domain image into a corresponding frequency domain amplitude map."

[0142] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: converting multiple spatial domain images in the image series into corresponding first frequency domain amplitude maps through Fourier transform; superimposing each first frequency domain amplitude map to construct a frequency domain amplitude sum map; looping through the following steps N1 to N5 until the comparison result meets the convergence condition: N1. Correct each spatial domain image in the image series based on the set drift curve; N2. Superimpose each corrected spatial domain image to construct a spatial domain sum Figure; N3, through Fourier transform, convert the spatial domain summation map into a second frequency domain amplitude map; N4, compare the frequency domain amplitude summation map and the second frequency domain amplitude map; when the comparison result does not meet the convergence condition, execute step N5; when the comparison result meets the convergence condition, execute step N6; N5: adjust the parameters of the preset drift curve, and use the adjusted drift curve as the set drift curve, and execute step N1; N6, determine that the set drift curve is the target drift curve, and output the target drift curve; use the target drift curve to perform image registration processing on the image series.

[0143] According to the technical solution of the embodiment of the present invention, a frequency domain amplitude summation map is obtained by superimposing the first frequency domain amplitude maps in the image sequence, and the frequency domain amplitude summation map is used as a reference map. The spatial domain images in the image series that have been corrected using the drift curve are used to construct a spatial domain summation map, and the frequency domain amplitude map corresponding to the spatial domain summation map (i.e., the second frequency domain amplitude map) is compared with the frequency domain amplitude summation map serving as the reference map. If the comparison result does not meet the convergence condition, the drift curve is further adjusted. If the convergence condition is met, the image series is subjected to image registration processing based on the drift curve that meets the convergence condition. Each step of the technical solution provided by the embodiment of the present invention is relatively simple, does not involve complex calculation processes, can complete image registration processing relatively quickly, can effectively improve image registration efficiency, can improve image calibration accuracy, and relatively small computing resources can meet the requirements of each step, effectively reducing the consumption of computing resources for image registration.

[0144] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. An image registration method, characterized in that: include: Converting the plurality of spatial domain images in the image series into corresponding first frequency domain amplitude images through Fourier transform; superimposing the first frequency domain amplitude graphs to construct a frequency domain amplitude sum graph; Repeat steps N1 to N5 until the comparison result meets the convergence condition: N1. Correcting each spatial domain image in the image series based on a set drift curve; N2. Superimpose the corrected spatial domain images to construct a spatial domain summation map; N3. Convert the spatial domain summation graph into a second frequency domain amplitude graph through Fourier transform; N4. Compare the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, execute step N5; if the comparison result meets the convergence condition, execute step N6; N5: Adjust the parameters of the preset drift curve, and use the adjusted drift curve as the set drift curve, and execute step N1; N6. Determine that the set drift curve is a target drift curve, and output the target drift curve; performing image registration processing on the image series using the target drift curve; The image registration method further includes: constructing an initial drift curve according to a displacement difference between a first spatial domain image and a last spatial domain image in the image series.

2. The image registration method according to claim 1, wherein: Step N5 includes: For each of the spatial domain images, perform the following operations: Parameters of the drift curve are adjusted for the spatial domain image, and the adjusted drift curve is set as the drift curve corresponding to the spatial domain image.

3. The image registration method according to claim 2, wherein: Step N1 includes: For the case where each of the spatial domain images corresponds to a different drift curve, The drift curve is used to correct the corresponding spatial domain image.

4. The image registration method according to claim 3, wherein: Step N2 includes: The spatial domain images corrected for the current cycle period of each spatial domain image are superimposed to construct a spatial domain summation graph; or, For each of the spatial domain images, perform the following operations: The spatial domain image corrected in the current cycle of the spatial domain image is superimposed with the spatial domain image corrected in the previous cycle of other spatial domain images to construct a spatial domain summation graph.

5. The image registration method according to any one of claims 1 to 4, characterized in that: Step N4 includes: Calculating an error between the frequency domain amplitude sum graph and the second frequency domain amplitude graph; When the error is less than or equal to a preset threshold, it is determined that the comparison result meets the convergence condition.

6. The image registration method according to claim 1, wherein: The performing image registration processing on the image series includes: Correcting each spatial domain image in the image series using the target drift curve; Each spatial domain image corrected by the target drift curve is superimposed.

7. An image registration device, characterized in that: include: An image conversion unit, a curve construction unit, and a registration processing unit, wherein: The image conversion unit is configured to convert the plurality of spatial domain images in the image series into corresponding first frequency domain amplitude maps through Fourier transform, and after receiving the spatial domain summation map sent by the curve construction unit, convert the spatial domain summation map into a second frequency domain amplitude map; The curve construction unit is configured to superimpose the first frequency domain amplitude graphs to construct a frequency domain amplitude sum graph; and cyclically execute the following steps M1 to M5 until the comparison result meets the convergence condition: M1. Correcting each spatial domain image in the image series based on a set drift curve; M2, superimpose the corrected spatial domain images to construct a spatial domain summation map; M3. Send the spatial domain sum image to the image conversion unit, and receive the second frequency domain amplitude image sent by the image conversion unit; M4. Compare the frequency domain amplitude sum graph and the second frequency domain amplitude graph; if the comparison result does not meet the convergence condition, execute step M5; if the comparison result meets the convergence condition, execute step M6; M5: Adjust the parameters of the preset drift curve, and use the adjusted drift curve as the set drift curve, and execute step M1; M6. Determine the set drift curve as a target drift curve, and output the target drift curve to the registration processing unit; The registration processing unit is configured to perform image registration processing on the image series using the target drift curve; The curve construction unit is further configured to construct an initial drift curve according to a displacement difference between a first spatial domain image and a last spatial domain image in the image series.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images