A full-process automated ultra-high resolution image registration method and device

Abstract

The present invention discloses a full-process automated super-high-resolution image registration method and apparatus. The method includes: reading and aligning super-high-resolution images to generate low-resolution image pairs, performing enhancement and background removal processing on the low-resolution image pairs, further extracting feature points and descriptors of the images using the SIFT algorithm, performing feature matching through the FLANN algorithm, and estimating the affine transformation matrix for low-resolution image registration; combining the scaling factors of the super-high-resolution and low-resolution images, adjusting the translation part parameters of the transformation matrix to obtain the affine transformation matrix on the super-high-resolution image; and finally completing the registration of the super-high-resolution image. The method of the present invention has the characteristics of full automation, high processing efficiency, and high precision, is applicable to the field of super-high-resolution image processing, and has broad application prospects especially in the precise registration of high-resolution images such as remote sensing images, medical images, and astronomical images.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision and image processing, and in particular, to a full-process automated super-high resolution image registration method and apparatus. Background Art

[0002] With the development of computer vision and image processing technologies, image registration technology has been widely applied in multiple fields (such as remote sensing, medical images, autonomous driving, etc.). The core of image registration is to align two or more images through spatial transformation so that the corresponding parts of the same object or scene in different images are precisely matched. As the image resolution continues to increase, the demand for super-high resolution images is increasing day by day. Traditional image registration methods face many challenges, especially when dealing with large-scale super-high resolution image data, there are many problems in terms of computational efficiency and memory consumption.

[0003] Commonly used image registration methods can generally be divided into four types: feature-based registration, region-based registration, transformation-based registration, and deep learning-based registration methods. Although existing methods have solved some problems in image registration to a certain extent, when facing super-high resolution images, there are still various challenges. Memory consumption problem: The size of super-high resolution images is usually very large, and memory consumption becomes a bottleneck in the image registration process. Traditional registration methods (such as using cv2.warpAffine for image transformation) are difficult to process large images with extremely high memory requirements, resulting in low efficiency in the registration process or even being unable to perform. Computational efficiency: As the image resolution increases, the computational complexity in the image registration process increases significantly. Especially when performing global optimization and feature matching, the amount of computation and time consumption are huge, making it difficult to meet the requirements of real-time or large-scale image processing. Adaptability problem: Existing registration methods usually assume that the transformation between images is relatively simple, while super-high resolution images usually involve more complex geometric transformations (such as rotation, scaling, perspective transformation, etc.). The accurate estimation of these transformations and the manual adjustment of parameters are crucial for registration accuracy, and the robustness of existing methods is poor in this case.

[0004] In view of the above problems, super-high resolution image registration technology urgently needs an efficient and robust solution that can break through the bottlenecks of memory and computational efficiency, and while ensuring high precision, improve the computational speed and adaptability. In particular, a method is needed that can perform fast preprocessing on low-resolution images to obtain a transformation matrix, and on this basis, adjust the transformation matrix to adapt to super-high resolution images, thereby solving the problems of excessive memory and computational amount. Summary of the Invention

[0005] In view of the deficiencies of the prior art, the purpose of the present invention is to provide a full - process automated ultra - high - resolution image registration method and device, which are used to solve the memory limitation and computational efficiency problems in traditional registration methods.

[0006] The purpose of the present invention is achieved through the following technical solutions: A full - process automated ultra - high - resolution image registration method and device, the method comprising the following steps:

[0007] (1) Ultra - high - resolution image reading and pre - processing: Read the ultra - high - resolution image pair, standardize and align the sizes of the image pair to ensure that the images are compared and registered at the same scale. Generate a low - resolution image pair with a resolution reduced to one - Nth through downsampling technology. Perform a series of image enhancement processes on the low - resolution images to improve the image quality. Then, perform background removal operations to eliminate irrelevant image regions; The ultra - high - resolution image pair includes an ultra - high - resolution floating image and an ultra - high - resolution reference image; The low - resolution image pair includes a low - resolution floating image and a low - resolution reference image;

[0008] (2) Extract feature points and descriptors of the pre - processed low - resolution image pair through the SIFT algorithm and use the FLANN algorithm for feature matching. Subsequently, use the affine transformation matrix estimation algorithm to perform translation and rotation transformations on the pre - processed low - resolution floating image to accurately align it with the pre - processed low - resolution reference image; Finally, obtain the affine transformation matrix for low - resolution image registration and complete the low - resolution image registration process;

[0009] (3) According to the affine transformation matrix of the low - resolution image, combined with the scaling factor between the ultra - high - resolution and low - resolution images, adjust the parameters of the translation part in the transformation matrix to obtain the affine transformation matrix on the ultra - high - resolution image; Based on the affine transformation matrix on the ultra - high - resolution image, obtain its inverse affine transformation matrix; The registration methods for the ultra - high - resolution floating image and the reference image include:

[0010] By cutting the target registration image into blocks, calculating its coordinates in the ultra - high - resolution floating image using the inverse affine transformation matrix, and efficiently copying the pixel values at the corresponding positions to the target registration image in combination with multi - threading technology; or

[0011] By processing the four vertex coordinates after cutting the target registration image into blocks, combining affine transformation estimation and coordinate system transformation, and finally converting it into the affine transformation of the ultra - high - resolution floating image block and performing in - memory block copying to generate the target registration image.

[0012] Further, the step (1) includes the following sub - steps:

[0013] (1.1) Read the ultra - high - resolution image pair, where the size of the ultra - high - resolution floating image is W hm ×Hhm The size of the ultra-high resolution reference image is W hf ×H hf , and they usually differ in resolution and need to be made the same size by padding or cropping. In principle, if the aspect ratio of the ultra-high resolution floating image is smaller than that of the ultra-high resolution reference image, additional background areas need to be padded in four directions. If the aspect ratio of the ultra-high resolution floating image is larger than that of the ultra-high resolution reference image, redundant parts need to be cropped in four directions, mainly to ensure that the two images are spatially aligned. The size of the aligned image pair should be W hf ×H hf .

[0014] (1.2) The ultra-high resolution image pair is downsampled by a factor of N (N is usually 16, 32, or 64) using downsampling technology to obtain a low resolution image pair, where the image size is W lf ×H lf . Different image enhancement techniques are applied to the low resolution image pair respectively. The color saturation and brightness of the low resolution floating image are enhanced to make its image colors more vivid and bright, better highlighting the colors of the tissue structure and image details. The contrast and sharpness of the low resolution reference image are enhanced to make the light and dark differences in the image more obvious and improve the clarity and recognition. Then, the low resolution image pair is converted from the BGR color space to the HSV color space. The upper and lower limits of the HSV threshold for white (background color) are set to (180, 20, 255) and (0, 0, 220) respectively. Threshold processing is used to create white masks respectively, and denoising processing is carried out to remove small noise points. After that, the masks are inverted so that the background area becomes black and the non-background area becomes white, preparing for subsequent operations. To further accurately extract regions, multiple regions that meet the conditions are merged and small regions are removed. Morphological closing operations are used to fill the holes in the masks to make the masks more complete. Finally, the regions of interest are extracted from the original images through mask operations, removing the background and retaining the target regions, thereby improving the processing efficiency and optimizing the accuracy of subsequent analysis.

[0015] Furthermore, step (2) includes the following sub-steps:

[0016] (2.1) Use the SIFT (Scale-Invariant Feature Transform) algorithm to extract feature points and compute descriptors for the low-resolution floating image and the low-resolution reference image. Specifically, by creating a cv2.SIFT_create object and calling the detectAndCompute method, obtain the feature points (kp1, kp2) and their corresponding descriptors (des1, des2) of the low-resolution floating image and the low-resolution reference image. Then, use the FLANN (Fast Library for Approximate Nearest Neighbors) algorithm for feature matching. Create a matcher through cv2.FlannBasedMatcher, set the index parameter to use the K-D tree algorithm, set the search parameter to check 500 times, and then use the knnMatch method for feature matching. By screening the matching results, select the good matches that meet the distance threshold (0.85) to provide reliable matching points for the subsequent transformation matrix estimation.

[0017] (2.2) If the number of good matching points is greater than or equal to 4, then use these matching points to estimate the affine transformation matrix M between the low-resolution floating image and the low-resolution reference image through cv2.estimateAffinePartial2D. l , M l contains transformation parameters such as translation, rotation, and scaling. Finally, use cv2.warpAffine combined with the matrix M l to register the low-resolution floating image and the reference image to make them precisely aligned.

[0018] Further, according to the affine transformation matrix of the low-resolution image, combined with the scaling factors of the super-high-resolution and low-resolution images, adjust the parameters of the translation part in the affine transformation matrix to obtain the affine transformation matrix on the super-high-resolution image, including:

[0019] Based on the affine transformation matrix M obtained in step (2) l , M l is a 2×3 matrix, , the parameter controls the linear part of the affine transformation, including rotation, scaling, and cropping, etc., controls the translation of the image. Therefore, if a super-high-resolution transformation is to be performed, the parameters of the translation part in M l need to be modified. Calculate the scaling ratios in the width and height directions as sf w and sf h respectively through the width and height of the super-high-resolution and low-resolution images. Thus, the affine transformation matrix on the super-high-resolution image can be obtained.

[0020] Further, based on the affine transformation matrix on the super-high-resolution image , expand it to a 3×3 matrix by adding its secondary coordinates, calculate its inverse matrix, and extract the first two rows of the inverse matrix to obtain a 2×3 inverse affine transformation matrix M h_inv .

[0021] Furthermore, by cutting the target registration image into blocks, using the inverse affine transformation matrix to calculate its coordinates in the ultra-high resolution floating image, and combining multi-threading technology to efficiently copy the pixel values at the corresponding positions to the target registration image, including:

[0022] Method 1: First, create a blank target registration image with a size of W hf ×H hf , which is the same size as the image after alignment with the ultra-high resolution image. Cut the blank target registration image into blocks with a block size of W b ×H b , obtain the corresponding coordinates of the pixels in each image block in the target registration image and convert them into homogeneous coordinate format to generate a matrix of coordinates to be processed. Through the dot product with the inverse affine transformation matrix M h_inv , obtain the corresponding coordinates in the ultra-high resolution floating image. At the same time, process the out-of-bounds coordinates and perform rounding operations to obtain the corresponding coordinate indices on the effective ultra-high resolution floating image. Finally, copy its pixel values to the corresponding positions on the target registration image. Use multi-threading technology to process all the image blocks, and finally obtain the ultra-high resolution registered image.

[0023] Furthermore, by processing the four vertex coordinates after cutting the target registration image into blocks, combining affine transformation estimation and coordinate system transformation, and finally converting it into the affine transformation of the ultra-high resolution floating image block and performing in-memory block copying to generate the target registration image, including:

[0024] Method 2: First, create a blank target registration image with a size of W hf ×H hf , which is the same size as the image after alignment with the ultra-high resolution image. Cut the blank target registration image into blocks with a block size of W b ×H b , obtain the original coordinate matrix C of the four vertices in the image block in the target registration image out and convert it into homogeneous coordinate format. Through the dot product with the inverse affine transformation matrix M h_inv , obtain the coordinate matrix C of the corresponding vertices in the ultra-high resolution floating image m , find the minimum values (C min_x , C min_y ) and the maximum values (C max_x , C max_y ) of its coordinates in the x and y directions. Obtain a new matrix C through normalizing the coordinate system m_nIn the same way, the normalized coordinate matrix C out 's coordinate system is used to obtain a new matrix C out_n , and C m_n is obtained through cv2.estimateAffinePartial2D. The affine transformation relationship matrix M out_n between C c can be obtained. Then, the image patches cropped from the super-high-resolution floating image can be transformed through cv2.warpAffine. The new image obtained corresponds to the image patches in the target registration image. Therefore, the pixel values of the image patches can be copied to their corresponding positions by means of memory copy. The corresponding other image patches are also obtained using the same operation method as above, and the combination of multi-threaded technology is more efficient, greatly improving the calculation efficiency compared with Method 1.

[0025] Furthermore, the affine transformation relationship matrix M c can be obtained not only by estimating the affine transformation but also by means of perspective transformation, direct linear transformation, least squares fitting, etc.

[0026] The present invention also provides a full-process automated super-high-resolution image registration device, including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, it is used to implement the above-mentioned full-process automated super-high-resolution image registration method.

[0027] The present invention also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements the above-mentioned full-process automated super-high-resolution image registration method.

[0028] The beneficial effects of the present invention are as follows: The present invention proposes an efficient and robust registration method for the problems of memory consumption, calculation efficiency, and adaptability in super-high-resolution image registration. This method performs rapid preprocessing on the low-resolution image to obtain a preliminary transformation matrix, and on this basis, adjusts the transformation matrix to adapt to the super-high-resolution image. Two methods for super-high-resolution image registration are introduced. Method 1 performs pixel-by-pixel coordinate mapping through the inverse affine transformation matrix and fills the target image, while Method 2 processes the four vertex coordinates of the image patches, combines affine transformation estimation and image patch affine transformation, providing a more efficient image registration method. These steps effectively break through the memory and calculation efficiency bottlenecks in the super-high-resolution image registration process, avoid the processing difficulties or low efficiency problems caused by excessive memory requirements in traditional methods, have significant practical value, are applicable to the field of super-high-resolution image processing, and have broad application prospects especially in the precise registration of high-resolution images such as remote sensing images, medical images, and astronomical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0030] Figure 1 It is a schematic flowchart of a full - process automated super - high - resolution image registration method provided by an embodiment of the present invention;

[0031] Figure 2 It is a structural block diagram of a full - process automated super - high - resolution image registration device provided by an embodiment of the present invention. Specific embodiments

[0032] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0033] It should be noted that, without conflict, the features in the following embodiments and implementation manners can be combined with each other.

[0034] Figure 1 It is a schematic flowchart of a full - process automated super - high - resolution image registration method provided by an embodiment of the present invention. As Figure 1 shown, the full - process automated super - high - resolution image registration method in the embodiments of the present invention may include the following steps:

[0035] Step S1: Reading and pre - processing of super - high - resolution pathological images HE and IHC, reading the image pair, standardizing and aligning the sizes of HE and IHC images to ensure that the image sizes are the same, and obtaining low - resolution HE and IHC images through down - sampling technology or image pyramids. Performing a series of image enhancement processes on the low - resolution images to improve the image quality. Then, performing background removal operations to eliminate irrelevant image regions;

[0036] Specifically, the step S1 includes the following sub - steps:

[0037] Step S11: Read the pathological images HE (Hematoxylin and Eosin) and IHC (Immunohistochemistry) respectively. The sizes of the images are 59887×49495 and 59844×49491 respectively. Mark the HE image as the floating image and the IHC image as the reference image. Keep the size of the HE image the same as that of the IHC image by padding or cropping. The principle of padding and cropping is that if the size of the HE image is smaller than that of the IHC image, additional background areas need to be padded in four directions; if the size of the HE image is larger than that of the IHC image, redundant background parts need to be cropped in four directions to ensure that the sizes of the HE image and the IHC image are both 59844×49491;

[0038] Step S12: Obtain pairs of low-resolution HE images and low-resolution IHC images that are reduced to 1 / N of the HE and IHC images through downsampling or image pyramids, where the sizes of the images are both 935×773. Enhance the color and brightness of the low-resolution HE image. Use ImageEnhance.Color to enhance the color saturation of the image, and set the enhancement factor to 1.2 to make the image color more vivid. Then use ImageEnhance.Brightness to enhance the brightness of the image, and also set the enhancement factor to 1.2 to make the image brighter and improve the visibility of the image. Enhance the contrast and sharpness of the low-resolution IHC image. Use ImageEnhance.Contrast to enhance the contrast of the image, and set the enhancement factor to 2.0 to make the light and dark differences in the image more obvious. Then, use ImageEnhance.Sharpness to enhance the sharpness of the image, and set the enhancement factor to 2.0 to make the details of the image clearer. Convert the low-resolution HE and IHC images from the BGR color space to the HSV color space. Set the upper and lower limits of the HSV threshold of white (background color) to (180, 20, 255) and (0, 0, 220) respectively, and use cv2.inRange for threshold processing to create white masks respectively. Use morphological opening operation to remove the noise in the masks. The opening operation uses a 5×5 kernel, and small areas in the masks are removed through erosion and dilation operations, and larger connected areas are retained. Then, invert the denoised white masks. By specifying an area threshold of 5000, filter out larger areas and merge them into a new mask. Use a 11×11 kernel for morphological closing operation to fill the small holes in the mask to make the mask more complete. Finally, through the mask images obtained above, perform background removal operations on the low-resolution HE and IHC images respectively to obtain low-resolution HE and IHC images that only contain the target foreground areas.

[0039] Step S2: For the preprocessed low-resolution HE image and low-resolution IHC image in Step S1, extract feature points and descriptors using the SIFT (Scale-Invariant Feature Transform) algorithm and perform feature matching using the FLANN (Fast Library for Approximate Nearest Neighbors) algorithm. Subsequently, use the affine transformation matrix estimation algorithm to perform translation and rotation transformations on the preprocessed low-resolution HE image to accurately align it with the preprocessed low-resolution IHC image. Finally, obtain the affine transformation matrix M l and complete the registration process of the low-resolution HE image and the low-resolution IHC image;

[0040] Specifically, the said Step S2 includes the following sub-steps:

[0041] Step S21: Use the SIFT (Scale-Invariant Feature Transform) algorithm to extract feature points and calculate descriptors for the preprocessed low-resolution HE image and low-resolution IHC image. By creating a cv2.SIFT_create object and calling the detectAndCompute method, obtain the feature points (kp1, kp2) and their corresponding descriptors (des1, des2) of the low-resolution HE image and low-resolution IHC image. Then, use the FLANN (Fast Library for Approximate Nearest Neighbors) algorithm for feature matching. Create a matcher through cv2.FlannBasedMatcher, set the index parameter to use the K-D tree algorithm, set the search parameter to check 500 times, and then use the knnMatch method for feature matching. Traverse the matching results and filter out good matches. For each pair of matching feature points, if the distance of the first nearest neighbor feature point is less than 0.85 times the distance of the second nearest neighbor feature point, it is considered a good match and added to the good_matches list to provide reliable matching points for subsequent transformation matrix estimation;

[0042] Step S22: If the number of good matching points is less than 4, it means there are insufficient matching points and registration cannot be performed. If the number of good matching points is greater than or equal to 4, then these good matching points can be used to estimate the affine transformation matrix M between the low-resolution HE image and the low-resolution IHC image through cv2.estimateAffinePartial2D l , M lIt contains transformation parameters such as translation, rotation, and scaling. Finally, the cv2.warpAffine function is used to apply the translation and rotation transformations for registration, transforming the preprocessed low-resolution HE image to the position aligned with the preprocessed low-resolution IHC image, and obtaining the registered low-resolution HE image. In this way, the entire process of low-resolution image registration is completed, enabling the precise alignment of the preprocessed low-resolution HE image with the low-resolution IHC image, providing a basis for subsequent image analysis and research.

[0043] Step S3: According to the affine transformation matrix M l , and the widths and heights of the ultra-high-resolution and low-resolution images, calculate the scaling ratios in the width and height directions as sf w and sf h . By modifying the parameters of the translation part in M l , obtain the affine transformation matrix M h on the ultra-high-resolution image. Two methods for registering ultra-high-resolution HE and IHC images are proposed, and either one can be used; Method 1 cuts the target registration image into blocks, calculates its coordinates in the ultra-high-resolution HE image using the inverse affine transformation matrix, and efficiently copies the pixel values at the corresponding positions to the target registration image by combining multi-threading technology. Method 2 processes the four vertex coordinates after cutting the target registration image into blocks, combines affine transformation estimation and coordinate system transformation, and finally converts it into the affine transformation of the image block and performs in-memory block copying to generate the target registration image.

[0044] Specifically, the step S3 includes the following sub-steps:

[0045] Step S31: According to the parameter characteristics of the affine transformation matrix M l obtained in step S22, it is necessary to modify the parameters of the translation part in M l to apply it to the registration of ultra-high-resolution HE images and ultra-high-resolution IHC images. According to the widths and heights of the ultra-high-resolution image and the low-resolution image, calculate the scaling ratios in the width and height directions as sf w and sf h . Thus, the affine transformation matrix M h on the ultra-high-resolution image can be obtained. Expand it into a 3×3 matrix by adding homogeneous coordinates, calculate its inverse matrix, and extract the first two rows of the inverse matrix to obtain the 2×3 inverse affine transformation matrix M h_inv .

[0046] Step S32: The following introduces two methods for registering ultra-high-resolution HE and IHC images.

[0047] Method 1: First, create a blank target registration image with a size of W hf × H hfFor the blank target registration image patch, the patch size is W b ×H b , obtain the corresponding coordinates of the pixels in each image patch in the target registration image and convert them into homogeneous coordinate format, generate the coordinate matrix to be processed, and through the dot product with the inverse affine transformation matrix M h_inv , obtain the corresponding coordinates in the ultra-high resolution HE image, and at the same time process the out-of-bounds coordinates and perform rounding and integer-taking operations to obtain the corresponding coordinate indices on the effective ultra-high resolution HE image. Finally, copy its pixel values to the corresponding positions on the target registration image. Use multi-threaded technology to process all image patches, and finally obtain the ultra-high resolution registration image.

[0048] Method 2: First, create a blank target registration image with a size of W hf ×H hf . For the blank target registration image patch, the patch size is W b ×H b , obtain the original coordinate matrix C of the four vertices in the image patch in the target registration image out and convert it into homogeneous coordinate format. Through the dot product with the inverse affine transformation matrix M h_inv , obtain the coordinate matrix C of the corresponding vertices in the ultra-high resolution HE image m , find the minimum values (C min_x , C min_y ) and the maximum values (C max_x , C max_y ) of its coordinates in the x and y directions, subtract the coordinates in the x and y directions in the coordinate matrix C m from C min_x and C min_y respectively, and then obtain a new coordinate matrix C m_n in this way of normalizing the coordinate system. In the same way, normalize the coordinate system of the coordinate matrix C out to obtain a new matrix C out_n , and obtain the affine transformation relationship matrix M m_n between C out_n and C c through cv2.estimateAffinePartial2D. Then, the upper left corner (C min_x , C min_y ) and the lower right corner (C max_x , C max_y) The image at the position is transformed through cv2.warpAffine, and the obtained new image corresponds to the image patch in the target registration image. Therefore, the pixel values of the image patch can be copied to the corresponding position in the target registration image by means of memory copying. The corresponding other image patches are also obtained using the same operation method as above, and the efficiency is higher in combination with multi-threaded technology. Compared with Method 1, the calculation efficiency is greatly improved. The above transformation matrix M c can be obtained not only by estimating the affine transformation but also by means of perspective transformation, direct linear transformation, least squares fitting, etc.

[0049] The present invention also provides a computer-readable storage medium storing a computer program that can be used to execute the Figure 1 provided full-process automated super-high-resolution image registration method.

[0050] The present invention also provides Figure 2 a schematic structural diagram of a full-process automated super-high-resolution image registration device corresponding to Figure 1 as shown. As Figure 2 described, at the hardware level, the full-process automated super-high-resolution image registration device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the Figure 1 described data acquisition method. Of course, in addition to the software implementation method, the present invention does not exclude other implementation methods, such as logical devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logical unit, and can also be hardware or a logical device.

[0051] Improvements to a technology can be clearly distinguished as either hardware improvements (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many method flow improvements today can be regarded as direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement to a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is an integrated circuit whose logical function is determined by the user programming the device. Designers can program themselves to "integrate" a digital system onto a single PLD, without having to ask a chip manufacturer to design and fabricate a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, which is similar to the software compiler used in program development and writing. The original code before compilation also has to be written in a specific programming language, which is called a hardware description language (HDL), and there is not just one type of HDL, but many types, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones currently are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that by simply performing a little logical programming on the method flow using the above-mentioned several hardware description languages and programming it into the integrated circuit, it is easy to obtain the hardware circuit that implements the logical method flow.

[0052] The controller can be implemented in any suitable manner. For example, the controller can take the form of, for example, a microprocessor or a processor and a computer-readable medium storing computer-readable program code (such as software or firmware) executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of the controller include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in the form of pure computer-readable program code, it is entirely possible to logically program the method steps to enable the controller to be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. to achieve the same function. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be regarded as the structures within the hardware component. Or even, the devices for implementing various functions can be regarded as either software modules for implementing the method or the structures within the hardware component.

[0053] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0054] For the convenience of description, when describing the above devices, they are described separately as various units according to their functions. Of course, when implementing the present invention, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0055] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program code.

[0056] The present invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each flow and / or block of the flowchart illustrations and / or block diagrams, and combinations of flows and / or blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart Figure 1 one flow or more flows and / or blocks Figure 1 or means for implementing the functions specified in one block or more blocks.

[0057] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instruction means that implement the functions specified in the flowchart Figure 1 one flow or more flows and / or blocks Figure 1 or means for implementing the functions specified in one block or more blocks.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process, such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one flow or more flows and / or blocks Figure 1 or means for implementing the functions specified in one block or more blocks.

[0059] In a typical configuration, a computing device includes one or more processors (CPUs), an input / output interface, a network interface, and memory.

[0060] The memory may include non-permanent memory in the form of computer readable media, random access memory (RAM), and / or non-volatile memory such as read only memory (ROM) or flash memory (flash RAM). Memory is an example of computer readable media.

[0061] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0062] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0063] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0065] Each embodiment in the present invention is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, reference can be made to the partial description of the method embodiment.

[0066] The above are only the embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and changes can be made to the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims

1. A full-process automated ultra-high resolution image registration method, characterized in that: The following steps are involved: (1) Read ultra-high-resolution image pairs, standardize and align the sizes of the image pairs, generate low-resolution image pairs by downsampling, and perform preprocessing; (2) The feature points and descriptors of the preprocessed low-resolution image pair are extracted by the SIFT algorithm and the feature matching is performed using the FLANN algorithm. Then, the preprocessed low-resolution floating image is translated and rotated using the affine transformation matrix estimation algorithm to align it with the preprocessed low-resolution reference image. Finally, the affine transformation matrix is ​​obtained and the registration process of the low-resolution image is completed. (3) According to the affine transformation matrix of the low-resolution image, combined with the scaling factors of the ultra-high-resolution and low-resolution images, the parameters of the translation part in the affine transformation matrix are adjusted to obtain the affine transformation matrix on the ultra-high-resolution image; Obtaining an inverse affine transformation matrix based on the affine transformation matrix on the ultra-high resolution image; The registration methods of super-high resolution floating images and super-high resolution reference images include: By cutting the target registration image into blocks, using the inverse affine transformation matrix to calculate its coordinates in the ultra-high resolution floating image, and combining multi-threading technology to copy the pixel values ​​of the corresponding positions to the target registration image; or The target registration image is generated by processing the coordinates of the four vertices of the target registration image after slicing, combining affine transformation estimation and coordinate system transformation, and finally converting it into the affine transformation of the ultra-high resolution floating image block and performing memory block copying.

2. A fully automated ultra-high resolution image registration method according to claim 1, characterized in that: The step (1) comprises the following sub-steps: (1.1) Obtain a super-high-resolution floating image and a super-high-resolution reference image, i.e., a super-high-resolution image pair; and unify their sizes by padding or cropping to ensure that the two images are spatially aligned. The size of the aligned image pair is W hf ×H hf ; (1.2) The super-high-resolution image pair is reduced to one-N by downsampling technology to obtain a low-resolution image pair with a size of W lf ×H lf ; Enhance the color saturation and brightness of the low-resolution floating image, and enhance the contrast and sharpness of the low-resolution reference image; Convert the low-resolution image pair from BGR color space to HSV color space, and perform mask creation and denoising of the white background area; The mask is inverted and morphological closing is performed to fill the holes in the mask to accurately extract the region of interest, remove the background and retain the target area.

3. The fully automated ultra-high resolution image registration method according to claim 1, characterized in that: The step (2) includes the following sub-steps: (2.1) Use the SIFT algorithm to extract feature points and calculate descriptors of the low-resolution floating image and the low-resolution reference image to obtain their feature points and descriptors; use the FLANN algorithm to perform feature matching and screen out good matching points that meet the distance threshold; (2.2) Based on these good matching points, the affine transformation matrix M between the low-resolution floating image and the low-resolution reference image is estimated by the cv2.estimateAffinePartial2D method l , using the estimated affine transformation matrix M l , the low-resolution float image and the low-resolution reference image are registered via cv2.warpAffine so that they are precisely aligned.

4. The method for full-process automated ultra-high resolution image registration according to claim 1, characterized in that: According to the affine transformation matrix of the low-resolution image, combined with the scaling factors of the ultra-high-resolution and low-resolution images, the parameters of the translation part in the affine transformation matrix are adjusted to obtain the affine transformation matrix on the ultra-high-resolution image, including: Based on the affine transformation matrix obtained in step (2) , and calculate the scaling factor sf according to the aspect ratio of the super-resolution and low-resolution images w and sf h , by modifying M l Parameters that control translation , get the affine transformation matrix on the super-high resolution image .

5. A fully automated ultra-high resolution image registration method according to claim 4, characterized in that: Based on the affine transformation matrix on the ultra-high resolution image, its inverse affine transformation matrix is ​​obtained, including: Affine transformation matrix based on ultra-high resolution images , by adding homogeneous coordinates to expand to a 3×3 matrix, calculating its inverse matrix, and extracting the first two rows of the inverse matrix to obtain a 2×3 inverse affine transformation matrix M h_inv .

6. The fully automated ultra-high resolution image registration method according to claim 1, characterized in that: By cutting the target registration image into blocks, using the inverse affine transformation matrix to calculate its coordinates in the ultra-high resolution floating image, and combining multi-threading technology to efficiently copy the pixel values ​​of the corresponding positions to the target registration image, including: Create a blank target registration image of size W hf ×H hf , align with the super-high resolution image, and cut the target registration image into blocks with a block size of W b ×H b ; Convert the matrix composed of the coordinates of all pixels in the image block into homogeneous coordinate format and the inverse affine transformation matrix M h_inv Perform dot multiplication to obtain the corresponding coordinates in the ultra-high resolution floating image, perform boundary checking and rounding operations; copy the pixel values ​​in the ultra-high resolution floating image to the target registration image, and process the image blocks through multi-threading technology to improve processing efficiency.

7. The fully automated ultra-high resolution image registration method according to claim 1, characterized in that: By processing the coordinates of the four vertices of the target registration image after slicing, combining affine transformation estimation and coordinate system transformation, and finally converting it into the affine transformation of the ultra-high resolution floating image block and performing a memory block copy, the target registration image is generated, including: Create a blank target registration image of size W hf ×H hf , align with the super-high resolution image, cut the target registration image into blocks, obtain the original coordinates of the four vertices in the image block in the target registration image to form the matrix C out , and convert it into homogeneous coordinate format by and inverse affine transformation matrix M h_inv The dot product of the corresponding vertex in the ultra-high resolution floating image is obtained by m , by normalizing the coordinate system, we get the matrix C m_n ; Similarly, normalize C out The coordinate system of the matrix C out_n , obtain C through cv2.estimateAffinePartial2D m_n and C out_n The affine transformation relationship matrix M between c , and then the image block can be affine transformed by cv2.warpAffine; the transformed image is copied to the corresponding position of the image block in the target registration image; the corresponding other image blocks are also obtained using the same operation method as above, and all image blocks are efficiently processed in combination with multi-threading technology.

8. The full-process automated ultra-high resolution image registration method according to claim 7, characterized in that: The affine transformation relationship matrix M c Other ways to obtain include perspective transformation, direct linear transformation or least squares fitting.

9. A fully automated ultra-high resolution image registration device, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a full-process automated ultra-high resolution image registration method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a full-process automated ultra-high resolution image registration method as described in any one of claims 1-8 is implemented.

Images