Full-process automatic ultrahigh-resolution image registration method and device

Abstract

The invention discloses a full-process automatic ultrahigh-resolution image registration method and device, and the method comprises the steps: carrying out the reading and alignment operation of an ultrahigh-resolution image, generating a low-resolution image pair, carrying out the enhancement and background removal processing of the low-resolution image pair, further extracting the feature points and descriptors of the image through employing an SIFT algorithm, and carrying out the recognition of the feature points and descriptors. Feature matching is carried out through an FLANN algorithm, and an affine transformation matrix of low-resolution image registration is estimated; adjusting translation part parameters of the transformation matrix by combining the scaling factors of the ultrahigh-resolution image and the low-resolution image to obtain an affine transformation matrix on the ultrahigh-resolution image; and finally finishing the registration of the ultrahigh-resolution image. The method has the advantages of being full-automatic, high in processing efficiency and high in precision, is suitable for the field of ultrahigh-resolution image processing, and particularly has wide application prospects in precise registration of high-resolution images such as remote sensing images, medical images and astronomical images.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a full-process automated ultra-high resolution image registration method and device. Background Art

[0002] With the development of computer vision and image processing technology, image registration technology has been widely used in many fields (such as remote sensing, medical imaging, autonomous driving, etc.). The core of image registration is to align two or more images through spatial transformation so that the corresponding parts of the same object or scene in different images can be accurately matched. As image resolution continues to increase, the demand for ultra-high resolution images is increasing. Traditional image registration methods face many challenges, especially when processing large-scale ultra-high resolution image data, there are many difficulties in computational efficiency and memory consumption.

[0003] Commonly used image registration methods can be roughly divided into four types: feature-based registration, region-based registration, transformation-based registration, and deep learning-based registration methods. Although existing methods have solved some problems in image registration to a certain extent, there are still various challenges when facing ultra-high resolution images. Memory consumption problem: The size of ultra-high resolution images is usually very large, and memory consumption becomes a bottleneck in the image registration process. Traditional registration methods (such as using cv2.warpAffine for image transformation) are difficult to handle large images with extremely high memory requirements, resulting in inefficient registration process or even inability to proceed. Computational efficiency: With the increase of image resolution, the computational complexity of the image registration process increases significantly, especially when performing global optimization and feature matching. The amount of calculation and time consumption are huge, which is difficult to meet the needs of real-time or large-scale image processing. Adaptability problem: Existing registration methods usually assume that the transformation between images is relatively simple, while ultra-high resolution images usually involve more complex geometric transformations (such as rotation, scaling, perspective transformation, etc.). The accurate estimation of these transformations and the manual adjustment of parameters are crucial to the registration accuracy, and the existing methods are less robust in this case.

[0004] In view of the above problems, ultra-high resolution image registration technology urgently needs an efficient and robust solution that can break through the bottleneck of memory and computing efficiency, and improve computing speed and adaptability while ensuring high accuracy. In particular, a method is needed to quickly preprocess low-resolution images to obtain transformation matrices, and then adjust the transformation matrices to adapt to ultra-high resolution images, thereby solving the problems of excessive memory and computing. Summary of the invention

[0005] In view of the deficiencies in the prior art, an object of the present invention is to provide a full-process automated ultra-high resolution image registration method and device to solve the problems of memory limitation and computational efficiency in traditional registration methods.

[0006] The object of the present invention is achieved through the following technical solution: a full-process automated ultra-high resolution image registration method and device, the method comprising the following steps:

[0007] (1) Reading and preprocessing ultra-high resolution images: reading ultra-high resolution image pairs, standardizing and aligning the sizes of the image pairs to ensure that the images are compared and aligned at the same scale, and generating low-resolution image pairs with a resolution reduced to one-Nth through downsampling technology. Perform a series of image enhancement processes on the low-resolution images to improve image quality. Next, perform background removal operations to eliminate irrelevant image areas; ultra-high resolution image pairs include ultra-high resolution floating images and ultra-high resolution reference images; low-resolution image pairs include low-resolution floating images and low-resolution reference images;

[0008] (2) The feature points and descriptors of the preprocessed low-resolution image pair are extracted by the SIFT algorithm and the feature matching is performed using the FLANN algorithm. Subsequently, the preprocessed low-resolution floating image is translated and rotated using the affine transformation matrix estimation algorithm to accurately align it with the preprocessed low-resolution reference image. Finally, the affine transformation matrix for low-resolution image registration is obtained and the low-resolution image registration process is completed.

[0009] (3) According to the affine transformation matrix of the low-resolution image, combined with the scaling factors of the ultra-high-resolution and low-resolution images, the parameters of the translation part in the transformation matrix are adjusted to obtain the affine transformation matrix on the ultra-high-resolution image; based on the affine transformation matrix on the ultra-high-resolution image, its inverse affine transformation matrix is ​​obtained; the registration method of the ultra-high-resolution floating image and the reference image includes:

[0010] By cutting the target registration image into blocks, using the inverse affine transformation matrix to calculate its coordinates in the ultra-high resolution floating image, and combining multi-threading technology to efficiently copy the pixel values ​​of the corresponding positions to the target registration image; or

[0011] The target registration image is generated by processing the coordinates of the four vertices of the target registration image after slicing, combining affine transformation estimation and coordinate system transformation, and finally converting it into the affine transformation of the ultra-high resolution floating image block and performing memory block copying.

[0012] Furthermore, the step (1) includes the following sub-steps:

[0013] (1.1) Read super-high resolution image pairs, where the super-high resolution floating image size is W hm ×Hhm , the size of the super-high resolution reference image is W hf ×H hf , they usually have different resolutions and need to be padded or cropped to make them uniform in size. In principle, if the aspect ratio of the super-high resolution floating image is smaller than that of the super-high resolution reference image, it is necessary to fill in additional background areas in four directions. If the aspect ratio of the super-high resolution floating image is larger than that of the super-high resolution reference image, it is necessary to crop the excess parts in four directions. The main purpose is to ensure that the two images are aligned in space. The size of the aligned image pair should be W hf ×H hf .

[0014] (1.2) The super-high-resolution image pair is reduced to one-N times (N is usually 16, 32 or 64) by downsampling technology to obtain a low-resolution image pair, where the image size is W lf ×H lf . Different image enhancement techniques are used for low-resolution image pairs to enhance the color saturation and brightness of low-resolution floating images, making the image colors more vivid and bright, and better highlighting the colors and image details of the tissue structure. The contrast and sharpness of the low-resolution reference image are enhanced to make the image light and dark differences more obvious and improve clarity and recognition. Then the low-resolution image pairs are converted from the BGR color space to the HSV color space, and the upper and lower limits of the threshold of the white (background color) HSV are set to (180, 20, 255) and (0, 0, 220) respectively. White masks are created using threshold processing, and denoising is performed to remove small noise points. After that, the mask is inverted to make the background area black and the non-background area white, in preparation for subsequent operations. In order to further accurately extract the area, multiple areas that meet the conditions are merged and small areas are removed. The morphological closing operation is used to fill the holes in the mask to make the mask more complete. Finally, the area of ​​interest is extracted from the original image through the mask operation, the background is removed, and the target area is retained, thereby improving the processing efficiency and optimizing the accuracy of subsequent analysis.

[0015] Furthermore, the step (2) includes the following sub-steps:

[0016] (2.1) Use the SIFT (Scale Invariant Feature Transform) algorithm to extract feature points and calculate descriptors for the low-resolution floating image and the low-resolution reference image. Specifically, by creating a cv2.SIFT_create object and calling the detectAndCompute method, the feature points (kp1, kp2) of the low-resolution floating image and the low-resolution reference image and their corresponding descriptors (des1, des2) are obtained. Next, the FLANN (Fast Nearest Neighbor Search Package) algorithm is used for feature matching. The matcher is created by cv2.FlannBasedMatcher, and the index parameter is set to use the KD tree algorithm. The search parameter sets the number of checks to 500, and then the knnMatch method is used for feature matching. By screening the matching results, good matches that meet the distance threshold (0.85) are selected to provide reliable matching points for subsequent transformation matrix estimation.

[0017] (2.2) If the number of good matching points is greater than or equal to 4, these matching points can be used to estimate the affine transformation matrix M between the low-resolution floating image and the low-resolution reference image through cv2.estimateAffinePartial2D l , M l Contains transformation parameters such as translation, rotation, and scaling. Finally, use cv2.warpAffine to combine the matrix M l , the low-resolution floating image and the reference image are registered so that the two are precisely aligned.

[0018] Furthermore, according to the affine transformation matrix of the low-resolution image, combined with the scaling factors of the ultra-high-resolution and low-resolution images, the parameters of the translation part in the affine transformation matrix are adjusted to obtain the affine transformation matrix on the ultra-high-resolution image, including:

[0019] Based on the affine transformation matrix M obtained in step (2) l , M l is a 2×3 matrix, ,parameter Control the linear part of affine transformations, including rotation, scaling, and shearing, Controls the translation of the image. Therefore, if you want to perform ultra-high resolution transformation, you need to modify M l The parameters of the translation part. The width and height of the super-high resolution and low-resolution images are used to calculate the scaling ratios in the width and height directions, respectively: sf w and sf h . From this, we can get the affine transformation matrix on the ultra-high resolution image .

[0020] Furthermore, based on the affine transformation matrix on the ultra-high resolution image , by adding the matrix whose secondary coordinates are expanded to 3×3, calculating its inverse matrix, and extracting the first two rows of the inverse matrix to obtain the 2×3 inverse affine transformation matrix M h_inv .

[0021] Furthermore, by slicing the target registration image, using the inverse affine transformation matrix to calculate its coordinates in the ultra-high resolution floating image, and combining multi-threading technology to efficiently copy the pixel values ​​of the corresponding positions to the target registration image, including:

[0022] Method 1: First create a blank target registration image of size W hf ×H hf , which is consistent with the image size after alignment with the super-high resolution image. Cut the blank target registration image into blocks with a block size of W b ×H b , obtain the corresponding coordinates of the pixels in each image block in the target registration image and convert them into homogeneous coordinate format, generate the coordinate matrix to be processed, and use the inverse affine transformation matrix M h_inv The dot product of is used to obtain the corresponding coordinates in the ultra-high resolution floating image. At the same time, the out-of-bounds coordinates are processed and rounded to get the corresponding coordinate index on the effective ultra-high resolution floating image. Finally, its pixel value is copied to the corresponding position on the target registration image. All image blocks are processed using multi-threading technology to obtain the ultra-high resolution registration image.

[0023] Furthermore, by processing the coordinates of the four vertices of the target registration image after slicing, combining affine transformation estimation and coordinate system transformation, and finally converting it into the affine transformation of the ultra-high resolution floating image block and performing a memory block copy, the target registration image is generated, including:

[0024] Method 2: First, create a blank target registration image of size W hf ×H hf , which is consistent with the image size after alignment with the super-high resolution image. Cut the blank target registration image into blocks with a block size of W b ×H b , get the original coordinate matrix C of the four vertices in the image block in the target registration image out And convert it to homogeneous coordinate format by and inverse affine transformation matrix M h_inv The dot product of the corresponding vertex in the ultra-high resolution floating image is obtained by m , find the minimum value of its coordinates in the x and y directions (C min_x ,C min_y ) and the maximum value (C max_x ,C max_y ), a new matrix C is obtained by normalizing the coordinate system m_nIn the same way, the normalized coordinate matrix C out coordinate system, and obtain a new matrix C out_n , obtain C through cv2.estimateAffinePartial2D m_n and C out_n The affine transformation relationship matrix M between c , and then the image block captured from the ultra-high resolution floating image can be transformed through cv2.warpAffine, and the obtained new image corresponds to the image block in the target registration image, so the pixel value of the image block can be copied to its corresponding position through memory copy. The corresponding other image blocks are also obtained using the same operation method as above, and the efficiency is higher when combined with multi-threading technology, which greatly improves the calculation efficiency compared with method 1.

[0025] Furthermore, the affine transformation relationship matrix M c In addition to estimating affine transformation, the acquisition of can also be achieved by perspective transformation, direct linear transformation and least squares fitting.

[0026] The present invention also provides a full-process automated ultra-high resolution image registration device, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned full-process automated ultra-high resolution image registration method.

[0027] The present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned full-process automated ultra-high resolution image registration method is implemented.

[0028] The beneficial effect of the present invention is that the present invention proposes an efficient and robust registration method for the problems of memory consumption, computational efficiency and adaptability in ultra-high resolution image registration. The method obtains a preliminary transformation matrix by performing rapid preprocessing on a low-resolution image, and adjusts the transformation matrix on this basis to adapt to the ultra-high resolution image, and introduces two ultra-high resolution image registration methods. Method 1 maps pixel-by-pixel coordinates and fills the target image through an inverse affine transformation matrix, while method 2 processes the four vertex coordinates of the image block, combines affine transformation estimation and image block affine transformation, and provides a more efficient image registration method. These steps effectively break through the bottleneck of memory and computational efficiency in the ultra-high resolution image registration process, avoid the processing difficulties or inefficiencies caused by excessive memory requirements in traditional methods, have significant practical value, are applicable to the field of ultra-high resolution image processing, and have broad application prospects in the precise registration of high-resolution images such as remote sensing images, medical images, and astronomical images. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0030] Figure 1 A schematic diagram of a process flow of a fully automated ultra-high resolution image registration method provided by an embodiment of the present invention;

[0031] Figure 2 A structural block diagram of a full-process automated ultra-high resolution image registration device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0034] Figure 1 The following is a flow chart of a fully automated ultra-high resolution image registration method provided by an embodiment of the present invention. Figure 1 As shown, the full-process automated ultra-high resolution image registration method in the embodiment of the present invention may include the following steps:

[0035] Step S1: Ultra-high resolution pathology image HE and IHC reading and preprocessing, read the image pair, standardize and align the size of HE and IHC images to ensure the consistency of image size, and obtain low-resolution HE and IHC images through downsampling technology or image pyramid. Perform a series of image enhancement processing on the low-resolution image to improve the image quality. Then, perform background removal operation to eliminate irrelevant image areas;

[0036] Specifically, step S1 includes the following sub-steps:

[0037] Step S11: Read the pathological images HE (Hematoxylin and Eosin) and IHC (Immunohistochemistry) respectively, the image sizes are 59887×49495 and 59844×49491 respectively, mark the HE image as a floating image, and mark the IHC image as a reference image. The size of the HE image is kept consistent with the size of the IHC image by padding or cropping. The padding and cropping principle is that if the size of the HE image is smaller than that of the IHC image, it is necessary to fill in additional background areas in four directions. If the size of the HE image is larger than that of the IHC image, it is necessary to crop the redundant background parts in four directions to ensure that the size of the HE image and the IHC image are consistent with each other, both of which are 59844×49491.

[0038] Step S12: Obtain a pair of low-resolution HE images and low-resolution IHC images that are reduced to one-Nth of the HE and IHC images by downsampling or image pyramid, where the image sizes are both 935×773. Perform color and brightness enhancement on the low-resolution HE images. Use ImageEnhance.Color to enhance the color saturation of the image, and set the enhancement factor to 1.2 to make the image color more vivid. Then use ImageEnhance.Brightness to enhance the brightness of the image, and also set the enhancement factor to 1.2 to make the image brighter and improve the visibility of the image. Perform contrast and sharpness enhancement on the low-resolution IHC image, use ImageEnhance.Contrast to enhance the contrast of the image, and set the enhancement factor to 2.0 to make the light and dark differences of the image more obvious, and then use ImageEnhance.Sharpness to enhance the sharpness of the image, and set the enhancement factor to 2.0 to make the details of the image clearer. Convert the low-resolution HE and IHC images from BGR color space to HSV color space, set the upper and lower limits of the white (background color) HSV threshold to (180, 20, 255) and (0, 0, 220) respectively, use cv2.inRange for threshold processing, create white masks respectively, and use morphological opening operation to remove noise in the mask. The opening operation uses a 5×5 kernel, and removes small areas in the mask through corrosion and expansion operations, retaining larger connected areas. Then, the denoised white mask is inverted. By specifying an area threshold of 5000, larger areas are screened out and merged into a new mask. Use a 11×11 kernel for morphological closing operation to fill the small holes in the mask to make the mask more complete. Finally, the background removal operation is performed on the low-resolution HE and IHC images respectively using the mask images obtained above to obtain low-resolution HE and IHC images containing only the target foreground area.

[0039] Step S2: For the low-resolution HE image and low-resolution IHC image preprocessed in step S1, feature points and descriptors are extracted by SIFT (Scale-Invariant Feature Transform) algorithm and feature matching is performed by FLANN (Fast Library for Approximate Nearest Neighbors) algorithm. Then, the preprocessed low-resolution HE image is translated and rotated using the affine transformation matrix estimation algorithm to accurately align it with the preprocessed low-resolution IHC image. Finally, the affine transformation matrix M is obtained. l And complete the registration process of low-resolution HE images and low-resolution IHC images;

[0040] Specifically, step S2 includes the following sub-steps:

[0041] Step S21: Use the SIFT (Scale Invariant Feature Transform) algorithm to extract feature points and calculate descriptors for the preprocessed low-resolution HE image and low-resolution IHC image. By creating a cv2.SIFT_create object and calling the detectAndCompute method, the feature points (kp1, kp2) of the low-resolution HE image and the low-resolution IHC image and their corresponding descriptors (des1, des2) are obtained. Next, the FLANN (Fast Nearest Neighbor Search Package) algorithm is used for feature matching. The matcher is created through cv2.FlannBasedMatcher, and the index parameter is set to use the KD tree algorithm. The search parameter sets the number of checks to 500, and then the knnMatch method is used for feature matching. Traverse the matching results and filter out good matches. For each pair of matching feature points, if the distance of the first nearest neighbor feature point is less than 0.85 times the distance of the second nearest neighbor feature point, it is considered a good match and is added to the good_matches list to provide reliable matching points for subsequent transformation matrix estimation;

[0042] Step S22: If the number of good matching points is less than 4, it means that there are not enough matching points and registration cannot be performed. If the number of good matching points is greater than or equal to 4, these good matching points can be used to estimate the affine transformation matrix M between the low-resolution HE image and the low-resolution IHC image through cv2.estimateAffinePartial2D l , M lThe translation, rotation, and scaling transformation parameters are included. Finally, the cv2.warpAffine function is used to apply translation and rotation transformations for registration, and the preprocessed low-resolution HE image is transformed to a position aligned with the preprocessed low-resolution IHC image to obtain the registered low-resolution HE image. In this way, the entire process of low-resolution image registration is completed, and the preprocessed low-resolution HE image can be accurately aligned with the low-resolution IHC image, providing a basis for subsequent image analysis and research.

[0043] Step S3: According to the affine transformation matrix M l , as well as the width and height of the ultra-high resolution and low-resolution images, the scaling ratios in the width and height directions are calculated as sf w and sf h By modifying M l The parameters of the translation part in the image are used to obtain the affine transformation matrix M on the ultra-high resolution image. h Two ultra-high resolution HE and IHC image registration methods are proposed, either of which can be used; the first method is to cut the target registration image into blocks, use the inverse affine transformation matrix to calculate its coordinates in the ultra-high resolution HE image, and combine multi-threading technology to efficiently copy the pixel values ​​of the corresponding positions to the target registration image. The second method is to process the coordinates of the four vertices of the target registration image after cutting, combine affine transformation estimation and coordinate system transformation, and finally convert it into the affine transformation of the image block and perform memory block copy to generate the target registration image.

[0044] Specifically, step S3 includes the following sub-steps:

[0045] Step S31: Based on the affine transformation matrix M obtained in step S22 l The parameter characteristics of M need to be modified l The parameters of the translation part can be used for the registration of super-high resolution HE images and super-high resolution IHC images. According to the width and height of the super-high resolution image and the low-resolution image, the scaling ratios in the width and height directions are calculated as sf w and sf h . Thus, the affine transformation matrix M on the ultra-high resolution image can be obtained h , by adding homogeneous coordinates to expand to a 3×3 matrix, calculating its inverse matrix, and extracting the first two rows of the inverse matrix to obtain a 2×3 inverse affine transformation matrix M h_inv .

[0046] Step S32: Two methods of ultra-high resolution HE and IHC image registration are introduced below.

[0047] Method 1: First create a blank target registration image of size W hf ×H hf. Cut the blank target registration image into blocks with a block size of W b ×H b , obtain the corresponding coordinates of the pixels in each image block in the target registration image and convert them into homogeneous coordinate format, generate the coordinate matrix to be processed, and use the inverse affine transformation matrix M h_inv The corresponding coordinates in the ultra-high resolution HE image are obtained by multiplying the dot of the pixels by 1 and 2, and the out-of-bounds coordinates are processed and rounded to get the corresponding coordinate index on the effective ultra-high resolution HE image. Finally, the pixel value is copied to the corresponding position on the target registration image. All image blocks are processed using multi-threading technology to obtain the ultra-high resolution registration image.

[0048] Method 2: First, create a blank target registration image of size W hf ×H hf . Cut the blank target registration image into blocks with a block size of W b ×H b , get the original coordinate matrix C of the four vertices in the image block in the target registration image out And convert it to homogeneous coordinate format by and inverse affine transformation matrix M h_inv The point product of the corresponding vertex in the ultra-high resolution HE image is obtained by m , find the minimum value of its coordinates in the x and y directions (C min_x ,C min_y ) and the maximum value (C max_x ,C max_y ), the coordinate matrix C m The x- and y-coordinates in the min_x and C min_y , and then a new coordinate matrix C is obtained by normalizing the coordinate system m_n In the same way, the normalized coordinate matrix C out coordinate system, and obtain a new matrix C out_n , obtain C through cv2.estimateAffinePartial2D m_n and C out_n The affine transformation relationship matrix M between c , and then the upper left corner (C min_x ,C min_y ), lower right corner (C max_x ,C max_y) position, and transform it through cv2.warpAffine. The new image obtained corresponds to the image block in the target registration image. Therefore, the pixel value of the image block can be copied to the corresponding position in the target registration image through memory copy. The corresponding other image blocks are also obtained using the same operation method as above, and the efficiency is higher when combined with multi-threading technology. Compared with method 1, the calculation efficiency is greatly improved. The above transformation matrix M c In addition to estimating affine transformation, the acquisition of can also be achieved by perspective transformation, direct linear transformation and least squares fitting.

[0049] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 A full-process automated super-high resolution image registration method is provided.

[0050] The present invention also provides Figure 2 The one shown corresponds to Figure 1 Schematic diagram of the full-process automated ultra-high resolution image registration device. Figure 2 As mentioned above, at the hardware level, the fully automated ultra-high resolution image registration device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Of course, in addition to the software implementation, the present invention does not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0051] For the improvement of a technology, it can be clearly distinguished whether it is a hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or a software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0052] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320, and the memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices for implementing various functions included therein may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0053] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0054] For the convenience of description, the above device is described as being divided into various units according to their functions. Of course, when implementing the present invention, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0055] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0056] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0057] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0058] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0059] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0060] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0061] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0062] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0063] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0065] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0066] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A full-process automated ultra-high resolution image registration method, characterized in that: The following steps are involved: (1) Read ultra-high-resolution image pairs, standardize and align the sizes of the image pairs, generate low-resolution image pairs by downsampling, and perform preprocessing; (2) The feature points and descriptors of the preprocessed low-resolution image pair are extracted by the SIFT algorithm and the feature matching is performed using the FLANN algorithm. Then, the preprocessed low-resolution floating image is translated and rotated using the affine transformation matrix estimation algorithm to align it with the preprocessed low-resolution reference image. Finally, the affine transformation matrix is ​​obtained and the registration process of the low-resolution image is completed. (3) According to the affine transformation matrix of the low-resolution image, combined with the scaling factors of the ultra-high-resolution and low-resolution images, the parameters of the translation part in the affine transformation matrix are adjusted to obtain the affine transformation matrix on the ultra-high-resolution image; Obtaining an inverse affine transformation matrix based on the affine transformation matrix on the ultra-high resolution image; The registration methods of super-high resolution floating images and super-high resolution reference images include: By cutting the target registration image into blocks, using the inverse affine transformation matrix to calculate its coordinates in the ultra-high resolution floating image, and combining multi-threading technology to copy the pixel values ​​of the corresponding positions to the target registration image; or The target registration image is generated by processing the coordinates of the four vertices of the target registration image after slicing, combining affine transformation estimation and coordinate system transformation, and finally converting it into the affine transformation of the ultra-high resolution floating image block and performing memory block copying.

2. A fully automated ultra-high resolution image registration method according to claim 1, characterized in that: The step (1) comprises the following sub-steps: (1.1) Obtain a super-high-resolution floating image and a super-high-resolution reference image, i.e., a super-high-resolution image pair; and unify their sizes by padding or cropping to ensure that the two images are spatially aligned. The size of the aligned image pair is W hf ×H hf ; (1.2) The super-high-resolution image pair is reduced to one-N by downsampling technology to obtain a low-resolution image pair with a size of W lf ×H lf ; Enhance the color saturation and brightness of the low-resolution floating image, and enhance the contrast and sharpness of the low-resolution reference image; Convert the low-resolution image pair from BGR color space to HSV color space, and perform mask creation and denoising of the white background area; The mask is inverted and morphological closing is performed to fill the holes in the mask to accurately extract the region of interest, remove the background and retain the target area.

3. The fully automated ultra-high resolution image registration method according to claim 1, characterized in that: The step (2) includes the following sub-steps: (2.1) Use the SIFT algorithm to extract feature points and calculate descriptors of the low-resolution floating image and the low-resolution reference image to obtain their feature points and descriptors; use the FLANN algorithm to perform feature matching and screen out good matching points that meet the distance threshold; (2.2) Based on these good matching points, the affine transformation matrix M between the low-resolution floating image and the low-resolution reference image is estimated by the cv2.estimateAffinePartial2D method l , using the estimated affine transformation matrix M l , the low-resolution float image and the low-resolution reference image are registered via cv2.warpAffine so that they are precisely aligned.

4. The method of full-process automated ultra-high resolution image registration according to claim 1, characterized in that: According to the affine transformation matrix of the low-resolution image, combined with the scaling factors of the ultra-high-resolution and low-resolution images, the parameters of the translation part in the affine transformation matrix are adjusted to obtain the affine transformation matrix on the ultra-high-resolution image, including: Based on the affine transformation matrix obtained in step (2) , and calculate the scaling factor sf according to the aspect ratio of the super-resolution and low-resolution images w and sf h , by modifying M l Parameters that control translation , get the affine transformation matrix on the ultra-high resolution image .

5. A fully automated ultra-high resolution image registration method according to claim 4, characterized in that: Based on the affine transformation matrix on the ultra-high resolution image, its inverse affine transformation matrix is ​​obtained, including: Affine transformation matrix based on ultra-high resolution images , by adding homogeneous coordinates to expand to a 3×3 matrix, calculating its inverse matrix, and extracting the first two rows of the inverse matrix to obtain a 2×3 inverse affine transformation matrix M h_inv .

6. The fully automated ultra-high resolution image registration method according to claim 1, characterized in that: By cutting the target registration image into blocks, using the inverse affine transformation matrix to calculate its coordinates in the ultra-high resolution floating image, and combining multi-threading technology to efficiently copy the pixel values ​​of the corresponding positions to the target registration image, including: Create a blank target registration image of size W hf ×H hf , align with the super-high resolution image, and cut the target registration image into blocks with a block size of W b ×H b ; Convert the matrix composed of the coordinates of all pixels in the image block into homogeneous coordinate format and the inverse affine transformation matrix M h_inv Perform dot multiplication to obtain the corresponding coordinates in the ultra-high resolution floating image, perform boundary checking and rounding operations; copy the pixel values ​​in the ultra-high resolution floating image to the target registration image, and process the image blocks through multi-threading technology to improve processing efficiency.

7. The fully automated ultra-high resolution image registration method according to claim 1, characterized in that: By processing the coordinates of the four vertices of the target registration image after slicing, combining affine transformation estimation and coordinate system transformation, and finally converting it into the affine transformation of the ultra-high resolution floating image block and performing a memory block copy, the target registration image is generated, including: Create a blank target registration image of size W hf ×H hf , align with the super-high resolution image, cut the target registration image into blocks, obtain the original coordinates of the four vertices in the image block in the target registration image to form the matrix C out , and convert it into homogeneous coordinate format by and inverse affine transformation matrix M h_inv The dot product of the corresponding vertex in the ultra-high resolution floating image is obtained by m , by normalizing the coordinate system, we get the matrix C m_n ; Similarly, normalize C out The coordinate system of the matrix C out_n , obtain C through cv2.estimateAffinePartial2D m_n and C out_n The affine transformation relationship matrix M between c , and then the image block can be affine transformed by cv2.warpAffine; the transformed image is copied to the corresponding position of the image block in the target registration image; the corresponding other image blocks are also obtained using the same operation method as above, and all image blocks are efficiently processed in combination with multi-threading technology.

8. The full-process automated ultra-high resolution image registration method according to claim 7, characterized in that: The affine transformation relationship matrix M c Other ways to obtain include perspective transformation, direct linear transformation or least squares fitting.

9. A fully automated ultra-high resolution image registration device, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a full-process automated ultra-high resolution image registration method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: A program is stored thereon, and when the program is executed by a processor, a full-process automated ultra-high resolution image registration method as described in any one of claims 1-8 is implemented.

Images