Image registration method, computer program product, storage medium and electronic device

Abstract

The application relates to the technical field of image processing, and provides an image registration method, a computer program product, a storage medium and electronic equipment. The image registration method comprises the following steps: acquiring a reference image and an intermediate image; taking the image registration reference image as a reference, calculating a homography matrix between the image registration intermediate image and the image registration reference image by using a registration algorithm based on the homography matrix, and calculating a homography optical flow corresponding to the image registration homography matrix; taking the image registration reference image as a reference, calculating a forward optical flow between the image registration intermediate image and the image registration reference image by using an optical flow-based registration algorithm; fusing the image registration homography optical flow and the image registration forward optical flow to obtain a fused optical flow; and transforming a to-be-registered image of image registration by using the image registration fused optical flow to obtain a registered image. The method realizes the complementary advantages of two types of image registration algorithms, and is beneficial to improving the image registration accuracy.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image registration method, a computer program product, a storage medium, and an electronic device. Background Technology

[0002] In the field of image processing, techniques that require processing multiple images captured at different times, such as image stitching, multi-frame noise reduction, super-resolution, and High Dynamic Range (HDR), often necessitate image registration. Otherwise, artifacts like ghosting and blurring can easily occur in the processed results, significantly degrading image quality. Image registration is commonly used to process two images captured at different times. One image is used as a reference image, and the other as the image to be registered. The image to be registered is transformed so that pixels at the same location in the actual scene are perfectly aligned in the two images after the transformation.

[0003] Common image registration algorithms fall into two categories: homography matrix-based registration algorithms and optical flow-based registration algorithms. The first type of algorithm performs well in images with global motion, and the registered image content does not exhibit local distortion. However, it performs poorly in images with localized motion. The second type of algorithm effectively handles images with localized motion, but in areas with no texture, repetitive texture, or moving areas, the registered image content is prone to distortion, leading to a decrease in image quality. Summary of the Invention

[0004] The purpose of this application is to provide an image registration method, a computer program product, a storage medium, and an electronic device to improve the above-mentioned technical problems.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] In a first aspect, embodiments of this application provide an image registration method, comprising: acquiring a reference image and an intermediate image; wherein the intermediate image is an image to be registered, or an image obtained by processing the image to be registered, the brightness of which is at the same level as the reference image; using the reference image as a reference, calculating the homography matrix between the intermediate image and the reference image using a homography matrix-based registration algorithm, and calculating the homography optical flow corresponding to the homography matrix; using the reference image as a reference, calculating the forward optical flow between the intermediate image and the reference image using an optical flow-based registration algorithm; fusing the homography optical flow and the forward optical flow to obtain a fused optical flow; and transforming the image to be registered using the fused optical flow to obtain a registered image.

[0007] In the above method, firstly, a homography-based registration algorithm is used to calculate the homography optical flow, and an optical flow-based registration algorithm is used to calculate the forward optical flow. Then, the fused optical flow is calculated based on these two optical flows. Finally, the fused optical flow is used to transform the image to be registered. This method achieves the complementary advantages of the two types of image registration algorithms. It can effectively handle the situation where there is local motion in the image and suppress the distortion in the registration result, thus significantly improving the image registration accuracy.

[0008] In one implementation of the first aspect, fusing the homography optical flow and the forward optical flow to obtain a fused optical flow includes: calculating an optical flow fusion mask based on the forward optical flow; wherein the pixel values ​​in the optical flow fusion mask characterize the fusion weights; and fusing the homography optical flow and the forward optical flow using the optical flow fusion mask to obtain the fused optical flow.

[0009] In the above implementation, a pixel-by-pixel weighted fusion of homography optical flow and forward optical flow is performed using an optical flow fusion mask. This approach yields a more accurate fusion result and allows the calculation of the optical flow fusion mask to determine whether to use more homography or forward optical flow at any pixel location in the fused optical flow. Homography optical flow does not exhibit distortion, while forward optical flow performs better in locally moving regions.

[0010] In one implementation of the first aspect, the step of calculating the optical flow fusion mask based on the forward optical flow includes: using the intermediate image as a reference, calculating the reverse optical flow between the intermediate image and the reference image using the optical flow-based registration algorithm; calculating the difference degree of corresponding pixel values ​​in the forward optical flow and the reverse optical flow to obtain a first difference degree image; and determining the optical flow fusion mask based on the first difference degree image.

[0011] Ideally, the opposite of the reverse optical flow is equal to the forward optical flow. Therefore, if the pixel values ​​of the forward optical flow and the reverse optical flow in the same region differ significantly (regardless of sign), it indicates that the forward optical flow is distorted in that region. In other words, in the above implementation, the distorted region in the forward optical flow can be detected by calculating the difference between the corresponding pixel values ​​in the forward and reverse optical flows, and the detection result is reflected in the first difference image.

[0012] Subsequently, based on the first difference image, a suitable optical flow fusion mask is determined. This allows for the fusion of more homography-based optical flow in distorted regions and more forward optical flow in non-distorted regions during optical flow fusion. Since there is no distortion in homography-based optical flow, this optical flow fusion effectively eliminates or at least suppresses the distortion present in the forward optical flow. Then, by using the fused optical flow to transform the image to be registered, a high-precision registration result can be obtained.

[0013] In one implementation of the first aspect, the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the homography optical flow, and the step of determining the optical flow fusion mask based on the first difference image includes: binarizing the first difference image to obtain a binarized image with pixel values ​​of 0 or 1; and performing morphological filtering on the binarized image to obtain the optical flow fusion mask.

[0014] The binarization described above refers to setting a threshold. If a pixel value in the first difference image is large (greater than the threshold), the corresponding pixel value is set to 1 in the binarized image; otherwise, it is set to 0. Morphological filtering (e.g., erosion, dilation) applied to the binarized image primarily serves to reduce noise; the resulting optical flow fusion mask is still binarized. Since the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to homography, a pixel value of 1 in the mask indicates that the pixel value at that location will be entirely based on the homography during optical flow fusion.

[0015] Furthermore, based on the calculation method of the first difference image, it can be seen that if a certain pixel value in the first difference image is large, it indicates that there is distortion in the forward optical flow at that pixel position. The pixel value of the optical flow fusion mask calculated according to the above implementation method is likely to be 1. Therefore, the pixel value of the fused optical flow at that pixel position will come entirely from the homography optical flow rather than the forward optical flow, which is beneficial to eliminate distortion in the forward optical flow and improve the image registration result.

[0016] In one implementation of the first aspect, the step of calculating the optical flow fusion mask based on the forward optical flow includes: calculating the difference degree of corresponding pixel values ​​in the forward optical flow and the homography optical flow to obtain a second difference degree image; and determining the optical flow fusion mask based on the second difference degree image.

[0017] Since homography corresponds to a global transformation, there is no local distortion in homography. Therefore, if the pixel value of the forward optical flow in a certain region differs significantly from the pixel value of the homography in the same region, it indicates that there is distortion in the forward optical flow in that region. In other words, in the above implementation, the distortion region in the forward optical flow can be detected by calculating the difference between the corresponding pixel values ​​in the forward optical flow and the homography. The detection result is reflected in the second difference image.

[0018] Subsequently, by reasonably determining the optical flow fusion mask based on the second difference image, more homography optical flow can be fused in distorted regions and more forward optical flow can be fused in non-distorted regions during optical flow fusion. Such optical flow fusion is equivalent to eliminating or at least suppressing the distortion present in the forward optical flow. Then, by using the fused optical flow to transform the image to be registered, a higher accuracy registration result can be obtained.

[0019] In one implementation of the first aspect, the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the homography optical flow, and the step of determining the optical flow fusion mask based on the second difference image includes: using a monotonically increasing transformation function to map the pixel values ​​in the second difference image to a range of 0 to 1 to obtain the optical flow fusion mask.

[0020] In the above implementation, since the transformation function is monotonically increasing, the pixel values ​​in the second difference image remain unchanged after mapping. That is, if a pixel value in the second difference image is large (or small), its corresponding pixel value in the optical flow fusion mask is also large (or small). Furthermore, since the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to homography, if a pixel value in the optical flow fusion mask is large (close to or equal to 1), it indicates that during optical flow fusion, the pixel value at that location will be largely or completely represented by the homography pixel value.

[0021] Furthermore, according to the calculation method of the second difference image, if a certain pixel value in the second difference image is large, it indicates that there is distortion in the forward optical flow at that pixel position. However, the pixel value of the optical flow fusion mask calculated according to the above implementation method is close to or equal to 1 at that pixel position. Therefore, the pixel value of the fused optical flow at that pixel position will mainly or completely come from the homography optical flow rather than the forward optical flow, which is beneficial to eliminate optical flow distortion and improve the registration result of the image.

[0022] In one implementation of the first aspect, the step of calculating the optical flow fusion mask based on the forward optical flow includes: using the intermediate image as a reference, calculating the reverse optical flow between the intermediate image and the reference image using the optical flow-based registration algorithm; calculating the difference degree of corresponding pixel values ​​in the forward optical flow and the reverse optical flow to obtain a first difference degree image; calculating the difference degree of corresponding pixel values ​​in the forward optical flow and the homography optical flow to obtain a second difference degree image; and calculating the optical flow fusion mask based on the first difference degree image and the second difference degree image.

[0023] As described above, the first difference image detects distorted regions based on the difference between forward and reverse optical flow, while the second difference image detects distorted regions based on the difference between forward and homography optical flow. The inventors discovered that these two methods are complementary in detecting distorted regions in forward optical flow. Therefore, by rationally determining the optical flow fusion mask based on the first and second difference images, the suppression effect on optical flow distortion during optical flow fusion can be enhanced (compared to methods that only use the first or second difference image to determine the optical flow fusion mask), thereby further improving image registration accuracy.

[0024] In one implementation of the first aspect, the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the homography optical flow. The step of calculating the optical flow fusion mask based on the first difference image and the second difference image includes: binarizing the first difference image to obtain a binarized image with pixel values ​​of 0 or 1; performing morphological filtering on the binarized image to obtain a first mask; mapping the pixel values ​​in the second difference image to a range of 0 to 1 using a monotonically increasing transformation function to obtain a second mask; superimposing the corresponding pixel values ​​from the first mask and the second mask to obtain a third mask; and mapping the pixel values ​​in the third mask to a range of 0 to 1 to obtain the optical flow fusion mask.

[0025] In the above implementation, since the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the homography optical flow, if a pixel value in the first mask is 1, it indicates that if optical flow fusion is performed using the first mask, the pixel value at that location will be entirely based on the homography optical flow. Thus, the first mask, to some extent, also represents the result of detecting distorted regions based on forward and reverse optical flow. Conversely, if a pixel value in the second mask is large (close to or equal to 1), it indicates that if optical flow fusion is performed using the second mask, the pixel value at that location will be largely or entirely based on the homography optical flow. Thus, the second mask, to some extent, also represents the result of detecting distorted regions based on forward and homography optical flow.

[0026] Furthermore, by superimposing the first and second masks, for regions where the pixel values ​​in both masks are relatively large, the pixel values ​​will increase further after superposition. Similarly, for regions where the pixel values ​​in only one mask are relatively large, the pixel values ​​will also become relatively large after superposition. It is evident that superimposing the two masks enhances the detection capability for distorted regions in the forward optical flow. Moreover, the final mapping step sets the pixel values ​​of the fused optical flow in the distorted regions to 1 or a value close to 1, meaning that homography is primarily or entirely used in the distorted regions, thereby improving image registration accuracy.

[0027] In one implementation of the first aspect, acquiring the reference image and the intermediate image includes: acquiring the reference image and the image to be registered; determining whether the brightness of the image to be registered is at the same level as the brightness of the reference image; if they are not at the same level, adjusting the brightness of the image to be registered to obtain the intermediate image whose brightness is at the same level as the reference image.

[0028] The inventors discovered that if the brightness difference between the image to be registered and the reference image is significant, the calculation of the homography matrix and forward optical flow will be negatively affected, resulting in a decrease in image registration accuracy. Therefore, if the brightness of the image to be registered and the reference image are not at the same level, the brightness of the image to be registered can be adjusted first to obtain an intermediate image with the same brightness as the reference image (i.e., the brightness-adjusted image to be registered). Then, the homography matrix and forward optical flow can be calculated based on the reference image and the intermediate image, and the calculation results are more accurate. Of course, if the brightness of the image to be registered and the reference image are already at the same level, no processing is required, and the subsequent calculation of the homography matrix and forward optical flow can be performed directly (i.e., the image to be registered is directly used as the intermediate image).

[0029] Secondly, embodiments of this application provide an image registration apparatus, comprising: an image acquisition module for acquiring a reference image and an intermediate image; wherein the intermediate image is an image to be registered, or an image obtained by processing the image to be registered, the brightness of which is at the same level as the reference image; a homography registration module for calculating the homography matrix between the intermediate image and the reference image using a homography matrix-based registration algorithm, and calculating the homography optical flow corresponding to the homography matrix, using the reference image as a reference; an optical flow registration module for calculating the forward optical flow between the intermediate image and the reference image using an optical flow-based registration algorithm, using the reference image as a reference; an optical flow fusion module for fusing the homography optical flow and the forward optical flow to obtain a fused optical flow; and an image transformation module for transforming the image to be registered using the fused optical flow to obtain a registered image.

[0030] Thirdly, embodiments of this application provide a computer program product, including computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation of the first aspect.

[0031] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0032] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores computer program instructions, and the computer program instructions are read and executed by the processor to perform the method provided in the first aspect or any possible implementation of the first aspect. Attached Figure Description

[0033] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0034] Figure 1 The flowchart of an image registration method provided in an embodiment of this application is shown;

[0035] Figure 2 The structure of an image registration device provided in an embodiment of this application is shown;

[0036] Figure 3 The structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0037] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0038] In computer vision, image registration is a fundamental task. Image registration algorithms mainly fall into two categories: homography matrix-based registration algorithms and optical flow-based registration algorithms.

[0039] The main idea of ​​homography matrix-based registration algorithms is to perform feature point detection and matching on the reference image and the image to be registered, thereby calculating a homography matrix. This homography matrix is ​​then used to transform the image to be registered, achieving image registration. Since the transformation corresponding to the homography matrix is ​​a global transformation (i.e., the same transformation is performed on every pixel of the image to be registered, such as rotation, scaling, and translation), this type of algorithm performs well in images with global motion, and the registered image content does not exhibit local distortion. However, it performs poorly in images with local motion. For example, if the image contains pedestrians, and the background motion direction is different from the pedestrian's motion direction, this type of algorithm will perform poorly on the pedestrian portion.

[0040] The main idea of ​​optical flow-based registration algorithms is to search for the motion vector of each pixel in the image to be registered relative to its corresponding pixel in the reference image. The set of these motion vectors can be called the optical flow field or optical flow. By transforming the image to be registered using the calculated optical flow, image registration can be achieved. Since the transformation corresponding to optical flow is a non-global transformation and this transformation has locality (i.e., different transformations can be performed on each pixel in the image to be registered), this type of algorithm can effectively handle cases where there is local motion in the image. However, in some textureless, repetitive texture, or moving regions (although optical flow-based registration algorithms can effectively handle local motion, not all moving regions can be handled well), the registered image content is prone to distortion, resulting in a decrease in image quality.

[0041] In summary, both types of algorithms have their own advantages and disadvantages. The technical solution of this application combines these two types of algorithms to achieve image registration, allowing them to complement each other's strengths. It not only achieves good registration even in images with local motion, but also effectively controls distortion in the registration results.

[0042] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0043] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0044] The terms “first,” “second,” etc., are used only to distinguish one entity or operation from another, and should not be construed as indicating or implying relative importance, nor as requiring or implying any such actual relationship or order between these entities or operations.

[0045] Figure 1 This application illustrates a flowchart of an image registration method provided in an embodiment. This method can be, but is not limited to, [methods described by] [the following]. Figure 3 The illustrated electronic device performs the operation; the structure of this electronic device can be referred to in the following description. Figure 3 The explanation. (Refer to...) Figure 1 The method includes:

[0046] Step S110: Obtain the reference image and the intermediate image.

[0047] The intermediate image is either the image to be registered, or, after processing the image to be registered, an image whose brightness is at the same level as the reference image. The relationship between the intermediate image and the image to be registered is explained in detail below:

[0048] As explained above, image registration involves an image pair, which includes a reference image and an image to be registered. Image registration is the process of transforming the image to be registered to align it with the reference image. The specific transformation performed on the image to be registered is determined through calculations based on the reference image and the image to be registered.

[0049] Therefore, for image registration, the first step is to acquire a reference image and an image to be registered. This application does not limit the method used to acquire these two images; for example, they can be acquired in real-time via a camera, read from a database, etc. The reference image and the image to be registered have the same size.

[0050] After analyzing various registration algorithms based on homography matrix and optical flow, the inventors found that if the brightness of the image to be registered differs greatly from that of the reference image, the accuracy of the homography matrix (step S120) and the forward optical flow (step S130) calculated in subsequent steps may decrease, thereby causing a decrease in image registration accuracy.

[0051] Therefore, after obtaining the reference image and the image to be registered, we can first determine whether their brightness is at the same level. For example, we can calculate the average pixel value of the reference image and the average pixel value of the image to be registered as their brightness. If the difference (absolute value) between these two average pixel values ​​is less than a threshold, it indicates that the brightness of the reference image and the image to be registered is at the same level; otherwise, it indicates that the brightness of the reference image and the image to be registered is not at the same level. Of course, besides the average pixel value, image brightness can also be defined in other ways, which will not be listed here.

[0052] In some implementations, to mitigate the aforementioned issue of decreased accuracy in homography matrix and forward optical flow calculations, if the brightness of the reference image and the image to be registered are determined to be at different levels, the brightness of the image to be registered can be adjusted first to obtain an intermediate image (i.e., the brightness-adjusted image to be registered) with the same brightness as the reference image. Subsequently, the homography matrix and forward optical flow are calculated based on the reference image and the intermediate image, resulting in higher accuracy. Conversely, if the brightness of the image to be registered and the reference image are determined to be at the same level, the homography matrix and forward optical flow can be calculated directly based on both images; in other words, the intermediate image in this case is the image to be registered itself.

[0053] The operation described above, which adjusts the brightness level of the image to be registered to be the same as that of the reference image, can be called a brightness uniformity operation. For example, histogram matching is a type of brightness uniformity operation. In addition, there are some brightness uniformity operations based on deep learning, such as inputting the image to be registered and the reference image into a neural network model, which outputs an intermediate image with the same brightness level as the reference image, and so on.

[0054] It should be noted that in some implementations, the image brightness level may not be determined, and the image to be registered may be used directly as an intermediate image. In this case, the accuracy of the homography matrix and forward optical flow calculations mainly depends on the robustness of the image registration algorithm in steps S120 and S130 to image brightness. Relatively speaking, if the image brightness level is determined in advance, the requirements for brightness robustness of the image registration algorithm in steps S120 and S130 can be reduced, thereby broadening the applicable scenarios of the image registration method.

[0055] Step S120: Using the reference image as a reference, calculate the homography matrix between the intermediate image and the reference image using a registration algorithm based on the homography matrix, and calculate the homography optical flow corresponding to the homography matrix.

[0056] In step S120, using the reference image as a reference means aligning the intermediate image to the reference image, or in other words, the calculated homography matrix can be used to transform the intermediate image into the coordinate system of the reference image. Registration algorithms based on the homography matrix have been previously described. For example, feature point detection and matching can be performed on the reference image and the image to be registered, and then the homography matrix can be calculated based on the positional relationship of the matched feature point pairs. Of course, each specific algorithm will differ in the method of feature point detection or matching. The homography matrix calculated in step S120 can be a 3×3 matrix.

[0057] It should be noted that the homography matrix-based registration algorithm in step S120 is not limited to the existing algorithms. It also includes future image registration algorithms that aim to calculate the homography matrix and transform the images to be registered based on the calculated homography matrix.

[0058] Furthermore, it should be noted that regardless of whether the intermediate image is the image to be registered itself or the image obtained after brightness adjustment, the homography matrix between the intermediate image and the reference image can be considered equivalent to the homography matrix between the image to be registered and the reference image, because brightness adjustment does not change the position of objects in the image.

[0059] After calculating the homography matrix, it can be easily converted into a corresponding optical flow, called homography optical flow. Homography optical flow can be represented as a matrix with the same dimensions as both the reference image and the image to be registered. Each pixel value (also called an optical flow value) in this matrix includes an x-component and a y-component. The specific conversion method is as follows:

[0060] For any coordinate (x, y) in the intermediate image, multiplying it by the homography matrix H will give a new coordinate (x', y'). Subtracting the original coordinate from the new coordinate will give the homography optical flow value flow_H(x, y) = (x'-x, y'-y) at coordinate (x, y). By traversing all coordinates in the intermediate image, the homography optical flow flow_H can be obtained.

[0061] Step S130: Using the reference image as a reference, calculate the forward optical flow between the intermediate image and the reference image using an optical flow-based registration algorithm.

[0062] In step S130, using the reference image as a reference means aligning the intermediate image to the reference image, or in other words, the calculated optical flow can be used to transform the intermediate image into the coordinate system of the reference image. The optical flow calculated using the reference image as a reference is called the forward optical flow, to distinguish it from the reverse optical flow discussed later. Optical flow-based registration algorithms have been introduced previously; for example, the Lucas-Kanade optical flow algorithm and the DIS optical flow algorithm can be used. The forward optical flow calculated in step S130 can be represented as a matrix with the same size as both the reference image and the image to be registered. Each pixel value (also called an optical flow value) in this matrix includes an x-component and a y-component.

[0063] It should be noted that the optical flow-based registration algorithm in step S130 is not limited to the existing algorithms. It also includes future image registration algorithms that aim to calculate optical flow and transform the images to be registered based on the calculated optical flow.

[0064] Furthermore, it should be noted that regardless of whether the intermediate image is the image to be registered itself or the image obtained after brightness adjustment, the optical flow between the intermediate image and the reference image can be considered equivalent to the optical flow between the image to be registered and the reference image, because brightness adjustment does not change the position of objects in the image.

[0065] It should be understood that the execution order of steps S120 and S130 is not limited. Step S120 can be executed first and then step S130, or step S130 can be executed first and then step S130, or steps S120 and S130 can be executed in parallel.

[0066] Step S140: The homography optical flow and the forward optical flow are fused to obtain the fused optical flow.

[0067] The fusion in step S140 refers to an operation that combines all or part of the information in the homography optical flow with all or part of the information in the forward optical flow to form a fused optical flow. The fused optical flow can be represented as a matrix with the same size as both the homography optical flow and the forward optical flow. Each pixel value (also called an optical flow value) in this matrix includes an x ​​component and a y component.

[0068] For example, the pixel values ​​at corresponding pixel positions of the homography optical flow and the forward optical flow can be weighted to obtain the pixel value of the fused optical flow at the same pixel position; another example is that a portion of the homography optical flow and the forward optical flow can be taken and concatenated to form the fused optical flow; yet another example is that the homography optical flow and the forward optical flow can be input into a neural network, which can then predict the fused optical flow, and so on. The first example above can also be described as obtaining the fused optical flow using a weighted summation method, which will be further elaborated later.

[0069] Step S150: Transform the image to be registered using fused optical flow to obtain the registered image.

[0070] The registration result of the image to be registered is called the registration image. Given the fused optical flow, the coordinate mapping relationship between the image to be registered and the registration image is explicit, that is:

[0071] x' = x + flow_result(x, y, 1)

[0072] y' = y + flow_result(x, y, 2)

[0073] Where (x,y) represents any coordinate in the image to be registered, flow_result represents the fused optical flow, flow_result(x,y) represents the optical flow value at (x,y), flow_result(x,y,1) represents the x component of the optical flow value, flow_result(x,y,2) represents the y component of the optical flow value, and (x',y') represents the coordinates after (x,y) is mapped, which are located in the coordinate system of the registered image.

[0074] Based on the coordinate mapping relationship described above, the registered image can be calculated in the following ways, but is not limited to:

[0075] result(x',y') = cur(x,y)

[0076] Here, result represents the registered image, and cur represents the image to be registered. Iterate through all coordinates in result. For any coordinate (x', y'), calculate its corresponding coordinate (x, y) in the coordinate system of cur according to the coordinate mapping relationship above. Then, the pixel value of cur at (x, y) is used as the pixel value of result at coordinate (x', y') result(x', y').

[0077] Two points need further clarification:

[0078] Firstly, given (x', y'), the coordinates (x, y) calculated based on the coordinate mapping relationship may not be within the image range of cur. In this case, cur(x, y) can be replaced with a nearest pixel value in cur, for example, cur(u, v), where (u, v) is the nearest coordinate to (x, y) within the image range of cur. Of course, other remedial measures can also be taken to determine the value of result(x', y').

[0079] Secondly, after calculating the coordinates (x, y) based on the coordinate mapping relationship, it is not necessary to directly assign cur(x, y) to result(x', y') as in the formula above. Some smoothing operations can also be performed. For example, take several pixel values ​​in the neighborhood of (x, y) in cur (such as cur(x, y), cur(x+1, y+1), cur(x-1, y-1), cur(x+1, y-1), cur(x-1, y+1), etc.), perform a weighted average, and then assign the mean value to result(x', y').

[0080] In summary, the image registration method provided in this application first calculates the homography optical flow using a homography matrix-based registration algorithm and the forward optical flow using an optical flow-based registration algorithm. Then, the fused optical flow is calculated based on these two optical flows. Finally, the fused optical flow is used to transform the image to be registered. This method achieves the complementary advantages of the two types of image registration algorithms, which can effectively register images with local motion and suppress distortion in the registration results, thereby significantly improving the image registration accuracy.

[0081] Below, based on the above embodiments, we will continue to describe how to calculate the fused optical flow by weighted summation (step S140).

[0082] In some implementations, an optical flow fusion mask can be calculated first, and then the homography optical flow and the forward optical flow can be weighted and summed using the optical flow fusion mask to obtain the fused optical flow. The optical flow fusion mask can be represented as a matrix with the same size as both the homography optical flow and the forward optical flow. The pixel values ​​in this matrix represent the fusion weights used during the weighted fusion process.

[0083] Since there are two objects in the weighted summation, namely the homography optical flow and the forward optical flow, there are also two required fusion weights, namely the fusion weight corresponding to the homography optical flow and the fusion weight corresponding to the forward optical flow. The sum of these two weights can be fixed, for example, the sum is 1. Therefore, as long as one of the fusion weights is known, the other fusion weight can also be calculated.

[0084] Therefore, as an alternative approach, the pixel value in the optical flow fusion mask can be one of the two fusion weights. The other fusion weight is obtained by subtracting the pixel value in the optical flow fusion mask from 1. The following text will primarily use this approach as an example to illustrate the calculation of the optical flow fusion mask. Understandably, in this approach, the pixel value in the optical flow fusion mask takes a value between 0 and 1. Whether it can take either 0 or 1 depends on the specific implementation.

[0085] For example, if the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the homography optical flow, then the formula for calculating the fused optical flow is as follows:

[0086] flow_result(x,y)=flow_H(x,y)×flow_fusion_mask(x,y)+flow(x,y)×(1-flow_fusion_mask(x,y))

[0087] Wherein, flow_result represents the fused optical flow, flow_H represents the homography optical flow, flow represents the forward optical flow, flow_fusion_mask represents the optical flow fusion mask, (x,y) represents any coordinate position, M(x,y) represents the pixel value of matrix M at coordinate (x,y), and M can be flow_result, flow_H, flow, or flow_fusion_mask.

[0088] By using an optical flow fusion mask to perform pixel-by-pixel weighted fusion of homography optical flow and forward optical flow, a more accurate fusion result can be obtained. Furthermore, the calculation of the optical flow fusion mask can determine whether to use more homography or forward optical flow at any pixel location in the fused optical flow, thus effectively combining the advantages of the two image registration algorithms mentioned above. Specifically, homography optical flow does not exhibit distortion, while forward optical flow performs better in locally moving regions.

[0089] Below, based on the above embodiments, we will continue to introduce some methods for calculating optical flow fusion masks. It should be understood that the methods for calculating optical flow fusion masks are not limited to the following:

[0090] Method 1: Directly use an optical flow fusion mask with a fixed pixel value (strictly speaking, the "calculation" of the optical flow fusion mask may not be performed at this time). As for how to set the pixel value in the optical flow fusion mask, it can be set according to experience, or it can be set specifically according to the different scenes in the reference image and the image to be registered.

[0091] For example, each pixel value in the optical flow fusion mask can be set to 1 / 2, which is equivalent to averaging the homography optical flow and the forward optical flow to calculate the fused optical flow.

[0092] Method 2: Calculate the optical flow fusion mask based at least on the forward optical flow. Method 2 can be further subdivided into several implementation methods, such as:

[0093] Method 2.1: Input the homography optical flow and the forward optical flow into a neural network, which then predicts the optical flow fusion mask. Alternatively, the neural network may have more inputs, such as a reference image and / or the image to be registered.

[0094] During training, the neural network can use the difference between the calculated fused image and the pre-prepared reference fused image as the loss. The neural network will learn how to allocate fusion weights to homography optical flow and forward optical flow at each pixel location of the optical flow fusion mask.

[0095] Method 2.2: First, using the intermediate image as a reference, the reverse optical flow between the intermediate image and the reference image is calculated using an optical flow-based registration algorithm. Then, the difference between the corresponding pixel values ​​in the forward and reverse optical flows is calculated to obtain the first difference image. Finally, the optical flow fusion mask is determined based on the first difference image.

[0096] Method 2.2 uses the intermediate image as a reference, meaning the reference image is aligned to the intermediate image, or in other words, the calculated optical flow can be used to transform the reference image into the coordinate system of the intermediate image. The optical flow calculated using the intermediate image as a reference is called the reverse optical flow. To distinguish it from the forward optical flow mentioned earlier, the reverse optical flow is calculated using the same algorithm as the forward optical flow. The reverse optical flow calculated in Method 2.2 can be represented as a matrix with the same size as the forward optical flow. Each pixel value (also called an optical flow value) in this matrix includes an x-component and a y-component.

[0097] The difference between corresponding pixel values ​​in the forward and reverse optical flow represents the degree of difference between two pixel values, but the specific definition of the difference is not limited. The first difference image has the same size as both the forward and reverse optical flow images. For example, one way to calculate the first difference image is as follows:

[0098] S(x,y)=abs(flow(x,y,1)+flow_inverse(x,y,1))+abs(flow(x,y,2)+flow_inverse(x,y,2))

[0099] Where S represents the first difference image, flow represents the forward optical flow, flow_inverse represents the reverse optical flow, abs represents the absolute value, (x,y) represents any coordinate position, M(x,y) represents the pixel value of matrix M at coordinate (x,y), M(x,y,1) represents the x component of M(x,y), M(x,y,2) represents the y component of M(x,y), and M can be S, flow, or flow_inverse.

[0100] It's important to note that the formulas above use "+" signs instead of "-" signs before `flow_inverse(x,y,1)` and `flow_inverse(x,y,2)`. At first glance, this might seem like no "difference" is being calculated. The reason is as follows: the algorithms used to calculate forward and reverse optical flow are the same, one using a reference image and the other using an intermediate image. Therefore, ideally, the difference between reverse and forward optical flow is exactly one "-" sign. Thus, when calculating the difference, a "-" sign is not needed before `flow_inverse(x,y,1)` and `flow_inverse(x,y,2)`. For example, according to this formula, `flow(10,10) = (2,3)`, `flow_inverse(10,10) = (-2,-3)`, then `S(x,y) = 0`.

[0101] Obviously, the first difference image can also be calculated using other methods, for example:

[0102] S(x,y)=sqrt((flow(x,y,1)+flow_inverse(x,y,1))^2+(flow(x,y,2)+flow_inverse(x,y,2))^2)

[0103] Here, ^2 represents squaring, sqrt represents sqrting, and the other parts of the formula will not be explained again.

[0104] As mentioned earlier, optical flow-based image registration algorithms may cause distortion in the registered image content. This distortion is caused by the algorithm's inaccurate calculation of some pixel values ​​in the forward optical flow, and can therefore be referred to as distortion in the forward optical flow.

[0105] If the optical flow-based image registration algorithm is ideal, there is no distortion in the forward optical flow, and the forward and reverse optical flows are opposite to each other (the opposite optical flow is when each pixel value in that optical flow is the opposite of its value). In this case, the first difference image can be an image with all pixel values ​​being 0. However, if the optical flow-based image registration algorithm is not ideal, distortion may occur in the forward optical flow. In the distorted regions, the pixel value difference between the forward and reverse optical flows is large, resulting in a larger pixel value in the calculated first difference image within the distorted regions. In other words, by calculating the difference in corresponding pixel values ​​between the forward and reverse optical flows, distorted regions in the forward optical flow can be detected, and the detection result is reflected in the first difference image.

[0106] If there are distorted regions in the forward optical flow, the pixel values ​​in these distorted regions should be corrected or replaced; otherwise, the quality of the registered image will be affected. However, since homography corresponds to a global transformation, there are no local distortions in homography. Therefore, after calculating the first difference image, the optical flow fusion mask can be reasonably determined based on it. This allows for the use of more or more pixel values ​​from the homography optical flow in distorted regions during optical flow fusion, and more or more pixel values ​​from the forward optical flow in non-distorted regions. This corrects or replaces the pixel values ​​in the distorted regions of the forward optical flow. Then, by using the fused optical flow to transform the image to be registered, a higher quality registered image can be obtained.

[0107] For example, when the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to homography optical flow, the optical flow fusion mask can be determined based on the first difference image as follows:

[0108] First, the first difference image is binarized to obtain a binary image with pixel values ​​of 0 or 1. Binarization involves setting a threshold; if a pixel value in the first difference image is large (greater than the threshold), the corresponding pixel value is set to 1 in the binary image; otherwise, the corresponding pixel value is set to 0. Then, morphological filtering is applied to the obtained binary image to obtain an optical flow fusion mask. Morphological filtering can include erosion and / or dilation operations, its main function being to reduce noise (e.g., isolated points) in the binary image and improve the quality of the fused optical flow mask. Note that morphological filtering does not change the binary characteristics of the image; that is, the optical flow fusion mask obtained after morphological filtering is still binary. Alternatively, the binary image can be directly used as the optical flow fusion mask, omitting the above morphological filtering step.

[0109] The following analysis explains why the optical flow fusion mask calculated using the above method is beneficial for eliminating distortion in the forward optical flow:

[0110] As can be seen from the calculation method of the first difference image, a large pixel value in the first difference image indicates that there is distortion in the forward optical flow at that pixel position. The pixel value at that pixel position in the optical flow fusion mask calculated in the above way is likely to be 1. Since the pixel value in the optical flow fusion mask represents the fusion weight corresponding to the homography optical flow, the pixel value of the fused optical flow at that pixel position will come entirely from the homography optical flow rather than the forward optical flow. Therefore, it is beneficial to eliminate the distortion in the forward optical flow and improve the registration result of the image.

[0111] If the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the forward optical flow, then only appropriate adjustments need to be made to the steps of determining the optical flow fusion mask based on the first difference image. For example, the logic of the binarization operation can be changed to: set a threshold, if a pixel value in the first difference image is large (greater than the threshold), then set the corresponding pixel value to 0 in the binarized image; otherwise, set the corresponding pixel value to 1 in the binarized image.

[0112] Method 2.3: First, calculate the difference between corresponding pixel values ​​in the forward optical flow and the homography optical flow to obtain the second difference image, and then determine the optical flow fusion mask based on the second difference image.

[0113] The difference between corresponding pixel values ​​in forward optical flow and homography represents the degree of difference between two pixel values, but the specific definition of the difference is not limited. The second difference image has the same size as both the forward and homography optical flow images. For example, referring to the calculation of the first difference image, one way to calculate the second difference image is as follows:

[0114] S_H(x,y)=abs(flow(x,y,1)-flow_H(x,y,1))+abs(flow(x,y,2)-flow_H(x,y,2))

[0115] Where S_H represents the second difference image, flow represents the forward optical flow, flow_H represents the homography optical flow, abs represents the absolute value, (x,y) represents any coordinate position, M(x,y) represents the pixel value of matrix M at coordinate (x,y), M(x,y,1) represents the x component of M(x,y), M(x,y,2) represents the y component of M(x,y), and M can be S_H, flow, or flow_H.

[0116] It should be noted that since both forward optical flow and homography optical flow are calculated based on a reference image, the above formulas use "-" signs instead of "+" signs before flow_inverse(x,y,1) and flow_inverse(x,y,2).

[0117] Obviously, the second difference image can also be calculated using other methods, for example:

[0118] S_H(x,y)=sqrt((flow(x,y,1)+flow_H(x,y,1))^2+(flow(x,y,2)+flow_H(x,y,2))^2)

[0119] Here, ^2 represents squaring, sqrt represents sqrting, and the other parts of the formula will not be explained again.

[0120] Since homography corresponds to a global transformation, there is no local distortion in homography. Therefore, if the pixel value of the forward optical flow in a certain region differs significantly from the pixel value of the homography in the same region, it indicates that there is distortion in the forward optical flow in that region. In other words, by calculating the difference between the corresponding pixel values ​​in the forward optical flow and the homography, the distorted region in the forward optical flow can be detected. The detection result is reflected in the second difference image.

[0121] After obtaining the second difference image, the optical flow fusion mask is reasonably determined based on it. This allows for the use of pixel values ​​from the homography optical flow, either more extensively or entirely, in distorted regions, and pixel values ​​from the forward optical flow, either more extensively or entirely, in non-distorted regions. Such optical flow fusion effectively eliminates or at least suppresses the distortion present in the forward optical flow. Then, by using the fused optical flow to transform the image to be registered, a higher quality registered image can be obtained.

[0122] For example, in the case where the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to homography optical flow, a monotonically increasing transformation function can be used to map the pixel values ​​in the second difference image to between 0 and 1 (whether the endpoint values ​​of 0 and 1 can be taken depends on the different implementations of the transformation function) to obtain the optical flow fusion mask.

[0123] The specific form of the transformation function is not limited. For example, a possible transformation function is as follows:

[0124] f(x)=1-exp(-(x-thresh)^2 / sigma)

[0125] Here, exp represents the exponential function, thresh and sigma are preset constants, and sigma > 0. Clearly, f(x) is increasing, and since 1 ≥ exp(-(x-thresh)^2 / sigma) > 0, therefore 1 > f(x) ≥ 0, thus mapping x to the range 0 to 1. Using this transformation function, the optical flow fusion mask can be calculated according to the following formula.

[0126] flow_fusion_mask(x,y)=1-exp(-(S_H(x,y)-thresh)^2 / sigma)

[0127] In this formula, `flow_fusion_mask` represents the optical flow fusion mask, `S_H` represents the second difference image, `(x,y)` represents any coordinate position, and `M(x,y)` represents the pixel value of matrix `M` at coordinates `(x,y)`. `M` can be either `flow_fusion_mask` or `S_H`. It should be noted that the `exp(-(S_H(x,y)-thresh)^2 / sigma)` part can be considered as a Gaussian distribution expression, resulting in a smoother fusion weight `flow_fusion_mask(x,y)` calculated using this formula, which helps reduce noise in the fused image.

[0128] In an alternative approach, the second difference image after pixel value mapping can be used as an optical flow fusion mask instead of being directly used. Instead, it can be binarized first and then the resulting binarized image can be used as an optical flow fusion mask. Furthermore, morphological filtering can be applied to the binarized image.

[0129] The following analysis explains why the optical flow fusion mask calculated using the above method is beneficial for eliminating distortion in the forward optical flow:

[0130] As can be seen from the calculation method of the second difference image, a larger pixel value in the second difference image indicates that there is distortion in the forward optical flow at that pixel location. Since the transformation function is monotonically increasing, the pixel values ​​in the second difference image remain unchanged after mapping. That is, if a pixel value in the second difference image is large (or small), its corresponding pixel value in the optical flow fusion mask is also large (or small). Therefore, the pixel value at the distortion location in the optical flow fusion mask calculated according to the above implementation method is close to or equal to 1. Furthermore, since the pixel value in the optical flow fusion mask represents the fusion weight corresponding to the homography optical flow, the pixel value of the fused optical flow at that pixel location will mainly or completely originate from the homography optical flow rather than the forward optical flow. Therefore, this is beneficial for eliminating distortion in the forward optical flow and improving the image registration result.

[0131] If the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the forward optical flow, then only appropriate adjustments need to be made to the steps of determining the optical flow fusion mask based on the second difference image. For example, the transformation function can be changed to a monotonically decreasing function.

[0132] Furthermore, in method 2.2, determining the optical flow fusion mask based on the first difference image can also be done without using a binarization method, but rather similar to method 2.3, by using a transformation function to map the pixel values ​​in the first difference image to a range of 0 to 1; correspondingly, in method 2.3, determining the optical flow fusion mask based on the second difference image can also be done without using a transformation function, but rather similar to method 2.2, by using a binarization operation to map the pixel values ​​in the second difference image to 0 or 1.

[0133] Method 2.4: First, using the intermediate image as a reference, the reverse optical flow between the intermediate image and the reference image is calculated using an optical flow-based registration algorithm, and the difference between corresponding pixel values ​​in the forward and reverse optical flows is calculated to obtain a first difference image; then, the difference between corresponding pixel values ​​in the forward and homography optical flows is calculated to obtain a second difference image; finally, an optical flow fusion mask is calculated based on the first and second difference images.

[0134] The steps for calculating the first difference image and the second difference image have been described in methods 2.2 and 2.3 respectively, and will not be repeated here. Furthermore, the order in which these two steps are executed is not limited, and they may be executed in parallel.

[0135] As described above, the first difference image detects distorted regions based on the difference between forward and reverse optical flow, while the second difference image detects distorted regions based on the difference between forward and homography optical flow. The inventors have discovered that these two methods are complementary in detecting distorted regions in forward optical flow. Therefore, by rationally determining the optical flow fusion mask based on the first and second difference images, the suppression effect on optical flow distortion during optical flow fusion can be enhanced (compared to implementations 2.2 and 2.3 that only use the first or second difference image to determine the optical flow fusion mask), thereby further improving image registration accuracy.

[0136] For example, when the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to homography optical flow, the optical flow fusion mask can be determined based on the first difference image and the second difference image as follows:

[0137] First, the first difference image is binarized to obtain a binary image with pixel values ​​of 0 or 1. Then, morphological filtering is performed on the binarized image to obtain the first mask. This step is similar to the steps in Method 2.2 for binarizing the first difference image and calculating the optical flow fusion mask, and will not be repeated here.

[0138] Then, using a monotonically increasing transformation function, the pixel values ​​in the second difference image are mapped to a range of 0 to 1, resulting in the second mask. This step is similar to the step in method 2.3 where the transformation function is used to map the second difference image into an optical flow fusion mask, and will not be repeated here. Additionally, it should be noted that the order in which this step and the previous step are executed is not limited, and they can be performed in parallel.

[0139] Next, the corresponding pixel values ​​from the first and second masks are superimposed to obtain the third mask. Since the first and second masks have the same size, their corresponding pixel values ​​can be superimposed. "Superimposition" here can refer to directly summing the corresponding pixel values ​​or to a weighted sum. To ensure that the summation values ​​are of the same order of magnitude (if two numbers are not of the same order of magnitude, the sum will largely depend on the larger of the two numbers, with the smaller contributing little to the result), the pixel values ​​in the masks were limited to between 0 and 1 during the calculation of the first and second masks.

[0140] Finally, the pixel values ​​in the third mask are mapped to a range of 0 to 1 to obtain the optical flow fusion mask. Since pixel values ​​were superimposed in the previous step, the pixel values ​​in the third mask may be greater than 1 (because there are no pixel values ​​less than 0 in the first and second masks, so there are no pixel values ​​less than 0 after superposition). However, the fusion weights are required to be between 0 and 1, so mapping is necessary. During mapping, it should be ensured that the larger pixel value, after mapping, should at least not be less than the smaller pixel value.

[0141] The mapping in this step can be implemented in several ways: for example, the transformation function f(x) given earlier can be used for mapping; for example, the maximum and minimum pixel values ​​in the third mask can be determined first, and then the maximum value can be mapped to 1, the minimum value to 0, and the remaining pixel values ​​can be mapped according to the proportional relationship between the maximum and minimum values; for example, all pixel values ​​greater than 1 in the third mask (if any) can be uniformly mapped to 1, and the remaining pixel values ​​can be mapped to the interval [0,1); for example, the third mask can be binarized (binarization is also a mapping method), and so on.

[0142] The following analysis explains why the optical flow fusion mask calculated using the above method can better eliminate distortions in the forward optical flow:

[0143] Since the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to homography optical flow, if a pixel value of 1 in the first mask is 1, it indicates that if optical flow fusion is performed using the first mask, the pixel value at that location will be entirely based on the homography optical flow. Therefore, the first mask, to some extent, also represents the result of detecting distorted regions based on forward and reverse optical flow. Conversely, if a pixel value in the second mask is large (close to or equal to 1), it indicates that if optical flow fusion is performed using the second mask, the pixel value at that location will be largely or entirely based on the homography optical flow. Therefore, the second mask, to some extent, also represents the result of detecting distorted regions based on forward and homography optical flow.

[0144] Furthermore, by superimposing the first and second masks, for regions where the pixel values ​​in both masks are relatively large, the pixel values ​​will increase further after superposition. Similarly, for regions where only one mask has a relatively large pixel value (i.e., distorted regions that one detection method failed to detect), the pixel values ​​will also become relatively large after superposition. It is evident that superimposing the two masks enhances the detection capability for distorted regions in the forward optical flow. Moreover, the final mapping step sets the pixel values ​​of the fused optical flow in the distorted regions to 1 or a value close to 1, meaning that homography is primarily or entirely used in the distorted regions, thereby improving image registration accuracy.

[0145] If the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the forward optical flow, then the calculation methods for the first and second masks can be appropriately adjusted. For specific methods, please refer to the relevant content in Methods 2.2 and 2.3, which will not be repeated here.

[0146] Figure 2 A functional block diagram of the image registration apparatus 200 provided in an embodiment of this application is shown. (Refer to...) Figure 2 The image registration device 200 includes:

[0147] Image acquisition module 210 is used to acquire a reference image and an intermediate image; wherein, the intermediate image is an image to be registered, or an image obtained by processing the image to be registered, whose brightness is at the same level as the reference image;

[0148] The homography registration module 220 is used to calculate the homography matrix between the intermediate image and the reference image using a homography matrix-based registration algorithm, with the reference image as a reference, and to calculate the homography optical flow corresponding to the homography matrix.

[0149] The optical flow registration module 230 is used to calculate the forward optical flow between the intermediate image and the reference image using an optical flow-based registration algorithm, with the reference image as a reference.

[0150] The optical flow fusion module 240 is used to fuse the homography optical flow and the forward optical flow to obtain a fused optical flow;

[0151] The image transformation module 250 is used to transform the image to be registered using the fused optical flow to obtain a registered image.

[0152] In one implementation of the image registration device 200, the optical flow fusion module 240 fuses the homography optical flow and the forward optical flow to obtain a fused optical flow, including: calculating an optical flow fusion mask based on the forward optical flow; wherein the pixel values ​​in the optical flow fusion mask represent the fusion weights; and fusing the homography optical flow and the forward optical flow using the optical flow fusion mask to obtain the fused optical flow.

[0153] In one implementation of the image registration device 200, the optical flow fusion module 240 calculates an optical flow fusion mask based on the forward optical flow, including: using the intermediate image as a reference, calculating the reverse optical flow between the intermediate image and the reference image using the optical flow-based registration algorithm; calculating the difference degree of corresponding pixel values ​​in the forward optical flow and the reverse optical flow to obtain a first difference degree image; and determining the optical flow fusion mask based on the first difference degree image.

[0154] In one implementation of the image registration device 200, the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the homography optical flow. The optical flow fusion module 240 determines the optical flow fusion mask based on the first difference image, including: binarizing the first difference image to obtain a binarized image with pixel values ​​of 0 or 1; and performing morphological filtering on the binarized image to obtain the optical flow fusion mask.

[0155] In one implementation of the image registration device 200, the optical flow fusion module 240 calculates an optical flow fusion mask based on the forward optical flow, including: calculating the difference degree of corresponding pixel values ​​in the forward optical flow and the homography optical flow to obtain a second difference degree image; and determining the optical flow fusion mask based on the second difference degree image.

[0156] In one implementation of the image registration device 200, the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the homography optical flow. The optical flow fusion module 240 determines the optical flow fusion mask based on the second difference image, including: using a monotonically increasing transformation function to map the pixel values ​​in the second difference image to a range of 0 to 1 to obtain the optical flow fusion mask.

[0157] In one implementation of the image registration device 200, the optical flow fusion module 240 calculates an optical flow fusion mask based on the forward optical flow, including: using the intermediate image as a reference, calculating the reverse optical flow between the intermediate image and the reference image using the optical flow-based registration algorithm; calculating the difference degree of corresponding pixel values ​​in the forward optical flow and the reverse optical flow to obtain a first difference degree image; calculating the difference degree of corresponding pixel values ​​in the forward optical flow and the homography optical flow to obtain a second difference degree image; and calculating the optical flow fusion mask based on the first difference degree image and the second difference degree image.

[0158] In one implementation of the image registration device 200, the pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the homography optical flow. The optical flow fusion module 240 calculates the optical flow fusion mask based on the first difference image and the second difference image, including: binarizing the first difference image to obtain a binarized image with pixel values ​​of 0 or 1; performing morphological filtering on the binarized image to obtain a first mask; using a monotonically increasing transformation function to map the pixel values ​​in the second difference image to a range of 0 to 1 to obtain a second mask; superimposing the corresponding pixel values ​​in the first mask and the second mask to obtain a third mask; and mapping the pixel values ​​in the third mask to a range of 0 to 1 to obtain the optical flow fusion mask.

[0159] In one implementation of the image registration device 200, the image acquisition module 210 acquires a reference image and an intermediate image, including: acquiring a reference image and an image to be registered; determining whether the brightness of the image to be registered is at the same level as the brightness of the reference image; if they are not at the same level, adjusting the brightness of the image to be registered to obtain the intermediate image whose brightness is at the same level as the reference image.

[0160] The image registration device 200 provided in this application embodiment has been described in the foregoing method embodiment in terms of its implementation principle and the resulting technical effects. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the method embodiment.

[0161] Figure 3 This illustration shows a possible structure of the electronic device 300 provided in an embodiment of this application. (Refer to...) Figure 3 The electronic device 300 includes a processor 310, a memory 320, and a communication interface 330. These components are interconnected and communicate with each other via a communication bus 340 and / or other forms of connection mechanism (not shown).

[0162] The processor 310 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 310 can be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it can also be a special-purpose processor, including a Neural-network Processing Unit (NPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Furthermore, when there are multiple processors 310, some can be general-purpose processors and others can be special-purpose processors.

[0163] The memory 320 includes one or more (only one is shown in the figure), which may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0164] Processor 310 and other possible components can access memory 320, read and / or write data therein. In particular, memory 320 can store one or more computer program instructions, which processor 310 can read and execute to implement the image registration method provided in the embodiments of this application.

[0165] Communication interface 330 includes one or more (only one is shown in the figure) that can be used to communicate directly or indirectly with other devices for data exchange. Communication interface 330 may include interfaces for wired and / or wireless communication. If communication with other devices is not required, electronic device 300 may not include communication interface 330.

[0166] Understandable. Figure 3 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown. Figure 3 The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 300 may be a physical device, such as a mobile phone, camera, camcorder, wearable device, tablet computer, PC, laptop computer, server, etc., or it may be a virtual device, such as a virtual machine, virtualization container, etc. Furthermore, electronic device 300 is not limited to a single device; it can also be a combination of multiple devices or a cluster of a large number of devices.

[0167] This application also provides a computer-readable storage medium storing computer program instructions. These computer program instructions are read and executed by a computer's processor to perform the image registration method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 3 The memory 320 in the electronic device 300.

[0168] This application also provides a computer program product, which includes computer program instructions. These computer program instructions are read and executed by a processor to perform the image registration method provided in this application.

[0169] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image registration method, characterized in that, include: Acquire a reference image and an intermediate image; wherein the intermediate image is the image to be registered, or an image obtained by processing the image to be registered, whose brightness is at the same level as the reference image; Using the reference image as a reference, a registration algorithm based on the homography matrix is ​​used to calculate the homography matrix between the intermediate image and the reference image, and the homography optical flow corresponding to the homography matrix is ​​calculated. Using the reference image as a reference, the forward optical flow between the intermediate image and the reference image is calculated using an optical flow-based registration algorithm; The homography optical flow and the forward optical flow are fused to obtain a fused optical flow; The image to be registered is transformed using the fused optical flow to obtain a registered image.

2. The image registration method according to claim 1, characterized in that, The process of fusing the homography optical flow and the forward optical flow to obtain a fused optical flow includes: The optical flow fusion mask is calculated based on the forward optical flow; wherein the pixel values ​​in the optical flow fusion mask represent the fusion weights; The homography optical flow and the forward optical flow are fused using the optical flow fusion mask to obtain the fused optical flow.

3. The image registration method according to claim 2, characterized in that, The calculation of the optical flow fusion mask based on the forward optical flow includes: Using the intermediate image as a reference, the reverse optical flow between the intermediate image and the reference image is calculated using the optical flow-based registration algorithm; Calculate the difference between corresponding pixel values ​​in the forward optical flow and the reverse optical flow to obtain a first difference image; The optical flow fusion mask is determined based on the first difference image.

4. The image registration method according to claim 2, characterized in that, The calculation of the optical flow fusion mask based on the forward optical flow includes: The difference between corresponding pixel values ​​in the forward optical flow and the homography optical flow is calculated to obtain a second difference image; The optical flow fusion mask is determined based on the second difference image.

5. The image registration method according to claim 2, characterized in that, The calculation of the optical flow fusion mask based on the forward optical flow includes: Using the intermediate image as a reference, the reverse optical flow between the intermediate image and the reference image is calculated using the optical flow-based registration algorithm; Calculate the difference between corresponding pixel values ​​in the forward optical flow and the reverse optical flow to obtain a first difference image; The difference between corresponding pixel values ​​in the forward optical flow and the homography optical flow is calculated to obtain a second difference image; The optical flow fusion mask is calculated based on the first difference image and the second difference image.

6. The image registration method according to claim 5, characterized in that, The pixel values ​​in the optical flow fusion mask represent the fusion weights corresponding to the homography optical flow. The step of calculating the optical flow fusion mask based on the first difference image and the second difference image includes: The first difference image is binarized to obtain a binarized image with pixel values ​​of 0 or 1; Morphological filtering is performed on the binarized image to obtain the first mask; Using a monotonically increasing transformation function, the pixel values ​​in the second difference image are mapped to a range of 0 to 1 to obtain the second mask; By superimposing the corresponding pixel values ​​from the first mask and the second mask, a third mask is obtained; The pixel values ​​in the third mask are mapped to a range of 0 to 1 to obtain the optical flow fusion mask.

7. The image registration method according to any one of claims 1-6, characterized in that, The acquisition of the reference image and intermediate image includes: Acquire the reference image and the image to be registered; Determine whether the brightness of the image to be registered is at the same level as the brightness of the reference image; If they are not at the same level, the brightness of the image to be registered is adjusted to obtain the intermediate image whose brightness is at the same level as the reference image.

8. A computer program product, characterized in that, It includes computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer program instructions, which are read and executed by the processor to perform the method of any one of claims 1-7.

Images