Image registration method, device, electronic device and computer readable storage medium

Abstract

Image registration method, device, electronic device and computer-readable storage medium. The present application relates to an image registration method, which globally registers a reference image block in a reference frame with an image block to be registered in a frame to be registered, and determines a global registration result of the reference image block. Block matching is performed on the reference image block in the reference frame with the image block to be registered in the frame to be registered, and the block matching result of the reference image block is determined. According to the global registration result and the block matching result, the target image registration result of the reference frame is determined. The registration effect of the global registration result and the block matching result can be comprehensively considered to determine the target image registration result of the reference frame. The accuracy of image registration of the reference frame and the frame to be registered is improved, and thus the noise reduction effect of the image obtained by fusing the reference frame and the frame to be registered is also improved. At the same time, because the registration effect of the global registration result and the block matching result is comprehensively considered, the difference in noise reduction effect at the boundaries of different regions on the fused image is small.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image registration method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the continuous development of image processing technology, image processing methods based on multi-frame images have emerged. Due to the influence of the camera module's own parameters, the information collected by a single frame image is always limited. Therefore, by using image processing methods based on multi-frame images, multiple frames of images can be processed to synthesize clearer images, thereby restoring more image details.

[0003] The first step in using the above image processing method is to register multiple frames of images. Image registration refers to the position correction of the non-reference image based on the reference image in the multiple frames of images, calculating the displacement of the reference image relative to the non-reference image, so that the reference image can be processed according to the displacement to obtain an image with a content close to that on the non-reference image.

[0004] However, after using traditional image registration methods to register multiple frames of images, the noise reduction effects in different regions of the obtained images are poor, and the noise reduction effects at the boundaries of different regions on the images are quite different. Summary of the invention

[0005] The embodiments of the present application provide an image registration method, an apparatus, an electronic device, and a computer-readable storage medium, which can improve the registration accuracy of the image and thereby improve the noise reduction effect of the image.

[0006] In one aspect, an image registration method is provided, the method comprising:

[0007] Performing global registration on the reference image block in the reference frame and the image block to be registered in the frame to be registered, and determining a global registration result of the reference image block;

[0008] Performing block matching on a reference image block in the reference frame and an image block to be registered in the frame to be registered, and determining a block matching result of the reference image block;

[0009] A target image registration result of the reference frame is determined according to the global registration result and the block matching result.

[0010] In another aspect, an image registration device is provided, the device comprising:

[0011] A global registration module, used to globally register the reference image block in the reference frame with the image block to be registered in the frame to be registered, and determine the global registration result of the reference image block;

[0012] A block matching module, used for performing block matching on a reference image block in the reference frame and an image block to be registered in the frame to be registered, and determining a block matching result of the reference image block;

[0013] The target image registration result determination module is used to determine the target image registration result of the reference frame according to the global registration result and the block matching result.

[0014] On the other hand, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the image registration method described above.

[0015] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image registration method described above are implemented.

[0016] The above-mentioned image registration method globally registers the reference image block in the reference frame with the image block to be registered in the frame to be registered, and determines the global registration result of the reference image block. The reference image block in the reference frame is block-matched with the image block to be registered in the frame to be registered, and determines the block matching result of the reference image block. Based on the global registration result and the block matching result, the target image registration result of the reference frame is determined.

[0017] Since global registration is used for static areas, a better registration effect can be achieved, while block matching is used for moving areas, a better registration effect can be achieved. Therefore, for the reference image block in the reference frame, global registration and block matching are performed with the image block to be registered in the frame to be registered, respectively, to determine the global registration result and block matching result of the reference image block. Finally, the target image registration result of the reference frame is determined based on the two registration results. That is, the registration effect of the global registration result and the block matching result can be comprehensively considered to determine the target image registration result of the reference frame. The accuracy of image registration between the reference frame and the frame to be registered is improved, and thus the denoising effect of the image obtained by fusing the reference frame and the frame to be registered is also improved. At the same time, because the registration effect of the global registration result and the block matching result is comprehensively considered, the denoising effect at the boundaries of different regions on the fused image is less different. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 A diagram of an application environment of an image registration method in an embodiment;

[0020] Figure 2 is a flow chart of an image registration method in one embodiment;

[0021] Figure 3 A schematic diagram of the structure of a first image pyramid of a reference frame and a second image pyramid of a frame to be registered in one embodiment;

[0022] Figure 4 for Figure 2 A flowchart of a method for determining a global registration result corresponding to a reference image block in a k-th layer of reference image from a k-th layer of images to be registered;

[0023] Figure 5 A flowchart of a method for calculating a set of affine transformation matrices between a reference frame and a frame to be registered in one embodiment;

[0024] Figure 6 is a schematic diagram of calculating a set of affine transformation matrices between a reference frame and a frame to be registered in one embodiment;

[0025] Figure 7 for Figure 5 In the flowchart of the method for mapping feature points corresponding to the original feature points of the reference image block by mapping the original feature points of the reference image block between the reference frame and the frame to be registered through the optical flow algorithm;

[0026] Figure 8 is a schematic diagram of performing block matching between a first layer reference image and a first layer to be registered image in one embodiment;

[0027] Fig. 9 is a schematic diagram of performing block matching between a second layer reference image and a second layer to-be-registered image in one embodiment;

[0028] Fig.10 is a flow chart of an image registration method in another embodiment;

[0029] Fig.11A A flowchart of an image registration method in yet another embodiment;

[0030] Fig. 11B FIG. 1 is a schematic diagram of image fusion of a reference frame and a frame to be registered in one embodiment;

[0031] Fig.12 is a flowchart of an image registration method in a specific embodiment;

[0032] Fig.13It is a schematic diagram of a multi-frame fusion result obtained after a traditional multi-frame fusion result and a multi-frame fusion result obtained after adopting the image registration method in the present application in one embodiment;

[0033] Fig.14 is a structural block diagram of an image registration device in one embodiment;

[0034] Fig.15 is a structural block diagram of an image registration device in another embodiment;

[0035] Fig.16 FIG. 1 is a schematic diagram of the internal structure of an electronic device in an embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first client may be referred to as a second client, and similarly, a second client may be referred to as a first client. Both the first client and the second client are clients, but they are not the same client.

[0038] Figure 1 FIG. 1 is an application scenario diagram of an image registration method in an embodiment. Figure 1 As shown, the application environment includes an electronic device 120, which obtains multiple frames of images by shooting the same scene through a camera module. A reference frame and a frame to be registered are determined from multiple frames of images, wherein the reference frame is an image with a better image effect determined from multiple frames of images, such as an image with a relatively high definition, and this application does not limit this. Among them, the frame to be registered is a part or all of the images of the multiple frames of images except the reference frame as the frame to be registered, and this application does not limit this. The electronic device 120 globally registers the reference image block in the reference frame with the image block to be registered in the frame to be registered, and determines the global registration result of the reference image block; performs block matching on the reference image block in the reference frame and the image block to be registered in the frame to be registered, and determines the block matching result of the reference image block; and determines the target image registration result of the reference frame according to the global registration result and the block matching result. Here, the electronic device 120 can be any terminal device such as a mobile phone, a tablet computer, a PDA (Personal Digital Assistant), a wearable device (smart watch, etc.), a smart home, etc.

[0039] Figure 2 FIG. 1 is a flow chart of an image registration method in one embodiment. The image registration method in this embodiment is executed on Figure 1 The electronic device in the example is used to describe. Figure 2 As shown, the image registration method includes steps 220 to 260.

[0040] Step 220 , globally aligning the reference image block in the reference frame with the image block to be registered in the frame to be registered, and determining a global alignment result of the reference image block.

[0041] Specifically, the electronic device 120 obtains multiple frames of images by shooting the same scene through a camera module, and determines a reference frame and a frame to be registered from the multiple frames of images. Here, the number of reference frames is generally one, and the number of frames to be registered is generally multiple, and of course the number of frames to be registered can also be one. Among them, the reference frame is an image with a better image effect determined from multiple frames of images, and the image effect here can refer to image parameters such as brightness or clarity. For example, the image with the highest brightness is determined from multiple frames of images as a reference frame, or the image with the highest clarity is determined from multiple frames of images as a reference frame, and this application does not limit this. Among them, the frame to be registered is to use part or all of the images of the multiple frames of images excluding the reference frame as the frame to be registered, and this application does not limit this. For example, from all images of the multiple frames of images excluding the reference frame, a partial image whose image effect is second only to the reference frame is selected as the frame to be registered.

[0042] After the reference frame and the frame to be registered are determined from the multiple frames, the reference frame is divided into reference image blocks according to a preset size, and the frame to be registered is divided into image blocks to be registered having the same size as the reference image blocks. That is, the reference frame is divided into reference image blocks according to the same division method, and the frame to be registered is divided into image blocks to be registered having the same size as the reference image blocks.

[0043] Then, each reference image block in the reference frame is globally registered with the image block to be registered in the frame to be registered, and the global registration result of the reference image block is determined. The global registration result of the reference image block refers to the globally registered image block corresponding to the reference image block determined from the frame to be registered and the position information of the globally registered image block in the frame to be registered. Specifically, when determining the global registration result of the reference image block, the reference image block in the reference frame can be globally registered with the image block to be registered in the frame to be registered based on the affine transformation matrix between the reference frame and the frame to be registered, and the global registration result of the reference image block can be determined.

[0044] Step 240 , block matching is performed on the reference image block in the reference frame and the image block to be registered in the frame to be registered, and a block matching result of the reference image block is determined.

[0045] The block matching result of the reference image block refers to the block matching image block corresponding to the reference image block determined from the frame to be registered and the position information of the block matching image block. Specifically, when determining the block matching result of the reference image block, each reference image block in the reference frame can be block matched with the image block to be registered in the frame to be registered to determine the block matching result of the reference image block. For example, the similarity between each reference image block in the reference frame and the image block to be registered in the frame to be registered is calculated, the image block to be registered with the highest similarity is used as the block matching image block of the reference image block, and the position information of the block matching image block is determined in the frame to be registered.

[0046] Step 260: Determine the target image registration result of the reference frame according to the global registration result and the block matching result.

[0047] For each reference image block on the reference frame, the global registration result of each reference image block and the block matching result of each reference image block are obtained on each frame to be registered through the above registration process. For each frame to be registered, based on the global registration result of each reference image block and the block matching result of each reference image block obtained on the frame to be registered, the target image registration result of the reference frame can be determined. Here, the target image registration result includes the target registration image block and the position information of the target registration image block in the frame to be registered.

[0048] Based on the target image registration result between the reference frame and each frame to be registered, the reference frame can be registered with each frame to be registered. After the registration, the registered reference frame can be fused with each frame to be registered to generate a fused image.

[0049] The image registration method in the embodiment of the present application can achieve a better registration effect by adopting global registration for the static area, and can achieve a better registration effect by adopting block matching for the moving area. Therefore, for the reference image block in the reference frame, global registration and block matching are respectively performed with the image block to be registered in the frame to be registered, and the global registration result and block matching result of the reference image block are determined. Finally, the target image registration result of the reference frame is determined based on the two registration results. That is, the registration effect of the global registration result and the block matching result can be comprehensively considered to determine the target image registration result of the reference frame. The accuracy of image registration of the reference frame and the frame to be registered is improved, and thus the noise reduction effect of the image obtained by fusing the reference frame and the frame to be registered is also improved. At the same time, because the registration effect of the global registration result and the block matching result are comprehensively considered, the difference in noise reduction effect at the boundaries of different regions on the image obtained by fusing the registered images is small.

[0050] In one embodiment, an image registration method is provided, further comprising:

[0051] The reference frame and the frame to be registered are respectively subjected to image preprocessing to obtain a first image pyramid of the reference frame and a second image pyramid of the frame to be registered; the first image pyramid includes n layers of reference images, and the second image pyramid includes n layers of images to be registered; wherein the n-th layer of reference image is the original image of the reference frame, and the n-th layer of image to be registered is the original image of the frame to be registered; the k-th layer of reference image in the first image pyramid corresponds to the k-th layer of image to be registered in the second image pyramid, 1≤k≤n, and k is a positive integer;

[0052] A reference image block is obtained from a k-th layer reference image in the first image pyramid, and an image block to be registered is obtained from a k-th layer image to be registered in the second image pyramid.

[0053] Specific, combined Figure 3 , which is a schematic diagram of the structures of a first image pyramid of a reference frame and a second image pyramid of a frame to be registered.

[0054] The image pyramid is a collection of images arranged in a pyramid shape with gradually decreasing resolutions. The bottom of the pyramid is a high-resolution image to be processed (e.g., the original image to be processed), and the top is a low-resolution image to be processed. When moving from the bottom to the top of the pyramid, the size and resolution of the image will decrease. Figure 3 As shown in FIG. 1 , it is a schematic diagram of inverting the pyramid, the first layer image is the top of the pyramid, the nth layer image is the bottom of the pyramid (original image), and images of different layers are obtained by downsampling the original image.

[0055] The image pyramid here can be a Gaussian pyramid or a Laplacian pyramid, which is not limited in this application. Among them, the Gaussian pyramid includes a series of downsampled images obtained by Gaussian smoothing and subsampling, that is, the k-th layer of the Gaussian pyramid can obtain the k+1 layer of Gaussian images by smoothing and subsampling. The Gaussian pyramid contains a series of low-pass filters, whose cutoff frequencies gradually increase by a factor of 2 from the upper layer to the lower layer, so the Gaussian pyramid can span a large frequency range. The Laplacian pyramid includes a series of images obtained by first reducing the original image and then enlarging it. The Laplacian pyramid can be understood as the inverse form of the Gaussian pyramid.

[0056] Specifically, image preprocessing is performed on the reference frame and each frame to be registered, respectively, to obtain a first image pyramid of the reference frame and a second image pyramid of the frame to be registered. The first image pyramid includes n layers of reference images (Gaussian images or Laplacian images), from bottom to top, the first layer reference image, the second layer reference image, and the nth layer reference image. The nth layer reference image is the original image of the reference frame.

[0057] Similarly, the second image pyramid includes n layers of images to be registered (Gaussian images or Laplacian images), which are the first layer of images to be registered, the second layer of images to be registered, and the nth layer of images to be registered from bottom to top. Among them, the nth layer of images to be registered is the original image of the frame to be registered. And the kth layer of reference images in the first image pyramid corresponds to the kth layer of images to be registered in the second image pyramid, 1≤k≤n, and k is a positive integer.

[0058] After constructing the first image pyramid of the reference frame and the second image pyramid of the frame to be registered, when performing image registration, the reference image blocks can be obtained from the k-th layer of the reference image in the first image pyramid, and the image blocks to be registered can be obtained from the k-th layer of the image to be registered in the second image pyramid. That is, the reference image blocks and the image blocks to be registered are obtained from the same layer of the first image pyramid and the second image pyramid, respectively, so as to perform global registration and block matching operations on the reference image blocks and the image blocks to be registered in the same layer.

[0059] In an embodiment of the present application, image preprocessing is performed on the reference frame and the frame to be registered, respectively, to obtain a first image pyramid of the reference frame and a second image pyramid of the frame to be registered. Based on images of different resolutions in the image pyramid, features of different scales on the image can be captured. Then, a reference image block is obtained from the k-th layer reference image in the first image pyramid, and an image block to be registered is obtained from the k-th layer image to be registered in the second image pyramid. Then, by globally registering and block matching the reference image block in the reference frame with the image block to be registered in the frame to be registered, features of different scales on the image can be better captured, thereby improving the accuracy of registration.

[0060] In one embodiment, step 220, globally registering the reference image block in the reference frame with the image block to be registered in the frame to be registered, and determining the global registration result of the reference image block, includes:

[0061] Starting from k=1, the following operations are performed iteratively: for the reference image block in the k-th layer reference image, a global registration result corresponding to the reference image block in the k-th layer reference image is determined from the k-th layer to-be-registered image; until k=n;

[0062] The global registration result includes the globally registered image block and the position information of the globally registered image block.

[0063] Specific, combined Figure 3As shown, a first image pyramid of the reference frame and a second image pyramid of each frame to be registered are constructed, and the first image pyramid includes n layers of reference images, and the second image pyramid includes n layers of images to be registered, 1≤k≤n, and k is a positive integer. Therefore, starting from k=1, for the reference image block A1 in the first layer of the reference image of the reference frame, the reference image block A1 is globally registered with the image block to be registered in the first layer of the image to be registered of the frame to be registered, and the global registration result corresponding to the reference image block in the first layer of the reference image is determined from the first layer of the image to be registered. That is, the globally registered image block B1 corresponding to the reference image block in the first layer of the reference image is determined from the first layer of the image to be registered, and the position information of the globally registered image block B1 is determined.

[0064] Then, let k=2, for the reference image block A2 in the second layer reference image of the reference frame, globally align the reference image block A2 with the image block to be registered in the second layer to be registered image of the frame to be registered, and determine the global alignment result corresponding to the reference image block in the second layer reference image from the second layer to be registered image. That is, determine the globally aligned image block B2 corresponding to the reference image block in the second layer reference image from the second layer to be registered image, and determine the position information of the globally aligned image block B2.

[0065] Iteratively execute the reference image block A in the k-th layer reference image of the reference frame k , the reference image block A k Perform global registration with the image block to be registered in the k-th layer of the image to be registered frame, and determine the global registration result corresponding to the reference image block in the k-th layer of the reference image from the k-th layer of the image to be registered. That is, determine the global registration image block B corresponding to the reference image block in the k-th layer of the reference image from the k-th layer of the image to be registered k , and determine the global registration image block B k The operation of the position information of the first image pyramid is repeated until k = n. For example, assuming n = 5, the 5th layer reference image of the first image pyramid is the reference frame original image, and the 5th layer reference image of the second image pyramid is the to-be-registered frame original image. The above operation is iterated 5 times, and the global registration results corresponding to the reference image blocks in the same layer of the reference image in the first image pyramid are obtained from each layer of the to-be-registered image in the second image pyramid.

[0066] In an embodiment of the present application, a first image pyramid of a reference frame and a second image pyramid of each frame to be registered are constructed, and the first image pyramid includes n layers of reference images, and the second image pyramid includes n layers of images to be registered. When globally registering the reference image block in the k-th layer of the reference image with the image block to be registered in the k-th layer of the image to be registered, global registration is performed in sequence from the top layer of the pyramid to the bottom layer of the pyramid, thereby realizing registration from a low-resolution image to a high-resolution image, that is, realizing a registration process from coarse to fine. In other words, the global registration results corresponding to the reference image blocks in the same layer of the reference image in the first image pyramid can be obtained from each layer of the image to be registered in the second image pyramid. Because the images of different layers in the pyramid have different resolutions, the global registration results of the reference image blocks of the reference frame are obtained from the images to be registered of different resolutions. It is avoided that the global registration results determined only from the frames to be registered miss important features, thereby improving the accuracy of the registration results.

[0067] In one embodiment, Figure 4 As shown, step 220, determining the global registration result corresponding to the reference image block in the k-th layer of reference image from the k-th layer of images to be registered, includes:

[0068] Step 222, calculate an affine transformation matrix set between the reference frame and the frame to be registered; the affine transformation matrix set includes an affine transformation matrix between the reference image block in the reference frame and the image block to be registered in the frame to be registered.

[0069] The affine transformation is actually the superposition of two transformations, namely linear transformation and translation transformation. The transformation between the reference frame and the frame to be registered can be equivalent to the affine transformation. Therefore, for the reference image block in the reference frame, its global registration result in the frame to be registered can be determined based on the affine transformation.

[0070] When calculating the set of affine transformation matrices between the reference frame and the frame to be registered, specifically, in the first step, the affine transformation matrices between each reference image block in the reference frame and the image block to be registered in a frame to be registered are calculated to obtain the local affine transformation matrix of each reference image block. In the second step, based on the affine transformation matrix of each reference image block, a set of affine transformation matrices of the reference frame and the frame to be registered is constructed. In the third step, the operations of the first and second steps are performed for each frame to be registered to generate a set of affine transformation matrices of the reference frame and each frame to be registered. That is, the set of affine transformation matrices includes the affine transformation matrices between the reference image blocks in the reference frame and the image blocks to be registered in each frame to be registered.

[0071] Step 224 , based on the set of affine transformation matrices between the reference frame and the frame to be registered, determine the global registration result corresponding to the reference image block in the k-th layer of reference image from the k-th layer of images to be registered.

[0072] A first image pyramid of the reference frame and a second image pyramid of each frame to be registered are pre-constructed, and the first image pyramid includes n layers of reference images, and the second image pyramid includes n layers of images to be registered, 1≤k≤n, and k is a positive integer. After obtaining the set of affine transformation matrices between the reference frame and the frame to be registered, for the reference image block A in the k-th layer of the reference image of the first image pyramid, k , we can find the reference image block A from the affine transformation matrix set k and the target affine transformation matrix between the image blocks to be registered in the k-th layer of the image to be registered in the second image pyramid.

[0073] Based on the target affine transformation matrix, the k-th layer of the image to be registered can be transformed to obtain the transformed image to be registered. The transformation processing here includes translation, rotation, scaling and other processing of the image to be registered, which is not limited in this application. Then, the reference image block A k The transformed image to be registered is overlapped, and the reference image block A is determined from the transformed image to be registered. k The image blocks to be registered that overlap in spatial position B k At this time, the image block B to be registered k is the reference image block A k The global registration image block, the image block to be registered B k The position information of the reference image block A k The position information of the global registration image block. Image block to be registered B k and the position information of the image block B to be registered constitute the global registration result of the reference image block.

[0074] The above operation is performed for each layer in the second image pyramid to generate reference image blocks A on each layer of the reference image in the first image pyramid. k The global registration result, that is, the image block B to be registered k And the position information of the image block B to be registered.

[0075] In an embodiment of the present application, first, the set of affine transformation matrices between the reference frame and the frame to be registered is calculated, and the registration relationship between the reference frame and the frame to be registered is obtained from the perspective of affine transformation. Then, based on the set of affine transformation matrices between the reference frame and the frame to be registered, the reference image blocks in the k-th layer of the image to be registered and the image blocks to be registered in the k-th layer of the reference image are globally registered, and the global registration results of the reference image blocks in each layer of the reference image are determined. Based on the set of affine transformation matrices, the global registration between the reference image blocks in the reference frame and the image blocks to be registered in the frame to be registered is accurately achieved.

[0076] In one embodiment, Figure 5 As shown, the set of affine transformation matrices between the reference frame and the frame to be registered is calculated, including:

[0077] Step 520: divide the reference frame into reference image blocks, and divide the frame to be registered into image blocks to be registered that have the same size as the reference image blocks.

[0078] Combination Figure 6 As shown, it is a schematic diagram of calculating the set of affine transformation matrices between a reference frame and a frame to be registered in one embodiment. The electronic device obtains multiple frames of images by shooting the same scene through a camera module, and determines a reference frame and multiple frames to be registered from the multiple frames of images. The reference frame is divided into reference image blocks according to a preset size, and the frame to be registered is divided into image blocks to be registered of the same size as the reference image blocks. For example, the reference frame is divided into 4*4 reference image blocks, and the frame to be registered is divided into 4*4 image blocks to be registered according to a size of 4*4 pixels. The size of the divided image blocks is not limited in this application. Of course, it can also be divided according to the size of the reference image block. For example, the reference frame is divided into multiple reference image blocks according to a size of 4*4 pixels, and the frame to be registered is divided into multiple image blocks to be registered according to a size of 4*4 pixels.

[0079] Step 540: For each reference image block, feature point detection is performed on the reference image block to obtain original feature points of the reference image block.

[0080] After dividing the reference frame into reference image blocks according to a preset size, for each reference image block, feature point detection is performed on the reference image block to obtain the original feature points of the reference image block. Here, feature point detection can be performed on the reference image block using a feature point detection algorithm. For example, feature point detection is performed using any of the Harris corner detection algorithm, SIFT (Scale-invariant feature transform) feature detection algorithm, SURF (Speeded Up Robust Features) algorithm, and ORB (Oriented FAST and Rotated BRIEF) algorithm, etc., and this application does not limit this.

[0081] Step 560 , mapping the original feature points of the reference image block between the reference frame and the frame to be registered by using an optical flow algorithm to obtain mapped feature points corresponding to the original feature points of the reference image block.

[0082] Among them, optical flow is the instantaneous speed of the pixel movement of a moving object in space on the observation imaging plane. The optical flow method is a method that uses the change of pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. Therefore, the original feature points of the reference image block are mapped to the frame to be registered through the optical flow algorithm to obtain the mapped feature points corresponding to the original feature points of the reference image block. It can be seen that the optical flow method can well capture the motion information on the image.

[0083] Step 580 , for each reference image block in the reference frame, based on the original feature points and the mapped feature points of the reference image block, determine a set of affine transformation matrices between the reference frame and the frame to be registered.

[0084] For each reference image block in the reference frame, feature point detection is performed on the reference image block to obtain the original feature points of the reference image block. Then, the original feature points of the reference image block are mapped to the frame to be registered through the optical flow algorithm to obtain the mapped feature points corresponding to the original feature points of the reference image block. Therefore, for each reference image block, the coordinates of the original feature points of the reference image block are obtained from the reference frame, and the coordinates of the mapped feature points of the reference image block are obtained from the frame to be registered. The distance between the coordinates is calculated based on the coordinates of the original feature points and the coordinates of the mapped feature points to obtain the affine transformation matrix H1 of the reference image block between the reference frame and the frame to be registered. The reference frame and the frame to be registered are overlapped in the Z-axis direction, and the original feature points of the reference image block are mapped between the reference frame and the frame to be registered through the optical flow algorithm. It can be understood that the reference frame and the frame to be registered are rotated along the Z-axis direction. The affine transformation matrix obtained at this time is For example, combined with Figure 6 As shown, the affine transformation matrix obtained at this time is The present application does not impose any specific limitation on the specific value of H1. Assume that the reference frame is divided into 4*4 reference image blocks, and the frame to be registered is divided into 4*4 image blocks to be registered according to the size of 4*4 pixels. Then, a 4*4 affine transformation matrix H1 can be obtained.

[0085] Finally, assuming that there are m frames to be registered, we can obtain the affine transformation matrices H1, H2, H3...Hm between the reference frame and each frame to be registered, and we can also obtain the affine transformation matrix set H = (H1, H2, H3...Hm) between the reference frame and each frame to be registered.

[0086] In an embodiment of the present application, when calculating the affine transformation matrix set between the reference frame and the frame to be registered, first, the reference frame is divided into reference image blocks, and the frame to be registered is divided into image blocks to be registered of the same size as the reference image blocks. Secondly, for each reference image block, feature point detection is performed on the reference image block to obtain the original feature points of the reference image block. The original feature points of the reference image block are mapped between the reference frame and the frame to be registered through the optical flow algorithm to obtain the mapped feature points corresponding to the original feature points of the reference image block. Finally, for each reference image block in the reference frame, the affine transformation matrix set between the reference frame and the frame to be registered is determined based on the original feature points and the mapped feature points of the reference image block. The optical flow method can more accurately capture the features of the moving area on the image. Therefore, the accuracy of the affine transformation matrix set between the reference frame and each frame to be registered determined based on the optical flow method is higher.

[0087] In one embodiment, Figure 7 As shown, step 560, mapping the original feature points of the reference image block between the reference frame and the frame to be registered by an optical flow algorithm to obtain mapped feature points corresponding to the original feature points of the reference image block, including:

[0088] Step 562: Map the original feature points of the reference image block to the frame to be matched using a first optical flow algorithm to obtain first mapped feature points corresponding to the original feature points of the reference image block.

[0089] The optical flow method here can be the LK (Lucas–Kanade) optical flow method. Of course, this application does not limit this. Among them, the LK optical flow algorithm is an optical flow estimation algorithm for two-frame differences. The first optical flow algorithm here is the forward optical flow algorithm, and the forward optical flow algorithm refers to an algorithm for calculating the forward optical flow from the reference frame to the frame to be matched. The original feature points of the reference image block are mapped to the frame to be matched through the forward optical flow algorithm to obtain the first mapping feature points corresponding to the original feature points of the reference image block.

[0090] Step 564: Map the first mapped feature points to the reference frame using a second optical flow algorithm to generate second mapped feature points corresponding to the original feature points of the reference image block.

[0091] The second optical flow algorithm here is the inverse optical flow algorithm, which refers to an algorithm that calculates the inverse optical flow from the frame to be matched to the reference frame. After the first mapping feature points corresponding to the original feature points of the reference image block are obtained through the forward optical flow algorithm, the first mapping feature points are mapped to the reference frame through the inverse optical flow algorithm to generate second mapping feature points corresponding to the original feature points of the reference image block. Here, the original feature points of the reference image block are mapped back to the reference image block through the forward optical flow algorithm and the inverse optical flow algorithm to obtain the second mapping feature points, but the second mapping feature points do not completely coincide with the original feature points.

[0092] Step 566: Calculate the distance between the second mapped feature point corresponding to the original feature point of the reference image block and the original feature point of the reference image block.

[0093] The second mapping feature point is the feature point obtained by mapping the original feature point of the reference image block back to the reference image block through the forward optical flow algorithm and the reverse optical flow algorithm. Calculate the distance between the second mapping feature point corresponding to the original feature point of the reference image block and the original feature point of the reference image block. Filter out the original feature points corresponding to the distance less than the preset threshold from the calculated distance, and use these original feature points as reliable feature points. Because the distance between the second mapping feature point and the original feature point of the reference image block is small, that is, the deviation between the second mapping feature point and the original feature point of the reference image block is small, it means that the accuracy of the feature points detected from the reference frame is high, that is, the reliability is high.

[0094] Step 568: Determine, from the first mapping feature points, the mapping feature points corresponding to the original feature points of the reference image block according to the distance.

[0095] At the same time, the first mapping feature point corresponding to the distance less than the preset threshold is selected from the calculated distance, that is, the first mapping feature point on the frame to be matched is used as the mapping feature point corresponding to the original feature point of the reference image block. In other words, the first mapping feature point corresponding to the credible feature point is used as the mapping feature point corresponding to the original feature point.

[0096] In an embodiment of the present application, the original feature points of the reference image block are first mapped to the frame to be registered through the forward optical flow algorithm and the reverse optical flow algorithm, and then mapped back to the reference image block to obtain the second mapping feature points. Then, the distance between the second mapping feature points corresponding to the original feature points of the reference image block and the original feature points of the reference image block is calculated. Finally, based on the distance, the first mapping feature points whose distances meet the preset conditions are determined from the first mapping feature points as the mapping feature points corresponding to the original feature points of the reference image block. Through the optical flow algorithm twice, the first mapping feature points with preset conditions are screened out from a large number of first mapping feature points corresponding to the original feature points as the mapping feature points corresponding to the original feature points of the reference image block. Furthermore, the accuracy of the affine transformation matrix subsequently calculated based on the original feature points and the mapping feature points is also improved.

[0097] In one embodiment, for each reference image block in the reference frame, based on the original feature points and the mapped feature points of the reference image block, a set of affine transformation matrices between the reference frame and the frame to be registered is determined, including:

[0098] For each reference image block in the reference frame, based on the original feature points and the mapped feature points of the reference image block, an affine transformation matrix between the reference image block and the image block to be registered in the frame to be registered is determined;

[0099] Based on the affine transformation matrix between the reference image block and the image block to be registered in the frame to be registered, an affine transformation matrix set between the reference frame and the frame to be registered is obtained.

[0100] For each reference image block in the reference frame, feature point detection is performed on the reference image block to obtain the original feature points of the reference image block. Then, the original feature points of the reference image block are mapped to the frame to be registered through the optical flow algorithm to obtain the mapped feature points corresponding to the original feature points of the reference image block. Therefore, for each reference image block, the coordinates of the original feature points of the reference image block are obtained from the reference frame, and the coordinates of the mapped feature points of the reference image block are obtained from the frame to be registered. Based on the coordinates of the original feature points and the coordinates of the mapped feature points, the distance between the coordinates is calculated to obtain the affine transformation matrix of the reference image block between the reference frame and the frame to be registered.

[0101] Finally, assuming that there are m frames to be registered, based on the affine transformation matrices H1, H2, H3...Hm between the reference frame and each frame to be registered, we get the affine transformation matrix set H = (H1, H2, H3...Hm) between the reference frame and each frame to be registered.

[0102] In an embodiment of the present application, first, the reference frame is divided into reference image blocks, and from the granularity of the reference image blocks, the affine transformation matrix between the reference image blocks and the image blocks to be registered in the frame to be registered can be more accurately calculated. Then, based on the affine transformation matrix between each reference image block in the reference frame and the image blocks to be registered in the frame to be registered, a set of affine transformation matrices between the reference frame and each frame to be registered is obtained. So that subsequently, based on the set of affine transformation matrices between the reference frame and each frame to be registered, the reference image blocks in the reference frame can be accurately globally registered with the image blocks to be registered in the frame to be registered, and a global registration result of the reference image blocks can be obtained.

[0103] In one embodiment, performing block matching on a reference image block in a reference frame and an image block to be registered in a frame to be registered to determine a block matching result of the reference image block includes:

[0104] The following operations are performed iteratively starting from k=1: for the reference image block in the k-th layer reference image, block matching is performed between the reference image block and the image block to be registered in the k-th layer image to be registered in combination with preset position information to obtain a block matching result corresponding to the reference image block in the k-th layer reference image; the block matching result includes the block matching image block and the position information of the block matching image block; until k=n.

[0105] The reference frame and the frame to be registered are preprocessed in advance to obtain a first image pyramid of the reference frame and a second image pyramid of the frame to be registered. Specifically, when performing block matching on the reference image block in the reference frame and the image block to be registered in the frame to be registered, the following operations are performed iteratively starting from k=1: for the reference image block in the k-th layer reference image, the reference image block is block matched with the image block to be registered in the k-th layer image to be registered in combination with the preset position information to obtain a block matching result corresponding to the reference image block in the k-th layer reference image; until k=n. The preset position information is the registration result of the reference image block in the previous layer of the reference image of this layer, specifically, the registration result obtained after the reference image block is globally registered and block matched in the previous layer of the reference image of this layer.

[0106] The process of performing block matching between the reference image block and the image block to be registered in the k-th layer of the image to be registered in combination with the preset position information to obtain the block matching result corresponding to the reference image block in the k-th layer of the reference image is specifically as follows: determining the candidate image block to be registered with the same preset position information from the k-th layer of the image to be registered; then calculating the first pixel difference between the reference image block and the image block to be registered in the neighborhood of the candidate image block to be registered; and finally, determining the block matching result of the reference image block from the neighborhood according to the first pixel difference. The block matching result includes the block matching image block on the k-th layer of the image to be registered and the position information of the block matching image block on the k-th layer of the image to be registered.

[0107] In an embodiment of the present application, when block matching is performed between the reference image block and the image block to be registered in the k-th layer of the image to be registered, block matching is performed sequentially from the top layer of the pyramid to the bottom layer of the pyramid, thereby realizing the registration from the low-resolution image to the high-resolution image, that is, realizing the registration process from coarse to fine. And in the process of block matching, the result of the global registration of the previous layer is combined, and the block matching can be corrected by the result of the global registration. Thus, while taking into account the better registration effect of global registration on the static area, the better registration effect of block matching on the moving area can also be taken into account. Therefore, multiple registration methods are used for each area on the image at the same time, not just a single registration method. Therefore, the noise reduction effect of different areas on the image obtained after registration is good, and the difference in noise reduction effect at the boundary of different areas on the image is small, that is, there will be no obvious difference in the noise morphology at the boundary of different areas on the image.

[0108] In one embodiment, if k=1, the preset position information is the position information of the reference image block in the k-th layer reference image, then the reference image block is block matched with the image block to be registered in the k-th layer image to be registered in combination with the preset position information to obtain a block matching result of the reference image block, including:

[0109] Determine, from the k-th layer of images to be registered, candidate image blocks to be registered having the same position information as the reference image blocks in the k-th layer of reference images;

[0110] Calculating a first pixel difference between the reference image block and the image block to be registered in the neighborhood of the candidate image block to be registered;

[0111] According to the first pixel difference, a block matching result of the reference image block is determined from within the neighborhood.

[0112] Combination Figure 8As shown, the k-th layer reference image is divided into reference image blocks according to the size of 8pixel*8pixel, and the k-th layer image to be registered is divided into image blocks to be registered according to the size of 8pixel*8pixel. Of course, the k-th layer reference image and the k-th layer image to be registered can also be divided according to other pixel sizes, which is not limited in this application. Starting from k=1, that is, starting from the first layer of the pyramid, the preset position information at this time is the position information (x1, y1, z1) of the reference image block A1 in the first layer reference image itself in the k-th layer reference image, for example (3, 3, 1). Among them, z=1 indicates that the reference image block A1 is located in the first layer reference image. For the reference image block A1 in the first layer reference image, the reference image block A1 is block matched with the image block to be registered in the first layer image to be registered in combination with the preset position information (x1, y1, z1), and the process of obtaining the block matching result corresponding to the reference image block A1 in the first layer reference image is specifically as follows:

[0113] First, determine the candidate image blocks to be registered from the first layer of images to be registered that have the same position information as the reference image blocks in the first layer of reference images. That is, determine the image block A1' to be registered at the preset position information (x1, y1, z1), for example (3, 3, 1), from the first layer of images to be registered as the candidate image blocks to be registered.

[0114] Secondly, the first pixel difference between the reference image block and the image block to be registered in the neighborhood of the candidate image block to be registered is calculated. Then, the neighborhood of the candidate image block to be registered A1' is first determined. For example, the neighborhood of the candidate image block to be registered A1' can be determined with a preset size with the image block to be registered A1' as the center. Here, the determined area can be a rectangle, of course, it can also be other shapes, which is not limited in this application. Combined with Figure 8 As shown, the determined area may be a rectangle of 5*5 image blocks. Then, the first pixel difference between the reference image block A1 and each image block to be registered in the neighborhood of the candidate image block to be registered A1' is calculated.

[0115] Finally, according to the first pixel difference, the block matching result of the reference image block is determined from the neighborhood. Based on the first pixel difference between the reference image block A1 and each image block to be registered in the neighborhood of the candidate image block to be registered A1`, the image block to be registered A1`` corresponding to the minimum first pixel difference is determined from the field as the block matching image block of the reference image block A1. And the position information of the block matching image block A1`` on the first layer of the image to be registered is recorded.

[0116] In an embodiment of the present application, for the first layer of the pyramid, first, a candidate image block to be registered having the same position information as the reference image block in the first layer of the reference image is determined from the first layer of the image to be registered. Secondly, the first pixel difference between the reference image block and the image block to be registered in the neighborhood of the candidate image block to be registered is calculated. Finally, based on the first pixel difference, the image block to be registered corresponding to the smallest first pixel difference is determined from the field as the block matching image block of the reference image block. After the candidate image block to be registered is obtained by mapping the reference image block to the image to be registered according to the preset position information, the comparison range is expanded by taking the neighborhood, thereby improving the accuracy of the block matching image block determined from the neighborhood.

[0117] In one embodiment, if k>1, the preset position information is the position information of the intermediate registration image block corresponding to the k-1th layer, then the reference image block is block matched with the image block to be registered in the kth layer of the image to be registered in combination with the preset position information to obtain a block matching result of the reference image block, including:

[0118] Determine, from the k-th layer of images to be registered, candidate image blocks to be registered having the same position information as the intermediate registered image blocks corresponding to the k-1th layer;

[0119] Calculating a first pixel difference between the reference image block and the image block to be registered in the neighborhood of the candidate image block to be registered;

[0120] According to the first pixel difference, a block matching result of the reference image block is determined from within the neighborhood.

[0121] Combination Fig. 9 As shown, k>1, that is, starting from k=2, that is, starting from the second layer of the pyramid, the preset position information is the position information of the intermediate registration image block corresponding to the previous layer, that is, the first layer. The process of determining the intermediate registration image block corresponding to the first layer is: obtaining the global registration result of the reference image block in the first layer reference image (global registration image block B1 and its position information (3,3,1)), the block matching result of the reference image block in the first layer reference image (block matching image block A1`` and its position information (2,5,1)); according to the global registration image result and the block matching result, determining the intermediate registration result corresponding to the reference image block in the first layer reference image. The intermediate registration result includes the intermediate registration image block and the position information of the intermediate registration image block. For example, from the global registration result and the block matching result, determine the result that is closest to the image content of the reference image block in the first layer reference image, that is, the result with the highest matching accuracy, as the intermediate registration result corresponding to the reference image block in the first layer reference image. For example, the block matching image block A1`` and its position information (2,5,1) are taken as the intermediate registration result.

[0122] For the second layer of the pyramid, the reference image block A2 in the second layer reference image is block matched with the image block to be registered in the second layer image to be registered in combination with the preset position information to obtain the block matching result of the reference image block, which is specifically:

[0123] First, determine the candidate image blocks to be registered that have the same position information as the intermediate registered image block A1`` corresponding to the first layer from the second layer of images to be registered. The position information of the intermediate registered image block A1`` corresponding to the first layer is (2,5,1), that is, determine the image block A2` to be registered at the preset position information, such as (2,5,1), from the second layer of images to be registered as the candidate image block to be registered.

[0124] Secondly, the first pixel difference between the reference image block A2 and the image block to be registered in the neighborhood of the candidate image block to be registered A2' is calculated. Then, the neighborhood of the candidate image block to be registered A2' is first determined. For example, the neighborhood of the candidate image block to be registered A2' can be determined with a preset size with the image block to be registered A2' as the center. Here, the determined area can be a rectangle, of course, it can also be other graphics, which is not limited in this application. Combined with Fig. 9 As shown, the determined area may be a rectangle of 5*5 image blocks. Then, the first pixel difference between the reference image block A2 and each image block to be registered in the neighborhood of the candidate image block to be registered A2' is calculated.

[0125] Finally, according to the first pixel difference, the block matching result of the reference image block is determined from the neighborhood. Based on the first pixel difference between the reference image block A2 and each image block to be registered in the neighborhood of the candidate image block to be registered A2`, the image block to be registered A2`` corresponding to the minimum first pixel difference is determined from the field as the block matching image block of the reference image block A2. And the position information of the block matching image block A2`` on the second layer of the image to be registered is recorded.

[0126] At the same time, for the second layer of the pyramid, after obtaining the affine transformation matrix set between the reference frame and the frame to be registered, for the reference image block A2 in the reference image of the second layer of the first image pyramid, the target affine transformation matrix between the reference image block A2 and the image block to be registered in the image to be registered in the second layer of the second image pyramid can be found from the affine transformation matrix set.

[0127] Based on the target affine transformation matrix, the second layer of the image to be registered can be transformed to obtain the transformed image to be registered. Then, the reference image block A2 is overlapped with the transformed image to be registered, and the image block B2 to be registered that overlaps with the reference image block A2 in spatial position is determined from the transformed image to be registered. At this time, the image block B2 to be registered is the global registration image block of the reference image block A2, and the position information of the image block B2 to be registered is the position information of the global registration image block of the reference image block A2. The image block B2 to be registered and the position information of the image block B2 to be registered constitute the global registration result of the reference image block.

[0128] Similarly, for the third layer of the pyramid, the reference image block A3 in the third layer reference image is block matched with the image block to be registered in the third layer to be registered image in combination with the preset position information to obtain the block matching result of the reference image block. The above operation is iterated until k=n.

[0129] In the embodiment of the present application, starting from the second layer of the pyramid, the preset position information is the position information of the intermediate registration image block corresponding to the previous layer. The intermediate registration image block corresponding to the previous layer is determined from the global registration result and block matching result of the previous layer. Therefore, the reference image block is block matched with the image block to be registered in the k-th layer of the image to be registered in combination with the preset position information to obtain the block matching result of the reference image block, which combines the result of the global registration of the previous layer, and realizes that the block matching can be corrected by the result of the global registration. Thus, while taking into account the better registration effect of global registration on the static area, the better registration effect of block matching on the moving area can also be taken into account. Therefore, multiple registration methods are used for each area on the image at the same time, not just a single registration method. Therefore, the noise reduction effect of different areas on the image obtained after registration is good, and the difference in noise reduction effect at the boundary of different areas on the image is small, that is, there will be no obvious difference in the noise morphology at the boundary of different areas on the image.

[0130] In one embodiment, Fig.10 As shown, an image registration method is provided, which also includes:

[0131] Step 1020 , obtaining the global registration result of the reference image blocks in the k-1th layer of reference images and the block matching result of the reference image blocks in the k-1th layer of reference images from the k-1th layer of images to be registered.

[0132] Step 1040, determining an intermediate registration result corresponding to the reference image block in the k-1th layer reference image according to the global registration result and the block matching result; the intermediate registration result includes the intermediate registered image block and position information of the intermediate registered image block.

[0133] After calculating the global registration result of the reference image block in the k-1 layer reference image and the block matching result of the reference image block in the k-1 layer reference image for the k-1 layer of the pyramid, an intermediate registration result corresponding to the reference image block in the k-1 layer reference image is determined from the global registration result and the block matching result. Specifically, the pixel difference between the reference image block and the global registration result and the block matching result is calculated respectively, and the result corresponding to the smaller pixel difference is obtained as the intermediate registration result corresponding to the reference image block. Similarly, the intermediate registration result includes the intermediate registered image block and the position information of the intermediate registered image block.

[0134] Step 1060: Use the position information of the intermediate registered image block corresponding to the k-1th layer as the preset position information.

[0135] After calculating the intermediate registration result corresponding to the reference image block in the k-1th layer of reference image, the position information of the intermediate registration image block corresponding to the k-1th layer is used as the preset position information. When k>1, for the kth layer of reference image, the candidate image block to be registered with the same position information as the intermediate registration image block corresponding to the k-1th layer can be determined from the kth layer of image to be registered. Calculate the first pixel difference between the reference image block and the image block to be registered in the neighborhood of the candidate image block to be registered. According to the first pixel difference, determine the block matching result of the reference image block from the neighborhood. Therefore, combined with the position information of the intermediate registration image block corresponding to the k-1th layer, the block matching result of the reference image block in the kth layer of reference image can be calculated.

[0136] In an embodiment of the present application, when calculating the block matching result of the reference image block in the k-th layer reference image, it is necessary to combine the position information of the intermediate registration image block corresponding to the k-1th layer. The position information of the intermediate registration image block corresponding to the k-1th layer can be determined from the global registration result of the reference image block in the k-1th layer reference image and the block matching result of the reference image block in the k-1th layer reference image. Therefore, by performing block matching of the reference image block with the image block to be registered in the k-th layer to be registered image in combination with the preset position information, the block matching result of the reference image block is obtained, which combines the result of the global registration of the previous layer, and realizes the ability to correct the current block matching through the result of the global registration. Thus, while taking into account the better registration effect of global registration on static areas, it is also possible to take into account the better registration effect of block matching on moving areas.

[0137] In one embodiment, determining the target image registration result of the reference frame according to the global registration result and the block matching result includes:

[0138] If k=n, for the reference image block in the n-th layer of reference image, determine the target image registration result of the n-th layer of reference image according to the global registration result of the reference image block in the n-th layer of reference image and the block matching result of the reference image block;

[0139] The target image registration result of the n-th layer reference image is used as the target image registration result of the reference frame.

[0140] Specifically, the following operations are performed iteratively starting from k=1: for the reference image block in the k-th layer reference image, determine the global registration result corresponding to the reference image block in the k-th layer reference image from the k-th layer to be registered image. For the reference image block in the k-th layer reference image, perform block matching on the reference image block and the image block to be registered in the k-th layer to be registered image in combination with the preset position information to obtain a block matching result corresponding to the reference image block in the k-th layer reference image; the block matching result includes the block matching image block and the position information of the block matching image block; until k=n.

[0141] If k=n, for the reference image block in the n-th layer reference image, the target image registration result of the n-th layer reference image is determined according to the global registration result of the reference image block in the n-th layer reference image and the block matching result of the reference image block. Specifically, the target image registration result of the n-th layer reference image is determined from the global registration result of the reference image block in the n-th layer reference image and the block matching result of the reference image block. The target image registration result of the n-th layer reference image is used as the target image registration result of the reference frame.

[0142] In an embodiment of the present application, the reference images of each layer of the first image pyramid of the reference frame are sequentially registered with the images to be registered of each layer of the second image pyramid of the frame to be registered, until the block matching results and global registration results between the bottom layer of the first image pyramid (the original image of the reference frame) and the bottom layer of the second image pyramid (the original image of the frame to be registered) are obtained. Then, the target image registration result of the n-th layer of the reference image is determined from the global registration results of the reference image blocks in the n-th layer of the reference image and the block matching results of the reference image blocks. The target image registration result of the n-th layer of the reference image is used as the target image registration result of the reference frame. The registration from low-resolution images to high-resolution images is realized, that is, the registration process from coarse to fine is realized, thereby improving the accuracy of the registration result.

[0143] In one embodiment, for a reference image block in an n-th layer of reference images, determining a target image registration result of the n-th layer of reference images according to a global registration result of the reference image block in the n-th layer of reference images and a block matching result of the reference image block includes:

[0144] For each reference image block in the n-th layer of reference image, calculate a second pixel difference between the reference image block and a global registration result of the reference image block, and calculate a third pixel difference between the reference image block and a block matching result of the reference image block;

[0145] Determining a target image registration result of the reference image block based on the second pixel difference and the third pixel difference;

[0146] Based on the target image registration results of each reference image block in the n-th layer reference image, a target image registration result of the n-th layer reference image is obtained.

[0147] Specifically, when determining the target image registration result of the n-th layer reference image, for each reference image block in the n-th layer reference image, a second pixel difference between the reference image block and the global registration result of the reference image block is calculated, and a third pixel difference between the reference image block and the block matching result of the reference image block is calculated. The pixel difference refers to the difference between pixels in the image block.

[0148] Determine the size between the second pixel difference and the third pixel difference. If the second pixel difference is less than the third pixel difference, use the global registration result of the reference image block as the target image registration result of the reference image block. If the second pixel difference is greater than the third pixel difference, use the block matching result of the reference image block as the target image registration result of the reference image block. If the second pixel difference is equal to the third pixel difference, use the block matching result of the reference image block or the global registration result corresponding to the reference image block as the target image registration result of the reference image block.

[0149] Based on the target image registration results of each reference image block in the n-th layer reference image, the target image registration result of the entire n-th layer reference image is obtained.

[0150] In the embodiment of the present application, after calculating the global registration result of the reference image block in the n-th layer reference image and the block matching result of the reference image block, the target image registration result of the reference image block is screened out from the pixel difference between the reference image block and the global registration result of the reference image block and the block matching result of the reference image block, and finally the target image registration result of the n-th layer reference image is obtained. From the perspective of pixel difference, the result with better actual registration effect can be screened out from the global registration result and the block matching result as the final target image registration result, thereby improving the accuracy of registration.

[0151] In one embodiment, image preprocessing is performed on the reference frame and the frame to be registered respectively to obtain a first image pyramid of the reference frame and a second image pyramid of the frame to be registered, including:

[0152] Generate a first image pyramid of the reference frame based on a target component in a YUV component of the reference frame; the target component includes a Y component;

[0153] A second image pyramid of the frame to be registered is generated based on the target component in the YUV component of the frame to be registered.

[0154] The reference frame and the frame to be registered here can be YUV images. "Y" represents brightness (Luminance, Luma), and "U" and "V" represent chrominance and concentration (Chrominance, Chroma). Of course, the reference frame and the frame to be registered can be images in formats such as RGB images and RGBW images, which are not limited in this application.

[0155] If the reference frame and the frame to be registered can be images in YUV format, since YUV images better retain the brightness information of the image compared to images in other formats, the first image pyramid of the reference frame and the second image pyramid of the frame to be registered can be constructed based on the reference frame and the frame to be registered in YUV format. The obtained first image pyramid and second image pyramid can also better retain the brightness information of the image.

[0156] Specifically, a first image pyramid of a reference frame can be generated based on a target component in a YUV component of the reference frame. Similarly, a second image pyramid of a frame to be registered can be generated based on a target component in a YUV component of the frame to be registered. The target component includes a Y component, that is, the target component only needs to include a Y component. For example, the target component includes a Y component, or the target component includes a Y component and a U component, or the target component includes a Y component, a U component, and a V component, which is not limited in this application.

[0157] In the embodiment of the present application, if the reference frame and the frame to be registered are YUV images, a first image pyramid of the reference frame can be generated based on the target component in the YUV component of the reference frame. Similarly, a second image pyramid of the frame to be registered can be generated based on the target component in the YUV component of the frame to be registered. The target component includes the Y component, that is, the target component only needs to include the Y component. Thus, in the process of generating the image pyramid, the amount of calculation can be reduced, thereby greatly improving the efficiency of image registration.

[0158] In one embodiment, Fig.11A As shown, an image registration method is provided, which also includes:

[0159] According to the target image registration result of the reference frame, the reference image block in the reference frame is fused with the image block to be registered in the frame to be registered to generate a fused image.

[0160] like Fig.11AAs shown, an image registration method is provided, comprising:

[0161] Step 1102, the electronic device captures the same scene through a camera module to obtain multiple frames of YUV images;

[0162] Step 1104, selecting a reference frame from the multiple YUV image frames by using a frame selection algorithm, and using the images other than the reference frame as the frames to be registered;

[0163] Step 1106, calculating a set of affine transformation matrices between the reference frame and the frame to be registered;

[0164] Step 1108, constructing a first image pyramid of the reference frame based on the Y component of the reference frame;

[0165] Step 1110, constructing a second image pyramid of the frame to be registered based on the Y component of the frame to be registered;

[0166] Step 1112, globally registering the reference image block in the reference frame with the image block to be registered in the frame to be registered, and determining a global registration result of the reference image block;

[0167] Step 1114, performing block matching on the reference image block in the reference frame and the image block to be registered in the frame to be registered, and determining a block matching result of the reference image block;

[0168] Step 1116, determining the target image registration result of the reference frame according to the global registration result and the block matching result;

[0169] Step 1118 , based on the target image registration result of the reference frame, the reference image block in the reference frame is fused with the image block to be registered in the frame to be registered to generate a fused image.

[0170] Specifically, first, each reference image block in the reference frame is globally registered with the image block to be registered in the frame to be registered, and the global registration result of the reference image block is determined. When determining the global registration result of the reference image block, the reference image block in the reference frame can be globally registered with the image block to be registered in the frame to be registered based on the affine transformation matrix between the reference frame and the frame to be registered, and the global registration result of the reference image block is determined.

[0171] Secondly, block matching is performed on the reference image blocks in the reference frame and the image blocks to be registered in the frame to be registered to determine the block matching results of the reference image blocks. When determining the block matching results of the reference image blocks, each reference image block in the reference frame can be block matched with the image blocks to be registered in the frame to be registered to determine the block matching results of the reference image blocks.

[0172] Finally, for each reference image block on the reference frame, the global registration result of each reference image block and the block matching result of each reference image block are obtained on each frame to be registered after the above registration process. For each frame to be registered, based on the global registration result of each reference image block and the block matching result of each reference image block obtained on the frame to be registered, the target image registration result of the reference frame can be determined. Here, the target image registration result includes the target registration image block and the position information of the target registration image block in the frame to be registered. Based on the target image registration result between the reference frame and each frame to be registered, the reference frame is registered with each frame to be registered. For the reference frame and multiple frames to be registered, the reference image block in the reference frame can be fused with the image block to be registered in each frame to be registered according to the target image registration result of the reference frame to generate a fused image.

[0173] like Fig. 11B As shown, it is a schematic diagram of image fusion of a reference frame and a frame to be registered in one embodiment. The left figure is a reference frame, and each reference image block in the reference frame is image registered with each frame to be registered, and a target registered image block of the reference image block on frame to be registered 1 is obtained, a target registered image block of the reference image block on frame to be registered 2 is obtained, a target registered image block of the reference image block on frame to be registered 3 is obtained, a target registered image block of the reference image block on frame to be registered 4 is obtained, a target registered image block of the reference image block on frame to be registered 5 is obtained, and so on. That is, a target registered image block corresponding to the reference image block will be obtained on each frame to be registered.

[0174] Then, when performing image fusion, for each reference image block, the target registration image block corresponding to the reference image block on each frame to be registered is fused with the reference image block to obtain a fusion result corresponding to the reference image block. The above fusion operation is performed on each reference image block in turn to finally obtain the fusion result of the entire reference frame, that is, the fused image.

[0175] After fusion, a high dynamic range image (HDR) can be generated. Compared with ordinary images, HDR images can provide more dynamic range and image details. Alternatively, the noise level of the fused image generated after fusion is greatly reduced, which improves the clarity of the image.

[0176] In the embodiment of the present application, since global registration is used for the static area, a better registration effect can be achieved, and block matching is used for the moving area, a better registration effect can be achieved. Therefore, for the reference image block in the reference frame, global registration and block matching are performed with the image block to be registered in the frame to be registered, respectively, to determine the global registration result and block matching result of the reference image block. Finally, based on the two registration results, the target image registration result of the reference frame is determined. That is, the registration effect of the global registration result and the block matching result can be comprehensively considered to determine the target image registration result of the reference frame. The accuracy of image registration between the reference frame and the frame to be registered is improved, and then the noise reduction effect of the image obtained by fusing the reference frame and the frame to be registered is also improved, and the clarity of the image is improved. At the same time, because the registration effect of the global registration result and the block matching result are comprehensively considered, the noise reduction effect at the boundary of different regions on the fused image is relatively small.

[0177] In a specific embodiment, Fig.12 As shown, an image registration method is provided, the method comprising:

[0178] Step 1202, the electronic device 120 uses a camera module to capture the same scene and obtain multiple frames of YUV images;

[0179] Step 1204, selecting a reference frame from the multiple YUV image frames by using a frame selection algorithm, and using the images other than the reference frame as the frames to be registered;

[0180] Step 1206, traverse all frames to be registered; if yes, proceed to 1222; if no, proceed to 1208;

[0181] Step 1208, constructing a first image pyramid of the reference frame based on the Y component of the reference frame; constructing a second image pyramid of the frame to be registered based on the Y component of the frame to be registered; the first image pyramid includes n layers of reference images, and the second image pyramid includes n layers of images to be registered; the k-th layer of reference images in the first image pyramid corresponds to the k-th layer of images to be registered in the second image pyramid, 1≤k≤n, and k is a positive integer;

[0182] Step 1210, starting from k=1, traverse all image layers of the pyramid in sequence; if yes, proceed to 1206; if no, proceed to 1212;

[0183] Step 1212, dividing the k-th layer reference image in the first image pyramid into reference image blocks of a preset size; dividing the k-th layer to-be-registered image in the second image pyramid into image blocks to be registered that have the same size as the reference image blocks;

[0184] Step 1214, for the reference image blocks in the k-th layer of reference image, determine the global registration result corresponding to the reference image block in the k-th layer of reference image from the k-th layer of images to be registered one by one;

[0185] Step 1216, for the reference image block in the k-th layer reference image, block matching is performed between the reference image block and the image block to be registered in the k-th layer image to be registered in combination with the preset position information, to obtain a block matching result corresponding to the reference image block in the k-th layer reference image;

[0186] Step 1218, selecting a result with a smaller pixel difference from the global registration result of the reference image block in the k-th layer reference image and the block matching result of the reference image block in the k-th layer reference image as an intermediate registration result of the reference image block in the k-th layer reference image;

[0187] Step 1220, using the position information of the intermediate registered image block in the intermediate registration result of the reference image block in the k-th layer reference image as the preset position information for the next iterative calculation, and looping through steps 1214 to 1220 until k=n;

[0188] Step 1222: The intermediate registration result of the reference image block in the k-th layer reference image is used as the final target image registration result of the reference frame.

[0189] like Fig.13 FIG. 1 is a schematic diagram of a multi-frame fusion result obtained after a traditional multi-frame fusion result and an image registration method in the present application in an embodiment. Fig.13 (a1) is the multi-frame fusion result obtained after image registration using only block matching. Fig.13 (b1) is the multi-frame fusion result obtained after image registration using the image registration method of the present application. Using block matching for image registration can achieve a good registration effect for the moving area, but the image registration effect is poor for the static area, resulting in a large noise in the static area in the multi-frame fusion result.

[0190] Fig.13 (a2) is the multi-frame fusion result obtained after only global registration is used for image registration. Fig.13 (b2) is the multi-frame fusion result obtained after image registration using the image registration method of the present application. In (a2), the entire image is registered by global registration. Then, global registration is used for the static area in the image, which can achieve a better registration effect, while the moving area (car) in the image cannot be accurately registered, resulting in a large noise in the moving area in the multi-frame fusion result. Among them, the moving area includes rigid motion area and non-rigid motion, etc., which is not limited in the present application.

[0191] Fig.13 (a3) is a partial enlarged view of a part of the area in (a2). Fig.13 (b3) is a partial enlarged view of a part of the area in Figure (b2).

[0192] from Fig.13 It can be seen from the figure that the multi-frame fusion result obtained after image registration using the image registration method of the present application has a better noise reduction effect than the multi-frame fusion result obtained after image registration using only block matching or feature point matching. Fig.13 In (a3), the noise form (noise size) is not consistent at the boundary between the static area and the moving area (car). That is, the noise reduction effect on both sides of the separation is quite different. Fig.13 In (b3), there is no obvious difference in the noise morphology between the static area and the moving area. Because the image registration method in this application absorbs the advantages of both global registration and block matching for both the static area and the moving area, the static area and the moving area can be accurately registered, and the noise reduction levels of the static area and the moving area in the multi-frame fusion results are similar, and there will be no inconsistency in noise morphology.

[0193] In an embodiment of the present application, when block matching is performed between the reference image block and the image block to be registered in the k-th layer of the image to be registered, block matching is performed sequentially from the top layer of the pyramid to the bottom layer of the pyramid, thereby realizing the registration from the low-resolution image to the high-resolution image, that is, realizing the registration process from coarse to fine. And in the process of block matching, the result of the global registration of the previous layer is combined, so that the block matching can be corrected by the result of the global registration. Thus, while taking into account the better registration effect of the global registration on the static area, the better registration effect of the block matching on the moving area can also be taken into account. Therefore, multiple registration methods are used for each area on the image at the same time, not just a single registration method. Therefore, the noise reduction effect of different areas on the image obtained after registration is good, and the difference in the noise reduction effect at the boundary of different areas on the image is small, that is, there will be no obvious difference in the noise morphology at the boundary of different areas on the image.

[0194] In one embodiment, Fig.14 As shown, an image registration device 1400 is provided, the device comprising:

[0195] A global registration module 1420 is used to perform global registration on the reference image block in the reference frame and the image block to be registered in the frame to be registered, and determine a global registration result of the reference image block;

[0196] The block matching module 1440 is used to perform block matching on the reference image block in the reference frame and the image block to be registered in the frame to be registered, and determine the block matching result of the reference image block;

[0197] The target image registration result determination module 1460 is used to determine the target image registration result of the reference frame according to the global registration result and the block matching result.

[0198] In one embodiment, Fig.15 As shown, an image registration device 1400 is provided, and the device also includes:

[0199] The image pyramid generation module 1480 is used to perform image preprocessing on the reference frame and the frame to be registered, respectively, to obtain a first image pyramid of the reference frame and a second image pyramid of the frame to be registered; the first image pyramid includes n layers of reference images, and the second image pyramid includes n layers of images to be registered; wherein the n-th layer of reference images is the original image of the reference frame, and the n-th layer of images to be registered is the original image of the frame to be registered; the k-th layer of reference images in the first image pyramid corresponds to the k-th layer of images to be registered in the second image pyramid, 1≤k≤n, and k is a positive integer;

[0200] The reference image block and image block to be registered acquisition module 1490 is used to acquire the reference image block from the k-th layer reference image in the first image pyramid, and acquire the image block to be registered from the k-th layer image to be registered in the second image pyramid.

[0201] In one embodiment, the global registration module 1420 is further configured to iteratively perform the following operations starting from k=1: for a reference image block in a k-th layer of reference images, determine a global registration result corresponding to the reference image block in the k-th layer of reference images from the k-th layer of images to be registered; until k=n;

[0202] The global registration result includes the globally registered image block and the position information of the globally registered image block.

[0203] In one embodiment, the global registration module 1420 includes:

[0204] An affine transformation matrix set calculation unit, used to calculate an affine transformation matrix set between a reference frame and a frame to be registered; the affine transformation matrix set includes an affine transformation matrix between a reference image block in the reference frame and an image block to be registered in the frame to be registered;

[0205] The global registration result determination unit is used to determine the global registration result corresponding to the reference image block in the k-th layer reference image from the k-th layer of images to be registered based on the affine transformation matrix set between the reference frame and the frame to be registered.

[0206] In one embodiment, an affine transformation matrix set calculation unit is used to divide a reference frame into reference image blocks, and divide a frame to be registered into image blocks to be registered of the same size as the reference image blocks; for each reference image block, perform feature point detection on the reference image block to obtain the original feature points of the reference image block; map the original feature points of the reference image block between the reference frame and the frame to be registered by an optical flow algorithm to obtain mapping feature points corresponding to the original feature points of the reference image block; for each reference image block in the reference frame, determine the affine transformation matrix set between the reference frame and the frame to be registered based on the original feature points and the mapping feature points of the reference image block.

[0207] In one embodiment, an affine transformation matrix set calculation unit is used to map the original feature points of a reference image block to a frame to be matched by a first optical flow algorithm to obtain first mapping feature points corresponding to the original feature points of the reference image block; map the first mapping feature points to a reference frame by a second optical flow algorithm to generate second mapping feature points corresponding to the original feature points of the reference image block; calculate the distance between the second mapping feature points corresponding to the original feature points of the reference image block and the original feature points of the reference image block; and determine, based on the distance, the mapping feature points corresponding to the original feature points of the reference image block from the first mapping feature points.

[0208] In one embodiment, the affine transformation matrix set calculation unit is used to determine, for each reference image block in the reference frame, an affine transformation matrix between the reference image block and the image block to be registered in the frame to be registered based on the original feature points and the mapped feature points of the reference image block; and obtain an affine transformation matrix set between the reference frame and the frame to be registered based on the affine transformation matrix between the reference image block and the image block to be registered in the frame to be registered.

[0209] In one embodiment, the block matching module 1440 is used to iteratively perform the following operations starting from k=1: for the reference image block in the k-th layer reference image, block matching is performed between the reference image block and the image block to be registered in the k-th layer image to be registered in combination with preset position information to obtain a block matching result corresponding to the reference image block in the k-th layer reference image; wherein the block matching result includes the block matching image block and the position information of the block matching image block; until k=n.

[0210] In one embodiment, if k=1, the preset position information is the position information of the reference image block in the k-th layer reference image, then the block matching module 1440 includes:

[0211] A candidate to-be-registered image block determining unit, used to determine, from the k-th layer of images to be registered, candidate to-be-registered image blocks having the same position information as the reference image blocks in the k-th layer of reference images;

[0212] A first pixel difference calculation unit, used to calculate a first pixel difference between the reference image block and the image block to be registered in the neighborhood of the candidate image block to be registered;

[0213] The block matching result determining unit is used to determine the block matching result of the reference image block in the neighborhood according to the first pixel difference.

[0214] In one embodiment, if k>1, the preset position information is the position information of the intermediate registration image block corresponding to the k-1th layer, then the block matching module 1440 includes:

[0215] A candidate to-be-registered image block determining unit, used to determine, from the k-th layer of images to be registered, candidate to-be-registered image blocks having the same position information as the intermediate registered image blocks corresponding to the k-1th layer;

[0216] A first pixel difference calculation unit, used to calculate a first pixel difference between the reference image block and the image block to be registered in the neighborhood of the candidate image block to be registered;

[0217] The block matching result determining unit is used to determine the block matching result of the reference image block in the neighborhood according to the first pixel difference.

[0218] In one embodiment, an image registration device is provided, further comprising:

[0219] A preset position information determination module is used to obtain the global registration result of the reference image block in the k-1th layer reference image and the block matching result of the reference image block in the k-1th layer reference image from the k-1th layer image to be registered; determine the intermediate registration result corresponding to the reference image block in the k-1th layer reference image according to the global registration result and the block matching result; the intermediate registration result includes the intermediate registration image block and the position information of the intermediate registration image block; and use the position information of the intermediate registration image block corresponding to the k-1th layer as the preset position information.

[0220] In one embodiment, the target image registration result determination module 1460 is used to determine the target image registration result of the n-th layer reference image according to the global registration result of the reference image block in the n-th layer reference image and the block matching result of the reference image block if k=n; and use the target image registration result of the n-th layer reference image as the target image registration result of the reference frame.

[0221] In one embodiment, the target image registration result determination module 1460 is further used to calculate, for each reference image block in the n-th layer reference image, a second pixel difference between the reference image block and the global registration result of the reference image block, and calculate a third pixel difference between the reference image block and the block matching result of the reference image block; determine the target image registration result of the reference image block based on the second pixel difference and the third pixel difference; and obtain the target image registration result of the n-th layer reference image based on the target image registration result of each reference image block in the n-th layer reference image.

[0222] In one embodiment, the target image registration result determination module 1460 is further configured to, if the second pixel difference is less than the third pixel difference, use the global registration result of the reference image block as the target image registration result of the reference image block; if the second pixel difference is greater than the third pixel difference, use the block matching result of the reference image block as the target image registration result of the reference image block; if the second pixel difference is equal to the third pixel difference, use the block matching result of the reference image block or the global registration result corresponding to the reference image block as the target image registration result of the reference image block.

[0223] In one embodiment, the reference frame and the frame to be registered are YUV images.

[0224] In one embodiment, the image pyramid generation module 1480 is further used to generate a first image pyramid of the reference frame based on the target component in the YUV component of the reference frame; the target component includes the Y component; and to generate a second image pyramid of the frame to be registered based on the target component in the YUV component of the frame to be registered.

[0225] In one embodiment, an image registration device is provided, further comprising:

[0226] The image fusion module is used to fuse the reference image block in the reference frame with the image block to be registered in the to-be-registered frame according to the target image registration result of the reference frame to generate a fused image.

[0227] It should be understood that, although the various steps in the above-mentioned flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the above-mentioned flow chart may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0228] The division of the modules in the above-mentioned image registration device is only for illustration. In other embodiments, the image registration device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned image registration device.

[0229] For the specific definition of the image registration device, please refer to the definition of the image registration method above, which will not be repeated here. Each module in the above-mentioned image registration device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0230] Fig.16 Schematic diagram of the internal structure of an electronic device in an embodiment. The electronic device can be any terminal device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a PDA (Personal Digital Assistant), a POS (Point of Sales), a car computer, a wearable device, etc. The electronic device includes a processor and a memory connected via a system bus. Among them, the processor may include one or more processing units. The processor may be a CPU (Central Processing Unit) or a DSP (Digital Signal Processing), etc. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program.

[0231] The computer program can be executed by a processor to implement an image registration method provided in the following embodiments. The internal memory provides a high-speed cache operating environment for the operating system computer program in the non-volatile storage medium.

[0232] The implementation of each module in the image registration device provided in the embodiment of the present application can be in the form of a computer program. The computer program can be run on an electronic device. The program module constituted by the computer program can be stored in the memory of the electronic device. When the computer program is executed by the processor, the steps of the method described in the embodiment of the present application are implemented.

[0233] The embodiment of the present application also provides a computer-readable storage medium, which is one or more non-volatile computer-readable storage media containing computer-executable instructions, and when the computer-executable instructions are executed by one or more processors, the processors execute the steps of the image registration method.

[0234] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the image registration method.

[0235] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include ROM (Read-Only Memory), PROM (Programmable Read-only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-only Memory) or flash memory. Volatile memory may include RAM (Random Access Memory), which is used as an external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), Double Data Rate DDRSDRAM (Double Data Rate Synchronous Dynamic Random Access memory), ESDRAM (Enhanced Synchronous Dynamic Random Access memory), SLDRAM (Sync Link Dynamic Random Access Memory), RDRAM (Rambus Dynamic Random Access Memory), and DRDRAM (Direct Rambus Dynamic Random Access Memory).

[0236] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image registration method, characterized in that: The method comprises: Performing global registration on a reference image block in a reference frame and an image block to be registered in a frame to be registered, and determining a global registration result of the reference image block, wherein the global registration is based on an affine transformation matrix between the reference frame and the frame to be registered, and determining the global registration result of the reference image block; Performing block matching on the reference image block in the reference frame and the image block to be registered in the frame to be registered, and determining the block matching result of the reference image block, wherein the block matching is used to calculate the similarity between each of the reference image blocks in the reference frame and the image block to be registered in the frame to be registered, and determining the block matching result of the reference image block according to each of the similarities; A target image registration result of the reference frame is determined according to the global registration result and the block matching result.

2. The method according to claim 1, characterized in that The method further comprises: Performing image preprocessing on the reference frame and the frame to be registered respectively to obtain a first image pyramid of the reference frame and a second image pyramid of the frame to be registered; the first image pyramid includes n layers of reference images, and the second image pyramid includes n layers of images to be registered; wherein the n-th layer of reference image is the original image of the reference frame, and the n-th layer of image to be registered is the original image of the frame to be registered; the k-th layer of reference image in the first image pyramid corresponds to the k-th layer of image to be registered in the second image pyramid, 1≤k≤n, and k is a positive integer; The reference image block is obtained from a k-th layer reference image in the first image pyramid, and the image block to be registered is obtained from a k-th layer image to be registered in the second image pyramid.

3. The method according to claim 2, characterized in that The globally registering the reference image block in the reference frame with the image block to be registered in the frame to be registered, and determining the global registration result of the reference image block, comprises: Starting from k=1, iteratively perform the following operations: for the reference image block in the k-th layer reference image, determine the global registration result corresponding to the reference image block in the k-th layer reference image from the k-th layer image to be registered; until k=n; The global registration result includes the globally registered image block and the position information of the globally registered image block.

4. The method according to claim 3, characterized in that: The step of determining, from the k-th layer of images to be registered, a global registration result corresponding to a reference image block in the k-th layer of reference images comprises: Calculating an affine transformation matrix set between the reference frame and the frame to be registered; the affine transformation matrix set includes an affine transformation matrix between a reference image block in the reference frame and an image block to be registered in the frame to be registered; Based on a set of affine transformation matrices between the reference frame and the frame to be registered, a global registration result corresponding to a reference image block in the k-th layer of reference image is determined from the k-th layer of images to be registered.

5. The method according to claim 4, characterized in that The step of calculating a set of affine transformation matrices between the reference frame and the frame to be registered comprises: Dividing the reference frame into reference image blocks, and dividing the frame to be registered into image blocks to be registered that have the same size as the reference image blocks; For each of the reference image blocks, performing feature point detection on the reference image block to obtain original feature points of the reference image block; Mapping original feature points of the reference image block between the reference frame and the frame to be registered by an optical flow algorithm to obtain mapping feature points corresponding to the original feature points of the reference image block; For each reference image block in the reference frame, a set of affine transformation matrices between the reference frame and the frame to be registered is determined based on the original feature points and the mapped feature points of the reference image block.

6. The method according to claim 5, characterized in that The mapping of the original feature points of the reference image block between the reference frame and the frame to be registered by using an optical flow algorithm to obtain mapping feature points corresponding to the original feature points of the reference image block includes: Mapping the original feature points of the reference image block to the frame to be registered by a first optical flow algorithm to obtain first mapped feature points corresponding to the original feature points of the reference image block; Mapping the first mapping feature points to the reference frame by a second optical flow algorithm to generate second mapping feature points corresponding to the original feature points of the reference image block; Calculating the distance between the second mapping feature point corresponding to the original feature point of the reference image block and the original feature point of the reference image block; According to the distance, a mapping feature point corresponding to the original feature point of the reference image block is determined from the first mapping feature points.

7. The method according to claim 5 or 6, characterized in that: The step of determining, for each reference image block in the reference frame, a set of affine transformation matrices between the reference frame and the frame to be registered based on the original feature points and the mapped feature points of the reference image block, comprises: For each reference image block in the reference frame, based on the original feature points and the mapped feature points of the reference image block, determine an affine transformation matrix between the reference image block and the image block to be registered in the frame to be registered; Based on the affine transformation matrix between the reference image block and the image block to be registered in the frame to be registered, a set of affine transformation matrices between the reference frame and the frame to be registered is obtained.

8. The method according to claim 3, characterized in that The step of performing block matching on the reference image block in the reference frame and the image block to be registered in the frame to be registered, and determining the block matching result of the reference image block comprises: The following operations are performed iteratively starting from k=1: for the reference image block in the k-th layer reference image, block matching is performed on the reference image block with the image block to be registered in the k-th layer image to be registered in combination with preset position information to obtain a block matching result corresponding to the reference image block in the k-th layer reference image; the block matching result includes the block matching image block and the position information of the block matching image block; until k=n.

9. The method according to claim 8, characterized in that If k=1, the preset position information is the position information of the reference image block in the k-th layer reference image, then the reference image block is block matched with the image block to be registered in the k-th layer image to be registered in combination with the preset position information to obtain the block matching result of the reference image block, including: Determine, from the k-th layer of images to be registered, candidate image blocks to be registered having the same position information as the reference image blocks in the k-th layer of reference images; Calculating a first pixel difference between the reference image block and an image block to be registered in a neighborhood of the candidate image block to be registered; A block matching result of the reference image block is determined from within the neighborhood according to the first pixel difference.

10. The method according to claim 8, characterized in that If k>1, the preset position information is the position information of the intermediate registered image block corresponding to the k-1th layer, then the reference image block is block matched with the image block to be registered in the kth layer of the image to be registered in combination with the preset position information to obtain the block matching result of the reference image block, including: Determine, from the k-th layer of images to be registered, candidate image blocks to be registered having the same position information as the intermediate registered image blocks corresponding to the k-1-th layer; Calculating a first pixel difference between the reference image block and an image block to be registered in a neighborhood of the candidate image block to be registered; A block matching result of the reference image block is determined from within the neighborhood according to the first pixel difference.

11. The method according to claim 10, characterized in that The method further comprises: Acquire, from the k-1th layer of images to be registered, a global registration result of the reference image blocks in the k-1th layer of reference images and a block matching result of the reference image blocks in the k-1th layer of reference images; Determine, according to the global registration result and the block matching result, an intermediate registration result corresponding to the reference image block in the k-1th layer reference image; the intermediate registration result includes the intermediate registered image block and position information of the intermediate registered image block; The position information of the intermediate registered image block corresponding to the k-1th layer is used as the preset position information.

12. The method according to claim 8, characterized in that Determining the target image registration result of the reference frame according to the global registration result and the block matching result includes: If k=n, for a reference image block in an n-th layer of reference image, determining a target image registration result of the n-th layer of reference image according to a global registration result of the reference image block in the n-th layer of reference image and a block matching result of the reference image block; The target image registration result of the n-th layer reference image is used as the target image registration result of the reference frame.

13. The method according to claim 12, characterized in that The step of determining the target image registration result of the n-layer reference image according to the global registration result of the reference image block in the n-layer reference image and the block matching result of the reference image block comprises: For each reference image block in the n-th layer reference image, calculating a second pixel difference between the reference image block and a global registration result of the reference image block, and calculating a third pixel difference between the reference image block and a block matching result of the reference image block; Determining a target image registration result of the reference image block based on the second pixel difference and the third pixel difference; Based on the target image registration results of the reference image blocks in the n-th layer reference image, the target image registration result of the n-th layer reference image is obtained.

14. The method according to claim 13, characterized in that The determining, based on the second pixel difference and the third pixel difference, a target image registration result of the reference image block includes: If the second pixel difference is less than the third pixel difference, taking the global registration result of the reference image block as the target image registration result of the reference image block; If the second pixel difference is greater than the third pixel difference, using the block matching result of the reference image block as the target image registration result of the reference image block; If the second pixel difference is equal to the third pixel difference, the block matching result of the reference image block or the global registration result corresponding to the reference image block is used as the target image registration result of the reference image block.

15. The method according to claim 2, characterized in that The reference frame and the frame to be registered are YUV images.

16. The method according to claim 15, characterized in that The performing image preprocessing on the reference frame and the frame to be registered respectively to obtain a first image pyramid of the reference frame and a second image pyramid of the frame to be registered includes: Generate a first image pyramid of the reference frame based on a target component in a YUV component of the reference frame; the target component includes a Y component; A second image pyramid of the frame to be registered is generated based on the target component in the YUV component of the frame to be registered.

17. The method according to claim 1, characterized in that The method further comprises: According to the target image registration result of the reference frame, the reference image block in the reference frame is fused with the image block to be registered in the frame to be registered to generate a fused image.

18. An image registration device, characterized in that: The device comprises: A global registration module, used to globally register the reference image block in the reference frame with the image block to be registered in the frame to be registered, and determine the global registration result of the reference image block, wherein the global registration is based on an affine transformation matrix between the reference frame and the frame to be registered, and determines the global registration result of the reference image block; a block matching module, configured to perform block matching on the reference image block in the reference frame and the image block to be registered in the frame to be registered, and determine the block matching result of the reference image block, wherein the block matching is used to calculate the similarity between each of the reference image blocks in the reference frame and the image block to be registered in the frame to be registered, and determine the block matching result of the reference image block according to each of the similarities; The target image registration result determination module is used to determine the target image registration result of the reference frame according to the global registration result and the block matching result.

19. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image registration method according to any one of claims 1 to 17.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 17 are implemented.

Images