Image alignment method, device, computer equipment and storage medium

Through the noise reduction processing and the use of optical flow information, the problem of poor alignment effect of image alignment method on noisy images is solved. By adding noise back-aligning images, the accuracy and quality of image alignment are improved.

CN114764751BActive Publication Date: 2024-12-06GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202110043709.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-13
Publication Date
2024-12-06
Estimated Expiration
2041-01-13

AI Technical Summary

Technical Problem

The existing image alignment methods have poor alignment effects on noisy images, especially when the non-rigid body motion and optical flow estimation errors are large, making it difficult to achieve good alignment effects.

Method used

By acquiring the first noise-reducing image of the original image and the second noise-reducing image of the reference frame image, a difference processing is performed to obtain the image noise, and then aligning the first noise-reducing image to the second noise-reducing image to obtain the initial alignment image, And add noise back to the initial alignment image according to the noise position information to improve the alignment effect.

Benefits of technology

Through noise reduction processing and the use of optical flow information, the accuracy and effect of image alignment are improved, the continuity and consistency of noise in non-moving areas are ensured, and the quality of image alignment is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114764751B_ABST
    Figure CN114764751B_ABST
Patent Text Reader

Abstract

The present application relates to an image alignment method, device, computer equipment and storage medium. The method comprises: obtaining a first denoised image corresponding to an original image and a second denoised image corresponding to a reference frame image of the original image; performing a difference process between the original image and the first denoised image to obtain the image noise of the original image; aligning the first denoised image with the second denoised image to obtain an initial aligned image corresponding to the original image; adding the image noise of the original image to the corresponding position in the initial aligned image according to the position information of the image noise in the original image to obtain a target aligned image corresponding to the original image. The present method can ensure the continuity and consistency of the noise in the non-moving area and improve the alignment effect of the image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image alignment method, apparatus, computer equipment and storage medium. Background Art

[0002] Image alignment is a basic processing technology in the field of image processing. After shooting multiple frames of images with a single camera, you can select any image from the multiple frames as a reference frame image, and use each frame image except the reference frame image as the original image. The technology of aligning each original image frame to the reference frame image is called multi-frame image alignment technology. Currently, most image alignment methods are based on affine transformation matrix or optical flow algorithm for alignment.

[0003] However, current alignment methods based on affine transformation matrices or optical flow algorithms rely on noise. If the image contains non-rigid motion or optical flow estimation produces errors, the alignment effect will be greatly affected. Therefore, traditional image alignment methods have poor alignment effects on noisy images. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, apparatus, computer device and storage medium, which can improve the image alignment effect on noisy images.

[0005] An image alignment method, the method comprising:

[0006] Acquire a first denoised image corresponding to an original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image of the multiple frames of original images except the reference frame image;

[0007] Performing a difference process between the original image and the first denoised image to obtain image noise of the original image;

[0008] Aligning the first denoised image with the second denoised image to obtain an initial aligned image corresponding to the original image;

[0009] According to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the initial aligned image to obtain a target aligned image corresponding to the original image.

[0010] In one embodiment, aligning the first denoised image with the second denoised image to obtain an initial aligned image corresponding to the image includes:

[0011] Inputting the first denoised image and the second denoised image into a pre-trained optical flow learning network to obtain optical flow information from the first denoised image to the second denoised image;

[0012] The first denoised image is aligned with the second denoised image according to the offset direction and offset in the optical flow information to obtain an initial aligned image corresponding to the original image.

[0013] In one of the embodiments, after aligning the first denoised image with the second denoised image to obtain an initial aligned image corresponding to the original image, the method further includes:

[0014] Correcting the initial aligned image to obtain a corrected aligned image;

[0015] The adding the image noise of the original image to the corresponding position in the initial aligned image according to the position information of the image noise in the original image to obtain the target aligned image corresponding to the original image includes:

[0016] According to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the corrected aligned image to obtain a target aligned image corresponding to the original image.

[0017] In one embodiment, the correcting the initial alignment image to obtain a corrected alignment image includes:

[0018] Inputting the initial aligned image, the second denoised image, and the optical flow information from the first denoised image to the second denoised image into a pre-trained correction model to obtain a first score map of the initial aligned image and a second score map of the second denoised image; the first score map is used to represent a first weight of each pixel in the initial aligned image, and the second score map is used to represent a second weight of each pixel in the second denoised image;

[0019] The initial aligned image and the second denoised image are fused according to the first score map and the second score map to obtain the corrected aligned image.

[0020] In one embodiment, the fusing the initial aligned image and the second denoised image according to the first score map and the second score map to obtain the corrected aligned image includes:

[0021] Acquire a first weight of each pixel in the initial aligned image from the first score map, and acquire a second weight of the corresponding pixel in the second denoised image from the second score map;

[0022] According to the first weight of each pixel in the initial aligned image and the second weight of the corresponding pixel in the second denoised image, the pixel values ​​of the pixel in the initial aligned image and the pixel values ​​of the corresponding pixel in the second denoised image are weighted to obtain the corrected aligned image.

[0023] In one embodiment, the adding the image noise of the original image to the corresponding position in the initial aligned image according to the position information of the image noise in the original image to obtain the target aligned image corresponding to the original image includes:

[0024] According to the position information of the image noise in the original image, obtaining the pixel value of the corresponding position in the initial aligned image;

[0025] The pixel value of the image noise position in the original image is added to the pixel value of the corresponding position to obtain a new pixel value, and the pixel value of the corresponding position of the initial aligned image is replaced by the new pixel value to obtain a target aligned image corresponding to the original image.

[0026] In one embodiment, the step of acquiring a first denoised image corresponding to an original image and a second denoised image corresponding to a reference frame image of the original image includes:

[0027] The original image and the reference frame image are respectively input into a pre-trained denoising model to obtain a first denoised image corresponding to the original image and a second denoised image corresponding to the reference frame image.

[0028] An image alignment device, comprising:

[0029] A denoised image acquisition module, used to acquire a first denoised image corresponding to an original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image of the multiple frames of original images except the reference frame image;

[0030] An image noise acquisition module, used for performing a difference process between the original image and the first noise reduction image to obtain the image noise of the original image;

[0031] An initial alignment image acquisition module, used to align the first denoised image with the second denoised image to obtain an initial alignment image corresponding to the original image;

[0032] The target alignment image acquisition module is used to add the image noise of the original image to the corresponding position in the initial alignment image according to the position information of the image noise in the original image, so as to obtain the target alignment image corresponding to the original image.

[0033] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0034] Acquire a first denoised image corresponding to an original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image of the multiple frames of original images except the reference frame image;

[0035] Performing a difference process between the original image and the first denoised image to obtain image noise of the original image;

[0036] Aligning the first denoised image with the second denoised image to obtain an initial aligned image corresponding to the original image;

[0037] According to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the initial aligned image to obtain a target aligned image corresponding to the original image.

[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0039] Acquire a first denoised image corresponding to an original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image of the multiple frames of original images except the reference frame image;

[0040] Performing a difference process between the original image and the first denoised image to obtain image noise of the original image;

[0041] Aligning the first denoised image with the second denoised image to obtain an initial aligned image corresponding to the original image;

[0042] According to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the initial aligned image to obtain a target aligned image corresponding to the original image.

[0043] The above-mentioned image alignment method, device, computer equipment and storage medium obtain a first denoised image corresponding to the original image and a second denoised image corresponding to the reference frame image of the original image, wherein the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image in the multiple frames of original images except the reference frame image, so as to perform a difference process between the original image and the first denoised image to obtain the image noise of the original image; then align the first denoised image with the second denoised image to obtain the initial aligned image corresponding to the first denoised image; finally, according to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the initial aligned image to obtain the target aligned image corresponding to the original image. The method performs denoising on the original image and the reference frame, so as to improve the accuracy of the estimated optical flow in the process of aligning the first denoised image after denoising with the second denoised image, thereby improving the alignment effect of the image. By first removing the noise from the original image and performing image alignment to obtain the initial aligned image, the image noise is pasted back to the initial aligned image to compensate for the noise in the moving and non-moving areas of the initial aligned image, thereby ensuring the continuity and consistency of the noise in the non-moving area to the greatest extent, and further improving the image alignment effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a schematic diagram of a flow chart of an image alignment method in one embodiment;

[0045] Figure 2 A schematic diagram of a process of image noise reduction in one embodiment;

[0046] Figure 3 is a schematic diagram of a process of optical flow estimation in one embodiment;

[0047] Figure 4 is a schematic diagram of a process of correcting an initial aligned image in one embodiment;

[0048] Figure 5 is a schematic flow chart of an image alignment method in another embodiment;

[0049] Figure 6 is a structural block diagram of an image alignment device in one embodiment;

[0050] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] In one embodiment, Figure 1 As shown, an image alignment method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0053] Step S102, obtaining a first denoised image corresponding to the original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image in the multiple frames of original images except the reference frame image.

[0054] The first denoised image refers to an image obtained after denoising the original image.

[0055] The second denoised image refers to an image obtained after denoising the reference frame image.

[0056] In a specific implementation, after obtaining multiple original images under different exposure parameters, any one of the original images can be used as a reference frame image, and the remaining original images can be aligned to the reference frame image to achieve multi-frame image alignment. Taking the alignment of one of the original images to the reference frame image as an example, the original image and the reference frame image are subjected to denoising respectively to obtain a first denoised image corresponding to the original image and a second denoised image corresponding to the reference frame image.

[0057] Furthermore, step S102 includes: inputting the original image and the reference frame image into a pre-trained denoising model respectively to obtain a first denoised image corresponding to the original image and a second denoised image corresponding to the reference frame image.

[0058] Among them, the pre-trained denoising model can be a model based on AINDNet (Adaptive InstanceNormalization Denoiser). The AINDNet algorithm is mainly used to process real-world noise. The original denoising network is first trained with synthetic noise, and then the denoising network is retrained using transfer learning, that is, the real noise data set is used to update the original trained model. The model has strong generalization ability, and the use of adaptive instance norms to build a denoiser can prevent network overfitting. It can be understood that this embodiment is only illustrated by a denoising model based on AINDNet, and is not used to limit the denoising model. In actual applications, other denoising models can also be used.

[0059] refer to Figure 2, is a schematic diagram of the process of image denoising in an embodiment, wherein the original image and the reference frame image are respectively input into a pre-trained AINDNet model to obtain a first denoised image corresponding to the original image and a second denoised image corresponding to the reference frame image.

[0060] Step S104: performing a difference process between the original image and the first noise reduction image to obtain the image noise of the original image.

[0061] Among them, image noise refers to the interference of random signals on the image during the acquisition or transmission process. Some random, discrete, and isolated pixels appear on the image. Image noise will interfere with the human eye's analysis of image information.

[0062] In the specific implementation, refer to Figure 2 After obtaining the original image and the first denoised image obtained by performing denoising processing on the original image, the image noise corresponding to the original image can be obtained by subtracting the original image from the first denoised image.

[0063] Step S106: align the first denoised image with the second denoised image to obtain an initial aligned image corresponding to the original image.

[0064] In a specific implementation, the alignment of the images can first obtain the position offset between the two images through an optical flow learning network, and align the two images according to the position offset. Specifically, the first denoised image and the second denoised image can be input into a pre-trained optical flow learning network to obtain the optical flow of the first denoised image to the second denoised image, that is, the offset of each pixel point, and a deformation function for moving the first denoised image to the second denoised image is constructed according to the offset. The first denoised image is further aligned to the second denoised image according to the deformation function to obtain the initial alignment image corresponding to the first denoised image, that is, the initial alignment image corresponding to the original image.

[0065] In this step, since real noise (such as current noise) may make it difficult to judge the dense optical flow (pixel by pixel) and it is difficult to produce an optical flow data set, it is more difficult to fit the optical flow estimation under noise with no obvious regularity. Therefore, in this embodiment, when calculating the optical flow during image alignment, the first denoised image and the second denoised image after denoising are input into the optical flow learning network. Calculating the optical flow through the denoised images can significantly improve the accuracy of the optical flow calculation.

[0066] Step S108 , according to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the initial aligned image to obtain a target aligned image corresponding to the original image.

[0067] In this step, in order to obtain a good multi-frame alignment effect, it is found through multiple experiments that optical flow is more sensitive to noise. Therefore, in this step, the image noise removed previously is pasted back to the obtained initial alignment image, and the alignment image after adding noise is used as the target alignment image corresponding to the original image. Specifically, the pixel value of the corresponding position in the initial alignment image can be obtained according to the position information of the image noise in the original image, the image noise at each position in the original image is added to the pixel value of the corresponding position in the initial alignment image to obtain a new pixel value, and the pixel value of the corresponding position in the initial alignment image is replaced with the new pixel value to obtain an alignment image with a new pixel value as the target alignment image corresponding to the original image.

[0068] In the above-mentioned image alignment method, a first denoised image corresponding to the original image and a second denoised image corresponding to the reference frame image of the original image are obtained, wherein the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image in the multiple frames of original images except the reference frame image, so as to perform a difference process between the original image and the first denoised image to obtain the image noise of the original image; then the first denoised image is aligned with the second denoised image to obtain the initial aligned image corresponding to the first denoised image; finally, according to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the initial aligned image to obtain the target aligned image corresponding to the original image. The method performs denoising on the original image and the reference frame, so as to improve the accuracy of the estimated optical flow in the process of aligning the first denoised image after denoising with the second denoised image, thereby improving the alignment effect of the images. By first removing the noise from the original image and performing image alignment to obtain the initial aligned image, the image noise is pasted back to the initial aligned image to compensate for the noise in the moving and non-moving areas of the initial aligned image, thereby ensuring the continuity and consistency of the noise in the non-moving area to the greatest extent, and further improving the image alignment effect.

[0069] It can be understood that the traditional method of extracting feature points by relying on different feature extraction operators, estimating the affine transformation matrix based on the feature points, and aligning based on the affine transformation matrix is ​​more dependent on noise. When the image contains non-rigid motion, it is difficult to achieve image alignment, and feature point extraction is more dependent on experience. When the feature point extraction is wrong, it greatly affects the estimation of the affine transformation matrix, resulting in poor alignment effect. The method based on optical flow estimation also relies on noise. When the optical flow estimation result produces an error, it is easy to cause ghosting. Therefore, the image alignment method provided in the above embodiment can solve the problem of traditional image alignment based on affine transformation matrix when containing non-rigid motion and the problem of ghosting caused by the method based on optical flow estimation.

[0070] In one embodiment, the above step S106 specifically includes: inputting the first denoised image and the second denoised image into a pre-trained optical flow learning network to obtain optical flow information from the first denoised image to the second denoised image; aligning the first denoised image to the second denoised image according to the offset direction and offset in the optical flow information to obtain an initial aligned image corresponding to the original image.

[0071] Among them, optical flow represents the movement of an object in an image between two consecutive frames due to camera movement or object movement. The optical flow is a 2D vector field, which represents the movement of a pixel from the first frame to the second frame, and is a dual-channel array.

[0072] In the specific implementation, refer to Figure 3 , is a flowchart of optical flow estimation in one embodiment, the first denoised image and the second denoised image are input into a pre-trained optical flow learning network based on PWC-Net (Pyramid, Warping, and Cost Volume, an algorithm including image pyramid, deformation and cost capacity function), and the optical flow information from the first denoised image to the second denoised image is obtained, wherein the number of channels of the optical flow is 2, and the offset direction u and offset v of each pixel point can be obtained by obtaining the optical flow, which are recorded as (u, v). Furthermore, according to the offset of each pixel point, a deformation function (warp function) for moving the pixel point in the first denoised image to the second denoised image can be obtained, and according to the deformation function, each pixel point in the first denoised image is moved to the corresponding position in the second denoised image, so that the first denoised image is aligned with the second denoised image, and the initial aligned image corresponding to the first denoised image is obtained, that is, the initial aligned image corresponding to the original image is obtained.

[0073] Among them, the optical flow learning network based on PWC-Net constructs feature pyramids for two input images respectively, and iterates from high-level features to the bottom layer through layer-by-layer feature alignment, and finally integrates a background network (Context) for fine-tuning the alignment effect to obtain the final optical flow output. The core idea is to introduce deformable convolution and occlusion-aware network to solve the occlusion problem in optical flow estimation.

[0074] It can be understood that this embodiment is only used to illustrate the optical flow learning network based on PWC-Net, and is not used to limit the optical flow learning network. For example, the optical flow learning network can also be replaced by MaskFlowNet (an asymmetric feature matching network based on learnable occlusion masks).

[0075] In this embodiment, an optical flow learning network based on PWC-Net is used to obtain an offset corresponding to the optical flow of a pixel point in the first denoised image moving to the second denoised image, a deformation function is constructed according to the offset, and finally the first denoised image is aligned to the second denoised image according to the constructed deformation function, so as to further add noise to the initial aligned image obtained by alignment to obtain a target aligned image.

[0076] In one embodiment, after the above step S106, it also includes: correcting the initial alignment image to obtain a corrected alignment image; then the above step S108 specifically includes: according to the position information of the image noise in the original image, adding the image noise of the original image to the corresponding position in the corrected alignment image to obtain a target alignment image corresponding to the original image.

[0077] In a specific implementation, taking into account the errors in optical flow estimation, the initial aligned image may have image edge problems that are not processed well. Therefore, after aligning the first denoised image to the second denoised image through optical flow to obtain the initial aligned image, this step also includes correcting the initial aligned image to obtain a corrected aligned image, and further adding the image noise of the original image to the corresponding position in the corrected aligned image according to the position information of the image noise in the original image to obtain a target aligned image corresponding to the original image.

[0078] In this embodiment, by correcting the initial aligned image and adding image noise to the corrected initial aligned image, the effect of optical flow on the real image is improved, thereby improving the alignment effect of the original image to the reference frame image.

[0079] In one embodiment, the step of correcting the initial aligned image to obtain a corrected aligned image includes: inputting the initial aligned image, the second denoised image, and the optical flow information from the first denoised image to the second denoised image into a pre-trained correction model to obtain a first score map of the initial aligned image and a second score map of the second denoised image; the first score map is used to represent the first weight of each pixel in the initial aligned image, and the second score map is used to represent the second weight of each pixel in the second denoised image; and the initial aligned image and the second denoised image are fused according to the first score map and the second score map to obtain the corrected aligned image.

[0080] Among them, the correction model can adopt the Unet correction model, specifically 1 / 2Unet, and the downsampling is increased to 4 times downsampling to ensure that the method can be run on the mobile phone. The training of the Unet correction model can first determine a larger weight of the regularization term, and focus the loss optimization on the regularization term, so that the correction model can find the location of inaccurate optical flow prediction on the basis of more initial aligned images, and perform image edge repair. After that, the learning rate is lowered, the proportion of the regularization term is reduced, and the loss optimization is focused on the mse loss. At the same time, with the increase of the regularization term loss (that is, the part of the second denoised image reused increases, and the proportion of the second denoised image increases), but because the learning rate is small at this time, the phenomenon of training collapse and the appearance of a completely white score image can be prevented. Through this training process, the processing effect of the motion boundary of the initial aligned image can be improved.

[0081] In the specific implementation, refer to Figure 4 , is a flow chart of correcting the initial aligned image in one embodiment. According to the specific implementation steps of step S104, the first denoised image and the second denoised image are input into the pre-trained optical flow learning network to obtain the optical flow information from the first denoised image to the second denoised image. Then, the initial aligned image, the second denoised image and the optical flow information are input into the pre-trained Unet correction model. After the sigmoid layer output, a first score map for representing the first weight of each pixel in the initial aligned image is obtained. According to the first score map, a second score map representing the second weight of each pixel in the second denoised image is determined, wherein the first score map, the second score map and the initial aligned image and the second denoised image are of the same size. Since the weights of the pixels in the image represented by the first score map and the second score map are in the interval [0, 1]. Then, the initial aligned image and the second denoised image can be fused according to the first score map, the second score map and the mapping function to obtain a corrected aligned image.

[0082] Furthermore, in one embodiment, the step of fusing the initial aligned image and the second denoised image according to the first score map and the second score map to obtain a revised aligned image includes: obtaining a first weight of each pixel in the initial aligned image from the first score map, and obtaining a second weight of the corresponding pixel in the second denoised image from the second score map; and performing weighted processing on the pixel value of each pixel in the initial aligned image and the pixel value of the corresponding pixel in the second denoised image according to the first weight of each pixel in the initial aligned image and the second weight of the corresponding pixel in the second denoised image to obtain a revised aligned image.

[0083] In a specific implementation, the mapping function is used to map the positions of the first score map, the initial alignment image, the second score map, and the target alignment image. For example, if the pixel value of the pixel point (i, j) in the initial alignment image is recorded as w i,j , the pixel value of the corresponding pixel point (i', j') in the second denoised image is recorded as r i,j The first weight of the pixel (i, j) obtained from the first score map of the initial aligned image is recorded as s i,j , then the second weight of the corresponding pixel point (i', j') obtained from the second score map of the second denoised image can be recorded as 1-s i,j , then the relationship between weighted processing of the pixel values ​​of the initial aligned image and the second denoised image according to the first weight and the second weight can be expressed as:

[0084] refine i,j =s i,j *w i,j +(1-s i,j )*r i,j

[0085] Among them, refine i,j represents the new pixel value of the pixel corresponding to the pixel (i, j) in the initial aligned image, where s i,j is a scalar.

[0086] In this embodiment, the initial aligned image, the second denoised image and the optical flow are input into a pre-trained correction model to obtain a first score map of the initial aligned image and a second score map of the second denoised image. The first weights of the pixels of the initial aligned image and the second weights of the corresponding pixels of the second denoised image are further obtained from the first score map and the second score map, respectively. The pixel values ​​of the initial aligned image and the second denoised image are weighted according to the first weight and the second weight to obtain new pixel values, which are used to replace the pixel values ​​in the initial aligned image to correct the initial aligned image and obtain a corrected aligned image. The method solves the problem of image edges that are not processed well by the optical flow by refining the image that is initially aligned according to the optical flow, improves the processing effect of the motion boundary of the initial aligned image, and further improves the image alignment effect of the target aligned image obtained after adding image noise.

[0087] In one embodiment, step S108 specifically includes: obtaining the pixel value of the corresponding position in the initial aligned image according to the position information of the image noise in the original image; adding the pixel value of the image noise position in the original image to the pixel value of the corresponding position to obtain a new pixel value, replacing the pixel value of the corresponding position of the initial aligned image with the new pixel value, and obtaining a target aligned image corresponding to the original image.

[0088] Among them, image noise affects the pixel values ​​of the image. For example, salt noise is white point noise with a pixel value of 255 randomly added to the image, and pepper noise is black point noise with a pixel value of 0 randomly added to the image.

[0089] In the specific implementation, since the intensity value interval of the image is [0, 255], when the pixel value of the image noise position in the original image is added to the pixel value of the corresponding position in the initial aligned image to obtain a new pixel value, the obtained new pixel value must also be within the interval of [0, 255]. For example, if the image noise at position (m, n) is 20, and the pixel value at position (m', n') corresponding to position (m, n) in the initial aligned image is 150, then the pixel value 150 at position (m', n') in the initial aligned image will be replaced by a new pixel value of 150+20=170. Similarly, the pixel value of each pixel in the initial aligned image is subjected to noise addition processing, thereby obtaining an aligned image containing image noise as the target aligned image of the original image.

[0090] In this embodiment, the image noise at each position in the original image is added to the pixel value at the corresponding position to obtain a new pixel value, and the pixel value at the corresponding position of the initial aligned image is replaced by the new pixel value, thereby adding the image noise to the corresponding position in the initial aligned image, compensating for the noise in the moving area and the non-moving area in the initial aligned image, ensuring the continuity and consistency of the noise in the non-moving area to the greatest extent, and further improving the image alignment effect.

[0091] In another embodiment, Figure 5 As shown, an image alignment method is provided, the method comprising the following steps:

[0092] Step S502, obtaining a first denoised image corresponding to the original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image of the multiple frames of original images except the reference frame image;

[0093] Step S504, performing a difference process between the original image and the first noise reduction image to obtain the image noise of the original image;

[0094] Step S506, inputting the first denoised image and the second denoised image into a pre-trained optical flow learning network to obtain optical flow information from the first denoised image to the second denoised image;

[0095] Step S508, aligning the first denoised image with the second denoised image according to the offset direction and offset in the optical flow information to obtain an initial aligned image corresponding to the original image;

[0096] Step S510, inputting the initial aligned image, the second denoised image, and the optical flow information from the first denoised image to the second denoised image into a pre-trained correction model to obtain a first score map of the initial aligned image and a second score map of the second denoised image;

[0097] Step S512, fusing the initial aligned image and the second denoised image according to the first score map and the second score map to obtain a corrected aligned image;

[0098] Step S514, according to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the corrected aligned image to obtain a target aligned image corresponding to the original image.

[0099] In this embodiment, on the one hand, by refining the initial aligned image aligned by optical flow, and on the other hand, by first removing the noise and then pasting it back to the corrected aligned image, on the one hand, the noise consistency of the non-moving area can be guaranteed to the greatest extent, so as to further perform the ghost removal task, and on the other hand, the effect of the optical flow and the refined network on the real picture is improved. And calculating the optical flow by the denoised image can significantly improve the accuracy of the optical flow, thereby improving the alignment effect of the initial aligned image, and then improving the image effect of the target aligned image. This method can be applied to the alignment of real images of non-rigid bodies, solving the core problem of HDR (high dynamic range image), the core function of the mobile phone album.

[0100] It should be understood that although Figure 1-5 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-5 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0101] In one embodiment, Figure 6 As shown, an image alignment device is provided, comprising: a noise reduction image acquisition module 602, an image noise acquisition module 604, an initial alignment image acquisition module 606 and a target alignment image acquisition module 608, wherein:

[0102] The denoised image acquisition module 602 is used to acquire a first denoised image corresponding to the original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image of the multiple frames of original images except the reference frame image;

[0103] An image noise acquisition module 604 is used to perform a difference process between the original image and the first noise reduction image to obtain the image noise of the original image;

[0104] An initial alignment image acquisition module 606 is used to align the first denoised image with the second denoised image to obtain an initial alignment image corresponding to the original image;

[0105] The target alignment image acquisition module 608 is used to add the image noise of the original image to the corresponding position in the initial alignment image according to the position information of the image noise in the original image, so as to obtain the target alignment image corresponding to the original image.

[0106] In one embodiment, the above-mentioned initial alignment image acquisition module 606 is specifically used to input the first denoised image and the second denoised image into a pre-trained optical flow learning network to obtain optical flow information from the first denoised image to the second denoised image; according to the offset direction and offset in the optical flow information, the first denoised image is aligned to the second denoised image to obtain an initial alignment image corresponding to the original image.

[0107] In one embodiment, the above-mentioned device also includes an image correction module, which is used to correct the initial alignment image to obtain a corrected alignment image; then the above-mentioned target alignment image acquisition module 608 is also used to add the image noise of the original image to the corresponding position in the corrected alignment image according to the position information of the image noise in the original image, so as to obtain a target alignment image corresponding to the original image.

[0108] In one embodiment, the above-mentioned image correction module is specifically used to input the initial aligned image, the second denoised image and the optical flow information from the first denoised image to the second denoised image into a pre-trained correction model to obtain a first score map of the initial aligned image and a second score map of the second denoised image; the first score map is used to characterize the first weight of each pixel in the initial aligned image, and the second score map is used to characterize the second weight of each pixel in the second denoised image; according to the first score map and the second score map, the initial aligned image and the second denoised image are fused to obtain a corrected aligned image.

[0109] In one embodiment, the above-mentioned image correction module is also used to obtain a first weight of each pixel in the initial aligned image from the first score map, and obtain a second weight of the corresponding pixel in the second denoised image from the second score map; according to the first weight of each pixel in the initial aligned image and the second weight of the corresponding pixel in the second denoised image, weighted processing is performed on the pixel value of the pixel in the initial aligned image and the pixel value of the corresponding pixel in the second denoised image to obtain a corrected aligned image.

[0110] In one embodiment, the target alignment image acquisition module 608 is specifically used to obtain the pixel value of the corresponding position in the initial alignment image according to the position information of the image noise in the original image; add the pixel value of the image noise position in the original image to the pixel value of the corresponding position to obtain a new pixel value, replace the pixel value of the corresponding position of the initial alignment image with the new pixel value, and obtain the target alignment image corresponding to the original image.

[0111] In one embodiment, the denoised image acquisition module 602 is specifically used to input the original image and the reference frame image into a pre-trained denoising model respectively to obtain a first denoised image corresponding to the original image and a second denoised image corresponding to the reference frame image.

[0112] It should be noted that the image alignment device of the present application corresponds one-to-one to the image alignment method of the present application. The technical features and beneficial effects described in the embodiment of the above-mentioned image alignment method are all applicable to the embodiment of the image alignment device. For specific contents, please refer to the description in the embodiment of the method of the present application. It will not be repeated here. This is hereby declared.

[0113] In addition, each module in the above-mentioned image alignment device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0114] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image alignment method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0115] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0116] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0117] Acquire a first denoised image corresponding to the original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image of the multiple frames of original images except the reference frame image;

[0118] Performing a difference process between the original image and the first denoised image to obtain image noise of the original image;

[0119] Aligning the first denoised image with the second denoised image to obtain an initial aligned image corresponding to the original image;

[0120] According to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the initial aligned image to obtain the target aligned image corresponding to the original image.

[0121] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0122] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0123] Acquire a first denoised image corresponding to the original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image of the multiple frames of original images except the reference frame image;

[0124] Performing a difference process between the original image and the first denoised image to obtain image noise of the original image;

[0125] Aligning the first denoised image with the second denoised image to obtain an initial aligned image corresponding to the original image;

[0126] According to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the initial aligned image to obtain the target aligned image corresponding to the original image.

[0127] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0129] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image alignment method, characterized in that: The method comprises: Acquire a first denoised image corresponding to an original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image of the multiple frames of original images except the reference frame image; Performing a difference process between the original image and the first denoised image to obtain image noise of the original image; Inputting the first denoised image and the second denoised image into a pre-trained optical flow learning network to obtain optical flow information from the first denoised image to the second denoised image; Aligning the first denoised image with the second denoised image according to the offset direction and offset in the optical flow information to obtain an initial aligned image corresponding to the original image; According to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the initial aligned image to obtain a target aligned image corresponding to the original image.

2. The method according to claim 1, characterized in that The method further comprises: Correcting the initial aligned image to obtain a corrected aligned image; The adding the image noise of the original image to the corresponding position in the initial aligned image according to the position information of the image noise in the original image to obtain the target aligned image corresponding to the original image includes: According to the position information of the image noise in the original image, the image noise of the original image is added to the corresponding position in the corrected aligned image to obtain a target aligned image corresponding to the original image.

3. The method according to claim 2, characterized in that The correcting the initial aligned image to obtain a corrected aligned image includes: Inputting the initial aligned image, the second denoised image, and the optical flow information from the first denoised image to the second denoised image into a pre-trained correction model to obtain a first score map of the initial aligned image and a second score map of the second denoised image; the first score map is used to represent a first weight of each pixel in the initial aligned image, and the second score map is used to represent a second weight of each pixel in the second denoised image; The initial aligned image and the second denoised image are fused according to the first score map and the second score map to obtain the corrected aligned image.

4. The method according to claim 3, characterized in that The fusing the initial aligned image and the second denoised image according to the first score map and the second score map to obtain the corrected aligned image includes: Acquire a first weight of each pixel in the initial aligned image from the first score map, and acquire a second weight of the corresponding pixel in the second denoised image from the second score map; According to the first weight of each pixel in the initial aligned image and the second weight of the corresponding pixel in the second denoised image, the pixel values ​​of the pixel in the initial aligned image and the pixel values ​​of the corresponding pixel in the second denoised image are weighted to obtain the corrected aligned image.

5. The method according to any one of claims 1 to 4, characterized in that: The adding the image noise of the original image to the corresponding position in the initial aligned image according to the position information of the image noise in the original image to obtain the target aligned image corresponding to the original image includes: According to the position information of the image noise in the original image, obtaining the pixel value of the corresponding position in the initial aligned image; The pixel value of the image noise position in the original image is added to the pixel value of the corresponding position to obtain a new pixel value, and the pixel value of the corresponding position of the initial aligned image is replaced by the new pixel value to obtain a target aligned image corresponding to the original image.

6. The method according to claim 1, characterized in that The acquiring a first denoised image corresponding to the original image and a second denoised image corresponding to the reference frame image of the original image includes: The original image and the reference frame image are respectively input into a pre-trained denoising model to obtain a first denoised image corresponding to the original image and a second denoised image corresponding to the reference frame image.

7. An image alignment device, characterized in that: The device comprises: A denoised image acquisition module, used to acquire a first denoised image corresponding to an original image and a second denoised image corresponding to a reference frame image of the original image; the reference frame image is a frame image selected from multiple frames of original images to be aligned, and the original image is any frame image of the multiple frames of original images except the reference frame image; An image noise acquisition module, used for performing a difference process between the original image and the first noise reduction image to obtain the image noise of the original image; an initial alignment image acquisition module, configured to input the first denoised image and the second denoised image into a pre-trained optical flow learning network to obtain optical flow information from the first denoised image to the second denoised image; align the first denoised image to the second denoised image according to an offset direction and an offset in the optical flow information to obtain an initial alignment image corresponding to the original image; The target alignment image acquisition module is used to add the image noise of the original image to the corresponding position in the initial alignment image according to the position information of the image noise in the original image, so as to obtain the target alignment image corresponding to the original image.

8. The device according to claim 7, characterized in that The device also includes: An image correction module is used to correct the initial alignment image to obtain a corrected alignment image; The target alignment image acquisition module is further used to add the image noise of the original image to the corresponding position in the corrected alignment image according to the position information of the image noise in the original image, so as to obtain the target alignment image corresponding to the original image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Image alignment method and mobile terminal

    CN107527360A

  • Image denoising method, device and system

    CN108898567A

  • Image denoising method and device

    CN111275653A

  • Color interpolation method and device, equipment and storage medium

    CN111510691A

  • Image denoising model training method, image denoising method and device and medium

    CN111709891A