Image processing method and device and storage medium

Abstract

The invention relates to an image processing method and device and a storage medium. The image processing method comprises the following steps: determining a reference frame image and a moving frame image in a photographing process; performing first registration processing on the reference frame image and the moving frame image to obtain a first registration image and a first motion feature; performing second registration processing based on the first registration image and the reference frame image to obtain a second motion feature; and carrying out image fusion to obtain a target image. According to the method and the device, the first registration image and the reference frame image are registered in the second registration, and the second motion feature and the first motion feature of the first registration image and the reference frame image are superposed to obtain the fused image after registration, so that the registration error rate caused by calculation of an interpolation method in the registration process is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, device and storage medium. Background Art

[0002] With the development of terminal technology, users have higher and higher requirements on the image quality displayed by terminal cameras.

[0003] In order to improve the image quality of the photographed image, a multi-frame fusion method is usually used to denoise the captured image to improve the image quality. In the multi-frame fusion denoising method, images with different frame numbers need to be registered.

[0004] In the related art, the registration process is relatively complicated and may lead to a reduction in the quality of the registered image, thereby affecting the user experience. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, device and storage medium.

[0006] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising determining a reference frame image and a moving frame image during a photographing process; performing a first registration process on the reference frame image and the moving frame image to obtain a first registered image and a first motion feature, wherein the first motion feature is a motion feature between the moving frame image and the first registered image; performing a second registration process based on the first registered image and the reference frame image to obtain a second motion feature, wherein the second motion feature is a motion feature between the reference frame image and the first registered image; performing image fusion based on the reference frame image, the first registered image, the moving frame, the first motion feature and the second motion feature to obtain a target image.

[0007] In one embodiment, the second registration processing based on the first registered image and the reference frame image includes: downsampling the reference frame image based on a first sampling ratio to obtain a first reference frame image, and downsampling the first registered image based on the first sampling ratio to obtain a first sampling image, wherein the first sampling ratio is greater than 0 and less than 1; constructing a first image pyramid based on the first reference frame image, wherein the first reference frame image is the first layer of the first image pyramid, the first image pyramid has N layers, and the N-th frame reference frame image corresponding to the N-th layer is obtained by downsampling the N-1-th frame reference frame image corresponding to the N-1-th layer based on the first sampling ratio. The method comprises the following steps: downsampling the image of the Nth frame and the image of the Nth frame according to a sampling ratio, and constructing a second image pyramid based on the first sampling image, wherein the first sampling image is the first layer of the second image pyramid, the second image pyramid is N layers, and the N-th frame sampling image corresponding to the N-th layer is obtained by downsampling the N-1-th frame sampling image corresponding to the N-1-th layer based on the first sampling ratio, wherein N is an integer greater than 1; determining a first optical flow feature based on the N-th frame reference frame image and the N-th frame sampling image, wherein the first optical flow feature is used to characterize the motion feature of the optical flow field between the N-th frame reference frame image and the N-th frame sampling second image; and iterating upsampling the first optical flow feature to obtain a second motion feature.

[0008] In another embodiment, the upsampling iteratively of the first optical flow feature to obtain the second motion feature includes: determining the initial value of the first optical flow feature and the first optical flow feature increment based on the N-th frame reference frame image and the N-th frame sampling image; performing feature fusion on the initial value of the first optical flow feature and the first optical flow feature increment to obtain the first optical flow feature; upsampling the first optical flow feature based on a second sampling ratio to obtain a second optical flow sampling feature, and determining the initial value of the second optical flow feature based on the second optical flow sampling feature, the second sampling ratio being the inverse of the first sampling ratio; determining the second optical flow feature increment based on the N-1-th frame reference frame image and the N-1-th frame sampling image; performing feature fusion on the initial value of the second optical flow feature and the second optical flow feature increment to obtain the second motion feature.

[0009] In another embodiment, determining the initial value of the first optical flow feature includes: determining a first target sub-region and a first adjacent region of the first target sub-region in the Nth frame sampling image based on a first search block; performing optical flow feature extraction on the first target sub-region and the first adjacent region respectively to obtain a first target optical flow feature corresponding to the first target sub-region and a first candidate optical flow feature corresponding to each first adjacent region; performing a similarity comparison between the first target optical flow feature and the first optical flow feature to obtain a first similarity result, and performing a similarity comparison between each first candidate optical flow feature and the first optical flow feature to obtain at least one second similarity result; if the first similarity result is greater than all the second similarity results, determining the first target optical flow feature as the second candidate optical flow feature; if there is at least one second similarity result greater than the first similarity result, determining the first candidate optical flow feature corresponding to the maximum value of the similarity result in the at least one second similarity result as the second candidate optical flow feature; and performing screening comparison between the second candidate optical flow feature and the zero vector to determine the initial value of the first optical flow feature.

[0010] In another embodiment, the screening and comparison of the second candidate optical flow feature with the zero vector to determine the initial value of the first optical flow feature includes: if the measurement value of the second candidate optical flow feature is less than or equal to the first measurement threshold, then determining the measurement value of the second optical flow candidate feature as the initial value of the first optical flow feature; if the measurement value of the second candidate optical flow feature is greater than the first measurement threshold, then performing a similarity comparison between the zero vector and the first optical flow feature to obtain a third similarity result, and determining the initial value of the first optical flow feature based on the third similarity result, including: if the third similarity result is greater than the first similarity result, and the difference between the third similarity result and the first similarity result is greater than the first similarity threshold, then determining the zero vector as the initial value of the first optical flow feature.

[0011] In another embodiment, determining the initial value of the second optical flow feature includes: determining a second target sub-region and a second adjacent region adjacent to the second target sub-region in the N-1th frame sampling image based on the second search block; performing optical flow feature extraction on the second target sub-region and the second adjacent region respectively to obtain a second target optical flow feature corresponding to the second target sub-region and a third candidate optical flow feature corresponding to each second adjacent region; performing a similarity comparison between the second target optical flow feature and the second optical flow sampling feature to obtain a fourth similarity result, and performing a similarity comparison between each third candidate optical flow feature and the second optical flow sampling feature to obtain at least one fifth similarity result; if the fourth similarity result is greater than all the fifth similarity results, determining the second target optical flow feature as the fourth candidate optical flow feature; if there is at least one fifth similarity result greater than the fourth similarity result, determining the third candidate optical flow feature corresponding to the maximum value of the similarity result in the at least one fifth similarity result as the fourth candidate optical flow feature; screening and comparing the fourth candidate optical flow feature with the zero vector to determine the initial value of the second optical flow feature.

[0012] In another embodiment, the screening and comparison of the fourth candidate optical flow feature with the zero vector to determine the initial value of the second optical flow feature includes: if the measurement value of the fourth candidate optical flow feature is less than or equal to the second measurement threshold, then determining the measurement value of the fourth optical flow candidate feature as the initial value of the second optical flow feature; if the measurement value of the fourth candidate optical flow feature is greater than the second measurement threshold, then performing a similarity comparison between the zero vector and the second optical flow sampling feature to obtain a sixth similarity result, and determining the initial value of the second optical flow feature based on the sixth similarity result, including: if the sixth similarity result is greater than the fourth similarity result, and the difference between the sixth similarity result and the first similarity result is greater than the second similarity threshold, then determining the zero vector as the initial value of the second optical flow feature.

[0013] In another embodiment, after obtaining the second motion feature, it also includes: performing feature fusion on the first motion feature and the second motion feature to obtain a third motion feature, and the third motion feature is used to characterize the motion features between the reference frame image and the moving frame image after the second registration process; determining the similarity of the first motion feature corresponding to the first image block as a fourth similarity result, and determining the similarity of the second image block corresponding to the third motion feature as a fifth similarity result, the first image block is determined based on the first registration image, and the second image block is determined based on the reference frame image; if the fourth similarity result is less than the fifth similarity result, determining the second optical flow feature increment; if the fourth similarity result is greater than or equal to the fifth similarity result, setting the second motion feature to zero, and then determining the second optical flow feature increment.

[0014] In another embodiment, the number of optical flows corresponding to any layer of the first image pyramid is determined based on the ratio of the reference frame image area of ​​the current layer to the area of ​​the second search block; the number of optical flows corresponding to any layer of the second image pyramid is determined based on the ratio of the sampled image area of ​​the current layer to the area of ​​the second search block.

[0015] According to a second aspect of an embodiment of the present disclosure, an image processing device is provided, comprising: an acquisition unit, configured to determine a reference frame image and a moving frame image during a photographing process; a registration unit, configured to perform a first registration process on the reference frame image and the moving frame image to obtain a first registered image and a first motion feature, wherein the first motion feature is a motion feature between the moving frame image and the first registered image; the registration unit is further configured to perform a second registration process based on the first registered image and the reference frame image to obtain a second motion feature, wherein the second motion feature is a motion feature between the reference frame image and the first registered image; and a processing unit, configured to perform image fusion based on the reference frame image, the first registered image, the moving frame, the first motion feature and the second motion feature to obtain a target image.

[0016] In one embodiment, the registration unit performs a second registration process based on the first registered image and the reference frame image, including: downsampling the reference frame image based on a first sampling ratio to obtain a first reference frame image, and downsampling the first registered image based on the first sampling ratio to obtain a first sampling image, wherein the first sampling ratio is greater than 0 and less than 1; constructing a first image pyramid based on the first reference frame image, wherein the first reference frame image is the first layer of the first image pyramid, the first image pyramid has N layers, and the N-th frame reference frame image corresponding to the N-th layer is obtained by the N-1-th frame reference frame image corresponding to the N-1-th layer based on the first image pyramid. The method comprises the following steps: downsampling the image of the Nth frame and the image of the Nth frame according to the first sampling ratio, and constructing a second image pyramid based on the first sampling image, wherein the first sampling image is the first layer of the second image pyramid, the second image pyramid is N layers, and the N-th frame sampling image corresponding to the N-th layer is obtained by downsampling the N-1-th frame sampling image corresponding to the N-1-th layer based on the first sampling ratio, wherein N is an integer greater than 1; determining a first optical flow feature based on the N-th frame reference frame image and the N-th frame sampling image, wherein the first optical flow feature is used to characterize the motion feature of the optical flow field between the N-th frame reference frame image and the N-th frame sampling second image; and iterating upsampling the first optical flow feature to obtain a second motion feature.

[0017] In another embodiment, the registration unit determines to iteratively upsample the first optical flow feature to obtain a second motion feature, including: determining an initial value of the first optical flow feature and a first optical flow feature increment based on the N-th frame reference frame image and the N-th frame sampling image; performing feature fusion on the initial value of the first optical flow feature and the first optical flow feature increment to obtain the first optical flow feature; upsampling the first optical flow feature based on a second sampling ratio to obtain a second optical flow sampling feature, and determining an initial value of the second optical flow feature based on the second optical flow sampling feature, the second sampling ratio being the inverse of the first sampling ratio; determining a second optical flow feature increment based on the N-1-th frame reference frame image and the N-1-th frame sampling image; performing feature fusion on the initial value of the second optical flow feature and the second optical flow feature increment to obtain the second motion feature.

[0018] In another embodiment, the registration unit determines the initial value of the first optical flow feature, including: determining a first target sub-region and a first adjacent region of the first target sub-region in the Nth frame sampling image based on a first search block; extracting optical flow features from the first target sub-region and the first adjacent region respectively to obtain a first target optical flow feature corresponding to the first target sub-region and a first candidate optical flow feature corresponding to each first adjacent region; performing a similarity comparison between the first target optical flow feature and the first optical flow feature to obtain a first similarity result, and performing a similarity comparison between each first candidate optical flow feature and the first optical flow feature to obtain at least one second similarity result; when the first similarity result is greater than all the second similarity results, determining the first target optical flow feature as the second candidate optical flow feature; when there is at least one second similarity result greater than the first similarity result, determining the first candidate optical flow feature corresponding to the maximum value of the similarity result in the at least one second similarity result as the second candidate optical flow feature; screening and comparing the second candidate optical flow feature with a zero vector to determine the initial value of the first optical flow feature.

[0019] In another embodiment, the registration unit screens and compares the second candidate optical flow feature with the zero vector to determine the initial value of the first optical flow feature, including: the registration unit determines the measurement value of the second optical flow candidate feature as the initial value of the first optical flow feature when the measurement value of the second candidate optical flow feature is less than or equal to the first measurement threshold; when the measurement value of the second candidate optical flow feature is greater than the first measurement threshold, performs a similarity comparison between the zero vector and the first optical flow feature to obtain a third similarity result, and determines the initial value of the first optical flow feature based on the third similarity result, including: if the third similarity result is greater than the first similarity result, and the difference between the third similarity result and the first similarity result is greater than the first similarity threshold, determines the zero vector as the initial value of the first optical flow feature.

[0020] In another embodiment, the registration unit determines the initial value of the second optical flow feature, including: determining a second target sub-region and a second adjacent region adjacent to the second target sub-region in the N-1th frame sampling image based on the second search block; extracting optical flow features from the second target sub-region and the second adjacent region respectively to obtain a second target optical flow feature corresponding to the second target sub-region and a third candidate optical flow feature corresponding to each second adjacent region; performing a similarity comparison between the second target optical flow feature and the second optical flow sampling feature to obtain a fourth similarity result, and performing a similarity comparison between each third candidate optical flow feature and the second optical flow sampling feature to obtain at least one fifth similarity result; when the fourth similarity result is greater than all the fifth similarity results, determining the second target optical flow feature as the fourth candidate optical flow feature; when there is at least one fifth similarity result greater than the fourth similarity result, determining the third candidate optical flow feature corresponding to the maximum similarity result in the at least one fifth similarity result as the fourth candidate optical flow feature; screening and comparing the fourth candidate optical flow feature with the zero vector to determine the initial value of the second optical flow feature.

[0021] In yet another embodiment, the registration unit screens and compares the fourth candidate optical flow feature with a zero vector to determine an initial value of the second optical flow feature, including:

[0022] In a case where the metric value of the fourth candidate optical flow feature is less than or equal to the second metric threshold, determining the metric value of the fourth candidate optical flow feature as an initial value of the second optical flow feature;

[0023] When the metric value of the fourth candidate optical flow feature is greater than the second metric threshold, performing a similarity comparison between the zero vector and the second optical flow sampling feature to obtain a sixth similarity result, and determining an initial value of the second optical flow feature based on the sixth similarity result, including:

[0024] When the sixth similarity result is greater than the fourth similarity result and a difference between the sixth similarity result and the fourth similarity result is greater than a second similarity threshold, a zero vector is determined as an initial value of the second optical flow feature.

[0025] In another embodiment, after the registration unit obtains the second motion feature, it also includes: performing feature fusion on the first motion feature and the second motion feature to obtain a third motion feature, wherein the third motion feature is used to characterize the motion feature between the reference frame image and the moving frame image after the second registration process; determining the similarity of the first motion feature corresponding to the first image block as a fourth similarity result, and determining the similarity of the second image block corresponding to the third motion feature as a fifth similarity result, the first image block is determined based on the first registration image, and the second image block is determined based on the reference frame image; when the fourth similarity result is less than the fifth similarity result, determining the second optical flow feature increment; when the fourth similarity result is greater than or equal to the fifth similarity result, after setting the second motion feature to zero, determining the second optical flow feature increment.

[0026] In another embodiment, the processing unit is used to: determine the number of optical flows corresponding to the first image pyramid of any layer based on the ratio of the reference frame image area of ​​the current layer to the area of ​​the second search block; determine the number of optical flows corresponding to the second image pyramid of any layer based on the ratio of the sampled image area of ​​the current layer to the area of ​​the second search block.

[0027] According to a third aspect of an embodiment of the present disclosure, an image processing device is provided, comprising: a processor; a memory for storing processor executable commands; wherein the processor is configured to: execute the image processing method described in the first aspect or any one embodiment of the first aspect.

[0028] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a device, the device is enabled to execute the image processing method described in the first aspect or any one of the embodiments of the first aspect.

[0029] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: after the first registration process obtains the first registration image and the first motion feature, the second registration is performed on the first registration image and the reference frame image, and the second motion feature of the two is superimposed with the first motion feature to obtain a registered and fused image, which reduces the number of times the interpolation method is used compared to the registration process in the related art. In this way, the registration error rate caused by the interpolation method calculation in the registration process is reduced.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0032] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0033] Figure 2 The invention is a flowchart showing a method for determining a reference frame image and a moving frame image according to an exemplary embodiment.

[0034] Figure 3 The figure is a flow chart of a second registration method according to an exemplary embodiment.

[0035] Figure 4 The figure is a flowchart of a first optical flow feature upsampling iterative method according to an exemplary embodiment.

[0036] Figure 5 The figure is a flowchart of a method for determining an initial value of a first optical flow feature according to an exemplary embodiment.

[0037] Figure 6 The flowchart is another method for determining an initial value of a first optical flow feature according to an exemplary embodiment.

[0038] Figure 7 The figure is a flow chart showing a method for determining an initial value of a second optical flow feature according to an exemplary embodiment.

[0039] Figure 8 The flowchart of another method for determining an initial value of a second optical flow feature is shown according to an exemplary embodiment.

[0040] Fig. 9 The figure is a flow chart of a second registration correction method according to an exemplary embodiment.

[0041] Fig. 10A It is a schematic flow chart of synthesizing YUV images of multiple frames with the same exposure according to an exemplary embodiment.

[0042] Fig. 10B The figure is a schematic flowchart of synthesizing RAW images of multiple frames with the same exposure according to an exemplary embodiment.

[0043] Fig. 10C It is a schematic flow chart of YUV image synthesis under multiple frames of different exposures according to an exemplary embodiment.

[0044] Fig. 10D The figure is a schematic flowchart of synthesizing multiple RAW images under different exposures according to an exemplary embodiment.

[0045] Fig.11 The figure is a block diagram of an image processing device according to an exemplary embodiment.

[0046] Fig.12 The invention is a block diagram of a device for image processing according to an exemplary embodiment.

[0047] Fig.13 The invention is a block diagram of a device for image processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0048] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0049] The image processing method involved in the embodiments of the present disclosure is applied to a terminal. The terminal involved in the present disclosure may also be referred to as a terminal device, a user equipment (User Equipment, UE), a mobile station (Mobile Station, MS), a mobile terminal (Mobile Terminal, MT), etc., which is a device that provides voice and / or data connectivity to a user. For example, the terminal may be a handheld device with a wireless connection function, a vehicle-mounted device, etc. At present, some examples of terminals are: a smart phone (Mobile Phone), a customer premises equipment (Customer Premise Equipment, CPE), a pocket computer (Pocket Personal Computer, PPC), a PDA, a personal digital assistant (Personal Digital Assistant, PDA), a laptop computer, a tablet computer, a wearable device, or a vehicle-mounted device, etc. It should be understood that the embodiments of the present disclosure do not limit the specific technology and specific device form adopted by the terminal.

[0050] It should be noted that, in order to facilitate understanding of the technical solution, the following embodiments are described using a terminal in the form of a smart phone as an example.

[0051] In the related art, when a user uses a terminal to take a photo, the movement of the photographed object itself or the brightness change of the environment may cause noise to the photographed image, such as blurry or uneven brightness of the photographed image.

[0052] With the development of terminal technology, users have higher and higher requirements for the image quality displayed by the terminal camera. In the process of taking pictures, in order to solve the above problems, the related technology usually adopts the method of multi-frame image fusion to reduce noise and improve the image quality. It can be understood that multi-frame image fusion mainly uses the continuous acquisition of multiple image frames including the photographed object in a short time, and performs fusion denoising based on the common elements in the image frames to generate a target image. Since users usually use handheld shooting. There may be slight shaking during the user's hand-holding process, which causes the lens to move slightly at the moment of taking pictures, or due to differences in exposure time, the continuous multi-frame images obtained look similar in content, but there is still obvious content movement, and the moving content is usually different from the relative position of the background in the image. Therefore, before fusing and denoising the image, it is necessary to perform registration operations on these multi-frame images so that the same content in the multi-frame images is aligned in a unified coordinate system (spatial coordinate system or color coordinate system), and then the registered images are fused in the time domain and frequency domain to obtain a denoised image.

[0053] In the related art, the registration of multiple frames of images is usually achieved through two registration processes. The first registration is a rough registration using the grid division method, and the first registration map is obtained using the interpolation method. The second registration is based on the optical flow algorithm. The constraint of feature consistency is added to the energy function term under the framework of the optical flow algorithm. The first registration map and the reference map are interpolated to obtain the second registration map. Finally, the second registration map is fused with the original image in the time domain and frequency domain to obtain the denoised image.

[0054] However, in the methods mentioned in the above-mentioned related technologies, on the one hand, since two interpolation methods are required to obtain the second registration map, this will bring a huge amount of calculation to the registration process and is prone to errors. It is understandable that the image after the difference calculation has a large loss on fine textures, which is not conducive to the final image fusion. On the other hand, since a more complex energy function is used for constraints in the optical flow algorithm framework, the calculation process is increased, which will reduce the real-time performance of the processing.

[0055] Based on this, in order to solve the above technical problems, the present disclosure proposes an image processing method, after the first registration process obtains the first registration image and the first motion feature, the second registration is performed on the first registration image and the reference frame image, and the second motion feature of the two is superimposed with the first motion feature to obtain a fused image after registration, which reduces the number of times the interpolation method is used compared to the registration process in the related art. In this way, the registration error rate caused by the interpolation method calculation in the registration process is reduced.

[0056] It should be noted that, in order to facilitate the understanding of the embodiments of the present disclosure, some terms used in the embodiments of the present disclosure are explained below to facilitate understanding by those skilled in the art.

[0057] Optical flow: Optical flow is a way to describe pixel changes in a continuous visual scene. In the context of object movement or camera perspective changes, the optical flow algorithm attempts to calculate the motion relationship between each pixel in the image.

[0058] Optical flow vector: The optical flow vector is a specific representation of this movement. It describes the displacement of a pixel from one frame to another in the form of a vector. For example, an optical flow vector (x, y) means that the pixel moves x units to the right and y units downward from one frame to another.

[0059] It should be noted that in the embodiments of the present disclosure, the terms "optical flow vector", "optical flow motion feature", "optical flow feature" and the like can be used interchangeably.

[0060] Optical flow vector increment: The optical flow vector increment is usually used to describe the change of the optical flow vector over time (or between image frames). If we label three consecutive frames of images as A, B, and C, the optical flow vector can describe the movement of pixels from A to B or B to C, and the optical flow vector increment can measure the change of the optical flow vector from A to B and B to C, which can also be understood as the change of the optical flow vector from A to C. If an object in the image moves 3 pixels, 2 pixels, and 1 pixel to the right in three consecutive frames, the optical flow vector between the first two frames and the last two frames is 3 and 2, and the increment of the optical flow vector is -1, indicating that the movement speed of the object is gradually decreasing.

[0061] It should be noted that in the embodiments of the present disclosure, the terms "optical flow vector increment", "optical flow feature increment", "optical flow motion feature increment" and the like can be used interchangeably.

[0062] Optical flow field: The optical flow field refers to the collection of optical flow vectors of all pixels in the entire image or an area. It vividly represents the movement direction and distance of each pixel in the image, thereby revealing the motion information of the image.

[0063] It should be noted that in the embodiments of the present disclosure, the terms "optical flow field", "optical flow motion field", "optical flow motion feature field", "optical flow motion vector field" and the like can be used interchangeably.

[0064] RAW image: In image processing, RAW image refers to unprocessed image data acquired by a digital camera, scanner or image scanning device. RAW images contain all data captured directly from the image sensor without any digital processing or compression, thus providing greater freedom in post-processing.

[0065] YUV: YUV is a color encoding method, especially commonly used in video systems. YUV does not refer to a specific color space, but a type of color space that decomposes color information into brightness (Luma) and chroma (Chroma). "Y" represents the brightness component, which contains black and white information and plays the role of brightness; "U" and "V" represent color difference: "U" is the difference between blue brightness and brightness Y, and "V" is the difference between red brightness and brightness Y, and the two together play the role of chroma.

[0066] Demosaic algorithm: refers to the process of creating a full-color image for a RAW image. This algorithm interpolates and estimates the other color channels at each pixel through the information of the surrounding pixels, thereby obtaining the complete red, green, and blue color information at each pixel. This process is called "demosaicing" or "demosaicing".

[0067] Gamma curve: refers to a nonlinear transformation function used to correct the brightness in an image. Its main purpose is to offset the effect of image display devices (such as computer monitors, TVs, etc.), where the brightness information is not linear when displaying images. Therefore, when processing images or video signals, it is necessary to adjust the brightness by applying gamma correction so that the effect of the image on the display device is closer to what the human eye sees.

[0068] Global homography matrix: The global homography matrix is ​​usually used to describe the planar projection change between two images. This change is represented by a matrix (such as a 3x3 matrix), which can include four basic types of transformations: translation, rotation, scaling, and perspective. The global homography matrix is ​​mainly used in image registration, stereo vision, image mosaicking and other fields to describe and process the geometric changes of images.

[0069] Upsampling: Upsampling is a technique to increase the resolution of an image, usually by increasing the number of pixels in the image. This method is sometimes also called image interpolation. For example, if you have a 100x100 pixel image, upsampling can convert it to a 200x200 pixel image. A common application of upsampling is image scaling. We can use upsampling to enlarge a small image or improve the quality and resolution of a large image. However, it should be noted that upsampling does not increase the information content of the image, because the newly added pixel values ​​are calculated entirely from the existing pixels through interpolation, rather than through new sampling.

[0070] Downsampling: Downsampling is the process of reducing the resolution or number of pixels in an image. This process usually involves a low-pass filter to remove high-frequency noise before reducing the resolution, and then resampling at a lower sampling rate, such as taking one pixel every few pixels, thereby reducing the size of the image. Downsampling is often used in applications where it is necessary to reduce the amount of storage or processing of the image, or to conform the image to a lower resolution. For example, in digital image compression, downsampling can effectively reduce the size of the image file; in multiscale image analysis, by creating a series of downsampled images, the image can be observed and processed from multiple scales.

[0071] Spatial propagation: "Spatial propagation" generally refers to the process of how information or features in an image propagate or diffuse in the spatial structure of the image. This concept often appears in various image processing and computer vision algorithms, especially when it comes to image enhancement, segmentation, denoising, feature extraction and other tasks. Spatial propagation can help us understand and utilize the local continuity and structural properties of images. For images, adjacent pixels usually have similar color and texture features, and the shape or object edges may change continuously in some way. These are all manifestations of the continuity and structural properties of images in space, and these properties can be applied in various image processing tasks through "spatial propagation".

[0072] Search Block: Search Block usually refers to a small image area used to search for a match. This method is often used in fields such as motion estimation and image or video compression.

[0073] Image Block: Image Block refers to the smaller rectangular areas that an image is divided into. These smaller rectangular areas can be overlapping or non-overlapping. Each image block contains relatively few pixels, and its size, shape, and number of pixel values ​​are greatly reduced compared to the original entire image.

[0074] Figure 1is a flowchart of an image processing method according to an exemplary embodiment. Figure 1 As shown, the image processing method is used in a terminal and includes the following steps.

[0075] In step S11, a reference frame image and a moving frame image are determined during the photographing process.

[0076] In step S12, a first registration process is performed on the reference frame image and the moving frame image to obtain a first registered image and a first motion feature.

[0077] In step S13, a second registration process is performed based on the first registered image and the reference frame image to obtain a second motion feature.

[0078] In step S14, image fusion is performed based on the reference frame image, the first registered image, the moving frame image, the first motion feature and the second motion feature to obtain a target image.

[0079] It should be understood that during the photo shooting process, the terminal will record multiple frames of images continuously within a period of time. For the convenience of description, the following content will refer to the multiple frames of images continuously recorded by the terminal within a period of time as candidate images. Among the candidate images, the one that meets the expected requirements is selected as the reference frame, and other image frames except the reference frame can be regarded as moving frames.

[0080] It should be understood that in multi-frame image fusion, a reference frame is a frame used as a comparison or benchmark to adjust other frame images. That is, during the first registration process, the moving frame will be adjusted based on the reference frame as a benchmark.

[0081] It should be understood that the target image can be understood as a denoised image obtained by fusion after registration.

[0082] In one implementation of the embodiment of the present disclosure, the first motion feature may include at least one of the following forms of expression: a first motion vector, a first motion vector field, and the like.

[0083] In one implementation of the embodiment of the present disclosure, the second motion feature may include at least one of the following forms of expression: a second motion vector, a second motion vector field, and the like.

[0084] The image processing method provided by the embodiment of the present disclosure performs registration on the reference frame image and the first registration image in the second registration process to obtain the second motion feature between the two, and fuses the first motion feature and the second motion feature in the image fusion process to obtain the denoised target image. Compared with the prior art, the use of the difference algorithm is reduced, the computational complexity is reduced, and the registration error rate caused by the interpolation calculation in the registration process is reduced, thereby improving the image quality of the denoised image.

[0085] Figure 2 is a flowchart of a method for determining a reference frame image and a moving frame image according to an exemplary embodiment. Figure 2 As shown, in one implementation of the embodiment of the present disclosure, determining a reference frame image and a moving frame image during a photographing process includes the following steps:

[0086] In step S21, an image frame with the highest definition among the candidate images is determined as a reference frame image.

[0087] In step S22, the brightness of the reference frame image is determined, and a moving frame that meets the requirements is determined based on the brightness of the reference frame image.

[0088] In one implementation of the embodiment of the present disclosure, the brightness of the reference frame image may be determined in the following manner, including:

[0089] Dividing the reference frame image into regions to obtain at least one sub-reference frame image;

[0090] Determine the brightness of the sub-reference frame images respectively, and solve the average brightness of the sub-reference frame images based on the brightness of all sub-reference frames;

[0091] The average brightness is determined as the brightness of the reference frame image.

[0092] In one implementation of the embodiment of the present disclosure, determining a moving frame that meets the requirements based on the brightness of a reference frame image includes:

[0093] Determining brightness of other frame images except the reference frame image in the candidate images;

[0094] The images whose brightness difference with the reference frame image is greater than a preset brightness threshold are eliminated, and the remaining images are determined as moving frame images.

[0095] The image processing method provided in the embodiment of the present disclosure can reduce the impact of noise caused by abnormal image exposure on subsequent registration results by screening reference frame images and moving frame images based on brightness, thereby improving the accuracy of the registration results and further improving the final image denoising results.

[0096] In one implementation of the disclosed embodiment, before the first registration processing is performed on the reference frame image and the moving frame image, if it is determined that the input reference frame image and the moving frame image are RAW images, they can also be preprocessed as follows to improve the image contrast, including: channel completion based on Demosaic; averaging the values ​​of the four channels based on the photosensitive array arrangements such as GRBG and RGGB in the Raw image, and then using gamma removal or histogram equalization to improve the contrast of the reference frame image and the contrast of the moving frame image.

[0097] The image processing method provided in the embodiment of the present disclosure can be more conducive to the operation of the registration process, optimize the registration result, and thus improve the quality of the denoised image by preprocessing the input reference frame image and the moving frame image based on the above-mentioned method when the image is a RAW image.

[0098] In one implementation of the embodiment of the present disclosure, the first registration processing method for the reference frame image and the moving frame image includes:

[0099] The motion change between the reference frame and the moving frame is determined based on the global homography matrix.

[0100] Optionally, determining the motion change between the reference frame and the moving frame includes at least one of the following: a translational motion change between the reference frame and the moving frame; or a rotational motion change between the reference frame and the moving frame.

[0101] The image processing method provided in the embodiment of the present disclosure performs the first registration of the reference frame and the moving frame through the global homography matrix, which can realize the initial registration of planes with more (or more obvious) registration feature points, and provide a technical basis for the subsequent registration of planes with fewer (or less obvious) feature points.

[0102] It should be understood that the feature point can be a predefined reference point in the image or a randomly selected point. The setting of the feature point is used to describe the motion change of the corresponding image area with the feature point.

[0103] Figure 3 is a flow chart of a second registration method according to an exemplary embodiment. Figure 3 As shown, in one implementation of the embodiment of the present disclosure, performing a second registration process based on the first registration image and the reference frame image includes the following steps:

[0104] In step S31, the reference frame image is downsampled based on a first sampling ratio to obtain a first reference frame image, and the first registered image is downsampled based on the first sampling ratio to obtain a first sampled image.

[0105] In step S32, a first image pyramid is constructed based on the first reference frame image, and a second image pyramid is constructed based on the first sampling image.

[0106] In step S33, a first optical flow feature is determined based on the N-th reference frame image and the N-th sampled image.

[0107] In step S34, the first optical flow feature is upsampled iteratively to obtain a second motion feature.

[0108] In an implementation manner of the embodiment of the present disclosure, a value range of the first sampling ratio includes: the first sampling ratio is greater than 0 and less than 1.

[0109] In one implementation of the disclosed embodiment, the first image pyramid has N layers, the first reference frame image is the first layer of the first image pyramid, and the N-th frame reference frame image corresponding to the N-th layer is obtained by downsampling the N-1-th frame reference frame image corresponding to the N-1-th layer based on a first sampling ratio, where N is an integer greater than 1.

[0110] For example, taking N equal to 3 and the first sampling ratio as 1 / 2, the first image pyramid has 3 layers, and the 3rd frame reference frame image corresponding to the 3rd layer is obtained by downsampling the 2nd frame reference frame image corresponding to the 2nd layer based on the first sampling ratio. The 2nd frame reference frame image corresponding to the 2nd layer is obtained by downsampling the 1st frame image corresponding to the 1st layer based on the first sampling ratio. For example, the 1st frame image corresponding to the 1st layer is a square with a side length of a and an area of ​​a. 2 (That is, a*a). The second frame image corresponding to the second layer should be a square with a side length of (1 / 2)a and an area of ​​(1 / 4)a. 2 (That is, (1 / 2)a*(1 / 2)a). The third frame image corresponding to the third layer should be a square with a side length of 1 / 2[(1 / 2)a] and an area of ​​(1 / 16)a. 2 (That is, 1 / 2[(1 / 2)a]*1 / 2[(1 / 2)a]).

[0111] In one implementation of the disclosed embodiment, the second image pyramid has N layers, the first frame sampling image corresponds to the first layer of the second image pyramid, and the N-th frame sampling image corresponding to the N-th layer is obtained by downsampling the N-1-th frame sampling image corresponding to the N-1-th layer based on the first sampling ratio, where N is an integer greater than 1.

[0112] It should be understood that the detailed content of the pyramid distribution of the second image is the same as or similar to the detailed content of the first pyramid distribution, and the detailed content of the pyramid distribution of the second image will not be described one by one here.

[0113] In one implementation of the embodiments of the present disclosure, the first optical flow feature is used to characterize the motion characteristics of the optical flow field between the N-th reference frame image and the N-th sampled image.

[0114] The image processing method provided by the embodiment of the present disclosure obtains a scaled image by downsampling the reference frame image and the first registered image. Since the noise in the scaled image is relatively small, the optical flow field motion features are solved in the scaled image, and then the optical flow field motion features are obtained by upsampling iteratively. Compared with directly solving the optical flow field motion features in a large-area image, this can reduce the interference of image noise on the optical flow field motion features and improve the accuracy of the registration results.

[0115] Figure 4 is a flowchart of a first optical flow feature upsampling iterative method according to an exemplary embodiment. Figure 4 As shown, in one implementation of the embodiment of the present disclosure, upsampling the first optical flow feature iteratively to obtain the second motion feature includes the following steps:

[0116] In step S41, an initial value of the first optical flow feature and an increment of the first optical flow feature are determined.

[0117] In step S42, feature fusion is performed on the initial value of the first optical flow feature and the first optical flow feature increment to obtain the first optical flow feature.

[0118] In step S43, the first optical flow feature is upsampled based on the second sampling ratio to obtain a second optical flow sampling feature, and an initial value of the second optical flow feature is determined based on the second optical flow sampling feature.

[0119] In step S44, a second optical flow feature increment is determined based on the N-1th reference frame image and the N-1th sampled image.

[0120] In step S45, feature fusion is performed on the initial value of the second optical flow feature and the second optical flow feature increment to obtain a second motion feature.

[0121] In one implementation of the embodiments of the present disclosure, an initial value of the first optical flow feature and an increment of the first optical flow feature are determined based on the N-th reference frame image and the N-th sampled image.

[0122] It should be understood that, following the above embodiment, the Nth reference frame image and the Nth sampled image are the top layer of the first image pyramid and the top layer of the second image pyramid, respectively, and the first optical flow feature is determined at the top layers of the two pyramids. Therefore, the initial optical flow feature determined based on the Nth reference frame image and the Nth sampled image is identified as the initial value of the first optical flow feature.

[0123] In one implementation of the disclosed embodiment, when the brightness is constant and the spatial consistency meets the requirements, the optical flow feature can be solved by the Lucas-Kanade method, which is based on the least squares method to solve the optical flow vector. The Lucas-Kanade method assumes that in a small window, all pixels have the same motion. Then, it minimizes the sum of the squares of the pixel differences of this window in the first image and the second image to obtain the motion parameter. The specific formula is as follows:

[0124] IxΔx+IyΔy=-It

[0125] Where Ix and Iy are the grayscale gradients of the image in the x and y directions, It is the grayscale change of the image in the time t direction, and Δx and Δy are the optical flow vectors, i.e., displacements. Δx and Δy can be obtained by solving this equation. Furthermore, based on the second-order derivative of Δx and Δy with respect to time t, the increment of the optical flow feature can be obtained.

[0126] In one implementation of the embodiment of the present disclosure, the second upsampling ratio is the inverse of the first downsampling ratio.

[0127] Exemplarily, following the above embodiment, the first downsampling ratio is 1 / 2, and the second upsampling ratio is 2. When the first optical flow feature is vector b, the second optical flow feature is vector 2b.

[0128] In one implementation of the disclosed embodiment, the second optical flow feature increment is solved by the Lucas-Kanade method. For detailed solution content, see the above-mentioned first optical flow feature increment solution content, which will not be repeated here.

[0129] In one implementation of the embodiments of the present disclosure, the number of iterations of the first optical flow feature is the same as the number of layers of the first image pyramid or the number of layers of the second image pyramid.

[0130] It should be understood that the purpose of the upsampling iteration based on the first optical flow feature is to obtain the optical flow feature between the reference frame image and the first registration result image. Therefore, the number of upsampling iterations should be the number of layers of the pyramid. Since the subsequent iterative process is the same or similar to the iterative process of this embodiment, for the sake of convenience, it will not be described one by one here.

[0131] The image processing method provided by the embodiment of the present disclosure obtains a scaled image by downsampling the reference frame image and the first registered image. Since the noise in the scaled image is relatively small, the optical flow field motion features are solved in the scaled image, and then the optical flow field motion features are obtained by upsampling iteratively. Compared with directly solving the optical flow field motion features in a large-area image, this can reduce the interference of image noise on the optical flow field motion features and improve the accuracy of the registration results.

[0132] Figure 5 is a flow chart showing a method for determining an initial value of a first optical flow feature according to an exemplary embodiment. Figure 5 As shown, in one implementation of the embodiment of the present disclosure, the following steps are included:

[0133] In step S51 , a first target sub-region and a first adjacent region are determined based on a first search block.

[0134] In step S52, optical flow features are extracted from the first target sub-region and the first adjacent region respectively to obtain a first target optical flow feature and a first candidate optical flow feature.

[0135] In step S53, a similarity comparison is performed between the first target optical flow feature and the first optical flow feature to obtain a first similarity result, and a similarity comparison is performed between each first candidate optical flow feature and the first optical flow feature to obtain at least one second similarity result.

[0136] In step S54, if the first similarity result is greater than all the second similarity results, the first target optical flow feature is determined to be the second candidate optical flow feature.

[0137] In step S55, if there is at least one second similarity result greater than the first similarity result, the first candidate optical flow feature corresponding to the maximum similarity result in the at least one second similarity result is determined as the second candidate optical flow feature.

[0138] In step S56, the second candidate optical flow feature is screened and compared with the zero vector to determine the initial value of the first optical flow feature.

[0139] In one implementation of the embodiment of the present disclosure, the first adjacent area is an area adjacent to the first target sub-area.

[0140] In one implementation of the embodiments of the present disclosure, a first target sub-region and a first adjacent region of the first target sub-region are determined in the Nth frame sampling image based on the first search block.

[0141] In one implementation of the disclosed embodiment, optical flow features are extracted from the first target sub-region and the first adjacent region respectively to obtain the first target optical flow feature corresponding to the first target sub-region and the first candidate optical flow feature corresponding to each first adjacent region.

[0142] In one implementation of the embodiments of the present disclosure, the similarity result can be solved by at least one of the following methods, including: a sum of squared differences (SSD) algorithm, or a sum of absolute differences (SAD) algorithm.

[0143] It should be understood that, based on the influence of spatial propagation, when confirming the initial value of the optical flow feature vector of the target area, the motion vector of the adjacent pixel area of ​​the target area needs to be considered. By performing similarity comparison with the first optical flow feature respectively, the optical flow feature of the target area is replaced with the optical flow feature closest to the first optical flow feature, which can reduce the error in the registration process.

[0144] Figure 6 is a flowchart of another method for determining an initial value of a first optical flow feature according to an exemplary embodiment. Figure 6 As shown, in one implementation of the embodiment of the present disclosure, determining the initial value of the first optical flow feature includes the following steps:

[0145] In step S61, the magnitude relationship between the metric value of the second candidate optical flow feature and the first metric threshold is determined.

[0146] In step S62, if the metric value of the second candidate optical flow feature is less than or equal to the first metric threshold, the metric value of the second optical flow candidate feature is determined to be the initial value of the first optical flow feature.

[0147] In step S63, if the metric value of the second candidate optical flow feature is greater than the first metric threshold, a similarity comparison is performed between the zero vector and the first optical flow feature to obtain a third similarity result, and the initial value of the first optical flow feature is determined based on the third similarity result.

[0148] In step S64, if the third similarity result is greater than the first similarity result, and the difference between the third similarity result and the first similarity result is greater than the second similarity threshold, the zero vector is determined as the initial value of the first optical flow feature.

[0149] In one implementation of the present disclosure, the metric value of the second candidate optical flow feature may be obtained based on at least one of the following methods, including:

[0150] ||w||1=|u|+|v|

[0151]

[0152] The second candidate optical flow feature can be expressed as w=(u,v), ||w||1 represents the measurement value of the second candidate optical flow feature obtained by solving the first method, and ||w||2 represents the measurement value of the second candidate optical flow feature obtained by solving the second method.

[0153] It should be understood that if the metric value of the second candidate optical flow feature is less than or equal to the first metric threshold, then the metric value of the second optical flow candidate feature is determined to be the initial value of the first optical flow feature, which can eliminate features with larger motion and reduce the risk of interference from abnormal data, thereby improving the accuracy of the optical flow feature calculation results.

[0154] It should be understood that if the third similarity result is greater than the first similarity result, and the difference between the third similarity result and the first similarity result is greater than the second similarity threshold, then determining the zero vector as the initial value of the first optical flow feature can avoid the movement of smaller amplitudes during actual movement being replaced by the zero vector, resulting in missing calculation data for tiny movements.

[0155] Figure 7 is a flow chart showing a method for determining an initial value of a second optical flow feature according to an exemplary embodiment. Figure 7 As shown, in one implementation of the embodiment of the present disclosure, determining the initial value of the second optical flow feature includes the following steps:

[0156] In step S71 , a second target sub-region and a second adjacent region adjacent to the second target sub-region are determined.

[0157] In step S72, optical flow features are extracted from the second target sub-region and the second adjacent region respectively to obtain a second target optical flow feature and a third candidate optical flow feature.

[0158] In step S73, a similarity comparison is performed between the second target optical flow feature and the second optical flow sampling feature to obtain a fourth similarity result, and a similarity comparison is performed between each third candidate optical flow feature and the second optical flow sampling feature to obtain at least one fifth similarity result.

[0159] In step S74, if the fourth similarity result is greater than all the fifth similarity results, the second target optical flow feature is determined to be the fourth candidate optical flow feature.

[0160] In step S75, if there is at least one fifth similarity result greater than the fourth similarity result, the third candidate optical flow feature corresponding to the maximum similarity result in the at least one fifth similarity result is determined as the fourth candidate optical flow feature.

[0161] In step S76, the fourth candidate optical flow feature is screened and compared with the zero vector to determine the initial value of the second optical flow feature.

[0162] In one implementation of the embodiments of the present disclosure, a second target sub-region and a second adjacent region of the second target sub-region are determined in the N-1th frame sampling image based on the second search block.

[0163] In one implementation of the disclosed embodiment, optical flow features are extracted from the second target sub-region and the second adjacent region respectively to obtain the second target optical flow features corresponding to the second target sub-region and the third candidate optical flow features corresponding to each first adjacent region.

[0164] It is understandable that for determining the initial value of the second optical flow feature, the optional implementation of steps S71 to S76 can be found in Figure 5 Optional implementations of steps S51 to S52, and Figure 5 Other related parts in the embodiments involved will not be described in detail here.

[0165] Figure 8 is a flowchart of another method for determining an initial value of a second optical flow feature according to an exemplary embodiment. Figure 8 As shown, in one implementation of the embodiment of the present disclosure, determining the initial value of the second optical flow feature includes the following steps:

[0166] In step S81, the magnitude relationship between the metric value of the fourth candidate optical flow vector and the second metric threshold is determined.

[0167] In step S82, if the metric value of the fourth candidate optical flow feature is less than or equal to the second metric threshold, the metric value of the fourth optical flow candidate feature is determined to be the initial value of the second optical flow feature.

[0168] In step S83, if the metric value of the fourth candidate optical flow feature is greater than the second metric threshold, a similarity comparison is performed between the zero vector and the second optical flow sampling feature to obtain a sixth similarity result, and the initial value of the second optical flow feature is determined based on the sixth similarity result.

[0169] In step S84, if the sixth similarity result is greater than the fourth similarity result, and the difference between the sixth similarity result and the first similarity result is greater than the second similarity threshold, the zero vector is determined as the initial value of the second optical flow feature.

[0170] It is understandable that for determining the initial value of the second optical flow feature, the optional implementation of steps S61 to S64 can refer to Figure 6 Optional implementations of steps S61 to S64, and Figure 6 Other related parts in the embodiments involved will not be described in detail here.

[0171] It should be understood that since the second optical flow feature and the first optical flow feature are obtained based on images of different sizes (the pyramid image of the N-1th layer and the pyramid image of the Nth layer), the second metric threshold and the first metric threshold may be equal, or the lower limit of the value range of the second metric threshold may be greater than the upper limit of the value range of the first metric threshold.

[0172] It should be understood that, based on the above embodiment, the noise of the pyramid image at the Nth layer is less than the noise of the pyramid image at the N-1th layer. Therefore, the second similarity threshold and the first similarity threshold may be equal, or the upper limit of the value range of the second similarity threshold may be less than the lower limit of the value range of the first similarity threshold.

[0173] Fig. 9 is a flow chart of a second registration correction method according to an exemplary embodiment. Fig. 9 As shown, in one implementation of the embodiment of the present disclosure, after obtaining the second motion feature, the following steps are also included:

[0174] In step S91: the first motion feature and the second motion feature are fused to obtain a third motion feature.

[0175] In step S92, the similarity between the first motion feature and the first image block is determined as a seventh similarity result, and the similarity between the second image block and the third motion feature is determined as an eighth similarity result.

[0176] In step S93, if the seventh similarity result is smaller than the eighth similarity result, the second optical flow feature increment is determined.

[0177] In step S94, if the seventh similarity result is greater than or equal to the eighth similarity result, the second motion feature is set to zero and then a second optical flow feature increment is determined.

[0178] In one implementation of the embodiments of the present disclosure, the third motion feature is used to characterize the motion feature between the reference frame image and the moving frame image after the second registration process.

[0179] In one implementation of the embodiment of the present disclosure, the first image block is determined based on the first registered image, and the second image block is determined based on the reference frame image.

[0180] Exemplarily, the image block selection range may be determined based on the search block size in the sparse optical flow, for example, may be 8*8.

[0181] It should be understood that if the seventh similarity result is smaller than the eighth similarity result, it is considered that the result after two registrations is closer to the actual image. Therefore, the optical flow feature increment iteration is performed based on the second optical flow feature corresponding to the third motion feature.

[0182] The third motion feature is considered to be a motion feature after two registrations. Therefore, the iterative process for the first optical flow vector increment has been completed, based on which, the second optical flow feature increment is iterated from the N-1 layer. Of course, it can also be iterated from the first optical flow feature increment starting from the Nth layer according to actual needs.

[0183] It should be understood that if the seventh similarity result is greater than or equal to the eighth similarity result, it is considered that the result after two registrations is more deviated from the actual image (that is, the second motion feature solved by the optical flow field iteration has obvious errors). Based on this, the second motion feature is proposed by setting zero, and the optical flow feature increment iteration is performed again to determine the second motion feature.

[0184] Optionally, the method of setting the second motion feature to zero includes: replacing the first motion feature with the second motion feature.

[0185] In one implementation of the disclosed embodiment, the number of optical flows corresponding to any layer of the first image pyramid is determined based on the ratio of the area of ​​the reference frame image of the current layer to the area of ​​the second search block;

[0186] The number of optical flows corresponding to any layer of the second image pyramid is determined based on the ratio of the area of ​​the sampled image of the current layer to the area of ​​the second search block.

[0187] The image processing method provided by the embodiment of the present disclosure can solve the above-mentioned optical flow characteristics and other factors based on the sparse optical flow algorithm framework. When using the sparse optical flow algorithm, the determination of the number of optical flows can be determined based on the relationship between the current image area and the area of ​​the search block used.

[0188] In one implementation of the disclosed embodiment, when performing non-same-exposure image registration, the long-exposure image blocks to be searched in the large format of the first image pyramid and / or the second image pyramid (the image format corresponding to the layers other than the Nth layer) will appear obviously overexposed. If an 8-bit image block is used for the search, the value in the image block corresponding to the pyramid will be close to 255. Using such an image block in a texture-free area to perform incremental calculation of the optical flow value is likely to cause an unstable numerical solution. Therefore, based on the optical flow vector value at the corresponding position of the previous pyramid layer and the downsampling ratio, the optical flow feature metric value at that position is replaced and calculated.

[0189] For example, take the N-2th layer image in the first image pyramid as an example, where the downsampling ratio of the first image pyramid is a. If the N-2th layer image is overexposed, the optical flow feature metric value of the N-2th layer image should be M / a, where M is the optical flow feature metric value of the N-1th layer image.

[0190] In one implementation of the disclosed embodiment, in response to the sum of the pixel values ​​in the image block being greater than or equal to the first pixel value threshold, the optical flow feature metric value at the position is replaced and calculated based on the optical flow vector value at the corresponding position of the previous pyramid layer and the downsampling ratio.

[0191] In one implementation of the disclosed embodiments, in response to the total amount of gradient change in one of the x direction or the y direction within the image block being less than a first gradient threshold, the optical flow feature metric value at the position is replaced and calculated based on the optical flow vector value at the corresponding position of the previous pyramid layer and the downsampling ratio.

[0192] In one implementation of the embodiments of the present disclosure, performing image fusion based on the reference frame image, the first registered image, the moving frame image, the first motion feature, and the second motion feature to obtain a target image includes:

[0193] If multiple RAW images are to be synthesized, multiple RAW images are directly registered and fused to obtain multiple frames of denoised RAW images, and then passed through the BPS (Bayer processing segment) and IPE (Image processing engine) modules to obtain RGB images.

[0194] If multiple frames of YUV images are synthesized, each frame of YUV passes through the BPS module. After the synthesis of multiple frames, a denoised YUV image will be obtained, and then it passes through the IPE module to obtain an RGB image.

[0195] In some embodiments, BPS: This hardware module processes the Raw image in Bayer format and finally obtains a YUV image.

[0196] IPE: This hardware module processes the YUV image and finally obtains the RGB image.

[0197] For example, Fig. 10A As shown, Fig. 10A This is a schematic flow chart of the synthesis of multiple YUV images with the same exposure. Fig. 10A In the process, each RAW image is passed through the BPS module to obtain a YUV image, and finally multiple YUV images are registered and fused, and the fused RGB image is obtained through the IPE module.

[0198] For example, Fig. 10B As shown, Fig. 10B This is a schematic flow chart for synthesizing multiple RAW images with the same exposure. Fig. 10B In the process, multiple RAW images are first registered and fused to obtain the fused image, and then the corresponding YUV image is obtained through the BPS module, and the YUV image is input into the IPR module to obtain the fused RGB image.

[0199] For example, Fig. 10C As shown, Fig. 10C This is a schematic flow chart of the synthesis of multiple frames of YUV images with different exposures. Fig. 10C In the , for yuv images with different exposure times, registration and fusion can be performed to obtain yuv images with high dynamic range, and finally the fused RGB image is obtained through the IPE module.

[0200] For example, Fig. 10D As shown, Fig. 10D This is a schematic flow chart for synthesizing multiple RAW images with different exposures. Fig. 10D In the , for RAW images with different exposure times, registration and fusion can be performed to obtain RAW images with high dynamic range, and then processed by the BPS module to obtain the corresponding YUV image, and finally the fused RGB image is obtained by the IPE module.

[0201] It should be understood that the registration and fusion process mentioned in the above embodiment can be referred to in Figure 1 Optional implementations of steps S11 to S14, and Figure 1 Other related parts in the embodiments involved will not be described in detail here.

[0202] It should be understood that the fused RGB image in the above embodiment is the target image.

[0203] It is understandable that the embodiments of the present disclosure may be implemented separately or in combination with each other. For example, the user-determined reference frame image and moving frame image may be implemented independently or together with the first registration or second registration operation. The present disclosure only lists the independent implementation methods of the embodiments, and the implementation methods of the combined application of the embodiments shall not be considered to exceed the scope of the technical solutions of the embodiments of the present disclosure.

[0204] Based on the same concept, an embodiment of the present disclosure also provides an image processing device.

[0205] It is understandable that in order to realize the above functions, the image processing device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0206] Fig.11 FIG. 1 is a block diagram of an image processing apparatus 100 according to an exemplary embodiment. Fig.11 The device includes an acquisition unit 101, a registration unit 102 and a processing unit 103.

[0207] An acquisition unit 101 is used to determine a reference frame image and a moving frame image during a photographing process;

[0208] The registration unit 102 is used to perform a first registration process on the reference frame image and the moving frame image to obtain a first registered image and a first motion feature, where the first motion feature is a motion feature between the moving frame image and the first registered image;

[0209] The registration unit 102 is further configured to perform a second registration process based on the first registered image and the reference frame image to obtain a second motion feature, where the second motion feature is a motion feature between the reference frame image and the first registered image;

[0210] The processing unit 103 is used to perform image fusion based on the reference frame image, the first registered image, the moving frame, the first motion feature and the second motion feature to obtain a target image.

[0211] In one implementation of the disclosed embodiment, the registration unit 102 performs a second registration process based on the first registered image and the reference frame image, including: based on a first sampling ratio, downsampling the reference frame image to obtain a first reference frame image, and based on the first sampling ratio, downsampling the first registered image to obtain a first sampling image, wherein the first sampling ratio is greater than 0 and less than 1; based on the first reference frame image, constructing a first image pyramid, the first reference frame image is the first layer of the first image pyramid, the first image pyramid is N layers, and the N-th frame reference frame image corresponding to the N-th layer is composed of the N-1-th frame reference frame image corresponding to the N-1-th layer The first sampling ratio is used for downsampling, and the second image pyramid is constructed based on the first sampling image, the first sampling image is the first layer of the second image pyramid, the second image pyramid is N layers, and the N-th frame sampling image corresponding to the N-th layer is obtained by downsampling the N-1-th frame sampling image corresponding to the N-1-th layer based on the first sampling ratio, wherein N is an integer greater than 1; a first optical flow feature is determined based on the N-th frame reference frame image and the N-th frame sampling image, and the first optical flow feature is used to characterize the motion feature of the optical flow field between the N-th frame reference frame image and the N-frame sampling second image; the first optical flow feature is upsampled iteratively to obtain a second motion feature.

[0212] In one implementation of the disclosed embodiments, the registration unit 102 iteratively upsamples the first optical flow feature to obtain a second motion feature, including: determining an initial value of the first optical flow feature and an increment of the first optical flow feature based on the N-th frame reference frame image and the N-th frame sampling image; performing feature fusion on the initial value of the first optical flow feature and the increment of the first optical flow feature to obtain the first optical flow feature; upsampling the first optical flow feature based on a second sampling ratio to obtain a second optical flow sampling feature, and determining an initial value of the second optical flow feature based on the second optical flow sampling feature, the second sampling ratio being the inverse of the first sampling ratio; determining a second optical flow feature increment based on the N-1-th frame reference frame image and the N-1-th frame sampling image; performing feature fusion on the initial value of the second optical flow feature and the increment of the second optical flow feature to obtain a second motion feature.

[0213] In one implementation of the disclosed embodiment, the registration unit 102 determines the initial value of the first optical flow feature, including: determining a first target sub-region and a first adjacent region of the first target sub-region in the Nth frame sampling image based on the first search block; extracting optical flow features from the first target sub-region and the first adjacent region respectively to obtain a first target optical flow feature corresponding to the first target sub-region and a first candidate optical flow feature corresponding to each first adjacent region; performing a similarity comparison between the first target optical flow feature and the first optical flow feature to obtain a first similarity result, and performing a similarity comparison between each first candidate optical flow feature and the first optical flow feature to obtain at least one second similarity result; determining the first target optical flow feature as the second candidate optical flow feature when the first similarity result is greater than all the second similarity results; determining the first candidate optical flow feature corresponding to the maximum value of the similarity result in the at least one second similarity result as the second candidate optical flow feature when there is at least one second similarity result greater than the first similarity result; and screening and comparing the second candidate optical flow feature with the zero vector to determine the initial value of the first optical flow feature.

[0214] In one implementation of the embodiments of the present disclosure, the registration unit 102 screens and compares the second candidate optical flow feature with the zero vector to determine the initial value of the first optical flow feature, including: when the measurement value of the second candidate optical flow feature is less than or equal to the first measurement threshold, determining the measurement value of the second optical flow candidate feature as the initial value of the first optical flow feature; when the measurement value of the second candidate optical flow feature is greater than the first measurement threshold, performing a similarity comparison between the zero vector and the first optical flow feature to obtain a third similarity result, and determining the initial value of the first optical flow feature based on the third similarity result, including: when the third similarity result is greater than the first similarity result, and the difference between the third similarity result and the first similarity result is greater than the first similarity threshold, determining the zero vector as the initial value of the first optical flow feature.

[0215] In one implementation of the disclosed embodiment, the registration unit 102 determines the initial value of the second optical flow feature, including: determining a second target sub-region and a second adjacent region adjacent to the second target sub-region in the N-1 frame sampling image based on the second search block; extracting optical flow features from the second target sub-region and the second adjacent region respectively to obtain a second target optical flow feature corresponding to the second target sub-region and a third candidate optical flow feature corresponding to each second adjacent region; performing a similarity comparison between the second target optical flow feature and the second optical flow sampling feature to obtain a fourth similarity result, and performing a similarity comparison between each third candidate optical flow feature and the second optical flow sampling feature to obtain at least one fifth similarity result; when the fourth similarity result is greater than all the fifth similarity results, determining the second target optical flow feature as the fourth candidate optical flow feature; when there is at least one fifth similarity result greater than the fourth similarity result, determining the third candidate optical flow feature corresponding to the maximum value of the similarity result in the at least one fifth similarity result as the fourth candidate optical flow feature; and screening and comparing the fourth candidate optical flow feature with the zero vector to determine the initial value of the second optical flow feature.

[0216] In one implementation of the embodiments of the present disclosure, the registration unit 102 screens and compares the fourth candidate optical flow feature with the zero vector to determine the initial value of the second optical flow feature, including: when the measurement value of the fourth candidate optical flow feature is less than or equal to the second measurement threshold, determining the measurement value of the fourth candidate optical flow feature as the initial value of the second optical flow feature; when the measurement value of the fourth candidate optical flow feature is greater than the second measurement threshold, performing a similarity comparison between the zero vector and the second optical flow sampling feature to obtain a sixth similarity result, and determining the initial value of the second optical flow feature based on the sixth similarity result, including: when the sixth similarity result is greater than the fourth similarity result, and the difference between the sixth similarity result and the fourth similarity result is greater than the second similarity threshold, determining the zero vector as the initial value of the second optical flow feature.

[0217] In one implementation of the disclosed embodiment, after the registration unit 102 obtains the second motion feature, it also includes: performing feature fusion on the first motion feature and the second motion feature to obtain a third motion feature, the third motion feature being used to characterize the motion feature between the reference frame image and the moving frame image after the second registration process; determining the similarity of the first motion feature corresponding to the first image block as a fourth similarity result, and determining the similarity of the second image block corresponding to the third motion feature as a fifth similarity result, the first image block being determined based on the first registration image, and the second image block being determined based on the reference frame image; when the fourth similarity result is less than the fifth similarity result, determining a second optical flow feature increment; when the fourth similarity result is greater than or equal to the fifth similarity result, after setting the second motion feature to zero, determining a second optical flow feature increment.

[0218] In one implementation of the disclosed embodiment, the processing unit 103 is used to: determine the number of optical flows corresponding to the first image pyramid of any layer based on the ratio of the reference frame image area of ​​the current layer to the area of ​​the second search block;

[0219] The number of optical flows corresponding to any layer of the second image pyramid is determined based on the ratio of the area of ​​the sampled image of the current layer to the area of ​​the second search block.

[0220] Fig.12 2 is a block diagram of an apparatus 200 for image processing according to an exemplary embodiment. For example, the apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0221] Reference Fig.12 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0222] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0223] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0224] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0225] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0226] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0227] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0228] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0229] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0230] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0231] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0232] Fig.13 is a block diagram of a device 300 for image processing according to an exemplary embodiment. For example, the device 300 may be provided as a server. Fig.13 The apparatus 300 includes a processing component 322, which further includes one or more processors, and a memory resource represented by a memory 332 for storing instructions executable by the processing component 322, such as an application. The application stored in the memory 332 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 322 is configured to execute instructions to perform the above-mentioned screen display method.

[0233] The device 300 may also include a power supply component 326 configured to perform power management of the device 300, a wired or wireless network interface 350 configured to connect the device 300 to a network, and an input / output (I / O) interface 358. The device 300 may operate based on an operating system stored in the memory 332, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0234] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0235] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0236] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.

[0237] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0238] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0239] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0240] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. An image processing method, characterized in that: include: Determine a reference frame image and a moving frame image during the photographing process; Performing a first registration process on the reference frame image and the moving frame image to obtain a first registered image and a first motion feature, where the first motion feature is a motion feature between the moving frame image and the first registered image; Performing a second registration process based on the first registered image and the reference frame image to obtain a second motion feature, where the second motion feature is a motion feature between the reference frame image and the first registered image; Image fusion is performed based on the reference frame image, the first registered image, the moving frame image, the first motion feature, and the second motion feature to obtain a target image.

2. The image processing method according to claim 1, characterized in that: The performing a second registration process based on the first registered image and the reference frame image includes: Based on a first sampling ratio, downsampling the reference frame image to obtain a first reference frame image, and based on the first sampling ratio, downsampling the first registered image to obtain a first sampled image, wherein the first sampling ratio is greater than 0 and less than 1; Based on the first reference frame image, construct a first image pyramid, the first reference frame image is the first layer of the first image pyramid, the first image pyramid has N layers, the N-th frame reference frame image corresponding to the N-th layer is obtained by downsampling the N-1-th frame reference frame image corresponding to the N-1-th layer based on the first sampling ratio, and based on the first sampling image, construct a second image pyramid, the first sampling image is the first layer of the second image pyramid, the second image pyramid has N layers, the N-th frame sampling image corresponding to the N-th layer is obtained by downsampling the N-1-th frame sampling image corresponding to the N-1-th layer based on the first sampling ratio, wherein N is an integer greater than 1; Determine a first optical flow feature based on the N-th reference frame image and the N-th sampled image, where the first optical flow feature is used to characterize a motion feature of an optical flow field between the N-th reference frame image and the N-th sampled image; The first optical flow feature is upsampled iteratively to obtain a second motion feature.

3. The image processing method according to claim 2, characterized in that: The iterative upsampling of the first optical flow feature to obtain a second motion feature includes: Determine an initial value of the first optical flow feature and a first optical flow feature increment based on the N-th reference frame image and the N-th sampled image; Performing feature fusion on the initial value of the first optical flow feature and the first optical flow feature increment to obtain the first optical flow feature; Upsampling the first optical flow feature based on a second sampling ratio to obtain a second optical flow sampling feature, and determining an initial value of the second optical flow feature based on the second optical flow sampling feature, where the second sampling ratio is the reciprocal of the first sampling ratio; Determine a second optical flow feature increment based on the N-1th reference frame image and the N-1th sampled image; The initial value of the second optical flow feature and the increment of the second optical flow feature are subjected to feature fusion to obtain the second motion feature.

4. The image processing method according to claim 3, characterized in that: Determining an initial value of the first optical flow feature includes: Determine a first target sub-region and a first adjacent region of the first target sub-region in the Nth frame sample image based on the first search block; Extracting optical flow features from the first target sub-region and the first adjacent region respectively to obtain a first target optical flow feature corresponding to the first target sub-region and a first candidate optical flow feature corresponding to each first adjacent region; Performing a similarity comparison between the first target optical flow feature and the first optical flow feature to obtain a first similarity result, and performing a similarity comparison between each first candidate optical flow feature and the first optical flow feature to obtain at least one second similarity result; If the first similarity result is greater than all the second similarity results, determining the first target optical flow feature as the second candidate optical flow feature; If there is at least one second similarity result greater than the first similarity result, determining the first candidate optical flow feature corresponding to the maximum similarity result in the at least one second similarity result as the second candidate optical flow feature; The second candidate optical flow feature is screened and compared with the zero vector to determine an initial value of the first optical flow feature.

5. The image processing method according to claim 4, characterized in that: The screening and comparing the second candidate optical flow feature with the zero vector to determine the initial value of the first optical flow feature includes: If the metric value of the second candidate optical flow feature is less than or equal to the first metric threshold, determining the metric value of the second optical flow candidate feature as the initial value of the first optical flow feature; If the metric value of the second candidate optical flow feature is greater than the first metric threshold, performing a similarity comparison between the zero vector and the first optical flow feature to obtain a third similarity result, and determining an initial value of the first optical flow feature based on the third similarity result, including: If the third similarity result is greater than the first similarity result, and a difference between the third similarity result and the first similarity result is greater than a first similarity threshold, a zero vector is determined as an initial value of the first optical flow feature.

6. The image processing method according to claim 3, characterized in that: The determining an initial value of the second optical flow feature comprises: Determine a second target sub-region and a second adjacent region adjacent to the second target sub-region in the N-1th frame sampling image based on the second search block; Extracting optical flow features from the second target sub-region and the second adjacent region respectively to obtain a second target optical flow feature corresponding to the second target sub-region and a third candidate optical flow feature corresponding to each second adjacent region; Performing a similarity comparison between the second target optical flow feature and the second optical flow sampling feature to obtain a fourth similarity result, and performing a similarity comparison between each third candidate optical flow feature and the second optical flow sampling feature to obtain at least one fifth similarity result; If the fourth similarity result is greater than all the fifth similarity results, determining the second target optical flow feature as the fourth candidate optical flow feature; If there is at least one fifth similarity result greater than the fourth similarity result, determining the third candidate optical flow feature corresponding to the maximum similarity result in the at least one fifth similarity result as the fourth candidate optical flow feature; The fourth candidate optical flow feature is screened and compared with the zero vector to determine an initial value of the second optical flow feature.

7. The image processing method according to claim 6, characterized in that: The screening and comparing the fourth candidate optical flow feature with the zero vector to determine the initial value of the second optical flow feature includes: If the metric value of the fourth candidate optical flow feature is less than or equal to the second metric threshold, determining the metric value of the fourth candidate optical flow feature as the initial value of the second optical flow feature; If the metric value of the fourth candidate optical flow feature is greater than the second metric threshold, performing a similarity comparison between the zero vector and the second optical flow sampling feature to obtain a sixth similarity result, and determining an initial value of the second optical flow feature based on the sixth similarity result, including: If the sixth similarity result is greater than the fourth similarity result, and a difference between the sixth similarity result and the fourth similarity result is greater than a second similarity threshold, a zero vector is determined as an initial value of the second optical flow feature.

8. The image processing method according to claim 1, characterized in that: After obtaining the second motion feature, the method further includes: Performing feature fusion on the first motion feature and the second motion feature to obtain a third motion feature, wherein the third motion feature is used to characterize the motion feature between the reference frame image and the moving frame image after the second registration process; Determine a similarity between the first motion feature and a first image block as a seventh similarity result, and determine a similarity between the second image block and the third motion feature as an eighth similarity result, wherein the first image block is determined based on the first registered image, and the second image block is determined based on the reference frame image; If the seventh similarity result is less than the eighth similarity result, determining a second optical flow feature increment; If the seventh similarity result is greater than or equal to the eighth similarity result, the second motion feature is set to zero and then the second optical flow feature increment is determined.

9. The image processing method according to claim 2, characterized in that: The number of optical flows corresponding to the first image pyramid at any level is determined based on the ratio of the area of ​​the reference frame image of the current level to the area of ​​the second search block; The number of optical flows corresponding to any layer of the second image pyramid is determined based on the ratio of the area of ​​the sampled image of the current layer to the area of ​​the second search block.

10. An image processing device, characterized in that: include: An acquisition unit, used to determine a reference frame image and a moving frame image during a photographing process; a registration unit, configured to perform a first registration process on the reference frame image and the moving frame image to obtain a first registered image and a first motion feature, wherein the first motion feature is a motion feature between the moving frame image and the first registered image; The registration unit is further configured to perform a second registration process based on the first registered image and the reference frame image to obtain a second motion feature, where the second motion feature is a motion feature between the reference frame image and the first registered image; A processing unit is used to perform image fusion based on the reference frame image, the first registered image, the moving frame, the first motion feature and the second motion feature to obtain a target image.

11. The image processing device according to claim 10, characterized in that: The registration unit performs a second registration process based on the first registration image and the reference frame image, including: Based on a first sampling ratio, downsampling the reference frame image to obtain a first reference frame image, and based on the first sampling ratio, downsampling the first registered image to obtain a first sampled image, wherein the first sampling ratio is greater than 0 and less than 1; Based on the first reference frame image, construct a first image pyramid, the first reference frame image is the first layer of the first image pyramid, the first image pyramid has N layers, the N-th frame reference frame image corresponding to the N-th layer is obtained by downsampling the N-1-th frame reference frame image corresponding to the N-1-th layer based on the first sampling ratio, and based on the first sampling image, construct a second image pyramid, the first sampling image is the first layer of the second image pyramid, the second image pyramid has N layers, the N-th frame sampling image corresponding to the N-th layer is obtained by downsampling the N-1-th frame sampling image corresponding to the N-1-th layer based on the first sampling ratio, wherein N is an integer greater than 1; Determine a first optical flow feature based on the N-th reference frame image and the N-th sampled image, where the first optical flow feature is used to characterize a motion feature of an optical flow field between the N-th reference frame image and the N-th sampled second image; The first optical flow feature is upsampled iteratively to obtain a second motion feature.

12. The image processing device according to claim 11, characterized in that: The registration unit determines to perform upsampling iteration on the first optical flow feature to obtain a second motion feature, including: Determine an initial value of the first optical flow feature and a first optical flow feature increment based on the N-th reference frame image and the N-th sampled image; Performing feature fusion on the initial value of the first optical flow feature and the first optical flow feature increment to obtain the first optical flow feature; Upsampling the first optical flow feature based on a second sampling ratio to obtain a second optical flow sampling feature, and determining an initial value of the second optical flow feature based on the second optical flow sampling feature, where the second sampling ratio is the reciprocal of the first sampling ratio; Determine a second optical flow feature increment based on the N-1th reference frame image and the N-1th sampled image; The initial value of the second optical flow feature and the increment of the second optical flow feature are subjected to feature fusion to obtain the second motion feature.

13. The image processing device according to claim 12, wherein the registration unit determines the initial value of the first optical flow feature, comprising: Determine a first target sub-region and a first adjacent region of the first target sub-region in the Nth frame sample image based on the first search block; Extracting optical flow features from the first target sub-region and the first adjacent region respectively to obtain a first target optical flow feature corresponding to the first target sub-region and a first candidate optical flow feature corresponding to each first adjacent region; Performing a similarity comparison between the first target optical flow feature and the first optical flow feature to obtain a first similarity result, and performing a similarity comparison between each first candidate optical flow feature and the first optical flow feature to obtain at least one second similarity result; In a case where the first similarity result is greater than all of the second similarity results, determining the first target optical flow feature as a second candidate optical flow feature; In a case where there is at least one second similarity result greater than the first similarity result, determining a first candidate optical flow feature corresponding to a maximum similarity result among the at least one second similarity result as a second candidate optical flow feature; The second candidate optical flow feature is screened and compared with the zero vector to determine an initial value of the first optical flow feature.

14. The image processing device according to claim 13, characterized in that: The registration unit screens and compares the second candidate optical flow feature with a zero vector to determine an initial value of the first optical flow feature, including: When the metric value of the second candidate optical flow feature is less than or equal to the first metric threshold, determining the metric value of the second optical flow candidate feature as the initial value of the first optical flow feature; When the metric value of the second candidate optical flow feature is greater than the first metric threshold, performing a similarity comparison between the zero vector and the first optical flow feature to obtain a third similarity result, and determining an initial value of the first optical flow feature based on the third similarity result, including: When the third similarity result is greater than the first similarity result and a difference between the third similarity result and the first similarity result is greater than a first similarity threshold, a zero vector is determined as an initial value of the first optical flow feature.

15. The image processing device according to claim 12, characterized in that: The registration unit determines an initial value of the second optical flow feature, including: Determine a second target sub-region and a second adjacent region adjacent to the second target sub-region in the N-1th frame sampling image based on the second search block; Extracting optical flow features from the second target sub-region and the second adjacent region respectively to obtain a second target optical flow feature corresponding to the second target sub-region and a third candidate optical flow feature corresponding to each second adjacent region; Performing a similarity comparison between the second target optical flow feature and the second optical flow sampling feature to obtain a fourth similarity result, and performing a similarity comparison between each third candidate optical flow feature and the second optical flow sampling feature to obtain at least one fifth similarity result; In a case where the fourth similarity result is greater than all the fifth similarity results, determining the second target optical flow feature as a fourth candidate optical flow feature; In a case where the at least one fifth similarity result is greater than the fourth similarity result, determining the third candidate optical flow feature corresponding to the maximum similarity result in the at least one fifth similarity result as the fourth candidate optical flow feature; The fourth candidate optical flow feature is screened and compared with the zero vector to determine an initial value of the second optical flow feature.

16. The image processing device according to claim 15, characterized in that: The registration unit screens and compares the fourth candidate optical flow feature with a zero vector to determine an initial value of the second optical flow feature, including: In a case where the metric value of the fourth candidate optical flow feature is less than or equal to the second metric threshold, determining the metric value of the fourth candidate optical flow feature as an initial value of the second optical flow feature; When the metric value of the fourth candidate optical flow feature is greater than the second metric threshold, performing a similarity comparison between the zero vector and the second optical flow sampling feature to obtain a sixth similarity result, and determining an initial value of the second optical flow feature based on the sixth similarity result, including: When the sixth similarity result is greater than the fourth similarity result and a difference between the sixth similarity result and the fourth similarity result is greater than a second similarity threshold, a zero vector is determined as an initial value of the second optical flow feature.

17. The image processing device according to claim 10, characterized in that: After the registration unit is aligned to the second motion feature, it also includes: Performing feature fusion on the first motion feature and the second motion feature to obtain a third motion feature, wherein the third motion feature is used to characterize the motion feature between the reference frame image and the moving frame image after the second registration process; Determine a similarity between the first motion feature and a first image block as a fourth similarity result, and determine a similarity between the second image block and the third motion feature as a fifth similarity result, wherein the first image block is determined based on the first registered image, and the second image block is determined based on the reference frame image; When the fourth similarity result is less than the fifth similarity result, determining a second optical flow feature increment; When the fourth similarity result is greater than or equal to the fifth similarity result, the second motion feature is set to zero and then the second optical flow feature increment is determined.

18. The image processing device according to claim 11, characterized in that: The processing unit is used for: Determine the number of optical flows corresponding to the first image pyramid of any layer based on the ratio of the area of ​​the reference frame image of the current layer to the area of ​​the second search block; The number of optical flows corresponding to any layer of the second image pyramid is determined based on the ratio of the area of ​​the sampled image of the current layer to the area of ​​the second search block.

19. An image processing device, characterized in that: include: a memory for storing processor executable instructions; Wherein, the processor is configured to: execute the image processing method described in any one of claims 1 to 9.

20. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the image processing method according to any one of claims 1 to 9.

Images