Image processing method, device and equipment

Abstract

The embodiments of the present application disclose an image processing method, an apparatus and a device, wherein the method comprises: obtaining feature information of a first area in a current image frame; wherein the first area comprises an area in the current image frame determined by performing motion estimation based on an optical flow method on the current image frame and a previous image frame; obtaining feature information of a second area in the current image frame; wherein the second area comprises an area corresponding to pixels whose correlation relationship between a pixel point among a plurality of first pixels of the current image frame and a pixel point among a plurality of second pixels of the previous image frame meets a condition; based on the feature information of the first area and the feature information of the second area, fusing the previous image frame and the current image frame to obtain a processed current image frame; wherein the processed current image frame is used as a previous image frame of a next image frame to process the next image frame.

Description

Technical Field

[0001] The embodiments of the present application relate to electronic technology, and relate to but are not limited to image processing methods, devices and equipment. Background Art

[0002] At present, the difficulty of multi-frame denoising algorithm is that when the global motion of the image acquisition module such as the camera and the local motion of the moving object occur at the same time, for example, when a handheld camera is moved and there is a moving object such as a car passing by, the amount of motion of the two frames before and after the captured image in the time domain varies greatly, making it difficult to detect the correlation between adjacent image frames, and thus it is difficult to accurately calculate the motion information of the object between adjacent image frames. Furthermore, in the process of multi-frame denoising, the accuracy of motion vector detection decreases, resulting in poor inter-frame alignment, so that ghosting, increased noise or blurring appear after the two frames are fused; it can be seen that in the related technology, in the process of capturing video images, when the global motion of the image acquisition device and the local motion of the moving object occur at the same time, the denoising effect of the video multi-frame is poor. Summary of the invention

[0003] Embodiments of the present application provide image processing methods, devices and equipment.

[0004] In a first aspect, an embodiment of the present application provides an image processing method for obtaining feature information of a first region in a current image frame; wherein the first region includes a region in the current image frame determined by performing motion estimation based on an optical flow method on the current image frame and a previous image frame;

[0005] Obtaining feature information of a second area in the current image frame; wherein the second area includes an area corresponding to a pixel point whose association relationship between a pixel point among a plurality of first pixel points in the current image frame and a pixel point among a plurality of second pixel points in the previous image frame meets a condition;

[0006] Based on the feature information of the first area and the feature information of the second area, the previous image frame and the current image frame are fused to obtain a processed current image frame; wherein the processed current image frame is used as the previous image frame of a next image frame, and the next image frame is processed.

[0007] In a second aspect, an embodiment of the present application provides an image processing device, comprising: a first obtaining module, configured to: obtain feature information of a first region in a current image frame; wherein the first region includes a region in the current image frame determined by performing motion estimation based on an optical flow method on the current image frame and a previous image frame;

[0008] A second obtaining module is configured to: obtain feature information of a second area in the current image frame; wherein the second area includes an area corresponding to a pixel point whose association relationship between a pixel point among the multiple first pixel points of the current image frame and a pixel point among the multiple second pixel points of the previous image frame meets a condition;

[0009] The processing module is configured to: based on the feature information of the first area and the feature information of the second area, fuse the previous image frame and the current image frame to obtain a processed current image frame; wherein the processed current image frame is used as the previous image frame of a next image frame to process the next image frame.

[0010] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the image processing method in the first aspect are implemented.

[0011] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the image processing method in the first aspect.

[0012] The embodiment of the present application provides an image processing method, apparatus and device, which obtains feature information of a first area in a current image frame; wherein the first area includes an area in the current image frame determined by performing motion estimation based on an optical flow method on the current image frame and a previous image frame; obtains feature information of a second area in the current image frame; wherein the second area includes an area corresponding to a pixel point whose association relationship between a pixel point among a plurality of first pixel points of the current image frame and a pixel point among a plurality of second pixel points of the previous image frame meets a condition; here, the feature information of the first area and the feature information of the second area are used to characterize the correlation between adjacent image frames, and further, based on the feature information of the first area and the feature information of the second area , the previous image frame and the current image frame are fused to obtain the processed current image frame; in this way, in the process of fusion processing, based on the feature information of the first area and the feature information of the second area, not only the motion information of the object between adjacent image frames can be accurately calculated, but also the accuracy of frame alignment is improved, thereby avoiding the occurrence of ghosting in fusion; at the same time, the processed current image frame is used as the previous image frame of the next image frame to process the next image frame; in this way, the problem of poor noise reduction effect of multiple video frames when the global motion of the image acquisition device and the local motion of the moving object occur simultaneously in the process of capturing video images in the related technology is solved, the noise reduction effect of multiple video frames is improved, and the video quality is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of the implementation flow of another image processing method provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of a process for fusing a previous image frame and a current image frame provided in an embodiment of the present application;

[0016] Figure 4 A schematic diagram of an implementation flow of another image processing method provided in an embodiment of the present application;

[0017] Figure 5 A schematic diagram of determining the characteristic information of an exemplary first region provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of determining the characteristic information of the second region provided in an exemplary embodiment of the present application;

[0019] Figure 7 A schematic diagram of an implementation flow of an image processing method provided in another embodiment of the present application;

[0020] Figure 8 A schematic diagram of an implementation flow of another image processing method provided in another embodiment of the present application;

[0021] Fig. 9 A schematic diagram of an exemplary area corresponding to a first parameter and an area corresponding to a second parameter provided in an embodiment of the present application;

[0022] Fig.10 Schematic diagram of exemplary images before and after fusion provided in the embodiment of the present application Figure 1 ;

[0023] Fig.11 Schematic diagram of exemplary images before and after fusion provided in the embodiment of the present application Figure 2 ;

[0024] Fig.12 A schematic diagram of fusing adjacent frames based on an image processing method in related art;

[0025] Fig.13 This is a schematic diagram of the fusion of adjacent frames based on the image processing method implemented in this application;

[0026] Fig.14 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0027] Fig.15A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0030] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] It should be pointed out that the terms "first\second\third" involved in the embodiments of the present application are merely to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0032] In the related art, in the process of multi-frame video denoising, the motion vector is first detected between the two consecutive frames, the previous frame is the image frame after denoising, and the current frame is the image frame with noise, and then the two frames are aligned through image twisting (warping) and other processes to minimize the difference in motion vectors between the two frames; then the two frames are fused to achieve denoising of the current frame. However, in the process of multi-frame video denoising, when the global motion of the image acquisition module and the local motion of the moving object occur simultaneously, such as when a handheld image acquisition module moves while a moving object such as a car passes by, the amount of motion of the two frames in the time domain varies greatly, the illumination varies greatly or there is motion occlusion, etc., which makes it difficult to detect the correlation between adjacent frames, and thus it is difficult to accurately calculate the motion information of objects between adjacent frames. The decrease in the accuracy of motion vector detection leads to poor inter-frame alignment, so that after the two frames are fused, problems such as ghosting, increased noise or blurring appear, which affects the final denoising effect.

[0033] The embodiment of the present application provides an image processing method, which is applied to an electronic device, referring to Figure 1 As shown, the method comprises the following steps:

[0034] Step 101: Obtain feature information of a first region in a current image frame.

[0035] The first area includes an area in the current image frame determined by performing motion estimation based on the optical flow method on the current image frame and the previous image frame.

[0036] In an embodiment of the present application, the electronic device may include mobile terminal devices such as mobile phones, tablet computers, laptop computers, personal digital assistants (PDAs), cameras, wearable devices, and fixed terminal devices such as desktop computers.

[0037] In some embodiments, the electronic device may include an image acquisition module, and the video image is acquired through the image acquisition module, and the video image includes multiple image frames, and then the multiple image frames are processed. Of course, the electronic device can also establish a communication connection with the image acquisition module to obtain the video image acquired by the image acquisition module, and then process the multiple image frames contained in the video image. Among them, the feature information of the first area is used to characterize the correlation between the two adjacent image frames, the current image frame and the previous image frame.

[0038] Step 102: Obtain feature information of a second region in the current image frame.

[0039] The second area includes an area corresponding to pixels whose association relationship between a pixel in a plurality of first pixels in the current image frame and a pixel in a plurality of second pixels in the previous image frame meets the condition.

[0040] The feature information of the second region is used to characterize the correlation between two adjacent image frames, the current image frame and the previous image frame. There may be no overlap between the first region and the second region, or there may be a partial overlap between the first region and the second region.

[0041] Step 103: Based on the feature information of the first region and the feature information of the second region, a fusion process is performed on the previous image frame and the current image frame to obtain a processed current image frame.

[0042] The processed current image frame is used as a previous image frame of the next image frame, and the next image frame is processed.

[0043] In an embodiment of the present application, after obtaining the characteristic information of the first area and the characteristic information of the second area, the electronic device can fuse the previous image frame and the current image frame based on the characteristic information of the first area and the characteristic information of the second area, that is, the correlation between adjacent image frames, to obtain a processed current image frame.

[0044] The image processing method provided by the embodiment of the present application obtains feature information of a first area in a current image frame; wherein the first area includes an area in the current image frame determined by performing motion estimation based on the optical flow method on the current image frame and the previous image frame; obtains feature information of a second area in the current image frame; wherein the second area includes an area corresponding to pixel points whose correlation relationship between a pixel point among a plurality of first pixel points of the current image frame and a pixel point among a plurality of second pixel points of the previous image frame meets a condition; here, the feature information of the first area and the feature information of the second area are used to characterize the correlation between adjacent image frames, and further, based on the feature information of the first area and the feature information of the second area, the feature information of the previous image frame is The image frame and the current image frame are fused to obtain a processed current image frame; in this way, in the process of fusion processing, based on the feature information of the first area and the feature information of the second area, not only the motion information of the object between adjacent image frames can be accurately calculated, but also the accuracy of frame alignment is improved, thereby avoiding ghosting in fusion; at the same time, the processed current image frame is used as the previous image frame of the next image frame to process the next image frame; in this way, the problem of poor noise reduction effect of multiple video frames when the global motion of the image acquisition device and the local motion of the moving object occur simultaneously in the process of capturing video images in the related technology is solved, the noise reduction effect of multiple video frames is improved, and the video quality is improved.

[0045] The embodiment of the present application provides an image processing method, which is applied to an electronic device, referring to Figure 2 As shown, the method comprises the following steps:

[0046] Step 201: Obtain feature information of a first region in a current image frame.

[0047] The first area includes an area in the current image frame determined by performing motion estimation based on the optical flow method on the current image frame and the previous image frame.

[0048] Step 202: Obtain feature information of a second region in the current image frame.

[0049] The second area includes an area corresponding to pixels whose association relationship between a pixel in a plurality of first pixels in the current image frame and a pixel in a plurality of second pixels in the previous image frame meets the condition.

[0050] In some embodiments of the present application, the feature information of the first region is used to identify the first region, and the feature information of the second region is used to identify the second region. That is, after the electronic device obtains the feature information for identifying the first region and the feature information for identifying the second region, it can perform noise reduction processing on multiple frames based on the feature information to obtain a high-quality image.

[0051] Step 203: Determine a third region and a fourth region in the current image frame based on the feature information of the first region and the feature information of the second region.

[0052] The third region represents a region where a local motion occurs in the current image frame relative to the previous image frame, and the fourth region represents a region where a global motion occurs in the current image frame relative to the previous image frame.

[0053] Step 204: Obtain a first parameter corresponding to the third region.

[0054] Here, the first parameter is used to reduce the ratio of the previous image frame when two frames are fused, and can also be understood as being used to increase the ratio of the current image frame when two frames are fused.

[0055] Step 205: Obtain a second parameter corresponding to a fifth region in a previous image frame that is associated with the fourth region.

[0056] Here, the second parameter is used to increase the ratio of the previous image frame when two frames are fused, and can also be understood as being used to reduce the ratio of the current image frame when two frames are fused.

[0057] Step 206: Based on the first parameter and the second parameter, the previous image frame and the current image frame are fused to obtain a processed current image frame.

[0058] The processed current image frame is used as a previous image frame of the next image frame, and the next image frame is processed.

[0059] In some embodiments of the present application, the pixel value of the third area of ​​the current target image frame is greater than the pixel value of the third area of ​​the current image frame, and / or the pixel value of the fifth area of ​​the previous target image frame is greater than the pixel value of the fifth area of ​​the previous image frame.

[0060] In the present application examples, see Figure 3 As shown, step 206 performs fusion processing on the previous image frame and the current image frame based on the first parameter and the second parameter to obtain the processed current image frame, including:

[0061] Step 206a: Adjust the pixel values ​​of the third area of ​​the current image frame based on the first parameter to obtain the current target image frame.

[0062] Here, the electronic device adjusts the pixel value of the third area of ​​the current image frame based on the first parameter, and may increase the pixel value of the third area of ​​the current image frame based on the first parameter to obtain the current target image frame.

[0063] Step 206b: adjust the pixel values ​​of the fifth area of ​​the previous image frame based on the second parameter to obtain a previous target image frame.

[0064] Here, the electronic device adjusts the pixel value of the fifth area of ​​the previous image frame based on the second parameter, and may increase the pixel value of the fifth area of ​​the previous image frame based on the second parameter to obtain the previous target image frame.

[0065] Step 206c: perform fusion processing on the current target image frame and the previous target image frame to obtain a processed current image frame.

[0066] Here, the electronic device performs fusion processing on the current target image frame and the previous target image frame, and may average two pixel values ​​of pixels in the current target image frame and the previous target image frame that have a one-to-one correspondence to obtain the processed current image frame.

[0067] For example, see Figure 4 As shown, first, the electronic device calculates the optical flow result from the previous image frame to the current image frame based on the previous image frame and the current image frame, and the optical flow result represents the area with optical flow. Secondly, the electronic device calculates the difference result between the previous image frame and the current image frame based on the previous image frame and the current image frame; the difference result represents the area where the absolute value of the difference is greater than the target threshold and represents the area with optical flow. Then, based on the optical flow result and the difference result, the electronic device determines that the moving object area in the current image frame includes a third area, namely the local motion area, and a fourth area, namely the global motion area. Furthermore, based on the attribute characteristics of the third area and the fourth area, the electronic device performs fusion processing on the previous image frame and the current image frame to obtain the processed current image frame. Finally, the electronic device uses the processed current image frame as the previous image frame of the next frame to perform multi-frame noise reduction processing.

[0068] For example, see Figure 5 As shown, the optical flow result from the previous image frame to the current image frame calculated by the electronic device is as follows Figure 5 The black block area in .

[0069] For example, see Figure 6 As shown, the difference between the previous image frame and the current image frame calculated by the electronic device is as follows: Figure 6 The black block area in .

[0070] As can be seen from the above, the image processing method provided by the embodiment of the present application uses the optical flow results and difference results between the previous and next frames to accurately distinguish the local motion area from the global motion area. Then, the attribute characteristics of the local motion area and the global motion area are determined based on the segmentation results, and different noise reduction parameters are used for the local motion area and the global motion area to adjust the fusion ratio of each area to achieve the purpose of eliminating the ghosting of the moving object and ensuring the noise reduction effect of other background areas at the same time.

[0071] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0072] Based on the foregoing, the embodiment of the present application provides an image processing method, which is applied to an electronic device, referring to Figure 7 As shown, the method comprises the following steps:

[0073] Step 301: downsample the current image frame to obtain a first target image frame.

[0074] In an embodiment of the present application, the electronic device downsamples the current image frame, such as reducing the size of the current image frame, to obtain a first target image frame, so as to increase the processing speed of motion area detection.

[0075] Step 302: downsample the previous image frame to obtain a second target image frame.

[0076] In an embodiment of the present application, the electronic device downsamples the previous image frame, such as reducing the size of the previous image frame, to obtain a second target image frame, so as to increase the processing speed of motion area detection.

[0077] Step 303: Obtain a first optical flow prediction region from the second target image frame to the first target image frame based on the optical flow method.

[0078] Step 304: Determine an area in the current image frame that has a mapping relationship with the first optical flow prediction area as the first area.

[0079] Here, the electronic device determines that the area in the current image frame that has a mapping relationship with the first optical flow prediction area is the first area, that is, maps the calculated optical flow result to the current image frame.

[0080] Step 305: Obtain feature information of the first region in the current image frame.

[0081] The first region includes a region in the current image frame determined by performing motion estimation based on the optical flow method on the current image frame and the previous image frame. Here, after the electronic device maps the calculated optical flow result to the current image frame, it marks the blocks with the optical flow to obtain feature information of the first region in the current image frame.

[0082] Step 306 : Obtain a second absolute value of the difference between the grayscale values ​​of the plurality of third pixel points of the first target image frame and the grayscale values ​​of the plurality of fourth pixel points of the second target image frame.

[0083] Step 307: Determine the area corresponding to the pixel points whose second absolute value is greater than the second threshold as the second optical flow prediction area.

[0084] Step 308: Determine the area in the current image frame that has a mapping relationship with the second optical flow prediction area as the second area.

[0085] Here, the electronic device determines that the area in the current image frame that has a mapping relationship with the second optical flow prediction area is the second area, that is, maps the calculated difference result to the current image frame.

[0086] Step 309: Obtain feature information of the second area in the current image frame.

[0087] The second region includes a region corresponding to pixels whose association relationship between a pixel point among the multiple first pixels of the current image frame and a pixel point among the multiple second pixels of the previous image frame meets the condition. Here, after the electronic device maps the calculated difference result to the current image frame, it marks the block with optical flow to obtain feature information of the second region in the current image frame.

[0088] Step 310: Based on the feature information of the first region and the feature information of the second region, a fusion process is performed on the previous image frame and the current image frame to obtain a processed current image frame.

[0089] The processed current image frame is used as a previous image frame of the next image frame, and the next image frame is processed.

[0090] It should be noted that, for the description of the same steps and the same contents in this embodiment as those in other embodiments, reference can be made to the description in other embodiments and will not be repeated here.

[0091] The embodiment of the present application provides an image processing method, referring to Figure 8 As shown, the method comprises the following steps:

[0092] Step 1. The electronic device calculates the optical flow result from the previous image frame (reference image) to the current image frame (base image). Here, in order to save processing speed, the reference image and the base image are first reduced in size, for example, both are reduced by 1 / 2. Then, the optical flow is detected based on the block matching algorithm on the reduced-size image. Here, the block size can be 16x16.

[0093] Furthermore, the electronic device maps the calculated optical flow results onto the base image and marks the blocks with optical flow as 0, such as Figure 5 Medium black block area.

[0094] Here, the electronic device may use a block matching optical flow calculation method to detect the optical flow of the moving object, thereby marking the area as an area with a larger motion vector.

[0095] Step 2: The electronic device calculates the difference between the reference image and the base image.

[0096] Here, in order to save processing speed, the reference image and base image are first reduced in size. Then, the absolute value of the difference between each pixel of the two frames is calculated on the reduced-size image. If the absolute value is less than the threshold, it is considered that the correlation between the two frames of the pixel is large. If it is greater than the threshold, it means that the difference between the two frames is large, which may be a moving object or an area with large lighting changes, and it is marked as 0. Furthermore, the electronic device maps the calculated difference result to the base image and marks the block with optical flow as 0, such as Figure 6 Medium black block area

[0097] Step 3: The electronic device detects the moving object area and the global motion area.

[0098] Here, the electronic device accurately detects the moving object area based on the optical flow results and difference results of the two frames before and after calculated in the above two steps. It should be noted that the inventors have experimentally learned that the optical flow results may also misdetect the global motion area of ​​the background, while the difference results can avoid this part. At the same time, the difference results may misdetect the part blocked by the moving object between the two frames, while the optical flow results can just cover this part. Therefore, after many experiments, the inventors have found that only the part where both results are 0 is the final local moving object area.

[0099] Step 4: The electronic device performs front and back frame fusion processing.

[0100] Here, after the electronic device detects the local moving object area and the global moving area, it provides different noise reduction parameters for different areas and adaptively adjusts the fusion ratio to perform fusion processing of the two frames.

[0101] In the embodiment of the present application, the local moving object area is prone to ghosting, so the ratio of the reference image when the two frames are fused is reduced, and the pixel values ​​of the base image are used as much as possible to avoid ghosting when fusion. Furthermore, in the embodiment of the present application, a smaller sigma value is provided to the area to reduce the proportion of the reference image as much as possible.

[0102] On the contrary, for global motion areas such as the background, in order to ensure the denoising effect, the pixel values ​​of the previous frame reference image that has been denoised will be used to the maximum extent, so a larger sigma will be provided to improve the fusion ratio of the reference image.

[0103] In the embodiment of the present application, exemplary parameters involved in the fusion of adjacent frames are shown in the following table:

[0104] Global Zone Local area Lambda 0.7 0.7 Sigma 16 8 diffVal 5 10 mu 0.9523448 0.457833362 weight 0.6666414 0.320483353

[0105] Exemplarily, the electronic device performs noise reduction processing on the fused image based on the above parameters by using the following formula. Here, the noise reduction filter may adopt a Gaussian noise reduction filter, and the calculation formula is as follows:

[0106] diffVal = |base_Y-ref_Y|

[0107]

[0108] weight=mu*lambda,lambda=0.7

[0109] Base_Y=base_Y*(1-weight)+ref_Y*weight

[0110] Here, the parameters involved in the above tables and formulas are explained, where Y represents the grayscale value, base_Y represents the grayscale value of the current image frame, ref_Y represents the grayscale value of the previous image frame, diffVal represents the absolute value of the difference between the grayscale value of the current image frame and the grayscale value of the previous image frame; mu represents the Gaussian denoising function value, sigma represents the coefficient of the Gaussian function, lambda represents the weight of the function value, weight represents the regional weight, and Base_Y represents the grayscale value of the current image frame after fusion.

[0111] For example, see Fig. 9 As shown, for the fused image, the sigma of the local motion area is 8, and the sigma of the global motion area is 14. Then, based on the above formula, the fused image is denoised to obtain the processed current image frame.

[0112] For example, see Fig.10 As shown, the embodiment of the present application uses different sigmas for noise reduction based on local and global motion areas, effectively eliminating ghosting around moving objects and ensuring the noise reduction effect in the background area.

[0113] For example, see Fig.11 As shown, in some embodiments, if the same sigma is used for fusion noise reduction, although the noise reduction effect of the background and other areas is guaranteed, it brings serious ghosting. It can be seen that by using different sigmas for noise reduction, the image quality of the processed current image frame is better.

[0114] It should be noted that when shooting videos, the image acquisition module and the moving object often move at the same time, so the global moving area of ​​the background is also included in the optical flow detection result. In addition, in low-light shooting scenes, due to the influence of lighting conditions, the grayscale change between the previous and next frames is too large, which will also cause inaccurate optical flow detection results. Fig.12 As shown, a multi-frame noise reduction method in related art is used to fuse the previous image frame and the current image frame, and the denoised current image frame still has noise and ghosting.

[0115] In order to obtain more accurate local motion region detection results, the image processing method provided in the embodiment of the present application not only considers the optical flow results, but also considers the grayscale difference results of the previous and next frames. Then, in the fusion processing stage of the two frames, the embodiment of the present application uses different sigma values ​​to perform noise reduction processing on the separated moving object area and the global motion area, see Fig.13 As shown, the image processing method provided in the present application is used to fuse the previous image frame and the current image frame, so as to adjust the fusion ratio of the moving object area and the global motion area to avoid ghosting on the moving object and ensure the noise reduction effect of the global motion area of ​​the background.

[0116] The embodiment of the present application can achieve the following beneficial effects: the local motion area and the global motion area are accurately distinguished by using the optical flow detection results and gray value difference results between the previous and next two frames. Then, according to the segmentation results, different noise reduction parameters are used to adjust the fusion ratio of each area for the local motion area and the global motion area to eliminate the ghosting of the moving object, while ensuring the noise reduction effect of other background areas.

[0117] An embodiment of the present application provides an image processing device, and each module included in the device can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; during the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0118] Fig.14 Schematic diagram of the composition structure of the image processing device according to the embodiment of the present application. Fig.14 As shown, the image processing device 400 includes a first obtaining module 401, a second obtaining module 402 and a processing module 403, wherein:

[0119] The first obtaining module 401 is configured to: obtain feature information of a first region in the current image frame; wherein the first region includes a region in the current image frame determined by performing motion estimation based on an optical flow method on the current image frame and the previous image frame;

[0120] The second obtaining module 402 is configured to: obtain feature information of a second area in the current image frame; wherein the second area includes an area corresponding to a pixel point whose association relationship between a pixel point among a plurality of first pixel points in the current image frame and a pixel point among a plurality of second pixel points in a previous image frame meets a condition;

[0121] The processing module 403 is configured to: based on the feature information of the first area and the feature information of the second area, fuse the previous image frame and the current image frame to obtain a processed current image frame; wherein the processed current image frame is used as the previous image frame of the next image frame to process the next image frame.

[0122] In other embodiments of the present application, the processing module 403 is configured to: determine a third area and a fourth area in the current image frame based on the feature information of the first area and the feature information of the second area; wherein the third area represents an area where the current image frame has a local motion relative to the previous image frame, and the fourth area represents an area where the current image frame has a global motion relative to the previous image frame;

[0123] Obtaining a first parameter corresponding to the third region;

[0124] obtaining a second parameter corresponding to a fifth area in a previous image frame associated with the fourth area;

[0125] Based on the first parameter and the second parameter, a fusion process is performed on the previous image frame and the current image frame to obtain a processed current image frame.

[0126] In other embodiments of the present application, the processing module 403 is configured to: adjust the pixel value of the third area of ​​the current image frame based on the first parameter to obtain the current target image frame;

[0127] Adjusting the pixel value of the fifth area of ​​the previous image frame based on the second parameter to obtain a previous target image frame;

[0128] The current target image frame and the previous target image frame are fused to obtain a processed current image frame.

[0129] In other embodiments of the present application, the pixel value of the third area of ​​the current target image frame is greater than the pixel value of the third area of ​​the current image frame, and / or the pixel value of the fifth area of ​​the previous target image frame is greater than the pixel value of the fifth area of ​​the previous image frame.

[0130] In other embodiments of the present application, the processing module 403 is configured to: downsample the current image frame to obtain a first target image frame;

[0131] Down-sampling the previous image frame to obtain a second target image frame;

[0132] Obtaining a first optical flow prediction region from the second target image frame to the first target image frame based on an optical flow method;

[0133] Determine an area in the current image frame that has a mapping relationship with the first optical flow prediction area as the first area.

[0134] In other embodiments of the present application, the association relationship meets the condition that a first absolute value of the difference between the grayscale value of the first pixel and the grayscale value of the second pixel is greater than a first threshold.

[0135] In other embodiments of the present application, the processing module 403 is configured to: downsample the current image frame to obtain a first target image frame;

[0136] Down-sampling the previous image frame to obtain a second target image frame;

[0137] Obtaining a second absolute value of a difference between grayscale values ​​of a plurality of third pixel points of the first target image frame and grayscale values ​​of a plurality of fourth pixel points of the second target image frame;

[0138] Determine an area corresponding to pixels whose second absolute value is greater than a second threshold as a second optical flow prediction area;

[0139] Determine an area in the current image frame that has a mapping relationship with the second optical flow prediction area as the second area.

[0140] In other embodiments of the present application, the characteristic information of the first area is used to identify the first area, and the characteristic information of the second area is used to identify the second area.

[0141] The image processing device provided by the embodiment of the present application obtains feature information of a first area in a current image frame; wherein the first area includes an area in the current image frame determined by performing motion estimation based on the optical flow method on the current image frame and the previous image frame; obtains feature information of a second area in the current image frame; wherein the second area includes an area corresponding to pixel points whose association relationship between a pixel point among a plurality of first pixel points of the current image frame and a pixel point among a plurality of second pixel points of the previous image frame meets a condition; here, the feature information of the first area and the feature information of the second area are used to characterize the correlation between adjacent image frames, and further, based on the feature information of the first area and the feature information of the second area, the feature information of the previous image frame is estimated. The image frame and the current image frame are fused to obtain a processed current image frame; in this way, in the process of fusion processing, based on the feature information of the first area and the feature information of the second area, not only the motion information of the object between adjacent image frames can be accurately calculated, but also the accuracy of frame alignment is improved, thereby avoiding ghosting in fusion; at the same time, the processed current image frame is used as the previous image frame of the next image frame to process the next image frame; in this way, the problem of poor noise reduction effect of multiple video frames when the global motion of the image acquisition device and the local motion of the moving object occur simultaneously in the process of capturing video images in the related technology is solved, the noise reduction effect of multiple video frames is improved, and the video quality is improved.

[0142] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0143] It should be noted that in the embodiment of the present application, when the above-mentioned image processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a desktop, a server, a television, an audio player, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0144] An embodiment of the present application provides an electronic device, Fig.15 A schematic diagram of a hardware entity of an electronic device according to an embodiment of the present application is shown in FIG. Fig.15As shown, the electronic device 500 includes a memory 501 and a processor 502. The memory 501 stores a computer program that can be run on the processor 502. When the processor 502 executes the program, the image processing method provided in the above embodiment is implemented.

[0145] It should be noted that the memory 501 is configured to store instructions and applications executable by the processor 502, and can also cache data to be processed or processed by the processor 502 and various modules in the electronic device 500 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0146] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the image processing method provided in the above embodiment is implemented.

[0147] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0148] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0149] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0150] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0151] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0152] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0153] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0154] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application can be essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable an electronic device (which can be a mobile phone, a tablet computer, a desktop, a server, a television, an audio player, etc.) to execute all or part of the methods of each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks. The methods disclosed in the several method embodiments provided in the present application can be arbitrarily combined without conflict to obtain new method embodiments. The features disclosed in the several product embodiments provided in the present application can be arbitrarily combined without conflict to obtain new product embodiments. The features disclosed in the several method or device embodiments provided in the present application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments. The above are only implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0155] Industrial Applicability

[0156] The embodiment of the present application provides an image processing method, apparatus and device, which obtains feature information of a first area in a current image frame; wherein the first area includes an area in the current image frame determined by performing motion estimation based on an optical flow method on the current image frame and a previous image frame; obtains feature information of a second area in the current image frame; wherein the second area includes an area corresponding to a pixel point whose association relationship between a pixel point among a plurality of first pixel points of the current image frame and a pixel point among a plurality of second pixel points of the previous image frame meets a condition; here, the feature information of the first area and the feature information of the second area are used to characterize the correlation between adjacent image frames, and further, based on the feature information of the first area and the feature information of the second area , the previous image frame and the current image frame are fused to obtain the processed current image frame; in this way, in the process of fusion processing, based on the feature information of the first area and the feature information of the second area, not only the motion information of the object between adjacent image frames can be accurately calculated, but also the accuracy of frame alignment is improved, thereby avoiding the occurrence of ghosting in fusion; at the same time, the processed current image frame is used as the previous image frame of the next image frame to process the next image frame; in this way, the problem of poor noise reduction effect of multiple video frames when the global motion of the image acquisition device and the local motion of the moving object occur simultaneously in the process of capturing video images in the related technology is solved, the noise reduction effect of multiple video frames is improved, and the video quality is improved.

Claims

1. An image processing method, characterized in that: include: Obtaining feature information of a first region in a current image frame; wherein the first region includes a region in the current image frame determined by performing motion estimation based on an optical flow method on the current image frame and a previous image frame; Obtaining feature information of a second area in the current image frame; wherein the second area includes an area corresponding to a pixel point whose association relationship between a pixel point among a plurality of first pixel points in the current image frame and a pixel point among a plurality of second pixel points in the previous image frame meets a condition; Based on the feature information of the first region and the feature information of the second region, the previous image frame and the current image frame are fused to obtain a processed current image frame; wherein the processed current image frame is used as a previous image frame of a next image frame, and the next image frame is processed; The step of fusing the previous image frame and the current image frame based on the feature information of the first region and the feature information of the second region to obtain the processed current image frame includes: Based on the feature information of the first region and the feature information of the second region, determining a third region and a fourth region in the current image frame; wherein the third region represents a region where a local motion occurs in the current image frame relative to the previous image frame, and the fourth region represents a region where a global motion occurs in the current image frame relative to the previous image frame; Obtaining a first parameter corresponding to the third area, wherein the first parameter is used to increase the ratio of the current image frame when two frames are fused; Obtaining a second parameter corresponding to a fifth area in the previous image frame associated with the fourth area, wherein the second parameter is used to increase a ratio of the previous image frame when two frames are fused; Based on the first parameter and the second parameter, the previous image frame and the current image frame are fused to obtain the processed current image frame.

2. The method according to claim 1, characterized in that: The step of performing fusion processing on the previous image frame and the current image frame based on the first parameter and the second parameter to obtain the processed current image frame includes: Adjusting the pixel values ​​of the third area of ​​the current image frame based on the first parameter to obtain a current target image frame; Adjusting the pixel value of the fifth area of ​​the previous image frame based on the second parameter to obtain a previous target image frame; The current target image frame and the previous target image frame are fused to obtain the processed current image frame.

3. The method according to claim 2, characterized in that: The pixel value of the third area of ​​the current target image frame is greater than the pixel value of the third area of ​​the current image frame, and / or the pixel value of the fifth area of ​​the previous target image frame is greater than the pixel value of the fifth area of ​​the previous image frame.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: down-sampling the current image frame to obtain a first target image frame; Downsampling the previous image frame to obtain a second target image frame; Obtaining a first optical flow prediction region from the second target image frame to the first target image frame based on an optical flow method; Determine an area in the current image frame that has a mapping relationship with the first optical flow prediction area as the first area.

5. The method according to any one of claims 1 to 3, characterized in that: The condition that the association relationship meets the condition indicates that a first absolute value of a difference between a grayscale value of the first pixel and a grayscale value of the second pixel is greater than a first threshold.

6. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: down-sampling the current image frame to obtain a first target image frame; Downsampling the previous image frame to obtain a second target image frame; Obtaining a second absolute value of a difference between grayscale values ​​of a plurality of third pixel points of the first target image frame and grayscale values ​​of a plurality of fourth pixel points of the second target image frame; Determine an area corresponding to pixels whose second absolute value is greater than a second threshold as a second optical flow prediction area; Determine an area in the current image frame that has a mapping relationship with the second optical flow prediction area as the second area.

7. The method according to any one of claims 1 to 3, characterized in that: The characteristic information of the first area is used to identify the first area, and the characteristic information of the second area is used to identify the second area.

8. An image processing device, characterized in that: include: A first obtaining module is configured to: obtain feature information of a first area in a current image frame; wherein the first area includes an area in the current image frame determined by performing motion estimation based on an optical flow method on the current image frame and a previous image frame; A second obtaining module is configured to: obtain feature information of a second area in the current image frame; wherein the second area includes an area corresponding to a pixel point whose association relationship between a pixel point among the multiple first pixel points of the current image frame and a pixel point among the multiple second pixel points of the previous image frame meets a condition; The processing module is configured to: perform fusion processing on the previous image frame and the current image frame based on the feature information of the first area and the feature information of the second area to obtain a processed current image frame; wherein the processed current image frame is used as a previous image frame of a next image frame to process the next image frame; the fusion processing on the previous image frame and the current image frame based on the feature information of the first area and the feature information of the second area to obtain the processed current image frame includes: determining a third area and a fourth area in the current image frame based on the feature information of the first area and the feature information of the second area ; wherein the third area represents an area where the current image frame has local motion relative to the previous image frame, and the fourth area represents an area where the current image frame has global motion relative to the previous image frame; a first parameter corresponding to the third area is obtained, and the first parameter is used to increase the ratio of the current image frame when the two frames are fused; a second parameter corresponding to the fifth area in the previous image frame associated with the fourth area is obtained, and the second parameter is used to increase the ratio of the previous image frame when the two frames are fused; based on the first parameter and the second parameter, the previous image frame and the current image frame are fused to obtain the processed current image frame.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the image processing method according to any one of claims 1 to 7 when executing the program.

Images