Image processing method and device and storage medium

Abstract

The invention relates to an image processing method and device and a storage medium. The image processing method comprises the steps that a second image sequence is determined according to a first image sequence, the first image sequence comprises multiple frames of images of the same picture collected in a scene with ambient light brightness lower than a threshold value, and the second image sequence is an image sequence obtained after registration of a first image and remaining images, the first image is the image with the highest definition in the multiple frames of images, and the residual images are other images except the first image in the first image sequence; determining a second image and a third image according to the first image and the second image sequence; and determining a target image according to the second image and the third image. According to the invention, effective de-noising of the image in the scene where the ambient light brightness is lower than the threshold value can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, and in particular to an image processing method, device and storage medium. Background Art

[0002] Image noise is a common problem in the field of image processing. In extremely dark scenes, the amount of light entering the image is very low, which makes the impact of image noise more significant. Therefore, there are higher requirements for the denoising accuracy of image processing methods in such scenes. However, the accuracy of current image denoising needs to be further improved. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, device and storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising: determining a second image sequence based on a first image sequence, the first image sequence comprising multiple frames of images of the same picture acquired in a scene where the ambient light brightness is lower than a threshold, the second image sequence being an image sequence obtained after registering the first image and the remaining images, the first image being an image with the highest clarity among the multiple frames of images, and the remaining images being other images in the first image sequence except the first image; determining a second image and a third image based on the first image and the second image sequence; and determining a target image based on the second image and the third image.

[0005] In one embodiment, the first image is determined in the following manner, including: using an edge detection operator to determine the horizontal gradient and the vertical gradient of each frame of the multiple images, and determining the global gradient mean based on the horizontal gradient and the vertical gradient; determining the first image based on the global gradient mean and the shooting parameters of each frame of the multiple images.

[0006] In one embodiment, determining the second image and the third image based on the first image and the second image sequence includes: denoising the first image to obtain the second image; performing exposure compensation on each image in the second image sequence based on an exposure compensation parameter to obtain an image sequence after exposure compensation, wherein the exposure compensation parameter is a ratio between an expected grayscale value and an actual grayscale value of any frame image in the multiple frames of images; and denoising the image sequence after exposure compensation to obtain the third image.

[0007] In one embodiment, determining the target image based on the second image and the third image includes: binarizing the image in the third image to obtain a binary mask image, wherein the binary mask image is used to distinguish between image areas with brightness greater than a threshold and image areas with brightness less than or equal to the threshold; based on the binarized image, performing reverse exposure compensation on the second image and the third image, and fusing the reverse exposure compensated second image and third image to obtain the target image.

[0008] In one embodiment, the reverse exposure compensation is performed on the second image and the third image based on the binarized image, including: extracting an image area in the second image whose brightness is greater than a threshold value through the binary mask image; performing reverse exposure compensation on the third image based on the exposure compensation parameters to obtain a reverse exposure compensated image; and extracting an image area in the reverse exposure compensated image whose brightness is less than or equal to the threshold value through the complement of the binary mask image.

[0009] In one embodiment, the image processing method further includes: performing post-processing on the target image, wherein the post-processing includes at least one of the following: filling simulated noise into the target image; detecting a target area in the target image whose grayscale value variation is greater than a threshold, and sharpening the target area.

[0010] According to a second aspect of an embodiment of the present disclosure, an image processing device is provided, comprising: a determination unit, used to determine a second image sequence based on a first image sequence, wherein the first image sequence includes multiple frames of images of the same picture captured in a scene where the ambient light brightness is lower than a threshold value, the second image sequence is an image sequence obtained after aligning the first image and the remaining images, the first image is an image with the highest clarity among the multiple frames of images, and the remaining images are other images in the first image sequence except the first image; a processing unit, used to determine a second image and a third image based on the first image and the second image sequence; and determine a target image based on the second image and the third image.

[0011] In one embodiment, the first image is determined in the following manner: using an edge detection operator to determine the horizontal gradient and the vertical gradient of each frame of the multiple images, and based on the horizontal gradient and the vertical gradient, determining the global gradient mean; based on the global gradient mean and the shooting parameters of each frame of the multiple images, determining the first image.

[0012] In one embodiment, the processing unit determines the second image and the third image based on the first image and the second image sequence in the following manner: denoising the first image to obtain the second image; performing exposure compensation on each image in the second image sequence based on an exposure compensation parameter to obtain an image sequence after exposure compensation, wherein the exposure compensation parameter is a ratio between an expected grayscale value and an actual grayscale value of any frame image in the multiple frames of images; denoising the image sequence after exposure compensation to obtain the third image.

[0013] In one embodiment, the processing unit determines the target image based on the second image and the third image in the following manner: binarizing the image in the third image to obtain a binary mask image, wherein the binary mask image is used to distinguish between image areas having brightness greater than a threshold and image areas having brightness less than or equal to the threshold; based on the binarized image, performing reverse exposure compensation on the second image and the third image, and fusing the reverse exposure compensated second image and third image to obtain the target image.

[0014] In one embodiment, the processing unit performs reverse exposure compensation on the second image and the third image based on the binarized image in the following manner: extracting an image area in the second image whose brightness is greater than a threshold value through the binary mask image; performing reverse exposure compensation on the third image based on the exposure compensation parameters to obtain a reverse exposure compensated image; and extracting an image area in the reverse exposure compensated image whose brightness is less than or equal to the threshold value through the complement of the binary mask image.

[0015] In one embodiment, the processing unit is further used to perform post-processing on the target image, and the post-processing includes at least one of the following: filling simulated noise into the target image; detecting a target area in the target image whose grayscale value variation is greater than a threshold, and sharpening the target area.

[0016] According to a third aspect of the present disclosure, an image processing device is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: execute the image processing method described in the aforementioned first aspect or any one of the embodiments of the first aspect.

[0017] According to a fourth aspect of the present disclosure, a storage medium is provided, wherein the storage medium stores instructions, and when the instructions are executed by a processor, the image processing method described in the first aspect or any one of the embodiments of the first aspect is executed.

[0018] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: by determining the image with the highest definition as the first image and performing image registration to obtain the second image sequence, image denoising is performed on the first image and the second image sequence respectively, effectively removing noise, retaining image details and dynamic range, and improving image quality. At the same time, by fusing the second image and the third image, the accuracy loss and information loss of some areas that may be introduced by the method of directly improving the image brightness are reduced.

[0019] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0021] Figure 1 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0022] Figure 2 The invention is a flowchart of a method for determining a first image according to an exemplary embodiment.

[0023] Figure 3 The figure is a flow chart of an image denoising method according to an exemplary embodiment.

[0024] Figure 4 The figure is a flow chart of an exposure fusion method according to an exemplary embodiment.

[0025] Figure 5 The figure is a flow chart of a reverse exposure compensation method according to an exemplary embodiment.

[0026] Figure 6 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0027] Figure 7 is a schematic diagram showing a process of denoising an extremely dark scene image according to an exemplary embodiment

[0028] Figure 8 is a block diagram of an image processing device according to an exemplary embodiment.

[0029] Fig. 9 The invention is a block diagram of a device for image processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0031] The image processing method provided by the embodiments of the present disclosure can be applied to a scene where an image is processed in a scene where the ambient light brightness is lower than a threshold, for example, a scene where a picture collected in an extremely dark scene where the ambient light intensity is less than or equal to 0.1 lux is subjected to denoising.

[0032] In the related art, directly denoising images in extremely dark scenes will result in a large loss of accuracy and fail to effectively retain local information in the image. On the other hand, in order to expand the image signal range, the method of directly increasing the brightness of the original image is often used, which can expand the image signal range to a certain extent. However, since the signal value of the highlight area where the image brightness is greater than the set threshold has approached or reached the maximum value of the image, this operation of directly increasing the image brightness will cause information loss in the highlight area of ​​the image and amplify the noise.

[0033] In view of this, an embodiment of the present disclosure provides an image processing method. Figure 1 The figure is a flowchart of an image method according to an exemplary embodiment.

[0034] In step S11 , a second image sequence is determined according to the first image sequence.

[0035] Among them, the first image sequence includes multiple frames of images of the same picture captured in a scene where the ambient light brightness is lower than a threshold value, the second image sequence is an image sequence obtained after aligning the first image and the remaining images, the first image is the image with the highest clarity among the multiple frames of images, and the remaining images are other images in the first image sequence except the first image.

[0036] The scene where the ambient light brightness is lower than the threshold may include an extremely dark scene where the ambient light intensity is less than or equal to 0.1 lux. In addition, the first image sequence may be an original image sequence acquired by the terminal device through an image acquisition device, or may be an image sequence acquired from other terminal devices.

[0037] The image with the highest definition among multiple frames of images is determined as a reference image and registered to obtain a second image sequence, which can improve the accuracy and stability of the registration.

[0038] In step S12, a second image and a third image are determined according to the first image and the second image sequence.

[0039] In the embodiment of the present disclosure, the first image and the second image sequence are subjected to image processing to obtain the second image and the third image. The image processing may be a different denoising process for the first image and the second image sequence, so that different regions of the second image and the third image are effectively denoised while ensuring accuracy.

[0040] In step S13, a target image is determined according to the second image and the third image.

[0041] The fusion process is to fuse the second image and the third image after image denoising. Specifically, the fusion process can be to extract different areas where noise is effectively removed in the second image and the third image and fuse them. For example, the pixel value of the highlight area in the second image with brightness greater than a set threshold and the pixel value of the non-highlight area in the third image with brightness less than or equal to the set threshold are extracted. The advantage of this is that the detail information of the highlight area can be retained, while exposure compensation is performed in the non-highlight area, thereby obtaining better image quality and detail retention capability.

[0042] In the image processing method provided by the embodiment of the present disclosure, multiple frames of images are collected in a scene where the ambient light brightness is lower than a threshold value, and after image registration is performed based on the image with the highest definition among the multiple frames of images, the images are denoised. After obtaining the second image and the third image after image denoising, the images of different regions where the noise is effectively removed in the second image and the third image are extracted and fused to obtain the target image, and the image noise is effectively removed without losing accuracy, and the information of the highlight area can be retained.

[0043] Figure 2 The invention is a flowchart of a method for determining a first image according to an exemplary embodiment.

[0044] In step S21, the edge detection operator is used to determine the horizontal gradient and the vertical gradient of each frame of the multiple frames, and the global gradient mean is determined based on the horizontal gradient and the vertical gradient.

[0045] In the embodiment of the present disclosure, the Sobel gradient operator may be used to determine the global gradient. The Sobel gradient operator is a commonly used image edge detection operator, which is used to calculate the gradient strength and gradient direction of each pixel in an image.

[0046] Edge detection operators can be used to determine edge positions in an image, and these operators are usually calculated based on the grayscale values ​​of pixels in the image. In the embodiments of the present disclosure, other edge detection operators can also be used to determine the global gradient.

[0047] For example, the Prewitt operator is used to determine the global gradient by calculating the horizontal and vertical gradients of each pixel in the image. The Roberts operator is used to calculate the gradient using a 2×2 template, which is more suitable for smaller images.

[0048] In step S22, a first image is determined based on the global gradient mean and the shooting parameters of each frame of the multiple frames.

[0049] The mean of the global gradient calculates the gradient information of the entire image and can be used to comprehensively evaluate the clarity of the image. If the mean is large, it means that there are more edges and details in the image, that is, the image is relatively clear. If the mean is small, it means that there are fewer edges and details in the image, that is, the image is relatively blurred. At the same time, the imaging device usually records some parameter information related to image capture, such as exposure time, sensitivity (ISO), focal length, etc. These parameter information can provide some indication of image quality and clarity.

[0050] Therefore, when selecting a clear frame, the shooting parameter information may also be comprehensively considered to determine the first image.

[0051] For example, a weighted function of a clarity index may be defined to weight the clarity score of the image and the shooting parameters, and the first image may be determined based on the weighted score.

[0052] In the disclosed embodiment, the first image sequence is acquired under set acquisition parameters, wherein the acquisition parameters include but are not limited to: exposure time, and / or sensitivity (ISO), etc. For example, in order to receive light for a longer time and thereby obtain more photon signals, a longer exposure time and / or a maximum sensitivity (ISO) analog gain are set.

[0053] In the method for determining the first image provided by the embodiment of the present disclosure, by selecting the image with the highest definition as the first image, the image detail information can be retained to the greatest extent and the image noise caused by motion blur or other factors can be reduced. The image with the highest definition can provide a more accurate reference, which is helpful for the accuracy and stability of subsequent image registration and processing steps. Therefore, determining the image with the highest definition as the first image has an important impact on the quality and accuracy of the entire image processing process.

[0054] Figure 3 The figure is a flow chart of an image denoising method according to an exemplary embodiment.

[0055] In step S31, the first image is denoised to obtain a second image.

[0056] The first image is subjected to image denoising to obtain a second image, wherein the first image is the selected image with the highest definition, and denoising the first image can retain the detail information of the highlight area of ​​the original image.

[0057] In step S32, exposure compensation is performed on each image in the second image sequence based on the exposure compensation parameter to obtain an image sequence after exposure compensation.

[0058] After obtaining the second image sequence through image registration, in order to reduce the accuracy loss after denoising caused by extremely low signal in the scene where the ambient light brightness is lower than the threshold, the exposure compensation step is taken. Exposure compensation is to adjust the brightness of the image by multiplying each frame in the image sequence with the exposure compensation parameter calculated in advance. The exposure compensation parameter is the ratio between the expected grayscale value and the actual grayscale value of any frame in the multiple frames.

[0059] In step S33, the exposure compensated image sequence is denoised to obtain a third image.

[0060] Through exposure compensation, the problem that image denoising causes a large loss of precision, which is caused by the extremely low amount of light entering the scene, and the image signal values ​​are concentrated in a smaller range, is solved. The result after exposure compensation is subjected to image denoising to obtain a third image in which the noise in the lower brightness area of ​​the original image is effectively removed. Among them, the image denoising method for the image sequence after exposure compensation can be to input the exposure compensation result into a multi-frame image denoising network. In the training stage, the image denoising network can be trained using simulated noise data, so that it can effectively remove noise in the image and improve the clarity and quality of the image.

[0061] In the image denoising method provided by the embodiment of the present disclosure, the first image is subjected to image denoising to obtain the second image, and the highlight area of ​​the first image can be effectively denoised. Exposure compensation is performed on each image in the second image sequence, and the brightness of each image frame can be adjusted to increase the visibility of details in the lower brightness area of ​​the original image, better remove noise in the image, and improve the image quality and detail retention capability. In addition, by performing image denoising on each image in the second image sequence to obtain the third image, the repeated information in the multi-frame image sequence can be fully utilized to reduce the impact of additional noise that may be caused by exposure compensation.

[0062] In the disclosed embodiment, since the images in the second image sequence may have some highlight areas, after exposure compensation, these areas may be overexposed due to excessive brightness. This overexposure problem will cause the loss of details and information of the highlight areas of the third image obtained after image denoising.

[0063] In view of this, an embodiment of the present disclosure provides an exposure fusion method. In the exposure fusion method provided by the embodiment of the present disclosure, a binary mask image is used to distinguish image areas with brightness greater than a threshold and image areas with brightness less than or equal to the threshold, and reverse exposure compensation is performed. Afterwards, the image areas with brightness greater than the threshold where noise is effectively removed in the second image and the image areas with brightness less than or equal to the threshold where noise is effectively removed in the third image after reverse exposure compensation are fused to obtain a target image.

[0064] Figure 4 The figure is a flow chart of an exposure fusion method according to an exemplary embodiment.

[0065] In step S41, the image in the third image is binarized to obtain a binarized mask image, which is used to distinguish between image areas with brightness greater than a threshold and image areas with brightness less than or equal to the threshold.

[0066] Binary Mask is a technique in image processing, which is used to represent pixels in a specific area of ​​an image. A binary mask is a two-dimensional array of the same size as the original image, in which there are only two possible element values, usually 0 and 1. In the disclosed embodiment, the pixel value whose image brightness is greater than the set threshold is recorded as 1, and the pixel value whose image brightness is less than or equal to the set threshold is recorded as 0. Among them, the set threshold can be set according to the current scene illumination after normalizing the image signal value. For example, in an extremely dark scene, the set threshold can be 0.99.

[0067] In step S42, based on the binarized image, reverse exposure compensation is performed on the second image and the third image, and the reverse exposure compensated second image and third image are fused to obtain a target image.

[0068] In the embodiment of the present disclosure, the third image after the exposure compensation processing is threshold processed to obtain a binary mask image that identifies the highlighted and non-highlighted areas in the image. And in the process of fusion, the areas where the noise of the second image and the third image is effectively removed are retained. Through the exposure fusion method in the embodiment of the present disclosure, the highlighted area where the noise is effectively removed in the second image and the non-highlighted area where the noise is effectively removed in the third image after exposure compensation can be effectively fused, and the final target image can better restore the details and dynamic range of the image and have a better image denoising effect.

[0069] Figure 5 The figure is a flow chart of a reverse exposure compensation method according to an exemplary embodiment.

[0070] In step S51, an image region having brightness greater than a threshold value in the second image is extracted by binarizing the mask image.

[0071] In the disclosed embodiment, the second image is multiplied by the binary mask image, thereby retaining the highlight area in the second image where the noise is effectively removed.

[0072] In step S52, reverse exposure compensation is performed on the third image based on the exposure compensation parameter to obtain a reverse exposure compensated image.

[0073] The reverse exposure compensation is to divide the signal value in the third image after the exposure compensation processing by the exposure compensation parameter to reversely compensate for the exposure gain effect thereof, so that the image is restored to the original exposure level.

[0074] In step S53, the image region whose brightness is less than or equal to the threshold value in the image after inverse exposure compensation is extracted through the complement of the binary mask image.

[0075] The non-highlight area in which the noise is effectively removed in the third image is extracted by multiplying the image after inverse exposure compensation with the complement of the binary mask.

[0076] In the disclosed embodiment, the third image is restored to the original exposure level through inverse exposure compensation, and the highlight area in the second image and the non-highlight area in the third image are clearly separated through a binary mask image. Combined with the subsequent fusion operation, image noise removal can be achieved without losing accuracy and retaining image highlight information.

[0077] Figure 6 The figure is a flowchart of an image processing method according to an exemplary embodiment.

[0078] In the present disclosure, Figure 6 As shown, first, multiple frames of images are collected when the ambient light brightness is lower than the threshold. The edge detection operator is used to calculate the horizontal and vertical gradients of each frame of the image, and the global gradient mean is determined based on these gradients. Combining the global gradient mean and the shooting parameters of each frame of the image, the image with the highest clarity is selected as the first image, and image registration is performed to obtain the second image sequence.

[0079] The first image is denoised to obtain a second image, and the second image sequence is denoised after exposure compensation to obtain a third image.

[0080] A binarized mask image is obtained by binarizing the third image, which is used to distinguish image areas with brightness greater than a threshold and brightness less than or equal to the threshold. The third image is reverse-exposure compensated, and then the second image and the third image after reverse-exposure compensation are fused based on the binarized image to obtain a target image. In the processed image, due to the influence of the denoising operation, a certain smearing feeling may be brought, that is, the image looks too smooth or lacks details. In view of this, an embodiment of the present disclosure provides an image post-processing method. In the image post-processing method provided in an embodiment of the present disclosure, image post-processing may include filling simulated noise into the target image and / or detecting areas in the target image where the grayscale value changes dramatically, and sharpening the target image according to the detection results.

[0081] The simulated noise added to the target image can be Poisson noise, Gaussian noise, etc. Poisson distribution is a statistical distribution commonly used to simulate image noise. By dividing the target image by the simulated gain and then applying Poisson distribution, noise of a certain intensity can be generated and added to the target image. Gaussian noise is a common type of random noise that has the characteristics of Gaussian distribution and has a corresponding relationship with the sensitivity (ISO). The purpose of this is to reduce the smearing feeling and make the image look more real and natural.

[0082] In the disclosed embodiment, an edge detection operator can also be used to detect areas where the grayscale value changes dramatically, that is, high-frequency texture information, and then the image is sharpened according to the detection result. The sharpening operation can enhance the edges and details of the image, making the image look clearer and more detailed.

[0083] An edge detection operator is a filter used to detect edge or texture features in an image. It can determine the boundary position in an image by calculating the gradient or second-order derivative of pixel values.

[0084] In the image post-processing method of the disclosed embodiment, by filling the target image with simulated noise and / or detecting the area in the target image where the gray value changes dramatically, the target image is sharpened according to the detection result, so as to further improve the image quality, enhance the visual effect, and make the image more in line with the perception of the human eye. By further optimizing the image based on the target image, the defects that may occur in the image processing process are compensated, and the appearance and authenticity of the image are improved.

[0085] In one embodiment, the image is processed using the image processing method in the embodiment of the present disclosure, for example, a denoising process is performed on an image collected in an extremely dark scene with a light intensity of <= 0.1 lux in the environment.

[0086] The present disclosure embodiment will now illustrate the above-mentioned image processing method with examples. In the following example, an extremely dark scene with a light intensity of <= 0.1 lux in the environment is used as the image acquisition scene, and the main frame I base As the first image, the registered I1~I6 are used as the second image sequence, and the denoised main frame image I dn As the second image. as the third image.

[0087] Figure 7 The figure is a schematic diagram showing a process of denoising an extremely dark scene image according to an exemplary embodiment.

[0088] In the embodiment of the present disclosure, in an extremely dark scene, a multi-frame image sequence is collected under the conditions of a longer exposure time and a higher sensitivity ISO. For example, the exposure time is set to 330ms and the sensitivity ISO analog gain is set to full, and 6 frames of images are collected.

[0089] The image sequence is preprocessed, and the preprocessing process may include: using the Sobel gradient operator to select the image with the highest definition as the main frame. base After that, the main frame and the other five frames are input into the pre-trained alignment network for image registration, and the five registered frames I2 to I6 and their corresponding confidence scores are obtained. If the confidence score is less than the preset threshold, it means that the registration fails, and the current frame needs to be discarded and the main frame replaces the current image.

[0090] In the embodiment of the present disclosure, the main frame is subjected to noise removal, and a pre-trained image denoising network is used to process the main frame to obtain a denoised main frame image I dn Multiply the registered main frame and other images in the image sequence by the exposure compensation parameter to obtain the image sequence after exposure compensation. Use the multi-frame image denoising network to denoise the exposure compensated image sequence to obtain the denoised image after exposure compensation

[0091] According to the denoised image Generate a binary mask image I mask , where pixels with a signal value greater than 0.99 are 1, and pixels with a signal value less than or equal to 0.99 are 0. Perform inverse exposure compensation. The result after inverse exposure compensation is fused with the original denoised image sequence according to the binary mask image to obtain the final target image I restore .

[0092] In the embodiment of the present disclosure, by adding simulated noise, the target image is backfilled with noise to reduce the smearing effect of the restored image. The noise backfilling can be performed in the following manner:

[0093]

[0094] Among them, K a Indicates the analog gain at the current sensitivity ISO. represents Poisson distribution, N g Indicates the Gaussian noise at the current sensitivity ISO. a and N g It only needs to be calibrated once before it can be used directly.

[0095] In the disclosed embodiment, the details of the target image are enhanced by edge detection, sharpening and other technologies, thereby improving the visual quality of the target image.

[0096] In the disclosed embodiment, a multi-frame image sequence is collected, and the main frame with the highest definition is selected as the reference for image registration. Through exposure compensation and denoising network processing, image noise removal and highlight information retention are achieved; finally, through inverse exposure compensation and exposure fusion, the final denoised image is obtained, which can maintain high image accuracy and clarity in extremely dark scenes, while effectively removing noise in the image.

[0097] Based on the same concept, an embodiment of the present disclosure also provides an image processing device.

[0098] It is understandable that in order to realize the above functions, the image processing device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0099] Figure 8 FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment. Figure 8 , the device includes a determining unit 101 and a processing unit 102.

[0100] A determining unit 101 is configured to determine a second image sequence according to a first image sequence, wherein the first image sequence includes multiple frames of images of the same picture acquired in a scene where the ambient light brightness is lower than a threshold, the second image sequence is an image sequence obtained after registering the first image and the remaining images, the first image is an image with the highest definition among the multiple frames of images, and the remaining images are other images in the first image sequence except the first image;

[0101] The processing unit 102 is configured to determine a second image and a third image according to the first image and the second image sequence; and determine a target image according to the second image and the third image.

[0102] In one embodiment, the first image is determined in the following manner, including: using an edge detection operator to determine the horizontal gradient and the vertical gradient of each frame of the multiple frames, and determining the global gradient mean based on the horizontal gradient and the vertical gradient; determining the first image based on the global gradient mean and the shooting parameters of each frame of the multiple frames.

[0103] In one embodiment, the processing unit 102 determines the second image and the third image based on the first image and the second image sequence in the following manner: denoising the first image to obtain the second image; performing exposure compensation on each image in the second image sequence based on an exposure compensation parameter to obtain an image sequence after exposure compensation, where the exposure compensation parameter is a ratio between an expected grayscale value and an actual grayscale value of any frame image in multiple frames of images; denoising the image sequence after exposure compensation to obtain a third image.

[0104] In one embodiment, the processing unit 102 determines the target image based on the second image and the third image in the following manner: binarizing the image in the third image to obtain a binarized mask image, the binarized mask image is used to distinguish between image areas with brightness greater than a threshold and image areas with brightness less than or equal to the threshold; based on the binarized image, performing inverse exposure compensation on the second image and the third image, and fusing the inverse exposure compensated second image and third image to obtain the target image.

[0105] In one embodiment, the processing unit 102 performs reverse exposure compensation on the second image and the third image based on the binarized image in the following manner: extracting an image area in the second image whose brightness is greater than a threshold value through a binary mask image; performing reverse exposure compensation on the third image based on an exposure compensation parameter to obtain a reverse exposure compensated image; and extracting an image area in the reverse exposure compensated image whose brightness is less than or equal to the threshold value through the complement of the binary mask image.

[0106] In one embodiment, the processing unit 102 is further used to: perform post-processing on the target image, the post-processing comprising at least one of the following: filling simulated noise into the target image; detecting a target area in the target image where the grayscale value change is greater than a threshold, and sharpening the target area.

[0107] Fig. 9 2 is a block diagram of an apparatus 200 for image processing according to an exemplary embodiment. For example, the apparatus 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0108] Reference Fig. 9 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0109] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0110] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0111] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0112] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0113] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0114] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0115] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0116] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0117] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0118] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0119] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0120] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0121] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0122] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0123] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0124] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. An image processing method, characterized in that: include: Determine a second image sequence according to a first image sequence, wherein the first image sequence includes multiple frames of images of the same picture acquired in a scene where the ambient light brightness is lower than a threshold, the second image sequence is an image sequence obtained after registering the first image and the remaining images, the first image is an image with the highest definition among the multiple frames of images, and the remaining images are other images in the first image sequence except the first image; Determine a second image and a third image according to the first image and the second image sequence; A target image is determined according to the second image and the third image.

2. The image processing method according to claim 1, characterized in that: The first image is determined by: Using an edge detection operator, determine the horizontal gradient and the vertical gradient of each frame of the multiple frames of images, and determine the global gradient mean based on the horizontal gradient and the vertical gradient; The first image is determined based on the global gradient mean and the shooting parameters of each frame of the multiple frames of images.

3. The image processing method according to claim 1, characterized in that: The determining the second image and the third image according to the first image and the second image sequence comprises: Denoising the first image to obtain the second image; performing exposure compensation on each image in the second image sequence based on an exposure compensation parameter to obtain an image sequence after exposure compensation, wherein the exposure compensation parameter is a ratio between an expected grayscale value and an actual grayscale value of any frame of the multiple images; De-noising the exposure-compensated image sequence to obtain the third image.

4. The image processing method according to claim 1, characterized in that: The determining the target image according to the second image and the third image comprises: Binarizing the image in the third image to obtain a binary mask image, wherein the binary mask image is used to distinguish between an image area whose brightness is greater than a threshold and an image area whose brightness is less than or equal to the threshold; Based on the binarized image, reverse exposure compensation is performed on the second image and the third image, and the reverse exposure compensated second image and third image are fused to obtain a target image.

5. The image processing method according to claim 4, characterized in that: The performing inverse exposure compensation on the second image and the third image based on the binary image includes: Extracting an image area in the second image whose brightness is greater than a threshold value by using the binary mask image; Performing inverse exposure compensation on the third image based on the exposure compensation parameter to obtain an image after inverse exposure compensation; An image region whose brightness is less than or equal to the threshold value in the image after reverse exposure compensation is extracted through the complement of the binary mask image.

6. The image processing method according to claim 1, characterized in that: The image processing method further comprises: Post-processing the target image, wherein the post-processing includes at least one of the following: Filling simulated noise into the target image; Detect a target area in the target image whose grayscale value variation is greater than a threshold, and sharpen the target area.

7. An image processing device, characterized in that: include: a determining unit, configured to determine a second image sequence according to a first image sequence, wherein the first image sequence includes multiple frames of images of the same picture acquired in a scene where the ambient light brightness is lower than a threshold, the second image sequence is an image sequence obtained after registering the first image and the remaining images, the first image is an image with the highest definition among the multiple frames of images, and the remaining images are other images in the first image sequence except the first image; A processing unit is used to determine a second image and a third image according to the first image and the second image sequence; and to determine a target image according to the second image and the third image.

8. The image processing device according to claim 7, characterized in that: The first image is determined by: Using an edge detection operator, determine the horizontal gradient and the vertical gradient of each frame of the multiple frames of images, and determine the global gradient mean based on the horizontal gradient and the vertical gradient; The first image is determined based on the global gradient mean and the shooting parameters of each frame of the multiple frames of images.

9. The image processing device according to claim 7, characterized in that: The processing unit determines the second image and the third image according to the first image and the second image sequence in the following manner: Denoising the first image to obtain the second image; performing exposure compensation on each image in the second image sequence based on an exposure compensation parameter to obtain an image sequence after exposure compensation, wherein the exposure compensation parameter is a ratio between an expected grayscale value and an actual grayscale value of any frame of the multiple images; De-noising the exposure-compensated image sequence to obtain the third image.

10. The image processing device according to claim 7, characterized in that: The processing unit determines the target image according to the second image and the third image in the following manner: Binarizing the image in the third image to obtain a binary mask image, wherein the binary mask image is used to distinguish between an image area whose brightness is greater than a threshold and an image area whose brightness is less than or equal to the threshold; Based on the binarized image, reverse exposure compensation is performed on the second image and the third image, and the reverse exposure compensated second image and third image are fused to obtain a target image.

11. The image processing device according to claim 10, characterized in that: The processing unit performs inverse exposure compensation on the second image and the third image based on the binarized image in the following manner: Extracting an image area in the second image whose brightness is greater than a threshold value by using the binary mask image; Performing inverse exposure compensation on the third image based on the exposure compensation parameter to obtain an image after inverse exposure compensation; An image region whose brightness is less than or equal to the threshold value in the image after reverse exposure compensation is extracted through the complement of the binary mask image.

12. The image processing device according to claim 7, characterized in that: The processing unit is further configured to perform post-processing on the target image, wherein the post-processing includes at least one of the following: Filling simulated noise into the target image; Detect a target area in the target image whose grayscale value variation is greater than a threshold, and sharpen the target area.

13. An image processing device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the method described in any one of claims 1 to 6.

14. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the method according to any one of claims 1 to 6.

Images