An image processing method, apparatus and storage medium

Abstract

This disclosure relates to an image processing method, apparatus, and storage medium. The image processing method includes: determining a second image sequence based on a first image sequence, wherein the first image sequence includes multiple frames of the same scene captured in a scene where the ambient light intensity is below a threshold, and the second image sequence is an image sequence obtained by registering the first image and remaining images, wherein the first image is the image with the highest clarity among the multiple frames, and the remaining images are all images in the first image sequence other than the first image; determining a second image and a third image based on the first image and the second image sequence; and determining a target image based on the second image and the third image. This disclosure enables effective noise reduction of images in scenes where the ambient light intensity is below a threshold.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and more particularly to an image processing method, apparatus, and storage medium. Background Technology

[0002] Image noise is a prevalent problem in image processing. In extremely dark scenes, the amount of light entering the image is very low, making the impact of image noise even more significant. Therefore, higher precision is required for denoising methods in such scenarios. However, the accuracy of current image denoising techniques needs further improvement. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this disclosure provides an image processing method, apparatus and storage medium.

[0004] According to a first aspect of the present disclosure, an image processing method is provided, the image processing method comprising: determining a second image sequence based on a first image sequence, wherein the first image sequence includes multiple frames of the same scene captured in a scene where the ambient light intensity is below a threshold, the second image sequence being an image sequence obtained by registering the first image and the remaining images, the first image being the image with the highest clarity among the multiple frames, and the remaining images being other images in the first image sequence besides the first image; determining a second image and a third image based on the first image and the second image sequence; and determining a target image based on the second image and the third image.

[0005] In one embodiment, the first image is determined by: using an edge detection operator to determine the horizontal and vertical gradients of each frame in the multi-frame images, and determining a global gradient mean based on the horizontal and vertical gradients; and determining the first image based on the global gradient mean and the shooting parameters of each frame in the multi-frame images.

[0006] In one embodiment, determining the second image and the third image based on the first image and the second image sequence includes: denoising the first image to obtain the second image; performing exposure compensation on each image in the second image sequence based on an exposure compensation parameter to obtain an exposure-compensated image sequence, wherein the exposure compensation parameter is the ratio between the expected grayscale value and the actual grayscale value of any frame in the multi-frame image; and denoising the exposure-compensated image sequence to obtain the third image.

[0007] In one embodiment, determining the target image based on the second image and the third image includes: performing binarization processing on the image in the third image to obtain a binarized mask image, wherein the binarized mask image is used to distinguish image regions with brightness greater than a threshold and image regions with brightness less than or equal to the threshold; performing inverse exposure compensation on the second image and the third image based on the binarized image, and fusing the inversely exposed second image and the third image to obtain the target image.

[0008] In one embodiment, the step of performing inverse exposure compensation on the second image and the third image based on the binarized image includes: extracting image regions in the second image whose brightness is greater than a threshold using the binarized mask image; performing inverse exposure compensation on the third image based on the exposure compensation parameters to obtain an image after inverse exposure compensation; and extracting image regions in the image after inverse exposure compensation whose brightness is less than or equal to the threshold using the complement of the binarized mask image.

[0009] In one embodiment, the image processing method further includes: post-processing the target image, the post-processing including at least one of the following: filling the target image with simulated noise; detecting target regions in the target image whose grayscale value changes are greater than a threshold, and sharpening the target regions.

[0010] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising: a determining unit, configured to determine a second image sequence based on a first image sequence, wherein the first image sequence includes multiple frames of the same scene captured in a scene where the ambient light intensity is below a threshold, the second image sequence being an image sequence obtained by registering the first image and remaining images, the first image being the image with the highest clarity among the multiple frames, and the remaining images being other images in the first image sequence besides the first image; and a processing unit, configured to determine a second image and a third image based on the first image and the second image sequence; and to determine a target image based on the second image and the third image.

[0011] In one embodiment, the first image is determined as follows: using an edge detection operator, the horizontal and vertical gradients of each frame in the multi-frame images are determined, and based on the horizontal and vertical gradients, a global gradient mean is determined; based on the global gradient mean and the shooting parameters of each frame in the multi-frame images, the first image is determined.

[0012] In one embodiment, the processing unit determines the second image and the third image based on the first image and the second image sequence as follows: denoising the first image to obtain the second image; performing exposure compensation on each image in the second image sequence based on an exposure compensation parameter to obtain an exposure-compensated image sequence, wherein the exposure compensation parameter is the ratio between the expected grayscale value and the actual grayscale value of any frame in the multi-frame image; and denoising the exposure-compensated image sequence to obtain the third image.

[0013] In one embodiment, the processing unit determines the target image based on the second image and the third image in the following manner: performing binarization processing on the image in the third image to obtain a binarized mask image, the binarized mask image being used to distinguish image regions with brightness greater than a threshold and image regions with brightness less than or equal to the threshold; based on the binarized image, performing inverse exposure compensation on the second image and the third image, and fusing the inverse exposure compensated second image and third image to obtain the target image.

[0014] In one embodiment, the processing unit performs inverse exposure compensation on the second image and the third image based on the binarized image in the following manner: extracting image regions in the second image with brightness greater than a threshold using the binarized mask image; performing inverse exposure compensation on the third image based on the exposure compensation parameters to obtain an image after inverse exposure compensation; and extracting image regions in the image after inverse exposure compensation with brightness less than or equal to the threshold using the complement of the binarized mask image.

[0015] In one embodiment, the processing unit is further configured to perform post-processing on the target image, the post-processing including at least one of the following: filling the target image with simulated noise; detecting target regions in the target image whose grayscale value changes are greater than a threshold, and sharpening the target regions.

[0016] According to a third aspect of this disclosure, an image processing apparatus is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the image processing method described in the first aspect or any embodiment of the first aspect.

[0017] According to a fourth aspect of this disclosure, a storage medium is provided that stores instructions, which, when executed by a processor, perform the image processing method described in the first aspect or any embodiment of the first aspect.

[0018] The technical solutions provided by the embodiments of this disclosure can include the following beneficial effects: by determining the image with the highest sharpness as the first image and performing image registration to obtain the second image sequence, and then performing image denoising on the first image and the second image sequence respectively, noise is effectively removed while preserving the details and dynamic range of the image, thereby improving image quality. Simultaneously, by fusing the second and third images, the accuracy loss and information loss in some areas that may be introduced by directly increasing image brightness are reduced.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0021] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0022] Figure 2 This is a flowchart illustrating a method for determining a first image according to an exemplary embodiment.

[0023] Figure 3 This is a flowchart illustrating an image denoising method according to an exemplary embodiment.

[0024] Figure 4 This is a flowchart illustrating an exposure fusion method according to an exemplary embodiment.

[0025] Figure 5 This is a flowchart illustrating an inverse exposure compensation method according to an exemplary embodiment.

[0026] Figure 6 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0027] Figure 7 This is a schematic diagram illustrating a denoising process for an extremely dark scene image according to an exemplary embodiment.

[0028] Figure 8 This is a block diagram of an image processing apparatus according to an exemplary embodiment.

[0029] Figure 9 This is a block diagram illustrating an apparatus for image processing according to an exemplary embodiment. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure.

[0031] The image processing method provided in this disclosure can be applied to scenarios where the ambient light intensity is below a threshold for image processing. For example, it can be used to denoise images acquired in extremely dark scenes where the ambient light intensity is <= 0.1 lux.

[0032] In related technologies, directly denoising images in extremely dark scenes results in significant accuracy loss and fails to effectively preserve local information within the image. On the other hand, to expand the image signal range, a common approach is to directly increase the brightness of the original image, which can expand the signal range to some extent. However, since the signal values ​​of highlight areas with brightness exceeding a set threshold are already close to or have reached the maximum value of the image, this direct brightness enhancement leads to information loss in the highlight areas and amplification of noise.

[0033] In view of this, embodiments of the present disclosure provide an image processing method. Figure 1 This is a flowchart illustrating an image method according to an exemplary embodiment.

[0034] In step S11, the second image sequence is determined based on the first image sequence.

[0035] The first image sequence includes multiple frames of the same scene captured in a scene where the ambient light is below a threshold. The second image sequence is an image sequence obtained by registering the first image and the remaining images. The first image is the image with the highest clarity among the multiple frames, and the remaining images are the other images in the first image sequence besides the first image.

[0036] Scenes where the ambient light intensity is below a threshold can include extremely dark scenes where the ambient light intensity is <= 0.1 lux. Furthermore, the first image sequence can be an original image sequence captured by the terminal device through an image acquisition device, or it can be an image sequence obtained from other terminal devices.

[0037] Determining the image with the highest resolution among multiple frames as the reference image and registering it to obtain the second image sequence can improve the accuracy and stability of the registration.

[0038] In step S12, the second image and the third image are determined based on the sequence of the first image and the second image.

[0039] In this embodiment of the disclosure, image processing is performed on the first image and the second image sequence to obtain the second image and the third image. The image processing may involve different denoising processes on the first image and the second image sequence, thereby effectively removing noise from different regions of the second image and the third image while maintaining accuracy.

[0040] In step S13, the target image is determined based on the second image and the third image.

[0041] The fusion process involves combining the denoised second image with the third image. Specifically, this can be achieved by extracting and fusing different regions in the second and third images where noise has been effectively removed. For example, pixel values ​​from highlight areas in the second image with brightness exceeding a set threshold and pixel values ​​from non-highlight areas in the third image with brightness less than or equal to the set threshold. This approach preserves detail in highlight areas while performing exposure compensation in non-highlight areas, resulting in better image quality and detail retention.

[0042] In the image processing method provided in this embodiment, multiple frames of images are acquired in a scene where the ambient light intensity is below a threshold. After image registration is performed using the image with the highest clarity among the multiple frames as a reference, the images are then denoised. After obtaining the denoised second and third images, images of different regions in the second and third images where noise has been effectively removed are extracted and fused to obtain the target image. This method effectively removes image noise without sacrificing accuracy and retains information about highlight areas.

[0043] Figure 2 This is a flowchart illustrating a method for determining a first image according to an exemplary embodiment.

[0044] In step S21, the edge detection operator is used to determine the horizontal and vertical gradients of each frame in the multi-frame images, and the global gradient mean is determined based on the horizontal and vertical gradients.

[0045] In this embodiment of the disclosure, the global gradient can be determined using the Sobel gradient operator, a commonly used image edge detection operator used to calculate the gradient intensity and gradient direction of each pixel in the image.

[0046] Edge detection operators can be used to determine the location of edges in an image; these operators are typically calculated based on the grayscale values ​​of pixels in the image. In this embodiment of the disclosure, other edge detection operators can also be used to determine the global gradient.

[0047] For example, the Prewitt operator is used to determine the global gradient by calculating the horizontal and vertical gradients of each pixel in the image. The Roberts operator, which uses a 2×2 template for gradient calculation, is more suitable for smaller images.

[0048] In step S22, the first image is determined based on the global gradient mean and the shooting parameters of each frame in the multi-frame images.

[0049] The mean of the global gradient calculates the gradient information of the entire image and can be used to comprehensively evaluate image sharpness. A larger mean indicates that there is more edge and detail information in the image, meaning the image is relatively sharp. A smaller mean indicates that there is less edge and detail information in the image, meaning the image is relatively blurry. Simultaneously, the imaging device usually records some parameter information related to image capture, such as exposure time, ISO, and focal length. This parameter information can provide some indications about image quality and sharpness.

[0050] Therefore, when selecting a clear frame, the shooting parameter information can also be taken into account to determine the first image.

[0051] For example, a weighted function for a sharpness index can be defined, which combines the image sharpness score with the shooting parameters in a weighted manner. The first image is then determined based on the weighted score.

[0052] In this embodiment of the disclosure, the first image sequence is acquired under set acquisition parameters, which include, but are not limited to, exposure time and / or ISO sensitivity. For example, in order to receive light for a longer period of time and thus obtain more photon signals, a longer exposure time is set and / or the ISO sensitivity is maximized.

[0053] In the method for determining the first image provided in this disclosure, by selecting the image with the highest sharpness as the first image, the detail information of the image can be preserved to the greatest extent, and image noise caused by motion blur or other factors can be reduced. The image with the highest sharpness can provide a more accurate reference, which helps to improve the accuracy and stability of subsequent image registration and processing steps. Therefore, determining the image with the highest sharpness as the first image has a significant impact on the quality and accuracy of the entire image processing workflow.

[0054] Figure 3 This is a flowchart illustrating an image denoising method according to an exemplary embodiment.

[0055] In step S31, the first image is denoised to obtain the second image.

[0056] The first image is denoised to obtain the second image. The first image is selected as the one with the highest clarity. Denoising the first image preserves the detail information in the highlight areas of the original image.

[0057] In step S32, exposure compensation is performed on each image in the second image sequence based on the exposure compensation parameters to obtain an exposure-compensated image sequence.

[0058] After obtaining the second image sequence through image registration, an exposure compensation step is implemented to reduce the accuracy loss after denoising caused by extremely low signal intensity in scenes where ambient light levels are below a threshold. Exposure compensation adjusts the image brightness by multiplying each frame in the image sequence by a pre-calculated exposure compensation parameter. The exposure compensation parameter is the ratio between the expected grayscale value and the actual grayscale value of any frame in the multi-frame image sequence.

[0059] In step S33, the image sequence after exposure compensation is denoised to obtain the third image.

[0060] By using exposure compensation, the problem of significant accuracy loss in image denoising caused by the concentration of image signal values ​​within a small range due to extremely low scene illumination is solved. Image denoising is then performed on the exposure-compensated result, yielding a third image where noise in the low-brightness areas of the original image has been effectively removed. One method for denoising the exposure-compensated image sequence is to input the exposure compensation result into a multi-frame image denoising network. During the training phase, simulated noise data can be used to train the image denoising network, enabling it to effectively remove noise from the image and improve image clarity and quality.

[0061] In the image denoising method provided in this disclosure, a second image is obtained by performing image denoising processing on a first image, which can effectively denoise the highlight areas of the first image. Exposure compensation is applied to each image in the second image sequence to adjust the brightness of each image frame, increasing the visibility of details in lower-brightness areas of the original image, better removing noise from the image, and improving image quality and detail preservation. Furthermore, by performing image denoising processing on each image in the second image sequence to obtain a third image, the repetitive information present in the multi-frame image sequence can be fully utilized, reducing the impact of additional noise that exposure compensation may introduce.

[0062] In this embodiment of the disclosure, since the images in the second image sequence may have some highlight areas, after exposure compensation, these areas may be overexposed due to excessive brightness. This overexposure problem will cause the loss of details and information in the highlight areas of the third image after image denoising processing.

[0063] In view of this, embodiments of this disclosure provide an exposure fusion method. In the exposure fusion method provided in this disclosure, a binary mask image is used to distinguish image regions with brightness greater than a threshold and image regions with brightness less than or equal to the threshold, and inverse exposure compensation is performed. Then, the image regions with brightness greater than the threshold whose noise has been effectively removed in the second image and the image regions with brightness less than or equal to the threshold whose noise has been effectively removed in the inverse exposure compensated third image are fused to obtain a target image.

[0064] Figure 4 This is a flowchart illustrating an exposure fusion method according to an exemplary embodiment.

[0065] In step S41, the image in the third image is binarized to obtain a binarized mask image. The binarized mask image is used to distinguish image regions with brightness greater than a threshold and image regions with brightness less than or equal to the threshold.

[0066] A binary mask is a technique in image processing used to represent pixels in a specific region of an image. A binary mask is a two-dimensional array of the same size as the original image, where each element has only two possible values, typically 0 and 1. In this embodiment, pixel values ​​with image brightness greater than a set threshold are recorded as 1, and pixel values ​​with image brightness less than or equal to the set threshold are recorded as 0. The set threshold can be determined by normalizing the image signal values ​​and then setting it based on the current scene illumination. For example, in an extremely dark scene, the set threshold could be 0.99.

[0067] In step S42, based on the binarized image, inverse exposure compensation is performed on the second and third images, and the inverse exposure compensated second and third images are fused to obtain the target image.

[0068] In this embodiment, the third image after exposure compensation is thresholded to obtain a binary mask image that identifies the bright and non-bright areas. Furthermore, during the fusion process, regions in both the second and third images where noise has been effectively removed are preserved. Through the exposure fusion method in this embodiment, the bright areas in the second image with effectively removed noise can be effectively fused with the non-bright areas in the third image after exposure compensation, resulting in a final target image that better restores image details and dynamic range, achieving a better image denoising effect.

[0069] Figure 5 This is a flowchart illustrating an inverse exposure compensation method according to an exemplary embodiment.

[0070] In step S51, the image region with brightness greater than the threshold in the second image is extracted by using a binarized mask image.

[0071] In this embodiment of the disclosure, multiplying the second image with the binarized mask image achieves the preservation of the bright areas in the second image where noise has been effectively removed.

[0072] In step S52, inverse exposure compensation is performed on the third image based on the exposure compensation parameters to obtain the inverse exposure compensated image.

[0073] Inverse exposure compensation involves dividing the signal value in the third image after exposure compensation processing by the exposure compensation parameter to inversely compensate for its exposure gain effect, so that the image is restored to the original exposure level.

[0074] In step S53, the image regions with brightness less than or equal to a threshold in the image after inverse exposure compensation are extracted by using the complement of the binarized mask image.

[0075] By multiplying the image after inverse exposure compensation with the complement of the binary mask, the non-highlighted areas in the third image with effectively removed noise are extracted.

[0076] In this embodiment, the third image is restored to its original exposure level by inverse exposure compensation, and the bright areas in the second image and the non-bright areas in the third image are clearly separated by a binary mask image. Combined with subsequent fusion operations, image noise removal can be achieved without loss of accuracy, while preserving the image highlight information.

[0077] Figure 6 This is a flowchart illustrating an image processing method according to an exemplary embodiment.

[0078] In this embodiment of the disclosure, such as Figure 6 As shown, multiple frames of images were first acquired when the ambient light level was below a threshold. An edge detection operator was used to calculate the horizontal and vertical gradients of each frame, and the global gradient mean was determined based on these gradients. Combining the global gradient mean and the shooting parameters of each frame, the image with the highest sharpness was selected as the first image, and image registration was performed to obtain the second image sequence.

[0079] The first image is denoised to obtain the second image, and the second image sequence is then denoised after exposure compensation to obtain the third image.

[0080] A binarized mask image is obtained by binarizing the third image to distinguish image regions with brightness greater than a threshold and those with brightness less than or equal to the threshold. Inverse exposure compensation is performed on the third image, and then the second image and the inversely exposed third image are fused based on the binarized image to obtain the target image. In the processed image, due to the denoising operation, a certain smearing effect may occur, meaning the image appears too smooth or lacks detail. In view of this, embodiments of this disclosure provide an image post-processing method. In the image post-processing method provided in embodiments of this disclosure, image post-processing may include filling the target image with simulated noise and / or detecting regions in the target image with drastic changes in grayscale values, and sharpening the target image based on the detection results.

[0081] The simulated noise added to the target image can be Poisson noise, Gaussian noise, etc. The Poisson distribution is a statistical distribution commonly used to simulate image noise. Dividing the target image by the simulated gain and then applying the Poisson distribution generates noise of a certain intensity, which is then added to the target image. Gaussian noise is a common type of random noise; it exhibits Gaussian distribution characteristics and has a corresponding relationship with ISO sensitivity. This is done to reduce the smearing effect, making the image look more realistic and natural.

[0082] In this embodiment of the disclosure, an edge detection operator can also be used to detect regions with drastic changes in grayscale values, i.e., high-frequency texture information, and then the image can be sharpened based on the detection results. The sharpening operation can enhance the edges and details of the image, making the image look clearer and more detailed.

[0083] An edge detection operator is a filter used to detect edge or texture features in an image. It determines the location of boundaries in an image by calculating the gradient or second derivative of pixel values.

[0084] In the image post-processing method of this disclosure, by filling the target image with simulated noise and / or detecting regions in the target image with drastic changes in grayscale values, and sharpening the target image based on the detection results, the image quality can be further improved, the visual effect enhanced, and the image made more consistent with human visual perception. By further optimizing the image based on the target image, defects that may occur during image processing are compensated for, and the visual appeal and realism of the image are improved.

[0085] In one embodiment, an image is processed using an image processing method according to the present disclosure. For example, noise reduction processing is performed on an image acquired in an extremely dark scene where the ambient light intensity is <= 0.1 lux.

[0086] The image processing method described above will be illustrated below with reference to embodiments of this disclosure. In the following examples, an extremely dark scene with an ambient light intensity <= 0.1 lux is used as the image acquisition scene, and the main frame I... base As the first image, the registered I1 to I6 are used as the second image sequence, with the denoised main frame image I... dn As the second image. The image after denoising. As a third image.

[0087] Figure 7 This is a schematic diagram illustrating a denoising process for an extremely dark scene image according to an exemplary embodiment.

[0088] In this embodiment of the disclosure, a multi-frame image sequence is acquired under extremely dark scene conditions with a long exposure time and a high ISO sensitivity. For example, an exposure time of 330ms and an ISO sensitivity with the analog gain set to maximum are used to acquire 6 frames of images.

[0089] The image sequence undergoes preprocessing, which may include selecting the image with the highest sharpness as the main frame using the Sobel gradient operator. The main frame I is selected. base Next, the main frame and the other 5 frames are input into a pre-trained alignment network for image registration, resulting in 5 registered frames I2 to I6 and their corresponding confidence scores. If the confidence score is less than a preset threshold, the registration has failed, and the current frame needs to be discarded and the main frame replaced.

[0090] In this embodiment of the disclosure, noise removal is performed on the main frame by using a pre-trained image denoising network to process the main frame, resulting in a denoised main frame image I. dn The registered main frame and other images in the image sequence are multiplied by the exposure compensation parameter to obtain the exposure-compensated image sequence. A multi-frame image denoising network is used to denoise the exposure-compensated image sequence, resulting in a denoised image after exposure compensation.

[0091] Based on the denoised image Generate binarized mask image I mask Pixels with signal values ​​greater than 0.99 are assigned a value of 1, and pixels with a value less than or equal to 0.99 are assigned a value of 0. This applies to the denoised image after exposure compensation. Perform inverse exposure compensation. Then, fuse the inverse exposure compensation result with the original denoised image sequence using a binary mask image to obtain the final target image I. restore .

[0092] In this embodiment of the disclosure, noise backfilling is performed on the target image by simulating the addition of noise, thereby reducing the smearing appearance of the restored image. The noise backfilling can be performed in the following ways:

[0093]

[0094] Among them, K a This indicates the analog gain at the current ISO sensitivity. Represents a Poisson distribution, N g This represents the Gaussian noise at the current ISO sensitivity. K a and N g It can be used directly after only one calibration.

[0095] In this embodiment of the disclosure, edge detection and sharpening techniques are used to enhance the details of the target image and improve its visual quality.

[0096] In this embodiment, a multi-frame image sequence is acquired, and the main frame with the highest clarity is selected as the reference for image registration. Through exposure compensation and denoising network processing, image noise removal and highlight information preservation are achieved. Finally, through inverse exposure compensation and exposure fusion, the final denoised image is obtained, which can maintain high precision and clarity in extremely dark scenes while effectively removing noise from the image.

[0097] Based on the same concept, embodiments of this disclosure also provide an image processing apparatus.

[0098] It is understood that the image processing apparatus provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0099] Figure 8 This is a block diagram of an image processing apparatus according to an exemplary embodiment. (Refer to...) Figure 8 The device includes a determining unit 101 and a processing unit 102.

[0100] The determining unit 101 is used to determine the second image sequence based on the first image sequence. The first image sequence includes multiple frames of the same scene captured in a scene where the ambient light is below a threshold. The second image sequence is an image sequence obtained by registering the first image and the remaining images. The first image is the image with the highest clarity among the multiple frames, and the remaining images are other images in the first image sequence besides the first image.

[0101] The processing unit 102 is configured to determine a second image and a third image based on the first image and the second image sequence; and to determine a target image based on the second image and the third image.

[0102] In one embodiment, the first image is determined by: using an edge detection operator to determine the horizontal and vertical gradients of each frame in the multi-frame images, and determining the global gradient mean based on the horizontal and vertical gradients; and determining the first image based on the global gradient mean and the shooting parameters of each frame in the multi-frame images.

[0103] In one embodiment, the processing unit 102 determines the second image and the third image based on the first image and the second image sequence in the following manner: denoising the first image to obtain the second image; performing exposure compensation on each image in the second image sequence based on an exposure compensation parameter to obtain an exposure-compensated image sequence, wherein the exposure compensation parameter is the ratio between the expected grayscale value and the actual grayscale value of any frame in the multi-frame image; and denoising the exposure-compensated image sequence to obtain the third image.

[0104] In one embodiment, the processing unit 102 determines the target image based on the second image and the third image in the following manner: the image in the third image is binarized to obtain a binarized mask image, which is used to distinguish image regions with brightness greater than a threshold and image regions with brightness less than or equal to the threshold; based on the binarized image, the second image and the third image are inversely exposed and compensated, and the inversely exposed and compensated second image and the third image are fused to obtain the target image.

[0105] In one embodiment, the processing unit 102 performs inverse exposure compensation on the second and third images based on the binarized image in the following manner: extracting the image region with brightness greater than a threshold in the second image through the binarized mask image; performing inverse exposure compensation on the third image based on the exposure compensation parameters to obtain the inverse exposure compensated image; and extracting the image region with brightness less than or equal to the threshold in the inverse exposure compensated image through the complement of the binarized mask image.

[0106] In one embodiment, the processing unit 102 is further configured to: perform post-processing on the target image, the post-processing including at least one of the following: filling the target image with simulated noise; detecting a target region in the target image whose grayscale value change is greater than a threshold, and sharpening the target region.

[0107] Figure 9 This is a block diagram illustrating an apparatus 200 for image processing according to an exemplary embodiment. For example, apparatus 200 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0108] Reference Figure 9 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0109] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0110] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0111] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0112] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0113] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0114] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0115] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0116] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0117] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0118] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0119] It is understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0120] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0121] It can be further understood that, unless otherwise specified, "connection" includes both direct connections where no other components exist between the two parties and indirect connections where other components exist between them.

[0122] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0123] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0124] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, include: A second image sequence is determined based on a first image sequence. The first image sequence includes multiple frames of the same scene captured in a scene where the ambient light is below a threshold. The second image sequence is an image sequence obtained by registering the first image and the remaining images. The first image is the image with the highest clarity among the multiple frames, and the remaining images are the other images in the first image sequence besides the first image. The first image is denoised to obtain the second image; For each image in the second image sequence, exposure compensation is performed based on the exposure compensation parameter to obtain an exposure-compensated image sequence. The exposure compensation parameter is the ratio between the expected gray value and the actual gray value of any frame in the multi-frame images. The exposure-compensated image sequence is denoised to obtain a third image; The image in the third image is binarized to obtain a binarized mask image. The binarized mask image is used to distinguish image regions with brightness greater than a threshold and image regions with brightness less than or equal to the threshold. The image regions with brightness greater than a threshold in the second image are extracted using the binarized mask image; Based on the exposure compensation parameters, the third image is subjected to inverse exposure compensation to obtain an image after inverse exposure compensation. The image regions with brightness less than or equal to the threshold in the inverse exposure compensated image are extracted by complementing the binarized mask image; The target image is obtained by fusing the image regions in the second image whose brightness is greater than the threshold and the image regions in the third image after inverse exposure compensation whose brightness is less than or equal to the threshold.

2. The image processing method according to claim 1, characterized in that, The first image is determined in the following ways: Using an edge detection operator, the horizontal and vertical gradients of each frame in the multi-frame images are determined, and the global gradient mean is determined based on the horizontal and vertical gradients. The first image is determined based on the global gradient mean and the shooting parameters of each frame in the multi-frame images.

3. The image processing method according to claim 1, characterized in that, The image processing method further includes: The target image is post-processed, and the post-processing includes at least one of the following: Simulated noise is filled into the target image; Detect target regions in the target image whose grayscale value changes are greater than a threshold, and sharpen the target regions.

4. An image processing apparatus, characterized in that, include: The determining unit is configured to determine a second image sequence based on a first image sequence. The first image sequence includes multiple frames of the same scene captured in a scene where the ambient light is below a threshold. The second image sequence is an image sequence obtained by registering the first image and the remaining images. The first image is the image with the highest clarity among the multiple frames, and the remaining images are other images in the first image sequence besides the first image. A processing unit is used to denoise the first image to obtain a second image; For each image in the second image sequence, exposure compensation is performed based on the exposure compensation parameter to obtain an exposure-compensated image sequence. The exposure compensation parameter is the ratio between the expected gray value and the actual gray value of any frame in the multi-frame images. The exposure-compensated image sequence is denoised to obtain a third image; The image in the third image is binarized to obtain a binarized mask image. The binarized mask image is used to distinguish image regions with brightness greater than a threshold and image regions with brightness less than or equal to the threshold. The image regions with brightness greater than a threshold in the second image are extracted using the binarized mask image; Based on the exposure compensation parameters, the third image is subjected to inverse exposure compensation to obtain an image after inverse exposure compensation. The image regions with brightness less than or equal to the threshold in the inverse exposure compensated image are extracted by complementing the binarized mask image; The target image is obtained by fusing the image regions in the second image whose brightness is greater than the threshold and the image regions in the third image after inverse exposure compensation whose brightness is less than or equal to the threshold.

5. The image processing apparatus according to claim 4, characterized in that, The first image is determined in the following way: Using an edge detection operator, the horizontal and vertical gradients of each frame in the multi-frame images are determined, and the global gradient mean is determined based on the horizontal and vertical gradients. The first image is determined based on the global gradient mean and the shooting parameters of each frame in the multi-frame images.

6. The image processing apparatus according to claim 4, characterized in that, The processing unit is further configured to perform post-processing on the target image, the post-processing including at least one of the following: Simulated noise is filled into the target image; Detect target regions in the target image whose grayscale value changes are greater than a threshold, and sharpen the target regions.

7. An image processing apparatus, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the method described in any one of claims 1 to 3.

8. A storage medium, characterized in that, The storage medium stores instructions that, when executed by the processor of the terminal, enable the terminal to perform the method described in any one of claims 1 to 3.

Images