Image processing method, device and storage medium

Abstract

The present disclosure relates to an image processing method, device and storage medium, the method comprising: acquiring a first image, a second image and a third image of a same target object; wherein an exposure of the first image is less than an exposure of the second image; the exposure of the second image is less than an exposure of the third image; performing first brightness processing on the first image to obtain a fourth image; wherein the first brightness processing is: performing brightening processing with different brightening multiples on at least two different brightness pixel points in the first image, and the brightening multiple corresponding to the pixel point is in an inverse proportional relationship with the brightness of the pixel point; and performing image fusion on the first image, the second image, the third image and the fourth image to obtain a fused image.

Description

Technical Field

[0001] This disclosure relates to the field of image technology, and in particular to an image processing method, apparatus and storage medium. Background Technology

[0002] To improve the imaging quality in high-contrast scenes with significant variations in brightness and darkness, multiple frames of input images with different exposures are typically acquired. Image fusion is then performed on these multiple frames to obtain a high dynamic range (HDR) image with rich brightness levels.

[0003] If the subject of the image is a moving object, such as a waving or shaking person, the HDR image obtained by fusing multiple frames of input images with different exposures may have blurred outlines of the moving object, which will seriously affect the image quality and reduce the user experience. Summary of the Invention

[0004] This disclosure provides an image processing method, apparatus, and storage medium.

[0005] According to a first aspect of the present disclosure, an image processing method is provided, comprising:

[0006] Acquire a first image, a second image, and a third image of the same target object; wherein the exposure of the first image is less than the exposure of the second image; and the exposure of the second image is less than the exposure of the third image.

[0007] The first image is subjected to a first brightness processing to obtain a fourth image; wherein the first brightness processing is: at least two pixels with different brightness in the first image are brightened by different brightening factors, and the brightening factor corresponding to the pixel is inversely proportional to the brightness of the pixel;

[0008] Image fusion is performed on the first image, the second image, the third image, and the fourth image to obtain a fused image. Optionally, the step of performing a first brightness processing on the first image to obtain the fourth image includes:

[0009] Based on the luminance components of the first image and the second image, a luminance mapping relationship between the first image and the second image is determined; the luminance mapping relationship is used to indicate the nonlinear relationship between the luminance components of each pixel in the first image and the luminance components of each pixel in the second image.

[0010] Based on the brightness mapping relationship, the brightness enhancement factor corresponding to each pixel in the first image is determined, and the brightness component of each pixel is brightened to obtain the fourth image with enhanced brightness.

[0011] Optionally, the step of performing a first brightness processing on the first image to obtain a fourth image includes:

[0012] A first brightness processing is performed on the brightness components of each pixel in the first image, and the brightness components of each pixel after the first brightness processing are determined as the brightness components of each pixel in the fourth image.

[0013] Based on the color components of one or more of the first image, the second image, and the third image, determine the color components of each pixel in the fourth image;

[0014] The fourth image is generated based on the brightness component and color component of each pixel in the fourth image after brightening.

[0015] Optionally, determining the brightness mapping relationship between the first image and the second image based on the brightness components of the first image and the second image includes:

[0016] The first and second images in RGB format are converted to YUV format to obtain the first and second images in YUV format.

[0017] Brightness sampling is performed on the same pixel positions of the first and second images in YUV format to obtain a set of pixels;

[0018] Obtain the first luminance component and the second luminance component of each pixel in the pixel set in the first image and the second luminance component in the second image;

[0019] The brightness mapping relationship between the first image and the second image is determined based on the first brightness component and the second brightness component.

[0020] Optionally, the step of fusing the first image, the second image, the third image, and the fourth image to obtain the fused image includes:

[0021] Based on the second image, the fourth image is subjected to ghosting removal processing to obtain the fifth image;

[0022] The first image, the third image, and the fifth image are fused to obtain a fused high dynamic range (HDR) image.

[0023] Optionally, the step of performing ghosting removal processing on the fourth image based on the second image to obtain the fifth image includes:

[0024] Based on the second image, the position of the target object area in the fourth image is corrected to obtain the fifth image; the position of the target object area in the fifth image is the same as the position of the target object area in the second image.

[0025] Optionally, the step of fusing the first image, the third image, and the fifth image to obtain a fused high dynamic range (HDR) image includes:

[0026] Obtain a first region from the first image, and obtain a second region from the third image;

[0027] The first region and the second region are fused into the fifth image to obtain a high dynamic range (HDR) image.

[0028] Wherein, the first region in the HDR image is the same as the first region in the first image; the second region in the HDR image is the same as the second region in the third image; the average brightness of the first region is greater than the average brightness of the first image, and the average brightness of the second region is less than the average brightness of the third image.

[0029] Optionally, acquiring the first image, second image, and third image of the same target object includes:

[0030] Obtain a request to generate an HDR image; wherein the generation request carries multiple different brightness requirement levels;

[0031] Based on the multiple different brightness requirement levels, the exposure time corresponding to the multiple different brightness requirement levels is determined;

[0032] First, second, and third images of the same target object were acquired based on multiple different exposure times.

[0033] According to a second aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0034] An acquisition module is used to acquire a first image, a second image, and a third image of the same target object; wherein the exposure of the first image is less than the exposure of the second image; and the exposure of the second image is less than the exposure of the third image.

[0035] A brightening module is used to perform a first brightness processing on the first image to obtain a fourth image; wherein the first brightness processing is: performing brightening processing on at least two pixels with different brightness in the first image by different brightening factors, and the brightening factor corresponding to the pixel is inversely proportional to the brightness of the pixel;

[0036] The processing module is used to perform image fusion on the first image, the second image, the third image, and the fourth image to obtain a fused image.

[0037] Optionally, the highlighting module is used for:

[0038] Based on the luminance components of the first image and the second image, a luminance mapping relationship between the first image and the second image is determined; the luminance mapping relationship is used to indicate the nonlinear relationship between the luminance components of each pixel in the first image and the luminance components of each pixel in the second image.

[0039] Based on the brightness mapping relationship, the brightness enhancement factor corresponding to each pixel in the first image is determined, and the brightness component of each pixel is brightened to obtain the fourth image with enhanced brightness.

[0040] Optionally, the highlighting module is used for:

[0041] A first brightness processing is performed on the brightness components of each pixel in the first image, and the brightness components of each pixel after the first brightness processing are determined as the brightness components of each pixel in the fourth image.

[0042] Based on the color components of one or more of the first image, the second image, and the third image, determine the color components of each pixel in the fourth image;

[0043] The fourth image is generated based on the brightness component and color component of each pixel in the fourth image after brightening.

[0044] Optionally, the brightening module is further configured to:

[0045] The first and second images in RGB format are converted to YUV format to obtain the first and second images in YUV format.

[0046] Brightness sampling is performed on the same pixel positions of the first and second images in YUV format to obtain a set of pixels;

[0047] Obtain the first luminance component and the second luminance component of each pixel in the pixel set in the first image and the second luminance component in the second image;

[0048] The brightness mapping relationship between the first image and the second image is determined based on the first brightness component and the second brightness component.

[0049] Optionally, the processing module is configured to:

[0050] Based on the second image, the fourth image is subjected to ghosting removal processing to obtain the fifth image;

[0051] The first image, the third image, and the fifth image are fused to obtain a fused high dynamic range (HDR) image.

[0052] Optionally, the processing module is further configured to:

[0053] Based on the second image, the position of the target object area in the fourth image is corrected to obtain the fifth image; the position of the target object area in the fifth image is the same as the position of the target object area in the second image.

[0054] Optionally, the processing module is further configured to:

[0055] Obtain a first region from the first image, and obtain a second region from the third image;

[0056] The first region and the second region are fused into the fifth image to obtain a high dynamic range (HDR) image.

[0057] Wherein, the first region in the HDR image is the same as the first region in the first image; the second region in the HDR image is the same as the second region in the third image; the average brightness of the first region is greater than the average brightness of the first image, and the average brightness of the second region is less than the average brightness of the third image.

[0058] Optionally, the acquisition module is used to:

[0059] Obtain a request to generate an HDR image; wherein the generation request carries multiple different brightness requirement levels;

[0060] Based on the multiple different brightness requirement levels, the exposure time corresponding to the multiple different brightness requirement levels is determined;

[0061] First, second, and third images of the same target object were acquired based on multiple different exposure times.

[0062] According to a third aspect of the present disclosure, an image processing apparatus is provided, comprising:

[0063] processor;

[0064] Memory used to store executable instructions;

[0065] The processor is configured to, when executing executable instructions stored in the memory, implement the steps of the image processing method according to the first aspect of the present disclosure.

[0066] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein when instructions in the storage medium are executed by a processor of an image processing apparatus, the image processing apparatus is enabled to perform steps in the image processing method as described in the first aspect of the present disclosure.

[0067] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0068] This embodiment of the present disclosure acquires multiple frames of images of the same target object with different exposure levels, and performs a first brightness processing on at least two pixels with different brightness levels in the first image with the lowest exposure level using different brightening factors to obtain a fourth image. Since the low-exposure first image better preserves the contour information of the target object, the fourth image obtained after the first brightness processing can not only preserve the contour information of the target object, but also reduce the overexposure of pixels with higher brightness in the image and the loss of effective pixel information.

[0069] By fusing the first, second, third, and fourth images with different exposures, a fused image with high dynamic range can be obtained. At the same time, the fused image can retain the contour information of the target object, reduce the blurring of the contour of moving objects, and improve the image quality of the fused image.

[0070] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0071] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0072] Figure 1 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 1 .

[0073] Figure 2 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 2 .

[0074] Figure 3 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 3 .

[0075] Figure 4 This is a schematic diagram illustrating a fitting curve between a first luminance component and a second luminance component according to an exemplary embodiment.

[0076] Figure 4This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 5 .

[0077] Figure 5 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 6 .

[0078] Figure 6 This is a schematic diagram illustrating the processing flow of an image acquisition device according to an exemplary embodiment.

[0079] Figure 7 This is a schematic diagram illustrating a sampling pixel according to an exemplary embodiment.

[0080] Figure 7 This is a comparative diagram based on linear brightness mapping processing.

[0081] Figure 8 This is a schematic diagram of the fitting curves of the first and second luminance components of a set of pixels according to an exemplary embodiment.

[0082] Figure 8 This is a schematic diagram illustrating the comparison of image effects after nonlinear brightening processing according to an exemplary embodiment.

[0083] Figure 9 This is a second image with normal exposure, as illustrated in an exemplary embodiment.

[0084] Figure 9 This is a first image of underexposure according to an exemplary embodiment.

[0085] Figure 10 This is a schematic diagram illustrating the effect of an image processing method according to an exemplary embodiment.

[0086] Figure 10 This is a comparison diagram of the effects of an image processing method and an image processing method based on bracketing exposure, according to an exemplary embodiment.

[0087] Figure 11 This is a comparison diagram illustrating the effects of fused images obtained based on nonlinear brightening processing and linear brightening processing, according to an exemplary embodiment.

[0088] Figure 11 This is a schematic diagram of the structure of an image processing apparatus according to an exemplary embodiment.

[0089] Figure 12 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. Detailed Implementation

[0090] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses consistent with some aspects of this disclosure as detailed in the appended claims.

[0091] In related technologies, in order to obtain high-quality images of high-contrast scenes, bracketing exposure is usually used to generate HDR images; for example, by acquiring multiple frames of images with different exposures, the multiple frames of images with different exposures are fused to obtain an HDR image.

[0092] There are two objective limitations to generating HDR images based on bracketing exposure: first, it requires multiple frames, which makes the entire exposure period for image acquisition relatively long; second, among the multiple frames with different exposure levels, there needs to be one high-brightness image, which means that acquiring the high-brightness image requires a long exposure time.

[0093] However, due to these two objective limitations, when the target object of the image acquisition is a moving object, the position of the target object in multiple frames with different exposures may be deviated and cannot be aligned; and the image with a longer exposure time loses the outline information of the target object, which may result in the area where the target object is located in the final HDR image being relatively blurry, i.e., there is ghosting, which reduces the image quality of HDR.

[0094] This disclosure provides an image processing method. Figure 12 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 13 ,like Figure 13 As shown, the method includes:

[0095] Step S101: Acquire a first image, a second image, and a third image of the same target object; wherein the exposure of the first image is less than the exposure of the second image; and the exposure of the second image is less than the exposure of the third image.

[0096] Step S102: Perform a first brightness processing on the first image to obtain a fourth image; wherein, the first brightness processing is: perform different brightening factor processing on at least two pixels with different brightness in the first image, and the brightening factor corresponding to the pixel is inversely proportional to the brightness of the pixel;

[0097] Step S103: Perform image fusion on the first image, the second image, the third image, and the fourth image to obtain the fused image.

[0098] In this embodiment of the disclosure, the image processing method can be applied to a mobile terminal equipped with an image acquisition device, such as a smartphone, tablet computer, or wearable electronic device, and the image acquisition device equipped on the mobile terminal acquires the first image, the second image, and the third image.

[0099] The image acquisition device generally refers to the device in a mobile terminal that can perform the photo-taking function, including a camera, as well as necessary processing and storage modules to complete the acquisition and transmission of images, and may also include some processing function modules.

[0100] Image acquisition devices can be cameras or video cameras, etc.

[0101] In this embodiment, the image processing method can also be applied to other devices besides the device that acquires the first, second, and third images (hereinafter referred to as the first device) or a cloud server. The other device may or may not have an image acquisition device, but it must possess image processing capabilities. When applied to other devices or a cloud server, the first, second, and third images sent by the first device are received by the other device or cloud server to obtain the first, second, and third images. For example, after the other device or cloud server performs image processing, i.e., obtains the fused image, it can be sent to the first device for display or storage.

[0102] In step S101, the target object can be a dynamic object, that is, the image processing method of this embodiment is an image obtained by capturing a moving target object.

[0103] The target object can be: a person, an animal, or a vehicle, etc.

[0104] The first image, the second image, and the third image are all images obtained with different exposure levels when the same target object is captured; wherein, the exposure level of the first image is less than that of the second image; and the exposure level of the second image is less than that of the third image.

[0105] In some embodiments, the first image, the second image, and the third image are multiple frames of images captured continuously by the same camera or camcorder, for example, multiple frames of images captured continuously after the camera has enabled the HDR capture mechanism.

[0106] It should be noted that the second image can be an exposed image with an exposure level within a preset exposure range. This exposure level being within the preset exposure range indicates that the highlight areas in the image will not be too bright or too dark; thus, the second image is a normally exposed image; consequently, the first image is an underexposed image, and the third image is a long-exposed image.

[0107] In step S102, the brightness factor corresponding to each pixel can be determined according to the brightness of each pixel in the first image. Based on the brightness factor of each pixel, the pixel is subjected to a first brightness processing to obtain the fourth image.

[0108] Here, at least two different pixels in the first image correspond to different brightening factors, and the brightening factor corresponding to the pixel is inversely proportional to the brightness of the pixel. That is, the higher the brightness of the pixel, the smaller the brightening factor corresponding to the pixel; the lower the brightness of the pixel, the larger the brightening factor corresponding to the pixel.

[0109] It should be noted that, considering that the brightness of different pixels in the first image may vary greatly, if each pixel in the first image is brightened by the same brightening factor, it may result in overexposed areas in the brightened image, and the effective pixel information in the first image may be lost.

[0110] Based on this, the embodiments of this disclosure reduce the area of ​​overexposed regions in the fourth image obtained by performing first brightness processing on at least two pixels with different brightness levels in the first image using different brightening factors.

[0111] In some embodiments, the first image can be divided into a first brightness region and a second brightness region according to the brightness components of each pixel in the first image; wherein the average brightness component of the pixels in the first brightness region is less than the average brightness component of the pixels in the second brightness region.

[0112] The pixels in the first brightness area are brightened using a first brightening factor, and the pixels in the second brightness area are brightened using a second brightening factor, to obtain the fourth image.

[0113] In this embodiment of the disclosure, the first brightening factor is greater than the second brightening factor.

[0114] Based on the comparison results between the brightness components of each pixel in the first image and the brightness threshold, each pixel in the first image can be divided into a first pixel and a second pixel; based on the positions of multiple first pixels, a first region in the first image can be determined; based on the positions of multiple second pixels, a second region in the first image can be determined.

[0115] Based on a first brightening factor corresponding to the first region, the luminance components of each first pixel in the first region are brightened; based on a second brightening factor corresponding to the second region, the luminance components of each second pixel in the second region are brightened to obtain a fourth image.

[0116] In this way, by using a larger first brightening factor to brighten the first region with lower average brightness in the first image, and using a smaller second brightening factor to brighten the second region with higher average brightness in the first image, the image regions with different brightness levels in the first image are brightened to different degrees. This not only significantly improves the brightness of the low-brightness regions in the first image, but also reduces the loss of pixel information in the high-brightness regions of the first image due to the brightening process, thus preserving more image detail information.

[0117] In step S103, the image can be obtained by fusing the low-exposure first image, the normally exposed second image, the high-exposure third image, and the fourth image after the first brightness processing.

[0118] It should be noted that although the first image retains the image information of the high-brightness area, the second image retains the image information of the normal brightness area, and the third image retains the image information of the low-brightness area, the longer exposure time of the second and third images leads to the loss of the outline information of the target object in the second and third images. On the other hand, due to the shorter exposure time of the first image, although the outline of the target object in the first image is relatively clear, the insufficient amount of light results in a significant lack of brightness and color information of the target object in the first image.

[0119] Therefore, by fusing the first image, the second image, the third image, and the fourth image, since the fourth image is obtained by performing the first brightness processing on the first image, the fourth image retains both the contour information of the target object and the brightness and color information of the target object; by fusing the first image, the second image, the third image, and the fourth image, the fused image retains not only the contour information, brightness and color information of the target object, but also the image information of high brightness areas, normal brightness areas, and low brightness areas.

[0120] Optionally, such as Figure 14 As shown, Figure 14 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 15 The step S102, which involves performing non-linear brightening processing on the first image to obtain a fourth image, includes:

[0121] Step S1021: Determine the nonlinear brightness mapping relationship between the first image and the second image based on the brightness components of the first image and the second image.

[0122] Step S1022: Based on the nonlinear brightness mapping relationship, determine the brightening factor corresponding to each pixel in the first image, and brighten the brightness component of each pixel to obtain the fourth image with enhanced brightness.

[0123] In this embodiment of the disclosure, the luminance components of each pixel in the first image and the luminance components of each pixel in the second image can be obtained respectively; based on the luminance components of each pixel in the first image and the luminance components of each pixel in the second image, the nonlinear relationship between the luminance components of pixels at the same position in the first image and the luminance components in the second image is determined, and the nonlinear relationship is determined as the luminance mapping relationship between the first image and the second image.

[0124] Based on the brightness mapping relationship, the brightness enhancement factor corresponding to each pixel with different brightness is determined; according to the brightness enhancement factor of each pixel with different brightness, the brightness component of the pixel is brightened to obtain the brightened fourth image.

[0125] Here, the brightness mapping relationship can be used to describe the changes in image brightness caused by different exposure levels; based on the brightness mapping relationship, at least two pixels with different brightness levels in the first image are brightened by different brightening factors, so that the image brightness of the fourth image obtained by the brightening process is approximately the image brightness of the image acquired under normal exposure. For example, the difference between the image brightness of the fourth image and the image brightness of the image acquired under normal exposure is less than a preset error threshold.

[0126] It should be noted that, based on the brightness mapping relationship between the first image and the second image, the first brightness processing of the first image is performed. The brightness distribution histogram of the pixels in the fourth image obtained by the first brightness processing is similar to that of the pixels in the second image, which is beneficial for achieving fine control over the bright and dark areas of the image.

[0127] In this way, on the one hand, it can reduce the occurrence of overexposed pixels and loss of effective pixel information in the image, and on the other hand, the brightness distribution of each pixel in the image conforms to the brightness distribution of pixels in an image under normal exposure conditions, thus better preserving the image content information of the first image and the image brightness information of the second image.

[0128] Optionally, such as Figure 15 As shown, Figure 16 This is a flowchart illustrating an image processing method according to an exemplary embodiment. Figure 16The step S102, which involves performing a first brightness processing on the first image to obtain a fourth image, includes:

[0129] Step S1023: Perform a first brightness processing on the brightness components of each pixel in the first image, and determine the brightness components of each pixel after the first brightness processing as the brightness components of each pixel in the fourth image.

[0130] Step S1024: Determine the color components of each pixel in the fourth image based on the color components of one or more of the first image, the second image, and the third image;

[0131] Step S1025: Generate the fourth image based on the brightness component and color component of each pixel in the fourth image after brightening.

[0132] In this embodiment of the disclosure, considering that the first image, the second image, and the third image are images with different exposures obtained for the same target object, the difference between the brightness components of each pixel in the first image, the second image, and the third image is large, while the difference between the color components is small.

[0133] Therefore, in order to obtain a fourth image with the same image content as the underexposed first image and the same image brightness as the normally exposed second image, the brightness and color components of each pixel in the first image can be processed separately.

[0134] By obtaining the luminance components of each pixel in the first image and performing a first luminance processing on the luminance components of each pixel, the luminance components of each pixel in the fourth image are obtained.

[0135] Here, the brightness mapping relationship between the first image and the second image can be obtained; based on the brightness mapping relationship, the brightening factor corresponding to each pixel in the first image is determined, and based on the brightening factor corresponding to each pixel, the brightness component of each pixel is brightened to obtain the brightness component of each pixel in the fourth image.

[0136] By acquiring the color components of each pixel in at least one of the first, second, and third images, the color components of each pixel in the fourth image are determined based on the color components of each pixel in the at least one image.

[0137] It should be noted that since the color component values ​​corresponding to the same pixel in the first, second, and third images are basically the same (within ±3), any one of the first, second, and third images can be determined as the target image, and the color components of the target image can be determined as the color components of each pixel in the fourth image.

[0138] In some embodiments, at least two images may be determined from the first image, the second image, and the third image; the color components of each pixel in the at least two images are obtained by weighting the color components of each pixel in the at least two images with the corresponding weights.

[0139] Optionally, determining the brightness mapping relationship between the first image and the second image based on the brightness components of the first image and the second image includes:

[0140] The first and second images in RGB format are converted to YUV format to obtain the first and second images in YUV format.

[0141] Brightness sampling is performed on the same pixel positions of the first and second images in YUV format to obtain a set of pixels;

[0142] Obtain the first luminance component and the second luminance component of each pixel in the pixel set in the first image and the second luminance component in the second image;

[0143] The brightness mapping relationship between the first image and the second image is determined based on the first brightness component and the second brightness component.

[0144] In this embodiment of the disclosure, in order to more conveniently perform image processing, such as brightening each pixel in the image and image fusion between images in YUV format, it is necessary to first convert the format of the first image, the second image, and the third image from RGB format to YUV format.

[0145] Here, the format conversion can be performed directly based on the image signal processor (ISP) in the image acquisition device.

[0146] It's important to note that in YUV format images, Y represents luminance, while U and V represent chrominance. Y alone results in a black and white image; adding UV creates a color image. RGB is a standard for representing color in the digital field, also known as a color space. In RGB format images, each pixel value is represented by three components: R, G, and B. A specific color is represented by different combinations of luminance values ​​from the three primary colors R, G, and B. If each component uses 8 bits, then one pixel uses 3 * 8 = 24 bits. Both YUV and RGB are used to represent color; however, their approaches and methods for describing color differ. RGB decomposes a color into a combination of the luminance of three pure colors, while YUV decomposes a color into a combination of one luminance and two chrominance values.

[0147] This embodiment of the disclosure converts the first image, the second image, and the third image into YUV format images, thereby allowing the luminance component of each pixel to be determined directly based on the Y value of each pixel in the first image, the second image, and the third image.

[0148] After obtaining a first image and a second image in YUV format, a set of pixels can be obtained by sampling the first image and the second image at equal intervals. The set of pixels includes at least the pixel values ​​of multiple pixels in the first image and the pixel values ​​of multiple pixels in the second image.

[0149] It should be noted that the number of pixels sampled can be set according to the total number of pixels in the image. In some embodiments, in order to accurately determine the nonlinear brightness mapping relationship between the first image and the second image, the number of pixels sampled is at least 0.1% of the total number of pixels in the image.

[0150] By obtaining the Y-values ​​of each pixel in the pixel set in the first image and the Y-values ​​in the second image, the first luminance component and the second luminance component of each pixel in the first image are determined. Based on the first and second luminance components of each pixel, a fitting analysis is performed to determine the fitting curve between the first and second luminance components. This fitting curve represents the luminance mapping relationship between the first and second images.

[0151] For example, the distribution of each pixel in the pixel set can be determined based on the first luminance component being the X value and the second luminance component being the Y value; a fitting curve between the first luminance component and the second luminance component can be obtained based on the least squares method. Figure 17 As shown, Figure 17 This is a schematic diagram illustrating a fitting curve between a first luminance component and a second luminance component according to an exemplary embodiment.

[0152] Optionally, such asFigure 18 As shown, Figure 18 This is a flowchart illustrating an image processing method according to an exemplary embodiment. ​ The step S103, which involves fusing the first image, the second image, the third image, and the fourth image to obtain a fused image, includes:

[0153] Step S1031: Based on the second image, perform ghosting removal processing on the fourth image to obtain the fifth image;

[0154] Step S1032: Perform image fusion on the first image, the third image, and the fifth image to obtain a fused high dynamic range (HDR) image.

[0155] Here, the ghosting removal process can be an image alignment process, which achieves the effect of eliminating ghosting in the fused image by performing image alignment processing on multiple frames of images.

[0156] It should be noted that, since the target object may be dynamic, the position of the target object may vary in three frames captured for the same object, resulting in a visual ghosting effect. For example, in three consecutive frames captured of a swaying arm, the arm might be positioned higher in the first frame, in the middle of the second frame, and lower in the third. Therefore, the resulting image after fusing these three frames will exhibit a ghosting effect.

[0157] The second image can be used as a reference image to perform image alignment processing on multiple frames (i.e., the first image, the third image, and the fourth image); and the multi-frame images after image alignment processing are fused to obtain the fused HDR image.

[0158] In this embodiment, ghosting in the fused HDR image is eliminated because the target object is in the same position in the multiple frames after image alignment processing. On the other hand, the fifth image retains the outline information, brightness and color information of the target object, the first image retains the image information of the high brightness area, and the third image retains the image information of the low brightness area. This means that the fused HDR image retains not only the outline information, brightness and color information of the target object, but also the image information of the high brightness area and the low brightness area.

[0159] Optionally, in step S1031, based on the fourth image, the second image is subjected to ghosting removal processing to obtain the fifth image, including:

[0160] Based on the second image, the position of the target object area in the fourth image is corrected to obtain the fifth image; the position of the target object area in the fifth image is the same as the position of the target object area in the second image.

[0161] It should be noted that since ghosting in the image is caused by the deviation in the position of the target object in multiple frames, in order to eliminate ghosting, the position of the target object in the multiple frames needs to be adjusted to be the same. When the position of the target object in the multiple frames is consistent, the fused image will not have ghosting visually.

[0162] The second image can be used as a reference image. Based on the position of the target object in the second image, the position of the target object in the fourth image is corrected so that the position of the target object in the corrected fourth image (i.e., the fifth image) is consistent with the position of the target object in the second image. Using the first and third images, dynamic range adjustment is performed on the pixels of the fifth image to obtain an HDR image.

[0163] Because the fourth image after brightness enhancement better preserves the image content information of the first image (including the outline information of the target object) and the image brightness information of the second image (including image detail information), it can effectively reduce the ghosting of the fused HDR image on the one hand, and the HDR image can better preserve the outline information and image detail information of the target object on the other hand.

[0164] In other embodiments, the position of the target object region in the fourth image can be corrected based on the second image; the target object region can be extracted from the corrected fourth image and fused into the second image to obtain the fifth image.

[0165] In this embodiment of the disclosure, considering that the image quality of the fourth image may be lower than that of the second image in certain specific scenarios, in order to improve the image quality of the fused HDR image, the position of the target object in the fourth image can be corrected based on the position of the target object in the second image; and the target object is cropped out from the corrected fourth image and fused into the target object in the second image to obtain the fifth image.

[0166] Because the fourth image retains the outline information of the target object, by fusing the area where the target object is located in the corrected fourth image into the second image, it is possible to compensate for the missing outline information of the target object in the image while maintaining the image quality, and also to effectively reduce the ghosting in the fused HDR image.

[0167] Optionally, step S1032, which involves image fusion of the first image, the third image, and the fifth image to obtain a fused high dynamic range (HDR) image, includes:

[0168] Obtain a first region from the first image, and obtain a second region from the third image;

[0169] The first region and the second region are fused into the fifth image to obtain the HDR image;

[0170] Wherein, the first region in the HDR image is the same as the first region in the first image; the second region in the HDR image is the same as the second region in the third image; the average brightness of the first region is greater than the average brightness of the first image, and the average brightness of the second region is less than the average brightness of the third image.

[0171] In this embodiment of the disclosure, considering that the first image with lower exposure can better present the image details of the bright areas, the third image with higher exposure can better present the image details of the dark areas.

[0172] Therefore, by fusing the first region in the first image whose average brightness is greater than the average brightness of the first image (i.e., the bright region of the first image) into the first region corresponding to the fifth image, and by fusing the second region in the third image whose average brightness is less than the average brightness of the third image (i.e., the low-bright region in the third image) into the second region corresponding to the fifth image, the fused HDR image can simultaneously present the image detail information of the bright and dark areas and the relatively clear outline information of the target object.

[0173] Optionally, obtaining the first image, second image, and third image of the same target object in step S101 includes:

[0174] Obtain a request to generate an HDR image; wherein the generation request carries multiple different brightness requirement levels;

[0175] Based on the multiple different brightness requirement levels, the exposure time corresponding to the multiple different brightness requirement levels is determined;

[0176] First, second, and third images of the same target object were acquired based on multiple different exposure times.

[0177] In this embodiment of the disclosure, the brightness requirement level can be described by the exposure value (EV); wherein, the multiple different brightness requirement levels carried in the generation request can be set by the user, or determined based on the ratio between bright and dark areas in the preview image during image preview.

[0178] The exposure time corresponding to the multiple different EV values ​​is determined based on the EV values ​​indicated by the multiple different brightness requirement levels in the HDR image generation request.

[0179] It's important to note that exposure time is the time interval between the shutter opening and closing. Different exposure times result in different levels of image exposure. A longer exposure time allows more light to enter the image, leading to a more exposed image. Conversely, a shorter exposure time allows less light to enter the image, resulting in a less exposed image.

[0180] The exposure of the image acquisition device can be controlled according to the exposure time corresponding to the multiple different EV values, so as to acquire three frames of images of the same target object with different exposures, namely the first image, the second image and the third image.

[0181] Since images with different exposures have different brightness and dynamic range, image fusion based on images of the same target object at different exposures can effectively improve the imaging quality of HDR images.

[0182] This disclosure also provides the following embodiments:

[0183] ​ This is a flowchart illustrating an image processing method according to an exemplary embodiment. ​ The image processing method is applied to a device that acquires the first image, the second image, and the third image, and the method includes:

[0184] Step S201: Obtain an HDR image generation request, wherein the generation request carries multiple different brightness requirement levels; based on the multiple different brightness requirement levels, determine the exposure time corresponding to the multiple different brightness requirement levels; based on the multiple different exposure times, acquire a first image, a second image, and a third image of the same target object.

[0185] In this example, after the mobile terminal obtains the HDR image generation request, it can decompose the generation request into multiple photo-taking sub-requests at the Hardware Abstraction Layer (HAL) according to the multiple brightness requirement levels in the generation request; and determine the exposure time corresponding to the multiple different brightness requirement levels according to the multiple different brightness requirement levels.

[0186] It should be noted that the brightness requirement level can be described by EV values. For example, EV1 describes an image with a brightness requirement that is twice as bright as EV0. The EV values ​​corresponding to each photo sub-request can be translated into specific exposure times using the AE algorithm.

[0187] It can be understood that the exposure time of the photo-taking sub-request is the shooting parameter of the image acquisition device of the mobile terminal; three frames of images can be acquired under the shooting parameter settings of each photo-taking sub-request. Here, exposure time refers to the time interval from shutter opening to closing. The longer the exposure time, the more light enters.

[0188] The first image, the second image, and the third image are multiple images continuously captured by the mobile terminal; wherein, the exposure of the first image is less than that of the second image; and the exposure of the second image is less than that of the third image.

[0189] Here, the first, second, and third images are RAW format image files. These RAW format image files contain raw RGB data, which is the data directly output by the sensor after photoelectric conversion and sampling by an ADC. This unprocessed data better reflects the characteristics of the image.

[0190] It should be noted that the second image can be an exposed image with an exposure level within a preset exposure range. This exposure level being within the preset exposure range indicates that the highlight areas in the image will not be overexposed or underexposed; thus, the second image is a normally exposed image; consequently, the first image is an underexposed image, and the third image is a long-exposed image.

[0191] Step S202: Convert the color space of the first image and the second image in RGB format to obtain the first image and the second image in YUV format; sample the brightness of the same pixel positions in the first image and the second image in YUV format to obtain a set of pixels;

[0192] In this example, the three images with different exposures can be converted to different color spaces, transforming the first, second, and third images in RGB format into the first, second, and third images in YUV format.

[0193] Here, the format conversion can be performed directly based on the ISP in the image acquisition device.

[0194] For example, such as ​ As shown, ​ This is a schematic diagram illustrating the processing flow of an image acquisition device according to an exemplary embodiment. A camera app on a mobile terminal sends an HDR image generation request to the camera sensor. Based on the generation request, the camera sensor acquires three RGB images with different exposures, namely an EV- image, an EV0 image, and an EV+ image, and sends these three RGB images with different exposures to an ISP. The ISP then performs format conversion on the three RGB images with different exposures to obtain three YUV images with different exposures.

[0195] After obtaining YUV images with different exposures, it is necessary to determine the nonlinear brightness mapping relationship between the first image and the second image based on the normally exposed second image and the underexposed first image, so that the underexposed first image can be nonlinearly brightened according to the nonlinear brightness mapping relationship.

[0196] In this example, the first luminance component of each sampled pixel in the first image and the second luminance component in the second image can be obtained by sampling the first image and the second image at equal intervals; based on the first luminance component and the second luminance component of each sampled pixel, a set of pixels is obtained.

[0197] Here, the sampling results are recorded in the form [y_ev-, y_ev0], where y_ev- represents the first luminance component of the sampled pixel in the underexposed first image; and y_ev0 represents the second luminance component of the sampled pixel in the normally exposed second image. It should be noted that the number of sampled pixels needs to occupy 0.1% of the total number of pixels in the entire image.

[0198] like ​ As shown, ​ This is a schematic diagram illustrating a sampling pixel according to an exemplary embodiment. Reference numeral 81 shows a sampling pixel in a normally exposed second image; reference numeral 82 shows a sampling pixel in an underexposed first image.

[0199] Step S203: Obtain the first luminance component and the second luminance component corresponding to each pixel in the pixel set in the first image and the second luminance component in the second image; determine the luminance mapping relationship between the first image and the second image based on the first luminance component and the second luminance component.

[0200] In this example, considering that the brightness of different pixels in the first image may vary greatly, if linear brightness mapping is performed on the first image, that is, if each pixel in the first image is brightened by the same brightening factor, it may result in overexposed areas in the brightened image, causing the loss of effective pixel information in the first image.

[0201] For example, such as ​ As shown, ​ This is a comparative diagram based on linear brightness mapping processing. Label 91 shows the pixel distribution curves for different pixel values ​​in the underexposed image, label 92 shows the pixel distribution curves for different pixel values ​​in the image after linear brightness mapping processing, and label 93 shows the pixel distribution curves for different pixel values ​​in the normally exposed image.

[0202] As shown in the figure, approximately 675,000 pixels in the image after linear brightness mapping have a pixel value greater than or equal to 250. This means that the overexposed area in the image after linear brightness mapping accounts for 33.5% of the entire image. In contrast, approximately 350,000 pixels in the normally exposed image of the same scene have a pixel value greater than or equal to 250, accounting for 17.4% of the entire image.

[0203] Based on this, the area of ​​overexposed regions in the fourth image obtained by non-linear brightening can be reduced by applying different brightening factors to at least two different pixels in the first image.

[0204] In order to reasonably determine the brightening factor of each pixel, a fitting analysis can be performed based on the first luminance component and the second luminance component of each sampled pixel in the pixel set to determine the fitting curve between the first luminance component and the second luminance component, which is the nonlinear luminance mapping relationship between the first image and the second image.

[0205] Here, the fitting curve between the first luminance component and the second luminance component can be determined using the least squares method.

[0206] For example, a fitting analysis is performed based on all [y_ev-, y_ev0] arrays of the pixel set; such as ​ As shown, ​ This is a schematic diagram illustrating the fitting curves of the first and second luminance components of a set of pixels according to an exemplary embodiment. The horizontal axis represents the first luminance component of the pixels in an underexposed first image, and the vertical axis represents the second luminance component of the pixels in a normally exposed second image. Reference numeral 101 represents an array [y_ev-, y_ev0], and reference numeral 102 represents the fitting curve between the first and second luminance components.

[0207] Step S204: Based on the brightness mapping relationship, determine the brightness factor corresponding to each pixel in the first image, and perform brightness enhancement processing on the brightness components of each pixel in the first image to obtain the brightness components of each pixel in the fourth image.

[0208] In this example, after determining the non-linear brightness mapping relationship between the first image and the second image, each pixel in the first image is brightened based on the non-linear brightness mapping relationship.

[0209] For example, the non-linear brightness mapping relationship between the first image and the second image is as follows:

[0210] y new =f(yev- );

[0211] Wherein, the y new The y represents the brightness component of a pixel in the fourth image; ev- Let f(·) be the first luminance component of a pixel in the first image, and let f(·) represent the fitting curve relationship between the first luminance component and the second luminance component.

[0212] This example uses a non-linear fitting curve between the first luminance component and the second luminance component to brighten the luminance components of each pixel in an underexposed first image, thereby raising the overall brightness of the first image to a normal exposure level.

[0213] Step S205: Determine the color components of each pixel in the fourth image based on the color components of one or more of the first image, the second image, and the third image; generate the fourth image based on the brightness components of each pixel after brightening and the color components in the fourth image.

[0214] In this example, after determining the luminance components of each pixel in the fourth image, any frame can be selected from the first, second, and third images as the target image, and the color components of each pixel in the target image are determined as the color components of each pixel in the fourth image; thus, the fourth image is generated based on the luminance and color components of each pixel in the fourth image.

[0215] It should be noted that before performing non-linear brightening processing on the image, the RGB format image is converted to YUV format. The conversion formula for RGB to YUV format is shown below:

[0216] Y = (77 × R + 150 × G + 29 × B) >> 8

[0217] U=((-44×R-87×G+131×B)>>8)+128

[0218] V=((131×R-110×G-21×B)>>8)+128

[0219] The calculation of color components (U and V values) ensures that the U and V values ​​of the same pixel after image conversion at different exposures differ by less than ±3. Therefore, regardless of whether the color components of each pixel in the first, second, or third image are used as the color components of each pixel in the fourth image, the generated fourth image will not be affected.

[0220] For example, such as ​ As shown, ​This is a schematic diagram illustrating the image effect comparison after nonlinear brightening processing according to an exemplary embodiment. As can be seen from the figure, the image brightness of the fourth image obtained by nonlinear brightening processing is significantly improved.

[0221] Step S206: Based on the second image, the position of the target object region in the fourth image is corrected to obtain the fifth image; the position of the target object region in the fifth image is the same as the position of the target object region in the second image.

[0222] In this example, image fusion can be performed based on the first image, the second image, the third image, and the fourth image to obtain the fused HDR image.

[0223] Here, considering that the exposure time of the second image is usually longer, moving objects in the second image are prone to ghosting and blurred edges. The exposure time of the first image is shorter, and the ghosting of moving objects in the first image is significantly better than that in the second image. However, due to insufficient light intake in the first image, the brightness information of the first image is severely lacking.

[0224] For example, such as ​ , ​ As shown, ​ This is a second image with normal exposure, as illustrated in an exemplary embodiment. ​ This is an example of an underexposed first image. In the second image, the hand area is relatively blurry and has motion blur; while in the first image, the hand area does not have obvious motion blur, but the brightness and color information of the first image are severely lacking.

[0225] Because the hand region in the final HDR image can only be derived from any one frame of the input image during image fusion processing, it cannot be created out of thin air; related technologies usually obtain the hand region from a normally exposed second image, so that the brightness and color information of the image are preserved in the final output HDR image after fusion, but this also causes the target object to be blurred.

[0226] Therefore, this example can obtain a fourth image by performing non-linear brightening processing on the first image, which has the same image content as the underexposed first image and the same image brightness as the normally exposed second image; based on the position of the target object in the second image, the position of the target object in the fourth image is corrected to obtain a fifth image.

[0227] For example, the area where the target object is located in the second image can be directly filled with the area where the target object is located in the fourth image, so that the target object in the processed second image has no afterimage and clear edge contour.

[0228] Step S207: Obtain a first region from the first image and a second region from the third image; fuse the first region and the second region into the fifth image to obtain a high dynamic range (HDR) image; wherein, the first region in the HDR image is the same as the first region in the first image; the second region in the HDR image is the same as the second region in the third image; the average brightness of the first region is greater than the average brightness of the first image, and the average brightness of the second region is less than the average brightness of the third image.

[0229] In this example, the low-brightness areas in the fifth image can be filled with the same areas in the long-exposure third image, and the high-brightness areas in the fifth image can be filled with the same areas in the underexposure first image, thus obtaining an HDR image with a large contrast between light and dark.

[0230] For example, taking the scenario of "waving from the window" as an example, such as ​ As shown, ​ This is a schematic diagram illustrating the effect of an image processing method according to an exemplary embodiment. The method involves performing non-linear brightening processing on an underexposed image to obtain a brightened frame image; and then, based on an HDR algorithm, fusing the underexposed image, the normally exposed image, the long-exposure image, and the brightened frame image to obtain a fused HDR image.

[0231] For example, to more clearly demonstrate the enhancement effect of the brightened frame image obtained by nonlinear brightening processing on the HDR image obtained by image fusion, this example shows a comparison between the HDR image obtained by the scheme of brightened frame image based on nonlinear brightening processing (i.e., image fusion based on underexposed image, normal exposure image, long exposure image and brightened frame image) and the HDR image obtained by the scheme of exposure bracketing (i.e., image fusion based on underexposed image, normal exposure image and long exposure image).

[0232] like ​ As shown, ​ This is a comparison diagram of the effects of an image processing method and an image processing method based on bracketing exposure, according to an exemplary embodiment. As shown in the diagram, the brightened frame image obtained by nonlinear brightening processing of the first image improves the image quality of the fused image and can effectively reduce ghosting (afterimages, motion blur) effects in high dynamic range scenes.

[0233] For example, to more clearly demonstrate the improvement effect of nonlinear brightening processing on the HDR image obtained by image fusion, this example shows a comparison between an HDR image obtained by a scheme based on brightened frame images obtained by nonlinear brightening processing (i.e., image fusion based on underexposed images, normally exposed images, long exposed images, and brightened frame images obtained by nonlinear brightening processing) and an HDR image obtained by a scheme based on brightened frame images obtained by linear brightening processing (i.e., image fusion based on brightened frame images obtained by linear brightening processing of underexposed images, normally exposed images, and long exposed images).

[0234] like ​ As shown, ​ This is a comparison diagram illustrating the effects of fused images obtained based on nonlinear brightening processing and linear brightening processing, according to an exemplary embodiment. As shown in the diagram, compared to the fused image obtained through linear brightening processing, the fused image obtained through nonlinear brightening processing can better control underexposed and overexposed areas in the image, reducing the loss of effective pixel information.

[0235] This disclosure also provides an image processing apparatus. ​ This is a schematic diagram illustrating the structure of an image processing apparatus according to an exemplary embodiment, such as... ​ As shown, the image processing device 100 includes:

[0236] The acquisition module 101 is used to acquire a first image, a second image, and a third image of the same target object; wherein the exposure of the first image is less than the exposure of the second image; and the exposure of the second image is less than the exposure of the third image.

[0237] The brightening module 102 is used to perform a first brightness processing on the first image to obtain a fourth image; wherein the first brightness processing is: performing brightening processing on at least two pixels with different brightness in the first image by different brightening factors, and the brightening factor corresponding to the pixel is inversely proportional to the brightness of the pixel.

[0238] The processing module 103 is used to perform image fusion on the first image, the second image, the third image and the fourth image to obtain a fused image.

[0239] Optionally, the brightening module 102 is used for:

[0240] Based on the luminance components of the first image and the second image, a luminance mapping relationship between the first image and the second image is determined; the luminance mapping relationship is used to indicate the nonlinear relationship between the luminance components of each pixel in the first image and the luminance components of each pixel in the second image.

[0241] Based on the brightness mapping relationship, the brightness enhancement factor corresponding to each pixel in the first image is determined, and the brightness component of each pixel is brightened to obtain the fourth image with enhanced brightness.

[0242] Optionally, the brightening module 102 is used for:

[0243] A first brightness processing is performed on the brightness components of each pixel in the first image, and the brightness components of each pixel after the first brightness processing are determined as the brightness components of each pixel in the fourth image.

[0244] Based on the color components of one or more of the first image, the second image, and the third image, determine the color components of each pixel in the fourth image;

[0245] The fourth image is generated based on the brightness component and color component of each pixel in the fourth image after brightening.

[0246] Optionally, the brightening module 102 is further configured to:

[0247] The first and second images in RGB format are converted to YUV format to obtain the first and second images in YUV format.

[0248] Brightness sampling is performed on the same pixel positions of the first and second images in YUV format to obtain a set of pixels;

[0249] Obtain the first luminance component and the second luminance component of each pixel in the pixel set in the first image and the second luminance component in the second image;

[0250] The brightness mapping relationship between the first image and the second image is determined based on the first brightness component and the second brightness component.

[0251] Optionally, the processing module 103 is configured to:

[0252] Based on the second image, the fourth image is subjected to ghosting removal processing to obtain the fifth image;

[0253] The first image, the third image, and the fifth image are fused to obtain a fused high dynamic range (HDR) image.

[0254] Optionally, the processing module 103 is further configured to:

[0255] Based on the second image, the position of the target object area in the fourth image is corrected to obtain the fifth image; the position of the target object area in the fifth image is the same as the position of the target object area in the second image.

[0256] Optionally, the processing module 103 is further configured to:

[0257] Obtain a first region from the first image, and obtain a second region from the third image;

[0258] The first region and the second region are fused into the fifth image to obtain a high dynamic range (HDR) image.

[0259] Wherein, the first region in the HDR image is the same as the first region in the first image; the second region in the HDR image is the same as the second region in the third image; the average brightness of the first region is greater than the average brightness of the first image, and the average brightness of the second region is less than the average brightness of the third image.

[0260] Optionally, the acquisition module 101 is configured to:

[0261] Obtain a request to generate an HDR image; wherein the generation request carries multiple different brightness requirement levels;

[0262] Based on the multiple different brightness requirement levels, the exposure time corresponding to the multiple different brightness requirement levels is determined;

[0263] First, second, and third images of the same target object were acquired based on multiple different exposure times.

[0264] ​ This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment. For example, apparatus 800 may be a mobile phone, a mobile computer, etc.

[0265] Reference ​ The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0266] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0267] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0268] Power supply component 806 provides power to various components of device 800. Power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 800.

[0269] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0270] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0271] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0272] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0273] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as Wi-Fi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0274] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0275] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0276] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0277] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Acquire a first image, a second image, and a third image of the same target object; wherein the exposure of the first image is less than the exposure of the second image; and the exposure of the second image is less than the exposure of the third image. The first image is subjected to a first brightness processing to obtain a fourth image; wherein the first brightness processing is: at least two pixels with different brightness in the first image are brightened by different brightening factors, and the brightening factor corresponding to the pixel is inversely proportional to the brightness of the pixel; The first image, the second image, the third image, and the fourth image are fused to obtain a fused image; The step of performing a first brightness processing on the first image to obtain a fourth image includes: The first and second images in RGB format are converted to YUV format to obtain the first and second images in YUV format. Based on the luminance components of the first image and the second image in YUV format, a luminance mapping relationship between the first image and the second image is determined; the luminance mapping relationship is used to indicate the non-linear relationship between the luminance components of each pixel in the first image and the luminance components of each pixel in the second image. Based on the brightness mapping relationship, the brightness enhancement factor corresponding to each pixel in the first image is determined, and the brightness component of each pixel is brightened to obtain the fourth image with enhanced brightness.

2. The method according to claim 1, characterized in that, The step of performing a first brightness processing on the first image to obtain a fourth image includes: Obtain the brightness mapping relationship between the first image and the second image; the brightness mapping relationship is used to indicate the non-linear relationship between the brightness components of each pixel in the first image and the brightness components of each pixel in the second image; Based on the brightness mapping relationship, the brightness factor corresponding to each pixel in the first image is determined, and based on the brightness factor corresponding to each pixel, the brightness component of each pixel is brightened to obtain the brightness component of each pixel in the fourth image. Based on the color components of one or more of the first image, the second image, and the third image, determine the color components of each pixel in the fourth image; The fourth image is generated based on the brightness component and color component of each pixel in the fourth image after brightening.

3. The method according to claim 1, characterized in that, Determining the luminance mapping relationship between the first image and the second image based on the luminance components of the first image and the second image in YUV format includes: Brightness sampling is performed on the same pixel positions of the first and second images in YUV format to obtain a set of pixels; Obtain the first luminance component and the second luminance component of each pixel in the pixel set in the first image and the second luminance component in the second image; The brightness mapping relationship between the first image and the second image is determined based on the first brightness component and the second brightness component.

4. The method according to claim 1, characterized in that, The step of fusing the first image, the second image, the third image, and the fourth image to obtain the fused image includes: Based on the second image, the fourth image is subjected to ghosting removal processing to obtain the fifth image; The first image, the third image, and the fifth image are fused to obtain a fused high dynamic range (HDR) image.

5. The method according to claim 4, characterized in that, The process of removing ghosting from the fourth image based on the second image to obtain the fifth image includes: Based on the second image, the position of the target object area in the fourth image is corrected to obtain the fifth image; the position of the target object area in the fifth image is the same as the position of the target object area in the second image.

6. The method according to claim 5, characterized in that, The step of fusing the first image, the third image, and the fifth image to obtain a fused high dynamic range (HDR) image includes: Obtain a first region from the first image, and obtain a second region from the third image; The first region and the second region are fused into the fifth image to obtain a high dynamic range (HDR) image. Wherein, the first region in the HDR image is the same as the first region in the first image; the second region in the HDR image is the same as the second region in the third image; the average brightness of the first region is greater than the average brightness of the first image, and the average brightness of the second region is less than the average brightness of the third image.

7. The method according to claim 1, characterized in that, The acquisition of the first, second, and third images of the same target object includes: Obtain a request to generate an HDR image; wherein the generation request carries multiple different brightness requirement levels; Based on the multiple different brightness requirement levels, the exposure time corresponding to the multiple different brightness requirement levels is determined; First, second, and third images of the same target object were acquired based on multiple different exposure times.

8. An image processing apparatus, characterized in that, include: An acquisition module is used to acquire a first image, a second image, and a third image of the same target object; wherein the exposure of the first image is less than the exposure of the second image; and the exposure of the second image is less than the exposure of the third image. A brightening module is used to perform a first brightness processing on the first image to obtain a fourth image; wherein the first brightness processing is: performing brightening processing on at least two pixels with different brightness in the first image by different brightening factors, and the brightening factor corresponding to the pixel is inversely proportional to the brightness of the pixel; The processing module is used to perform image fusion on the first image, the second image, the third image, and the fourth image to obtain a fused image; The brightening module is further configured to perform color space conversion on the first and second images in RGB format to obtain the first and second images in YUV format; determine the brightness mapping relationship between the first and second images based on the brightness components of the first and second images in YUV format; the brightness mapping relationship is used to indicate the non-linear relationship between the brightness components of each pixel in the first image and the brightness components of each pixel in the second image; based on the brightness mapping relationship, determine the brightening factor corresponding to each pixel in the first image, and perform brightening processing on the brightness components of each pixel to obtain a fourth image with enhanced brightness.

9. The apparatus according to claim 8, characterized in that, The highlighting module is used for: Obtain the brightness mapping relationship between the first image and the second image; the brightness mapping relationship is used to indicate the non-linear relationship between the brightness components of each pixel in the first image and the brightness components of each pixel in the second image; Based on the brightness mapping relationship, the brightness factor corresponding to each pixel in the first image is determined, and based on the brightness factor corresponding to each pixel, the brightness component of each pixel is brightened to obtain the brightness component of each pixel in the fourth image. Based on the color components of one or more of the first image, the second image, and the third image, determine the color components of each pixel in the fourth image; The fourth image is generated based on the brightness component and color component of each pixel in the fourth image after brightening.

10. The apparatus according to claim 8, characterized in that, The brightening module is also used for: Brightness sampling is performed on the same pixel positions of the first and second images in YUV format to obtain a set of pixels; Obtain the first luminance component and the second luminance component of each pixel in the pixel set in the first image and the second luminance component in the second image; The brightness mapping relationship between the first image and the second image is determined based on the first brightness component and the second brightness component.

11. The apparatus according to claim 8, characterized in that, The processing module is used for: Based on the second image, the fourth image is subjected to ghosting removal processing to obtain the fifth image; The first image, the third image, and the fifth image are fused to obtain a fused high dynamic range (HDR) image.

12. The apparatus according to claim 11, characterized in that, The processing module is further configured to: Based on the second image, the position of the target object area in the fourth image is corrected to obtain the fifth image; the position of the target object area in the fifth image is the same as the position of the target object area in the second image.

13. The apparatus according to claim 12, characterized in that, The processing module is further configured to: Obtain a first region from the first image, and obtain a second region from the third image; The first region and the second region are fused into the fifth image to obtain a high dynamic range (HDR) image. Wherein, the first region in the HDR image is the same as the first region in the first image; the second region in the HDR image is the same as the second region in the third image; the average brightness of the first region is greater than the average brightness of the first image, and the average brightness of the second region is less than the average brightness of the third image.

14. The apparatus according to claim 8, characterized in that, The acquisition module is used for: Obtain a request to generate an HDR image; wherein the generation request carries multiple different brightness requirement levels; Based on the multiple different brightness requirement levels, the exposure time corresponding to the multiple different brightness requirement levels is determined; First, second, and third images of the same target object were acquired based on multiple different exposure times.

15. An image processing apparatus, characterized in that, include: processor; Memory used to store executable instructions; The processor is configured to implement the image processing method according to any one of claims 1-7 when executing executable instructions stored in the memory.

16. A non-transitory computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the image processing apparatus, the image processing apparatus is able to perform the image processing method according to any one of claims 1-7.

Images