Image processing method and related device

Abstract

The present application relates to the field of electronic technology, and provides an image processing method and a related device. The method is applied to an electronic device. The electronic device can obtain at least one first image and at least one second image in response to a first operation of a user on a shooting interface, wherein the at least one first image and the at least one second image are obtained in response to shooting an HDR image; the at least one first image and the at least one second image are fused to obtain an HDR image; the at least one first image is denoised to obtain a non-HDR image; then the chromaticity of the HDR image is corrected according to the chromaticity and brightness of the non-HDR image to obtain a target chromaticity, and finally the HDR image with the target chromaticity is saved, so as to achieve the purpose of making the color of the HDR image close to the real color and improve the user experience.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method and related devices. Background Art

[0002] With the advancement of electronic technology, the shooting function of electronic devices (such as mobile phones) has developed rapidly. In order to improve the shooting quality, some mobile phones have integrated high-dynamic range (HDR) function, so that the images output by the mobile phone can present a larger dynamic range.

[0003] In order to realize the HDR function, multiple frames of images with different exposure times are usually fused to output an HDR image with more details in both brighter and darker areas. However, the fused HDR image has color cast problems of varying degrees, which affects the user experience. How to correct the color of HDR images is an urgent problem to be solved. Summary of the invention

[0004] An image processing method and a related device provided in an embodiment of the present application can correct the color of an HDR image so that the color of the HDR image is close to the real color.

[0005] In a first aspect, the present application provides an image processing method, which can be applied to an electronic device with a shooting function. In the method, the electronic device obtains at least one first image (for example, the number of first images is 4) and at least one second image (for example, the number of second images is 2) in response to a first operation of a user on a shooting interface, wherein the first operation user triggers the electronic device to shoot a high dynamic range HDR image, the at least one first image includes a normal exposure image, and the at least one second image includes a short exposure image; the at least one first image and the at least one second image are merged to obtain the HDR image; the at least one first image is denoised to obtain a non-HDR image; the target chromaticity of the HDR image is determined according to the chromaticity and brightness of the non-HDR image; and the HDR image with the target chromaticity is saved.

[0006] In one implementation, the electronic device may also acquire at least one third image (for example, the number of third images is 1) in response to the user's first operation on the shooting interface, and the at least one third image includes a long exposure image.

[0007] Among them, at least one first image can be a preview frame displayed on the shooting interface of the electronic device, the color presented by the first image can be considered as the real color of the photographed object, and at least one second image or at least one second image and at least one third image are collected to obtain an HDR image.

[0008] Among them, the present application performs a separate denoising process on at least one first image, which can reduce the noise interference problem to the greatest extent, so that the determined target chromaticity is as close to the real color as possible.

[0009] It is understandable that in order to display a wider range of brightness changes from the darkest to the brightest, HDR technology is proposed. HDR technology usually continuously collects two or more frames of images with different exposure times (for example, at least one first image, at least one second image, and at least one third image), and fuses these images to obtain an HDR image. In the fusion process, the image collected under normal exposure time (i.e., the first image) can be used as a reference image, and the best exposure value in the image collected under abnormal exposure time (such as a short frame image and a long frame image) can be fused into the first image, so as to correct the missing details in the first image and finally obtain an HDR image. However, in the fusion correction process, due to the nonlinear mapping of the neural network model, the pixel values ​​of the non-overexposed area of ​​the fused HDR image may deviate from the first image (for example, the ratio between the red component R, the green component G, and the blue component B deviates), thereby causing the color of the HDR image to deviate. It is understandable that the non-overexposed area can be considered as an area in the first image with normal brightness and no overexposure or underexposure, that is, an area that does not need to be corrected. Therefore, because the first image is a preview frame displayed in the shooting interface, the color presented by the first image can be used as the real color, so that the color of the HDR image can be corrected by the color of the first image (for example, the chromaticity of the YUV color space) and the color of the HDR image (for example, the chromaticity of the YUV color space), so that the color of the HDR is close to the real color.

[0010] In a possible implementation of the first aspect, determining a target chromaticity of the HDR image according to the chromaticity and brightness of the non-HDR image includes:

[0011] Determining a first luminance and a first chrominance of the non-HDR image, and a second chrominance of the HDR image;

[0012] The first chromaticity of the non-HDR image and the second chromaticity of the HDR image are fused according to the first luminance of the non-HDR image to determine the target chromaticity of the HDR image.

[0013] The target chromaticity is mainly used to correct the color of the normal area, and for HDR images and non-HDR images, the brightness of the normal area is completely consistent. Therefore, in this application, the chromaticity of the non-HDR image and the chromaticity of the HDR image can be fused according to the brightness of the non-HDR image in the YUV space to obtain the target chromaticity, thereby achieving the purpose of correcting the color cast of the HDR image.

[0014] In a possible implementation of the first aspect, fusing the first chromaticity of the non-HDR image and the second chromaticity of the HDR image according to the first brightness of the non-HDR image to determine the target chromaticity of the HDR image includes:

[0015] determining the first color weight according to a first brightness of the non-HDR image;

[0016] A target chromaticity corresponding to the HDR image is determined according to the first color weight, the first chromaticity, and the second chromaticity.

[0017] It is understandable that for different brightness in the image, the corresponding chromaticity fusion processing is different. In the present application, the color weight is used to determine the proportion of the first chromaticity and the second chromaticity in the chromaticity fusion, so different color weights can correspond to different chromaticity fusion processing, and different target chromaticities can be determined based on different color weights, so that different target chromaticities can match different brightness, so that the final color-corrected image can present better colors.

[0018] In a possible implementation of the first aspect, determining the first color weight according to the first brightness of the non-HDR image includes:

[0019] Determining a first brightness weight relationship, wherein the first brightness weight relationship is used to represent a relationship between a brightness value within a first brightness range and a preset weight range, and the first brightness range is used to indicate the brightness of the non-HDR image;

[0020] The first color weight is determined from the first brightness weight relationship according to a first brightness of the non-HDR image.

[0021] The first brightness weight relationship includes the corresponding relationship between the horizontal coordinate and the vertical coordinate, the horizontal coordinate is the brightness value after the non-HDR image is converted to the YUV color space and normalized, the brightness value includes the maximum brightness value and the minimum brightness value in the non-HDR image, and the vertical coordinate is the weight value required for chromaticity fusion, and the weight value range is [0,1]. Therefore, the corresponding color weight can be determined from the first brightness weight relationship according to the current brightness.

[0022] In a possible implementation of the first aspect, the first brightness range includes a first brightness threshold and a second brightness threshold, the second brightness threshold is greater than the first brightness threshold, the preset weight range includes a first weight value and a second weight value, and the second weight value is greater than the first weight value;

[0023] The determining the first color weight from the first brightness weight relationship according to the first brightness of the non-HDR image includes:

[0024] When the first brightness of the non-HDR image is less than or equal to the first brightness threshold, the first color weight is the second weight value;

[0025] When the first brightness of the non-HDR image is greater than or equal to the second brightness threshold, the first color weight is the first weight value;

[0026] When the first brightness of the non-HDR image is less than the second brightness threshold and greater than the first brightness threshold, the first color weight is determined according to the first brightness, the second weight value, the first brightness threshold and the second brightness threshold.

[0027] Exemplarily, when the first brightness of the non-HDR image is less than the second brightness threshold and greater than the first brightness threshold, the first color weight is determined based on linear interpolation according to the first brightness, the second weight value, the first brightness threshold and the second brightness threshold.

[0028] Among them, the situation where the brightness threshold is equal to the first brightness threshold can also be placed in another branch of the embodiment of the present application, for example, the first brightness of the non-HDR image is less than the second brightness threshold and greater than or equal to the first brightness threshold; the situation where the brightness threshold is equal to the second brightness threshold can also be placed in another branch of the embodiment of the present application, for example, the first brightness of the non-HDR image is less than or equal to the second brightness threshold and greater than the first brightness threshold.

[0029] In the embodiment of the present application, the first brightness threshold corresponds to the lowest brightness in the non-HDR image. When the brightness value is less than or equal to the first brightness threshold, it means that the chromaticity deviation of the area corresponding to the brightness value is large, and its color weight corresponds to the maximum weight value, which belongs to the fixed fusion weight; the second brightness threshold corresponds to the highest brightness in the non-HDR image. When the brightness value is greater than or equal to the second brightness threshold, it means that the area corresponding to the brightness value belongs to the overexposed area. Since the present application mainly deals with the color cast of the non-overexposed area, the minimum weight value corresponding to the corresponding color weight can achieve the purpose of not color fusion of the area after processing the area based on the minimum weight value. The brightness between the first brightness threshold and the second brightness threshold belongs to the common threshold in non-HDR images, so the corresponding weight can be determined according to the specific brightness value, so as to adapt to different brightness and achieve the purpose of color correction in different regions.

[0030] In a possible implementation of the first aspect, determining a target chromaticity corresponding to the HDR image according to the first color weight, the first chromaticity, and the second chromaticity includes:

[0031] Taking a sliding window as a unit, respectively, averaging the first chromaticity corresponding to each pixel in the non-HDR image and the second chromaticity corresponding to each pixel in the HDR image to obtain a first average chromaticity of the non-HDR image and a second average chromaticity of the HDR image;

[0032] A target chromaticity of the HDR image is determined according to a first color weight of the sliding window in the non-HDR image, the first average chromaticity, and the second average chromaticity.

[0033] In this application, in order to reduce the impact of individual abnormal pixels or local color blocks on the overall result, a pixel-by-pixel sliding window process can be used to determine the average chromaticity and its corresponding color weight in units of sliding windows. In addition, the use of sliding window processing can reduce the computational complexity and reduce the storage space requirements.

[0034] In a possible implementation of the first aspect, the performing averaging processing on the first chromaticity corresponding to each pixel in the non-HDR image and the second chromaticity corresponding to each pixel in the HDR image in units of sliding windows to obtain the first average chromaticity and the second average chromaticity includes:

[0035] Performing edge detection on the non-HDR image to determine an edge area of ​​the non-HDR image;

[0036] Determine whether the pixels in the sliding window of the non-HDR image and the pixels in the sliding window of the HDR image are located in the edge area. If it is determined that neither of them is located in the edge area, average the first chromaticity corresponding to the pixels in the sliding window of the non-HDR image and the second chromaticity corresponding to the pixels in the sliding window of the HDR image to obtain the first average chromaticity and the second average chromaticity.

[0037] In a possible implementation of the first aspect, the method further includes:

[0038] If it is determined that there are pixels within the sliding window of the non-HDR image and / or the pixels within the sliding window of the HDR image located in the edge area, the first chromaticities corresponding to the pixels within the sliding window of the non-HDR image located in the edge area and / or the second chromaticities corresponding to the pixels within the sliding window of the HDR image located in the edge area are averaged to obtain the first average chromaticity and / or the second average chromaticity.

[0039] In the present application, the information of the edge area is added in the sliding window processing, which can avoid color crosstalk, improve the accuracy of the target chromaticity, and thus ensure that the final color is close to the real color.

[0040] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the image processing method described in the first aspect or any possible implementation of the first aspect.

[0041] In a third aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the image processing method described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0042] In a possible implementation, the chip or chip system described above in the embodiment of the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).

[0043] In a fourth aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a processor, the computer executes the image processing method described in the first aspect or any possible implementation of the first aspect.

[0044] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a communication device, enables the communication device to execute an image processing method as described in the first aspect or any possible implementation of the first aspect.

[0045] It should be understood that the description of technical features, technical solutions, beneficial effects or similar language in this application does not imply that all features and advantages can be realized in any single embodiment. On the contrary, it is understood that the description of features or beneficial effects means that specific technical features, technical solutions or beneficial effects are included in at least one embodiment. Therefore, the description of technical features, technical solutions or beneficial effects in this specification does not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions and beneficial effects described in the present embodiment can also be combined in any appropriate manner. Those skilled in the art will understand that the embodiment can be realized without one or more specific technical features, technical solutions or beneficial effects of a specific embodiment. In other embodiments, additional technical features and beneficial effects can also be identified in a specific embodiment that does not embody all embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The following is an introduction to the drawings used in the embodiments of the present application.

[0047] Figure 1 is a schematic diagram of a color cast HDR image provided in an embodiment of the present application;

[0048] Figure 2 is a structural diagram of an electronic device 100 provided in an embodiment of the present application;

[0049] Figure 3 The software architecture of the electronic device of the embodiment of the present application is exemplarily shown;

[0050] Figure 4 is a structural schematic diagram of a high dynamic range processing module provided in an embodiment of the present application;

[0051] Figure 5 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0052] Figure 6 is a schematic diagram of a first brightness weight relationship provided in an embodiment of the present application;

[0053] Figure 7 It is a schematic diagram of color correction of a high dynamic range image based on a non-high dynamic range image provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.

[0055] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0056] The term "user interface (UI)" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. A control, also called a widget, is a basic element of a user interface. Typical controls include a toolbar, menu bar, text box, button, scrollbar, picture and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as HTML, CSS, and JavaScript. A node corresponds to a control or attribute in the interface. After parsing and rendering, the node is presented as user-visible content. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code. For example, HTML is defined by 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0057] The most common form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operation that is displayed in a graphical manner. It can be an icon, window, control or other interface element displayed on the display screen of an electronic device, where a control can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

[0058] In order to clearly describe the technical solutions of the embodiments of the present application, some terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0059] 1. The red green blue (RGB) color space or RGB domain is a color space based on the additive color model, which refers to a color model related to the structure of the human visual system. In the RGB color space, each pixel consists of three components: red (R), green (G), and blue (B), and the value of this component ranges from 0 to 255.

[0060] For example, when the exposure value increases, the RGB value of each pixel in the image will also increase accordingly, causing the overall image to become brighter; when the exposure value decreases, the RGB value of each pixel in the image will also decrease accordingly, causing the overall image to become darker.

[0061] 2. YUV color space or YUV domain is a color space based on the subtractive color model, which refers to a color encoding method. Y represents brightness (Luminance or Luma), U and V represent chrominance (Chrominance or Chroma), which can specifically represent the blue and red in hue (Chroma). When encoding photos or videos, considering human perception, it is allowed to reduce the bandwidth of chrominance, that is, the grayscale value.

[0062] Among them, the proper terms such as Y'UV, YUV, YCbCr, and YPbPr can all be called YUV, and there are overlaps with each other.

[0063] Among them, RGB color space focuses on the human eye's perception of color, while YUV color space focuses on the visual sensitivity to brightness. RGB color space and YUV color space can be converted to each other, and the conversion formula is as follows:

[0064] The calculation formula for RGB to YUV is:

[0065] Y=0.299R 0.587G 0.114B;

[0066] U= 0.14713R 0.28886G 0.436BU;

[0067] V=0.615R 0.51499G 0.10001B.

[0068] The calculation formula for YUV to RGB:

[0069] R=Y 1.13983V;

[0070] G=Y 0.39465U 0.58060V;

[0071] B=Y 2.03211U.

[0072] 3. Pixels and pixel points. Pixels are the smallest units in digital images. They represent a specific spatial location and color information. Each pixel has its own grayscale value or color value. Pixel points refer to the smallest controllable light point on an electronic device. They are the basic building blocks of the smallest unit in the picture, such as the physical light point of a display screen. Each pixel corresponds to one or more pixel points.

[0073] The grayscale value or color value corresponding to a pixel can be called a pixel value. In different color space models, pixel values ​​are represented differently. For example, in the RGB color space, a pixel value refers to a set of three primary color components corresponding to each pixel, and the three primary color components correspond to the red component R, the green component G, and the blue component B. For a grayscale image, a pixel value is a single value.

[0074] 4. Exposure area: The brightness of different areas of the subject is called exposure area. These exposure areas are reflected in the grayscale value in the final image. The grayscale value can reflect the lighting conditions in the natural environment, that is, the subject and the background should be within a reasonable brightness range.

[0075] Exemplarily, the grayscale value generally ranges from 0 to 255, with a grayscale value of 255 representing white, a grayscale value of 0 representing black, and the values ​​in between representing different degrees of gray. Generally speaking, the grayscale value of the normally exposed area (also referred to as the non-overexposed area) is usually controlled to be around 50% of the entire grayscale range, such as around 127 or 128. This value can ensure that the brightness and contrast of the image are optimal. The grayscale value of the overexposed area usually approaches or reaches the maximum grayscale value (such as 255), which may cause loss of image details. The grayscale value of the under-exposed area usually approaches or reaches the minimum grayscale value, such as less than 0.5.

[0076] 5. High Dynamic Range (HDR) imaging technology is a technology used to capture and present a wider range of brightness and color. This technology expands the dynamic range of the camera by combining multiple images with different exposures. The resulting high dynamic range HDR image can show a full range from the brightest details to the darkest details.

[0077] In photography, exposure refers to the amount of light that passes through the lens and hits the photosensitive element. This amount is determined by three parameters: aperture, shutter speed, and sensitivity (ISO).

[0078] The multiple images with different exposures include one or more of the following normal exposure images, short frame images, and long frame images:

[0079] Normal exposure image: or N-frame image, which is the image captured by the camera when the exposure value is 0 exposure value (EV). In other words, the exposure value of the normal exposure image is 0EV. The 0EV here is a relative value, not an exposure value of 0. For example, exposure value = exposure time * sensitivity (ISO). Assuming that the normal exposure image is captured when the ISO is 200 and the exposure time is 50 milliseconds, the actual exposure value corresponding to 0EV is the product of 200 and 50 milliseconds.

[0080] Short-frame image: also called S-frame image or short-exposure image, which is an image captured by the camera when the exposure value is less than 0EV. In other words, the exposure value of the short-frame image is less than 0EV, such as -2EV, -4EV, etc.

[0081] Long frame image: also called L frame image or long exposure image, which is an image taken by the camera with an exposure value greater than 0EV. In other words, the exposure value of the short frame image is greater than 0EV, such as the exposure value of the long frame image is 2EV, 4EV, etc.

[0082] It is understandable that, because different areas of the subject have different lighting conditions (for example, some areas are too dark, some areas are too bright, and some areas have normal brightness), the normal exposure image obtained by shooting with normal exposure includes one or more of the normal exposure area, the overexposure area, and the underexposure area. Among them, the brightness of the normal exposure area is neither too bright nor too dark, and can be in color; the brightness of the overexposure area is too bright and appears white; the brightness of the underexposure area is too dark and appears black.

[0083] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0084] In some shooting scenes (for example, there is a light source in the shooting picture), the shooting picture may have a large difference in brightness and darkness. Therefore, the image obtained by taking a normal exposure value is prone to bright areas being too bright (for example, overexposed areas) or dark areas being too dark (for example, underexposed areas). In this case, the captured image may lose details in bright or dark areas, affecting the image quality. In order to improve the image quality, the electronic device can provide an HDR function, and the image captured based on the HDR function can be called an HDR image. Among them, an HDR image is an image with a high dynamic range, that is, the difference between the brightest and darkest parts in the HDR image is large, so that it can better reflect the visual effect in the real shooting environment and be closer to the effect seen by the human eye.

[0085] However, HDR images processed by electronic devices may have color cast problems. For example, see Figure 1 , Figure 1 Schematic diagram of a color cast HDR image provided by an embodiment of the present application. Figure 1 As shown, the electronic device 100 uses a camera to shoot an object 200 in an actual scene in HDR mode, thereby displaying an image of the object 200 in the preview interface 101 corresponding to the HDR mode. Then, in response to an operation such as a user clicking on the photo control 312, the electronic device 100 may use the HDR function to shoot images with different exposure values ​​in the same scene, thereby obtaining a normal exposure image and an abnormal exposure image. Among them, the abnormal exposure image may include a short frame image and / or a long frame image. Next, the electronic device 100 performs a fusion process on the images with different exposure values ​​based on an internally deployed image processing model (for example, a high dynamic fusion model based on a neural network), that is, after fusing the normal exposure image and the abnormal exposure image, an HDR image 102 corresponding to the object 200 may be output. From Figure 1 It can be seen that the HDR image 102 corresponding to the object 200 is compared with the object 200 in the actual scene and the object 200 in the preview interface 101. The color of the HDR image finally output by the electronic device 100 deviates from the real color in the actual scene. For example, the color presented by the first area 201 in the HDR image 102 is compared with the real color in the actual scene (for example, the color presented by the first area 201 in the object 200 or the color presented by the first area in the preview interface 101). Exemplarily, in terms of display effect, the color cast problem may cause the entire or partial area of ​​the image to present a hue that is biased to a certain color to a certain extent, such as reddish, greenish or bluish.

[0086] The image of the object 200 displayed in the preview interface 101 is a normally exposed image, and the color presented by the image may be close to the real color.

[0087] The first area 201 is a normal exposure area, which can also be called a non-overexposed area. It is understandable that in an actual shooting scene (for example, there is a light source in the shooting picture), the light intensity reflected by the first area 201 of the shooting object 200 is normal intensity, and the light is neither too strong nor too weak. Therefore, when the electronic device 100 takes a picture of the shooting object 200, the first area 201 in the HDR image obtained by taking the picture is a normal exposure area.

[0088] In the multi-frame fusion scheme, the image processing model deployed in the electronic device 100 is trained based on a sample set containing a normal exposure image and an abnormal exposure image. Therefore, in actual use, the electronic device 100 first uses the normal exposure image as a reference image, and then inputs the abnormal exposure image and the reference image into the neural network model. In the neural network model, the abnormal exposure image is aligned with the reference image, and then the optimal exposure value in the abnormal exposure image is fused into the reference image, so as to correct the missing details in the reference image and finally obtain an HDR image. For example, in the case where there may be a large difference in light and dark in the captured image, there are one or more of the normal exposure area, overexposure area and underexposure area in the normal exposure image. The electronic device can correct the underexposure area or overexposure area in the normal exposure image according to the abnormal exposure image, and the HDR image finally obtained by fusion can make the dark details brighter and the bright details undistorted.

[0089] It is understandable that since the neural network model is a nonlinear mapping, the color ratio of different channels (such as RGB channels) will change. Therefore, for the image input into the model and the image output from the model, the ratio between the red component R, green component G and blue component B of the output image is different from the ratio between the red component R, green component G and blue component B of the input image. Therefore, the output image has a color deviation.

[0090] In addition, there is a difference between the exposure values ​​of the normal exposure image (also called the medium frame image) and the abnormal exposure image, and the exposure values ​​are not aligned. For example, assuming that the ISO of the normal exposure image is 100, and the ISO of the short frame image in the abnormal exposure image is 25, the exposure value theoretically differs by 4 times. Therefore, theoretically, the pixel value of the short frame image in the non-overexposed area is multiplied by 4 to be consistent with the pixel value of the medium frame image. However, in actual operation, when the electronic device is exposed, due to the accuracy of the hardware device, there is a convergence problem in the exposure reference. The actual exposure value difference between the normal exposure image and the abnormal exposure image is not the same as the theoretical difference, which will also affect the ratio between the red component R, green component G and blue component B of the fused HDR image, thereby causing color cast.

[0091] In addition, the signal-to-noise ratio of short-frame images in abnormal images is usually poor, resulting in large color noise, and the color noise is mainly concentrated in the non-overexposed area (also called dark area). Generally speaking, the neural network model uses the entire image for fusion, so during the fusion process, the non-overexposed area will inevitably introduce some short-frame information, which makes the output HDR image easily affected by color noise and causes color cast.

[0092] Therefore, the HDR image obtained by fusing the normal exposure image and the abnormal exposure image through the image processing model has different degrees of color deviation, so the color deviation of the HDR image is a common problem in the HDR scene. The applicant has found that the probability of color cast of the fused image of equal exposure frame images (for example, fusing multiple frames of normal exposure images to obtain a fused image) is low. In view of this, the present application provides an image processing method, which is applied to electronic devices. The electronic device can use the fused image of equal exposure frame images to perform color correction on the HDR image, thereby outputting an HDR image without color cast.

[0093] First of all, the electronic device in the embodiments of the present application can be a smart screen device, a smart television (television, TV), a mobile phone, a tablet computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (personal digital assistant, PDA), a wearable device (such as: a smart watch, a smart bracelet), and other devices with display functions. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0094] For example, taking the electronic device as a mobile phone, Figure 2 is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application. That is, exemplary, Figure 2 The electronic device shown may be a mobile phone.

[0095] like Figure 2 As shown, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 129, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0096] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0097] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0098] The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions. The processor 110 can also be provided with a memory for storing instructions and data.

[0099] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0100] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0101] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), miniled, microled, micro-oled, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0102] The electronic device can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, etc. In some embodiments, the shooting function includes shooting an HDR image.

[0103] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can also perform algorithm optimization on the noise and brightness of the image. The ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0104] The camera 193 is used to capture still images or videos. In some embodiments, the electronic device may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0105] Exemplarily, the photographed object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format.

[0106] In an embodiment of the present application, the camera 193 may include one or more of the following cameras: a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, a zoom camera, or a depth camera. Among them, the telephoto camera has a small shooting range and is suitable for shooting distant objects; the wide-angle camera has a larger shooting range; the ultra-wide-angle camera has a larger shooting range than the wide-angle camera and is suitable for shooting panoramic and other large-screen scenes. The depth camera can be used to measure the object distance of the object to be photographed, that is, to measure the depth information of the object to be photographed, for example, it may include a three-dimensional (3D) depth camera, a time of flight (TOF) depth camera, or a binocular depth camera.

[0107] The camera 193 may include a main camera and a secondary camera. The main camera may be used to capture images, for example, it may include a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, or a zoom camera, etc. The secondary camera may be used for ranging or other auxiliary functions, for example, it may include a depth camera.

[0108] The camera 193 may include a front camera and / or a rear camera. The front camera may include one or more main cameras, and the rear camera may also include one or more main cameras. When capturing an image, the target main camera used by the electronic device to capture the image may be a default main camera or a main camera selected by the user.

[0109] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0110] In an embodiment of the present application, the electronic device 100 can acquire images through shooting functions provided by ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0111] The external memory interface 120 can be used to connect to an external non-volatile memory to expand the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, image files such as photos obtained in the "photographing" mode or the "recording" mode are stored in the external non-volatile memory.

[0112] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor, etc. For example, after the camera application is turned on, the camera application can call the audio module 170 to obtain the ambient audio signal, and combine it with the image stream generated by the camera and other devices to form a video.

[0113] The gyro sensor 180B can be used to detect the magnitude of the angular velocity of the electronic device 100 in various directions (generally three axes). The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions. The gyro sensor 180B and the acceleration sensor 180E can be used to determine the motion posture of the electronic device 100. In the embodiment of the present application, the data reported by the acceleration sensor 180E and the gyro sensor 180B can be used to determine the motion state of the device, and then assist in determining whether the image at the corresponding moment will be jittery, ghosting, etc.

[0114] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0115] Figure 3 The software architecture of the electronic device according to the embodiment of the present application is exemplified.

[0116] like Figure 3 As shown, the system of the electronic device 100 may adopt a layered architecture, which divides the system into several layers, each layer having a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer (framework), the hardware abstract layer (HAL), the driver layer, and the hardware layer. Among them:

[0117] The application layer (application) can include a series of application packages. For example, the application package can include camera applications, gallery applications and other applications. Camera applications may include but are not limited to: UI module, camera module, gallery module and so on. Among them, the UI module can be a camera UI (cameraUI) module, which can be mainly responsible for the human-computer interaction of the camera application, such as controlling the preview interface and the display of the preview screen therein, receiving and responding to user operations occurring in the preview interface. The camera module provides camera functions, focus functions, HDR functions and so on. The gallery module can be used to store clear photos taken by users in the file system or a specific database of the electronic device for retrieval by applications such as gallery.

[0118] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. It mainly involves the camera framework, which can include camera management and camera device access interfaces, playing a role of connecting the upper and lower parts. It can interact with camera applications through the application API and interact with HAL through the HAL interface definition language (HIDL). It can also include a window manager. The camera application and the gallery application can display the photos taken to the user with the support of the window manager.

[0119] The hardware abstraction layer (HAL) is an interface layer between the application framework layer and the driver layer, which provides a virtual hardware platform for the operating system. Exemplarily, the hardware abstraction layer may include a camera hardware abstraction layer and a high dynamic range processing module. Among them, the camera hardware abstraction layer may provide virtual hardware for camera device 1 (first camera), camera device 2 (second camera) and more camera devices.

[0120] A variety of image processing algorithms are stored in the high dynamic range processing module. For example, in the embodiment of the present application, the high dynamic range processing module may include one or more of the following: high dynamic fusion algorithm, equal exposure denoising algorithm, color conversion algorithm and color correction algorithm. Among them:

[0121] The high dynamic fusion algorithm can be used to determine a group of images with different exposure values ​​through different exposure times, and then fuse the above group of images with different exposure values ​​through a preset fusion algorithm to determine an HDR image containing a full brightness range. Wherein, the group of images with different exposure values ​​includes one or more of normal exposure images and abnormal exposure images (including short frame images and / or long frame images).

[0122] The equal exposure denoising algorithm can be used to perform fusion denoising on images with the same exposure value in a group of images with different exposure values, thereby determining a non-HDR image. The images with the same exposure value include one or more normal exposure images.

[0123] The color conversion algorithm can be used to convert HDR images and non-HDR images from a first color space to a second color space, or to convert a color cast corrected image from a second color space to a first color space. The first color space is an additive color model based on the three primary colors of red, blue and green, namely the RGB color space, and the second color space is a subtractive color model based on a brightness signal (Y) and two chrominance signals (U and V), namely the YUV color space.

[0124] The color correction algorithm can be used to determine a first brightness and a first chromaticity of a non-HDR image and a second chromaticity of an HDR image, and then used to perform color correction on the second chromaticity of the HDR image according to the first brightness of the non-HDR image and the first chromaticity of the non-HDR image to determine a target chromaticity of the HDR image.

[0125] In a possible implementation, the color correction algorithm may also be used to determine a first color weight based on a first brightness corresponding to a pixel in a non-HDR image, and determine a target chromaticity corresponding to a pixel in an HDR image based on the first color weight, the first chromaticity, and the second chromaticity.

[0126] In one implementation, the color correction algorithm is further used to determine an edge region in the non-HDR image after normalizing the first brightness of the non-HDR image, wherein the edge region includes a region where pixel values ​​change dramatically, such as the outline of a portrait.

[0127] In one implementation, the color correction algorithm is also used to perform pixel-by-pixel sliding window processing on the first chromaticity of the non-HDR image and the second chromaticity of the HDR image, so as to determine the first average chromaticity of the non-HDR image and the second average chromaticity of the HDR image. Further, it can be used to calculate the first average chromaticity and the second average chromaticity according to the edge area. Still further, it can be used to determine the target chromaticity corresponding to the pixel point in the HDR image according to the first color weight, the first average chromaticity and the second average chromaticity.

[0128] The driver layer is a layer between hardware and software, which includes drivers for various hardware. The driver layer may include camera drivers, digital signal processor drivers, and image processor drivers. Among them, the camera driver is used to drive the image sensor of one or more cameras in the camera to collect images and drive the image signal processor to pre-process the image. The digital signal processor driver is used to drive the digital signal processor to process images. The image processor driver is used to drive the graphics processor to process images.

[0129] The hardware layer may include a camera, an image signal processor, a digital signal processor, and an image processor. The camera may include one or more image sensors of the camera (e.g., image sensor 1, image sensor 2, etc.). Optionally, the camera may also include a camera motor, a lens, a TOF sensor, and the like.

[0130] In the embodiment of the present application, the workflow of the software system of the electronic device 100 is exemplified in combination with the process of shooting an HDR image.

[0131] The system desktop can receive the user's operation of opening the camera application, for example, the user first clicks the desktop icon of "camera". In response to the user operation of opening the camera application, the system can start the cameraUI module and display the preview interface. The cameraUI module can be responsible for the human-computer interaction of the camera application, such as controlling the display of the preview interface and the interface elements therein, and monitoring and responding to user operations occurring in the preview interface. After starting the camera application, the camera device in the camera hardware abstraction layer, such as camera device 1, is called to send an instruction to start the camera application. The camera hardware abstraction layer sends the instruction to the camera device driver of the driver layer, and the camera device driver can start the sensor corresponding to the camera device, and then collect the image light signal through the sensor, and transmit the image light signal to the image signal processor for preprocessing to obtain multiple preview stream image frames including the photographed object. At this time, the preview stream image frame can be called the original image stream. Then, the original image stream is transmitted back to the hardware abstraction layer through the camera device driver. The camera hardware abstraction layer calls the camera device driver, and the camera device driver transmits the image frame back to the hardware abstraction layer, and the hardware abstraction layer returns the image frame containing the photographed object to the camera framework layer. After receiving the image frame from the hardware abstraction layer, the camera framework can pass the image frame to the cameraUI in the camera application to trigger the cameraUI to display the preview image frame in the preview interface. The user interface that provides shooting and display services can be displayed, also known as the preview interface, in which the image frames in the image stream captured by the camera are displayed.

[0132] In an embodiment of the present application, the cameraUI module responds to the user operation of turning on the HDR function by the user, and can start the HDR function, so that the camera application is in HDR mode. Further, in HDR mode, the cameraUI module responds to the user operation of the user clicking the photo control, and can then send an instruction to shoot an HDR image by calling the camera device in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the driver layer, and the camera device driver collects the image light signals corresponding to different exposure values ​​through the sensor, and transmits the image light signals corresponding to the different exposure values ​​to the image signal processor for preprocessing to obtain a normal exposure image and an abnormal exposure image. The camera hardware abstraction layer can give the normal exposure image and the abnormal exposure image to the high dynamic range processing module. Based on the support of the digital signal processor and the image processor, the high dynamic range processing module can call one or more processing algorithms such as the high dynamic fusion algorithm, the equal exposure denoising algorithm, the color conversion algorithm and the color correction algorithm to process the image to obtain a color-corrected HDR image. The hardware abstraction layer sends the color-corrected HDR image to the camera framework layer. After receiving the color-corrected HDR image from the hardware abstraction layer, the camera framework can pass the color-corrected HDR image to the cameraUI in the camera application to trigger the cameraUI to save the color-corrected HDR image.

[0133] See also Figure 4 , Figure 4 Schematic diagram of a high dynamic range processing module provided in an embodiment of the present application. Figure 4 The high dynamic range processing module 400 shown includes but is not limited to a high dynamic fusion model 401, a non-high dynamic fusion model 402, a color conversion module 403, and a color correction module 404. The high dynamic range processing module 400 can be implemented by hardware or software, or by a combination of hardware and software. When implemented by hardware, the high dynamic range processing module can be a logic circuit, an integrated circuit, etc. When implemented by software, the high dynamic range processing module can be a general processor, which is implemented by reading software code stored in a memory.

[0134] In one implementation, after the electronic device acquires images with different exposure values, the images with different exposure values ​​may be input into the high dynamic range fusion model 401 in the high dynamic range processing module 400, and the high dynamic range fusion model 401 performs fusion processing on the images with different exposure values ​​based on the high dynamic range fusion algorithm to obtain an HDR image. The images with different exposure values ​​include at least one first image acquired by a first exposure value and at least one second image acquired by a second exposure value, at least one first image includes a normal exposure image, at least one second image includes a short exposure image and a long exposure image, the first exposure value and the second exposure value are different, the first exposure value is for obtaining a normal exposure image, and the second exposure value is for obtaining a short exposure image or a long exposure image.

[0135] In one implementation, after the electronic device acquires images with different exposure values, images with the same exposure value among the images with different exposure values ​​may be input into the non-high dynamic fusion model 402, and the non-high dynamic fusion model 402 performs denoising on the images with the same exposure value based on the equal exposure denoising fusion algorithm to obtain a non-HDR image. The images with the same exposure value include at least one first image, that is, at least one normal exposure image.

[0136] Exemplarily, the denoising algorithm adopted by the non-high dynamic fusion model 402 includes, but is not limited to, a multi-frame denoising algorithm, a RAW domain denoising algorithm, and a neural network-based denoising algorithm.

[0137] It should be noted that, in order to ensure that the images output by the non-high dynamic fusion model 402 and the high dynamic fusion model 401 are completely aligned, the at least one first image input into the non-high dynamic fusion model 402 is consistent with the at least one first image input into the high dynamic fusion model 401 in terms of quantity, acquisition time, and exposure value. For example, if the number of first images input into the high dynamic fusion model 401 is 4, then the number of first images input into the non-high dynamic fusion model 402 is also 4.

[0138] Therefore, after acquiring the HDR image and the non-HDR image, the HDR image and the non-HDR image can be input into the color conversion module 403, and the color conversion module 403 converts the HDR image and the non-HDR image from the first color space (i.e., RGB color space) to the second color space (i.e., YUV color space) based on the color conversion algorithm, so that the first brightness and the first chromaticity of the non-HDR image, and the second brightness and the second chromaticity of the HDR image can be acquired.

[0139] Furthermore, the color correction module 404 may fuse the first chromaticity and the second chromaticity according to the first brightness of the non-HDR image based on a color correction algorithm to obtain a target chromaticity, and finally obtain an image with the second brightness and the target chromaticity.

[0140] Furthermore, the color correction module 404 inputs the image with the second brightness and the target chromaticity to the color conversion module 403, and the color conversion module 403 converts the image with the second brightness and the target chromaticity from the second color space to the first color space, thereby obtaining an HDR image with the target color.

[0141] See also Figure 5 , Figure 5 is a flow chart of an image processing method provided in an embodiment of the present application, which can be applied to Figure 2 or Figure 3 The electronic device shown can also be applied to Figure 4 The module shown in the figure includes but is not limited to the following steps:

[0142] Step S501: in response to a first operation of a user on a shooting interface, acquiring at least one first image and at least one second image.

[0143] The first operation is used to trigger the electronic device to shoot an HDR image. In one implementation, the HDR image may be an HDR photo, that is, the first operation may be used to trigger the electronic device to shoot a photo in HDR mode. In another implementation, the HDR image may be an image in an HDR video, that is, the first operation may be used to trigger the electronic device to shoot a video in HDR mode.

[0144] For example, the first operation may be a user clicking a related control on a shooting interface of a camera application displayed on the electronic device. For example, the first operation may be a user clicking a photo control (e.g. Figure 1 For another example, the first operation may be a click operation of a video recording control on a shooting interface by a user in an HDR mode of the electronic device.

[0145] It is understandable that the first operation may also be a preset gesture input by the user in the shooting interface, and the present application does not impose any restrictions on how to trigger the operation of shooting HDR images.

[0146] Specifically, the electronic device responds to a first operation on the shooting interface to obtain at least one frame of first image captured by a first exposure value and at least one frame of second image captured by a second exposure value in the same scene, wherein the second exposure value is less than the first exposure value, at least one frame of the first image includes a normal exposure image, and at least one frame of the first image includes a short exposure image.

[0147] In one implementation, the electronic device responds to a first operation on a shooting interface to obtain at least one frame of first image captured by a first exposure value, at least one frame of second image captured by a second exposure value, and at least one frame of third image captured by a third exposure value in the same scene, wherein the second exposure value is less than the first exposure value, the third exposure value is greater than the first exposure value, and at least one frame of the third image includes a long exposure image.

[0148] Exemplarily, the difference in exposure value is mainly achieved by adjusting the sensitivity ISO and exposure time. The electronic device continuously shoots at least one first image according to the preset ISO and exposure time, and then reduces the exposure value of the at least one first image to shoot at least one second image. For example, the exposure value of the first image is 0EV, and the exposure value of the first image is reduced by 1 to 3 EV to shoot the second image. Furthermore, the exposure value of at least one first image can be increased to shoot at least one third image. For example, the exposure value of the first image is 0EV, and the exposure value of the first image is increased by 1 to 3 EV to shoot the second image.

[0149] It can be understood that the preview image displayed in the shooting interface of the camera application is called the first image, that is, the normally exposed image.

[0150] Step S502: performing fusion processing on at least one first image and at least one second image to obtain an HDR image.

[0151] Specifically, the electronic device fuses at least one first image and at least one second image based on a high dynamic fusion algorithm to obtain an HDR image. Alternatively, when the electronic device acquires a third image, it may fuse at least one first image, at least one second image, and at least one third image based on a high dynamic fusion algorithm to obtain an HDR image. That is, in a high dynamic fusion transition, the electronic device may use the best details corresponding to the exposure value in each image to synthesize a final HDR image, and the HDR image may provide more dynamic range and image details, thereby better reflecting the visual scene in a real environment.

[0152] Step S503: performing denoising processing on at least one first image to obtain a non-HDR image.

[0153] In the present application, at least one first image includes a normally exposed image, that is, an image captured based on a normal exposure value. Therefore, for a normally exposed area in the first image, the normally exposed area can be within an appropriate brightness range, neither underexposed nor overexposed, and can present good details and colors. However, a normally exposed area in an HDR image obtained by fusing at least one first image and at least one second image may have a color cast. Since the color of the normally exposed area in the at least one first image is a real color, the color of the HDR can be corrected through the at least one first image.

[0154] Exemplarily, in order to ensure the quality of non-HDR images, an equal exposure denoising algorithm may be used to perform fusion denoising processing on at least one first image to obtain a non-HDR image. Taking a denoising algorithm based on a neural network as an example, in order to ensure the generation of a non-HDR image without color cast, a non-high dynamic fusion model may be obtained according to neural network model training, and then at least one first image may be fused and denoised based on the trained non-high dynamic fusion model. In this way, the non-HDR image output by the non-high dynamic fusion model has a better denoising effect, and can also ensure that it will not be interfered by noise information during the color correction process, and the image clarity is high, and it is also convenient to extract the edge area of ​​the non-HDR image required in the color correction process.

[0155] It should be noted that the at least one first image in this step is consistent with the at least one first image used in high dynamic range fusion in step S502 in terms of quantity, acquisition time and exposure value.

[0156] Step S504: Determine the target chromaticity of the HDR image according to the chromaticity and brightness of the non-HDR image.

[0157] Among them, both the non-HDR image and the HDR image are in the linear domain and the brightness of the normal exposure area is consistent. Specifically, the electronic device can perform color correction on the HDR image according to the chromaticity and brightness of the non-HDR image, so that the target chromaticity of the HDR image is close to the real color, that is, the color of the HDR image obtained after the color correction processing is basically similar to the color of the first image subjected to high dynamic range fusion.

[0158] In a possible implementation, the electronic device converts the non-HDR image and the HDR image from the RGB color space to the YUV color space, thereby obtaining the first brightness and first chromaticity of the non-HDR image, and the first brightness and first chromaticity of the HDR image. The first brightness refers to the Y component of the non-HDR image in the YUV color space, and the first chromaticity refers to the UV component of the non-HDR image in the YUV color space. The second brightness refers to the Y component of the HDR image in the YUV color space, and the second chromaticity refers to the UV component of the HDR image in the YUV color space. Therefore, the first chromaticity of the non-HDR image and the second chromaticity of the HDR image are fused in the YUV color space according to the first brightness of the non-HDR image to obtain the target chromaticity, so as to achieve the purpose of correcting the color cast of the HDR image.

[0159] In one implementation, the electronic device may determine a first brightness weight relationship according to a first brightness range of the non-HDR image, and then determine a first color weight from the first brightness weight relationship according to the first brightness of the HDR image. The first brightness weight relationship is used to represent the relationship between the brightness value of the first brightness range and the preset weight range, and the first brightness range is used to indicate the brightness of the non-HDR image.

[0160] For example, see Figure 6 , Figure 6 is a schematic diagram of a first brightness weight relationship provided in an embodiment of the present application. Figure 6 As shown, the horizontal axis is used to represent the brightness value of the non-HDR image, and can be specifically used to represent the brightness value within the first brightness range, including the first brightness threshold ( ) and the second brightness threshold ( ), the first brightness threshold is less than the second brightness threshold, the first brightness threshold is used to indicate the lowest brightness in the non-HDR image, and the second brightness threshold is used to indicate the highest brightness in the non-HDR image. For example, the Y component of the non-HDR image is counted to obtain a Y component interval [maximum Y component, minimum Y component], the first brightness threshold = minimum Y component The first preset ratio may be any one of 20% to 40%; the second brightness threshold = the maximum Y component The second preset ratio can be any one of 80% to 90%. The ordinate is used to represent the preset weight value, specifically to represent the weight within the preset weight range, specifically including the first weight value ( ) and the second weight value ( ), the second weight value is greater than the first weight value, the first weight value may be a value close to 1 in [0,1], for example, the first weight value is 0.8, and the second weight value may be a value close to 0 in [0,1], for example, the second weight value is 0. In the present application, in order to perform different degrees of color correction on areas corresponding to different brightnesses, different brightnesses may correspond to different preset weights. Figure 6 It can be seen that when the brightness value ( ) is less than or equal to the first brightness threshold, the corresponding weight value is the maximum value in the preset weight range, that is, it is the second weight, which belongs to the fixed fusion weight; when the brightness value ( ) is less than the second brightness threshold and greater than the first brightness threshold, the first color weight value ( ) decreases with the increase of brightness value, and can be based on the first brightness threshold ( ), the second brightness threshold ( ), the first weight value ( ), the second weight value ( ) and the current brightness value ( ) to determine the first color weight value; when the brightness value ( ) is greater than the second brightness threshold, the corresponding weight value is the minimum in the preset weight range, that is, it is the first weight.

[0161] In some embodiments, when the electronic device determines the first color weight from the first brightness weight relationship according to the first brightness of the non-HDR image, in one implementation, if the first brightness of the non-HDR image is less than or equal to the first brightness threshold, Figure 6 In the first brightness weight relationship shown, the first color weight can be determined as the second weight value.

[0162] In one implementation, if the first brightness of the non-HDR image is greater than or equal to the second brightness threshold, Figure 6 In the first brightness weight relationship shown, the first color weight can be determined as the first weight value. It can be understood that the first weight value is close to zero, which is equivalent to not performing color fusion when the first brightness is greater than the maximum value of the image brightness, so as to ensure that the color of the overexposed area is not fused.

[0163] In one implementation, if the first brightness of the non-HDR image is less than the second brightness threshold and greater than the first brightness threshold, then according to the first brightness ( ), the second weight value ( ), the first brightness threshold ( ) and the second brightness threshold ( )according to Figure 6 The first color weight is calculated by using the formula of the first color weight shown in ).

[0164] In some embodiments, when the electronic device determines the target chromaticity corresponding to the HDR image based on the first color weight, the first chromaticity, and the second chromaticity, for example, the electronic device may average the first chromaticity corresponding to each pixel in the non-HDR image and the second chromaticity corresponding to each pixel in the HDR image to determine the first average chromaticity of the non-HDR image and the second average chromaticity of the HDR image, and then determine the target chromaticity of the HDR image based on the first color weight, the first average chromaticity, and the second average chromaticity.

[0165] In a possible implementation, the electronic device may perform pixel-by-pixel sliding window processing on the chromaticity of the HDR image and the chromaticity of the non-HDR image, and in the sliding window processing, determine the weight coefficient required for chromaticity fusion according to the first brightness weight relationship. Specifically, in the sliding window processing, the first chromaticity corresponding to each pixel in the non-HDR image and the second chromaticity corresponding to each pixel in the HDR image are averaged in units of sliding windows to obtain the first average chromaticity and the second average chromaticity, and then the target chromaticity of the HDR image can be determined according to the first color weight of the sliding window in the non-HDR image, the first average chromaticity, and the second average chromaticity.

[0166] In one implementation, in the process of determining the average chromaticity, it is necessary to determine whether the current pixel is located in the edge area, because the changes between the pixel values ​​in the edge area are relatively drastic. For example, the area where the current pixel is located includes the current pixel and pixel 1, pixel 1 and the current pixel are two adjacent pixels, the pixel value of pixel 1 is 255, the pixel value of the current pixel is 2, the difference between the pixel values ​​of pixel 1 and the current pixel is large, and the pixel value changes are relatively drastic, so the area where the current pixel is located belongs to the edge area. Because the pixels with relatively drastic changes may affect the average chromaticity of the sliding window, the pixel points in the edge area may not be processed. That is, the electronic device performs edge detection on the non-HDR image to determine the edge area of ​​the non-HDR image; determines whether the pixels in the sliding window of the non-HDR image and the pixels in the sliding window of the high dynamic range image are located in the edge area, if it is determined that both are located in the edge area, no processing is performed; if it is determined that both are not located in the edge area, the first chromaticity corresponding to the pixels in the sliding window of the non-HDR image and the second chromaticity corresponding to the pixels in the sliding window of the high dynamic range image are averaged to obtain the first average chromaticity and the second average chromaticity.

[0167] In one implementation, if it is determined that there are pixels located in the edge area among the pixels in the sliding window of the non-HDR image (i.e., the current pixel) and the pixels in the sliding window of the high dynamic range image (i.e., the current pixel), the first chromaticity corresponding to the pixels located in the edge area in the sliding window of the non-HDR image and the second chromaticity corresponding to the pixels located in the edge area in the sliding window of the high dynamic range image are averaged to obtain a first average chromaticity and a second average chromaticity. For example, assuming that the sliding window includes the current pixel, pixel 2, pixel 3, and pixel 4, if the current pixel is located in the edge area but pixel 4 and pixel 3 are located in the edge area, it means that there are pixels located in the edge area, so the chromaticity corresponding to pixel 4 and pixel 3 may not be used in calculating the average chromaticity, and only the pixels in the sliding window that are not located in the edge area, such as the current pixel and pixel 2, may be used.

[0168] Step S505: Save the HDR image with target chromaticity.

[0169] Specifically, after determining the target chromaticity, a new HDR image after color cast correction is obtained based on the second brightness and target chromaticity of the HDR image, and then the HDR image is converted from YUV color to RGB color, and then other processing is performed on the HDR image in the RGB color space, such as lens shading correction (LSC), color correction matrix (CCM), auto white balance (AWB), etc. Finally, the electronic device saves an HDR image that is close to the real color.

[0170] See also Figure 7 , Figure 7 is a schematic diagram of color correction of a high dynamic range image based on a non-high dynamic range image provided by an embodiment of the present application. Figure 7 As shown, after receiving the user's photo operation, the electronic device obtains a non-high dynamic range image and a dynamic range image based on the image collected in the same scene, converts the non-high dynamic range image from the RGB color space to the YUV color space, thereby obtaining a first brightness (Y component) and a first chromaticity (UV component); converts the high dynamic range image from the RGB color space to the YUV color space, thereby obtaining a second brightness (Y component) and a second chromaticity (UV component). Then, the electronic device performs pixel-by-pixel sliding window processing on the non-high dynamic range image and the high dynamic range image, and the size of the sliding window is wb*hb, and wb*hb is less than the resolution of the image. Exemplarily, in the sliding window processing process, the first brightness of the non-high dynamic range image is first normalized (for example, the Canny algorithm) to extract the edge area, and then it is determined whether the current pixel is located in the edge area. If it is not located in the edge area, the current pixel is not processed, that is, the current pixel is not color corrected, and the chromaticity of the current pixel is maintained; if it is not located in the edge area, the current pixel is color corrected. Among them, the current pixel can be considered as a pixel located at the center point of the sliding window during the sliding window processing. Exemplarily, in the process of color correction of the current pixel, the first average chromaticity of the non-high dynamic range image and the second average chromaticity of the high dynamic range image are calculated, that is, the first chromaticity and the second chromaticity corresponding to each pixel in the sliding window of the non-high dynamic range image and the high dynamic range image are respectively counted, and the first chromaticity and the second chromaticity corresponding to each pixel are respectively summed and averaged to obtain the first average chromaticity (u_lite, v_lite) and the second average chromaticity (u_hdr, v_hdr). The electronic device can also perform weighted average calculation or subtraction calculation on the first average chromaticity and the second average chromaticity in the sliding window to obtain the weighted average chromaticity (u_diff, v_diff). Exemplarily, taking subtraction calculation as an example, u_diff=u_lite-u_hdr, v_diff=v_lite-v_hdr.

[0171] Next, the electronic device can determine the color weight corresponding to the current pixel based on the brightness corresponding to the current pixel, where the color weight is used to achieve chromaticity fusion of the non-high dynamic range image and the high dynamic range image, for example, the second average chromaticity and the weighted average chromaticity are fused together in a certain proportion to obtain the target chromaticity of the current pixel.

[0172] Exemplarily, in the process of determining the color weight, different brightness weight relationships may be used for different shooting scenes, that is, the weights corresponding to different brightness values ​​are different. It is understandable that for HDR modes at different times, the brightness range corresponding to the horizontal axis in the brightness weight relationship may be determined according to the Y value of the non-high dynamic range image determined in the HDR mode, and the first brightness threshold and the second brightness threshold may be specifically determined. For example, in the HDR mode at the first moment, the first brightness threshold (such as the first value) and the second brightness threshold (such as the second value) corresponding to the horizontal axis in the brightness weight relationship may be determined according to the acquisition of the non-high dynamic range image at the first moment; in the HDR mode at the second moment, the first brightness threshold (such as the third value) and the second brightness threshold (such as the fourth value) corresponding to the horizontal axis in the brightness weight relationship may be determined according to the acquisition of the non-high dynamic range image at the second moment, the first value and the third value may be equal or unequal, and the second value and the fourth value may be equal or unequal, depending on the brightness of the image acquired at the current moment.

[0173] For example, the electronic device may be configured to: Figure 6 The brightness weight relationship shown in FIG. 1 is used to determine the magnitude of the first brightness. If the first brightness is greater than or equal to the second brightness threshold value ( ), then the chroma fusion is not performed on the current pixel; if the first brightness is less than or equal to the first brightness threshold ( ), the color weight is the second weight value, and the target chromaticity (u_new, v_new) = the second average chromaticity + weighted average chromaticity The second weight value, that is, u_new = u_hdr + u_diff ,v_new= v_hdr+ v_diff ; If the first brightness is less than the second brightness threshold and greater than the first brightness threshold, the color weight ( ) The specific value is determined by the linear interpolation formula, target chromaticity (u_new, v_new) = second average chromaticity + weighted average chromaticity Color weight value, that is, u_new = u_hdr + u_diff ,v_new= v_hdr+ v_diff , exemplarily, the linear interpolation formula for determining the first color weight value is: .

[0174] Finally, after the non-high dynamic range image and the high dynamic range image are processed pixel by pixel by sliding window, the chromaticity corresponding to each sliding window can be determined, wherein the chromaticity corresponding to each sliding window can be the target chromaticity after chromaticity fusion, or the current chromaticity without chromaticity fusion. At this point, the electronic device obtains a new HDR image after color cast correction based on the second brightness of the high dynamic range image, the target chromaticity corresponding to the second brightness, or the current chromaticity corresponding to the second brightness. Furthermore, the electronic device can convert the HDR image from the YUV color space to the RGB color space.

[0175] It should be noted that the first chromaticity mentioned in this application only refers to the chromaticity corresponding to the first image, and does not represent a specific value. Similarly, the second chromaticity only refers to the chromaticity corresponding to the second image, and does not represent a specific value.

[0176] It should be understood that each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0177] The present application also provides an electronic device, which may include: a memory and a processor, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0178] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0179] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0180] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0181] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0182] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0183] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0184] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which enables a computer to execute the method executed by the electronic device in any of the above embodiments when the computer program is executed.

[0185] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0186] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0187] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk), etc.

[0188] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0189] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. An image processing method, characterized in that: The method comprises: In response to a first operation of a user on a shooting interface, acquiring at least one first image and at least one second image, wherein the first operation is used to trigger the electronic device to shoot a high dynamic range (HDR) image, the at least one first image includes a normal exposure image, and the at least one second image includes a short exposure image; Performing fusion processing on the at least one first image and the at least one second image to obtain the HDR image; Performing denoising processing on the at least one first image to obtain a non-HDR image; Determining a first luminance and a first chrominance of the non-HDR image, and a second chrominance of the HDR image; Determining a first color weight according to a first brightness of the non-HDR image; determining a target chromaticity of the HDR image according to the first color weight, the first chromaticity, and the second chromaticity; The HDR image having the target chromaticity is saved.

2. The method according to claim 1, characterized in that: The determining a first color weight according to a first brightness of the non-HDR image comprises: Determining a first brightness weight relationship, wherein the first brightness weight relationship is used to represent a relationship between a brightness value within a first brightness range and a preset weight range, and the first brightness range is used to indicate the brightness of the non-HDR image; The first color weight is determined from the first brightness weight relationship according to a first brightness of the non-HDR image.

3. The method according to claim 2, characterized in that The first brightness range includes a first brightness threshold and a second brightness threshold, the second brightness threshold is greater than the first brightness threshold, the preset weight range includes a first weight value and a second weight value, the second weight value is greater than the first weight value; The determining the first color weight from the first brightness weight relationship according to the first brightness of the non-HDR image includes: When the first brightness of the non-HDR image is less than or equal to the first brightness threshold, the first color weight is the second weight value; When the first brightness of the non-HDR image is greater than or equal to the second brightness threshold, the first color weight is the first weight value; When the first brightness of the non-HDR image is less than the second brightness threshold and greater than the first brightness threshold, the first color weight is determined according to the first brightness, the second weight value, the first brightness threshold and the second brightness threshold.

4. The method according to any one of claims 1 to 3, characterized in that: The determining the target chromaticity of the HDR image according to the first color weight, the first chromaticity, and the second chromaticity comprises: Taking a sliding window as a unit, respectively, averaging the first chromaticity corresponding to each pixel in the non-HDR image and the second chromaticity corresponding to each pixel in the HDR image to obtain a first average chromaticity of the non-HDR image and a second average chromaticity of the HDR image; A target chromaticity of the HDR image is determined according to a first color weight of the sliding window in the non-HDR image, the first average chromaticity, and the second average chromaticity.

5. The method according to claim 4, characterized in that The method of performing averaging processing on the first chromaticity corresponding to each pixel in the non-HDR image and the second chromaticity corresponding to each pixel in the HDR image in units of sliding windows to obtain the first average chromaticity of the non-HDR image and the second average chromaticity of the HDR image includes: Performing edge detection on the non-HDR image to determine an edge area of ​​the non-HDR image; Determine whether the pixels in the sliding window of the non-HDR image and the pixels in the sliding window of the HDR image are located in the edge area. If it is determined that neither of them is located in the edge area, average the first chromaticity corresponding to the pixels in the sliding window of the non-HDR image and the second chromaticity corresponding to the pixels in the sliding window of the HDR image to obtain the first average chromaticity and the second average chromaticity.

6. The method according to claim 5, characterized in that The method further comprises: If it is determined that there are pixels within the sliding window of the non-HDR image and the pixels within the sliding window of the HDR image that are located in the edge area, the first chromaticities corresponding to the pixels within the sliding window of the non-HDR image and the second chromaticities corresponding to the pixels within the sliding window of the HDR image that are located in the edge area are averaged to obtain the first average chromaticity and the second average chromaticity.

7. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory; wherein the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions so that the electronic device executes the method as described in any one of claims 1 to 6.

8. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes one or more processors, and the processor is used to call computer instructions so that the electronic device executes the method as described in any one of claims 1 to 6.

9. A computer program product comprising instructions, characterized in that When the computer program product is run on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 6.

10. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 6.

Images