Image generation method, electronic device, and storage medium

Abstract

An image generation method, an electronic device and a storage medium, relating to the field of terminal technology, the method can make the color of the generated image close to the actual color of the photographed object or the shooting environment. Specifically, the electronic device responds to the shooting instruction, obtains the first YUV image and obtains the second YUV image. Among them, the first YUV image is a YUV image generated based on the ISP algorithm, and the second YUV image is a YUV image generated based on the AI ​​algorithm. The electronic device then adjusts the second chromaticity value corresponding to each pixel in the second YUV image according to the adjustment coefficient corresponding to the second YUV image to obtain a third YUV image. Among them, for any pixel point, the third chromaticity value corresponding to the pixel point in the third YUV image is closer to the first chromaticity value than the second chromaticity value corresponding to the pixel point, and the first chromaticity value is the chromaticity value corresponding to the pixel point in the first YUV image. The electronic device then generates an image corresponding to the shooting instruction based on the third YUV image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of terminal technology, and in particular to an image generating method, an electronic device, and a storage medium. Background Art

[0002] With the advancement of science and technology, the photography capabilities of electronic devices are constantly improving, resulting in an increase in users' demand for the shooting effects of electronic devices. In order to meet user needs, during the shooting process of electronic devices, electronic devices can enhance images through artificial intelligence (AI) algorithms to output higher quality images. Enhancement processing may include: noise reduction processing, dark light enhancement processing, defogging processing, rain removal processing, etc.

[0003] However, images enhanced by electronic devices through AI algorithms have color cast problems, that is, the color of the enhanced image deviates from the actual color of the photographed object or the shooting environment. Summary of the invention

[0004] The embodiments of the present application provide an image generation method, an electronic device, and a storage medium, which can make the color of the generated image close to the actual color of the photographed object or the shooting environment.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an image generation method is provided, which is applied to an electronic device. The electronic device responds to a shooting instruction to obtain a first YUV image and a second YUV image. The first YUV image is a YUV image generated based on an ISP algorithm, and the second YUV image is a YUV image generated based on an AI algorithm. The electronic device then adjusts the second chromaticity value corresponding to each pixel in the second YUV image according to the adjustment coefficient corresponding to the second YUV image to obtain a third YUV image. For any pixel, the third chromaticity value corresponding to the pixel in the third YUV image is closer to the first chromaticity value than the second chromaticity value corresponding to the pixel, and the first chromaticity value is the chromaticity value corresponding to the pixel in the first YUV image. The electronic device then generates an image corresponding to the shooting instruction based on the third YUV image.

[0007] In the above solution, the electronic device uses the adjustment coefficient corresponding to the second YUV image to adjust the chromaticity value of the second YUV image as a whole, rather than performing specific adjustment processing on a single pixel point at a pixel point granularity, so that the chromaticity of the second YUV image as a whole can be corrected, that is, the chromaticity of the second YUV image as a whole is closer to the overall chromaticity of the first YUV image. This not only achieves chromaticity correction of the second YUV image, but also avoids the introduction of noise.

[0008] In a possible implementation of the first aspect, the second YUV image includes a plurality of regions, each region having a corresponding adjustment coefficient. For each region, the electronic device adjusts the second chromaticity value corresponding to each pixel in the region according to the adjustment coefficient corresponding to the region to obtain a third YUV image.

[0009] In the above scheme, the electronic device makes corresponding adjustments to each area in the second YUV image through an adjustment coefficient, wherein the area includes multiple pixels. Therefore, when adjusting the chromaticity values ​​of multiple pixels in the same area, a unified adjustment coefficient is used instead of weightedly fusing the chromaticity values ​​of the pixels in the first YUV image and the chromaticity values ​​of the pixels in the second YUV image to obtain the chromaticity value of each pixel. This can avoid the problem of inaccurate target UV value after weighted fusion due to noise in the first YUV image, thereby avoiding or reducing the noise in the final output image.

[0010] In another possible implementation of the first aspect, the adjustment coefficient corresponding to each region includes a first adjustment coefficient and a second adjustment coefficient, and the second chromaticity value corresponding to each pixel includes a first value and a second value, the first value is the chromaticity value corresponding to the U channel, and the second value is the chromaticity value corresponding to the V channel. For each region, the electronic device adjusts the first value corresponding to each pixel in the region according to the first adjustment coefficient corresponding to the region. For each region, the electronic device adjusts the second value corresponding to each pixel in the region according to the second adjustment coefficient corresponding to the region.

[0011] In the above scheme, the image is divided into blocks to obtain different areas, and different adjustment coefficients are set for different areas for adjustment. Different adjustment coefficients are also set for the same area when adjusting the U value and V value. This can make the adjusted U image and V image closer to the reference U image and V image. The finer the granularity during the adjustment process, the better the effect of the adjusted U image and V image.

[0012] In another possible implementation of the first aspect, the first YUV image includes a first chromaticity diagram, and the second YUV image includes a second chromaticity diagram. For each region, the electronic device linearly transforms the second chromaticity value corresponding to each pixel in the region in the second chromaticity diagram according to the adjustment coefficient corresponding to the region to obtain a third chromaticity diagram. Among them, for any pixel, the third chromaticity value corresponding to the pixel in the third chromaticity diagram is closer to the first chromaticity value corresponding to the pixel in the first chromaticity diagram than the second chromaticity value corresponding to the pixel. The electronic device then generates a third YUV image based on the third chromaticity diagram and the first brightness diagram in the second YUV image.

[0013] In the above scheme, the electronic device performs a linear transformation on the second chromaticity diagram based on the second chromaticity diagram to obtain a corresponding third chromaticity diagram, so that the third chromaticity value corresponding to the third chromaticity diagram is closer to the first chromaticity value corresponding to the pixel in the first chromaticity diagram than the second chromaticity value corresponding to the pixel point, thereby making the color of the third chromaticity diagram close to the color of the first chromaticity diagram, that is, making the color of the third chromaticity diagram close to the actual captured color, thereby solving the color cast problem of the image.

[0014] In another possible implementation of the first aspect, for each region, the electronic device determines a first objective function corresponding to the region, wherein the first objective function is used to characterize a first chromaticity difference corresponding to the region, the first chromaticity difference refers to a difference between an adjusted chromaticity value corresponding to the region and a chromaticity value corresponding to the region in the first YUV image, and the adjusted chromaticity value corresponding to the region can be characterized by an adjustment coefficient to be solved and a chromaticity value corresponding to the region in the second YUV image. The electronic device then obtains the adjustment coefficient when the first objective function takes a minimum value.

[0015] In the above scheme, the electronic device determines the adjustment coefficient through the minimum value of the first objective function, so that the difference between the chromaticity value obtained by the adjustment coefficient and the chromaticity value of the first YUV image can be minimized, thereby making the adjusted color close to the color of the first YUV image, that is, making the adjusted color close to the actual captured color, solving the problem of image color cast.

[0016] In another possible implementation of the first aspect, the adjusted chromaticity value corresponding to the region includes the adjusted chromaticity value corresponding to each pixel in the region. The adjusted chromaticity value corresponding to the region is characterized by the adjustment coefficient to be solved and the second chromaticity value corresponding to each pixel in the region. The first chromaticity difference is determined based on the second chromaticity difference of each pixel in the region, and the second chromaticity difference of each pixel refers to the difference between the adjusted chromaticity value corresponding to the pixel and the first chromaticity value corresponding to the pixel.

[0017] In the above scheme, the first chromaticity difference is determined based on the second chromaticity difference of each pixel in the area. In this way, the chromaticity difference of each pixel in the area can be used as a reference for the adjustment coefficient, so that the final adjustment coefficient is more accurate, and the image color after the adjustment coefficient is adjusted is closer to the color of the first YUV image.

[0018] In another possible implementation of the first aspect, the electronic device determines a second objective function, wherein the second objective function is used to characterize a third chromaticity difference, the third chromaticity difference refers to the sum of the fourth chromaticity differences of each pixel in the first YUV image, and the fourth chromaticity difference of each pixel refers to the difference between the adjusted chromaticity value corresponding to the pixel and the first chromaticity value corresponding to the pixel. The adjusted chromaticity value corresponding to the pixel is characterized by the adjustment coefficient to be solved and the second chromaticity value corresponding to the pixel in the second YUV image. The electronic device then obtains the adjustment coefficient when the second objective function takes the minimum value.

[0019] In the above solution, the electronic device can use the same adjustment coefficient to adjust the chromaticity value of the pixel points in the entire second YUV image, wherein the adjustment coefficient is the value when the second objective function takes the minimum value. In this way, the difference between the chromaticity value obtained by the adjustment coefficient and the chromaticity value of the first YUV image is minimized, and the adjusted color is close to the color of the first YUV image, that is, the adjusted color is close to the actual captured color, solving the color cast problem of the picture.

[0020] In another possible implementation of the first aspect, the second YUV image is generated by the first original image, and the first original image is one of the multiple original images collected in response to the shooting instruction. The first YUV image is: the electronic device obtains multiple second original images, wherein the multiple second original images are included in the multiple original images. The electronic device then determines the fourth YUV images corresponding to the multiple second original images respectively. The electronic device then fuses the multiple fourth YUV images to obtain the first YUV image.

[0021] In the above solution, the electronic device fuses the multiple fourth YUV images to obtain the first YUV image, which can make the obtained first YUV more accurate and make the color in the first YUV image closer to the actual color when the first original image was taken, so as to facilitate the subsequent more accurate color correction of the YUV image.

[0022] In another possible implementation of the first aspect, the electronic device determines weights corresponding to the plurality of fourth YUV images, respectively. The time difference corresponding to each fourth YUV image is negatively correlated with the weight, and the time difference refers to the difference between the first time and the second time, the first time is the generation time of the second original image corresponding to the fourth YUV image, and the second time is the generation time of the first original image. The electronic device then performs weighted fusion on each fourth YUV image according to the weights corresponding to each fourth YUV image, to obtain the first YUV image.

[0023] In the above scheme, when the generation time of the second original image corresponding to the fourth YUV image is closer to the generation time of the first original image, the weight of the image is higher, and the electronic device performs weighted fusion based on the image weight and the image. In this way, the obtained first YUV can be made more accurate, and the color in the first YUV image can be made closer to the actual color when the first original image was taken, so as to facilitate more accurate color correction of the YUV image later.

[0024] In another possible implementation of the first aspect, the multiple original images are original images continuously captured in response to a shooting instruction in a continuous shooting mode. Alternatively, the multiple original images are original images captured in a single shooting mode for image preview, and the capture time of the multiple original images and the time of receiving the shooting instruction meet a preset proximity condition.

[0025] In the above solution, the electronic device collects multiple original images, which can also facilitate more accurate acquisition of the first YUV image based on the multiple original images.

[0026] In a second aspect, an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor; the memory is coupled to the processor. The memory stores a computer program code, the computer program code comprises a computer instruction, and when the computer instruction is executed by the processor, the electronic device executes the method in the first aspect and any possible implementation thereof.

[0027] In a third aspect, an embodiment of the present application provides a computer storage medium, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect and any possible implementation thereof.

[0028] In a fourth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of the first aspect and any possible implementation thereof. The computer may be an electronic device in the second aspect and any possible implementation thereof.

[0029] It can be understood that the beneficial effects that can be achieved by the electronic device of the second aspect, the computer storage medium of the third aspect, and the computer program product of the fourth aspect provided above can be referred to as the beneficial effects in the first aspect and any possible implementation method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of image comparison provided in an embodiment of the present application;

[0031] Figure 2A schematic diagram of the principle of an image generation method provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of another image generation method provided in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of an image generation effect provided in an embodiment of the present application;

[0034] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0035] Figure 6 A schematic diagram of the system structure of an electronic device provided in an embodiment of the present application;

[0036] Figure 7 A schematic diagram of the system structure of another electronic device provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of a principle for generating a first U image and a first V image provided in an embodiment of the present application;

[0038] Fig. 9 A schematic diagram of a principle for confirming the weight corresponding to an image provided in an embodiment of the present application;

[0039] Fig.10 A schematic diagram of another image generation method provided in an embodiment of the present application;

[0040] Fig.11 A schematic diagram of another image generation method provided in an embodiment of the present application;

[0041] Fig.12 A schematic diagram of another image generation method provided in an embodiment of the present application;

[0042] Fig.13 A schematic diagram of the principle of another image generation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0044] Before introducing the embodiments of the present application, the following explanation is given to the relevant professional terms involved in the embodiments of the present application.

[0045] 1. YUV

[0046] YUV is a color encoding method that uses brightness and chromaticity to specify the color of a pixel. Y represents the brightness value, while U and V represent the chromaticity. Among them, U represents the blue chromaticity information, and V represents the red chromaticity information. The image generated by following the YUV color encoding method is a YUV image.

[0047] A YUV image can include images corresponding to the Y channel, U channel, and V channel. The image corresponding to the Y channel can be called a Y image, which is used to represent the brightness value (i.e., Y value) of the Y channel corresponding to each pixel. The image corresponding to the U channel can be called a U image, which is used to represent the chromaticity value (i.e., U value) of the U channel corresponding to each pixel. The image corresponding to the V channel can be called a V image, which is used to represent the chromaticity value (i.e., V value) of the V channel corresponding to each pixel.

[0048] 2. RAW images

[0049] RAW images, also known as original images, refer to the original data that the image sensor converts the captured light source signal into a digital signal without any modification or compression processed inside the camera. RAW images contain the original color information, brightness value information or other sensor information captured from the image sensor.

[0050] 3. RGB

[0051] RGB is a color model that obtains various colors by changing the three color channels of red, green, and blue and superimposing them on each other. An image generated according to the RGB color model is an RGB image, which is a color image.

[0052] 4. Image Noise

[0053] There is often unnecessary or redundant interference information in the image data, which can be called image noise. For example, image noise can be manifested as random changes in image brightness or color, which are not inherent in the photographed object itself but are generated during the photographing or data transmission process.

[0054] 5. Color Noise

[0055] Colored spots or patches of random color that appear in images captured by a camera. Color noise is usually reddish or blue in color. Color noise is more noticeable in images captured in low light.

[0056] 6. Color Diffusion

[0057] Color diffusion refers to the gradual spread of color from one area to an adjacent area, resulting in a change in color distribution.

[0058] 7. Image Signal Processor

[0059] The Image Signal Processor (ISP) is a hardware component or software module that receives the raw image data transmitted by the photosensitive element and converts the raw image data into image information suitable for display, storage or further analysis through a series of algorithms and processing flows.

[0060] 8. Deep Learning Algorithms

[0061] Deep learning algorithm refers to a machine learning algorithm based on deep neural network models and methods, and is a type of artificial intelligence (AI) algorithm.

[0062] 9. Professional Mode

[0063] Professional mode refers to a shooting mode in which an electronic device shoots under set professional shooting parameters. That is, the professional mode supports users to customize and adjust shooting parameters.

[0064] When the electronic device is in professional mode, the image processing process of the captured image is completely implemented by the native ISP of the chip. That is, in professional mode, the electronic device uses the native ISP algorithm to process the original image (or RAW image) captured, and no longer uses the AI ​​algorithm for image enhancement. The mode of image processing completely relying on the native ISP algorithm is not limited to the professional mode, but can also be other modes, and there is no limitation on this.

[0065] It should be understood that the colors of images obtained by the native ISP algorithm are close to the colors of the actual objects and scenes. Since the images captured in the professional mode are obtained using the ISP algorithm and no AI algorithm is used for image enhancement, the images captured in the professional mode do not have color cast problems, but may have other problems such as large image noise and darker dark areas.

[0066] Moreover, in this case, it is difficult for electronic devices to perform noise reduction on images generated by the ISP algorithm. Specifically, it is difficult for electronic devices to control the intensity of noise reduction when processing image noise. If the noise reduction is insufficient, the image obtained through the final output still has color noise. If the noise reduction is too strong, the image is over-smoothed, resulting in unnecessary propagation and change of colors between pixels in the image, i.e., color diffusion.

[0067] The embodiments of the present application are described in detail below.

[0068] When an electronic device is shooting, the shooting scene may be complex. For example, the shooting scene may be rainy or foggy, or the light in the shooting scene is dim. The image quality obtained in these complex shooting scenes may be poor.

[0069] In order to obtain higher quality images, during the shooting process, electronic devices do not directly generate the captured images based on the collected original images (RAW images), but use AI algorithms to enhance the original images to obtain the captured images. However, the images obtained by electronic devices through AI algorithms have color cast problems, that is, there is a deviation between the colors in the image and the colors in the actual shooting scene.

[0070] It should be understood that the use of an AI algorithm to enhance the original image may also be referred to as AI enhancement processing. The AI ​​enhancement processing that causes the color cast problem may include at least one of noise reduction processing, dark light enhancement processing, rain removal processing, or defogging processing using an AI algorithm. Exemplarily, rain removal processing may be the use of an AI algorithm to remove rain from an image taken in a rainy day scene, and defogging processing may be the use of an AI algorithm to remove fog from an image taken in a foggy day scene. It should be understood that the AI ​​enhancement processing that causes the color cast problem may also include other processing, which is not limited to this.

[0071] To facilitate understanding of the color cast problem, the following article uses the AI ​​algorithm to perform dark light enhancement processing as an example for illustration.

[0072] When an electronic device is shooting in a low-light environment (such as at night), the dark areas of the generated image may have lost details or be blurred, that is, the dark areas of the image may lack details. In order to solve the problem of insufficient details in dark areas of images shot by electronic devices in low-light environments, the electronic device may process the original image captured by an AI algorithm (such as a deep learning algorithm) to make the details of the dark areas in the generated image clear, thereby better displaying the dark details in the image. However, the images generated by the AI ​​algorithm have a color cast problem.

[0073] The images output by electronic devices after using AI algorithms are as follows: Figure 1 As shown in (a) in the figure, the image output by the electronic device using the ISP algorithm is as follows Figure 1 As shown in (b) in . Figure 1 (a) and Figure 1 (b) in the figure is an image taken with the same or similar shooting angle for the same subject. Figure 1 (a) and Figure 1 As shown in (b), the colors are different. Figure 1 There is color deviation in (a). Figure 1 (b) in the figure is close to the actual color of the subject.

[0074] In some schemes, such as Figure 2 As shown, in order to solve the color cast problem of the image processed by the AI ​​algorithm, the electronic device can input the RAW image into the ISP algorithm and the AI ​​algorithm, obtain the first YUV image (i.e., the first YUV image corresponding to the ISP algorithm) through the ISP algorithm, and obtain the second YUV image (i.e., the second YUV image corresponding to the AI ​​algorithm) through the AI ​​algorithm, and then perform UV fusion on the first YUV image and the second YUV image, thereby achieving chromaticity adjustment of the final output image. For example, the electronic device can perform UV fusion on the first YUV image and the second YUV image in a linear weighted manner.

[0075] In the following, we will introduce the specific processing of UV fusion in some schemes.

[0076] In some schemes, the electronic device performs UV fusion on the first YUV image and the second YUV image based on pixel granularity.

[0077] Specifically, the electronic device determines the first UV weight corresponding to the pixel in the first YUV image and the second UV weight corresponding to the pixel in the second YUV image based on the brightness value corresponding to each pixel in the second YUV image. The UV value corresponding to the pixel in the first YUV image is recorded as UV1, and the UV value corresponding to the pixel in the second YUV image is recorded as UV2. For each pixel in the second YUV image, the electronic device linearly weights the UV1 and UV2 corresponding to the pixel based on the first UV weight and the second UV weight corresponding to the pixel, and uses the linearly weighted UV value as the corrected UV value finally corresponding to the pixel.

[0078] The sum of the first UV weight and the second UV weight is 1. Exemplarily, the function of the weighted formula may be: target UV = UV1*α+UV2*(1-α), where UV1 represents the UV value corresponding to the pixel point in the first YUV image, UV2 represents the UV value corresponding to the pixel point in the second YUV image, and α [0, 1], α is the first UV weight, 1-α is the second UV weight.

[0079] Specifically, Figure 3 As shown, for each pixel in the second YUV image, the first UV weight and the second UV weight are determined according to the brightness value corresponding to the pixel in the second YUV image, which can be divided into the following three cases for discussion.

[0080] The first case: when the brightness value corresponding to the pixel in the second YUV image is lower than the first threshold, the first UV weight corresponding to the pixel is 1, and the second UV weight corresponding to the pixel is 0. Therefore, UV1*1+UV2*0 can get the target UV value of the pixel. This is equivalent to that for each pixel in the second YUV image whose brightness value is lower than the first threshold, the electronic device can use the UV value corresponding to the pixel in the first YUV image (i.e., UV1) as the target UV value of the pixel (i.e., the corrected UV value).

[0081] The second case: when the brightness value corresponding to the pixel in the second YUV image is higher than the first threshold but lower than the second threshold, the first UV weight α and the second UV weight 1-α corresponding to the pixel are not 0, and the first UV weight α and the second UV weight 1-α are not 1, and the target UV value corresponding to the pixel is obtained by UV1*α+UV2*(1-α). This is equivalent to that for each pixel in the second YUV image whose brightness value is higher than the first threshold but lower than the second threshold, the electronic device can linearly weight the UV1 and UV2 corresponding to the pixel, and use the linearly weighted UV value as the target UV value corresponding to the pixel.

[0082] For example, assuming that the first UV weight is 0.3, the second UV weight is 0.7, and the target UV value corresponding to the pixel point is UV1*0.3+UV2*0.7.

[0083] In the third case, when the brightness value of the pixel in the second YUV image is higher than the second threshold, the first UV weight corresponding to the pixel is 0, and the second UV weight corresponding to the pixel is 1. Therefore, UV1*0+UV2*1 can get the target UV value of the pixel. This is equivalent to that, for each pixel in the second YUV image whose brightness value is higher than the second threshold, the electronic device can use the UV value corresponding to the pixel in the second YUV image (i.e., UV2) as the target UV value of the pixel (i.e., the corrected UV value).

[0084] It is understandable that in this solution, the electronic device performs UV fusion on the first YUV image and the second YUV image by a weighted method, which has limited improvement on the color cast problem and also introduces noise problems when improving the color cast problem. The specific analysis is as follows:

[0085] In the first case above, the electronic device uses the UV value corresponding to the pixel point in the first YUV image as the target UV value, but the YUV image obtained by the ISP algorithm has a problem of excessive image noise. Therefore, when the brightness value of the second YUV image is lower than the first threshold, the image after color correction has a noise problem.

[0086] When the brightness value of the second YUV image is greater than the first threshold value but lower than the second threshold value (i.e., the second case), the target UV value of each pixel point finally output by the electronic device is obtained by integrating the UV value of the first YUV image. However, due to the noise problem of the first YUV image, that is, the pixel color in the first YUV image will randomly change, resulting in inaccurate UV values ​​of some pixels in the first YUV image, and then the target UV values ​​of some pixels will also be inaccurate, and the final output image will also have noise problems. In addition, the UV value of each pixel point finally output by the electronic device is obtained based on the UV value on the second UV image, and the second YUV image has a color cast problem, that is, the UV value of the pixel point in the second YUV image is inaccurate, and then the target UV value obtained according to the UV value in the second YUV image is also inaccurate, and then the color corresponding to the target UV value obtained by the electronic device is still different from the color in the actual shooting environment, that is, the degree of improvement of the color cast problem is limited.

[0087] When the brightness value of the second YUV image is higher than the second threshold value (ie, the third case), the target UV value obtained by the electronic device is the UV value on the second YUV image, but the YUV image obtained by the AI ​​algorithm has a color cast problem, so the color cast problem is still not solved.

[0088] It can be seen that in the above solution, the color cast problem cannot be effectively solved and image noise problem will be introduced.

[0089] Therefore, in order to solve the above problems, an embodiment of the present application provides an image generation method, which is executed by an electronic device. Specifically, the electronic device uses the first U image corresponding to the U channel of the first YUV image and the first V image corresponding to the V channel as references, and adjusts the color of the second U image corresponding to the U channel of the second YUV image and the color of the second V image corresponding to the V channel based on the color (or chromaticity) of the first U image and the first V image, respectively, so that the colors of the adjusted third U image and third V image are close to the colors of the actual shooting object and shooting environment.

[0090] In some embodiments, the electronic device responds to the shooting instruction to obtain the first YUV image and the second YUV image. The electronic device adjusts the second chromaticity value corresponding to each pixel in the second YUV image according to the adjustment coefficient corresponding to the second YUV image to obtain a third YUV image. Among them, for any pixel, the third chromaticity value corresponding to the pixel in the third YUV image is closer to the first chromaticity value than the second chromaticity value corresponding to the pixel, and the first chromaticity value is the chromaticity value corresponding to the pixel in the first YUV image. The electronic device then generates an image corresponding to the shooting instruction based on the third YUV image.

[0091] The shooting instruction may be a shooting instruction of the electronic device in a photo shooting scene, or may be a shooting instruction of the electronic device in a video recording scene. If it is a shooting instruction in a photo shooting scene, the shooting instruction may be an instruction in a single photo shooting scene (or single shooting mode), or the shooting instruction may be an instruction in a multiple photo shooting scene (or continuous shooting mode).

[0092] In the above embodiment, the adjustment coefficient corresponding to the second YUV image is used to adjust the chromaticity value of the second YUV image as a whole, rather than performing specific adjustment processing on a single pixel point at a pixel point granularity, so that the chromaticity of the second YUV image as a whole can be corrected, that is, the chromaticity of the second YUV image as a whole is closer to the overall chromaticity of the first YUV image. This not only achieves chromaticity correction of the second YUV image, but also avoids the introduction of noise.

[0093] In this embodiment, the chromaticity value refers to at least one of the U value corresponding to the pixel in the U image or the V value corresponding to the pixel in the V image. Exemplarily, the electronic device adjusts the U value of each pixel in the second U image and the V value of each pixel in the second V image by adjusting the coefficient, so that the U value corresponding to the third U image obtained after adjustment is close to the U value corresponding to the first U image, and the V value corresponding to the third V image is close to the V value corresponding to the first V image, thereby achieving the color of the adjusted third U image and the third V image close to the color of the actual shooting object and shooting environment.

[0094] In some embodiments, the electronic device may use the same adjustment coefficient to adjust the entire second YUV image, and the electronic device may also divide the second YUV image into different regions, each region having a corresponding adjustment coefficient. The adjustment coefficient may be one or more. For example, each region may correspond to one or more adjustment coefficients.

[0095] The following is a specific description by taking an example where an electronic device divides the second YUV image to obtain different regions, sets a different adjustment coefficient for each region, and adjusts each region by using the adjustment coefficient.

[0096] Specifically, the electronic device can adjust the chromaticity value of each region according to the adjustment coefficient corresponding to each region in the second YUV image, obtain the adjusted chromaticity value of each region, and finally obtain the third YUV image. For each region in the second YUV image, the adjustment coefficient corresponding to the region is determined according to the chromaticity difference between the region and the corresponding region, wherein the corresponding region refers to the region corresponding to the region in the first YUV image. It should be understood that the two regions having a corresponding relationship in the first YUV image and the second YUV image have the same position and size in the image.

[0097] It should be understood that in the above-mentioned scheme for UV fusion, if there is noise in the first YUV image, then the UV values ​​of some pixels in the first YUV image are not accurate enough. Then, taking the pixel as the granularity, the UV value corresponding to the pixel in the first YUV image and the UV value corresponding to the second YUV image are weightedly fused (i.e., the above-mentioned UV fusion), which will cause noise in some pixels after UV fusion. In an embodiment of the present application, the electronic device makes corresponding adjustments to each area in the second YUV image through an adjustment coefficient, wherein the area includes multiple pixels. Therefore, when adjusting the chromaticity value of multiple pixels in the same area, a unified adjustment coefficient is used instead of weighted fusion of the chromaticity value of the pixel in the first YUV image and the chromaticity value of the pixel in the second YUV image to obtain the chromaticity value of each pixel. This can avoid the problem that the target UV value after weighted fusion is also inaccurate due to the presence of noise in the first YUV image, thereby avoiding or reducing the noise of the final output image.

[0098] Moreover, in this embodiment, the chromaticity value of the second YUV image is adjusted by adjusting the coefficient so that the chromaticity value of the third YUV image obtained after the adjustment is close to the chromaticity value of the first YUV image, that is, the color of the adjusted third YUV image is close to the color of the actual shooting object and shooting environment.

[0099] It can be understood that in this embodiment, the electronic device combines the advantages of the first YUV image and the second YUV image, uses the first YUV image obtained by the ISP algorithm as a reference image, and performs color correction on the second YUV image of the AI ​​algorithm, that is, the electronic device uses the adjustment coefficient to adjust the chromaticity value of each pixel in the second YUV, so that the adjusted chromaticity value is close to the chromaticity value of the first YUV image, thereby making the color of the corrected U image and V image close to the actual color of the photographed object, and because the pixels in each area are adjusted as a whole, there will be no noise problem. This allows the final image to retain the low noise characteristics of the image corresponding to the AI ​​algorithm and solve the color cast problem of the image corresponding to the AI ​​algorithm. The electronic device then combines the Y image obtained by the AI ​​algorithm and the third YUV image after color correction to obtain the final YUV image, and converts the YUV image into an RGB image.

[0100] The following takes the scenario of performing image noise reduction processing during photo shooting (referred to as "noise reduction scenario") as an example to specifically explain the image processing method.

[0101] In the noise reduction scenario, electronic devices can use AI algorithms (such as deep learning algorithms) to reduce noise on images, but the images obtained by the AI ​​algorithm are prone to color cast problems, but the image noise is small. The image processed by the ISP algorithm has large noise, but the image color is close to the actual shooting color.

[0102] Specifically, the image output based on the ISP algorithm (that is, the image processed by the ISP algorithm) can be found in Figure 4 As shown in 401 in FIG. 4 , the UV image (or referred to as the first chromaticity diagram) corresponding to the image 401 is as follows: Figure 4 The color in the UV image 402 is similar to the actual color of the photographed object, but the noise is large, that is, there are more color noise points and smaller color blocks in the UV image 402. It should be understood that the UV image refers to an image obtained by combining the U image of the U channel and the V image of the V channel.

[0103] For images based on AI algorithm output (i.e. images processed by AI algorithm), please refer to Figure 4 The UV image (or second chromaticity diagram) corresponding to the image 403 is shown as Figure 4 The color in the UV image 404 deviates from the actual color of the photographed object, but the noise is small, that is, the color block in the UV image 404 is large, and the randomly appearing color noise is small.

[0104] The electronic device may adjust the image 404 according to the adjustment coefficient to obtain an adjusted image. Figure 4 The image 405 in is the adjusted UV image (or the third chromaticity image). Further, the electronic device can combine the adjusted UV image with the Y image (i.e., the brightness image) of the Y channel output by the AI ​​algorithm to obtain the target YUV image. The electronic device then converts the target YUV image into an RGB image 406.

[0105] In this embodiment, the electronic device can use the image generation method of the embodiment of the present application, combined with the characteristic that the image corresponding to the AI ​​algorithm has low noise, and the characteristic that the image corresponding to the ISP algorithm is close to the actual color of the photographed object, to obtain an image with low image noise and color close to the actual color of the photographed object.

[0106] It should be understood that the color cast problem caused by image enhancement through AI algorithms can be solved by the image generation method. For example, in image enhancement scenarios such as noise reduction of multiple images, dark light enhancement of images, defogging of images, and deraining of images, the color cast problem can be solved by the image generation method of the embodiment of the present application.

[0107] In the above embodiments, the application scenarios of the image generation method in the embodiments of the present application are only illustrated by taking the noise reduction scene as an example, and it should not be limited. For example, in the rain removal scene, the electronic device can obtain the image after rain removal through the AI ​​algorithm, but the image after rain removal has a color cast problem. In this case, the electronic device can use the first U image and the first V image obtained by the ISP algorithm as reference images, and use the image processing method in the embodiments of the present application to perform color correction on the third U image and the third V image after rain removal. In this way, the electronic device can finally output an image without rain and the color is close to the actual captured color.

[0108] Exemplarily, the electronic device may be a mobile phone, a tablet computer, a smart remote controller, a wearable device (such as a smart bracelet, a smart watch or smart glasses), a PDA, an augmented reality (AR) / virtual reality (VR) device. Alternatively, the mobile phone 500 may also be other types of electronic devices such as a portable multimedia player (PMP), a media player, etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0109] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0110] The mobile phone 500 may include a processor 510, an external memory interface 520, an internal memory 521, a universal serial bus (USB) interface 530, a charging management module 540, a power management module 541, a battery 542, an antenna 1, an antenna 2, a mobile communication module 550, a wireless communication module 560, an audio module 570, a speaker 570A, a receiver 570B, a microphone 570C, an earphone interface 570D, a sensor module 580, a button 590, a motor 591, an indicator 592, cameras 1~N593, a display screen 1~N594, and a subscriber identification module (SIM) card interface 1~N595, etc.

[0111] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 500. In other embodiments of the present application, the mobile phone 500 may include more or fewer components than those shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0112] The processor 510 may include one or more processing units, for example, the processor 510 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0113] The controller may be the nerve center and command center of the mobile phone 500. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0114] In some embodiments, the image generation method of the embodiment of the present application can be executed in the processor 510 and corresponding instructions can be generated.

[0115] The processor 510 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 510 is a cache memory. The memory may store instructions or data that the processor 510 has just used or cyclically used. If the processor 510 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 510, and thus improves the efficiency of the system.

[0116] In some embodiments, the processor 510 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0117] The mobile phone 500 implements the display function through a GPU, a display screen 594, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 594 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 510 may include one or more GPUs, which execute program instructions to generate or change display information.

[0118] The display screen 594 is used to display images, videos, etc. The display screen 594 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone 500 may include 1 or N display screens 594, where N is a positive integer greater than 1.

[0119] In some embodiments of the present application, the display screen 594 can display an image generated using the image processing method in the embodiments of the present application, which is an image with better quality after the color cast problem is corrected.

[0120] The mobile phone 500 can realize the shooting and taking pictures functions through the ISP, the camera 593, the video codec, the GPU, the display screen 594 and the application processor.

[0121] ISP is used to process the data fed back by camera 593. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 593. Exemplarily, ISP can process the RAW image captured by camera 593 and generate the corresponding YUV image.

[0122] The camera 593 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the mobile phone 500 may include 1 or N cameras 593, where N is a positive integer greater than 1. In some embodiments, the camera 593 can capture RAW images. In the embodiment of the present application, the ISP algorithm and the AI ​​algorithm can perform image processing based on the RAW image captured by the camera 593 to generate corresponding YUV images, for example, a first YUV image and a second YUV image.

[0123] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the mobile phone 500 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0124] Video codecs are used to compress or decompress digital videos. Mobile phone 500 may support one or more video codecs. Thus, mobile phone 500 may play or record videos in various coding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0125] NPU is a neural network (NN) computing processor. It can quickly process input information and continuously self-learn by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain. NPU can realize applications such as intelligent cognition of mobile phones 500, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0126] In some embodiments, the sensor module 580 may include a touch sensor ( Figure 5 The touch sensor may be disposed on the display screen 594, and the touch sensor and the display screen 594 may form a touch screen, also known as a "touch screen". The touch sensor is used to detect touch operations acting on or near it. The touch sensor may transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation may be provided through the display screen 594. In other embodiments, the touch sensor may also be disposed on the surface of the mobile phone 500, which is different from the position of the display screen 594.

[0127] In some embodiments, the electronic device receives a user's touch operation through the display screen 594, and the touch operation is used to instruct the electronic device to perform a shooting operation. In response to the user's touch operation, the electronic device obtains the first YUV image obtained by the ISP algorithm and the second YUV image obtained by the AI ​​algorithm. The electronic device then uses the first U image and the first V image corresponding to the first YUV image as reference images, and performs color correction on the second U image and the second V image corresponding to the second YUV image, respectively, to obtain a third U image and a third V image after color correction. The electronic device then combines the Y image in the second YUV image with the third U image and the third V image after color correction to obtain the final output YUV image.

[0128] Figure 6 A schematic diagram of the system structure of an electronic device provided in an embodiment of the present application.

[0129] The software system of the mobile phone 500 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to exemplify the software structure of the mobile phone 500.

[0130] The layered architecture divides the system into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the system is divided into five layers, from top to bottom: application layer, application framework layer, hardware abstraction layer, driver layer, and hardware layer.

[0131] The application layer may include a series of application packages. In the embodiment of the present application, the application package may include a camera, a gallery, etc.

[0132] The application framework layer provides an application programming interface (API) and a programming framework for the application of the application layer. The application framework layer includes some predefined functions. In an embodiment of the present application, the application framework layer may include a camera access interface, wherein the camera access interface may include camera management and camera devices. The camera access interface is used to provide an application programming interface and a programming framework for camera applications.

[0133] The hardware abstraction layer is an interface layer between the application framework layer and the driver layer, and provides a virtual hardware platform for the operating system. In the embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer and a camera algorithm library.

[0134] Among them, the camera hardware abstraction layer can provide virtual hardware of camera device 1, camera device 2 or more camera devices. The camera algorithm library may include running code and data for implementing the image generation method provided in the embodiment of the present application. Among them, the camera algorithm library may include an ISP algorithm module, an AI algorithm module, and a color cast correction module. Exemplarily, the color cast correction module is used to execute the image generation method provided in the embodiment of the present application, and the chromaticity values ​​in the YUV image generated by the ISP algorithm module (such as the chromaticity values ​​of the U channel and the chromaticity values ​​of the V channel) are used to perform color cast correction on the chromaticity values ​​in the YUV image generated by the AI ​​algorithm module.

[0135] The driver layer is a layer between hardware and software. The driver layer includes drivers for various hardware. The driver layer may include camera device drivers, digital signal processor drivers, and graphics processor drivers, etc.

[0136] Among them, the camera device driver is used to drive the image sensor of the camera to collect images. The digital signal processor driver is used to drive the digital signal processor to process images. The graphics processor driver is used to drive the graphics processor to process images. It should be understood that the digital signal processor and the graphics processor are used to provide hardware support for the operation of the algorithms in the camera algorithm library.

[0137] like Figure 7 As shown, the image generation method in the embodiment of the present application is specifically described below in combination with the software system structure:

[0138] (1) The electronic device drives the camera to collect raw images.

[0139] Exemplarily, in response to a user's operation of opening a camera application, for example, clicking on a camera application icon, the electronic device starts the camera application by calling a camera access interface of the application framework layer, and enters the camera interface to display the current shooting picture on the screen, and then sends an instruction to start the camera by calling the virtual hardware of the camera device (camera device 1 and / or other camera devices) in the camera hardware abstraction layer. The camera hardware abstraction layer sends the instruction to the camera device driver of the kernel layer. The camera device driver can start the corresponding camera and collect image light signals through the camera. The camera can transmit the collected image light signals to the image signal processor for preprocessing to obtain the original image (RAW image).

[0140] (2) The camera inputs the raw image into the ISP algorithm module in the camera algorithm library.

[0141] (3) The electronic device inputs the original image into the AI ​​algorithm module in the camera algorithm library.

[0142] The camera can transmit the above raw image to the camera algorithm library in the camera hardware abstraction layer through the camera device driver. The camera algorithm library stores the program code for implementing the image generation method provided in the embodiment of the present application. The raw image can be input into the ISP algorithm module and the AI ​​algorithm module in the camera algorithm library respectively.

[0143] It should be understood that the above steps (2) and (3) are only used to illustrate that the original image captured by the camera will be input into the ISP algorithm module and the AI ​​algorithm module respectively, and do not limit the order of input. In addition, in other examples, the camera can also directly send the original image to the camera algorithm library, and the camera algorithm library will input the original image into the AI ​​algorithm module and the ISP algorithm module for processing.

[0144] (4) The ISP algorithm module generates the first YUV image and inputs it into the color cast correction module in the camera algorithm library.

[0145] (5) The AI ​​algorithm module generates a second YUV image and inputs it into the color cast correction module in the camera algorithm library.

[0146] Exemplarily, each algorithm module in the camera algorithm library executes the above-mentioned image generation method based on the hardware support provided by the digital signal processor and the graphics processor. The ISP algorithm module can process the original image to obtain a first RGB image, and convert the first RGB image into a first YUV image. The AI ​​algorithm module can process the original image to obtain a second RGB image, and convert the second RGB image into a second YUV image.

[0147] Similarly, steps (4) and (5) are also not limited in the order of execution.

[0148] (6) The color cast correction module returns the color-corrected RGB image to the image library.

[0149] Exemplarily, the color cast correction module can use the first U image and the first V image corresponding to the first YUV image as reference images, and perform color correction on the second U image and the second V image corresponding to the second YUV image, respectively, to obtain a third U image and a third V image after color correction. The color cast correction module can then combine the Y image in the second YUV image with the third U image and the third V image after color correction to obtain a final output YUV image. The color cast correction module can convert the final YUV image into an RGB image and return it to the image library.

[0150] The image processing method in the embodiment of the present application is described in detail below through relevant drawings.

[0151] As described above, the electronic device (such as the color cast correction module in the electronic device) performs color correction on the second U image and the second V image based on the first U image and the first V image, respectively. Therefore, before the electronic device performs color correction, it is necessary to first obtain the first U image, the first V image, the second U image, and the second V image, that is, it is necessary to first obtain the first YUV image and the second YUV image. In the following, two methods for obtaining the first YUV image and the second YUV image will be introduced. It should be understood that there may be other methods for obtaining the first YUV image and the second YUV image. Here, only the following two methods are used as examples, and all the methods are not listed one by one.

[0152] Method 1: Obtain the first YUV image and the second YUV image through a single original image.

[0153] In the case of inputting a single original image, the electronic device inputs the single original image into the ISP algorithm to obtain an RGB image output by the ISP, and the electronic device inputs the single original image into the AI ​​algorithm to obtain an RGB image output by the AI ​​algorithm. The electronic device then converts the RGB image output by the ISP into a first YUV image, and converts the RGB image output by the AI ​​into a second YUV image.

[0154] Method 2: Obtain the first YUV image and the second YUV image through multiple original images.

[0155] In some embodiments, after receiving the shooting instruction, the electronic device can collect multiple original images, and input one of the multiple original images into the AI ​​algorithm to obtain a second YUV image.

[0156] In some embodiments, when the electronic device collects multiple original images, the electronic device inputs multiple second original images among the multiple original images into the ISP algorithm to obtain multiple fourth YUV images accordingly. The electronic device then fuses the multiple fourth YUV images to obtain the first YUV image.

[0157] The multiple original images may be original images continuously captured by the electronic device in a continuous shooting mode in response to a shooting instruction. They may also be original images captured by the electronic device in a single shooting mode for image preview, and the capture time of the multiple original images and the time of receiving the shooting instruction meet a preset proximity condition. Exemplarily, the multiple original images may include multiple preview images captured in a preview process before receiving the shooting instruction and images captured when receiving the photo taking instruction.

[0158] In some embodiments, the electronic device merges the plurality of fourth YUV images into one first YUV image by weighted averaging.

[0159] Specifically, the electronic device determines the weights corresponding to the plurality of fourth YUV images. The time difference corresponding to each fourth YUV image is negatively correlated with the weight, and the time difference refers to the difference between the first time and the second time, the first time is the generation time of the second original image corresponding to the fourth YUV image, and the second time is the generation time of the first original image. The electronic device then performs weighted fusion on each fourth YUV image according to the weights corresponding to each fourth YUV image to obtain the first YUV image.

[0160] Specifically, the electronic device converts multiple RGB images obtained by the ISP algorithm into multiple fourth YUV images, and then determines the weights corresponding to each fourth YUV image, wherein among the multiple images obtained by the ISP algorithm, the image whose generation time is closer to the image corresponding to the current AI algorithm has a greater weight, and the image whose generation time is farther from the image corresponding to the current AI algorithm has a smaller weight. The electronic device then performs weighted averaging on each fourth YUV image according to the weights of each fourth YUV image to obtain the first YUV image.

[0161] In some embodiments, the electronic device may further merge multiple UV images (ie, U images and V images) corresponding to the multiple fourth YUV images into one UV image.

[0162] In one example, the electronic device inputs multiple original images into the ISP algorithm to obtain multiple RGB images. The electronic device then converts the multiple RGB images into multiple YUV images, and performs weighted averaging on the U image and the V image in the multiple YUV images, and uses the weighted averaged U image and the V image as the first U image and the first V image in the first YUV image.

[0163] Specifically, Figure 8 As shown, the electronic device aligns the second RGB image obtained by the AI ​​algorithm and the multiple first RGB images obtained by the ISP algorithm. The electronic device then converts the multiple first RGB images into multiple fourth YUV images. For example, the fourth YUV image 1, the fourth YUV image 2...the fourth YUV image N. Each fourth YUV image corresponds to a fourth V image and a fourth U image. For example, the fourth YUV image 1 corresponds to the fourth V image 1 and the fourth U image 1, the fourth YUV image 2 corresponds to the fourth V image 2 and the fourth U image 2, and the fourth YUV image N corresponds to the fourth V image N and the fourth U image N. The electronic device performs weighted averaging on the multiple fourth U images to obtain the first U image after weighted averaging, and performs weighted averaging on the multiple fourth V images to obtain the first V image.

[0164] In some embodiments, the processing process of the electronic device performing weighted averaging on multiple fourth U images and multiple fourth V images may include: the electronic device first determines the weights corresponding to each fourth U image and each fourth V image, then weighted sums each fourth U image and each fourth V image, and performs averaging calculations to obtain the weighted averaged first U image and first V image.

[0165] In some embodiments, the weights corresponding to each fourth U image and each fourth V image are determined by the time difference between the image generation time corresponding to the current AI algorithm and the image generation time obtained by the ISP algorithm. Specifically, among the multiple images obtained by the ISP algorithm, the image whose generation time is closer to the image generation time corresponding to the current AI algorithm has a greater weight, and the image whose generation time is farther from the image generation time corresponding to the current AI algorithm has a smaller weight.

[0166] like Fig. 9 As shown, Q1 is the weight corresponding to image 2 obtained by the ISP algorithm, and Q2 is the weight corresponding to image 3 obtained by the ISP algorithm. Since the time difference between the generation time of image 1 and the generation time of image 3 corresponding to the AI ​​algorithm is less than the time difference between the generation time of image 2 and the generation time of image 3, that is, (t3-t2)>(t2-t1), Q1>Q2.

[0167] Next, we will take the first method to obtain a YUV image as an example to describe in detail how to perform color correction.

[0168] In some embodiments, the electronic device can make the colors in the second YUV image close to the colors in the first YUV image by filtering, thereby making the colors in the first YUV image close to the actual captured colors.

[0169] Specifically, the electronic device inputs the RAW image into the ISP algorithm and the AI ​​algorithm respectively, and can obtain the first YUV image and the second YUV image. The first YUV image includes the first Y image, the first U image and the first V image. The second YUV image may include the second Y image, the second U image and the second V image. Fig.10 As shown, the electronic device uses the second U image and the second V image obtained by the AI ​​algorithm as input images, and uses the first U image and the first V image obtained by the ISP algorithm as reference images (or guide images), and obtains the third U image and the third V image through filtering, so that the third U image is close to the first U image, and the third V image is close to the first V image. Exemplarily, during the filtering process, the electronic device can generate an adjustment coefficient, and then adjust the U value of each pixel in the second U image based on the adjustment coefficient to obtain the third U image, and adjust the V value of each pixel in the second V image to obtain the third V image.

[0170] It should be understood that the third U image is close to the first U image, which means that for each pixel in the second U image, the U value corresponding to the pixel in the third U image is close to the U value corresponding to the pixel in the first U image. In addition, the third V image is close to the first V image, which means that the V value corresponding to the pixel in the third U image is close to the V value corresponding to the pixel in the first U image.

[0171] The electronic device may combine the second Y image in the second YUV image with the filtered third V image and the third U image to obtain a color-corrected YUV image, and then convert the color-corrected YUV image into an RGB image.

[0172] In some embodiments, the electronic device can implement filtering processing by guided filtering, and can also implement filtering processing by bilateral filtering to correct the color of the second YUV image and solve the color cast problem. In the embodiments of the present application, no limitation is imposed on the filtering method.

[0173] The following takes guided filtering as an example to explain the filtering process in more detail.

[0174] In some embodiments, the process of adjusting each region by the electronic device through the adjustment coefficient may include: for each region, the electronic device linearly transforms the second chromaticity value corresponding to each pixel point in the region in the second chromaticity diagram (i.e., the second U image and the second V image) according to the adjustment coefficient corresponding to the region, to obtain a third chromaticity diagram (i.e., the third U image and the third V image). For any pixel point, the third chromaticity value corresponding to the pixel point in the third chromaticity diagram is closer to the first chromaticity value corresponding to the pixel point in the first chromaticity diagram (i.e., the first U image and the first V image) than the second chromaticity value corresponding to the pixel point. The electronic device then generates a third YUV image based on the third chromaticity diagram and the first brightness diagram in the second YUV image.

[0175] The electronic device may divide the image into blocks through the window to obtain multiple areas.

[0176] In some embodiments, Fig.11 As shown, the U value of the second U image and the U value of the third U image satisfy a linear relationship, and the V value of the second V image and the V value of the third V image also satisfy a linear relationship. The linear function can be: .

[0177] When q represents the U value corresponding to the area in the third U image, I represents the U value corresponding to the area in the second U image. Alternatively, when q represents the V value corresponding to the area in the third V image, I represents the V value corresponding to the area in the second V image. For any pixel point, the electronic device adjusts the coefficients (i.e., a and b) so that the difference between the U value corresponding to the pixel point in the third U image and the U value corresponding to the pixel point in the first U image is smaller than the difference between the U value corresponding to the pixel point in the second U image and the U value corresponding to the pixel point in the first U image, that is, when q=pn, n is minimized.

[0178] In some embodiments, before the electronic device performs a linear transformation according to the adjustment coefficient corresponding to the second chromaticity value, the process of determining the adjustment coefficient may include: for each region, the electronic device determines the first objective function corresponding to the region. The first objective function is used to characterize the first chromaticity difference corresponding to the region, and the first chromaticity difference refers to the difference between the adjusted chromaticity value corresponding to the region and the chromaticity value corresponding to the region in the first YUV image. The adjusted chromaticity value corresponding to the region is characterized by the adjustment coefficient to be solved and the chromaticity value corresponding to the region in the second YUV image. The electronic device then obtains the adjustment coefficient when the first objective function takes the minimum value.

[0179] Exemplarily, the first objective function can be , used to characterize the first chromaticity difference corresponding to each area. The electronic device can use the minimum distance function: The minimum value of the first objective function is solved, and the values ​​of a and b when the first objective function takes the minimum value are used as the adjustment coefficients corresponding to each area.

[0180] In some embodiments, the adjusted chromaticity value corresponding to the region includes the adjusted chromaticity value corresponding to each pixel in the region. The adjusted chromaticity value corresponding to the region is characterized by the adjustment coefficient to be solved and the second chromaticity value corresponding to each pixel in the region. The first chromaticity difference is determined based on the second chromaticity difference of each pixel in the region, and the second chromaticity difference of each pixel refers to the difference between the adjusted chromaticity value corresponding to the pixel and the first chromaticity value corresponding to the pixel.

[0181] Specifically, the electronic device can use the distance function: Determine the minimum difference between the U value of each pixel in the area of ​​the third U image and the U value of the corresponding pixel in the first U image, and determine the minimum difference between the V value of each pixel in the area of ​​the third V image and the V value of the corresponding pixel in the first V image.

[0182] There is a linear relationship between the V value of each pixel in the area of ​​the third V image and the V value of the corresponding pixel in the second V image, and there is a linear relationship between the U value of each pixel in the area of ​​the third U image and the U value of the corresponding pixel in the second U image, and the linear function can be: . and represents the adjustment coefficient corresponding to the kth second region, w represents the region, represents the kth region. When representing the U value of the i-th pixel in the k-th second area, Characterizes the U value of the i-th pixel in the k-th third area, When V represents the V value of the i-th pixel in the k-th second area, Characterizes the V value of the i-th pixel in the k-th third area. The electronic device then substitutes the linear function into the above distance function, and its minimum distance function can also be: The second area refers to the area after the second U image and the second V image are divided into blocks. The third area refers to the area obtained by the electronic device adjusting the pixel points in multiple second areas through the adjustment coefficient, and the electronic device can obtain a third YUV image through multiple third areas.

[0183] When the electronic device determines the minimum difference between the U value of each pixel point in the area of ​​the third U image and the U value of the corresponding pixel point in the first U image, and determines the minimum difference between the V value of each pixel point in the area of ​​the third V image and the V value of the corresponding pixel point in the first V image, it can be obtained. and The value of Representation.

[0184] in, Characterizes the number of pixels in the second region. When representing the U value corresponding to the i-th pixel in the k-th second area, Characterizes the mean value of U value in the k-th second region, Characterizes the mean value of U value in the k-th first region, Used to characterize the variance of the U value in the kth second region. When representing the V value corresponding to the i-th pixel in the k-th second area, Characterizes the mean value of V value in the k-th second region, Characterizes the mean value of V value in the kth first region, It is used to characterize the variance of the V value in the kth second region. The first region refers to a region in the first U image and the first V image that corresponds to the position and size of the second region.

[0185] is a preset adjustable value, used to make the U value and V value in the third area close to the U value and V value in the second area, or to make the U value and V value in the third area close to the U value and V value in the first area. is a larger value, then The smaller, for example is close to 0. In this case, the U value of the pixel in the kth third region is the average of the U values ​​of the kth first region, and the V value of the pixel in the kth third region is the average of the V values ​​of the kth first region, that is, Approximately .

[0186] It is used to characterize the covariance between the second area and the first area, and can also be used Characterization, It can also be used to characterize the variance of the second region, or Representation, that is . You can also use Indicates, that is, .

[0187] In some other embodiments, the minimum distance function may also be: , after the electronic device solves and Same as above.

[0188] In some embodiments, the electronic device divides the second YUV image into blocks to obtain multiple areas, and the adjustment coefficient corresponding to each area includes a first adjustment coefficient and a second adjustment coefficient. The first adjustment coefficient is used to adjust the U value of each pixel in the area, and the second adjustment coefficient is used to adjust the V value of each pixel in the area. The chromaticity value corresponding to each pixel in the second YUV image can be recorded as a second chromaticity value. The second chromaticity value corresponding to each pixel may include a first value (or called a U value) and a second value (or called a V value), the first value is the chromaticity value corresponding to the U channel, and the second value is the chromaticity value corresponding to the V channel. The electronic device can adjust the first value corresponding to each pixel in the area according to the first adjustment coefficient corresponding to the area for each area. The electronic device can also adjust the second value corresponding to each pixel in the area according to the second adjustment coefficient corresponding to the area for each area.

[0189] Specifically, the electronic device can adjust the U value of each pixel in the second area by the first adjustment coefficient corresponding to each second area, and adjust the V value of each pixel in the second area by the second adjustment coefficient corresponding to each second area, to obtain multiple adjusted third areas, and the U value and V value of the pixel in the third area are close to the U value and V value in the first area.

[0190] It can be understood that in this embodiment, the image is divided into blocks to obtain different areas, and different adjustment coefficients are set for different areas for adjustment, and different adjustment coefficients are also set for the same area when adjusting the U value and V value. This can make the adjusted U image and V image closer to the reference U image and V image. The finer the granularity during the adjustment process, the better the effect of the adjusted U image and V image.

[0191] In some embodiments, after obtaining the adjustment coefficient corresponding to each area through the above steps, the electronic device can The second V image and the second U image obtained by the AI ​​algorithm are filtered to obtain a filtered third V image and a third U image, that is, a color-corrected third V image and a third U image. The electronic device then obtains the Y channel in the second YUV image corresponding to the second Y image, and combines the filtered third V image and the third U image to obtain the final YUV image.

[0192] In some embodiments, if For very small increments, assuming and Much smaller than ,but The value of is approximately equal to 0. In this case, Approximately ,Right now, Approximately , the V value of each pixel in the kth third region is approximately equal to the average value of the V value of the kth first region, or the U value of each pixel in the kth third region is approximately equal to the average value of the U value of the kth first region.

[0193] An example, Fig.12 (a) is the first YUV image. Fig.12 (b) is the second YUV image. Fig.12 (c) is the third YUV image. The second YUV image is smooth. In this case, Much smaller than , the electronic device can directly use the mean value of the first YUV image as the adjusted value of the second YUV image, that is, the electronic device can output Fig.12 Middle (c).

[0194] In some embodiments, the electronic device may also retain the edge of the image during filtering. The edge of the image is caused by the discontinuity or mutation of the gray value in the image, that is, the edge of the image is the area in the image where the gray value changes most significantly. For example, Fig.13 (a) is the image before filtering, which is in a step-like shape, that is, the image has a grayscale mutation. Fig.13 After filtering (a) in the figure, we can get Fig.13 (b) in which the filtered image retains the image edge, that is, the edge information of the image is retained.

[0195] In some embodiments, as described above, the entire second YUV image may correspond to the same adjustment coefficient, and the electronic device may use the same adjustment coefficient to adjust the chromaticity value of the entire second YUV image. Specifically, the electronic device determines a second objective function, wherein the second objective function is used to characterize the third chromaticity difference, the third chromaticity difference refers to the sum of the fourth chromaticity differences of each pixel in the first YUV image, and the fourth chromaticity difference of each pixel refers to the difference between the adjusted chromaticity value corresponding to the pixel and the first chromaticity value corresponding to the pixel. The adjusted chromaticity value corresponding to the pixel is characterized by the adjustment coefficient to be solved and the second chromaticity value corresponding to the pixel in the second YUV image. The electronic device then obtains the adjustment coefficient when the second objective function takes the minimum value.

[0196] In this embodiment, the electronic device may not block the second YUV image, and determine the adjustment coefficient by the difference between the chromaticity value of each pixel in the third YUV image and the chromaticity value of the corresponding pixel in the first YUV image. That is, the electronic device may adjust the entire second YUV image by the second objective function, that is, by determining the minimum difference between the U value and the V value of each pixel in the third YUV image and the U value and the V value of the corresponding pixel in the first YUV image, and determining the adjustment coefficient at the minimum difference.

[0197] Some other embodiments of the present application provide an electronic device, which may include: a memory and one or more processors. The memory is coupled to the processor. The memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs each function or step in the above method embodiment.

[0198] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the electronic device in the above-mentioned method embodiment.

[0199] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes each function or step executed by the electronic device in the above method embodiment.

[0200] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0201] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0202] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0203] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0204] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0205] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. An image generation method, characterized in that: Used in electronic equipment, including: In response to a shooting instruction, a first YUV image and a second YUV image are acquired; the first YUV image is a YUV image generated based on an ISP algorithm; and the second YUV image is a YUV image generated based on an AI algorithm; According to the adjustment coefficient corresponding to the second YUV image, the second chromaticity value corresponding to each pixel in the second YUV image is adjusted to obtain a third YUV image; wherein, for any pixel, the third chromaticity value corresponding to the pixel in the third YUV image is closer to the first chromaticity value than the second chromaticity value corresponding to the pixel; and the first chromaticity value is the chromaticity value corresponding to the pixel in the first YUV image; Based on the third YUV image, generating an image corresponding to the shooting instruction; Before the second chromaticity value corresponding to each pixel point in the second YUV image is adjusted according to the adjustment coefficient corresponding to the second YUV image to obtain the third YUV image, the method includes: Determine a second objective function; the second objective function is used to characterize a third chromaticity difference; the third chromaticity difference refers to the sum of the fourth chromaticity differences of each pixel in the first YUV image; the fourth chromaticity difference of each pixel refers to the difference between the adjusted chromaticity value corresponding to the pixel and the first chromaticity value corresponding to the pixel in the first YUV image; the adjusted chromaticity value corresponding to the pixel is characterized by the adjustment coefficient to be solved and the second chromaticity value; The adjustment coefficient when the second objective function takes a minimum value is obtained.

2. The method according to claim 1, characterized in that The second YUV image includes a plurality of regions; each region has a corresponding adjustment coefficient; The step of adjusting the second chromaticity value corresponding to each pixel in the second YUV image according to the adjustment coefficient corresponding to the second YUV image to obtain the third YUV image includes: For each of the regions, the second chromaticity values ​​corresponding to each pixel in the region are adjusted according to the adjustment coefficient corresponding to the region to obtain a third YUV image.

3. The method according to claim 2, characterized in that The adjustment coefficient corresponding to each of the regions includes a first adjustment coefficient and a second adjustment coefficient; the second chromaticity value corresponding to each pixel includes a first value and a second value; the first value is the chromaticity value corresponding to the U channel; the second value is the chromaticity value corresponding to the V channel; The step of adjusting, for each of the regions, the second chromaticity value corresponding to each pixel point in the region according to the adjustment coefficient corresponding to the region, to obtain the third YUV image comprises: For each of the regions, adjusting the first value corresponding to each pixel in the region according to the first adjustment coefficient corresponding to the region; For each of the regions, the second value corresponding to each pixel in the region is adjusted according to the second adjustment coefficient corresponding to the region.

4. The method according to claim 2, characterized in that: The first YUV image includes a first chromaticity diagram; the second YUV image includes a second chromaticity diagram; The step of adjusting, for each of the regions, the second chromaticity value corresponding to each pixel point in the region according to the adjustment coefficient corresponding to the region, to obtain the third YUV image comprises: For each of the regions, according to the adjustment coefficient corresponding to the region, a linear transformation is performed on the second chromaticity value corresponding to each pixel in the region in the second chromaticity diagram to obtain a third chromaticity diagram; for any pixel, the third chromaticity value corresponding to the pixel in the third chromaticity diagram is closer to the first chromaticity value corresponding to the pixel in the first chromaticity diagram than the second chromaticity value corresponding to the pixel; The third YUV image is generated according to the third chromaticity image and the first brightness image in the second YUV image.

5. The method according to claim 2, characterized in that: Before adjusting, for each of the regions, the second chromaticity value corresponding to each pixel point in the region according to the adjustment coefficient corresponding to the region, the method further includes: For each region, determine a first objective function corresponding to the region; the first objective function is used to characterize a first chromaticity difference corresponding to the region; the first chromaticity difference refers to a difference between an adjusted chromaticity value corresponding to the region and a chromaticity value corresponding to the region in the first YUV image; the adjusted chromaticity value corresponding to the region is characterized by an adjustment coefficient to be solved and a chromaticity value corresponding to the region in the second YUV image; Obtain an adjustment coefficient when the first objective function takes a minimum value.

6. The method according to claim 5, characterized in that The adjusted chromaticity value corresponding to the region includes the adjusted chromaticity value corresponding to each pixel point in the region; the adjusted chromaticity value corresponding to the region is represented by the adjustment coefficient to be solved and the second chromaticity value corresponding to each pixel point in the region; The first chromaticity difference is determined based on the second chromaticity difference of each pixel in the area; the second chromaticity difference of each pixel refers to the difference between the adjusted chromaticity value corresponding to the pixel and the first chromaticity value corresponding to the pixel.

7. The method according to any one of claims 1 to 6, characterized in that The second YUV image is generated from a first original image; the first original image is one of a plurality of original images collected in response to the shooting instruction; The obtaining of the first YUV image comprises: Acquire a plurality of second original images; the plurality of second original images are included in the plurality of original images; Determine a fourth YUV image corresponding to each of the plurality of second original images; The first YUV image is obtained by fusing the plurality of the fourth YUV images.

8. The method according to claim 7, characterized in that The fusing the plurality of fourth YUV images to obtain the first YUV image comprises: Determine the weights corresponding to the plurality of fourth YUV images respectively; the time difference corresponding to each of the fourth YUV images is negatively correlated with the weight; the time difference refers to the difference between the first time and the second time; the first time is the generation time of the second original image corresponding to the fourth YUV image, and the second time is the generation time of the first original image; The fourth YUV images are weightedly fused according to the weights respectively corresponding to the fourth YUV images to obtain the first YUV image.

9. The method according to claim 7, characterized in that: The plurality of original images are original images continuously acquired in response to the shooting instruction in a continuous shooting mode; or, The multiple original images are original images captured in a single-shooting mode and used for image preview; the acquisition time of the multiple original images and the time of receiving the shooting instruction meet a preset proximity condition.

10. An electronic device, characterized in that: The electronic device comprises at least: a memory and one or more processors; the memory is used to store computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 9.

12. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

Images