Image processing method and device, electronic equipment and readable storage medium

Abstract

The invention relates to an image processing method and device, electronic equipment and a readable storage medium. The method comprises the following steps: acquiring a first image and a second image based on use of the same target scene; wherein the shooting flash intensities of the second image and the first image are different; acquiring a third image containing single light source information according to the first image and the second image; the single light source information indicates that a flash lamp serves as a single light source; performing gain processing on the second image to obtain a first gain image, and performing gain processing on the third image to obtain a second gain image; and obtaining a target enhanced image based on the first gain image and the second gain image. According to the invention, the photographing effect can be ensured.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] As users use electronic devices more frequently, they have higher requirements for the photographic effects of electronic devices. Flash is an important functional device module or peripheral for photography, and is widely used in various electronic devices (including smart mobile terminals, professional cameras, professional lighting peripherals, etc.). However, due to the power consumption requirements and thermal design requirements of the equipment, the current flash power is not high enough, which affects the photographic effect. Summary of the invention

[0003] Based on this, it is necessary to provide an image processing method, device, electronic device and readable storage medium that can ensure the photographic effect in response to the above technical problems.

[0004] In a first aspect, the present application provides an image processing method, which is applied to an electronic device having a camera and a flash shooting function, and the method comprises:

[0005] Acquire a first image and a second image based on the same target scene; wherein the second image and the first image have different shooting flash intensities;

[0006] According to the first image and the second image, a third image including single light source information is acquired; the single light source information indicates that a flash is used as a single light source;

[0007] Performing gain processing on the second image to obtain a first gain image, and performing gain processing on the third image to obtain a second gain image;

[0008] A target enhanced image is obtained based on the first gain image and the second gain image.

[0009] In one of the embodiments, the first image includes at least one first image frame, and the second image includes at least one second image frame;

[0010] Acquiring a first image and a second image based on the same target scene includes:

[0011] A first image frame and a second image frame are determined from a plurality of target image frames; the plurality of target image frames are a plurality of original image frames obtained in a flashing mode of a flash lamp.

[0012] In one embodiment, the time interval between the first image frame and the second image frame is less than a preset time interval.

[0013] In one embodiment, acquiring a third image containing single light source information according to the first image and the second image includes:

[0014] Aligning pixels of the first image frame with the second image frame to obtain an aligned image frame;

[0015] Performing pixel gain processing on the aligned image frame to obtain a gain aligned image frame;

[0016] Obtaining a difference image frame according to the second image frame and the gain alignment image frame;

[0017] Performing filtering on the difference image frame to obtain a third image frame;

[0018] A third image is obtained based on the third image frame.

[0019] In one embodiment, obtaining a third image based on the third image frame includes:

[0020] When the number of the third image frame is one, using the third image frame as the third image;

[0021] When there are multiple third image frames, the multiple third image frames are averaged to obtain the third image.

[0022] In one embodiment, the gain parameter of the pixel gain processing is related to the total gain coefficient of the second image frame and the total gain coefficient of the first image frame; the total gain coefficient of the second image frame is related to the shutter time for capturing the second image frame, and the total gain coefficient of the first image frame is related to the shutter time for capturing the first image frame.

[0023] In one embodiment, performing gain processing on the second image to obtain a first gain image, and performing gain processing on the third image to obtain a second gain image, comprises:

[0024] Determining a first gain coefficient and a second gain coefficient according to the shooting scene information;

[0025] The first gain image is obtained by performing gain processing on the second image using the first gain coefficient, and the second gain image is obtained by performing gain processing on the third image using the second gain coefficient.

[0026] In one embodiment, the shooting scene information includes scene semantic information; and determining the first gain coefficient and the second gain coefficient according to the shooting scene information includes:

[0027] According to the scene semantic information, the brightness coefficient of the foreground subject and the brightness coefficient of the background are determined;

[0028] Based on the first brightness coefficient and the background brightness coefficient, a first gain coefficient is obtained; the first brightness coefficient represents the brightness coefficient of the flash light to the background area;

[0029] Based on the second brightness coefficient, the first gain coefficient, the foreground subject brightness coefficient and the background brightness coefficient, a second gain coefficient is obtained; the second brightness coefficient represents the brightness coefficient of the flash light to the foreground area.

[0030] In one embodiment, the first brightness coefficient includes the brightness coefficient of the background area of ​​the second image; the second brightness coefficient includes the brightness coefficient of the foreground area of ​​the second image and the brightness coefficient of the foreground area of ​​the third image; the method further includes:

[0031] According to the second image and the third image, a target image including scene light source information is acquired; the scene light source information indicates that scene light other than the flash is used as the light source;

[0032] Among them, the brightness coefficient of the background area of ​​the second image is determined based on the average brightness of the background area of ​​the second image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the background area of ​​the second image is positively correlated with the average brightness of the background area of ​​the second image and negatively correlated with the average brightness of the foreground area of ​​the target image; the brightness coefficient of the foreground area of ​​the second image is determined based on the average brightness of the foreground area of ​​the second image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the foreground area of ​​the second image is positively correlated with the average brightness of the foreground area of ​​the second image and negatively correlated with the average brightness of the foreground area of ​​the target image; the brightness coefficient of the foreground area of ​​the third image is determined based on the average brightness of the foreground area of ​​the third image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the foreground area of ​​the third image is positively correlated with the average brightness of the foreground area of ​​the third image and negatively correlated with the average brightness of the foreground area of ​​the target image.

[0033] In one of the embodiments, the shooting scene information further includes semantic mask information; and the method further includes:

[0034] According to the semantic mask information, the foreground area and the background area in the second image, the foreground area and the background area in the third image, and the foreground area and the background area in the target image are identified.

[0035] In one embodiment, acquiring a target image containing scene light source information according to the second image and the third image includes:

[0036] Performing filtering on the second image to obtain a filtered image;

[0037] A target image is obtained according to the filtered image, the third image and the target coefficient; wherein the target coefficient is determined based on the flash power value of the second image and the flash power value of the third image, and the target coefficient is positively correlated with the flash power value of the second image and negatively correlated with the flash power value of the third image.

[0038] In a second aspect, the present application further provides an image processing device, which is applied to an electronic device having a camera and a flash shooting function, and the device comprises:

[0039] A flash image acquisition module, used to acquire a first image and a second image based on the same target scene; wherein the second image and the first image have different shooting flash intensities;

[0040] An image acquisition module is used to acquire a third image including single light source information according to the first image and the second image; the single light source information indicates that a flash is used as a single light source;

[0041] A gain module, configured to perform gain processing on the second image to obtain a first gain image, and perform gain processing on the third image to obtain a second gain image;

[0042] The image acquisition module is used to obtain a target enhanced image based on the first gain image and the second gain image.

[0043] In a third aspect, the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0044] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when the computer program is executed by a processor.

[0045] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0046] The above-mentioned image processing method, device, electronic device and readable storage medium can obtain a first image and a second image based on the same target scene, where the shooting flash intensity of the second image and the first image is different, and then based on the first image and the second image, a third image containing single light source information is obtained, wherein the single light source information indicates that the flash is used as a single light source, and by gaining the second image and the third image respectively, a target enhanced image is finally obtained. The embodiment of the present application can clearly control the software simulation of the flash power according to user needs to obtain a more realistic effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A diagram showing an application environment of an image processing method in an embodiment;

[0049] Figure 2 is a schematic flow chart of an image processing method in one embodiment;

[0050] Figure 3 is a schematic diagram of a process of acquiring a third image in an embodiment;

[0051] Figure 4 A schematic diagram of a process for obtaining a gain image in one embodiment;

[0052] Figure 5 A schematic diagram of a flow chart of obtaining a gain coefficient in one embodiment;

[0053] Figure 6 A schematic diagram of a specific process of an image processing method in one embodiment;

[0054] Figure 7 is a schematic diagram of a specific flow chart of an image processing method in another embodiment;

[0055] Figure 8 is a structural block diagram of an image processing device in one embodiment;

[0056] Fig. 9 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0059] It is understood that in this application, terms such as "first", "second", etc. are only used to distinguish similar objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. It is understood that "at least one" means one or more, and "multiple" means two or more, for example, "multiple frames" means two or more frames.

[0060] When used herein, the singular forms "a", "an", and "said / the" may also include plural forms, unless the context clearly indicates otherwise. It should also be understood that the terms "include / comprise" or "have" and the like specify the presence of stated features, wholes, steps, operations, components, parts, or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts, or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0061] Photography is an extremely widespread application that is both practical and artistic. Photography is also a key function and key technology of various current electronic devices (such as terminal devices, including smart mobile terminals). Flash is an important functional device module or peripheral for photography, and is widely present in various terminal devices (including smart mobile terminals, professional cameras, professional lighting peripherals, etc.). Flash is not only a fill-in light device for dim light at the moment of photography, but also an important, popular, and multi-functional photography style for shooting "strong flash" images, which can obtain effects including but not limited to "foreground subject brightness coefficient" and "portrait light and shadow three-dimensional". For example, the "foreground subject brightness coefficient" is to brighten the "foreground subject" with a higher-power flash, making its brightness higher than the background, to obtain the effect of "highlighting the subject".

[0062] Currently, electronic devices are limited by the power consumption requirements and thermal design requirements of the equipment. The flash power is not high enough to directly meet the user's needs for "strong flash" style shooting. It is usually significantly weaker than a dedicated camera with a significantly higher power built-in flash or a separate high-power external flash. In the traditional solution, in order to meet the user's needs for "strong flash" style shooting, electronic devices first optimize the flash-related hardware design, including the use of multiple lights, zoom lampshades, etc., but they are also limited by thermal design and lead to increased hardware costs. The second is to use software to perform post-processing enhancement of the "strong flash" style on the photos taken with the existing flash, but it is usually impossible to accurately distinguish between "foreground and background depth", "flash shadow", etc., resulting in false post-processing enhancement effects. There is currently no relevant software algorithm, and the software algorithm itself is difficult to clearly control higher flash power, calculate "foreground and background depth", and calculate "light and shadow brought by the flash", resulting in false or even abnormal software algorithm effects.

[0063] For example, conventional solutions use the difference between flash-illuminated images and non-flash-illuminated images to extract foreground, or use flash-illuminated images as training data to obtain better de-reflected images, or use low-light enhancement methods, with brightness enhancement in the first step and color enhancement in the second step. In this regard, the present application provides a flash enhancement method without displaying the foreground calculation, which can involve a high light ratio effect, which is different from low-light enhancement.

[0064] The image processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the imaging unit 102 outputs the image being captured to the processor 104, and the memory 106 stores the captured image, which can be acquired and processed by the processor 104. Exemplarily, the processor 104 may include one or more processors of CPU (Central Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processing Unit), NPU (Neural-network Processing Unit), and the deep neural network model usually runs on a GPU or DSP or NPU processor. The input and output unit 108 includes but is not limited to a display, a touch screen, etc., and the user interacts through the input and output unit, including displaying images, etc.

[0065] Based on the above-mentioned traditional technology, the embodiment of the present application proposes a camera flash photography enhancement solution, which can clearly control the software simulation of the flash power according to user needs, and avoid direct calculation of "foreground and background depth" and "light and shadow brought by the flash", so as to obtain a more realistic effect. It should be noted that the beneficial effects or technical problems solved by the embodiment of the present application are not limited to this one, and can also be other implicit or related problems. For details, please refer to the description of the following embodiment.

[0066] Before introducing the specific embodiments of the present application, the professional terms involved in the present application are explained first:

[0067] DNN: Deep Neural Network, deep neural network.

[0068] CNN: Convolutional Neural Network, convolutional neural network.

[0069] AWB: Automatic White Balance, automatic white balance.

[0070] ISP link: Image Signal Processing Pipeline, image signal processing link or image information processing link, hereinafter referred to as image processing link.

[0071] OBC: Optical Black Correct, optical dark area correction.

[0072] RAW: RAW Image Format, usually refers to the Bayer format data output by the image sensor.

[0073] Mask: Mask information, generally refers to semantic segmentation mask information (Semantic mask image).

[0074] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0075] In an exemplary embodiment, Figure 2 As shown, an image processing method is provided, and the method is applied to an electronic device with a camera and a flash light shooting function as an example for explanation. It can be understood that the method can be applied to Figure 1 The processor in the embodiment can also be applied to a flash enhancement module, and can also be applied to a system including an exposure control module, a flash enhancement module and an image signal processing link (ISP), and is implemented through interaction between modules and links. The method includes the following steps:

[0076] Step 202: Acquire a first image and a second image based on the same target scene; wherein the second image and the first image have different shooting flash intensities.

[0077] The target scene refers to the scene that the camera on the electronic device shoots. For example, a flash can be used to shoot the first image and the second image in the same target scene. The first image and the second image can be understood as image data based on the image captured when the flash is emitting light. Furthermore, as a scene for image acquisition, the target scene can include a variety of different types of objects (i.e., the subject being photographed) and backgrounds. Different objects can have the same or different backgrounds. For example, buildings, people, animals, plants, or vehicles can all be objects in the target scene.

[0078] Exemplarily, the first image and the second image can be acquired in a flash photography mode, such as a flash mode. Optionally, the flash photography mode may include but is not limited to a flash photography mode. Further, multiple frames of images can be continuously acquired for the same target scene during a flash photography process, thereby acquiring the first image and the second image. Specifically, the first image and the second image can be original RAW images (for example, original RAW frames); exemplarily, the flash enhancement module can acquire the first image and the second image through an exposure control module (AE, Auto Exposure). Optionally, the flash enhancement module can include a flash enhancement first module, and the flash enhancement first module can be used to acquire the first image and the second image.

[0079] Taking a flash Flash photography process as an example, relevant original RAW frames are obtained in the flash Flash photography process, including a first original RAW frame and a second original RAW frame (the first image can use the first original RAW frame, and the second image can use the second original RAW frame); it should be noted that the present application outputs relevant original RAW frames obtained in a flash Flash photography process, including but not limited to the above-mentioned first original RAW frame and the second original RAW frame.

[0080] Among them, the shooting flash intensity of the second image is different from that of the first image. The shooting flash intensity in the embodiment of the present application can be simply referred to as flash intensity. Exemplarily, the flash intensity of the second image is greater than that of the first image. For example, the first image is a raw RAW frame with no flash or weak flash, and the second image is a raw RAW frame with strong flash. Exemplarily, the shutter time (shutter) and related gain (including sensor gain) corresponding to the raw RAW frame with weak flash, the raw RAW frame with no flash, and the raw RAW frame with strong flash are different. It can be understood that the above-mentioned raw RAW frame with no flash or weak flash, and the raw RAW frame with strong flash can also take other forms, not limited to the forms mentioned in the above-mentioned embodiments, as long as it can achieve that the flash intensity of the second image is greater than that of the first image.

[0081] The embodiment of the present application lays the foundation for the subsequent acquisition of the third image by acquiring the first image and the second image with different flash intensities, and performs post-processing enhancement in the style of "strong flash" on the image taken with the existing flash through a software algorithm, thereby avoiding direct calculation of the "light and shadow caused by the flash", achieving clear control of higher flash power, and obtaining a more realistic effect.

[0082] Step 204 , obtaining a third image including single light source information according to the first image and the second image; the single light source information indicates that a flash is used as a single light source.

[0083] Specifically, when a first image and a second image having a flash intensity greater than that of the first image are acquired, a third image may be acquired based on the first image and the second image, and the third image includes single light source information, wherein the single light source information may indicate that a flash is used as a single light source. Optionally, the flash enhancement first module may be used to acquire the third image based on the first image and the second image.

[0084] Exemplarily, the third image may be a RAW image; optionally, containing single light source information may mean that the third image is a RAW image "completely generated by flash illumination and does not contain illumination from other light sources", that is, a complete flash RAW image; it can also be understood that the third image is a complete flash RAW frame and does not contain information brought by other light sources.

[0085] Regarding the method of acquiring the third image, the third image can be acquired by subtracting the first image and the second image after pixel alignment and gain are performed. In a possible implementation, taking the first image using the first original RAW frame and the second image using the second original RAW frame as an example, the first original RAW frame can be matched with the second original RAW frame, and processing including but not limited to pixel alignment and gain processing is performed, and the two frames are subtracted and filtered, and a complete flash RAW frame (which can be called a third RAW frame) is output as the third image.

[0086] It is understandable that the third image may be acquired in other forms, not limited to the forms mentioned in the above embodiments, as long as it can acquire an image containing single light source information, where the single light source is a flash.

[0087] By acquiring the third image, the embodiment of the present application can avoid the direct calculation of the "light and shadow caused by the flash", and at the same time facilitate clear control of the software simulation of the flash power according to user needs, thereby avoiding the direct calculation of the "depth of the foreground and background" and improving the photographic effect.

[0088] Step 206: Perform gain processing on the second image to obtain a first gain image, and perform gain processing on the third image to obtain a second gain image.

[0089] Specifically, the second image may be subjected to gain processing to obtain a first gain image, and when the third image is obtained, the third image may be subjected to gain processing to obtain a second gain image. Optionally, the flash enhancement module may further include a flash enhancement second module, which may be used to obtain the second image and the third image transmitted by the flash enhancement first module, and further obtain the first gain image and the second gain image.

[0090] In some embodiments, gain parameters, such as gain coefficients, for performing gain processing on the second image and the third image may be calculated respectively, so as to obtain a target enhanced image by summing them. Exemplarily, the gain parameters may be determined according to user requirements, such as the user's photography stylization requirements for "software simulating higher flash power effect" or "higher foreground and background light ratio contrast effect", or other requirements, including but not limited to "simulating lower flash power", etc.

[0091] The embodiment of the present application can clearly control the software simulation of the flash power according to user needs, and can avoid the direct calculation of "depth of foreground and background" and "light and shadow caused by the flash", so as to obtain a more realistic effect.

[0092] Step 208: Obtain a target enhanced image based on the first gain image and the second gain image.

[0093] Specifically, when the first gain image and the second gain image are obtained, a target enhanced image can be obtained based on the first gain image and the second gain image. Exemplarily, the flash enhancement second module may include a flash enhancement control module, which can be used to obtain a target enhanced image based on the first gain image and the second gain image.

[0094] Optionally, the first gain image and the second gain image may be added to obtain a target enhanced image. Taking the flash enhancement control module as an example, the flash enhancement control module may be used to calculate gain parameters for performing gain processing on the second image and the third image, respectively, and sum them to obtain the target enhanced image. It is understood that the above-mentioned method of obtaining the target enhanced image may also adopt other forms, not limited to the forms mentioned in the above-mentioned embodiments, as long as it can achieve the target enhanced image based on the first gain image and the second gain image.

[0095] In a possible implementation, the target enhanced image may refer to an image frame containing flash enhancement, and the image frame containing flash enhancement may subsequently obtain a flash enhanced image through an image processing link (ISP); taking the second image using the second original RAW frame and the third image using the third RAW frame as an example, the flash enhancement control module may be used to calculate the gain coefficients of the second original RAW frame and the third RAW frame respectively, and sum them to obtain a fourth RAW frame (i.e., an image frame containing flash enhancement), and then the fourth RAW frame is obtained through a subsequent image processing link (ISP) to obtain a flash enhanced image (i.e., an image with an enhanced flash effect), and the image type of the flash enhanced image may be an RGB image or a YUV image. Among them, the fourth RAW frame may be called the fourth RAW image (Raw_4), i.e., a flash enhanced RAW image.

[0096] It should be noted that the embodiment of the present application does not specifically limit or describe the image processing link (ISP), which may include but is not limited to demosaicing, white balance, gamma correction, jpeg compression encoding, etc. Furthermore, post-processing algorithms such as red-eye removal can also be applied to obtain better results. In addition, although the description method of flash-enhanced images is adopted, the flash enhancement in the embodiment of the present application not only includes enhancing the flash effect and providing the high light ratio required by the user, but can also be used to reduce the light ratio, such as retaining the image quality of a strong flash but not retaining the high light ratio. This application does not limit this.

[0097] In the above-mentioned image processing method, the software is used to perform post-processing enhancement of the relevant photographic style on the image taken with the existing flash. For example, the software simulation of the flash power is clearly controlled according to user needs, which can avoid direct calculation of the "depth of foreground and background" and "light and shadow caused by the flash" to obtain a more realistic effect. In addition, the software process of the embodiment of the present application can be better embedded in the ISP process and is highly practical.

[0098] It should be noted that the flash enhancement module in the embodiment of the present application includes but is not limited to a first flash enhancement module, a second flash enhancement module and a flash enhancement control module, and the above modules may refer to the functional modules divided by the chip. Furthermore, the above modules may be implemented in whole or in part by software, hardware and a combination thereof. The above modules may be embedded in or independent of the processor in the electronic device in the form of hardware, or may be stored in the memory in the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0099] In practical applications, the acquisition of the first image, the second image, and the third image can be based on multiple original RAW frames acquired during a single photo-taking process, and multiple frame information can be applied to obtain better image quality using a corresponding multi-frame algorithm. In one embodiment, the first image includes at least one first image frame, and the second image includes at least one second image frame; the acquisition of the first image and the second image based on the same target scene includes:

[0100] A first image frame and a second image frame are determined from a plurality of target image frames; the plurality of target image frames are a plurality of original image frames obtained in a flashing mode of a flash lamp.

[0101] Specifically, the flash mode of the flash may refer to turning on the flash to fill in the light, and the flash function may be realized through the flash mode. In an embodiment of the present application, a plurality of original image frames (i.e., original RAW frames) may be acquired in the flash mode of the flash through an exposure control module, and the plurality of original image frames may be used as a plurality of target image frames. Optionally, the plurality of original image frames may be processed in the RAW domain to obtain a plurality of target image frames, wherein the above-mentioned RAW domain processing may be performed by the exposure control module.

[0102] In the embodiment of the present application, the first image includes at least one first image frame, and the second image includes at least one second image frame, and the first image frame and the second image frame can be original RAW frames. Exemplarily, taking the first image frame as the first original RAW frame and the second image frame as the second original RAW frame as an example, in the embodiment of the present application, at least one pair of original RAW frames must be obtained, one of which has no flash or weak flash (collectively referred to as weak flash) and the other has strong flash. In addition, the embodiment of the present application is not limited to using only one pair of original RAW frames, and multiple pairs of original RAW frames can also be used to obtain a more accurate full flash RAW image (i.e., the third image). Taking the application of a Flash mode photo-taking process as an example, the present application obtains multiple frames of original RAW frames through a Flash mode photo-taking process, applies multiple frame information, and adopts a multi-frame algorithm to obtain better image quality.

[0103] Regarding RAW domain processing, an embodiment of the present application proposes that through RAW domain processing, under the premise of absolute pixel alignment and without considering related random noise, the ratio of the RAW data of the original RAW frame can be made to have a preset relationship, such as a linear relationship, with the shutter time and related gain.

[0104] In one possible implementation, the RAW domain processing of multiple original image frames may refer to optical dark correction (OBC). In the embodiment of the present application, optical dark correction (OBC) is performed on each original RAW frame, so that under the premise of absolute pixel alignment and without considering correlated random noise, the ratio of the RAW data of each original RAW frame is linearly related to the shutter time and the related gain.

[0105] Furthermore, regarding the method of determining the first image frame and the second image frame from multiple target image frames, the first image frame and the second image frame may be determined from multiple target image frames based on flash intensity and time interval. For example, in a case where the flash intensity of the second image frame is greater than that of the first image frame, due to the existence of camera motion and motion of the photographed object, the time interval between the first image frame and the second image frame needs to meet corresponding conditions, such as a preset time interval, to ensure as much as possible that the obtained image frames are pixel-aligned.

[0106] Exemplarily, taking the flash mode as the flash light Flash photography mode (referred to as the flash light Flash mode) as an example, the flash light Flash photography mode is generally divided into three stages, namely, pre-flash, interval, and main flash, and each stage is generally on the order of 100ms or less. Among them, the camera imaging unit (i.e., the optical sensor) can generally be controlled by an exposure control module (AE). In a flash light Flash photography process, for the same target scene, multiple frames of original RAW frames can be continuously acquired, corresponding to weak flash original RAW frames, no flash original RAW frames, and strong flash original RAW frames. Exemplarily, the time interval between adjacent acquired original RAW frames can be as low as 10ms. Optionally, AE control is generally to avoid overexposure and underexposure, etc., and the shutter time (shutter) and related gain (including sensor gain) corresponding to each frame of the weak flash original RAW frame, the no flash original RAW frame, and the strong flash original RAW frame are different. It should be noted that the embodiment of the present application does not specifically limit the AE control and original RAW frame acquisition of a flash light Flash photography process.

[0107] In a flash mode photography process, at least one original RAW frame without flash or weak flash (first original RAW frame) and one original RAW frame with strong flash (second original RAW frame) are obtained, and due to the movement of the camera and the movement of the photographed object, the time interval between the two frames should be as short as possible (for example, controlled at the order of 10ms) to ensure that the obtained RAW frame image is pixel-aligned. It can be understood that the present application outputs the relevant original RAW frames obtained in a flash mode photography, including but not limited to the above-mentioned first original RAW frame and second original RAW frame.

[0108] In one embodiment, the time interval between the first image frame and the second image frame is less than a preset time interval.

[0109] Specifically, in the embodiment of the present application, the time interval between the first image frame and the second image frame is less than a preset time interval, and the preset time interval can be used to ensure that the time interval between the two frames is as short as possible to ensure that the obtained RAW frame image is pixel-aligned. Exemplarily, the preset time interval can be controlled at the order of 10ms.

[0110] It can be understood that the time interval between the first image frame and the second image frame is less than the preset time interval, but due to the objective existence of camera motion and subject motion, there will still be a certain degree of global or local pixel offset (i.e., pixel misalignment). In this regard, in one embodiment, Figure 3 As shown, step 204 includes steps 302 to 310. Among them:

[0111] Step 302: align pixels of the first image frame with the second image frame to obtain an aligned image frame.

[0112] Specifically, the embodiment of the present application performs pixel alignment on the first image frame to the second image frame to avoid pixel offset. Pixel alignment can be pixel-by-pixel alignment, and the embodiment of the present application does not specifically limit or describe the specific pixel alignment process, including but not limited to algorithms such as optical flow alignment and feature point alignment.

[0113] Taking the first image frame as the first original RAW frame (ori_raw_1), the second image frame as the second original RAW frame (ori_raw_2), and the preset time interval controlled at the order of 10ms as an example, although the time interval is as low as 10ms, due to the objective existence of camera motion and subject motion, there is still a certain degree of global or local pixel offset (i.e., pixel misalignment). For this, the embodiment of the present application proposes to perform pixel alignment processing on the first original RAW frame (ori_raw_1) to the second original RAW frame (ori_raw_2) to obtain a pixel-aligned first RAW frame (i.e., aligned image frame), and the pixel-aligned first RAW frame can be pixel_align_raw_1.

[0114] Step 304 , performing pixel gain processing on the aligned image frame to obtain a gain aligned image frame.

[0115] Specifically, when the aligned image frame is obtained, the aligned image frame may be pixel aligned, for example, alignment gain processing may be performed pixel by pixel. Exemplarily, the aligned image frame may be pixel by pixel gain processed to obtain a gain aligned image frame.

[0116] Taking the aligned image frame as the pixel-aligned first RAW frame (pixel_align_raw_1) as an example, alignment gain processing can be performed pixel by pixel to obtain the gain-aligned and pixel-aligned first RAW frame (gain_pixel_align_raw_1), that is, the gain-aligned image frame.

[0117] In some embodiments, the gain parameters of the pixel gain processing are related to the total gain coefficient of the second image frame and the total gain coefficient of the first image frame; the total gain coefficient of the second image frame is related to the shutter time for capturing the second image frame, and the total gain coefficient of the first image frame is related to the shutter time for capturing the first image frame.

[0118] Specifically, in the embodiment of the present application, the gain parameter of the pixel gain processing is related to the total gain coefficient of the second image frame and the total gain coefficient of the first image frame, wherein the total gain coefficient of the second image frame is related to the shutter time for shooting the second image frame, and the total gain coefficient of the first image frame is related to the shutter time for shooting the first image frame. Exemplarily, the gain parameter of the pixel gain processing in the present application may be the ratio of the total gain coefficient of the second image frame to the total gain coefficient of the first image frame, wherein the total gain coefficient may be the product of the shutter time and the relevant gain.

[0119] Take the first image frame as the first original RAW frame (ori_raw_1), the second image frame as the second original RAW frame (ori_raw_2), the aligned image frame as the pixel-aligned first RAW frame (pixel_align_raw_1), and the gain-aligned image frame as the gain-aligned and pixel-aligned first RAW frame (gain_pixel_align_raw_1) as an example, let the shutter time (shutter) of the first original RAW frame (ori_raw_1) be shutter_1, and the related gain (including sensor gain) be gain_1, then calculate its total gain coefficient: Total_gain_parameter_1= shutter_1*gain_1.

[0120] Similarly, the total gain coefficient of the second original RAW frame (ori_raw_2) is calculated as: Total_gain_parameter_2=shutter_2*gain_2. And generally speaking, in order to avoid overexposure, Total_gain_parameter_2 will be smaller than Total_gain_parameter_1 in AE.

[0121] Further, the pixel-aligned first RAW frame (pixel_align_raw_1) is subjected to pixel-by-pixel alignment gain processing to obtain the gain-aligned and pixel-aligned first RAW frame (gain_pixel_align_raw_1), wherein: .

[0122] Step 306: Obtain a difference image frame according to the second image frame and the gain alignment image frame.

[0123] Specifically, a difference image frame can be obtained according to the second image frame and the gain-aligned image frame, for example. The second image frame and the gain-aligned image frame are subtracted to obtain a difference image frame. Taking the second image frame as the second original RAW frame (ori_raw_2) and the gain-aligned image frame as the gain-aligned and pixel-aligned first RAW frame (gain_pixel_align_raw_1) as an example, the difference RAW image, i.e., the difference image frame, is obtained by subtracting the gain-aligned and pixel-aligned first RAW frame (gain_pixel_align_raw_1) from the second original RAW frame.

[0124] Exemplarily, the second image frame and the gain aligned image frame may be subtracted pixel by pixel to obtain a difference image frame. That is, the above subtraction operation may be pixel by pixel processing, wherein, theoretically, after the subtraction, the RAW data of any pixel position is greater than 0, and then the difference RAW image may be understood as a RAW image "generated entirely by flash light and excluding the original ambient light" (only_include_flash_raw), that is, only_include_flash_raw=ori_raw_2-gain_pixel_align_raw_1.

[0125] Step 308: filter the difference image frame to obtain a third image frame.

[0126] Specifically, since pixels cannot be completely aligned, there is random noise and other reasons, including but not limited to the resolution not needing to be pixel-by-pixel, the embodiment of the present application proposes filtering the difference image frame to obtain a third image frame.

[0127] Taking the difference image frame as a RAW image (only_include_flash_raw) as an example, the only_include_flash_raw image can be filtered to obtain a complete flash RAW frame (the third RAW frame, Raw_3), that is, the third image frame. It can be understood that the present application embodiment does not specifically limit and illustrate the filtering process, and can use including but not limited to mean filtering, Gaussian filtering, guided filtering, etc.

[0128] Among them, regarding outputting the complete flash RAW frame (the third RAW frame), let the flash power value of the second original RAW frame be strong_flash_power, and the flash power value of the first original RAW frame be weak_flash_power, then the flash power value corresponding to the third RAW frame is diff_flash_power, wherein diff_flash_power=strong_flash_power-weak_flash_power.

[0129] Step 310: Obtain a third image based on the third image frame.

[0130] Specifically, the third image can be obtained based on the third image frame. According to the number of the third image frames, different methods can be used to obtain a more accurate third image, such as a more accurate full flash RAW image.

[0131] It can be understood that the number of the third image frames depends on the number of the first image frames and the second image frames. When the first image frame and the second image frame are a pair of original RAW frames, and one of the frames has no flash or weak flash and the other has strong flash, the number of the third image frame is one. For the first image frame and the second image frame, the present application is not limited to using only one pair of original RAW frames, and multiple pairs of original RAW frames can also be used. In this case, the number of the third image frames is multiple.

[0132] In some embodiments, obtaining a third image based on the third image frame includes:

[0133] When the number of the third image frame is one, using the third image frame as the third image;

[0134] When there are multiple third image frames, the multiple third image frames are averaged to obtain the third image.

[0135] Specifically, when the number of the third image frame is one, the third image frame can be used as the third image. When the number of the third image frames is multiple, the multiple third image frames can be averaged to obtain the third image, that is, when multiple pairs of original RAW frames are used as the first image frame and the second image frame, the full flash RAW images calculated from the multiple pairs of original RAW frames can be averaged, thereby obtaining a more accurate full flash RAW image.

[0136] Regarding the acquisition of the target enhanced image, in an exemplary embodiment, as Figure 4 As shown, step 206 includes steps 402 to 404. Among them:

[0137] Step 402: Determine a first gain coefficient and a second gain coefficient according to shooting scene information.

[0138] Specifically, the embodiment of the present application can determine the first gain coefficient and the second gain coefficient according to the shooting scene information. The shooting scene information can be related to the target scene when the flash is used. For example, the shooting scene information can also be related to user requirements, such as photography style requirements.

[0139] Optionally, the flash power value and the above-mentioned shooting scene information may be input into the flash enhancement control module to determine the first gain coefficient and the second gain coefficient. The embodiment of the present application does not specifically limit more input conditions of the flash enhancement control module.

[0140] Step 404: Perform gain processing on the second image using the first gain coefficient to obtain a first gain image, and perform gain processing on the third image using the second gain coefficient to obtain a second gain image.

[0141] Specifically, the first gain image can be obtained by performing gain processing on the second image using the first gain coefficient, and the second gain image can be obtained by performing gain processing on the third image using the second gain coefficient. It can be understood that the above gain processing can be digital gain processing.

[0142] As described above, the embodiment of the present application determines the first gain coefficient and the second gain coefficient according to the shooting scene information, so that the software simulation of the flash power can be clearly controlled according to user needs, and direct calculation of factors such as "depth of foreground and background" and "light and shadow caused by the flash" can be avoided to obtain a more realistic effect.

[0143] Regarding user needs, the following embodiments are described exemplarily according to user needs of "software simulating higher flash power effect" or "higher foreground and background light ratio contrast effect". In an exemplary embodiment, the shooting scene information may include scene semantic information, such as Figure 5 As shown, step 402 includes steps 502 to 506. Among them:

[0144] Step 502: Determine the foreground subject brightness coefficient and the background brightness coefficient according to the scene semantic information.

[0145] Specifically, as an input condition of the flash enhancement control module, the shooting scene information may include scene semantic information. Optionally, the scene semantic information may include semantic information for describing scenes such as sunset, indoor restaurant lights, etc.

[0146] Among them, user needs can be determined based on scene semantic information, such as user needs for "foreground and background highlight ratio" and "background brightness". Exemplarily, the embodiment of the present application can characterize the user's needs for "foreground and background highlight ratio" and "background brightness" through the foreground subject brightness coefficient and the background brightness coefficient. For example, the background brightness coefficient can represent the brightness coefficient (background_rate_target) of the user's demand for "background brightness", and based on the foreground subject brightness coefficient and the background brightness coefficient, the ratio (portrait_background_target) of the demand for "foreground and background highlight ratio" can be determined.

[0147] It should be noted that the embodiments of the present application do not limit the method for obtaining the foreground subject brightness coefficient, the background brightness coefficient, the foreground and background highlight ratio, and the background brightness, and are not limited to the forms mentioned in the above embodiments, as long as they can represent user needs and / or scene requirements.

[0148] Step 504, obtaining a first gain coefficient based on the first brightness coefficient and the background brightness coefficient; the first brightness coefficient represents the brightness coefficient of the flash light to the background area.

[0149] Specifically, the first brightness coefficient may represent the brightness coefficient of the flash to the background area, that is, the brightness coefficient of the flash to the background. The scene area in the image in the embodiment of the present application may include a foreground area and a background area. In actual applications, when a user takes a photo through an electronic device, the framing range often includes the photographed subject and the background, wherein the above-mentioned photographed subject (foreground subject) may be the foreground area (referred to as the foreground), and the background may be the background area (referred to as the background). Optionally, taking a portrait shooting scene as an example, for a portrait shooting scene, the foreground subject may be set as a portrait mask area (mask), and the background area may be set as a non-portrait mask area (the area other than the portrait mask area).

[0150] Further, the embodiment of the present application can obtain a first gain coefficient based on the first brightness coefficient and the background brightness coefficient. For example, the first gain coefficient can be the ratio of the background brightness coefficient to the first brightness coefficient. It should be noted that the first brightness coefficient can be related to the second image, such as the brightness coefficient of the flash corresponding to the second image to the background (ori_raw_2_background_brightness_rate).

[0151] Step 506, obtaining a second gain coefficient based on the second brightness coefficient, the first gain coefficient, the foreground subject brightness coefficient and the background brightness coefficient; the second brightness coefficient represents the brightness coefficient of the flash to the foreground area.

[0152] Specifically, the second brightness coefficient may represent a brightness coefficient of the flash to the foreground area, such as a brightness coefficient of the flash to the foreground subject. Exemplarily, the second brightness coefficient may be related to the second image and the third image, such as a brightness coefficient of the flash to the foreground subject corresponding to the second image (ori_raw_2_portrait_brightness_rate), and a brightness coefficient of the flash to the foreground subject corresponding to the third image (raw_3_portrait_brightness_rate).

[0153] Further, the second gain coefficient can be obtained based on the second brightness coefficient, the first gain coefficient, the foreground subject brightness coefficient and the background brightness coefficient. It can be understood that the first gain coefficient and the second gain coefficient can also be in other forms, not limited to the forms mentioned in the above embodiment, as long as they can achieve a more realistic enhancement effect.

[0154] In the above, based on the foreground subject brightness coefficient, the background brightness coefficient, the brightness coefficient of the flash to the foreground subject, and the brightness coefficient of the flash to the background, the first gain coefficient and the second gain coefficient are determined, so that the embodiment of the present application can clearly control higher flash power according to user needs, and can accurately distinguish between "foreground and background depth", "flash shadow", etc., to obtain a more realistic post-processing enhancement effect.

[0155] In one embodiment, the first brightness coefficient may include a brightness coefficient of a background area of ​​the second image; the second brightness coefficient may include a brightness coefficient of a foreground area of ​​the second image and a brightness coefficient of a foreground area of ​​the third image; the method may further include:

[0156] According to the second image and the third image, a target image including scene light source information is acquired; the scene light source information indicates that scene light other than the flash is used as the light source;

[0157] Among them, the brightness coefficient of the background area of ​​the second image is determined based on the average brightness of the background area of ​​the second image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the background area of ​​the second image is positively correlated with the average brightness of the background area of ​​the second image and negatively correlated with the average brightness of the foreground area of ​​the target image; the brightness coefficient of the foreground area of ​​the second image is determined based on the average brightness of the foreground area of ​​the second image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the foreground area of ​​the second image is positively correlated with the average brightness of the foreground area of ​​the second image and negatively correlated with the average brightness of the foreground area of ​​the target image; the brightness coefficient of the foreground area of ​​the third image is determined based on the average brightness of the foreground area of ​​the third image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the foreground area of ​​the third image is positively correlated with the average brightness of the foreground area of ​​the third image and negatively correlated with the average brightness of the foreground area of ​​the target image.

[0158] Specifically, the flash enhancement control module can calculate the foreground subject brightness coefficient, the background brightness coefficient, the brightness coefficient of the flash to the foreground subject (such as the second brightness coefficient) and the brightness coefficient of the flash to the background (such as the first brightness coefficient) in this photo shooting based on the second image and the third image.

[0159] Among them, the first brightness coefficient may include the brightness coefficient of the background area of ​​the second image, and the brightness coefficient of the background area of ​​the second image may refer to the brightness coefficient of the flash corresponding to the second image for the background (ori_raw_2_background_brightness_rate); further, the second brightness coefficient may include the brightness coefficient of the foreground area of ​​the second image and the brightness coefficient of the foreground area of ​​the third image, wherein the brightness coefficient of the foreground area of ​​the second image may refer to the brightness coefficient of the flash corresponding to the second image for the foreground subject (ori_raw_2_portrait_brightness_rate), and the brightness coefficient of the foreground area of ​​the third image may refer to the brightness coefficient of the flash corresponding to the third image for the foreground subject (raw_3_portrait_brightness_rate).

[0160] In order to obtain the above-mentioned first brightness coefficient and second brightness coefficient, it is necessary to obtain a target image. For example, the embodiment of the present application can obtain a target image containing scene light source information based on the second image and the third image. The scene light source information indicates that the scene light other than the flash is used as the light source, that is, the target image can be understood as an image "generated by the original scene light when the flash is turned off". When the target image is obtained, the average brightness of the foreground area of ​​the target image and the average brightness of the background area of ​​the target image can be obtained, laying a foundation for the subsequent acquisition of relevant brightness coefficients.

[0161] Further, the brightness coefficient of the background area of ​​the second image is determined based on the average brightness of the background area of ​​the second image and the average brightness of the foreground area of ​​the target image. The brightness coefficient of the background area of ​​the second image is positively correlated with the average brightness of the background area of ​​the second image and negatively correlated with the average brightness of the foreground area of ​​the target image. In some embodiments, the brightness coefficient of the background area of ​​the second image includes but is not limited to the ratio of the average brightness of the background area of ​​the second image to the average brightness of the foreground area of ​​the target image.

[0162] The brightness coefficient of the foreground area of ​​the second image is determined based on the average brightness of the foreground area of ​​the second image and the average brightness of the foreground area of ​​the target image. The brightness coefficient of the foreground area of ​​the second image is positively correlated with the average brightness of the foreground area of ​​the second image and negatively correlated with the average brightness of the foreground area of ​​the target image. In one possible implementation, the brightness coefficient of the foreground area of ​​the second image includes but is not limited to the ratio of the average brightness of the foreground area of ​​the second image to the average brightness of the foreground area of ​​the target image.

[0163] In addition, the brightness coefficient of the foreground area of ​​the third image is determined based on the average brightness of the foreground area of ​​the third image and the average brightness of the foreground area of ​​the target image. The brightness coefficient of the foreground area of ​​the third image is positively correlated with the average brightness of the foreground area of ​​the third image and negatively correlated with the average brightness of the foreground area of ​​the target image. In one embodiment, the brightness coefficient of the foreground area of ​​the third image includes but is not limited to the ratio of the average brightness of the foreground area of ​​the third image to the average brightness of the foreground area of ​​the target image.

[0164] Taking the second image using the second original RAW frame (ori_raw_2) and the third image using the third RAW frame (Raw_3) as an example, according to the flash power value strong_flash_power of the second original RAW frame (ori_raw_2) and the flash power value diff_flash_power corresponding to the third RAW frame (Raw_3), the RAW image (close_flash_raw) of "flash off state, generated by the original light of the scene", that is, the target image, can be calculated.

[0165] Furthermore, the average brightness close_flash_raw_portrait_brightness (i.e., the average brightness of the foreground area of ​​the target image) can be calculated for the foreground area (e.g., the portrait mask area) of the target image (close_flash_raw), and the average brightness close_flash_raw_background_brightness (i.e., the average brightness of the background area of ​​the target image) can be calculated for the background area (e.g., the non-portrait mask area) of the target image (close_flash_raw).

[0166] The method of obtaining the brightness coefficient ori_raw_2_portrait_brightness_rate of the flash light corresponding to the second original RAW frame (ori_raw_2) with a flash power value of strong_flash_power to the foreground subject is to first calculate the average brightness ori_raw_2_portrait_brightness of the foreground area (such as the portrait mask area) of the second original RAW frame, and then calculate the ratio with close_flash_raw_portrait_brightness, that is:

[0167] .

[0168] The method of obtaining the brightness coefficient ori_raw_2_background_brightness_rate of the background corresponding to the flash power value strong_flash_power of the second original RAW frame (ori_raw_2) can be to first calculate the average brightness ori_raw_2_background_brightness of the background area (such as the non-portrait mask area) of the second original RAW frame, and then calculate the ratio with close_flash_raw_portrait_brightness, that is:

[0169] .

[0170] The method for obtaining the brightness coefficient raw_3_portrait_brightness_rate of the flash light corresponding to the flash power value diff_flash_power for the foreground subject of the third RAW frame (Raw_3) is also to first calculate the average brightness raw_3_portrait_brightness of the foreground area (such as the portrait mask area) of the third RAW frame, and then calculate the ratio with close_flash_raw_portrait_brightness, that is:

[0171] .

[0172] It can be understood that the embodiments of the present application do not limit the method for obtaining the above-mentioned brightness coefficient and the method for calculating the average brightness, and are not limited to the forms mentioned in the above embodiments, as long as the relevant brightness coefficient can be obtained.

[0173] When the above brightness coefficients are obtained, the gain coefficient ori_raw_2_gain (i.e., the first gain coefficient) of the second original RAW frame (ori_raw_2) and the gain coefficient raw_3_gain (i.e., the second gain coefficient) of the third RAW frame (Raw_3) can be calculated according to the user's requirements for the "highlight ratio of foreground and background" and "brightness of background". For example, the brightness coefficient required by the user for "brightness of background" is background_rate_target, and the ratio required for the "highlight ratio of foreground and background" is portrait_background_target. By summing the gain weights, the flash-enhanced RAW image (the fourth RAW image, Raw_4) can include:

[0174] Based on the following formula, the first gain coefficient (ori_raw_2_gain) is obtained:

[0175] ;

[0176] The second gain factor (raw_3_gain) is based on the following formula:

[0177] ;

[0178] Based on the following formula, the fourth RAW image (Raw_4) is obtained: .

[0179] Thereafter, the fourth RAW image containing the flash enhancement is processed through a subsequent image processing link (ISP) to obtain a flash enhancement image (ie, a target enhancement image).

[0180] In some embodiments, the shooting scene information may further include semantic mask information; and the method may further include:

[0181] According to the semantic mask information, the foreground area and the background area in the second image, the foreground area and the background area in the third image, and the foreground area and the background area in the target image are identified.

[0182] Specifically, the embodiments of the present application can distinguish the foreground area and the background area through semantic mask information. For example, the foreground area and the background area in the second image, the foreground area and the background area in the third image, and the foreground area and the background area in the target image can be identified based on the semantic mask information.

[0183] It is understandable that the present application does not impose any specific restrictions on the semantic mask information of the foreground subject and the background. Taking the flash photography mode as an example, the flash enhancement control module can input the flash power value in this flash photography, the semantic mask information of the foreground subject and the background, and the distance of the foreground subject. It should be noted that the present application embodiment does not impose any specific restrictions on the calculation of the above-mentioned input conditions.

[0184] Regarding acquisition of the target image, in one embodiment, acquiring the target image including scene light source information according to the second image and the third image includes:

[0185] Performing filtering on the second image to obtain a filtered image;

[0186] A target image is obtained according to the filtered image, the third image and the target coefficient; wherein the target coefficient is determined based on the flash power value of the second image and the flash power value of the third image, and the target coefficient is positively correlated with the flash power value of the second image and negatively correlated with the flash power value of the third image.

[0187] Specifically, the second image may be filtered to obtain a filtered image, wherein the filtering may be low-frequency filtering. Since high-frequency information between two frames cannot be aligned, the present application uses filtering to align the low-frequency filtered image (i.e., the filtered image) with the third image.

[0188] Furthermore, the target image can be obtained according to the filtered image, the third image and the target coefficient, wherein the target coefficient is determined based on the flash power value of the second image and the flash power value of the third image, and the target coefficient is positively correlated with the flash power value of the second image and negatively correlated with the flash power value of the third image. Optionally, the target coefficient includes but is not limited to the ratio of the flash power value of the second image to the flash power value of the third image; in some embodiments, the target image can be obtained by subtracting the filtered image from the product of the third image and the target coefficient.

[0189] For example, taking the second image using the second original RAW frame (ori_raw_2) and the third image using the third RAW frame (Raw_3) as an example, according to the flash power value of the second original RAW frame (ori_raw_2) being strong_flash_power and the flash power value corresponding to the third RAW frame (Raw_3) being diff_flash_power, the Raw image (close_flash_raw) of "flash off state, generated by the original light of the scene", i.e., the target image, can be calculated based on the following formula:

[0190] ; Among them, filter (ori_raw_2) represents the filtered image.

[0191] In some embodiments, for the average brightness close_flash_raw_portrait_brightness of the foreground area (eg, the portrait mask area) of the target image (close_flash_raw), a brightness coefficient close_flash_raw_portrait_brightness_rate of close_flash_raw_portrait_brightness may be set to 1.0.

[0192] For the average brightness close_flash_raw_background_brightnes of the background area (such as the non-portrait mask area) of the target image (close_flash_raw), the brightness coefficient close_flash_raw_background_brightness_rate corresponding to close_flash_raw_background_brightnes can be confirmed by the following formula:

[0193] .

[0194] In practical applications, in the process of acquiring the flash enhanced RAW image (the fourth RAW image, Raw_4), the ratio of close_flash_raw_portrait_brightness_rate to close_flash_raw_background_brightness_rate can be determined first. If the ratio is > 0.5 (which can be understood as a hyperparameter of the embodiment of the present application, without specific limitation), it means that the foreground subject (such as the portrait subject) and the background in the scene are at the same depth, and the light ratio cannot be changed, and the processing ends. If the ratio is < 0.5, continue. Usually, the ratio is close to 0.0. In order to simplify the description, the embodiment of the present application can assume that the above ratio is close to 0.0, and then use the above formula to acquire the fourth RAW image.

[0195] The above image processing method has clear control over the software simulation of flash power according to user needs, and avoids direct calculation of "depth of foreground and background" and "light and shadow caused by flash", so as to obtain a more realistic effect. The software process of the embodiment of the present application can be well embedded in the ISP process, which is highly practical.

[0196] In order to further explain the present application scheme, a specific example is given below. Figure 6 As shown, the image processing method is applied to a flash enhancement module as an example for explanation. The flash enhancement module may include a first flash enhancement module and a second flash enhancement module. The first flash enhancement module may be used to obtain a first image and a second image from an exposure control module, wherein the flash intensity of the second image may be greater than that of the first image. Then, the first flash enhancement module obtains a third image containing single light source information based on the first image and the second image. The second flash enhancement module may be used to perform gain processing on the second image to obtain a first gain image, and perform gain processing on the third image to obtain a second gain image, and finally obtain a target enhanced image based on the first gain image and the second gain image.

[0197] For more information about the ISP embedding process and the specific operations of the above modules, please refer to Figure 7 ,like Figure 7 As shown, taking the flash light Flash photography mode as an example, for the flash light Flash photography mode, through the exposure control module (AE), at least one pair of original RAW frames with a time interval as low as 10ms are obtained from the optical sensor, one of which is no flash or weak flash (the first original RAW frame), and the other is strong flash (the second original RAW frame). The first original RAW frame is pixel-aligned and gain-aligned to the second original RAW frame, and then subtracted and filtered to obtain a RAW image (the third RAW frame) that is "completely generated by flash light illumination and does not contain other light sources". Among them, the flash enhancement module can calculate the foreground subject brightness coefficient, the background brightness coefficient, the flash light brightness coefficient for the foreground subject, and the flash light brightness coefficient for the background, etc. according to the input strong flash frame and the mask information of the foreground subject and the background, and then calculate the respective gains of the full flash RAW and the strong flash RAW according to the user's requirements such as "foreground brightness" and "foreground subject brightness coefficient", and sum them to obtain an image frame containing flash enhancement, and then after subsequent ISP processing, a flash enhanced image (which can be an RGB image or a YUV image) is obtained.

[0198] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0199] Based on the same inventive concept, the embodiment of the present application also provides an image processing device for implementing the above-mentioned image processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations on the image processing method above, and will not be repeated here.

[0200] In an exemplary embodiment, Figure 8 As shown, an image processing device is provided, comprising:

[0201] The flash image acquisition module 901 is used to acquire a first image and a second image based on the same target scene; wherein the second image and the first image are shot with different flash intensities;

[0202] An image acquisition module 902 is used to acquire a third image including single light source information according to the first image and the second image; the single light source information indicates that a flash is used as a single light source;

[0203] A gain module 903, configured to perform gain processing on the second image to obtain a first gain image, and perform gain processing on the third image to obtain a second gain image;

[0204] The image acquisition module 904 is used to obtain a target enhanced image based on the first gain image and the second gain image.

[0205] In one embodiment, the first image includes at least one first image frame, and the second image includes at least one second image frame; the flash image acquisition module 901 is used to determine the first image frame and the second image frame from a plurality of target image frames; the plurality of target image frames are a plurality of original image frames obtained in the flashing mode of the flash light.

[0206] In one embodiment, the time interval between the first image frame and the second image frame is less than a preset time interval.

[0207] In one embodiment, the image acquisition module 902 includes:

[0208] An alignment module, used for performing pixel alignment of the first image frame with the second image frame to obtain an aligned image frame;

[0209] A pixel-by-pixel gain module, used for performing pixel gain processing on the aligned image frame to obtain a gain-aligned image frame;

[0210] A subtraction module, used for obtaining a difference image frame according to the second image frame and the gain alignment image frame;

[0211] A filtering module, used for filtering the difference image frame to obtain a third image frame;

[0212] The image acquisition module is used to obtain a third image based on the third image frame.

[0213] In one of the embodiments, the image acquisition module is used to use the third image frame as the third image when the number of the third image frame is one; and to average the plurality of third image frames to obtain the third image when the number of the third image frames is multiple.

[0214] In one embodiment, the gain parameter of the pixel gain processing is related to the total gain coefficient of the second image frame and the total gain coefficient of the first image frame; the total gain coefficient of the second image frame is related to the shutter time for capturing the second image frame, and the total gain coefficient of the first image frame is related to the shutter time for capturing the first image frame.

[0215] In one embodiment, the gain module 903 includes:

[0216] A gain coefficient determination module, used to determine a first gain coefficient and a second gain coefficient according to shooting scene information;

[0217] The gain processing module is used to perform gain processing on the second image by using a first gain coefficient to obtain a first gain image, and to perform gain processing on the third image by using a second gain coefficient to obtain a second gain image.

[0218] In one of the embodiments, the shooting scene information includes scene semantic information; a gain processing module is used to determine the foreground subject brightness coefficient and the background brightness coefficient according to the scene semantic information; a first gain coefficient is obtained based on the first brightness coefficient and the background brightness coefficient; the first brightness coefficient represents the brightness coefficient of the flash to the background area; and a second gain coefficient is obtained based on the second brightness coefficient, the first gain coefficient, the foreground subject brightness coefficient and the background brightness coefficient; the second brightness coefficient represents the brightness coefficient of the flash to the foreground area.

[0219] In one embodiment, the first brightness coefficient includes the brightness coefficient of the background area of ​​the second image; the second brightness coefficient includes the brightness coefficient of the foreground area of ​​the second image and the brightness coefficient of the foreground area of ​​the third image; the device further includes:

[0220] A target image acquisition module, used to acquire a target image including scene light source information according to the second image and the third image; the scene light source information indicates that scene light other than the flash is used as the light source;

[0221] Among them, the brightness coefficient of the background area of ​​the second image is determined based on the average brightness of the background area of ​​the second image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the background area of ​​the second image is positively correlated with the average brightness of the background area of ​​the second image and negatively correlated with the average brightness of the foreground area of ​​the target image; the brightness coefficient of the foreground area of ​​the second image is determined based on the average brightness of the foreground area of ​​the second image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the foreground area of ​​the second image is positively correlated with the average brightness of the foreground area of ​​the second image and negatively correlated with the average brightness of the foreground area of ​​the target image; the brightness coefficient of the foreground area of ​​the third image is determined based on the average brightness of the foreground area of ​​the third image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the foreground area of ​​the third image is positively correlated with the average brightness of the foreground area of ​​the third image and negatively correlated with the average brightness of the foreground area of ​​the target image.

[0222] In one embodiment, the shooting scene information further includes semantic mask information; and the device further includes:

[0223] The region recognition module is used to recognize the foreground region and the background region in the second image, the foreground region and the background region in the third image, and the foreground region and the background region in the target image according to the semantic mask information.

[0224] In one embodiment, the target image acquisition module is used to filter the second image to obtain a filtered image; and obtain the target image based on the filtered image, the third image and the target coefficient; wherein the target coefficient is determined based on the flash power value of the second image and the flash power value of the third image, and the target coefficient is positively correlated with the flash power value of the second image and negatively correlated with the flash power value of the third image.

[0225] Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in an electronic device in the form of hardware, or can be stored in a memory in an electronic device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0226] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0227] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0228] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0229] In one embodiment, the electronic device in the embodiment of the present application may be a device with a camera and a flash shooting function.

[0230] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0231] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0232] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0233] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0234] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0235] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, applied to an electronic device with a camera and a flash shooting function, characterized in that: The method comprises: Acquire a first image and a second image of the same target scene; wherein the second image and the first image are shot with different flash intensities; Acquire a third image including single light source information according to the first image and the second image; the single light source information indicates that the flashlight is used as a single light source; Performing gain processing on the second image to obtain a first gain image, and performing gain processing on the third image to obtain a second gain image; A target enhanced image is obtained based on the first gain image and the second gain image.

2. The method according to claim 1, characterized in that The first image includes at least one first image frame, and the second image includes at least one second image frame; Acquiring a first image and a second image of the same target scene, including: Determine the first image frame and the second image frame from a plurality of target image frames; The multiple target image frames are multiple original image frames obtained in the flash mode of the flash light.

3. The method according to claim 2, characterized in that The time interval between the first image frame and the second image frame is less than a preset time interval.

4. The method according to claim 2, characterized in that: Acquiring a third image containing single light source information according to the first image and the second image, comprising: Perform pixel alignment on the first image frame and the second image frame to obtain an aligned image frame; Performing pixel gain processing on the aligned image frame to obtain a gain aligned image frame; Obtaining a difference image frame according to the second image frame and the gain alignment image frame; Performing filtering processing on the difference image frame to obtain a third image frame; Based on the third image frame, the third image is obtained.

5. The method according to claim 4, characterized in that Based on the third image frame, obtaining the third image includes: When the number of the third image frame is one, using the third image frame as the third image; When there are multiple third image frames, the multiple third image frames are averaged to obtain the third image.

6. The method according to claim 4, characterized in that The gain parameters of the pixel gain processing are related to the total gain coefficient of the second image frame and the total gain coefficient of the first image frame; the total gain coefficient of the second image frame is related to the shutter time for shooting the second image frame, and the total gain coefficient of the first image frame is related to the shutter time for shooting the first image frame.

7. The method according to any one of claims 1 to 6, characterized in that: Performing gain processing on the second image to obtain a first gain image, and performing gain processing on the third image to obtain a second gain image, comprising: Determining a first gain coefficient and a second gain coefficient according to the shooting scene information; The first gain image is obtained by performing gain processing on the second image using the first gain coefficient, and the second gain image is obtained by performing gain processing on the third image using the second gain coefficient.

8. The method according to claim 7, characterized in that The shooting scene information includes scene semantic information; determining the first gain coefficient and the second gain coefficient according to the shooting scene information includes: Determining a foreground subject brightness coefficient and a background brightness coefficient according to the scene semantic information; Based on the first brightness coefficient and the background brightness coefficient, the first gain coefficient is obtained; the first brightness coefficient represents the brightness coefficient of the flash light to the background area; The second gain coefficient is obtained based on the second brightness coefficient, the first gain coefficient, the foreground subject brightness coefficient and the background brightness coefficient; the second brightness coefficient represents the brightness coefficient of the flash to the foreground area.

9. The method according to claim 8, characterized in that The first brightness coefficient includes the brightness coefficient of the background area of ​​the second image; the second brightness coefficient includes the brightness coefficient of the foreground area of ​​the second image and the brightness coefficient of the foreground area of ​​the third image; the method further includes: Acquire a target image containing scene light source information according to the second image and the third image; the scene light source information indicates that scene light other than the flash light is used as a light source; The brightness coefficient of the background area of ​​the second image is determined based on the average brightness of the background area of ​​the second image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the background area of ​​the second image is positively correlated with the average brightness of the background area of ​​the second image and negatively correlated with the average brightness of the foreground area of ​​the target image; The brightness coefficient of the foreground area of ​​the second image is determined based on the average brightness of the foreground area of ​​the second image and the average brightness of the foreground area of ​​the target image, and the brightness coefficient of the foreground area of ​​the second image is positively correlated with the average brightness of the foreground area of ​​the second image and negatively correlated with the average brightness of the foreground area of ​​the target image; The brightness coefficient of the foreground area of ​​the third image is determined based on the average brightness of the foreground area of ​​the third image and the average brightness of the foreground area of ​​the target image. The brightness coefficient of the foreground area of ​​the third image is positively correlated with the average brightness of the foreground area of ​​the third image and negatively correlated with the average brightness of the foreground area of ​​the target image.

10. The method according to claim 9, characterized in that The shooting scene information also includes semantic mask information; the method also includes: According to the semantic mask information, the foreground area and the background area in the second image, the foreground area and the background area in the third image, and the foreground area and the background area in the target image are identified.

11. The method according to claim 9, characterized in that Acquiring a target image containing scene light source information according to the second image and the third image, including: Performing filtering on the second image to obtain a filtered image; Obtaining the target image according to the filtered image, the third image and the target coefficient; The target coefficient is determined based on the flash power value of the second image and the flash power value of the third image, and the target coefficient is positively correlated with the flash power value of the second image and negatively correlated with the flash power value of the third image.

12. An image processing device, applied to an electronic device having a camera and a flash shooting function, characterized in that: The device comprises: A flash image acquisition module, used to acquire a first image and a second image of the same target scene; wherein the second image and the first image are shot with different flash intensities; An image acquisition module, configured to acquire a third image including single light source information according to the first image and the second image; the single light source information indicates that the flashlight is used as a single light source; a gain module, configured to perform gain processing on the second image to obtain a first gain image, and perform gain processing on the third image to obtain a second gain image; The image acquisition module is used to obtain a target enhanced image based on the first gain image and the second gain image.

13. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images