Image processing method and device, electronic equipment and storage medium

Abstract

The application discloses an image processing method and device, electronic equipment and storage medium; the method comprises the following steps: obtaining scene information of a preview picture of a to-be-acquired scene; based on the scene information, at least three frames of to-be-processed images corresponding to different exposure parameters are acquired; the to-be-processed images comprise an alignment frame as an exposure parameter reference, a portrait recognition frame with an exposure parameter smaller than and closest to the exposure parameter corresponding to the alignment frame, and an exposure processing frame; the exposure processing frame comprises at least one underexposed frame with an exposure parameter smaller than the exposure parameter corresponding to the portrait recognition frame; the alignment frame and the exposure processing frame are fused to obtain a fused frame; the portrait position of the portrait recognition frame is processed to obtain a portrait mask image; the portrait mask image and the fused frame are compared to obtain a portrait area of the fused frame; and the portrait area is processed based on the alignment frame and the underexposed frame to obtain a target image. Through implementation of the method, the image quality of a portrait can be improved and more portrait details can be displayed when a large-contrast scene is shot.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and includes, but is not limited to, an image processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] When shooting portraits in high-contrast scenes such as backlighting, focusing on the subject can result in an overexposed background with no detail. Exposure fusion technology is used to combine the overexposed background with the underexposed darker areas to restore detail in the overexposed background. However, this introduces a problem: in high-contrast portraits, the subject may appear darker, and due to various sensor limitations, the darker areas will be darker and grayer overall after fusion. Consequently, the final portrait will be even darker and may exhibit some color cast. Summary of the Invention

[0003] In view of this, the image processing method, apparatus, electronic device and storage medium provided in the embodiments of this application are used to improve the image quality of portraits and display more portrait details.

[0004] The first aspect of this application provides an image processing method, including:

[0005] Obtain scene information from a preview of the scene to be captured;

[0006] Based on the scene information, at least three frames of images to be processed are acquired with different exposure parameters corresponding to the scene information; the images to be processed include: an alignment frame as a reference for the exposure parameters, a portrait recognition frame with an exposure parameter less than and closest to the exposure parameter corresponding to the alignment frame, and an exposure processing frame; the exposure processing frame includes at least one underexposed frame with an exposure parameter less than the exposure parameter corresponding to the portrait recognition frame.

[0007] The aligned frame and the exposure processing frame are fused together to obtain a fused frame;

[0008] The position of the human face in the human face recognition frame is processed to obtain a human face mask image;

[0009] The portrait mask image and the fused frame are compared to obtain the portrait region of the fused frame;

[0010] The image of the portrait area is processed based on the alignment frame and the underexposed frame to obtain the target image.

[0011] In this embodiment of the application, obtaining the scene information of the preview image of the scene to be captured includes:

[0012] The overexposure level of the preview image is detected; the overexposure level is calculated from the variance of the grayscale histogram of the preview image, the number of high-brightness pixels, and the total number of pixels.

[0013] The scene information is determined by comparing the degree of overexposure with a pre-set exposure threshold table; the exposure threshold table includes the correspondence between the degree of overexposure and the scene, as well as the exposure parameters selected for different scenes.

[0014] In this embodiment of the application, when the exposure processing frame is an underexposed frame, the step of fusing the alignment frame and the exposure processing frame to obtain a fused frame includes:

[0015] The aligned frame and the underexposed frame are fused together to obtain a fused frame.

[0016] In this embodiment of the application, when the exposure processing frame further includes an overexposed frame with exposure parameters higher than those corresponding to the alignment frame, the step of fusing the alignment frame and the exposure processing frame to obtain a fused frame includes:

[0017] The overexposed frame, the underexposed frame, and the aligned frame are fused together to obtain the fused frame.

[0018] In this embodiment of the application, when the exposure processing frame includes multiple underexposed frames, the step of fusing the alignment frame and the exposure processing frame to obtain a fused frame includes:

[0019] The underexposed frames from the multiple frames are fused together to obtain the target underexposed frame;

[0020] The overexposed frame, the target underexposed frame, and the aligned frame are fused together to obtain the fused frame.

[0021] In this embodiment of the application, the step of processing the portrait position of the portrait recognition frame to obtain a portrait mask image includes:

[0022] The facial recognition frame is subjected to image enhancement processing; the image enhancement processing includes increasing the brightness of the facial recognition frame;

[0023] The human face recognition frame after image enhancement processing is used to obtain a process image through a deep learning algorithm;

[0024] Based on the process image, the edges of the human face in the enhanced human face recognition frame are smoothed to obtain the human face mask image.

[0025] In this embodiment of the application, the step of processing the portrait area based on the aligned frame and the underexposed frame to obtain the target image includes:

[0026] The brightness of the portrait area is adjusted based on the alignment frame and the fusion frame.

[0027] Based on the aligned frame and the underexposed frame, the saturation of the portrait area is adjusted to obtain the target image after brightness and saturation adjustment.

[0028] In this embodiment of the application, when the exposure processing frame includes multiple underexposed frames, the step of processing the portrait area based on the alignment frame and the underexposed frames to obtain the target image includes:

[0029] The brightness of the portrait area is adjusted based on the alignment frame and the fusion frame.

[0030] Based on the alignment frame and the target underexposed frame, the saturation of the portrait area is adjusted to obtain the target image after brightness and saturation adjustment.

[0031] In this embodiment of the application, adjusting the brightness of the portrait area based on the aligned frame and the fused frame includes:

[0032] Obtain the corresponding region between the alignment frame and the portrait region;

[0033] Calculate the mean and variance of brightness for the portrait region of the fused frame and the corresponding region of the aligned frame, respectively.

[0034] The brightness of the human figure region in the fused frame is adjusted based on the mean brightness and the variance of brightness.

[0035] In this embodiment of the application, adjusting the saturation of the portrait area based on the aligned frame and the underexposed frame includes:

[0036] Obtain the corresponding region between the underexposed frame and the portrait area;

[0037] Obtain the average saturation and average brightness of the corresponding regions of the portrait area and the underexposed frame;

[0038] Based on the average saturation and average brightness of the corresponding areas of the portrait area and the underexposed frame, determine the overexposure status of the corresponding areas of the portrait area and the underexposed frame;

[0039] Based on the overexposure situation, the saturation of the portrait area is adjusted by selecting either the alignment frame or the underexposure frame.

[0040] In this embodiment of the application, adjusting the saturation of the portrait area based on the aligned frame and the target underexposed frame includes:

[0041] Obtain the corresponding region between the target underexposed frame and the portrait area;

[0042] Obtain the average saturation and average brightness of the corresponding regions of the portrait area and the underexposed target frame;

[0043] Based on the average saturation and average brightness of the corresponding regions of the portrait region and the target underexposed frame, determine the overexposure status of the corresponding regions of the portrait region and the target underexposed frame;

[0044] The saturation of the portrait area is adjusted according to the overexposure situation.

[0045] A second aspect of this application provides an image processing apparatus, comprising:

[0046] The detection module is used to detect and obtain scene information from the preview of the scene to be captured;

[0047] The acquisition module is used to acquire at least three frames of images to be processed based on the scene information and different exposure parameters corresponding to the scene information; the images to be processed include: an alignment frame as an exposure parameter reference, a portrait recognition frame with an exposure parameter less than and closest to the exposure parameter corresponding to the alignment frame, and an exposure processing frame; the exposure processing frame includes at least one underexposed frame with an exposure parameter less than the exposure parameter corresponding to the portrait recognition frame.

[0048] A fusion module is used to perform fusion processing on the aligned frame and the exposure processing frame to obtain a fused frame;

[0049] The segmentation module is used to process the human image position of the human image recognition frame to obtain a human image mask image, and compare the human image mask image with the fused frame to obtain the human image region of the fused frame;

[0050] The adjustment module is used to perform image processing on the portrait area based on the alignment frame and the underexposed frame to obtain the target image.

[0051] A third aspect of this application provides an electronic device, including:

[0052] Memory containing executable program code;

[0053] and the processor coupled to the memory;

[0054] The processor calls the executable program code stored in the memory. When the executable program code is executed by the processor, the processor implements the method provided in the embodiments of this application.

[0055] The fourth embodiment of this application provides a computer-readable storage medium storing executable program code thereon. When the executable program code is executed by a processor, it implements the method provided in the embodiment of this application.

[0056] The image processing method, apparatus, electronic device, and storage medium provided in this application embodiment obtain scene information by detecting a preview of the scene to be captured, and acquire at least three images containing an alignment frame corresponding to the preview and serving as an exposure reference based on the scene information. Using the exposure parameters of the alignment frame as a reference, the image with exposure parameters smaller than and closest to the alignment frame among the at least three acquired images is selected as the portrait recognition frame. Since the exposure parameters of the portrait alignment frame are smaller than those of the alignment frame, the overexposed area of ​​this portrait recognition frame is smaller, enabling the segmentation of a portrait mask image that is accurate and sufficient for subsequent operations. Because the exposure parameters of the portrait recognition frame and the alignment frame are close, the suppression effect on the high-exposure area is weak during fusion processing, so the portrait recognition frame does not participate in exposure fusion. Instead, an underexposed frame with an exposure parameter smaller than that of the portrait recognition frame is selected from the at least three images and subjected to exposure fusion processing with the alignment frame to obtain a fused frame. Since the portrait portion of the obtained fused frame is generally dark and needs adjustment, the portrait area of ​​the fused frame is first obtained using a portrait mask image. Then, the brightness and saturation of the portrait area of ​​the fused frame are adjusted using an alignment frame and an underexposed frame. Color restoration and brightness compensation are performed on the portrait to improve the image quality and show more portrait details. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments and the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating the image processing method disclosed in the embodiments of this application;

[0059] Figure 2 This is a flowchart illustrating the image processing method for obtaining a human face mask image disclosed in the embodiments of this application;

[0060] Figure 3 This is a flowchart illustrating an image processing method for scenario 1 in this application embodiment;

[0061] Figure 4This is a flowchart illustrating an image processing method for scenario 2 in this application embodiment;

[0062] Figure 5 This is a flowchart illustrating an image processing method for scenario 3 in an embodiment of this application;

[0063] Figure 6 This is a structural illustration of an image processing apparatus disclosed in an embodiment of this application;

[0064] Figure 7 This is a structural illustration of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0067] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0068] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0069] It is understood that the electronic devices involved in the embodiments of this application may include general handheld screen electronic user terminals, such as mobile phones, smartphones, portable terminals, terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), laptops, notebooks, wireless broadband (Wibro) terminals, tablet computers (PCs), smart PCs, point of sale (POS) terminals, and in-vehicle computers, etc.

[0070] Electronic devices can also include wearable devices. Wearable devices are portable electronic devices that can be worn directly on the user's body or integrated into the user's clothing or accessories. Wearable devices are not just hardware devices; they can also achieve powerful intelligent functions through software support, data interaction, and cloud server interaction, such as computing, positioning, and alarm functions. They can also connect to mobile phones and various terminals. Wearable devices can include, but are not limited to, wrist-supported devices (such as watches, wristbands, etc.), foot-supported devices (such as shoes, socks, or other leg-wearing products), head-supported devices (such as glasses, helmets, headbands, etc.), as well as smart clothing, backpacks, canes, accessories, and other non-mainstream product forms.

[0071] Photography is one of the ways to record beautiful moments in life. It not only preserves memories but can also be used to create works of art, disseminate information, and record history. Photography can encompass a variety of themes, such as landscapes, people, animals, and architecture. With the development of technology, photography has become more common and convenient. People can take and share photos and videos anytime, anywhere using their mobile phones, making it a hobby and career choice for many. However, when shooting portraits in high-contrast scenes such as sunlight or backlighting, focusing on the subject can result in an overexposed background with no detail. In such cases, image processing techniques are used to merge the overexposed background with the underexposed darker areas to restore detail in the overexposed background. This introduces a problem: in high-contrast portrait photography, the subject may appear darker, and due to various sensor limitations, the darker areas will appear darker and grayer overall after merging. Consequently, the merged portrait will be even darker and may exhibit some color cast.

[0072] It is evident that the image processing methods of related technologies still have shortcomings in processing portraits in high-contrast scenes.

[0073] In view of this, embodiments of this application provide an image processing method that can retain more portrait detail information in high-contrast scenes and improve the imaging effect of portraits.

[0074] The image processing method provided in this application will now be described in detail with reference to the accompanying drawings.

[0075] Please refer to Figure 1 This is a schematic diagram illustrating the implementation flow of the image processing method according to an embodiment of this application. Figure 1 As shown, the method includes:

[0076] Step 101: Obtain scene information from the preview of the scene to be captured.

[0077] In this embodiment of the application, the scene information includes the overexposure degree of the scene, which can be understood as the overexposure degree of the preview image of the scene to be captured.

[0078] For example, when an electronic device uses a camera or other imaging device to capture a specified scene, the preview screen only provides a preview of the scene to be captured and does not store the image in the electronic device's memory. The preview screen of the scene to be captured refers to the image information acquired in real-time by the electronic device through the camera or other imaging device. In some embodiments, the preview screen can be understood as the image displayed on the monitor or viewfinder before the electronic device captures the image; this image is not stored in the electronic device's memory.

[0079] The overexposure level of the preview image of the scene to be captured is determined by an overexposure judgment algorithm. This overexposure judgment algorithm can select one frame of the real-time preview image as input or multiple frames of the real-time preview image as input, and there is no limitation here.

[0080] In some embodiments, when the input to the overexposure determination algorithm is a preview frame, a grayscale histogram is calculated for the input preview frame. The grayscale histogram describes the number of pixels at each gray level in the image, reflecting the frequency of each gray level. The mean and variance of the grayscale histogram are calculated. Pixels with brightness values ​​higher than a pre-set brightness threshold are marked as high-brightness pixels. The number of high-brightness pixels and the total number of pixels in the preview frame are counted, and the proportion of high-brightness pixels to the total number of pixels is calculated. The overexposure level is obtained by multiplying the variance of the grayscale histogram by the proportion of high-brightness pixels.

[0081] In some embodiments, when the input to the overexposure determination algorithm is a multi-frame preview image, the selection of the multi-frame preview image can be either consecutive or at dynamic time intervals; this is not limited here. The number of frames in the multi-frame preview image is also not limited here. For an overexposure determination algorithm with multiple frames as input, the overexposure degree of each frame can be calculated using the same method as for an overexposure determination algorithm with only one frame, and the final overexposure degree can be obtained through the mean or maximum / minimum method.

[0082] Step 102: Based on scene information, acquire at least three frames of images to be processed with different exposure parameters corresponding to the scene information; the images to be processed include alignment frames, portrait recognition frames, and exposure processing frames; the exposure processing frames include at least one underexposed frame.

[0083] In this embodiment, the electronic device stores a pre-set exposure threshold table. This table includes the correspondence between overexposure levels and scenes, as well as the corresponding exposure parameters selected for different scenes. After obtaining scene information from a preview of the scene to be captured, the overexposure parameter included in the scene information is compared with the pre-set exposure threshold table to determine the corresponding exposure parameters for different scenes. The correspondence between overexposure levels and scenes in the exposure threshold table means that the table presets multiple different scenes, each with a corresponding exposure level range. The exposure level value calculated from the preview image is compared with the exposure level range in the exposure threshold table. When the exposure level value of the preview image falls within a certain range, it indicates that the preview image corresponds to the specified scene within that range. The number of frames in the image to be processed and the exposure parameters corresponding to each frame are determined based on the scene corresponding to the preview image.

[0084] It should be noted that this application does not restrict the method of generating the pre-set exposure threshold table. That is to say, the overexposure range of the exposure threshold table and the selection of exposure parameters under different scenarios can be confirmed by the electronic device through a large amount of experimental data and stored in the memory before leaving the factory, or can be set by the user of the electronic device according to their own needs before using the image processing method, or the electronic device can make corresponding adjustments after detecting the hardware status of the device.

[0085] In this embodiment, the exposure parameters may include, but are not limited to, a reference exposure, an exposure compensation value, and the actual exposure. The reference exposure refers to the exposure corresponding to the metering data collected by the camera's metering module. The exposure compensation value refers to the exposure increased or decreased relative to the reference exposure. The actual exposure refers to the true exposure of the electronic device when capturing the image. The actual exposure can be determined based on the reference exposure and the exposure compensation value. For example, the exposure compensation value is EV+1, where EV refers to the difference between the exposure corresponding to the metering data collected by the camera's metering module (i.e., the reference exposure) and the actual exposure. "+" indicates increased exposure, "-" indicates decreased exposure, and the value "1" indicates the level of exposure compensation. EV+1 means increasing the exposure by one stop relative to the base exposure, so the actual exposure can be twice the base exposure. The exposure compensation value can be controlled by changing the exposure time or adjusting the aperture size, and is not limited here.

[0086] In this embodiment, an alignment frame is selected as the exposure parameter reference. The exposure parameter reference refers to defining the exposure of this alignment frame as the reference exposure of the image processing method. By default, this alignment frame corresponds to the preview image of the scene to be captured in step 101. This correspondence can be understood as using the preview image as the alignment frame, selecting one from multiple preview images, or selecting the exposure parameters of the preview image as the exposure parameters of the alignment frame; no limitation is made here.

[0087] It should be noted that the image processing method provided in this application embodiment is applied to high-contrast scenes. Contrast ratio is one of the important parameters in photography, referring to the proportion of light received by the dark and bright sides of a subject under lighting conditions. The magnitude of the contrast ratio affects aspects such as image contrast, dynamic range, and detail. High-contrast scenes include landscapes under sunlight, indoor backlighting, nighttime lighting, and other scenes that create significant contrast between light and dark.

[0088] In some embodiments, although the exposure parameters of the aligned frame are used as the reference exposure, the reference exposure is not equivalent to the combination of an exposure time of 1 second and an aperture of f / 1.0 or its equivalent combination, but is used as a reference for other frames to be processed in the image processing method.

[0089] In this embodiment, after acquiring an alignment frame as a reference for exposure parameters, the image processing method provided in this embodiment also needs to select a portrait recognition frame from at least three acquired images whose exposure parameters are less than and closest to the exposure parameters corresponding to the alignment frame. Since the alignment frame has obvious overexposed areas in high-contrast scenes, selecting the alignment frame for portrait segmentation cannot obtain a portrait mask image with sufficient accuracy for subsequent operations. Therefore, in this embodiment, a portrait recognition frame with an exposure parameter less than and closest to the exposure parameters corresponding to the alignment frame is selected from at least three acquired images. The exposure compensation of the portrait recognition frame is negative, and this portrait recognition frame has fewer overexposed areas compared to the alignment frame, thereby obtaining a portrait mask image with sufficient accuracy.

[0090] It should be noted that in this embodiment, there are at least two images with exposure parameters less than the alignment frame, that is, at least two images with negative exposure compensation. The reason for not selecting images with even smaller exposure parameters is that images with smaller exposure parameters have lower brightness, specifically manifested as dark images, which results in insufficient precision of the segmented portrait mask image for subsequent operations. Therefore, this application needs to process the portrait recognition frames selected by the above method to obtain the portrait mask image. The processing method is as follows.

[0091] Step 103: Process the human image position of the human image recognition frame to obtain the human image mask image.

[0092] After the electronic device selects the portrait recognition frame with an exposure parameter smaller than and closest to the alignment frame from at least three captured images, a series of portrait segmentation processing methods are required to obtain the portrait mask image. The portrait segmentation processing methods include image enhancement processing of the portrait recognition frame, obtaining a process image of the image-enhanced portrait recognition frame through a deep learning algorithm, and smoothing the portrait edges of the image-enhanced portrait recognition frame based on the process image to obtain the portrait mask image.

[0093] In this embodiment, image enhancement processing is performed on the acquired facial recognition frames. Due to the accuracy issues during training of deep learning algorithms, gamma stretching is performed during training to darken the image and simulate a dark image when data feature changes are made. This is used to derive the model for prediction. It should be noted that although deep learning algorithms require dark images as input, the brightness value of the facial recognition frame is lower than that of the alignment frame, but still lower than the brightness requirement of the input image for deep learning algorithms. Therefore, before input, the facial recognition frames need to be data-enhanced by gamma brightening to align the image brightness with the input brightness of the model.

[0094] In this embodiment, the image-enhanced portrait recognition frame is processed using a deep learning algorithm to obtain a process image. The input portrait recognition frame, representing an instance to be segmented, is then processed by a neural network to segment the required portrait instances based on category similarities and differences. Considering the need for a smaller model size on mobile devices, a lightweight network model is generally chosen to balance accuracy and speed. Mainstream network models include MobileNetV2, MobileUNet, HRNet, and ERFNet, among others, and are not limited here. In one embodiment, the ErfNet deep learning network model is selected to perform portrait segmentation on the image-enhanced portrait recognition frame. First, the image of the image-enhanced portrait recognition frame is adjusted to a pixel size of 144*144 to align with the input requirements of the deep learning network model and to accelerate the model's computation speed. Subsequent deep learning model calculations are then performed to obtain the process image. It should be noted that there are many methods for adjusting image pixel size in deep learning, such as max pooling, average pooling, and convolutional downsampling, and are not limited here.

[0095] It should be noted that while the process image obtained through deep learning algorithms can describe the approximate location of a person, it lacks precision. In some embodiments, the pixel size of the process image obtained through deep learning algorithms may be smaller than the pixel size of the acquired image to be processed. The process image needs to be restored to its original size to obtain a suitable portrait mask image for subsequent processing. The original-sized process image obtained by this method will have a jagged edge and still requires further processing to obtain the portrait mask image. Common methods for adjusting image size include bilinear interpolation, nearest neighbor interpolation, and image pyramids, which are not limited here.

[0096] In this embodiment, the image edges of the enhanced portrait recognition frame are smoothed based on the process image to obtain the portrait mask image. Using guided filtering on the process image (resized to its original size) and the enhanced portrait recognition frame can smooth and conform to the portrait contours, resulting in the portrait mask image. The guided filtering method is an image filtering technique that uses a guide image to filter the target image (input image), making the final output image generally similar to the target image, but with textures similar to the guide image.

[0097] Step 104: Perform a fusion process on the exposure processing frame and the alignment frame to obtain a fused frame.

[0098] In this application example, the fusion process includes calculating the brightness weight and saturation weight of multiple images to be fused, calculating the fusion weight of the images to be fused when performing fusion processing based on the brightness weight and saturation weight of the multiple images to be fused, and fusion of the multiple images to be fused according to the fusion weight to obtain a fused image.

[0099] Step 105: Compare the portrait mask image and the fused frame to obtain the portrait region of the fused frame.

[0100] Step 106: Process the image of the portrait area based on the aligned frame and the underexposed frame to obtain the target image.

[0101] In some embodiments, when there is only one underexposed frame in the exposure processing frame, the brightness of the portrait area is adjusted according to the alignment frame and the fusion frame; the saturation of the portrait area is adjusted according to the alignment frame and the underexposed frame to obtain the target image after brightness and saturation adjustment.

[0102] Brightness adjustment can be achieved by obtaining the corresponding area of ​​the aligned frame and the portrait area, calculating the mean and variance of brightness of the portrait area and the corresponding area of ​​the aligned frame respectively, and adjusting the brightness of the portrait area based on the mean and variance of brightness.

[0103] Saturation adjustment can be achieved by obtaining the corresponding regions of the aligned frame and the portrait area, obtaining the corresponding regions of the underexposed frame and the portrait area, obtaining the average saturation and average brightness of the corresponding regions of the aligned frame and the underexposed frame, determining the overexposure of the portrait area and the corresponding regions of the underexposed frame based on the average saturation and average brightness of the corresponding regions of the aligned frame and the underexposed frame, and adjusting the saturation of the portrait area based on the overexposure.

[0104] In the above technical solution, by performing color restoration and brightness compensation on the portrait area of ​​the fused frame image, the overall image quality of the target image is high, and the image quality of the portrait is improved, displaying more portrait details.

[0105] In some embodiments, after obtaining the target image, the target image is also converted to an output domain before being output. This output domain can be understood as a color model, a mathematical model used to represent and describe colors. Common color models include RGB, YUV, CMYK, HSB, etc.

[0106] Please see Figure 2 This is a flowchart illustrating the image processing method for obtaining a portrait mask image disclosed in the embodiments of this application, which may include the following steps:

[0107] Step 201: Perform image enhancement processing on the collected facial recognition frames.

[0108] Step 202: The image enhancement process of the human face recognition frame is used to obtain the process image through a deep learning algorithm.

[0109] Step 203: Based on the process image, smooth the edges of the human face recognition frame after image enhancement processing to obtain the human face mask image.

[0110] The electronic device acquires at least three frames of images to be fused based on different exposure parameters. These three frames are then fused to obtain a single fused image. Compared to the original three frames, the fused image has a wider dynamic range, more balanced exposure, and preserves details in both light and dark areas.

[0111] It should be noted that, in the embodiments of this application, the number of images to be fused for exposure fusion is not limited.

[0112] In some embodiments, based on scene information, the electronic device acquires images to be processed with different exposure parameters corresponding to the scene information, which may contain more than three frames. The images to be processed may also include overexposed frames and multiple underexposed frames.

[0113] The example scenario is described as follows:

[0114] Scenario 1: The electronic device collects scene information and the images to be processed with different exposure parameters include an aligned frame, a portrait recognition frame, and an underexposed frame.

[0115] Scenario 2: The images to be processed, which are collected by electronic devices and have different exposure parameters corresponding to scene information, include an alignment frame, a portrait recognition frame, an underexposed frame, and an overexposed frame.

[0116] Scenario 3: The image to be processed, corresponding to different exposure parameters and scene information acquired by the electronic device, contains one aligned frame, one portrait recognition frame, multiple underexposed frames, and one overexposed frame. The multiple underexposed frames can be understood as the image to be processed containing at least two underexposed frames, and the specific number is not limited.

[0117] Please see Figure 3 Here is a flowchart of an image processing method for scenario 1 above, which may include the following steps:

[0118] Step 301: Obtain scene information from the preview screen of the scene to be collected.

[0119] Step 302: Based on scene information, collect aligned frames, portrait recognition frames, and underexposed frames with different exposure parameters corresponding to the scene information.

[0120] Step 303: Process the human image position in the human image recognition frame to obtain the human image mask image.

[0121] Step 304: Perform a fusion process on the underexposed frame and the aligned frame to obtain a fused frame.

[0122] Step 305: Compare the portrait mask image and the fused frame to obtain the portrait region of the fused frame.

[0123] Step 306: Process the image of the portrait area based on the aligned frame and the underexposed frame to obtain the target image.

[0124] Please see Figure 4 Here is a flowchart of an image processing method for scenario 2 above, which may include the following steps:

[0125] Step 401: Obtain scene information from the preview screen of the scene to be collected.

[0126] Step 402: Based on scene information, collect aligned frames, portrait recognition frames, underexposed frames, and overexposed frames with different exposure parameters corresponding to the scene information.

[0127] In some embodiments, the electronic device also needs to acquire an overexposed frame with exposure parameters higher than the alignment frame used as the exposure reference, and the exposure compensation of the overexposed frame is positive.

[0128] Step 403: Process the human image position in the human image recognition frame to obtain the human image mask image.

[0129] Step 404: Perform fusion processing on underexposed frames, overexposed frames, and aligned frames to obtain fused frames.

[0130] In some embodiments, the electronic device performs fusion processing on underexposed frames, overexposed frames, and aligned frames to obtain a fused frame. The overexposed frame is used to brighten areas with low actual exposure in other images to be fused, thereby enriching the image details.

[0131] Step 405: Compare the portrait mask image and the fused frame to obtain the portrait region of the fused frame.

[0132] Step 406: Process the image of the portrait area based on the aligned frame and the underexposed frame to obtain the target image.

[0133] Please see Figure 5 Here is a flowchart of an image processing method for scenario 3 above, which may include the following steps:

[0134] Step 501: Obtain scene information from the preview screen of the scene to be collected.

[0135] Step 502: Based on scene information, collect aligned frames, portrait recognition frames, overexposed frames, and multiple underexposed frames with different exposure parameters corresponding to the scene information.

[0136] In some embodiments, based on scene information obtained from the preview image, the electronic device acquires an image to be processed containing multiple underexposed frames. These multiple underexposed frames can be used to reduce overexposure in the fused frames during subsequent fusion processing, thereby enriching the image details of the fused frames.

[0137] In some embodiments, the electronic device selects one of the underexposed frames with higher image quality to improve the image quality of the fused frame after fusion processing. The image quality can be determined based on noise information and detail information contained in multiple underexposed frame images.

[0138] Step 503: Process the human image position in the human image recognition frame to obtain the human image mask image.

[0139] Step 504: Fuse the multiple underexposed frames to obtain the target underexposed frame.

[0140] In some embodiments, the method for fusing multiple underexposed frames to obtain a target underexposed frame is as follows: the fusing process includes calculating the brightness weight and saturation weight of the multiple underexposed frames, calculating the fusion weight of the image to be fused when performing fusion processing based on the brightness weight and saturation weight of the multiple underexposed frame images, and fusing the multiple underexposed frame images according to the fusion weight to obtain the target underexposed frame.

[0141] Step 505: Perform fusion processing on the underexposed frame, overexposed frame and aligned frame of the target to obtain the fused frame.

[0142] Step 506: Compare the portrait mask image and the fused frame to obtain the portrait region of the fused frame.

[0143] Step 507: Process the image of the portrait area based on the aligned frame and the underexposed target frame to obtain the target image.

[0144] Please see Figure 6 This is a structural illustration of an image processing apparatus disclosed in an embodiment of this application. Figure 6 As shown, the image processing device includes a detection module 601, an acquisition module 602, a fusion module 603, a segmentation module 604, and an adjustment module 605, wherein:

[0145] The detection module 601 is used to detect and obtain scene information of the preview screen of the scene to be collected;

[0146] The acquisition module 602 is used to acquire at least three frames of images to be processed based on the scene information and different exposure parameters corresponding to the scene information; the images to be processed include: an alignment frame as an exposure parameter reference, a portrait recognition frame with an exposure parameter less than and closest to the exposure parameter corresponding to the alignment frame, and an exposure processing frame; the exposure processing frame includes at least one underexposed frame with an exposure parameter less than the exposure parameter corresponding to the portrait recognition frame.

[0147] The fusion module 603 is used to perform fusion processing on the images to be fused to obtain a fused image;

[0148] The segmentation module 604 is used to process the portrait position of the portrait recognition frame to obtain a portrait mask image, and compare the portrait mask image with the fused frame to obtain the portrait region of the fused frame;

[0149] The adjustment module 605 is used to perform image processing on the portrait area according to the alignment frame and the underexposed frame to obtain the target image.

[0150] In this embodiment of the application, the detection module 601 can be used to: detect the overexposure level of the preview image; the overexposure level is calculated from the variance of the grayscale histogram of the preview image, the number of high-brightness pixels and the total number of pixels; determine the scene information by comparing the overexposure level with a pre-set exposure threshold table; the exposure threshold table includes the correspondence between the overexposure level and the scene, as well as the exposure parameters selected for different scenes.

[0151] In this embodiment of the application, the fusion module 603 can be used to: calculate the brightness weight and saturation weight of the image to be fused; calculate the fusion weight of the image to be fused when performing the fusion process based on the brightness weight and saturation weight of the image to be fused; and weight the image to be fused according to the fusion weight to obtain the fused image.

[0152] In this embodiment, the segmentation module 604 can be used to: perform image enhancement processing on the portrait recognition frame; the image enhancement processing includes increasing the brightness of the portrait recognition frame; obtain a process image from the image-enhanced portrait recognition frame using a deep learning algorithm; smooth the portrait edges of the image-enhanced portrait recognition frame based on the process image to obtain the portrait mask image; compare the portrait mask image and the fused frame to obtain the portrait region of the fused frame.

[0153] In some embodiments, the acquisition module 602 is also used to acquire overexposed frames with exposure parameters higher than the aligned frame.

[0154] In some embodiments, the acquisition module 602 is also used to acquire overexposed frames with exposure parameters higher than the alignment frame and multiple underexposed frames with exposure parameters lower than the portrait recognition frame.

[0155] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0156] It should be noted that, in the embodiments of this application... Figure 6 The module division of the illustrated electronic device is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into a single processing unit, exist as separate physical units, or be integrated into a single unit. The integrated units described above can be implemented in hardware, as software functional units, or in a combination of both.

[0157] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0158] Please see Figure 7 This is a structural illustration of an electronic device disclosed in an embodiment of this application. Figure 7 As shown, the electronic device 700 may include one or more of the following components: a processor 701 and a memory 702 coupled to the processor 701, wherein the memory 702 may store one or more application programs, the one or more application programs may be configured to be executed by one or more processors 701, and the one or more programs are configured to perform the methods as described in the above embodiments.

[0159] The electronic device's processor provides computing and control capabilities. Its memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The electronic device's database stores data. Its communication interface allows communication with external terminals. When executed by the processor, the computer program implements an image processing method.

[0160] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method provided in the above embodiments.

[0161] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0162] In one embodiment, the electronic device provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 7The image processing method operates on the electronic device shown. The electronic device's memory can store the various program modules required to compose this image processing method, for example, Figure 7 The non-volatile storage medium, internal memory, etc., are shown. The computer program, composed of various program modules, causes the processor to execute the steps of the image processing methods in the various embodiments of this application described in this specification.

[0163] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0164] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0165] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0166] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0167] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0168] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0169] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0170] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0171] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0172] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0173] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0174] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, include: Obtain scene information from a preview of the scene to be captured; Based on the scene information, at least three frames of images to be processed with different exposure parameters corresponding to the scene information are collected; The image to be processed includes: an alignment frame as a reference for exposure parameters, a portrait recognition frame whose exposure parameters are less than and closest to the exposure parameters corresponding to the alignment frame, and an exposure processing frame; the exposure processing frame includes at least one underexposed frame whose exposure parameters are less than the exposure parameters corresponding to the portrait recognition frame. The aligned frame and the exposure processing frame are fused together to obtain a fused frame; The position of the human face in the human face recognition frame is processed to obtain a human face mask image; The portrait mask image and the fused frame are compared to obtain the portrait region of the fused frame; The image of the portrait area is processed based on the alignment frame and the underexposed frame to obtain the target image; The step of processing the portrait area based on the aligned frame and the underexposed frame to obtain the target image includes: The brightness of the portrait area is adjusted based on the alignment frame and the fusion frame. Based on the aligned frame and the underexposed frame, the saturation of the portrait area is adjusted to obtain the target image after brightness and saturation adjustment.

2. The method of claim 1, wherein, The scene information obtained from the preview of the scene to be captured includes: The overexposure level of the preview image is detected; the overexposure level is calculated from the variance of the grayscale histogram of the preview image, the number of high-brightness pixels, and the total number of pixels. The scene information is determined by comparing the degree of overexposure with a pre-set exposure threshold table; the exposure threshold table includes the correspondence between the degree of overexposure and the scene, as well as the exposure parameters selected for different scenes.

3. The method of claim 1, wherein, When the exposure processing frame is an underexposed frame, the step of fusing the alignment frame and the exposure processing frame to obtain a fused frame includes: The aligned frame and the underexposed frame are fused together to obtain a fused frame.

4. The method of claim 1, wherein, When the exposure processing frame further includes an overexposed frame with exposure parameters higher than the exposure parameters corresponding to the alignment frame, the step of fusing the alignment frame and the exposure processing frame to obtain a fused frame includes: The overexposed frame, the underexposed frame, and the aligned frame are fused together to obtain the fused frame.

5. The method of claim 4, wherein, When the exposure processing frame includes multiple underexposed frames, the step of fusing the aligned frame and the exposure processing frame to obtain a fused frame includes: The underexposed frames from the multiple frames are fused together to obtain the target underexposed frame; The overexposed frame, the target underexposed frame, and the aligned frame are fused together to obtain the fused frame.

6. The method according to any one of claims 1 to 5, characterized in that, The process of processing the human image position in the human image recognition frame to obtain a human image mask image includes: The facial recognition frame is subjected to image enhancement processing; the image enhancement processing includes increasing the brightness of the facial recognition frame; The human face recognition frame after image enhancement processing is used to obtain a process image through a deep learning algorithm; Based on the process image, the edges of the human face in the enhanced human face recognition frame are smoothed to obtain the human face mask image.

7. The method of claim 5, wherein, The step of processing the portrait area based on the aligned frame and the underexposed frame to obtain the target image includes: The brightness of the portrait area is adjusted based on the alignment frame and the fusion frame. Based on the alignment frame and the target underexposed frame, the saturation of the portrait area is adjusted to obtain the target image after brightness and saturation adjustment.

8. The method according to any one of claims 1 to 7, characterized in that, The step of adjusting the brightness of the portrait area based on the aligned frame and the merged frame includes: Obtain the corresponding region between the alignment frame and the portrait region; Calculate the mean and variance of brightness for the portrait region of the fused frame and the corresponding region of the aligned frame, respectively. The brightness of the human figure region in the fused frame is adjusted based on the mean brightness and the variance of brightness.

9. The method of claim 1, wherein, The step of adjusting the saturation of the portrait area based on the aligned frame and the underexposed frame includes: Obtain the corresponding region between the alignment frame and the portrait region; Obtain the corresponding region between the underexposed frame and the portrait area; Obtain the average saturation and average brightness of the corresponding region of the aligned frame and the corresponding region of the underexposed frame; Based on the average saturation and average brightness of the corresponding regions of the aligned frame and the corresponding regions of the underexposed frame, determine the overexposure status of the portrait region and the corresponding regions of the underexposed frame; Based on the overexposure situation, the saturation of the portrait area is adjusted by selecting either the alignment frame or the underexposure frame.

10. The method of claim 7, wherein, The step of adjusting the saturation of the portrait area based on the aligned frame and the target underexposed frame includes: Obtain the corresponding region between the alignment frame and the portrait region; Obtain the corresponding region between the target underexposed frame and the portrait area; Obtain the average saturation and average brightness of the corresponding region of the aligned frame and the corresponding region of the target underexposed frame; Based on the average saturation and average brightness of the corresponding regions of the aligned frame and the corresponding regions of the target underexposed frame, determine the overexposure status of the portrait region and the corresponding regions of the target underexposed frame; The saturation of the portrait area is adjusted according to the overexposure situation.

11. An image processing apparatus, comprising: The detection module is used to detect and obtain scene information from the preview of the scene to be captured; The acquisition module is used to acquire at least three frames of images to be processed based on the scene information and different exposure parameters corresponding to the scene information; The image to be processed includes: an alignment frame as a reference for exposure parameters, a portrait recognition frame whose exposure parameters are less than and closest to the exposure parameters corresponding to the alignment frame, and an exposure processing frame; the exposure processing frame includes at least one underexposed frame whose exposure parameters are less than the exposure parameters corresponding to the portrait recognition frame. A fusion module is used to perform fusion processing on the images to be fused to obtain a fused image. The fusion processing on the images to be fused to obtain a fused image includes: performing fusion processing on the alignment frame and the exposure processing frame to obtain a fused frame. The segmentation module is used to process the human image position of the human image recognition frame to obtain a human image mask image, and compare the human image mask image with the fused frame to obtain the human image region of the fused frame; The adjustment module is used to perform image processing on the portrait area based on the alignment frame and the underexposed frame to obtain the target image; The adjustment module is also used to: adjust the brightness of the portrait area according to the alignment frame and the fusion frame; Based on the aligned frame and the underexposed frame, the saturation of the portrait area is adjusted to obtain the target image after brightness and saturation adjustment.

12. An electronic device, comprising: include: Memory containing executable program code; and the processor coupled to the memory; The processor calls the executable program code stored in the memory, and when the executable program code is executed by the processor, the processor implements the method as described in any one of claims 1-10.

13. A computer-readable storage medium having stored thereon an executable program code, characterized in that, When the executable program code is executed by the processor, it implements the method as described in any one of claims 1-10.

Images