Image processing method and device, electronic equipment and computer storage medium

Abstract

The embodiment of the invention discloses an image processing method, which comprises the steps of determining a brightness mask image of a current frame, determining an intensity parameter image of the current frame, performing image enhancement processing on the current frame to obtain the enhanced current frame, and determining the intensity parameter image of the current frame according to the intensity parameter image when the current frame is subjected to brightness processing, and fusing the current frame and the enhanced current frame by using the brightness mask image and the intensity parameter image of the current frame to obtain an output image. The embodiment of the invention further provides an image processing device, electronic equipment and a computer storage medium.

Description

Technical Field

[0001] This application relates to the technology of image processing, and in particular, to an image processing method, apparatus, electronic device, and computer storage medium. Background Art

[0002] Currently, the shooting and dissemination of videos have long become an essential part of everyone's life, but there are always some pain points in the shooting and optimization of night scene images or video effects.

[0003] In the related art, in the processing of night scene images, due to the extremely large contrast between light and dark in the night scene, details in the dark part are likely to be lost in the processed image, while the highlight part is likely to be overexposed, affecting the image quality of the night scene image. Summary of the Invention

[0004] Embodiments of this application provide an image processing method, apparatus, electronic device, and computer storage medium, which can improve the image quality of night scene images.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide an image processing method, including:

[0007] Determine the luminance mask image of the current frame;

[0008] Determine the intensity parameter image of the current frame; wherein, the intensity parameter image is the intensity parameter map when performing luminance processing on the current frame;

[0009] Perform image enhancement processing on the current frame to obtain the current frame after enhancement processing;

[0010] Use the luminance mask image of the current frame and the intensity parameter image to fuse the current frame and the current frame after enhancement processing to obtain an output image.

[0011] In a second aspect, embodiments of this application provide an image processing apparatus, including:

[0012] A first determination module, configured to determine the luminance mask image of the current frame;

[0013] A second determination module, configured to determine the intensity parameter image of the current frame; wherein, the intensity parameter image is the intensity parameter map when performing luminance processing on the current frame;

[0014] A processing module, configured to perform image enhancement processing on the current frame to obtain the current frame after enhancement processing;

[0015] A fusion module, configured to fuse the current frame and the enhanced current frame by using the luminance mask image of the current frame and the intensity parameter image, so as to obtain an output image.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor and a storage medium storing processor-executable instructions; the storage medium depends on the processor to execute operations through a communication bus, and when the instructions are executed by the processor, the image processing method described in one or more of the above embodiments is executed.

[0017] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing executable instructions, and when the executable instructions are executed by one or more processors, the processor executes the image processing method described in one or more of the above embodiments.

[0018] An embodiment of the present application provides an image processing method, apparatus, electronic device, and computer storage medium. Determine the luminance mask image of the current frame, and determine the intensity parameter image of the current frame. The intensity parameter image is: the intensity parameter map when performing luminance processing on the current frame. Perform image enhancement processing on the current frame to obtain the enhanced current frame. Use the luminance mask image and the intensity parameter image of the current frame to fuse the current frame and the enhanced current frame to obtain an output image. That is to say, in the embodiment of the present application, by determining the luminance mask image of the current frame, the bright and dark regions can be identified, and the intensity parameter image can identify the intensity parameters for performing luminance processing on different luminance regions. In this way, through the two, the current frame and the enhanced current frame after image enhancement processing are fused, so that different parameters can be used to fuse different luminance regions of the current frame. Then, for a night scene image with a large contrast, different processing can be performed on different luminance regions, improving the image quality of the night scene image. Description of the Drawings

[0019] Figure 1 It is a schematic flowchart of an optional image processing method provided by an embodiment of the present application;

[0020] Figure 2 It is a schematic flowchart of an example of an optional image processing method provided by an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of an optional sigmod mapping curve provided by an embodiment of the present application;

[0022] Figure 4a It is a schematic diagram of an optional luminance mask image provided by an embodiment of the present application Figure 1 ;

[0023] Figure 4bSchematic diagram of an optional luminance mask image provided by an embodiment of the present application Figure 2 ;

[0024] Figure 4c Schematic diagram of an optional luminance mask image provided by an embodiment of the present application Figure 3 ;

[0025] Figure 5 Schematic diagram of an optional luminance and intensity parameter mapping curve provided by an embodiment of the present application;

[0026] Figure 6 Schematic diagram of an optional permeability enhancement curve provided by an embodiment of the present application;

[0027] Figure 7 Schematic diagram of the structure of an optional image processing device provided by an embodiment of the present application;

[0028] Figure 8 Schematic diagram of the structure of an optional ISP provided by an embodiment of the present application;

[0029] Figure 9 Schematic diagram of the structure of an optional electronic device provided by an embodiment of the present application Figure 1 ;

[0030] Figure 10 Schematic diagram of the structure of an optional electronic device provided by an embodiment of the present application Figure 2 。 Detailed implementation manners

[0031] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application.

[0032] In the related art, regarding the relationship between the dynamic range and the tone, first, the definitions of the dynamic range and the tone are clarified.

[0033] Dynamic Range: It refers to the difference between the brightest and darkest areas that a camera can capture. In a night scene video, the common challenge for the dynamic range is how to simultaneously retain the details in the dark areas (such as stars in the night sky, street shadows, etc.) and the brightness in the highlight parts (such as street lights, car headlights, etc.). If the dynamic range is insufficient, overexposure (loss of details in the highlight part) or underexposure (loss of details in the dark part) will occur in the picture.

[0034] Tonal Range: It is the way and level of the transition of light and dark, and tone in an image. The tonal relationship determines the brightness, contrast, and shadow depth in an image. A well-designed tonal range can make the picture look more hierarchical and natural. In a night scene video, the importance of the tonal relationship is particularly prominent because it determines how to present the contrast between low-light areas and strong light sources.

[0035] It can be seen that the adjustment of dynamic range and tonal range not only involves the adjustment of exposure and contrast, but also includes color mapping. Especially in night scenes, a higher color temperature and the use of cold tones may cause the shadow areas to look "dead" or "depressing", while the highlight areas may appear too dazzling.

[0036] Aiming at the technical problem of poor image effects in night scene image processing, the embodiment of the present application provides a method for processing an image. Figure 1 As shown in Figure 1 the following, the method for processing an image can include:

[0037] S101: Determine the brightness mask image of the current frame;

[0038] The method for processing an image provided by the embodiment of the present application can be applied to an Image Signal Processing (ISP), or can be applied to the processor of an electronic device. Here, the embodiment of the present application does not make specific limitations on this.

[0039] After obtaining the current frame, the brightness mask image of the current frame can be determined. Here, the current frame can be RAW data collected by an image sensor. Of course, it can also be other image data. Here, the embodiment of the present application does not make specific limitations on this.

[0040] Among them, the brightness mask image of the current frame can be determined according to the brightness image of the current frame. Among them, the brightness mask image of the current frame can be directly determined according to the current frame, or the current frame can be downsampled first, and the image obtained by downsampling can be used to determine the brightness mask image of the current frame. In addition, after obtaining the brightness mask image of the current frame, the brightness mask image of the current frame can be corrected on this basis to update and obtain the brightness mask image of the current frame. Here, the embodiment of the present application does not make specific limitations on this.

[0041] S102: Determine the intensity parameter image of the current frame;

[0042] After obtaining the current frame, the intensity parameter image of the current frame can also be determined according to the current frame, where the intensity parameter image is: the intensity parameter map when performing brightness processing on the current frame; that is, the intensity parameters for performing brightness processing on the current frame can be determined according to the current frame, so as to obtain the intensity parameter image.

[0043] It can be seen from the intensity parameter image that for different brightnesses of the current frame, the corresponding intensity parameters are different. In this way, different intensity parameters can be determined for different brightness regions of the current frame, so as to perform different intensity processing on different brightness regions in the current frame.

[0044] S103: Perform image enhancement processing on the current frame to obtain the current frame after enhancement processing;

[0045] In addition, after obtaining the current frame, image enhancement processing is also performed on the current frame. Here, the image enhancement processing can adopt enhancement methods in the spatial domain, can also adopt enhancement methods in the frequency domain, and can also adopt the method of an Artificial Intelligence (AI) model for image enhancement. Of course, here, the method of first setting a mapping relationship and then using the mapping relationship can also be adopted to perform image enhancement processing on the current frame. Here, the embodiments of the present application do not make specific limitations in this regard.

[0046] It should be noted that when the current frame is an image collected by an image sensor, here, during the processing of the current frame, the image data of the current frame can be first converted to obtain an RGB image, and then the RGB image is subjected to image enhancement processing to obtain the current frame after enhancement processing.

[0047] S104: Use the brightness mask image and intensity parameter image of the current frame to fuse the current frame and the current frame after enhancement processing to obtain an output image.

[0048] After obtaining the brightness mask image of the current frame, the intensity parameter image of the current frame, and the current frame after enhancement processing through the above S101 - S103, in S104, the current frame and the current frame after enhancement processing can be fused by using the brightness mask image and the intensity parameter image of the current frame to obtain an output image.

[0049] Here, the fusion weights can be determined by using the brightness mask image and the intensity parameter image of the current frame, so as to realize the fusion of the current frame and the current frame after enhancement processing. Of course, the method of an AI model can also be adopted by using the brightness mask image and the intensity parameter image of the current frame to realize the fusion of the current frame and the current frame after enhancement processing to obtain an output image. Here, the embodiments of the present application do not make specific limitations in this regard.

[0050] In an alternative embodiment, to obtain the luminance mask image of the current frame, determining the luminance mask image of the current frame may include:

[0051] Determine the luminance image of the current frame;

[0052] Map the luminance image according to a preset luminance mapping relationship to obtain the mapped luminance image;

[0053] Based on the relationship between the luminance values in the mapped luminance image and a preset luminance threshold, reset the mapped luminance image to obtain the luminance mask image of the current frame.

[0054] It can be understood that after obtaining the current frame, to determine the luminance image of the current frame, a preset luminance formula can be used here to determine the luminance image of the current frame, and then the preset luminance mapping relationship is used to map the luminance image of the current frame, so that the mapped luminance image can be obtained.

[0055] Here, the purpose of adopting the above-mentioned preset luminance mapping relationship is to widen the luminance values in the luminance image of the current frame and map different luminance values for different luminance values. For example, when using the sigmod function for mapping, the values in the dark area can be made closer to 0 after mapping, the values in the bright area can be made closer to 1 after mapping, and the luminance values in the middle area between the dark area and the bright area are between 0 and 1. In this way, the luminance values in the mapped luminance image are widened in the middle area.

[0056] Then, compare the luminance values in the mapped luminance image with a preset luminance threshold. Here, a first preset luminance threshold and a second preset luminance threshold can be set, and the first preset luminance threshold is less than the second preset luminance threshold.

[0057] Then, set the luminance values less than the first preset luminance threshold to 0, and set the luminance values greater than the second preset luminance threshold to 255.

[0058] In this way, by adopting the above-mentioned preset luminance mapping relationship to obtain the mapped luminance image and based on the relationship between the luminance values of the mapped luminance image and the preset luminance threshold, the luminance mask image of the current frame can be obtained, so that the obtained luminance mask image can reflect different luminance regions of the image, which is convenient for fusing the current frame and the enhanced current frame to improve the image quality.

[0059] Further, in an alternative embodiment, to update the luminance mask image of the current frame, S101 may include:

[0060] Perform image recognition on the current frame to obtain the target region of the current frame;

[0061] Set the pixel values of the target area in the luminance mask image of the current frame to 0 to update the luminance mask image of the current frame.

[0062] Understandably, based on the image data of the current frame, the luminance mask image of the current frame can be determined. After performing image recognition on the current frame to obtain the target area of the current frame, the pixel values of the target area in the luminance mask image of the current frame are then set to 0, thereby enabling the update of the luminance mask image of the current frame.

[0063] Among them, the target area can be the area containing the target object obtained by performing image recognition on the current frame. The target object can be a person, an animal, a scene, etc. Here, the embodiments of the present application do not make specific limitations in this regard.

[0064] That is to say, here, after obtaining the luminance mask image of the current frame, the target area obtained by performing image recognition on the luminance mask image of the current frame can be determined, and then the pixel values of the target area in the luminance mask image of the current frame are reset, so that the target area is used as the dark area. In this way, the luminance mask image of the current frame is updated.

[0065] It should be noted that when determining the luminance mask image of the current frame, if the downsampled image of the current frame is used to determine the luminance mask image of the current frame, then when fusing it with the intensity parameter image for the current frame and the enhanced current frame, the size of the luminance mask image of the current frame needs to be extended (resize) so that the extended size is consistent with the current frame.

[0066] In this way, resetting the pixel values of the target area in the luminance mask image of the current frame plays a role in protecting the target area, thereby improving the image quality of the target area in the night scene image.

[0067] In order to obtain the target area, in an optional embodiment, performing image recognition on the current frame to obtain the target area of the current frame may include:

[0068] Performing image recognition on the current frame to obtain the face area of the current frame;

[0069] Taking the face area as a reference to determine the target area.

[0070] Understandably, image recognition can be first performed on the current frame to identify the face area. The face area can be used as the target area, or the target area can be determined based on the face area. For example, taking the face area as a reference and using a preset value as the diameter to draw a circle to obtain a circular area that includes the face area, and taking the circular area as the target area.

[0071] In this way, the face region in the current frame can be recognized, and the target region can be determined based on it, so that the target region is obtained based on the face region. Thus, the pixel values in the brightness mask image can be reset based on the face region to protect the face region, so that targeted processing can be performed in this region to improve the image quality.

[0072] Further, in order to determine the target region based on the face region, in an alternative embodiment, determining the target region based on the face region may include:

[0073] Based on the face region, determine the adjacent region of the face region;

[0074] Determine the face region and the adjacent region as the target region.

[0075] It can be understood that first, based on the face region, the adjacent region of the face region can be determined. Here, the face region is generally a face frame obtained through image recognition. Based on the image block of the face frame, determine the adjacent image blocks of the image block of the face frame and use them as the adjacent regions of the face region.

[0076] Then, use the face region and the adjacent region as the target region. In this way, the target region not only includes the face region but also the adjacent region of the face region, so that not only the face region is protected, but also the adjacent region of the face region is protected, thus more effectively protecting the face region and enabling targeted processing of the face region and its adjacent region, improving the image quality.

[0077] In order to obtain a smoother brightness mask image of the current frame, in an alternative embodiment, the above method may further include:

[0078] Perform smoothing processing on the brightness mask image of the current frame to update and obtain the brightness mask image of the current frame.

[0079] It can be understood that here, after obtaining the brightness mask image of the current frame, smoothing processing can be performed on the brightness mask image of the current frame. Here, the brightness mask image of the current frame can be smoothed by setting a smoothing window, so that the brightness values of the smoothed brightness mask image of the current frame are smoother.

[0080] In this way, the brightness mask image of the current frame can be further smoothed, which helps to fuse the current frame and the enhanced current frame to improve the image quality of the output image.

[0081] For two consecutive frames of dynamically changing images, in an alternative embodiment, the above method may further include:

[0082] Perform a fusion process on the luminance mask image of the current frame and the luminance mask image of the previous frame of the current frame to update and obtain the luminance mask image of the current frame.

[0083] Understandably, in multi-frame fusion, the object in the target area will move frequently. For example, in shooting, the face frame will move frequently. To eliminate the discontinuity of the luminance mask image in the time domain caused by the frequent movement of the face, in the embodiments of the present application, the luminance mask image of the current frame can be processed for temporal balance.

[0084] Here, a fusion process can be performed on the luminance mask image of the current frame and the luminance mask image of the previous frame of the current frame to update and obtain the luminance mask image of the current frame. The fusion process here mainly determines the respective weight values for the pixel values at each position in the two images, and then performs weighted summation on them, so that the luminance mask image of the current frame can be updated.

[0085] In this way, the luminance mask image of the current frame is processed for temporal balance to eliminate the problem that the luminance mask images of the previous and current frames are discontinuous due to the frequent movement of the object in the target area, making the updated luminance mask image of the current frame have temporal correlation and improving the image quality on the basis of considering the influence of the previous and current frames.

[0086] In an optional embodiment, to obtain the intensity parameter image of the current frame, S102 may include:

[0087] Determine the luminance image of the current frame;

[0088] Map the luminance image according to the preset mapping relationship between luminance and intensity parameters to obtain the intensity parameter image of the current frame.

[0089] Understandably, first determine the luminance image of the current frame. Here, the luminance value at each position can be determined by using the values of each channel, so as to determine the luminance image of the current frame. Then, use the preset mapping relationship between luminance and intensity parameters to map the luminance image, so as to map the intensity parameter corresponding to each position, and then obtain the intensity parameter image of the current frame.

[0090] Among them, the preset mapping relationship between luminance and intensity parameters is a curve with a positive correlation between luminance and intensity parameters.

[0091] In this way, through the preset mapping relationship between luminance and intensity parameters, the intensity parameter image of the current frame can be determined, which helps to determine different intensity parameter values for luminance processing and provides favorable data for the fusion of the current frame and the current frame after enhancement processing.

[0092] To achieve image enhancement for the current frame, in an optional embodiment, S103 may include:

[0093] Determine the mapping relationship diagram of the current frame;

[0094] Map the current frame according to the mapping relationship diagram of the current frame to obtain the enhanced current frame.

[0095] It can be understood that after obtaining the current frame, first determine the mapping relationship diagram of the current frame. Here, for each position of the current frame, determine a mapping relationship, so as to determine the mapping relationship diagram of the current frame.

[0096] Among them, when determining a mapping relationship for each position of the current frame, the mapping relationship corresponding to the pixel value of each position can be determined according to the preset corresponding relationship, or the mapping relationship of each position can be determined by using the AI model. Here, the embodiments of the present application do not make specific limitations on this.

[0097] After obtaining the mapping relationship diagram of the current frame, for each position, the corresponding mapping relationship can be used to determine the pixel value of this position after mapping, so as to obtain the pixel value of each position after mapping, which is the enhanced current frame.

[0098] In this way, by first determining the mapping relationship diagram of the current frame and then using the mapping relationship diagram of the current frame to map the current frame, the enhanced current frame is obtained, so that the enhanced current frame realizes image enhancement compared with the current frame, which is helpful for fusion with the current frame.

[0099] To determine the mapping relationship diagram of the current frame, in an optional embodiment, determining the mapping relationship of the current frame may include:

[0100] Determine the minimum value of each channel of the current frame;

[0101] Determine the minimum value of the minimum value of each channel and the preset multiple of the luminance image of the current frame as the target parameter map;

[0102] Input the target parameter map into the preset function to obtain the mapping relationship diagram of the current frame.

[0103] It can be understood that taking the current frame as a RAW image as an example, first convert the RAW image into an RGB image, and then for each position in the RGB image, determine the minimum value of each channel, and compare it with the preset multiple of the luminance image of the current frame. Here, for each position point, compare the minimum value of each channel with the preset multiple of the luminance value of this position point in the luminance image of the current frame, and take the minimum value obtained by comparison as the target parameter, so as to form the target parameter map.

[0104] Among them, the above preset multiple is a value in the range of 0 - 0.5. Then, after obtaining the target parameter map, the target parameter map can be used as the parameter of the preset function to obtain the mapping relationship at each position, thereby obtaining the mapping relationship map of the current frame.

[0105] In this way, by determining the mapping relationship map of the current frame in the above manner of determining the target parameter map, the pixel values at different positions can be mapped using different mapping relationships, so as to implement different image enhancement methods for different positions to process the current frame, making the image enhancement effect of the current frame after enhancement processing better, which helps to improve the image quality of the output image.

[0106] For two consecutive frames of dynamically changing images, in an optional embodiment, the above method may further include:

[0107] Perform a fusion process on the mapping relationship map of the current frame and the mapping relationship map of the previous frame of the current frame to update and obtain the mapping relationship map of the current frame.

[0108] It can be understood that in multi-frame fusion, the objects in adjacent two frames will move frequently. For example, in shooting, the face frame will move frequently. In order to eliminate the discontinuity of the mapping relationship map of the current frame caused by the frequent movement of the face in two consecutive frames, in the embodiment of the present application, a temporal balance process can be performed on the mapping relationship map of the current frame.

[0109] Here, a fusion process can be performed on the mapping relationship map of the current frame and the mapping relationship map of the previous frame of the current frame to update and obtain the mapping relationship map of the current frame. The fusion process here is mainly to determine the respective weight values for the target parameters at each position in the two images, and then perform a weighted sum on them, so as to update and obtain the mapping relationship map of the current frame.

[0110] In this way, a temporal balance process is performed on the mapping relationship map of the current frame to eliminate the problem that the frequent movement of the objects in adjacent two frames makes the mapping relationship maps of the front and back two frames discontinuous, so that the updated mapping relationship map of the current frame can consider the changes in adjacent frames in multi-frame fusion, has temporal correlation, and improves the image quality of the night scene image on the basis of considering the influence of the front and back frames.

[0111] In an optional embodiment, to obtain the output image, S104 may include:

[0112] Determine the fusion weight according to the luminance mask image and the intensity parameter image of the current frame;

[0113] Use the fusion weight to perform a weighted sum on the current frame and the current frame after enhancement processing to obtain the output image.

[0114] Understandably, after determining the luminance mask image and the intensity parameter image of the current frame, the fusion weights can be determined based on these two images. The fusion weights include the fusion weight of the current frame and the fusion weight of the current frame after enhancement processing, and the sum of the fusion weight of the current frame and the fusion weight of the current frame after enhancement processing is 1.

[0115] After obtaining the fusion weight of the current frame and the fusion weight of the current frame after enhancement processing, the current frame and the current frame after enhancement processing are weighted and summed using the fusion weight of the current frame and the fusion weight of the current frame after enhancement processing, thereby obtaining the output image.

[0116] In this way, the luminance mask image and the intensity parameter image of the current frame are used to determine the fusion weights, thereby realizing the fusion of the current frame and the current frame after enhancement processing to obtain the output image. Considering the intensity of local mapping in the fusion using the luminance mask image and the intensity of global mapping in the intensity parameter image enables different intensities to be used for fusion at different positions, improving the image quality of the output image.

[0117] Further, in order to obtain the fusion weight of the current frame and the fusion weight of the current frame after enhancement processing, in an optional embodiment, determining the fusion weights based on the luminance mask image and the intensity parameter image of the current frame may include:

[0118] Determining the fusion weight of the current frame after enhancement processing in the fusion weights according to the product of the luminance mask image and the intensity parameter image of the current frame;

[0119] Determining the difference obtained by subtracting the fusion weight of the current frame after enhancement processing from 1 as the fusion weight of the current frame.

[0120] Understandably, after obtaining the luminance mask image and the intensity parameter image of the current frame, for each position of the image, the pixel values at the corresponding positions on the luminance mask image and the intensity parameter image of the current frame are multiplied to obtain the fusion weight of the current frame after enhancement processing, and then the value obtained by subtracting the fusion weight of the current frame after enhancement processing from 1 is used as the fusion weight of the current frame.

[0121] After determining the fusion weight of the current frame and the fusion weight of the current frame after enhancement processing, fusing the current frame and the current frame after enhancement processing can obtain the output image.

[0122] In this way, determining the fusion weights through the above method enables the consideration of not only the intensity of local mapping but also the intensity of global mapping in the fusion, so that different intensities can be used for fusion at different positions, improving the image quality of the output image.

[0123] When the image processing method provided by the embodiments of the present application is applied to an ISP, in an optional embodiment, the current frame is an image acquired from an image sensor.

[0124] It can be understood that when the above image processing method is adopted in the ISP, the current frame is an image acquired from an image sensor, that is, a RAW image.

[0125] It should be noted that in the above process of performing image enhancement processing on the current frame to obtain the enhanced current frame, it is necessary to first convert the RAW image to obtain an RGB image, and then perform enhancement processing on the RGB image to obtain the enhanced current frame.

[0126] In this way, by executing the above image processing method in the ISP, different positions of the night scene image can be processed differently in the ISP, which is beneficial to processing the night scene image before post-image processing and improves the image quality of the night scene image.

[0127] The following is an example to describe the image processing method described in one or more of the above embodiments.

[0128] In this example, the algorithm processing nodes are performed in the RAW domain and the YUV domain respectively. First, calculate the Highlight mask representing highlights (equivalent to the above brightness mask image) in the RAW domain, and transfer it to the YUV domain. Combine information such as the face region (face roi) and face luminance (face luma) to construct a temporally smoothed Local mask (equivalent to the updated brightness mask image) for realizing local tone enhancement.

[0129] For the YUV node data, first convert it to grayscale data, and through image mapping (remap) processing, obtain the global Global mask (equivalent to the above intensity parameter image).

[0130] Through the three RGB channels, combined with the design principle of "dark channel prior", judge the permeability enhancement weight value according to the minimum channel, and combine the Global mask and the Local mask to generate the final tone enhancement result.

[0131] Figure 2 For the flow schematic diagram of an example of an optional image processing method provided by the embodiments of the present application, as Figure 2 shown, the image processing method may include:

[0132] S201: Obtain an input image (RAW Input); respectively execute S202 and S207;

[0133] Among them, the input image is a RAW image.

[0134] S202: Determine the Highlight mask based on RAW Input; execute S203;

[0135] Specifically, the calculation of the Highlight mask is as follows:

[0136] In this example, the Highlight mask is calculated from the RAW domain data to indicate the location of the highlight area. Since the data in the RAW domain is linear data and has not undergone non - linear processing units such as local tone mapping and gamma, the over - exposed area is calculated at the node to represent the highlight information. The calculation steps are as follows:

[0137] (a) RAW downsampling:

[0138]

[0139] (b) Generate a grayscale thumbnail:

[0140] Gray ds = 0.2989 * R_ds+0.5870 * (Gr_ds + Gb_ds) / 2+0.1140 * B_ds (3)

[0142] (c) Take the maximum value within each block:

[0143] Gray_ds_block = max(Gray_ds[32 * 32]) (4)

[0144] Among them, take the maximum value every 32 * 32, and the block size can be adjusted according to performance and effect.

[0145] (d) sigmod mapping:

[0146] Gray_ds_block_sigmod = sigmod(Gray_ds_block) (5)

[0147] Among them, Figure 3 is a schematic diagram of an optional sigmod mapping curve provided by the embodiment of the present application. As Figure 3 shown, the sigmod function form, and the sigmod value range in this example is 0 - 1.

[0148] (e) Obtain the Highlight mask:

[0149] Highlight Mask = 0 if(Gray_ds_block_sigmod < min_thres) (6)

[0150] Highlight Mask = 255 if (Gray_ds_block_sigmod > max_thres) (7)

[0151] S203: Face protection; Execute S204;

[0152] S204: Remap; Execute S205;

[0153] S205: Temporal balance; Execute S206;

[0154] S206: Local mask;

[0155] Specifically, in S203 - S206, the RAW domain transfers the Highlight mask to the current node. The obtained video frame is used to identify the face bounding box, which is converted into small - size coordinates and marked as a shadow on the Highlight mask.

[0156] Figure 4a Schematic diagram of an optional luminance mask image provided by an embodiment of the present application Figure 1 , as Figure 4a shown, is the determined Highlight mask. Figure 4b Schematic diagram of an optional luminance mask image provided by an embodiment of the present application Figure 2 , as Figure 4b shown, is the Highlight mask with the face bounding box identified. Figure 4c Schematic diagram of an optional luminance mask image provided by an embodiment of the present application Figure 3 , as Figure 4c shown, because the Highlight mask is small in size. For example, the input data is 4096 * 3072; the Highlight mask is 64 * 48; the face coordinate points are mapped to the position points on the small mask, and the mask values of 9 pixel points in its neighborhood of 3 * 3 are all set to 0, that is, the area protecting the face, reducing the intensity of the face. After filtering the small - size mask with reduced intensity combined with the face bounding box to a certain size, its size is extended (resized) to the size of the large image, obtaining Figure 4c the result, which is the Local mask.

[0157] Meanwhile, since the face bounding box moves frequently in the video, therefore, the Local mask combined with face protection needs to save the Local mask of the previous frame in the time domain, and balance the Local mask of the current frame and the Local mask of the previous frame in the time domain. The specific formula is as follows:

[0158] Mask_cur = alpha * Mask_Pre + (1 - alpha) * Mask_cur(8)

[0159] Among them, Mask_cur represents the Local mask of the current frame, Mask_Pre represents the Local mask of the previous frame, and alpha represents the weight value, generally 0.2.

[0160] S207: Execute the ISP Pipeline; execute S208;

[0161] S208: Based on Input, obtain the RGB image; respectively execute S209 and S212;

[0162] S209: RGB2Gray; execute S210;

[0163] S210: remap; execute S211;

[0164] S211: Global mask; execute S214;

[0165] Specifically, the Global mask guides the processing intensity of different pixels of different images. Therefore, it is obtained by remapping with the brightness value. First, convert the RGB input to a grayscale (Gray) image and perform a brightness relationship mapping on the Gray image.

[0166] In Figure 5 is a schematic diagram of an optional brightness and intensity parameter mapping curve provided by an embodiment of the present application. As Figure 5 shown, by mapping the grayscale image through this mapping, the enhancement weight corresponding to the entire image is obtained. The meaning represented is: the processing intensity is the largest at the highlight, the intensity in the middle tone decays rapidly, and the processing intensity is the weakest in the dark part. This is to prevent problems such as over - strong contrast stretching, resulting in dead black in the dark part and excessive contrast in the whole image.

[0167] S212: Min(R, G, B); execute S213;

[0168] S213: Clarity; execute S214;

[0169] In the design of the permeability enhancement curve, the permeability enhancement curve mapping combines the design idea of the "dark channel". Calculate the minimum value of the three channels for the RGB input data to obtain the dark channel prior map Min(R, G, B). After the RAW to RGB data, perform this step of processing.

[0170] Min(R, G, B) = Min(R, G, B) (9)

[0171]

[0172] Among them, Select between 0 and 0.5.

[0173]

[0174] Among them, maxValue is obtained according to the bit width of the image. For example, when the bit width is 8bit, maxvlaue = 255; γ is adjusted around 1. For example, r = 1.1.

[0175] Figure 6 It is a schematic diagram of an optional permeability enhancement curve provided by an embodiment of the present application. As Figure 6 shown, the solid line is the curve where the input and output are equal, and the dotted line is the permeability enhancement curve. Among them, the curves corresponding to different setting parameters are also different, and the starting point, ending point, and intensity of the mapping all have differences in combination with the image information.

[0176] In addition, the permeability enhancement curve of the current frame and the permeability enhancement curve of the next frame can be balanced in the time domain. Similar to the balancing method of the above Local mask, it will not be elaborated here.

[0177] S214: Use Local mask and Global mask to blend Clarity and Input to obtain Output;

[0178] S215: Output Output.

[0179] After obtaining the Global mask, which marks the intensity of the global mapping, and the Local mask, which identifies the intensity of the local processing, it is possible to protect areas such as the face and the ground to avoid the problem of excessive contrast stretching.

[0180] That is to say, after obtaining the Global mask and the Local mask of the current frame, multiply the two masks to obtain the intensity that should take effect for each pixel point of the current frame.

[0181] Output = Clarity * Global mask * Local mask + (1 - Global mask * Localmask) * Input (12)

[0183] To ensure the temporal stability of the video stream, after obtaining the processing output of the current frame, it is still necessary to balance the clarity mapping of the current frame in the time domain to ensure that the video picture has no flicker.

[0184] This example can be used for input images with any bit width. For example, 8 bits correspond to a data range of (0 to 255), and 10 bits correspond to a data range of (0 to 1024). This solution makes distinctions based on the input format and can be successfully processed. At the same time, the filtering method described in this example is recommended to be mean filtering, but filtering in other image data domains is also possible.

[0185] It can be seen that the night scene tone enhancement technology of this example, combined with the data stream scheme design of ISP, is used for video streams; the tone enhancement scheme under night scenes combines portrait or face information to make special processing strategies to prevent excessive stretching of the contrast in the face area and avoid the problem of dirty faces; the scheme for the photography domain has temporal smoothing processing to ensure the stability in video streams.

[0186] This example proposes a tone enhancement algorithm for night scene videos. The designed architecture combines the imaging data stream to calculate the highlight mask in the linear domain, reducing the real-time computation amount in the YUV domain. Different from the previous algorithm schemes that only perform post-processing, this example can protect the portrait area in combination with the time domain and cause no loss to the portrait during the contrast stretching process. At the same time, a method combining global mapping weights and local mapping weights is designed to achieve contrast stretching with adaptive intensity.

[0187] In this example, a tone permeability enhancement algorithm for night scene videos is designed. The algorithm combines the real-time shooting data stream of the camera and designs a tone processing scheme combining the RAW domain and the YUV image domain. The algorithm can be used for tone enhancement of night scene video images, improving the local contrast without over-stretching areas such as portraits and the ground.

[0188] The embodiment of this application provides a method for processing images, which determines the brightness mask image of the current frame and determines the intensity parameter image of the current frame. The intensity parameter image is the intensity parameter map when performing brightness processing on the current frame. The current frame is subjected to image enhancement processing to obtain the enhanced current frame. The current frame and the enhanced current frame are fused using the brightness mask image and the intensity parameter image of the current frame to obtain the output image. That is to say, in the embodiment of this application, by determining the brightness mask image of the current frame, the bright and dark areas can be identified, and the intensity parameter image can identify the intensity parameters for performing brightness processing in different brightness areas. In this way, through the two, the current frame and the enhanced current frame after image enhancement processing are fused, enabling different parameters to be used for fusion in different brightness areas of the current frame. Then, for night scene images with large contrast, different processing can be performed on different brightness areas, improving the image quality of night scene images.

[0189] Based on the same inventive concept as the foregoing embodiment, the embodiment of this application provides an image processing device. Figure 7The structural schematic diagram of an optional image processing device provided by an embodiment of the present application is as follows. Figure 7 As shown in the figure, the image processing device includes: a first determination module 71, a second determination module 72, a processing module 73, and a fusion module 74. Among them,

[0190] The first determination module 71 is used to determine the luminance mask image of the current frame.

[0191] The second determination module 72 is used to determine the intensity parameter image of the current frame. Among them, the intensity parameter image is the intensity parameter map when performing luminance processing on the current frame.

[0192] The processing module 73 is used to perform image enhancement processing on the current frame to obtain the current frame after enhancement processing.

[0193] The fusion module 74 is used to use the luminance mask image and the intensity parameter image of the current frame to fuse the current frame and the current frame after enhancement processing to obtain an output image.

[0194] In an optional embodiment, the first determination module 71 is specifically used to: determine the luminance mask image of the current frame, including: determining the luminance image of the current frame; performing mapping on the luminance image according to a preset luminance mapping relationship to obtain the mapped luminance image; resetting the mapped luminance image based on the relationship between the luminance value in the mapped luminance image and a preset luminance threshold to obtain the luminance mask image of the current frame.

[0195] In an optional embodiment, the device is further used to: perform image recognition on the current frame to obtain the target area of the current frame; set the pixel values of the target area in the luminance mask image of the current frame to 0 to update and obtain the luminance mask image of the current frame.

[0196] In an optional embodiment, when the first determination module 71 performs image recognition on the current frame to obtain the target area of the current frame, it includes: performing image recognition on the current frame to obtain the face area of the current frame; determining the target area based on the face area.

[0197] In an optional embodiment, when the first determination module 71 determines the target area based on the face area, it includes: determining the adjacent area of the face area based on the face area; determining the face area and the adjacent area as the target area.

[0198] In an optional embodiment, the device is further used to: perform smoothing processing on the luminance mask image of the current frame to update and obtain the luminance mask image of the current frame.

[0199] In an alternative embodiment, the apparatus is further configured to: perform a fusion process on the luminance mask image of the current frame and the luminance mask image of the previous frame of the current frame to update and obtain the luminance mask image of the current frame.

[0200] In an alternative embodiment, the second determination module 72 is specifically configured to: determine the luminance image of the current frame; perform mapping on the luminance image according to a preset mapping relationship between luminance and intensity parameters to obtain the intensity parameter image of the current frame.

[0201] In an alternative embodiment, the processing module 73 is specifically configured to: determine the mapping relationship graph of the current frame; perform mapping on the current frame according to the mapping relationship graph of the current frame to obtain the current frame after enhancement processing.

[0202] In an alternative embodiment, when the processing module 73 determines the mapping relationship of the current frame, it includes: determining the minimum value of each channel of the current frame; determining the minimum value among the minimum values of each channel and a preset multiple of the luminance image of the current frame as the target parameter graph; inputting the target parameter graph into a preset function to obtain the mapping relationship graph of the current frame.

[0203] In an alternative embodiment, the apparatus is further configured to: perform a fusion process on the mapping relationship graph of the current frame and the mapping relationship graph of the previous frame of the current frame to update and obtain the mapping relationship graph of the current frame.

[0204] In an alternative embodiment, the fusion module 74 is specifically configured to: determine a fusion weight according to the luminance mask image and the intensity parameter image of the current frame; use the fusion weight to perform weighted summation on the current frame and the current frame after enhancement processing to obtain an output image.

[0205] In an alternative embodiment, when the fusion module 74 determines the fusion weight according to the luminance mask image and the intensity parameter image of the current frame, it includes: determining the fusion weight of the current frame after enhancement processing in the fusion weight according to the product of the luminance mask image and the intensity parameter image of the current frame; determining the difference obtained by subtracting the fusion weight of the current frame after enhancement processing from one as the fusion weight of the current frame.

[0206] In an alternative embodiment, the current frame is an image acquired from an image sensor.

[0207] In practical applications, the above-mentioned first determination module 71, second determination module 72, processing module 73, and fusion module 74 can be implemented by a processor located on an image processing device, specifically implemented by a CPU, a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0208] An embodiment of the present application provides an ISP. Figure 8 As a schematic structural diagram of an optional ISP provided by an embodiment of the present application, as Figure 8 shown, an embodiment of the present application provides an ISP800, and the ISP800 includes:

[0209] A processor 81, configured to call and run a computer program from a memory, so that a device installed with the ISP800 executes the method described in one or more of the above embodiments.

[0210] A transceiver 82, configured to receive and send information during the process of receiving and sending information between the device and the ISP800.

[0211] Figure 9 As a schematic structural diagram of an optional electronic device provided by an embodiment of the present application Figure 1 , as Figure 9 shown, an embodiment of the present application provides an electronic device 900, including:

[0212] An ISP900, a processor 91, and a storage medium 92 storing executable instructions of the processor; the storage medium 92 depends on the processor 91 to execute operations through a communication bus 93.

[0213] Figure 10 As a schematic structural diagram of an optional electronic device provided by an embodiment of the present application Figure 2 , as Figure 10 shown, an embodiment of the present application provides an electronic device 1000, including:

[0214] A processor 101 and a storage medium 102 storing executable instructions of the processor; the storage medium 102 depends on the processor 101 to execute operations through a communication bus 103. When the instructions are executed by the processor, the image processing method executed on the processor side in one or more of the above embodiments is executed.

[0215] It should be noted that in actual application, each component in the computer device is coupled together through the communication bus 103. It can be understood that the communication bus 103 is used to realize the connection and communication between these components. In addition to the data bus, the communication bus 103 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clear illustration, in Figure 10 all kinds of buses are labeled as the communication bus 103.

[0216] An embodiment of the present application provides a computer storage medium storing executable instructions. When the executable instructions are executed by one or more processors, the processors execute the image processing method described in the above one or more embodiments.

[0217] Among them, the computer-readable storage medium can be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.

[0218] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program codes.

[0219] This application is described with reference to the flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices produce means for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or one or more of the blocks.

[0220] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufactured article including instruction means that implement the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or one or more of the blocks.

[0221] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more of the flows Figure 1 one or more of the flows and / or blocks Figure 1 or one or more of the blocks.

[0222] As mentioned above, it is only a preferred embodiment of the present application and is not used to limit the protection scope of the present application.

Claims

1. A method for processing an image, characterized in that: include: Determine the brightness mask image of the current frame; Determine an intensity parameter image of the current frame; wherein the intensity parameter image is: an intensity parameter map when brightness processing is performed on the current frame; Performing image enhancement processing on the current frame to obtain an enhanced current frame; The current frame and the enhanced current frame are fused using the brightness mask image of the current frame and the intensity parameter image to obtain an output image.

2. The method according to claim 1, characterized in that The step of determining the brightness mask image of the current frame includes: Determining a brightness image of the current frame; According to a preset brightness mapping relationship, the brightness image is mapped to obtain a mapped brightness image; Based on the relationship between the brightness value in the mapped brightness image and the preset brightness threshold, the mapped brightness image is reset to obtain the brightness mask image of the current frame.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Performing image recognition on the current frame to obtain a target area of ​​the current frame; The pixel value of the target area in the brightness mask image of the current frame is set to 0 to update the brightness mask image of the current frame.

4. The method according to claim 3, characterized in that The performing image recognition on the current frame to obtain a target area of ​​the current frame includes: Performing image recognition on the current frame to obtain a face region of the current frame; The target area is determined based on the face area.

5. The method according to claim 4, characterized in that The step of determining the target area based on the face area includes: Taking the face region as a reference, determining an adjacent region of the face region; The face area and the adjacent area are determined as the target area.

6. The method according to claim 1, characterized in that The method further comprises: The brightness mask image of the current frame is smoothed to update the brightness mask image of the current frame.

7. The method according to claim 1, characterized in that The method further comprises: The brightness mask image of the current frame and the brightness mask image of the previous frame of the current frame are fused to update the brightness mask image of the current frame.

8. The method according to claim 1, characterized in that Determining the intensity parameter image of the current frame includes: Determining a brightness image of the current frame; According to a preset mapping relationship between brightness and intensity parameters, the brightness image is mapped to obtain an intensity parameter image of the current frame.

9. The method according to claim 1, characterized in that: The performing image enhancement processing on the current frame to obtain the enhanced current frame includes: Determine a mapping relationship diagram of the current frame; According to the mapping relationship diagram of the current frame, the current frame is mapped to obtain the enhanced current frame.

10. The method according to claim 9, characterized in that The determining of the mapping relationship of the current image frame includes: Determine the minimum value of each channel of the current frame; Determine the minimum value of each channel and the preset multiple of the brightness image of the current frame as the target parameter map; The target parameter graph is input into a preset function to obtain a mapping relationship graph of the current frame.

11. The method according to claim 9, characterized in that The method further comprises: The mapping relationship diagram of the current frame and the mapping relationship diagram of the previous frame of the current frame are fused to update the mapping relationship diagram of the current frame.

12. The method according to claim 1, characterized in that The step of fusing the current frame and the enhanced current frame using the brightness mask image and the intensity parameter image to obtain an output image includes: Determining a fusion weight according to the brightness mask image of the current frame and the intensity parameter image; The fusion weight is used to perform a weighted summation on the current frame and the current frame after the enhancement process to obtain the output image.

13. An image processing device, characterized in that: include: A first determination module, used to determine a brightness mask image of a current frame; A second determination module is used to determine an intensity parameter image of the current frame; wherein the intensity parameter image is: an intensity parameter map when brightness processing is performed on the current frame; A processing module, used for performing image enhancement processing on the current frame to obtain the current frame after the enhancement processing; The fusion module is used to fuse the current frame and the enhanced current frame using the brightness mask image of the current frame and the intensity parameter image to obtain an output image.

14. An electronic device, characterized in that: include: A processor and a storage medium storing instructions executable by the processor; The storage medium relies on the processor to perform operations through a communication bus, and when the instructions are executed by the processor, the image processing method described in any one of claims 1 to 12 is executed.

Images