Image processing method and device, electronic equipment and readable storage medium

Abstract

The application discloses an image processing method, belonging to the field of image processing, which comprises the following steps: constructing a haze perception depth image according to a first image and a second image, the shooting magnification of the first image and the second image being different, the haze perception depth image comprising at least one of pixel depth information, pixel edge information and pixel brightness information; obtaining a haze image feature map according to the haze perception depth image, the first image and the second image, the haze image feature map being used for indicating the concentration of haze in different regions of the image; and performing image reconstruction processing on the first image and the second image according to the haze feature image, so as to generate a third image, the third image being the image after haze removal of the first image and the second image.

Description

Technical Field

[0001] This application belongs to the field of image processing, and specifically relates to an image processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] When users need to take images and there is smog in the shooting environment, the higher the smog concentration, the worse the image quality.

[0003] However, most existing image dehazing methods are based on deep learning, which cannot fully utilize the depth information of foggy images. As a result, the dehazing images still have hazy spots or the dehazing effect is unnatural, resulting in poor image quality when electronic devices take pictures in foggy environments. Summary of the Invention

[0004] The purpose of this application is to provide an image processing method, apparatus, electronic device, and readable storage medium that can improve the image quality of images captured by electronic devices in foggy environments.

[0005] In a first aspect, embodiments of this application provide an image processing method, which includes: constructing a haze perception depth image based on a first image and a second image, wherein the first image and the second image have different magnifications, and the haze perception depth image includes at least one of pixel depth information, pixel edge information, and pixel brightness information; obtaining a haze image feature map based on the haze perception depth image, the first image, and the second image, wherein the haze image feature map is used to indicate the concentration of fog in different regions of the image; and performing image reconstruction processing on the first image and the second image based on the haze feature image to generate a third image, wherein the third image is an image after dehazing the first image and the second image.

[0006] Secondly, embodiments of this application provide an image processing apparatus, comprising: a construction module and a processing module; the construction module is configured to construct a haze perception depth image based on a first image and a second image, wherein the first image and the second image have different magnifications, and the haze perception depth image includes at least one of pixel depth information, pixel edge information, and pixel brightness information; the processing module is configured to obtain a haze image feature map based on the haze perception depth image, the first image, and the second image, wherein the haze image feature map is used to indicate the concentration of haze in different regions of the image, and to perform image reconstruction processing on the first image and the second image based on the haze feature map to generate a third image, wherein the third image is the image after dehazing the first image and the second image.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0011] In this embodiment, the electronic device can construct a haze perception depth image based on a first image and a second image, wherein the first image and the second image have different magnifications. The haze perception depth image includes at least one of pixel depth information, pixel edge information, and pixel brightness information. Then, the electronic device can obtain a haze image feature map based on the haze perception depth image, the first image, and the second image. The haze image feature map is used to indicate the concentration of fog in different areas of the image. The first image and the second image are then reconstructed based on the haze feature map to generate a third image, which is the dehazed version of the first image and the second image. Since the electronic device can construct a haze perception depth image including pixel information based on images with different magnifications and further obtain a haze image feature map, and since the haze image feature map can indicate the concentration of fog in the image, it can determine which areas in the first image and the second image contain fog, and the fog concentration information in those areas. Therefore, based on the fog concentration information in those areas, the image of those areas can be processed to obtain an image without fog, i.e., a high-quality image without fog. This improves the image quality of images captured by the electronic device in hazy environments. Attached Figure Description

[0012] Figure 1 This is a flowchart of an image processing method provided in an embodiment of this application;

[0013] Figure 2(a) is one of the schematic diagrams of a mobile phone interface provided in an embodiment of this application;

[0014] Figure 2(b) is a second example of a mobile phone interface provided in an embodiment of this application;

[0015] Figure 3 This is a structural diagram of a convolutional neural network provided in an embodiment of this application;

[0016] Figure 4 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0017] Figure 5 This is one of the hardware structure diagrams of an electronic device provided in the embodiments of this application;

[0018] Figure 6 This is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0021] Due to pollution, the frequency of smog is increasing. When smog occurs, if users take photos with electronic devices, such as mobile phones, the image quality will be degraded, and this deterioration increases with the concentration of smog. Existing image processing methods include physical methods and deep learning methods, but these methods struggle to fully utilize the depth information inherent in foggy images, potentially leading to unnatural dehazing results.

[0022] Currently, existing mobile phones typically have two cameras. Therefore, the image processing method in this application is based on the dual-camera lens in the mobile phone and uses perceived image depth information to dehaze the image, thereby making full use of the depth, brightness, and edge information of the hazy image.

[0023] When smog occurs, photos taken with a mobile phone suffer from quality degradation, and the greater the smog, the worse the image quality. Existing dehazing methods are divided into physical methods and deep learning methods. Physical methods cannot adaptively adjust parameters for different scenes, and the resulting transmittance maps are difficult to reconstruct from fog-free images. Deep learning methods struggle to effectively learn the characteristics of smog and also find it difficult to reconstruct fog-free images. Neither of these methods achieves satisfactory dehazing results, impacting the shooting experience.

[0024] This invention is based on the dual-camera system of a commonly used mobile phone: simultaneously capturing foggy images using a 1X main camera and a 2X secondary camera, and then using the dual cameras to calculate a fog-perceived depth map. This depth map integrates edge, brightness, and depth information from the foggy image. Next, during the learning process for the foggy image, filter parameters are adaptively adjusted to extract features from areas with heavier fog. Then, based on the fog-perceived depth map, atmospheric light values ​​are calculated after eliminating interference from bright light. Simultaneously, using the foggy image as input and the fog-perceived depth map as a guiding image, a guided filter is used to refine the transmittance map. Finally, a fog-free image is reconstructed using the foggy image, the refined transmittance map, and the atmospheric light values.

[0025] The image dehazing provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0026] This application provides an image processing method. Figure 1 A flowchart illustrating an image processing method provided in an embodiment of this application is shown. This method can be applied to an electronic device, which includes a first camera and a second camera. Figure 1 As shown, the image processing method provided in this application embodiment may include the following steps 201 to 203.

[0027] Step 201: The electronic device constructs a haze perception depth image based on the first image and the second image.

[0028] In this embodiment of the application, the first image and the second image are captured at different magnifications, and the haze perception depth map includes at least one of pixel depth information, pixel edge information and pixel brightness information.

[0029] Optionally, in this embodiment of the application, the electronic device may include two cameras, namely a first camera (1X camera) and a second camera (2X camera). When the user needs to take pictures in a foggy environment, the user can turn on the first camera and the second camera and take pictures simultaneously through the first camera and the second camera to obtain two foggy images, namely the first image and the second image. Then the electronic device can process the two foggy images to obtain one image, and then construct a fog and haze perception depth image based on the one image; or, after obtaining the first image and the second image, the electronic device can construct a fog and haze perception depth image based on the first image and the second image.

[0030] Optionally, in this embodiment of the application, the first image is a foggy image captured by the electronic device using the first camera, and the second image can be a foggy image captured by the second camera.

[0031] Optionally, in this embodiment, the sensor corresponding to the 1X camera has a larger sensor size, allowing for clearer results through the ROI; while the 2X camera uses a 50mm telephoto lens, which has a different focal length than the 1X camera, enabling better acquisition of more accurate depth information. The 1X and 2X cameras are used to design 1X and 2X methods for generating depth information. A 50mm focal length is the most consistent with human eye perception (other mobile phones do not have a 2X sensor; they often use a cropped 1X sensor).

[0032] Optionally, in this embodiment of the application, when the electronic device takes pictures through the first camera and the second camera, it can identify the focal length of the first camera and the second camera online.

[0033] Optionally, in this embodiment of the application, when a user takes a picture of a foggy image, the user can enable the depth map defogging mode and take the picture in this mode; or, the user can go to the photo album of the electronic device, find the image that needs to be defogging, select the depth map defogging function, and thus trigger the electronic device to process the foggy image.

[0034] For example, as shown in Figure 2(a), the electronic device displays a shooting interface, where the user can input settings and select to enable the "Depth Map Dehazing" mode, thereby triggering the electronic device to take a picture in this mode; or, as shown in Figure 2(b), the user selects the image to be dehazed from the images stored in the electronic device, and then the electronic device can display the image. The user can then long-press the image and click the "Depth Map Dehazing" function icon to start the "Depth Map Dehazing" function, triggering the electronic device to dehaze the image.

[0035] Optionally, in the embodiments of this application, the haze perception depth image integrates edge information, brightness information, and depth information of the hazy image.

[0036] Optionally, in the embodiments of this application, step 201 above can be specifically implemented by steps 201a to 201f below.

[0037] Step 201a: The electronic device performs image matching on the first image and the second image to obtain matching information.

[0038] Optionally, in this embodiment of the application, the electronic device may use a first camera and a second camera to perform dual-target positioning, calibration and matching on the first image and the second image to obtain matching information.

[0039] Step 201b: The electronic device obtains the depth information of each pixel in the first image and the second image based on the matching information.

[0040] Optionally, in this embodiment of the application, after obtaining the matching information, the electronic device can obtain the depth information of each pixel in the first image and the corresponding pixel in the second image based on the matching information of the first image and the second image, thereby obtaining the depth information of each pixel in the first image and the second image.

[0041] Step 201c: The electronic device obtains a depth image based on the depth information.

[0042] Optionally, in this embodiment of the application, the depth image includes depth information of each pixel in the first image and the second image.

[0043] Optionally, in this embodiment of the application, the electronic device can obtain the depth information of each pixel in the first image and the second image based on the matching information, and then obtain a depth image based on the depth information, wherein the depth image records the depth information of each pixel in the first image and the second image.

[0044] Step 201d: The electronic device obtains a fog image based on the first image, the second image, and the depth image.

[0045] Optionally, in this embodiment of the application, since haze points in the air may cause too much sunlight and other diffuse reflected light to be captured by the first and second cameras of the electronic device, the electronic device can use the brightness feature differences of haze points to extract haze point features in the first and second images.

[0046] Optionally, in this embodiment of the application, a convolutional neural network can be used to learn from an image without a haze to obtain the feature information of the haze-free image. Then, the learned result, i.e., the feature information of the haze-free image, can be subtracted from the first image and the second image to obtain the haze image.

[0047] Optionally, in this embodiment of the application, the convolutional neural network includes three haze convolutional blocks, namely haze convolutional block 1 to haze convolutional block 3. Each convolutional block includes three convolutional kernels of size 5*5, stride 1, and padding 2, and a ReLU activation function. Figure 3 Provides a structural diagram of a convolutional neural network, such as Figure 3 As shown, the depth image and the first and second images after convolution by haze convolution blocks 1 to 3 are fused to obtain the fused feature Fd. The fused feature Fd is input into the perceptual convolution block I to obtain the haze image. The perceptual convolution block I contains two 5×5 convolution kernels with a stride of 1, a padding of 2, and a ReLU activation function.

[0048] Optionally, in this embodiment of the application, the fog image includes fog information for each pixel in the first image and the second image.

[0049] Step 201e: The electronic device performs edge detection on the first image and the second image to obtain an edge-detected image.

[0050] Optionally, in this embodiment of the application, the electronic device may use the Canny operator to perform edge detection on the first image and the second image to form an edge detection map, wherein the edge detection map records the edge information of each pixel in the first image and the second image.

[0051] Optionally, in this embodiment of the application, the fog image includes edge information of each pixel in the first image and the second image.

[0052] Step 201f: The electronic device constructs a haze perception depth image based on the depth image, fog image, and edge detection image.

[0053] Optionally, in this embodiment of the application, the electronic device can integrate the depth image, the fog image, and the edge detection image to construct a comprehensive perception map, namely, a fog and haze perception depth image.

[0054] Optionally, in this embodiment of the application, the electronic device can use the Merge function to overlay information from the depth image, the fog image, and the edge detection image to construct a fog perception depth image.

[0055] The Merge function is as follows:

[0056] I perception =Merge(I perception1 ,I perception2 ,I perception3 );

[0057] It should be noted that I perception1 For depth images, I perception2 For foggy images, I perception3 This is an edge detection image.

[0058] Step 202: The electronic device obtains a haze image feature map based on the haze perception depth image, the first image, and the second image.

[0059] In this embodiment of the application, the above-mentioned haze image feature map is used to indicate the concentration of fog in different areas of the image.

[0060] Optionally, in this embodiment of the application, the fog perception depth map includes edge information, brightness information and depth information. Based on this information, the concentration information of fog-containing areas in the first image and the second image, or the concentration information of fog in different areas, can be calculated, providing a learning basis for fog image filtering methods.

[0061] Optionally, in the embodiments of this application, step 202 can be implemented by steps 202a and 202b as described below.

[0062] Step 202a: The electronic device obtains concentration information based on the haze sensing depth map.

[0063] In this embodiment of the application, the concentration information is used to indicate the concentration value of fog in the first image and the second image.

[0064] Optionally, in this embodiment of the application, the electronic device can calculate the fog concentration information in different regions of the first image and the second image based on at least one of the pixel depth information, pixel edge information and pixel brightness information included in the fog perception depth map.

[0065] Optionally, in this embodiment of the application, the electronic device can divide the first image and the second image into M regions respectively, and then calculate the fog concentration information of each region in the first image and the second image according to the fog perception depth map; or, the electronic device can divide the fog-included regions in the first image and the second image into regions according to the fog perception depth map, and then calculate the fog concentration information of different regions in the first image and the second image.

[0066] Step 202b: The electronic device inputs the first image and the second image into the Gaussian filter function, and obtains the haze image feature map based on the concentration information.

[0067] Optionally, in this embodiment of the application, the electronic device uses a Gaussian filtering function based on the concentration information, takes the first image and the second image as input images, and adjusts the neighborhood range parameter and standard deviation parameter to obtain a haze image feature map.

[0068] Optionally, in this embodiment, the neighborhood range parameter and the standard deviation parameter are parameters in the Gaussian filter function.

[0069] Optionally, in this embodiment, the electronic device can dynamically adjust the intensity information of the Gaussian kernel parameter in the Gaussian filter function based on the concentration information, thereby achieving adaptive filtering. The first and second images are used as input images to ultimately obtain a haze image feature map, i.e., a haze image feature map I. feature .

[0070] Among them, there is a fog image feature map I feature It can be obtained using the following formula.

[0071] I feature =Dehaze_filter(in,gausFilter);

[0072] Here, Dehaze_fliter represents the filtering function using a Gaussian kernel, in represents the input, and gausFilter represents the Gaussian kernel function, which includes the neighborhood range and standard deviation. These two parameters can be dynamically adjusted based on the fog concentration information.

[0073] Step 203: The electronic device performs image reconstruction processing on the first and second images based on the haze feature images to generate the third image.

[0074] In this embodiment of the application, the third image is the image after dehazing the first image and the second image.

[0075] This application provides an image processing method. An electronic device can construct a haze perception depth image based on a first image and a second image, wherein the first image and the second image have different magnifications. The haze perception depth image includes at least one of pixel depth information, pixel edge information, and pixel brightness information. Then, the electronic device can obtain a haze image feature map based on the haze perception depth image, the first image, and the second image. The haze image feature map is used to indicate the concentration of fog in different areas of the image. The first image and the second image are then reconstructed based on the haze feature image to generate a third image, which is the image after dehazing the first image and the second image. Because electronic devices can construct a haze perception depth image including pixel information based on images captured at different magnifications, and further obtain a haze image feature map, since this haze image feature map can indicate the concentration of fog in the image, it is possible to determine which areas in the first and second images contain fog, as well as the fog concentration information in those areas. Therefore, based on the fog concentration information in those areas, the image of those areas is processed to obtain an image without fog, that is, a high-quality image-free image can be obtained. In this way, the image quality of images captured by electronic devices in hazy environments is improved.

[0076] Optionally, in the embodiments of this application, step 203 can be implemented by steps 203a and 203b as described below.

[0077] Step 203a: The electronic device obtains the first atmospheric light value and the first transmittance estimation image based on the haze image feature map.

[0078] Optionally, in this embodiment of the application, the first transmittance estimation image can be a coarse transmittance map, that is, an image after a coarse estimate of the transmittance of the first image and the second image.

[0079] Optionally, in the embodiments of this application, step 203a can be implemented by the following steps 203a1 and 203a2.

[0080] Step 203a1: The electronic device obtains the first atmospheric light value based on the haze image feature map.

[0081] Optionally, in the embodiments of this application, step 203a1 can be implemented by steps 11 to 16 as described below.

[0082] Step 11: The electronic device uses a fog image learning network to divide the fog image feature map into N regions according to the fog concentration.

[0083] Where N is an integer greater than 1.

[0084] Optionally, in this embodiment of the application, the electronic device obtains the haze concentration region in the haze image feature map based on the haze image learning network, and divides the different regions into R1 to RN, so that the haze image feature map is divided into N regions according to the haze concentration.

[0085] Step 12: For each of the N regions, the electronic device calculates the dark channel map for each region.

[0086] Optionally, in this embodiment of the application, the electronic device locates a certain region RI and calculates the dark channel map R of that region. dark .

[0087] Step 13: The electronic device determines, based on the dark channel map of each region, at least one atmospheric light value in the dark channel map of each region that has a brightness value greater than a preset threshold.

[0088] Optionally, in this embodiment of the application, the electronic device determines the atmospheric light value with a brightness value of the top one-thousandth in the region RI based on the dark channel map of each region.

[0089] Step 14: For each atmospheric light value in at least one atmospheric light value, the electronic device determines the pixel value of the coordinates corresponding to each atmospheric light value to obtain at least one pixel value.

[0090] Optionally, in this embodiment of the application, the electronic device locates the corresponding pixel in region RI for each atmospheric light value in the first one-thousandth of atmospheric light values, so as to obtain a pixel value corresponding to each atmospheric light value, thereby obtaining at least one pixel value corresponding to at least one atmospheric light value.

[0091] Step 15: The electronic device determines the atmospheric light value of each region based on at least one atmospheric light value and at least one pixel value.

[0092] Optionally, in this embodiment of the application, the electronic device adds each atmospheric light value to the coordinate value of its corresponding pixel to obtain at least one summed value. The average value of the at least one summed value is taken to determine an atmospheric light value AI, which is the atmospheric light value corresponding to region RI. Thus, the atmospheric light values ​​A1 to AN corresponding to regions R1 to RN can be obtained, that is, the N atmospheric light values ​​corresponding to the N regions respectively.

[0093] Step 16: The electronic device obtains the first atmospheric light value based on the atmospheric light value of each region.

[0094] Optionally, in this embodiment of the application, the electronic device integrates A1 to AN and takes the average value of A1 to AN, thereby achieving the first atmospheric light value A. final .

[0095] Step 203a2: The electronic device obtains a first transmittance estimation image based on the haze image feature map and the first atmospheric light value.

[0096] Optionally, in the embodiments of this application, step 203a2 can be implemented by steps 21 and 22 as described below.

[0097] Step 11: The electronic device obtains the transmittance estimation image corresponding to the atmospheric light value of each region based on the haze image feature map and the atmospheric light value of each region included in the first atmospheric light value.

[0098] Optionally, in this embodiment of the application, the electronic device uses the haze image feature map as the input image and obtains a rough transmittance map corresponding to the atmospheric light value of each region based on the atmospheric light value of each region in the first atmospheric light value.

[0099] Optionally, in this embodiment of the application, the electronic device uses the haze image feature map as the input image, and based on the atmospheric light value of each region, such as the atmospheric light value AI of region RI, obtains the coarse filter transmittance map TI corresponding to region RI based on the dark channel prior theory, so as to obtain the coarse transmittance maps T1 to TN corresponding to regions R1 to RN, that is, N coarse transmittance maps corresponding to N regions respectively.

[0100] Step 12: The electronic device obtains the first transmittance estimation image based on the transmittance estimation image corresponding to the atmospheric light value of each region.

[0101] Optionally, in this embodiment of the application, the electronic device obtains a first transmittance estimation image based on a coarse transmittance map corresponding to the atmospheric light value of each region.

[0102] Optionally, in this embodiment of the application, the electronic device can obtain a coarse transmittance map consisting of T1 to TN from N coarse transmittance maps, which is the first transmittance estimation image.

[0103] Step 203b: The electronic device performs image reconstruction processing on the first image and the second image based on the haze perception depth image, the first transmittance estimation image and the first atmospheric light value, to generate a third image.

[0104] Optionally, in the embodiments of this application, step 203b can be implemented by the following steps 203b1 and 203b2.

[0105] Step 203b1: The electronic device uses the haze perception depth map as the guiding image and the first transmittance estimation image as the input image. Based on fast guided filtering, the transmittance of the first transmittance estimation image is refined to obtain the second transmittance estimation image.

[0106] Optionally, in this embodiment of the application, the electronic device uses the haze perception depth map as the guiding image and the first transmittance estimation image as the input image. Based on fast guided filtering, the transmittance of the first transmittance estimation image is refined to obtain the image with refined transmittance, which is the second transmittance estimation image.

[0107] Optionally, in this embodiment of the application, the electronic device uses the haze perception depth map as the guiding image and the first transmittance estimation image as the input image. Based on fast guided filtering, the transmittance of the first transmittance estimation image is refined to obtain the second transmittance estimation image.

[0108] Optionally, in this embodiment of the application, the electronic device can use a haze perception depth map as the guiding image and a first transmittance estimation image as the input image. Based on fast guided filtering, the transmittance of the first transmittance estimation image is refined to obtain a fourth image t. final .

[0109] Step 203b2: Based on the dark channel prior theory, the electronic device uses the second transmittance estimation image as the guiding image, and performs image reconstruction processing on the first image and the second image according to the first atmospheric light value to generate the third image.

[0110] Optionally, in this embodiment of the application, the electronic device uses the second transmittance estimation image as a guide image based on the dark channel prior theory, and performs image reconstruction processing on the first image and the second image according to the first atmospheric light value to generate the third image.

[0111] Optionally, in this embodiment of the application, the electronic device can use the second transmittance estimation image as the guide image, the first image and the second image as the input images, and reconstruct the third image, i.e. the fog-free image, based on the first atmospheric light value of the dark channel prior theory.

[0112] Optionally, in this embodiment of the application, the electronic device can obtain a fog-free image using the following formula.

[0113] J(x) = DarkChannel(t) final A final );

[0114] Where DarkChannel represents the prior reconstruction formula for the dark channel.

[0115] The image processing method provided in this application can be executed by an image processing device. This application uses an image processing device executing the image processing method as an example to illustrate the image processing device provided in this application.

[0116] Figure 4 A schematic diagram of a possible structure of the image processing apparatus involved in an embodiment of this application is shown. For example... Figure 4 As shown, the image processing device 40 may include: a building module 41 and a processing module 42.

[0117] The construction module 41 is used to construct a haze perception depth image based on the first image and the second image. The first image and the second image have different magnifications. The haze perception depth image includes at least one of pixel depth information, pixel edge information, and pixel brightness information. The processing module 42 is used to obtain a haze image feature map based on the haze perception depth image, the first image, and the second image. The haze image feature map is used to indicate the concentration of haze in different areas of the image. The module also performs image reconstruction processing on the first image and the second image based on the haze feature map to generate a third image, which is the dehazed version of the first image and the second image.

[0118] This application provides an image processing device. Since the electronic device can construct a haze perception depth image including pixel information based on images captured at different magnifications, and further obtain a haze image feature map, and since this haze image feature map can indicate the concentration of haze in the image's haze-containing areas, it can determine which areas in the first and second images contain haze, and the haze concentration information in those areas. Therefore, based on the haze concentration information of the haze-containing areas, the image of those areas is processed to obtain an image without haze, i.e., a high-quality image-free image. This improves the image quality of images captured by the electronic device in hazy environments.

[0119] In one possible implementation, the processing module 42 is specifically used to obtain a first atmospheric light value and a first transmittance estimation image based on the feature map of the haze image; and to perform image reconstruction processing on the first image and the second image based on the haze perception depth image, the first transmittance estimation image and the first atmospheric light value to generate a third image.

[0120] In one possible implementation, the construction module 41 is specifically used to perform image matching on the first image and the second image to obtain matching information; based on the matching information, to obtain depth information of each pixel in the first image and the second image; based on the depth information, to obtain a depth image; based on the first image, the second image, and the depth image, to obtain a fog image; to perform edge detection on the first image and the second image to obtain an edge detection image; and based on the depth image, the fog image, and the edge detection image, to construct a haze perception depth image.

[0121] In one possible implementation, the processing module 42 is specifically used to obtain concentration information based on the haze perception depth map; the concentration information is used to indicate the concentration value of the fog in the first image and the second image; and the first image and the second image are input into a Gaussian filtering function to obtain a haze image feature map based on the concentration information.

[0122] In one possible implementation, the processing module 42 is specifically used to obtain a first atmospheric light value based on the haze image feature map; and to obtain a first transmittance estimation image based on the haze image feature map and the first atmospheric light value.

[0123] In one possible implementation, the processing module 42 is specifically used to obtain a transmittance estimation image corresponding to the atmospheric light value of each region based on the feature map of the haze image and the atmospheric light value of each region included in the first atmospheric light value; and to obtain a first transmittance estimation image based on the transmittance estimation image corresponding to the atmospheric light value of each region.

[0124] In one possible implementation, the processing module 42 is specifically used to take the haze perception depth map as the guiding image and the first transmittance estimation image as the input image, refine the transmittance of the first transmittance estimation image based on fast guided filtering to obtain a second transmittance estimation image; and based on the dark channel prior theory, use the second transmittance estimation image as the guiding image, and perform image reconstruction processing on the first image and the second image according to the first atmospheric light value to generate a third image.

[0125] The image processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0126] The image processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0127] The image processing apparatus provided in this application embodiment can implement the various processes implemented in the above method embodiment, and will not be described again here to avoid repetition.

[0128] Optionally, such as Figure 5 As shown, this application embodiment also provides an electronic device 6000, including a processor 6001 and a memory 6002. The memory 6002 stores a program or instructions that can run on the processor 6001. When the program or instructions are executed by the processor 6001, they implement the various steps of the above-described image processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0129] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0130] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0131] The electronic device 100 includes, but is not limited to, components such as: radio frequency unit 101, network module 102, audio output unit 103, input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, and processor 110.

[0132] Those skilled in the art will understand that the electronic device 100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0133] The processor 110 is used to construct a haze perception depth image based on a first image and a second image. The first image and the second image have different magnifications. The haze perception depth image includes at least one of pixel depth information, pixel edge information, and pixel brightness information. Then, based on the haze perception depth image, the first image, and the second image, a haze image feature map is obtained. The haze image feature map is used to indicate the concentration of fog in different areas of the image. The processor 110 is used to perform image reconstruction processing on the first image and the second image based on the haze feature image to generate a third image. The third image is the image after dehazing the first image and the second image.

[0134] This application provides an electronic device that can construct a haze perception depth image including pixel information based on images captured at different magnifications, and further obtain a haze image feature map. Since the haze image feature map can indicate the concentration of areas containing fog in the image, it can determine which areas in the first and second images contain fog, as well as the fog concentration information in those areas. Based on the fog concentration information in those areas, the image of those areas is processed to obtain an image without fog, i.e., a high-quality image-free image. This improves the image quality of images captured by the electronic device in hazy environments.

[0135] Optionally, the processor 110 is specifically configured to obtain a first atmospheric light value and a first transmittance estimation image based on the feature map of the haze image; and to perform image reconstruction processing on the first image and the second image based on the haze perception depth image, the first transmittance estimation image and the first atmospheric light value to generate a third image.

[0136] Optionally, the processor 110 is specifically configured to perform image matching on the first image and the second image to obtain matching information; obtain depth information of each pixel in the first image and the second image based on the matching information; obtain a depth image based on the depth information; obtain a fog image based on the first image, the second image and the depth image; perform edge detection on the first image and the second image to obtain an edge detection image; and construct a haze perception depth image based on the depth image, the fog image and the edge detection image.

[0137] Optionally, the processor 110 is specifically used to obtain concentration information based on the haze perception depth map; the concentration information is used to indicate the concentration value of the fog in the first image and the second image; and input the first image and the second image into a Gaussian filtering function to obtain a haze image feature map based on the concentration information.

[0138] Optionally, the processor 110 is specifically configured to obtain a first atmospheric light value based on the haze image feature map; and to obtain a first transmittance estimation image based on the haze image feature map and the first atmospheric light value.

[0139] Optionally, the processor 110 is specifically configured to obtain a transmittance estimation image corresponding to the atmospheric light value of each region based on the feature map of the haze image and the atmospheric light value of each region included in the first atmospheric light value; and to obtain a first transmittance estimation image based on the transmittance estimation image corresponding to the atmospheric light value of each region.

[0140] Optionally, the processor 110 is specifically configured to use the haze perception depth map as the guiding image and the first transmittance estimation image as the input image, refine the transmittance of the first transmittance estimation image based on fast guided filtering to obtain a second transmittance estimation image; and based on the dark channel prior theory, use the second transmittance estimation image as the guiding image, and perform image reconstruction processing on the first image and the second image according to the first atmospheric light value to generate a third image.

[0141] It should be understood that, in this embodiment, the input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 107 includes at least one of a touch panel 1071 and other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0142] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0143] Processor 110 may include one or more processing units; optionally, processor 110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 110.

[0144] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described image processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0145] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0146] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described image processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0147] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0148] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described image processing method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0149] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0151] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: Based on the first image and the second image, a haze perception depth image is constructed. The first image and the second image have different magnifications. The first image and the second image are images obtained by two cameras simultaneously. The haze perception depth image includes pixel depth information and also includes at least one of pixel edge information and pixel brightness information. Based on the haze perception depth image, the first image, and the second image, a haze image feature map is obtained, which is used to indicate the concentration of haze in different areas of the image; Based on the haze image feature map, the first atmospheric light value and the first transmittance estimation image are obtained; Using the haze perception depth map as the guiding image and the first transmittance estimation image as the input image, the transmittance of the first transmittance estimation image is refined based on fast guided filtering to obtain the second transmittance estimation image. Based on the dark channel prior theory, the second transmittance estimation image is used as the guiding image. Based on the first atmospheric light value, the first image and the second image are reconstructed to generate the third image.

2. The method according to claim 1, characterized in that, The step of constructing a haze perception depth map based on the first image and the second image includes: Image matching is performed on the first image and the second image to obtain matching information; Based on the matching information, the depth information of each pixel in the first image and the second image is obtained; Based on the depth information, a depth image is obtained; A fog image is obtained based on the first image, the second image, and the depth image; Edge detection is performed on the first image and the second image to obtain an edge-detected image; The haze perception depth image is constructed based on the depth image, the fog image, and the edge detection image.

3. The method according to claim 1, characterized in that, The step of obtaining a haze image feature map based on the haze perception depth map, the first image, and the second image includes: Based on the haze perception depth map, concentration information is obtained; the concentration information is used to indicate the concentration value of fog in the first image and the second image; The first image and the second image are input into a Gaussian filter function, and the haze image feature map is obtained based on the concentration information.

4. The method according to claim 1, characterized in that, The step of obtaining the first atmospheric light value and the first transmittance estimation image based on the haze image feature map includes: The first atmospheric light value is obtained based on the haze image feature map; The first transmittance estimation image is obtained based on the haze image feature map and the first atmospheric light value.

5. The method according to claim 4, characterized in that, The step of obtaining the first transmittance estimation image based on the haze image feature map and the first atmospheric light value includes: Based on the haze image feature map and the atmospheric light value of each region included in the first atmospheric light value, a transmittance estimation image corresponding to the atmospheric light value of each region is obtained; The first transmittance estimation image is obtained based on the transmittance estimation image corresponding to the atmospheric light value of each region.

6. An image fogging device, characterized in that, The device includes: a construction module and a processing module; the construction module is used to construct a haze perception depth image based on a first image and a second image, wherein the first image and the second image have different magnifications and are images obtained simultaneously by two cameras; the haze perception depth image includes pixel depth information and also includes at least one of pixel edge information and pixel brightness information. The processing module is used to obtain a haze image feature map based on the haze perception depth image, the first image, and the second image. The haze image feature map is used to indicate the concentration of haze in different areas of the image. The processing module is further configured to obtain a first atmospheric light value and a first transmittance estimation image based on the haze image feature map; The processing module is further configured to use the haze perception depth map as a guide image and the first transmittance estimation image as an input image, and refine the transmittance of the first transmittance estimation image based on fast guided filtering to obtain a second transmittance estimation image. The processing module is further configured to use the second transmittance estimation image as a guide image based on the dark channel prior theory, and to perform image reconstruction processing on the first image and the second image according to the first atmospheric light value to generate a third image.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the image processing method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the image processing method as described in any one of claims 1 to 5.

Images