Image processing method and device, electronic device, and storage medium

By filtering the brightness map maximum value and calculating the weight map of multi-frame images, and combining pyramid technology for image fusion, the problem of black edges in images in night scenes or darker light scenes is solved, improving image quality and user experience.

CN112651918BActive Publication Date: 2024-11-22BEIJING XIAOMI PINECONE ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110055602.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-01-15
Publication Date
2024-11-22
Estimated Expiration
2041-01-15

AI Technical Summary

Technical Problem

In night scenes or darker scenes, the exposure differences between multiple frames of images are large, resulting in areas with high brightness/contrast in the fused image that are prone to black edges, affecting image quality.

Method used

By filtering the brightness map of multiple frame images, the bright area map is determined, the weight map is calculated based on the bright area map, and the image fusion is combined with the Laplace pyramid and the weight pyramid to suppress negative Laplace brightness values ​​and reduce black Edge phenomenon.

Benefits of technology

It effectively reduces the black edge phenomenon in the fused image and improves the image display effect. It is especially suitable for night scenes or darker shooting and video synthesis, improving the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112651918B_ABST
    Figure CN112651918B_ABST
Patent Text Reader

Abstract

The present disclosure is about an image processing method and device, an electronic device, and a storage medium. The method includes: performing a predetermined number of maximum filtering on the brightness map of the first image in the multi-frame image to obtain a first bright area map; determining a first weight map based on the first bright area map; determining a target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multi-frame image, and the second image corresponding to each image in the multi-frame image; wherein the second image corresponding to each image is the image corresponding to the layer with the largest image resolution in the Laplacian pyramid corresponding to each image; and the first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image. The present disclosure realizes the image effect when shooting images, is more suitable for night scene image photography, improves the display effect of images or videos, and improves user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to image fusion technology, and in particular to an image processing method and device, an electronic device, and a storage medium. Background Art

[0002] At present, in order to improve the brightness of the captured image, multiple images with different exposures are generally obtained, and the multiple frames of images with different exposures are fused according to the corresponding weights to generate a fused image with better dynamic range, color, noise, details, etc., thereby improving the quality of the captured image and improving the user experience. However, when shooting images in night scenes, due to the large exposure differences of the input multiple frames, the brightness differences between the foreground and background in the image are also large, and black edges are easily generated at the edges of the areas with high brightness / contrast in the fused image. Summary of the invention

[0003] The present disclosure provides an image processing method and device, an electronic device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0005] Performing maximum value filtering for a predetermined number of times on the brightness map of the first image in the multiple frames of images to obtain a first bright area map;

[0006] determining a first weight map based on the first bright area map;

[0007] A target image is determined according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images; wherein the second image corresponding to each image is an image corresponding to a layer with the largest image resolution in the Laplacian pyramid corresponding to each image; and the first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

[0008] Optionally, determining a first weight map based on the first bright area map includes:

[0009] Obtaining a weight curve and a second image corresponding to the first image; wherein the weight curve is a functional relationship between a weight and a pixel brightness value in an image;

[0010] The first weight map is determined according to the second image, the first bright area map and the weight curve.

[0011] Optionally, determining the first weight map according to the second image, the first bright area map and the weight curve includes:

[0012] If the brightness value of the pixel in the second image is less than 0, the brightness weight value of the pixel in the first bright area image is used as the corresponding weight in the first weight image, wherein the brightness weight value of the pixel in the first bright area image is: calculating the square of the difference between the brightness of the pixel in the first bright area image and the first set value, calculating the inverse of the ratio of the square to the second set value, and using e as the base to obtain the exponent of the inverse as the brightness weight value of the pixel in the first bright area image;

[0013] If the brightness value of a pixel in the second image is greater than or equal to 0, the weight of the pixel greater than or equal to 0 in the first weight map is 1.

[0014] Optionally, determining the target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images includes:

[0015] Performing upsampling processing on the first fused image to obtain a first fused image after upsampling processing;

[0016] Obtaining a third image according to the first weight map, the original weight map corresponding to the first image, and the second image corresponding to the first image;

[0017] Performing weighted processing on the original weight map corresponding to other images in the multiple frames of images except the first image and the second image to obtain a fourth image;

[0018] The first fused image after the downsampling process, the third image and the fourth image are fused to obtain the target image.

[0019] Optionally, before determining the target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images, the method further includes:

[0020] Obtaining a Laplacian pyramid corresponding to each image and a weight pyramid corresponding to each image;

[0021] The first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

[0022] Optionally, determining the first fused image according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image includes:

[0023] For each layer of the Laplacian pyramid corresponding to each image, the following processing is performed: acquiring the image of the layer in the Laplacian pyramid corresponding to each image to obtain the image corresponding to the layer for each image; acquiring the weight map of the layer in the weight pyramid corresponding to each image to obtain the weight map corresponding to the layer for each image; weighting the image corresponding to the layer for each image and the weight map corresponding to the layer for each image to obtain a fused image corresponding to the layer; wherein the fused images corresponding to all the layers form a first pyramid;

[0024] The first fused image is determined according to the first pyramid.

[0025] Optionally, determining the first fused image according to the first pyramid includes:

[0026] Starting from N being the first specified number, loop through the following steps until N is the second specified number:

[0027] Upsampling the image corresponding to the Nth layer in the first pyramid to obtain an upsampled image;

[0028] Fusing the upsampled image with the image corresponding to the N-1th layer in the first pyramid to obtain a fused image corresponding to the N-1th layer; updating the image corresponding to the N-1th layer to the fused image corresponding to the N-1th layer, and performing N minus one;

[0029] The first designated number is the layer number of the layer with the lowest resolution in the first pyramid, the second designated number is the layer number of the layer with the highest resolution in the first pyramid + 1, and N is an integer greater than the second designated number and less than the first designated number.

[0030] The first fused image is the fused image corresponding to the Nth layer obtained after the cycle is completed.

[0031] Optionally, obtaining the Laplacian pyramid corresponding to each image includes:

[0032] Down-sampling each of the images respectively to obtain a Gaussian pyramid layer of each of the images;

[0033] The current Laplacian pyramid layer of each image is obtained by:

[0034] Upsampling a Gaussian pyramid layer above the current layer of each image to obtain an upsampled layer, and performing Gaussian blur processing on the upsampled layer to obtain a first Gaussian pyramid layer;

[0035] Performing a subtraction operation on the current Gaussian pyramid layer of each image and the first Gaussian pyramid layer to obtain a current Laplacian pyramid layer;

[0036] The Laplacian pyramid layers obtained layer by layer are combined to form a Laplacian pyramid corresponding to each image.

[0037] Optionally, obtaining a weight pyramid corresponding to each image includes:

[0038] Obtaining an original weight map corresponding to each image in the multiple frames of images, and downsampling the original weight map of each image to obtain a weight map of the second layer of each image;

[0039] The weight map of the second layer of each image is downsampled again to obtain the weight map of the third layer of each image, until a weight map corresponding to each layer of the Laplacian pyramid of each image is obtained.

[0040] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0041] A filtering unit, configured to perform a maximum value filtering for a predetermined number of times on the brightness map of the first image in the multiple frames of images to obtain a first bright area map;

[0042] A first determining unit, configured to determine a first weight map based on the first bright area map;

[0043] The second determination unit is used to determine the target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images; wherein the second image corresponding to each image is the image corresponding to the layer with the largest image resolution in the Laplacian pyramid corresponding to each image; and the first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

[0044] Optionally, the first determining unit is further configured to:

[0045] Obtaining a weight curve and a second image corresponding to the first image; wherein the weight curve is a functional relationship between a weight and a pixel brightness value in an image;

[0046] The first weight map is determined according to the second image, the first bright area map and the weight curve.

[0047] Optionally, the first determining unit is further configured to:

[0048] If the brightness value of the pixel in the second image is less than 0, the brightness weight value of the pixel in the first bright area image is used as the corresponding weight in the first weight image, wherein the brightness weight value of the pixel in the first bright area image is: calculating the square of the difference between the brightness of the pixel in the first bright area image and the first set value, calculating the inverse of the ratio of the square to the second set value, and using e as the base to obtain the exponent of the inverse as the brightness weight value of the pixel in the first bright area image;

[0049] If the brightness value of a pixel in the second image is greater than or equal to 0, the weight of the pixel greater than or equal to 0 in the first weight map is 1.

[0050] Optionally, the second determining unit is further configured to:

[0051] Performing upsampling processing on the first fused image to obtain a first fused image after upsampling processing;

[0052] Obtaining a third image according to the first weight map, the original weight map corresponding to the first image, and the second image corresponding to the first image;

[0053] Performing weighted processing on the original weight map corresponding to other images in the multiple frames of images except the first image and the second image to obtain a fourth image;

[0054] The first fused image after the downsampling process, the third image and the fourth image are fused to obtain the target image.

[0055] Optionally, the device further comprises:

[0056] An acquisition unit, used to acquire a Laplacian pyramid corresponding to each image and a weight pyramid corresponding to each image;

[0057] The third determining unit is configured to determine the first fused image according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

[0058] Optionally, the third determining unit is further configured to:

[0059] For each layer of the Laplacian pyramid corresponding to each image, the following processing is performed: acquiring the image of the layer in the Laplacian pyramid corresponding to each image to obtain the image corresponding to the layer for each image; acquiring the weight map of the layer in the weight pyramid corresponding to each image to obtain the weight map corresponding to the layer for each image; weighting the image corresponding to the layer for each image and the weight map corresponding to the layer for each image to obtain a fused image corresponding to the layer; wherein the fused images corresponding to all the layers form a first pyramid;

[0060] The first fused image is determined according to the first pyramid.

[0061] Optionally, the third determining unit is further configured to:

[0062] Starting from N being the first specified number, loop through the following steps until N is the second specified number:

[0063] Upsampling the image corresponding to the Nth layer in the first pyramid to obtain an upsampled image;

[0064] Fusing the upsampled image with the image corresponding to the N-1th layer in the first pyramid to obtain a fused image corresponding to the N-1th layer; updating the image corresponding to the N-1th layer to the fused image corresponding to the N-1th layer, and performing N minus one;

[0065] The first designated number is the layer number of the layer with the lowest resolution in the first pyramid, the second designated number is the layer number of the layer with the highest resolution in the first pyramid + 1, and N is an integer greater than the second designated number and less than the first designated number.

[0066] The first fused image is the fused image corresponding to the Nth layer obtained after the cycle is completed.

[0067] Optionally, the acquisition unit is further used to:

[0068] Down-sampling each of the images respectively to obtain a Gaussian pyramid layer of each of the images;

[0069] The current Laplacian pyramid layer of each image is obtained by:

[0070] Upsampling a Gaussian pyramid layer above the current layer of each image to obtain an upsampled layer, and performing Gaussian blur processing on the upsampled layer to obtain a first Gaussian pyramid layer;

[0071] Performing a subtraction operation on the current Gaussian pyramid layer of each image and the first Gaussian pyramid layer to obtain a current Laplacian pyramid layer;

[0072] The Laplacian pyramid layers obtained layer by layer are combined to form a Laplacian pyramid corresponding to each image.

[0073] Optionally, the acquisition unit is further used to:

[0074] Obtaining an original weight map corresponding to each image in the multiple frames of images, and downsampling the original weight map of each image to obtain a weight map of the second layer of each image;

[0075] The weight map of the second layer of each image is downsampled again to obtain the weight map of the third layer of each image, until a weight map corresponding to each layer of the Laplacian pyramid of each image is obtained.

[0076] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor and a memory for storing processor executable instructions, wherein the processor is configured to execute the steps of the above-mentioned image processing method when calling the executable instructions in the memory.

[0077] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps of the above-mentioned image processing method.

[0078] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0079] In an embodiment of the present disclosure, when shooting an image, if the ambient light is too dark or the current scene is night scene shooting, a bright area map is determined in the root of multiple exposure frames, a first weight map is determined based on the first bright area map, and a target image is determined according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames, and the second image corresponding to each image in the multiple frames; through the aforementioned fusion processing, the black edge phenomenon of the fused image is greatly reduced, and the display effect of the fused image is further improved. The embodiment of the present disclosure realizes the image effect when shooting an image, and is more suitable for image photography and video synthesis in night scenes or when the light is low, thereby improving the display effect of the image or video and improving the user experience.

[0080] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0082] Figure 1 A schematic diagram of an image processing method flow chart shown in an embodiment of the present disclosure;

[0083] Figure 2 A schematic diagram of the synthesis of two frames of images;

[0084] Figure 3 This is a schematic diagram of the Gaussian pyramid layer;

[0085] Figure 4A schematic diagram of an image processing method flow chart shown in an embodiment of the present disclosure;

[0086] Figure 5 A schematic diagram of the structure of an image processing device according to an embodiment of the present disclosure;

[0087] Figure 6 A block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0088] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and apparatuses consistent with some aspects of the present disclosure as detailed in the appended claims.

[0089] Figure 1 FIG. 1 is a flow chart of an image processing method according to an embodiment of the present disclosure. Figure 1 As shown, the image processing method of the embodiment of the present disclosure includes the following steps:

[0090] S11, performing maximum value filtering for a predetermined number of times on the brightness map of the first image in the multiple frames of images to obtain a first bright area map.

[0091] The multiple exposure images are fused from the highest layer to the second layer according to the upsampling method to obtain the first image.

[0092] In the embodiment of the present disclosure, the first image of multi-frame exposure fusion refers to an image that is fused through two or more images with different exposures. The different exposure images may include under-exposed images and may also include over-exposed images. According to the weight values ​​of the brightness of the different exposure images, multiple frame images are fused to generate a fused image.

[0093] In the embodiment of the present disclosure, a brightness Laplacian pyramid layer is constructed for each image in the multiple exposure images. Here, constructing the Laplacian pyramid layer of the image includes:

[0094] Each of the images is downsampled to obtain a Gaussian pyramid layer of each image; specifically, firstly, a Gaussian blur process is performed on the image of the next layer of each image in the multiple exposure images, and then the even rows and columns of the image after Gaussian blur are deleted to obtain a Gaussian pyramid layer of the current layer. The aforementioned Gaussian pyramid layer process is performed on each of the images in sequence for a set number of times, so as to obtain a Gaussian pyramid layer of a set number of layers of each image. For example, according to the accuracy requirements of image processing, any image can be processed with a Gaussian pyramid layer, such as an image can be processed into a Gaussian pyramid layer of 8 layers, 4 layers, 16 layers, etc.

[0095] The current Laplacian pyramid layer of each image is obtained by:

[0096] Upsample the Gaussian pyramid layer above the current layer of each image to obtain an upsampled layer, perform Gaussian blur processing on the upsampled layer to obtain a first Gaussian pyramid layer; subtract the current Gaussian pyramid layer of each image from the first Gaussian pyramid layer to obtain the current Laplacian pyramid layer; combine the Laplacian pyramid layers obtained layer by layer to form a Laplacian pyramid corresponding to each image. In other words, the Laplacian pyramid layer is composed of a series of images obtained by subtracting the source image from the image that is first reduced and then enlarged, i.e., upsampled. The upsampling process here includes: mapping the pixel at position (x, y) in the image to be upsampled to the position (2x+1, 2y+1) of the target image, and filling the area in the target image where no pixels are mapped with 0, i.e., upsampling the image to be processed. Figure 3 As shown, Figure 3 A schematic diagram of a Gaussian pyramid layer is shown.

[0097] In the embodiment of the present disclosure, the method further includes obtaining a weight pyramid corresponding to each image, which specifically includes:

[0098] Obtaining an original weight map corresponding to each image in the multiple frames of images, and downsampling the original weight map of each image to obtain a weight map of the second layer of each image;

[0099] The weight map of the second layer of each image is downsampled again to obtain the weight map of the third layer of each image, until a weight map corresponding to each layer of the Laplacian pyramid of each image is obtained.

[0100] In the disclosed embodiment, fusion weights are also constructed for the Laplacian pyramid layers in the multiple exposure images; when image fusion is performed, fusion is performed layer by layer according to the fusion weights, and then the images fused layer by layer are fused. Figure 2 A schematic diagram of the synthesis of two frames of images, such as Figure 2As shown in the figure, the leftmost part of the figure is the input original image. The image to be processed is decomposed by Laplacian pyramid (see the white box and its outer area in the second picture on the left of the figure). Correspondingly, a Gaussian pyramid corresponding to the weight map is constructed (see the white box and its outer area in the third picture on the left of the figure). Then, the weight pyramid is fused with the Laplacian pyramid to obtain the result image. In this example, the processing method for fusion of three or more frames of images is the same as Figure 2 The processing methods of the two frames of images shown are exactly the same. The image fusion here is to multiply the brightness values ​​of the pixels in the two or more frames of images by their corresponding weight values ​​and then superimpose them to generate a fused image.

[0101] In the disclosed embodiment, when fusing multiple frames of images, the fusion is performed starting from the top layer of the Laplacian pyramid layer and downward to the second layer in the Laplacian pyramid layer, that is, the first layer of the image in the Laplacian pyramid layer is not fused first.

[0102] In the disclosed embodiment, when it is determined that the brightness value of the central pixel in the brightness map of the normally exposed image is greater than the pixel with the largest brightness value among the neighboring pixels, the brightness value of the central pixel is set to the maximum brightness value among the neighboring pixels. Maximum filtering is to first sort the surrounding pixels and the central pixel according to the brightness value of the pixel, and then compare the brightness value of the central pixel with the pixel with the largest brightness. If the brightness value of the central pixel is greater than the maximum brightness value of the surrounding pixels, the brightness value of the central pixel is set to the maximum brightness value of the surrounding pixels.

[0103] When it is determined that the number of current maximum value filtering has not reached the set number of times, the brightness value of the central pixel in the bright area map after the previous maximum value filtering is set to the maximum brightness value among the adjacent pixels until the number of maximum value filtering reaches the set number of times, thereby obtaining the first bright area map.

[0104] S12: Determine a first weight map based on the first bright area map.

[0105] In the embodiment of the present disclosure, a weight curve and a second image corresponding to the first image are obtained; wherein the weight curve is a functional relationship between a weight and a pixel brightness value in an image;

[0106] Determining the first weight map according to the second image, the first bright area map and the weight curve. Specifically, determining the first weight map includes:

[0107] If the brightness value of the pixel in the second image is less than 0, the brightness weight value of the pixel in the first bright area image is used as the corresponding weight in the first weight image, wherein the brightness weight value of the pixel in the first bright area image is: calculating the square of the difference between the brightness of the pixel in the first bright area image and the first set value, calculating the inverse of the ratio of the square to the second set value, and using e as the base to obtain the exponent of the inverse as the brightness weight value of the pixel in the first bright area image;

[0108] If the brightness value of a pixel in the second image is greater than or equal to 0, the weight of the pixel greater than or equal to 0 in the first weight map is 1.

[0109] The specific calculation formula is as follows:

[0110] If L1[index]<0, then W[index]=ylut(Ω1[index]);

[0111] Otherwise, W[index]=1.

[0112] W[index] is the weight of the pixel index, L1[index] is the pixel corresponding to the pixel index in the second image; ylut(Ω1[index]) represents the brightness weight value calculation function of the pixel index in the first bright area image, Ω1[index] represents the pixel index in the first bright area image;

[0113] y_lut(x)=exp(-(x-mu)*(x-mu) / sigma), x is the current pixel brightness value, mu is the first set value, and sigma is the second set value.

[0114] In the embodiment of the present disclosure, if the brightness value of the pixel point of the first layer image in the Laplacian pyramid layer of the normal exposure image is less than 0, the square of the difference between the first set constant and the brightness value of the pixel point with the same coordinate in the first bright area map and the first layer image, and the ratio of the square of the difference to the second set constant are calculated, and the suppression weight of the pixel point with the brightness value less than 0 in the first layer image is set to the exponent of the opposite number of the ratio with e as the base; if the brightness value of the pixel point of the first layer image of the normal exposure image is greater than or equal to 0, the suppression weight of the pixel point greater than or equal to 0 is set to 1. Here, the first set value mu includes 204, and the second set value sigma includes 2×14×14.

[0115] In the disclosed embodiment, the purpose of determining the first weight map is to suppress the black edge forming area in the normally exposed image in the original captured image, especially to suppress the negative Laplace brightness value therein, so as to remove the black edge phenomenon in the image.

[0116] S13. Determine a target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images.

[0117] In the disclosed embodiment, the second image corresponding to each image is an image corresponding to a layer with the largest image resolution in the Laplacian pyramid corresponding to each image; and the first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

[0118] Determining the target image specifically includes: performing upsampling processing on the first fused image to obtain a first fused image after the upsampling processing; here, the first fused image can be a Laplacian pyramid layer fused image of the normally exposed image in the multiple frames of images.

[0119] A third image is obtained according to the first weight map, the original weight map corresponding to the first image, and the second image corresponding to the first image; here, the original weight map is the weight map of the first image, that is, the weight map that has not been downsampled. The second image refers to the first layer of the Laplacian pyramid layer image after the Laplacian pyramid layer processing is performed on the normal exposure image in the multiple frames.

[0120] Performing weighted processing on the original weight map corresponding to other images in the multiple frames of images except the first image and the second image to obtain a fourth image;

[0121] The first fused image after the downsampling process, the third image and the fourth image are fused to obtain the target image.

[0122] In the embodiment of the present disclosure, before determining the target image, the method further includes:

[0123] Acquire the Laplacian pyramid layer corresponding to each image and the weight pyramid respectively corresponding to the Laplacian pyramid layer of each image; and determine the first fused image according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

[0124] Determining the first fused image specifically includes:

[0125] For each layer of the Laplacian pyramid corresponding to each image, the following processing is performed: acquiring the image of the layer in the Laplacian pyramid corresponding to each image to obtain the image corresponding to the layer for each image; acquiring the weight map of the layer in the weight pyramid corresponding to each image to obtain the weight map corresponding to the layer for each image; weighting the image corresponding to the layer for each image and the weight map corresponding to the layer for each image to obtain a fused image corresponding to the layer; wherein the fused images corresponding to all the layers form a first pyramid;

[0126] The first fused image is determined according to the first pyramid.

[0127] Determining the first fused image according to the first pyramid includes:

[0128] Starting from N being the first specified number, loop through the following steps until N is the second specified number:

[0129] Upsampling the image corresponding to the Nth layer in the first pyramid to obtain an upsampled image;

[0130] Fusing the upsampled image with the image corresponding to the N-1th layer in the first pyramid to obtain a fused image corresponding to the N-1th layer; updating the image corresponding to the N-1th layer to the fused image corresponding to the N-1th layer, and performing N minus one;

[0131] The first designated number is the layer number of the layer with the lowest resolution in the first pyramid, the second designated number is the layer number of the layer with the highest resolution in the first pyramid + 1, and N is an integer greater than the second designated number and less than the first designated number.

[0132] The first fused image is the fused image corresponding to the Nth layer obtained after the cycle is completed.

[0133] In the embodiment of the present disclosure, after obtaining the target image, the corresponding color channel information such as UV value is superimposed to generate an RGB image, which is output and displayed. The technical solution of the embodiment of the present disclosure can be applied to scenes such as image capture or video capture.

[0134] The essence of the technical solution of the embodiment of the present disclosure is further explained below through specific examples.

[0135] Figure 4 FIG. 1 is a flow chart of an image processing method according to an embodiment of the present disclosure. Figure 4As shown, in this example, three frames of input images are used for image fusion, and the three frames are respectively an underexposed image ev-, a normally exposed image ev0, and an overexposed image ev+. It should be noted that the present disclosure only uses three frames of images as an example to explain the principle. When the number of input frames is ≥ 2 frames, the same logic can be applied for image fusion processing. In the embodiment of the present disclosure, the fusion of multiple frames of images containing partially identical content is also supported, that is, each two adjacent frames of images in the multiple frames contain the same content, and the effect equivalent to wide-angle shooting is achieved through fusion. Among them, the same content in the multiple frames can be processed according to the relevant means of the embodiment of the present disclosure.

[0136] In the embodiment of the present disclosure, as an example, ev represents the exposure value of the image, which is related to the aperture value and shutter speed of the lens. The image processing method of the embodiment of the present disclosure is particularly suitable for image fusion in night scenes, or for image fusion in scenes with dark ambient light. The image processing method of the embodiment of the present disclosure includes the following processing steps:

[0137] Step 1. Input multiple frames of exposure images: ev-, ev0, ev+, the brightness maps corresponding to the input image frames are y0, y1, y2, and the UV channel color information is not considered; in the embodiment of the present disclosure, the exposure condition of the image, the controller of the electronic device can determine whether the currently captured image is normally exposed, overexposed or underexposed by the amount of light entering the shooting lens and the shooting time.

[0138] Step 2. Construct the Laplacian pyramid pyrY0, pyrY1, pyrY2 of the brightness image y, calculate the original weights W0, W1, W2 of the three frames input according to the weight curve, and construct the weight pyramid pyrW0, pyrW1, pyrW2;

[0139] Step 3. According to the Laplace pyramid layer, the three frames of images are fused layer by layer to obtain a fused pyramid image. Starting from the highest layer of the Laplace pyramid layer, the image is synthesized by upsampling and stacking layer by layer. The embodiment of the present disclosure only upsamples to the second layer of the Laplace pyramid layer, and the obtained synthesized image Y1 has a resolution of 1 / 4 of the size of the original captured image;

[0140] Y1=pyrMerge(pyrW02*pyrY02+pyrW12*pyrY12+pyrW22*pyrY22), frommax_layerto 2 ndlayer; where pyrW02, pyrW12, and pyrW22 represent the fusion weights of the three frames of images ev-, ev0, and ev+ from the highest layer to the second layer of the Laplace pyramid layer; pyrY02, pyrY12, and pyrY22 represent the image brightness maps of the three frames of images ev-, ev0, and ev+ from the highest layer to the second layer of the Laplace pyramid layer;

[0141] Step 4. Input the brightness image of the ev0 frame as the initial bright area image Ω0 in the night scene;

[0142] Step 5. Perform a maximum filter (max filter) on the bright area image Ω0 for a set number of times to obtain a bright area image Ω1 covering the area where the black edge is formed;

[0143] Ω1=max_filter(Ω0).

[0144] Maximum filtering means that the surrounding pixels and the central pixel are first sorted according to the brightness value of the pixel, and then the brightness value of the central pixel is compared with the pixel with the maximum brightness. If the brightness value of the central pixel is greater than the maximum brightness value of the surrounding pixels, the brightness value of the central pixel is set to the maximum brightness value of the surrounding pixels. The setting times can be set according to the processing accuracy of the image to be processed and the computing power of the processor, and can be set to 2, 3, 5, etc.

[0145] Step 6. In the bright area map Ω1, the higher the brightness of the area, the stronger the black edge attenuation needs to be. The suppression weight W for suppressing the black edge is determined according to the constructed weight curve y_lut and the bright area map Ω1.

[0146] y_lut(x)=exp(-(x-mu)×(x-mu) / sigma), x is the current pixel brightness value, the default value of mu is 204, and the default value of sigma is 2×14×14; the values ​​of mu and sigma are both empirical values, and the embodiments of the present disclosure are not limited to the aforementioned values ​​and can be set to other values ​​according to actual needs. Those skilled in the art should understand that no matter what values ​​mu and sigma are set to, they are equivalent replacements for the technical solutions of the embodiments of the present disclosure.

[0147] If L1[index]<0, then W[index]=ylut(Ω1[index]);

[0148] Otherwise, W[index]=1.

[0149] W[index] is the weight of the pixel point index, L1[index] is the pixel point corresponding to the pixel point index in the second image; ylut(Ω1[index]) represents the brightness weight value calculation function of the pixel point index in the first bright area image, and Ω1[index] represents the pixel point of the pixel point index in the first bright area image.

[0150] Step 7. In the process of pyramid fusion upsampling reconstruction, at the maximum resolution layer, i.e., the first layer, the negative Laplacian image layer L1 corresponding to ev0 is attenuated according to the weight map W to achieve the purpose of removing black lines.

[0151] Y01=upscale(Y1)+w01×L01+W×w11×L11+w21×L21

[0152] Among them, w01, w11, and w21 represent the fusion weights of the first layer of the Laplacian pyramid layers of the three frames of images ev-, ev0, and ev+ respectively; L01, L11, and L21 represent the image brightness maps of the first layer of the Laplacian pyramid layers of the three frames of images ev-, ev0, and ev+ respectively.

[0153] Step 8. Output Y001 without black edges, superimpose UV information, and obtain the ideal RGB image.

[0154] The embodiments of the present disclosure achieve image effects when shooting images, and are more suitable for image photography and video synthesis in night scenes or in dim light, thereby improving the display effect of images or videos and improving user experience.

[0155] Figure 5 FIG. 1 is a schematic diagram of the structure of an image processing device according to an embodiment of the present disclosure. Figure 5 As shown, the image processing device of the embodiment of the present disclosure includes:

[0156] A filtering unit 50 is used to perform a maximum value filtering for a predetermined number of times on the brightness map of the first image in the multiple frames of images to obtain a first bright area map;

[0157] A first determining unit 51, configured to determine a first weight map based on the first bright area map;

[0158] The second determination unit 52 is used to determine the target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images; wherein the second image corresponding to each image is the image corresponding to the layer with the largest image resolution in the Laplacian pyramid corresponding to each image; and the first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

[0159] Optionally, the first determining unit 51 is further configured to:

[0160] Obtaining a weight curve and a second image corresponding to the first image; wherein the weight curve is a functional relationship between a weight and a pixel brightness value in an image;

[0161] The first weight map is determined according to the second image, the first bright area map and the weight curve.

[0162] Optionally, the first determining unit 51 is further configured to:

[0163] If the brightness value of the pixel in the second image is less than 0, the brightness weight value of the pixel in the first bright area image is used as the corresponding weight in the first weight image, wherein the brightness weight value of the pixel in the first bright area image is: calculating the square of the difference between the brightness of the pixel in the first bright area image and the first set value, calculating the inverse of the ratio of the square to the second set value, and using e as the base to obtain the exponent of the inverse as the brightness weight value of the pixel in the first bright area image;

[0164] If the brightness value of a pixel in the second image is greater than or equal to 0, the weight of the pixel greater than or equal to 0 in the first weight map is 1.

[0165] Specifically, the first weight map is determined based on the following formula:

[0166] If L1[index]<0, then W[index]=ylut(Ω1[index]);

[0167] Otherwise, W[index]=1.

[0168] W[index] is the weight of the pixel index, L1[index] is the pixel corresponding to the pixel index in the second image; ylut(Ω1[index]) represents the brightness weight value calculation function of the pixel index in the first bright area image, Ω1[index] represents the pixel index in the first bright area image;

[0169] y_lut(x)=exp(-(x-mu)*(x-mu) / sigma), x is the current pixel brightness value, mu is the first set value, and sigma is the second set value.

[0170] Optionally, the second determining unit 52 is further configured to:

[0171] Performing upsampling processing on the first fused image to obtain a first fused image after upsampling processing;

[0172] Obtaining a third image according to the first weight map, the original weight map corresponding to the first image, and the second image corresponding to the first image;

[0173] Performing weighted processing on the original weight map corresponding to other images in the multiple frames of images except the first image and the second image to obtain a fourth image;

[0174] The first fused image after the downsampling process, the third image and the fourth image are fused to obtain the target image.

[0175] exist Figure 5 Based on the image processing device shown in the figure, the image processing device of the embodiment of the present disclosure further includes:

[0176] Get unit( Figure 5 ), used to obtain the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image;

[0177] The third determination unit ( Figure 5 ), which is used to determine the first fused image according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

[0178] Optionally, the third determining unit is further configured to:

[0179] For each layer of the Laplacian pyramid corresponding to each image, the following processing is performed: acquiring the image of the layer in the Laplacian pyramid corresponding to each image to obtain the image corresponding to the layer for each image; acquiring the weight map of the layer in the weight pyramid corresponding to each image to obtain the weight map corresponding to the layer for each image; weighting the image corresponding to the layer for each image and the weight map corresponding to the layer for each image to obtain a fused image corresponding to the layer; wherein the fused images corresponding to all the layers form a first pyramid;

[0180] The first fused image is determined according to the first pyramid.

[0181] Optionally, the third determining unit is further configured to:

[0182] Starting from N being the first specified number, loop through the following steps until N is the second specified number:

[0183] Upsampling the image corresponding to the Nth layer in the first pyramid to obtain an upsampled image;

[0184] Fusing the upsampled image with the image corresponding to the N-1th layer in the first pyramid to obtain a fused image corresponding to the N-1th layer; updating the image corresponding to the N-1th layer to the fused image corresponding to the N-1th layer, and performing N minus one;

[0185] The first designated number is the layer number of the layer with the lowest resolution in the first pyramid, the second designated number is the layer number of the layer with the highest resolution in the first pyramid + 1, and N is an integer greater than the second designated number and less than the first designated number.

[0186] The first fused image is the fused image corresponding to the Nth layer obtained after the cycle is completed.

[0187] Optionally, the acquisition unit is further used to:

[0188] Down-sampling each of the images respectively to obtain a Gaussian pyramid layer of each of the images;

[0189] The current Laplacian pyramid layer of each image is obtained by:

[0190] Upsampling a Gaussian pyramid layer above the current layer of each image to obtain an upsampled layer, and performing Gaussian blur processing on the upsampled layer to obtain a first Gaussian pyramid layer;

[0191] Performing a subtraction operation on the current Gaussian pyramid layer of each image and the first Gaussian pyramid layer to obtain a current Laplacian pyramid layer;

[0192] The Laplacian pyramid layers obtained layer by layer are combined to form a Laplacian pyramid corresponding to each image.

[0193] Optionally, the acquisition unit is further used to:

[0194] Obtaining an original weight map corresponding to each image in the multiple frames of images, and downsampling the original weight map of each image to obtain a weight map of the second layer of each image;

[0195] The weight map of the second layer of each image is downsampled again to obtain the weight map of the third layer of each image, until a weight map corresponding to each layer of the Laplacian pyramid of each image is obtained.

[0196] In an exemplary embodiment, the filtering unit 50, the first determination unit 51, the second determination unit 52, the third determination unit and the acquisition unit, etc. can be implemented by one or more central processing units (CPU), graphics processing units (GPU), baseband processors (BP), application specific integrated circuits (ASIC), digital signal processors (DSP), programmable logic devices (PLD), complex programmable logic devices (CPLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers (MCU), microprocessors, or other electronic components to execute the aforementioned image processing method.

[0197] Regarding the image processing device in the above embodiment, the specific manner in which each module and unit performs operations has been described in detail in the embodiment of the device, and will not be elaborated here.

[0198] Figure 6 is a block diagram of an electronic device 800 according to an exemplary embodiment. Figure 6 As shown, the electronic device 800 supports multi-screen output, and the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0199] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0200] The memory 804 is configured to store various types of data to support operations on the device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0201] The power component 806 provides power to the various components of the electronic device 800. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

[0202] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0203] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0204] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0205] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0206] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as Wi-Fi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0207] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the image processing steps of the above-described embodiments.

[0208] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, and the instructions can be executed by a processor 820 of an electronic device 800 to complete the steps of the image processing method of the above embodiment. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0209] The embodiment of the present disclosure also records a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the control method of the aforementioned embodiment. The method includes:

[0210] Performing maximum value filtering for a predetermined number of times on the brightness map of the first image in the multiple frames of images to obtain a first bright area map;

[0211] determining a first weight map based on the first bright area map;

[0212] A target image is determined according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images; wherein the second image corresponding to each image is an image corresponding to a layer with the largest image resolution in the Laplacian pyramid corresponding to each image; and the first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

[0213] Optionally, determining a first weight map based on the first bright area map includes:

[0214] Obtaining a weight curve and a second image corresponding to the first image; wherein the weight curve is a functional relationship between a weight and a pixel brightness value in an image;

[0215] The first weight map is determined according to the second image, the first bright area map and the weight curve.

[0216] Optionally, determining the first weight map according to the second image, the first bright area map and the weight curve includes:

[0217] If the brightness value of the pixel in the second image is less than 0, the brightness weight value of the pixel in the first bright area image is used as the corresponding weight in the first weight image, wherein the brightness weight value of the pixel in the first bright area image is: calculating the square of the difference between the brightness of the pixel in the first bright area image and the first set value, calculating the inverse of the ratio of the square to the second set value, and using e as the base to obtain the exponent of the inverse as the brightness weight value of the pixel in the first bright area image;

[0218] If the brightness value of a pixel in the second image is greater than or equal to 0, the weight of the pixel greater than or equal to 0 in the first weight map is 1.

[0219] The specific calculation method of the first weight map is as follows:

[0220] If L1[index]<0, then W[index]=ylut(Ω1[index]);

[0221] Otherwise, W[index]=1.

[0222] W[index] is the weight of the pixel index, L1[index] is the pixel corresponding to the pixel index in the second image; ylut(Ω1[index]) represents the brightness weight value calculation function of the pixel index in the first bright area image, Ω1[index] represents the pixel index in the first bright area image;

[0223] y_lut(x)=exp(-(x-mu)*(x-mu) / sigma), x is the current pixel brightness value, mu is the first set value, and sigma is the second set value.

[0224] Optionally, determining the target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images includes:

[0225] Performing upsampling processing on the first fused image to obtain a first fused image after upsampling processing;

[0226] Obtaining a third image according to the first weight map, the original weight map corresponding to the first image, and the second image corresponding to the first image;

[0227] Performing weighted processing on the original weight map corresponding to other images in the multiple frames of images except the first image and the second image to obtain a fourth image;

[0228] The first fused image after the downsampling process, the third image and the fourth image are fused to obtain the target image.

[0229] Optionally, before determining the target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images, the method further includes:

[0230] Obtaining a Laplacian pyramid corresponding to each image and a weight pyramid corresponding to each image;

[0231] The first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

[0232] Optionally, determining the first fused image according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image includes:

[0233] For each layer of the Laplacian pyramid corresponding to each image, the following processing is performed: acquiring the image of the layer in the Laplacian pyramid corresponding to each image to obtain the image corresponding to the layer for each image; acquiring the weight map of the layer in the weight pyramid corresponding to each image to obtain the weight map corresponding to the layer for each image; weighting the image corresponding to the layer for each image and the weight map corresponding to the layer for each image to obtain a fused image corresponding to the layer; wherein the fused images corresponding to all the layers form a first pyramid;

[0234] The first fused image is determined according to the first pyramid.

[0235] Optionally, determining the first fused image according to the first pyramid includes:

[0236] Starting from N being the first specified number, loop through the following steps until N is the second specified number:

[0237] Upsampling the image corresponding to the Nth layer in the first pyramid to obtain an upsampled image;

[0238] Fusing the upsampled image with the image corresponding to the N-1th layer in the first pyramid to obtain a fused image corresponding to the N-1th layer; updating the image corresponding to the N-1th layer to the fused image corresponding to the N-1th layer, and performing N minus one;

[0239] The first designated number is the layer number of the layer with the lowest resolution in the first pyramid, the second designated number is the layer number of the layer with the highest resolution in the first pyramid + 1, and N is an integer greater than the second designated number and less than the first designated number.

[0240] The first fused image is the fused image corresponding to the Nth layer obtained after the cycle is completed.

[0241] Optionally, obtaining the Laplacian pyramid corresponding to each image includes:

[0242] Down-sampling each of the images respectively to obtain a Gaussian pyramid layer of each of the images;

[0243] The current Laplacian pyramid layer of each image is obtained by:

[0244] Upsampling a Gaussian pyramid layer above the current layer of each image to obtain an upsampled layer, and performing Gaussian blur processing on the upsampled layer to obtain a first Gaussian pyramid layer;

[0245] Performing a subtraction operation on the current Gaussian pyramid layer of each image and the first Gaussian pyramid layer to obtain a current Laplacian pyramid layer;

[0246] The Laplacian pyramid layers obtained layer by layer are combined to form a Laplacian pyramid corresponding to each image.

[0247] Optionally, obtaining a weight pyramid corresponding to each image includes:

[0248] Obtaining an original weight map corresponding to each image in the multiple frames of images, and downsampling the original weight map of each image to obtain a weight map of the second layer of each image;

[0249] The weight map of the second layer of each image is downsampled again to obtain the weight map of the third layer of each image, until a weight map corresponding to each layer of the Laplacian pyramid of each image is obtained.

[0250] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0251] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Performing maximum value filtering for a predetermined number of times on the brightness map of the first image in the multiple frames of images to obtain a first bright area map; determining a first weight map based on the first bright area map; Determine a target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images; wherein the second image corresponding to each image is an image corresponding to a layer with the largest image resolution in the Laplacian pyramid corresponding to each image; the first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image; the original weight map is a weight map that has not been downsampled, and the weight pyramid is obtained by downsampling the original weight map layer by layer; The determining of the first weight map based on the first bright area map includes: Obtaining a weight curve and a second image corresponding to the first image; wherein the weight curve is a functional relationship between a weight and a pixel brightness value in an image; The first weight map is determined according to the second image, the first bright area map and the weight curve.

2. The method according to claim 1, characterized in that Determining the first weight map according to the second image, the first bright area map and the weight curve includes: If the brightness value of the pixel in the second image is less than 0, the brightness weight value of the pixel in the first bright area image is used as the corresponding weight in the first weight image, wherein the brightness weight value of the pixel in the first bright area image is: calculating the square of the difference between the brightness of the pixel in the first bright area image and the first set value, calculating the inverse of the ratio of the square to the second set value, and using e as the base to obtain the exponent of the inverse as the brightness weight value of the pixel in the first bright area image; If the brightness value of a pixel in the second image is greater than or equal to 0, the weight of the pixel greater than or equal to 0 in the first weight map is 1.

3. The method according to claim 1, characterized in that Determining a target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images includes: Performing upsampling processing on the first fused image to obtain a first fused image after upsampling processing; Obtaining a third image according to the first weight map, the original weight map corresponding to the first image, and the second image corresponding to the first image; Performing weighted processing on the original weight map corresponding to other images in the multiple frames of images except the first image and the second image to obtain a fourth image; The first fused image after the upsampling process, the third image and the fourth image are fused to obtain the target image.

4. The method according to claim 1, characterized in that Before determining the target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images, the method further includes: Obtaining a Laplacian pyramid corresponding to each image and a weight pyramid corresponding to each image; The first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

5. The method according to claim 4, characterized in that The determining the first fused image according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image includes: For each layer of the Laplacian pyramid corresponding to each image, the following processing is performed: acquiring the image of the layer in the Laplacian pyramid corresponding to each image to obtain the image corresponding to the layer for each image; acquiring the weight map of the layer in the weight pyramid corresponding to each image to obtain the weight map corresponding to the layer for each image; weighting the image corresponding to the layer for each image and the weight map corresponding to the layer for each image to obtain a fused image corresponding to the layer; wherein the fused images corresponding to all the layers form a first pyramid; The first fused image is determined according to the first pyramid.

6. The method according to claim 5, characterized in that The determining the first fused image according to the first pyramid includes: Starting from N being the first specified number, loop through the following steps until N is the second specified number: Upsampling the image corresponding to the Nth layer in the first pyramid to obtain an upsampled image; Fusing the upsampled image with the image corresponding to the N-1th layer in the first pyramid to obtain a fused image corresponding to the N-1th layer; updating the image corresponding to the N-1th layer to the fused image corresponding to the N-1th layer, and performing N minus one; The first designated number is the number of the layer with the lowest resolution in the first pyramid, the second designated number is the number of the layer with the highest resolution in the first pyramid plus 1, and N is an integer greater than the second designated number and less than the first designated number. The first fused image is the fused image corresponding to the Nth layer obtained after the cycle is completed.

7. The method according to claim 4 or 5, characterized in that: The obtaining of the Laplacian pyramid corresponding to each image includes: Down-sampling each of the images respectively to obtain a Gaussian pyramid layer of each of the images; The current Laplacian pyramid layer of each image is obtained by: Upsampling a Gaussian pyramid layer above the current layer of each image to obtain an upsampled layer, and performing Gaussian blur processing on the upsampled layer to obtain a first Gaussian pyramid layer; Performing a subtraction operation on the current Gaussian pyramid layer of each image and the first Gaussian pyramid layer to obtain a current Laplacian pyramid layer; The Laplacian pyramid layers obtained layer by layer are combined to form a Laplacian pyramid corresponding to each image.

8. The method according to claim 4 or 5, characterized in that: Obtaining the weight pyramid corresponding to each image, including: Obtaining an original weight map corresponding to each image in the multiple frames of images, and downsampling the original weight map of each image to obtain a weight map of the second layer of each image; The weight map of the second layer of each image is downsampled again to obtain the weight map of the third layer of each image, until a weight map corresponding to each layer of the Laplacian pyramid of each image is obtained.

9. An image processing device, characterized in that: The device comprises: A filtering unit, configured to perform a maximum value filtering for a predetermined number of times on the brightness map of the first image in the multiple frames of images to obtain a first bright area map; A first determining unit is configured to obtain a weight curve and a second image corresponding to the first image; wherein the weight curve is a functional relationship between a weight and a pixel brightness value in an image; and determine a first weight map according to the second image, the first bright area map and the weight curve; The second determination unit is used to determine the target image according to the first weight map, the first fused image, the original weight map corresponding to each image in the multiple frames of images, and the second image corresponding to each image in the multiple frames of images; wherein the second image corresponding to each image is the image corresponding to the layer with the largest image resolution in the Laplacian pyramid corresponding to each image; the first fused image is determined according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image; the original weight map is a weight map that has not been downsampled, and the weight pyramid is obtained by downsampling the original weight map layer by layer.

10. The device according to claim 9, characterized in that The first determining unit is further configured to: If the brightness value of the pixel in the second image is less than 0, the brightness weight value of the pixel in the first bright area image is used as the corresponding weight in the first weight image, wherein the brightness weight value of the pixel in the first bright area image is: calculating the square of the difference between the brightness of the pixel in the first bright area image and the first set value, calculating the inverse of the ratio of the square to the second set value, and using e as the base to obtain the exponent of the inverse as the brightness weight value of the pixel in the first bright area image; If the brightness value of a pixel in the second image is greater than or equal to 0, the weight of the pixel greater than or equal to 0 in the first weight map is 1.

11. The device according to claim 9, characterized in that The second determining unit is further configured to: Performing upsampling processing on the first fused image to obtain a first fused image after upsampling processing; Obtaining a third image according to the first weight map, the original weight map corresponding to the first image, and the second image corresponding to the first image; Performing weighted processing on the original weight map corresponding to other images in the multiple frames of images except the first image and the second image to obtain a fourth image; The first fused image after the upsampling process, the third image and the fourth image are fused to obtain the target image.

12. The device according to claim 9, characterized in that The device also includes: An acquisition unit, used to acquire a Laplacian pyramid corresponding to each image and a weight pyramid corresponding to each image; The third determining unit is configured to determine the first fused image according to the Laplacian pyramid corresponding to each image and the weight pyramid corresponding to each image.

13. The device according to claim 12, characterized in that The third determining unit is further configured to: For each layer of the Laplacian pyramid corresponding to each image, the following processing is performed: acquiring the image of the layer in the Laplacian pyramid corresponding to each image to obtain the image corresponding to the layer for each image; acquiring the weight map of the layer in the weight pyramid corresponding to each image to obtain the weight map corresponding to the layer for each image; weighting the image corresponding to the layer for each image and the weight map corresponding to the layer for each image to obtain a fused image corresponding to the layer; wherein the fused images corresponding to all the layers form a first pyramid; The first fused image is determined according to the first pyramid.

14. The device according to claim 13, characterized in that The third determining unit is further configured to: Starting from N being the first specified number, loop through the following steps until N is the second specified number: Upsampling the image corresponding to the Nth layer in the first pyramid to obtain an upsampled image; Fusing the upsampled image with the image corresponding to the N-1th layer in the first pyramid to obtain a fused image corresponding to the N-1th layer; updating the image corresponding to the N-1th layer to the fused image corresponding to the N-1th layer, and performing N minus one; The first designated number is the number of the layer with the lowest resolution in the first pyramid, the second designated number is the number of the layer with the highest resolution in the first pyramid plus 1, and N is an integer greater than the second designated number and less than the first designated number. The first fused image is the fused image corresponding to the Nth layer obtained after the cycle is completed.

15. The device according to claim 12 or 13, characterized in that The acquisition unit is further used for: Down-sampling each of the images respectively to obtain a Gaussian pyramid layer of each of the images; The current Laplacian pyramid layer of each image is obtained by: Upsampling a Gaussian pyramid layer above the current layer of each image to obtain an upsampled layer, and performing Gaussian blur processing on the upsampled layer to obtain a first Gaussian pyramid layer; Performing a subtraction operation on the current Gaussian pyramid layer of each image and the first Gaussian pyramid layer to obtain a current Laplacian pyramid layer; The Laplacian pyramid layers obtained layer by layer are combined to form a Laplacian pyramid corresponding to each image.

16. The device according to claim 12 or 13, characterized in that The acquisition unit is further used for: Obtaining an original weight map corresponding to each image in the multiple frames of images, and downsampling the original weight map of each image to obtain a weight map of the second layer of each image; The weight map of the second layer of each image is downsampled again to obtain the weight map of the third layer of each image, until a weight map corresponding to each layer of the Laplacian pyramid of each image is obtained.

17. An electronic device, characterized in that: The electronic device comprises: a processor and a memory for storing processor executable instructions, wherein the processor is configured to execute the steps of the image processing method according to any one of claims 1 to 8 when calling the executable instructions in the memory.

18. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the steps of the image processing method according to any one of claims 1 to 8.