Image fusion method, device, electronic device, storage medium, and product

Abstract

The present application relates to an image fusion method, device, electronic device, storage medium, and product. The method includes: obtaining multiple frames of original images shot under different exposure parameters, stretching the multiple frames of original images to obtain multiple frames of intermediate images; wherein the bit width of the intermediate images is greater than the bit width of the original images. Calculating the fusion weights of the multiple frames of intermediate images based on the multiple frames of original images, and performing image fusion on the multiple frames of intermediate images based on the fusion weights to obtain a target image. Since the multiple frames of original images shot under different exposure parameters can respectively capture image details of different dynamic ranges, and the multiple frames of original images are stretched, the bit width of the obtained multiple frames of intermediate images is larger and can contain more image information. Then, calculating the fusion weights of the multiple frames of intermediate images based on the multiple frames of original images, and performing image fusion on the multiple frames of intermediate images based on the fusion weights, a target image with a high dynamic range containing more image information can be obtained.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image fusion method, device, electronic device, storage medium, and product. Background Art

[0002] With the rapid development of computer technology, image processing-related technologies have also developed rapidly. As a result, various fields have put forward widespread demands for high-quality images, and high-quality images can be high dynamic range images, namely HDR (High Dynamic Range) images. Specifically, ordinary images taken by electronic devices at different exposure times can be synthesized to obtain high dynamic range images. Since high dynamic range images are images synthesized based on multiple frames of ordinary images, high dynamic range images can provide more dynamic range and image details than ordinary images, and can better restore the scene observed by the human eye.

[0003] However, the image quality of high dynamic range images synthesized by traditional methods is still low and cannot meet people's increasingly higher requirements for image quality. Summary of the Invention

[0004] The embodiments of the present application provide an image fusion method, apparatus, electronic device, and computer-readable storage medium, which can improve the image quality of synthesized high dynamic range images.

[0005] In one aspect, an image fusion method is provided, the method comprising:

[0006] Acquire multiple frames of original images captured under different exposure parameters;

[0007] Stretching the multiple frames of original images to obtain multiple frames of intermediate images, wherein the bit width of the intermediate images is greater than the bit width of the original images;

[0008] The fusion weights of the multiple frames of intermediate images are calculated according to the multiple frames of original images, and the multiple frames of intermediate images are fused based on the fusion weights to obtain a target image.

[0009] In another aspect, an image fusion device is provided, comprising:

[0010] The original image acquisition module is used to acquire multiple frames of original images shot under different exposure parameters;

[0011] an intermediate image acquisition module, configured to perform stretching processing on the multiple frames of original images to obtain multiple frames of intermediate images; the bit width of the intermediate images is greater than the bit width of the original images;

[0012] The image fusion module is used to calculate the fusion weights of the multiple frames of intermediate images according to the multiple frames of original images, and perform image fusion on the multiple frames of intermediate images based on the fusion weights to obtain a target image.

[0013] On the other hand, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs the steps of the image fusion method described above.

[0014] On the other hand, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image fusion method described above are implemented.

[0015] On the other hand, a computer program product is provided, comprising a computer program, which implements the steps of the image fusion method described above when executed by a processor.

[0016] The above-mentioned image fusion method, device, electronic device, storage medium, and product obtain multiple frames of original images captured under different exposure parameters, stretch the multiple frames of original images, and obtain multiple frames of intermediate images; wherein the bit width of the intermediate images is greater than the bit width of the original images. The fusion weights of the multiple frames of intermediate images are calculated based on the multiple frames of original images, and the multiple frames of intermediate images are fused based on the fusion weights to obtain the target image. Since the multiple frames of original images captured under different exposure parameters can respectively capture image details of different dynamic ranges, and the multiple frames of original images are stretched, the resulting multiple frames of intermediate images have a larger bit width and can contain more image information. Therefore, the fusion weights of the multiple frames of intermediate images are calculated based on the multiple frames of original images, and the multiple frames of intermediate images are fused based on the fusion weights to obtain a high dynamic range target image containing more image information. In addition, the image quality of the synthesized high dynamic range image is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A diagram showing an application environment of an image fusion method in one embodiment;

[0019] Figure 2 is a flowchart of an image fusion method in one embodiment;

[0020] Figure 3 is a schematic diagram of a long-exposure image, a medium-exposure image, and a short-exposure image obtained by photographing the same scene in one embodiment;

[0021] Figure 4 for Figure 2 Flowchart of the method for performing image fusion and obtaining the target image;

[0022] Figure 5 1 is a schematic diagram of a process of converting a long-exposure image into a grayscale image and then converting the grayscale image into an image pyramid in one embodiment;

[0023] Figure 6 A flowchart of a method for calculating a fusion weight map if the fusion weight map includes a first fusion weight map in one embodiment;

[0024] Figure 7 for Figure 6 A flowchart of a method for calculating a first weight pyramid corresponding to a first image pyramid;

[0025] Figure 8 A schematic diagram of a process of calculating a first weight pyramid corresponding to a first image pyramid based on a long-exposure image in one embodiment;

[0026] Figure 9 1. A schematic diagram of a process of calculating a first weighted pyramid corresponding to a first image pyramid based on a first image pyramid of a mid-exposure image in one embodiment;

[0027] Figure 10 2. A schematic diagram of a process of calculating a first weighted pyramid corresponding to a first image pyramid based on a short-exposure image in one embodiment;

[0028] Figure 11 Schematic diagram of a process of generating a first fused weight map based on a first weight pyramid in one embodiment;

[0029] Figure 12 Flowchart of a method for calculating a fusion weight map of an intermediate image corresponding to a grayscale image based on an image pyramid corresponding to the grayscale image and a weight pyramid corresponding to the image pyramid in one embodiment;

[0030] Figure 13 is a schematic diagram of a process of equalizing a brightness histogram of a grayscale image in one embodiment;

[0031] Figure 14 for Figure 12 A flowchart of a method for calculating a second weighted pyramid corresponding to the second image pyramid based on the second image pyramid and the second Gaussian curve corresponding to the grayscale image;

[0032] Figure 15 FIG1 is a schematic diagram of a process of generating a second fused weight map based on a second weight pyramid in one embodiment;

[0033] Figure 16 Schematic diagram of a process of multiplying a first fusion weight map and a second fusion weight map to obtain a new fusion weight map in one embodiment;

[0034] Figure 17 A schematic diagram of a process for generating multiple frames of original images in one embodiment;

[0035] Figure 18 is a flowchart of an image fusion method in another embodiment;

[0036] Figure 19 is a schematic diagram of an image fusion method in a specific embodiment;

[0037] Figure 20 is a structural block diagram of an image fusion device in one embodiment;

[0038] Figure 21 for Figure 20 Structural block diagram of the image fusion module;

[0039] Figure 22 for Figure 21 The structural block diagram of the fusion weight graph calculation unit;

[0040] Figure 23 FIG. 1 is a schematic diagram of the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0042] It will be understood that the terms "first," "second," and so forth, used herein may be used to describe various elements, but these elements are not limited by these terms. These terms are used solely to distinguish a first element from another element. For example, a first image pyramid may be referred to as a second image pyramid, and similarly, a second image pyramid may be referred to as a first image pyramid, without departing from the scope of this application. Both the first image pyramid and the second image pyramid are image pyramids, but they are not identical image pyramids.

[0043] Figure 1 FIG. 1 is a schematic diagram of an application environment of an image fusion method in an embodiment. Figure 1As shown, the application environment includes an electronic device 120, which acquires multiple frames of original images captured under different exposure parameters; stretches the multiple frames of original images to obtain multiple frames of intermediate images; the bit width of the intermediate images is greater than the bit width of the original images; calculates fusion weights for the multiple frames of intermediate images based on the multiple frames of original images, and fuses the multiple frames of intermediate images based on the fusion weights to obtain a target image. The electronic device 120 may be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices may include smart watches, smart bracelets, head-mounted devices, and the like.

[0044] Figure 2 FIG. 1 is a flow chart of an image fusion method in one embodiment. The image fusion method in this embodiment is based on the image fusion method running on Figure 1 The electronic equipment in the example is used to describe it. Figure 2 As shown, the image fusion method includes steps 220 to 260, wherein:

[0045] Step 220: Acquire multiple frames of original images captured under different exposure parameters.

[0046] The light or brightness information of the shooting scene is uncertain. In order to capture clear images, the exposure parameters of the camera can be set to different values, and multiple frames of original images can be captured. The exposure parameters here include but are not limited to one or more of the sensitivity, exposure amount, exposure time, etc. When other parameters remain unchanged, increasing the exposure time will result in overexposure (long exposure) exposure parameters; when other parameters remain unchanged, shortening the exposure time will result in underexposure (short exposure) exposure parameters. It can be set that when the shooting button is pressed, multiple frames of original images are continuously captured under different shooting parameters, thereby obtaining multiple frames of original images captured under different exposure parameters. Such as Figure 3 As shown, from left to right are the long exposure image taken by the electronic device under overexposure parameters, the medium exposure image taken under normal exposure parameters, and the short exposure image taken under underexposure parameters of the same scene.

[0047] Multiple frames of raw images captured at different exposure parameters can capture image details in different dynamic ranges within the scene. The "dynamic range" of a scene refers to the range of luminosity distribution in the image, from the darkest shadows to the brightest highlights.

[0048] The multi-frame original image here can be an RGB three-channel image (abbreviated as RGB image) or an RGBW four-channel image (abbreviated as RGBW image), which is not limited in this application. Among them, the RGB three-channel image includes images of three channels: R channel (red channel), G channel (green channel), and B channel (blue channel), that is, the original image includes red pixels (R pixels), green pixels (G pixels), and blue pixels (B pixels). Among them, the RGBW four-channel image includes images of four channels: R channel (red channel), G channel (green channel), B channel (blue channel), and W channel (white channel), that is, the original image includes red pixels (R pixels), green pixels (G pixels), blue pixels (B pixels) and full-color pixels (or called W pixels).

[0049] Step 240 : stretching the multiple frames of original images to obtain multiple frames of intermediate images; wherein the bit width of the intermediate images is greater than the bit width of the original images.

[0050] Stretching refers to linearly stretching the image to be adjusted. Linear stretching transforms all pixels in the image according to a linear transformation function. Linear stretching can include direct linear stretching, cropped linear stretching, and segmented stretching. The principle behind linear stretching is that an image with a luminance or chrominance value range of 0-127 cannot display clearer details. However, stretching the luminance or chrominance values ​​of that image, i.e., stretching the luminance or chrominance value range from 0-127 to 0-255, results in a stretched image that displays clearer details.

[0051] Generally, the bit width of the intermediate image (Middle Image) obtained after stretching the original image is greater than the bit width of the original image. For example, in the above example, an image frame with a luminance or chrominance value range of 0-255 is an 8-bit image. For example, the output format of an 8-bit image is 8 bits, which means that the luminance or chrominance values ​​of the image are stored in a variable with a bit width of 8 bits. Correspondingly, an image frame with a luminance or chrominance value range of 0-127 is a 6-bit image. For example, the output format of a 6-bit image is 6 bits, which means that the luminance or chrominance values ​​of the image are stored in a variable with a bit width of 6 bits. In this example, the multiple original image frames are all 6-bit images, and the multiple intermediate image frames are all 8-bit images. Therefore, the bit width of the intermediate image (Middle Image) is greater than the bit width of the original images. In other words, the multiple intermediate images (Middle Image) contain more image information than the multiple original image frames. Of course, in this example, multiple frames of original images may all be 10-bit images, and multiple frames of intermediate images (Middle Image) may all be 16-bit images, and this application does not impose any limitation on this.

[0052] Step 260 , calculating the fusion weights of the multiple frames of intermediate images according to the multiple frames of original images, and performing image fusion on the multiple frames of intermediate images based on the fusion weights to obtain a target image.

[0053] Because the bit width of the multi-frame intermediate image (Middle Image) is larger than the bit width of the original image, using the multi-frame intermediate image (Middle Image) to calculate the fusion weight of the multi-frame intermediate image (Middle Image) is computationally intensive and places a heavy load on the CPU or GPU. However, since the bit width of the multi-frame original image is smaller than that of the middle image (Middle Image), the fusion weight of the multi-frame intermediate image (Middle Image) can be calculated based on the multi-frame original image, reducing the computational effort and avoiding a heavy load on the CPU or GPU.

[0054] Specifically, the fusion weight map of the intermediate image (MiddleImage) corresponding to each frame of the original image can be calculated, and then the weighted fusion processing of multiple frames of intermediate images (MiddleImage) can be performed based on the fusion weight map of each intermediate image (MiddleImage) to obtain the target image.

[0055] In an embodiment of the present application, multiple frames of original images taken under different exposure parameters are obtained, and the multiple frames of original images are stretched to obtain multiple frames of intermediate images (Middle Image); wherein the bit width of the intermediate image (Middle Image) is greater than the bit width of the original image. The fusion weights of the multiple frames of intermediate images (Middle Image) are calculated based on the multiple frames of original images, and the multiple frames of intermediate images (Middle Image) are fused based on the fusion weights to obtain a target image. Since the multiple frames of original images taken under different exposure parameters can capture image details of different dynamic ranges respectively, and the multiple frames of original images are stretched, the bit width of the obtained multiple frames of intermediate images (Middle Image) is larger and can contain more image information. Therefore, the fusion weights of the multiple frames of intermediate images (Middle Image) are calculated based on the multiple frames of original images, and the multiple frames of intermediate images (Middle Image) are fused based on the fusion weights to obtain a target image with a high dynamic range containing more image information. In addition, the image quality of the synthesized high dynamic range image is improved.

[0056] In the previous embodiment, the process of obtaining multiple frames of original images shot under different exposure parameters, stretching the multiple frames of original images to obtain multiple frames of intermediate images (Middle Image), calculating the fusion weights of the multiple frames of intermediate images (Middle Image) based on the multiple frames of original images, and performing image fusion on the multiple frames of intermediate images (Middle Image) based on the fusion weights to obtain the target image was described. In this embodiment, Figure 4 As shown, the specific implementation steps of step 260, calculating the fusion weights of multiple intermediate images (Middle Image) based on multiple original images, and performing image fusion on the multiple intermediate images (Middle Image) based on the fusion weights to obtain the target image, include:

[0057] Step 262 : For each original image in the multiple frames of original images, convert the original image into a grayscale image.

[0058] Because multiple original images captured under different exposure parameters can capture details in different dynamic ranges within a scene, meaning the brightness of the same area within multiple original images varies, the primary goal of image fusion is to obtain a target image with a more uniform brightness distribution. Grayscale images can well reflect the brightness distribution within an image. Therefore, when performing image fusion, each original image in the multiple original images can first be converted into a grayscale image. Then, based on the grayscale image, a fusion weight map for the corresponding intermediate image is calculated.

[0059] Assuming the original image is an RGB image, when converting each original image in multiple frames into a grayscale image, an averaging method, a weighted averaging method, or other methods may be used to convert the RGB image into a grayscale image, although this application does not limit this. The bit width of the resulting grayscale image is consistent with the bit width of the original image. Therefore, to reduce the amount of computation, the grayscale image may be right-shifted to obtain a grayscale image with a smaller bit width.

[0060] The averaging method refers to a method of averaging the values ​​of the RGB three channels corresponding to the same pixel to obtain the grayscale value of the pixel.

[0061] For example, the formula using the averaging method is as follows:

[0062] I(x,y)=1 / 3×I_R(x,y)+1 / 3×I_G(x,y)+1 / 3×I_B(x,y) Formula (1-1)

[0063] Here, I_R(x,y), I_G(x,y), and I_B(x,y) are the R, G, and B channel values ​​of pixel I at (x,y) in the original image. I(x,y) represents the converted grayscale value corresponding to pixel I. I(x,y) is then right-shifted to obtain a grayscale value Gray_Y with a smaller bit width. For example, if the original multi-frame image is a 10-bit image, assuming Gray_Y = I(x,y)>>2, this means that I(x,y) is right-shifted by 2 bits to obtain a grayscale value Gray_Y with a bit width of 8. This converts the original 10-bit image into an 8-bit grayscale image.

[0064] The weighted average method refers to a method of performing a weighted average of the RGB three-channel values ​​corresponding to the same pixel to obtain the grayscale value of the pixel. For example, the formula for the weighted average method is as follows:

[0065] I(x,y)=0.3×I_R(x,y)+0.59×I_G(x,y)+0.11×I_B(x,y) Formula (1-2)

[0066] Then, I(x,y) is right-shifted to obtain a grayscale value Gray_Y with a smaller bit width. For example, if the original image of multiple frames is a 10-bit image, assuming Gray_Y = I(x,y)>>2, it means that I(x,y) is right-shifted by 2 bits to obtain a grayscale value Gray_Y with a bit width of 8. In other words, the original 10-bit image is converted into an 8-bit grayscale image.

[0067] Of course, you can also use the following formula to convert the 10-bit original image into an 8-bit grayscale image (GrayImage):

[0068] Gray_Y=([77,150,29]*[I_R(x,y),I_G(x,y),I_B(x,y)])>>10 Formula (1-3)

[0069] Here, multiplying I_R(x,y), I_G(x,y), and I_B(x,y) by 77 / 150 / 29, and then summing the multiplication results, is equivalent to expanding the 10-bit width of I_R(x,y), I_G(x,y), and I_B(x,y) to 18. Therefore, right-shifting ([77,150,29]*[I_R(x,y),I_G(x,y),I_B(x,y)]) by 10 bits yields the grayscale value Gray_Y with a bit width of 8. This converts the original 10-bit image into an 8-bit grayscale image.

[0070] Step 264 : Calculate the fusion weight map of the intermediate image corresponding to the grayscale image based on the grayscale image.

[0071] After obtaining a grayscale image with a smaller bit width, the fusion weight map of the intermediate image corresponding to the grayscale image can be calculated based on the grayscale image. Since the grayscale image with a smaller bit width is obtained, the amount of calculation required to calculate the fusion weight map can be greatly reduced by calculating the fusion weight map based on the grayscale image.

[0072] Specifically, the fusion weight map of the middle image (Middle Image) corresponding to the grayscale image (GrayImage) can be calculated in combination with the image pyramid of the grayscale image (GrayImage). The image pyramid is a collection of images arranged in a pyramid shape with gradually decreasing resolutions. The bottom of the pyramid is the high-resolution image to be processed (such as the original grayscale image (Gray Image)), and the top is the low-resolution image to be processed. When the pyramid moves from the bottom to the top, the size and resolution of the image will decrease. Figure 5 FIG. 1 is a schematic diagram of a process of converting a long exposure image into a grayscale image and then converting the grayscale image into an image pyramid in one embodiment. Figure 5 The figure shows an inverted pyramid. The L1 layer image is the base of the pyramid (the original grayscale image), and the Ln layer image is the top of the pyramid. Downsampling the original grayscale image yields images of different layers. Similarly, we can obtain image pyramids for medium-exposure images and short-exposure images.

[0073] The image pyramid here can be a Gaussian pyramid or a Laplacian pyramid, which is not limited in this application. Among them, the Gaussian pyramid includes a series of downsampled images obtained by Gaussian smoothing and subsampling, that is, the k-th layer of the Gaussian pyramid can obtain the k+1 layer of Gaussian images by smoothing and subsampling. The Gaussian pyramid contains a series of low-pass filters, whose cutoff frequency gradually increases by a factor of 2 from the upper layer to the lower layer, so the Gaussian pyramid can span a large frequency range. The Laplacian pyramid includes a series of images obtained by first reducing and then enlarging the original image. The Laplacian pyramid can be understood as the inverse form of the Gaussian pyramid.

[0074] Step 266 : Perform weighted fusion processing on the multiple frames of intermediate images based on the fusion weight map of each intermediate image to obtain a target image.

[0075] The fusion weight map of each intermediate image contains the fusion weight corresponding to each pixel in the intermediate image. Therefore, after obtaining the fusion weight map of each intermediate image, the fusion weight corresponding to the pixel at the same position in each intermediate image can be obtained from the fusion weight map of the intermediate image. Then, the pixel value of the pixel is multiplied by the fusion weight corresponding to the pixel to be fused to obtain the pixel value to be fused. Finally, the pixel values ​​to be fused corresponding to the pixels at the same position in each intermediate image are added together to obtain the pixel value after fusion.

[0076] Finally, the target image can be obtained based on the fused pixel values ​​of all pixels at the same position in each intermediate image.

[0077] In an embodiment of the present application, when performing image fusion on multiple frames of intermediate images (Middle Image), the main purpose is to obtain a target image with a more uniform brightness distribution. The grayscale image (Gray Image) can well reflect the brightness distribution in the image, so when performing image fusion, for each original image in the multiple frames of original images, the original image can be first converted into a grayscale image (Gray Image). Then, based on the grayscale image (Gray Image), the fusion weight map of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image) is calculated. Finally, based on the fusion weight map of each intermediate image (Middle Image), weighted fusion processing is performed on the multiple frames of intermediate images (Middle Image) to obtain the target image. In this way, since the grayscale image (Gray Image) can well reflect the brightness distribution in the image, the fusion weight map of the intermediate image (Middle Image) calculated based on the grayscale image (Gray Image) can better fuse the brightness of each intermediate image (Middle Image). Finally, a weighted fusion process is performed on multiple intermediate images based on the fusion weight maps of each intermediate image to obtain the target image. This improves the uniformity of the brightness distribution in the target image. This results in a high dynamic range target image containing more image information. This, in turn, improves the quality of the resulting high dynamic range image.

[0078] In the previous embodiment, the specific implementation steps for calculating the fusion weights of multiple intermediate images based on multiple original images and fusing the multiple intermediate images based on the fusion weights to obtain the target image were described. In this embodiment, step 264 is further described in detail. The specific implementation method for calculating the fusion weight map of the intermediate image corresponding to the grayscale image based on the grayscale image includes:

[0079] According to the image pyramid corresponding to the gray image and the weight pyramid corresponding to the image pyramid, a fusion weight map of the intermediate image corresponding to the gray image is calculated.

[0080] Specifically, after obtaining the image pyramids for the long-exposure image, the medium-exposure image, and the short-exposure image, a weighted pyramid corresponding to the image pyramid is calculated using Gaussian curves. A Gaussian curve corresponding to the image pyramid for the long-exposure image is preset. Similarly, a Gaussian curve corresponding to the image pyramid for the medium-exposure image is preset, and a Gaussian curve corresponding to the image pyramid for the short-exposure image is preset.

[0081] Among them, the corresponding Gaussian curves can be set based on experience according to the brightness distribution of the image pyramid of the long exposure image, the image pyramid of the medium exposure image, and the image pyramid of the short exposure image. Specifically, the following formula can be used to generate the Gaussian curve:

[0082]

[0083] Among them, σ is the variance parameter and μ is the mean parameter. Both parameters are set based on experience according to the brightness distribution of the image pyramid.

[0084] Based on the Gaussian curve, we can get a lookup table (Lut). Assuming that the grayscale image of the long exposure image is an 8-bit image, the lookup table records the output y = Weight after inputting any integer in x∈[0,255] into the above formula (1-5). expo The mapping relationship between them. That is, the Lut lookup table records the mapping relationship between 256 pairs of (x, y).

[0085] Then, for the image pyramid of the long-exposure image, the corresponding weight pyramid is calculated using the corresponding Gaussian curve. Similarly, for the image pyramid of the medium-exposure image, the corresponding weight pyramid is calculated using the corresponding Gaussian curve. For the image pyramid of the short-exposure image, the corresponding weight pyramid is calculated using the corresponding Gaussian curve. In this case, the corresponding weight pyramid can be calculated directly using the image pyramid in conjunction with the Lut lookup table. Because the Lut lookup table records the mapping between the input and output of the Gaussian curve, the weight pyramid corresponding to the image pyramid can be calculated directly using the image pyramid in conjunction with the Lut lookup table. This allows the Lut lookup table to be directly queried based on the pixel value of a pixel in the image pyramid to obtain the fusion weight corresponding to that pixel value. Furthermore, the weight pyramid corresponding to the image pyramid is obtained based on the fusion weights corresponding to the pixel values ​​of all pixels in the image pyramid. This eliminates the need to perform actual calculations based on the pixel values ​​of the image pyramid in conjunction with the Gaussian curve; the fusion weight corresponding to that pixel value can be obtained by directly querying the Lut lookup table. This accelerates image fusion processing and improves image processing efficiency.

[0086] In the embodiments of the present application, corresponding Gaussian curves are empirically configured based on the brightness distribution of the image pyramids for long-exposure images, medium-exposure images, and short-exposure images. A Lut lookup table (LUT) can be pre-derived based on the Gaussian curves. Therefore, a direct query of the LUT table based on the pixel value of a pixel in the image pyramid can obtain the weight corresponding to that pixel value. This accelerates image fusion processing and improves image processing efficiency.

[0087] In the previous embodiment, it is described that the fusion weight map of the middle image corresponding to the gray image (Gray Image) is calculated based on the image pyramid corresponding to the gray image (Gray Image) and the weight pyramid corresponding to the image pyramid. In this embodiment, Figure 6 As shown, a detailed description is given of a specific implementation method for calculating a fusion weight map of an intermediate image (Middle Image) corresponding to a grayscale image (Gray Image) according to an image pyramid corresponding to the grayscale image (Gray Image) and a weight pyramid corresponding to the image pyramid if the fusion weight map includes a first fusion weight map (exposure_map), including:

[0088] Step 620: Create a first image pyramid corresponding to the grayscale image.

[0089] For static scenes, multiple frames of original images captured at different exposure parameters are acquired. Since these images capture static scenes and do not contain moving objects, ghosting caused by moving objects will not appear in these images. Therefore, if these images capture static scenes, when fusing these intermediate images based on fusion weights, the final target image can be obtained by simply fusing them based on a common fusion weight map (exposure_map), without the need for a deghosting weight map.

[0090] Specific, combined Figure 5 Figure 2 shows a schematic diagram of converting a long-exposure image's grayscale image into a first image pyramid. The long-exposure image's grayscale image is scaled sequentially according to a preset scaling ratio to generate a first image pyramid corresponding to the long-exposure image's grayscale image (referred to as the long-exposure image's first image pyramid). The medium-exposure image's grayscale image is scaled sequentially according to a preset scaling ratio to generate a first image pyramid corresponding to the medium-exposure image's grayscale image (referred to as the medium-exposure image's first image pyramid). The short-exposure image's grayscale image is scaled sequentially according to a preset scaling ratio to generate a first image pyramid corresponding to the short-exposure image's grayscale image (referred to as the short-exposure image's first image pyramid). The first image pyramid includes n layers of first reference images, with the first reference image in layer 1 being the original grayscale image. Here, the preset scaling ratio can be determined based on experience. For example, assuming the preset scaling ratio is 1 / 2 and the resolution of the original grayscale image is 3840×2160, the resolution of the first reference image of the second layer is 1920×1080, the resolution of the first reference image of the third layer is 960×540, and so on, until the first reference image of the nth layer is generated. Of course, this application is not limited to this.

[0091] Of course, when scaling a grayscale image to generate a first image pyramid corresponding to the grayscale image, the grayscale image can also be scaled sequentially according to different preset scaling ratios. For example, when scaling the first reference image of the first layer, it is scaled based on a preset scaling ratio of 1 / 2; when scaling the first reference image of the second layer, it is scaled based on a preset scaling ratio of 1 / 5; when scaling the first reference image of the third layer, it is scaled based on a preset scaling ratio of 1 / 4; and so on, until the first reference image of the nth layer is generated. Of course, this application is not limited to this.

[0092] Step 640 : Calculate a first weight pyramid corresponding to the first image pyramid based on the first image pyramid and a first Gaussian curve corresponding to the grayscale image.

[0093] After obtaining the first image pyramids of the corresponding grayscale images (GrayImage) for the long-exposure image, the medium-exposure image, and the short-exposure image, a first weighted pyramid corresponding to the first image pyramid is calculated based on the first image pyramid combined with the first Gaussian curve corresponding to the grayscale image (GrayImage).

[0094] Specifically, first, a first Gaussian curve corresponding to a grayscale image is obtained. For grayscale images of long, medium, and short exposure images, the parameters of the empirically set first Gaussian curve are different due to the different brightness distributions of these three types of images. Consequently, the generated first Gaussian curves are also different. Next, a first weighted image corresponding to each first reference image is calculated based on the first reference image of each layer in the first image pyramid and the first Gaussian curve. Based on the pixel values ​​of pixels in the first reference image of the first layer through the first reference image of the nth layer in the first image pyramid and the first Gaussian curve, a fusion weight corresponding to the pixel value is calculated, thereby obtaining a first weighted image corresponding to each first reference image of each layer. Alternatively, the Lut lookup table corresponding to the first reference image of each layer can be directly queried based on the pixel values ​​of pixels in the first reference image of the first layer through the nth layer in the first image pyramid to obtain the fusion weight corresponding to the pixel value, thereby obtaining a first weighted image corresponding to each first reference image of each layer.

[0095] Finally, a first weighted pyramid corresponding to the first image pyramid is generated based on the first weighted images corresponding to each first reference image in the first image pyramid. That is, a first weighted pyramid corresponding to the first image pyramid is generated based on the first weighted images corresponding to the first reference images of the first layer through the first reference images of the nth layer.

[0096] Step 660: Reconstruct the first weight pyramid to generate a first fusion weight map of the intermediate image corresponding to the grayscale image.

[0097] Based on the first reference image of the first layer up to the first reference image of the nth layer in the first image pyramid, a first weight pyramid is obtained, which includes the first weight image of the first layer up to the first weight image of the nth layer. This means that the first weight pyramid contains weight information for each layer of the first reference image obtained by downsampling the grayscale image (Gray Image). Therefore, when it is necessary to obtain a first fused weight map (exposure_map) of an intermediate image (MiddleImage) corresponding to the grayscale image (Gray Image), it is necessary to upsample and accumulate the first weight images of each layer in the first weight pyramid to obtain a first fused weight map (exposure_map) having the same resolution as the grayscale image (Gray Image).

[0098] In the embodiment of the present application, a first image pyramid corresponding to a grayscale image (Gray Image) is created. Based on the first image pyramid and a first Gaussian curve corresponding to the grayscale image (Gray Image), a first weight pyramid corresponding to the first image pyramid is calculated. The first weight pyramid is reconstructed to generate a first fused weight map (exposure_map) of an intermediate image (Middle Image) corresponding to the grayscale image (Gray Image).

[0099] First, the grayscale image (Gray Image) is downsampled to obtain the first reference image of the first layer through the first reference image of the nth layer. A first image pyramid corresponding to the grayscale image (GrayImage) is generated based on the first reference images of the first layer through the first reference images of the nth layer. Then, based on the pixel values ​​of the pixels in the first reference image of the first layer through the first reference image of the nth layer in the first image pyramid, a fusion weight corresponding to the pixel value is calculated, thereby obtaining a first weight image corresponding to the first reference image of each layer. A first weight pyramid corresponding to the first image pyramid is then generated based on the first weight image of each layer. Finally, the first weight images of each layer in the first weight pyramid are upsampled and accumulated to obtain a first fused weight map (exposure_map) with the same resolution as the grayscale image (Gray Image). In this way, by first downsampling the grayscale image (Gray Image) and combining it with the first Gaussian curve to generate the first weight pyramid, weight information can be extracted from images of different resolutions, improving the accuracy of the extracted weight information. On the other hand, the first weight images of each layer in the first weight pyramid are upsampled and accumulated to obtain a first fused weight map (exposure_map) with the same resolution as the grayscale image (Gray Image). In this way, since the first fused weight map and the grayscale image (Gray Image) have the same resolution, the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image) can be fused based on the first fused weight map to obtain the target image.

[0100] In one embodiment, Figure 7 As shown, step 640, based on the first image pyramid and the first Gaussian curve corresponding to the grayscale image (GrayImage), calculates a first weight pyramid corresponding to the first image pyramid, including:

[0101] Step 642: Obtain a first Gaussian curve corresponding to the grayscale image.

[0102] The first Gaussian curve corresponding to the gray image can be set based on empirical values. Specifically, the following formula can be used to generate the first Gaussian curve:

[0103]

[0104] Among them, σ is the variance parameter and μ is the mean parameter. Both parameters are set based on experience according to the brightness distribution of the gray image.

[0105] Based on the first Gaussian curve, we can get a lookup table (Lut). Assuming that the grayscale image (GrayImage) is an 8-bit image, the lookup table records the output y=Weight after inputting any integer in x∈[0,255] into the above formula (1-5). exp The mapping relationship between o. That is, the Lut lookup table records the mapping relationship between 256 pairs of (x, y).

[0106] Since the variance and mean parameters used to generate the first Gaussian curve are set empirically based on the brightness distribution of the grayscale image, the variance and mean parameters for the long, medium, and short exposure grayscale images differ due to the different brightness distributions. Consequently, the resulting first Gaussian curves are also different.

[0107] Specifically, for the grayscale image of the long exposure image (Gray Image), the first Gaussian curve corresponding to the grayscale image of the long exposure image (GrayImage) is as follows: Figure 8 As shown in (a), the first Gaussian curve corresponding to the grayscale image of the long exposure image tends to cover the pixel area with lower brightness, so that the dark area in the target image generated by the fusion mainly takes the information of the long exposure image. Figure 9 As shown in (a), the first Gaussian curve corresponding to the grayscale image of the medium exposure image tends to cover the pixel area with medium brightness, so that the medium brightness area in the target image generated by the fusion mainly takes the information of the medium exposure image. For the grayscale image of the short exposure image, the first Gaussian curve corresponding to the grayscale image of the short exposure image is as follows: Figure 10 As shown in (a), it can be seen that the first Gaussian curve corresponding to the grayscale image of the short-exposure image tends to cover the pixel area with higher brightness, so that the bright area in the final fused target image mainly takes the information in the short-exposure image.

[0108] Step 644 , calculating a first weighted image corresponding to each first reference image based on the first reference image of each layer in the first image pyramid in combination with the first Gaussian curve;

[0109] Step 646 : Generate a first weight pyramid corresponding to the first image pyramid based on the first weight images corresponding to the first reference images in the first image pyramid.

[0110] like Figure 8 FIG. 1 is a schematic diagram of generating a first weighted pyramid based on a first image pyramid of a long exposure image in one embodiment. For the first image pyramid of the long exposure image, based on the pixel values ​​of the first reference image of the first layer to the first reference image of the nth layer in the first image pyramid, combined with Figure 8 The first Gaussian curve shown in (a) is used to calculate the fusion weight corresponding to the pixel value of the pixel, resulting in a first initial fusion weight map corresponding to the first reference image of the first layer through the first reference image of the nth layer. This first initial fusion weight map is then normalized to generate a normalized first initial fusion weight map. Based on the normalized first initial fusion weight maps of the first layer through the nth layer, a first weight pyramid corresponding to the first image pyramid of the long-exposure image is generated.

[0111] like Figure 9 FIG. 1 is a schematic diagram of generating a first weighted pyramid based on a first image pyramid of a mid-exposure image in one embodiment. For the first image pyramid of the mid-exposure image, based on the pixel values ​​of the first reference image of the first layer to the first reference image of the nth layer in the first image pyramid, combined with Figure 9 The first Gaussian curve shown in (a) is used to calculate the fusion weight corresponding to the pixel value of the pixel, resulting in a first initial fusion weight map corresponding to the first reference image of the first layer through the first reference image of the nth layer. This first initial fusion weight map is then normalized to generate a normalized first initial fusion weight map. Based on the normalized first initial fusion weight maps of the first layer through the nth layer, a first weight pyramid is generated, corresponding to the first image pyramid of the intermediate exposure image.

[0112] like Figure 10 FIG. 1 is a schematic diagram of generating a first weighted pyramid based on a first image pyramid of a short-exposure image in one embodiment. For the first image pyramid of the short-exposure image, based on the pixel values ​​of the first reference image of the first layer to the first reference image of the nth layer in the first image pyramid, combined with Figure 10The first Gaussian curve shown in (a) is used to calculate the fusion weight corresponding to the pixel value of the pixel, resulting in a first initial fusion weight map corresponding to the first reference image of the first layer through the first reference image of the nth layer. This first initial fusion weight map is then normalized to generate a normalized first initial fusion weight map. Based on the normalized first initial fusion weight maps of the first layer through the nth layer, a first weight pyramid corresponding to the first image pyramid of the short-exposure image is generated.

[0113] In the embodiment of the present application, first, different first Gaussian curves are set corresponding to the grayscale images (GrayImage) of the long-exposure image, the medium-exposure image, and the short-exposure image, respectively. Next, the first reference images of each layer in the first image pyramid of the long-exposure image are combined with the corresponding first Gaussian curves to calculate a first weight pyramid corresponding to the first image pyramid of the long-exposure image. Similarly, for the first image pyramids of the medium-exposure image and the short-exposure image, the first weight pyramid corresponding to the first image pyramid of the medium-exposure image and the first weight pyramid corresponding to the first image pyramid of the short-exposure image are calculated, respectively. In this way, different first Gaussian curves are used to extract different first weights for different brightness regions from the first image pyramids of the long-exposure image, the medium-exposure image, and the short-exposure image.

[0114] For example, the weights extracted from the first image pyramid of the long-exposure image are weighted more heavily toward dark areas; the weights extracted from the first image pyramid of the medium-exposure image are weighted more heavily toward medium-brightness areas; and the weights extracted from the first image pyramid of the short-exposure image are weighted more heavily toward bright areas. Consequently, the dark areas in the final fused target image primarily draw information from the long-exposure image, the medium-brightness areas primarily draw information from the medium-exposure image, and the bright areas primarily draw information from the short-exposure image. Ultimately, this results in a better fusion effect for the target image.

[0115] In one embodiment, step 660, reconstructing the first weight pyramid to generate a first fused weight map (exposure_map) of the intermediate image (Middle Image) corresponding to the grayscale image (GrayImage) includes:

[0116] Upsampling and accumulating the first weight images of each layer in the first weight pyramid, starting from the first weight image of the nth layer, until the first weight image of the 1st layer, to generate a first fused weight map corresponding to the first weight pyramid;

[0117] The first fused weight map corresponding to the first weight pyramid is used as the first fused weight map (exposure_map) of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image).

[0118] Specifically, the first weight pyramids corresponding to the grayscale images (Gray Image) of the long-exposure image, the medium-exposure image, and the short-exposure image are reconstructed to generate the first fused weight maps (exposure_map) of the intermediate images (Middle Image) corresponding to the grayscale images (Gray Image). Specifically, the first fused weight map (exposure_map) of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image) of the long-exposure image is generated; the first fused weight map (exposure_map) of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image) of the medium-exposure image is generated; and the first fused weight map (exposure_map) of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image) of the short-exposure image is generated.

[0119] like Figure 11 As shown, the first fusion weight map (exposure_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (Gray Image) of the long exposure image is used as an example for explanation. Figure 11 The left side is the first weight pyramid corresponding to the long exposure image, which includes n layers of first weight images. Among them, the first weight image of the L1 layer is the bottom of the pyramid, and the first weight image of the Ln layer is the top of the pyramid. The first step is to upsample and add the first weight image of the Ln layer in sequence until the first weight image of the 1st layer is reached, thereby generating a first fused weight map corresponding to the first weight pyramid. It can be understood as: starting from k=n, iteratively performing the following operations: upsampling the first weight image of each layer in the first weight pyramid, starting from the first weight image of the kth layer, in sequence to generate a new first weight image; superimposing the new first weight image with the first weight image of the k-1th layer to generate a new first weight image of the k-1th layer; until k=1, generating the first fused weight map (exposure_map) corresponding to the first weight pyramid; 1≤k≤n, and k is a positive integer.

[0120] Specifically, the first weight image of the Ln layer is upsampled to generate a new weight image with the same resolution as the first weight image of the Ln-1 layer; the new weight image is then accumulated with the first weight image of the Ln-1 layer, and the accumulated result is used as the new first weight image of the Ln-1 layer; then, the new first weight image of the Ln-1 layer is upsampled to generate a new weight image with the same resolution as the first weight image of the Ln-2 layer; the new weight image is then accumulated with the first weight image of the Ln-2 layer, and the accumulated result is used as the new first weight image of the Ln-2 layer; the above operations are repeated until the first weight image of the first layer is obtained, and a first fused weight map corresponding to the first weight pyramid is generated.

[0121] At this time, a frame of the first fusion weight map is output corresponding to a first weight pyramid, that is, a frame of the first fusion weight map is output for the first weight pyramid corresponding to the long exposure image; a frame of the first fusion weight map is output for the first weight pyramid corresponding to the medium exposure image; and a frame of the first fusion weight map is output for the first weight pyramid corresponding to the short exposure image.

[0122] In the second step, the first fused weight map corresponding to the first weight pyramid is used as the first fused weight map (exposure_map) for the intermediate image (Middle Image) corresponding to the grayscale image (GrayImage). Specifically, the first fused weight map outputted by the first weight pyramid for the long-exposure image is used as the first fused weight map (exposure_map) for the intermediate image (Middle Image) corresponding to the long-exposure image; the first fused weight map outputted by the first weight pyramid for the medium-exposure image is used as the first fused weight map (exposure_map) for the medium-exposure image; and the first fused weight map outputted by the first weight pyramid for the short-exposure image is used as the first fused weight map (exposure_map) for the short-exposure image. Finally, three first fused weight maps (exposure_maps) are generated for the three types of images with different exposure parameters: the long-exposure image, the medium-exposure image, and the short-exposure image.

[0123] In an embodiment of the present application, after generating the first weight pyramid corresponding to the grayscale images (Gray Image) of the long exposure image, the medium exposure image, and the short exposure image, since the first weight pyramid contains multiple layers of first weight images, it is necessary to process the multiple layers of first weight images to generate a frame of the first fused weight map. Here, the first weight images of each layer in the first weight pyramid are upsampled and accumulated starting from the first weight image of the nth layer to generate the first fused weight map corresponding to the first weight pyramid. By upsampling, the resolution of the accumulated first weight images can be guaranteed to be consistent; by accumulating, the weight information of the multiple layers of first weight images can be retained in the first fused weight map that is finally output. Ultimately, the accuracy and comprehensiveness of the weight information in the first fused weight map are improved.

[0124] In the aforementioned embodiment, it is described that the fusion weight map includes a first fusion weight map (exposure_map). In this embodiment, it is further described that the fusion weight map also includes a second fusion weight map (deghost_map). Figure 12 As shown, according to the image pyramid corresponding to the gray image and the weight pyramid corresponding to the image pyramid, the fusion weight map of the intermediate image corresponding to the gray image is calculated, including:

[0125] Step 1220: Create a second image pyramid corresponding to the grayscale image.

[0126] For motion scenes, multiple frames of original images captured at different exposure parameters are acquired. Since these frames capture motion and contain moving objects, and the positions of these moving objects vary across the original images, ghosting may occur in these frames. Therefore, if these frames capture motion, when fusing these intermediate images based on fusion weights, it is necessary to consider not only the first fusion weight map (exposure_map) but also the second fusion weight map (deghost_map) to obtain the final target image. This second fusion weight map (deghost_map), also known as a deghosting weight map, is primarily used to remove ghosting from an image.

[0127] In order to accurately calculate the motion area, it is necessary to perform brightness alignment on the input grayscale image. Then, a second image pyramid is created based on the brightness aligned grayscale image. The process of creating the second image pyramid for the brightness aligned grayscale image is similar to Figure 5 The process of creating a first image pyramid corresponding to the grayscale image directly based on the grayscale image is the same as that shown in , and will not be repeated here. The second image pyramid includes n layers of second reference images, where the second reference image in the first layer is the original grayscale image after brightness alignment.

[0128] Step 1240 : Calculate a second weight pyramid corresponding to the second image pyramid based on the second image pyramid and the second Gaussian curve corresponding to the grayscale image.

[0129] After obtaining the second image pyramids of the corresponding brightness-aligned grayscale images for the long, medium, and short exposure images, a second weighted pyramid corresponding to the second image pyramid is calculated based on the second image pyramid combined with the second Gaussian curve corresponding to the grayscale image.

[0130] Specifically, first, a second Gaussian curve corresponding to the grayscale image is obtained. This second Gaussian curve can be empirically determined based on the distribution of motion regions in the second image pyramids of the long-exposure image, the medium-exposure image, and the short-exposure image. Thus, the second Gaussian curves corresponding to the grayscale images of the long-exposure image, the medium-exposure image, and the short-exposure image are obtained.

[0131] Then, for the second image pyramid of the long-exposure image, a second Gaussian curve corresponding to the grayscale image (Gray Image) of the long-exposure image is obtained. Based on the second image pyramid of the long-exposure image and the second Gaussian curve corresponding to the grayscale image (GrayImage) of the long-exposure image, a second weight pyramid corresponding to the second image pyramid of the long-exposure image is calculated. Similarly, a second weight pyramid corresponding to the second image pyramid of the medium-exposure image and a second weight pyramid corresponding to the second image pyramid of the short-exposure image are calculated.

[0132] Step 1260: reconstruct the second weight pyramid to generate a second fusion weight map of the intermediate image corresponding to the grayscale image.

[0133] Based on the second reference image of the first layer to the second reference image of the nth layer in the second image pyramid, a second weight pyramid is obtained, which includes the second weight image of the first layer to the second weight image of the nth layer, that is, the second weight pyramid is obtained. The second weight pyramid includes weight information of the second reference image of each layer obtained by downsampling the grayscale image (Gray Image). Therefore, when it is necessary to obtain a second fused weight map (exposure_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (Gray Image), it is necessary to upsample and accumulate the second weight images of each layer in the second weight pyramid to obtain a frame of the second fused weight map (exposure_map) with the same resolution as the grayscale image (Gray Image).

[0134] In this embodiment of the present application, a second image pyramid corresponding to the grayscale image (Gray Image) is created. Based on the second image pyramid and a second Gaussian curve corresponding to the grayscale image (Gray Image), a second weight pyramid corresponding to the second image pyramid is calculated. The second weight pyramid is reconstructed to generate a second fused weight map (exposure_map) of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image).

[0135] First, the grayscale image (Gray Image) is downsampled to obtain second reference images from the first layer to the nth layer. A second image pyramid corresponding to the grayscale image (GrayImage) is generated based on the second reference images from the first layer to the nth layer. Then, based on the pixel values ​​of the pixels in the second reference images from the first layer to the nth layer in the second image pyramid, a second Gaussian curve is combined to calculate the fusion weights corresponding to the pixel values. This results in a second weight image corresponding to each layer of the second reference image. A second weight pyramid corresponding to the second image pyramid is then generated based on each layer of the second weight image. Finally, the second weight images of each layer in the second weight pyramid are upsampled and accumulated to obtain a second fused weight map (exposure_map) with the same resolution as the grayscale image (Gray Image). In this way, by first downsampling the grayscale image (Gray Image) and combining it with the second Gaussian curve to generate the second weight pyramid, weight information can be extracted from images of different resolutions, improving the accuracy of the extracted weight information. On the other hand, the second weight images of each layer in the second weight pyramid are upsampled and accumulated to obtain a second fused weight map (exposure_map) with the same resolution as the grayscale image (Gray Image). In this way, since the second fused weight map and the grayscale image (Gray Image) have the same resolution, the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image) can be fused based on the second fused weight map to obtain the target image.

[0136] In the previous embodiment, step 1220, creating a second image pyramid corresponding to the gray image, includes:

[0137] Brightness alignment is performed on a grayscale image based on a reference grayscale image to generate a grayscale image after brightness alignment; the reference grayscale image is a grayscale image corresponding to a first original image, and the first original image is an original image corresponding to an overexposure parameter.

[0138] In order to accurately calculate the motion area, it is necessary to perform brightness alignment on the input grayscale image. Then, a second image pyramid is created based on the brightness-aligned grayscale image. Specifically, a reference grayscale image can be screened out from the input grayscale image, and any frame in the input grayscale image can be used as the reference grayscale image. For example, the grayscale image corresponding to the long-exposure image is used as the reference grayscale image. Among them, the long-exposure image is the first original image, that is, the original image corresponding to the overexposure parameter. Then, based on the grayscale image (Gray Image) of the long-exposure image, all input grayscale images (Gray Image) are brightness-aligned to generate a brightness-aligned grayscale image (Gray Image). That is, based on the grayscale image (Gray Image) of the long-exposure image, the grayscale images (Gray Images) of the long-exposure image, the medium-exposure image, and the short-exposure image are brightness-aligned to generate a brightness-aligned grayscale image (Gray Image).

[0139] The brightness-aligned grayscale image (Gray Image) is scaled in sequence according to a preset scaling ratio to generate a second image pyramid corresponding to the brightness-aligned grayscale image (Gray Image); the second image pyramid includes n layers of second reference images, wherein the second reference image of the first layer is the original image of the brightness-aligned grayscale image (Gray Image).

[0140] Specifically, the grayscale images of the brightness-aligned long-exposure image, the medium-exposure image, and the short-exposure image are scaled in sequence according to a preset scaling ratio to generate a second image pyramid corresponding to the brightness-aligned long-exposure image (hereinafter referred to as the second image pyramid of the long-exposure image), a second image pyramid corresponding to the brightness-aligned medium-exposure image (hereinafter referred to as the second image pyramid of the medium-exposure image), and a second image pyramid corresponding to the brightness-aligned short-exposure image (hereinafter referred to as the second image pyramid of the short-exposure image). The second image pyramid includes n layers of second reference images, wherein the first layer of the second reference image is the original grayscale image corresponding to the brightness-aligned long-exposure image, the original grayscale image corresponding to the medium-exposure image, or the original grayscale image corresponding to the short-exposure image.

[0141] In an embodiment of the present application, when creating a second image pyramid corresponding to a grayscale image (Gray Image), first, the grayscale image (Gray Image) is brightness-aligned based on a reference grayscale image to generate a grayscale image (GrayImage) after brightness alignment. Secondly, the grayscale image (Gray Image) after brightness alignment is scaled in sequence according to a preset scaling ratio to generate a second image pyramid corresponding to the grayscale image (Gray Image) after brightness alignment. Brightness alignment of the grayscale image (Gray Image) based on the reference grayscale image is performed, and subsequently a second image pyramid is created based on the grayscale image after brightness alignment, which can accurately reflect the motion area in the second image pyramid. Furthermore, the accuracy of the second weight pyramid (ghost removal weight pyramid) subsequently calculated based on the second image pyramid is improved. Ultimately, the ghost removal effect in the image fusion process is improved.

[0142] In the previous embodiment, brightness alignment is performed on the grayscale image based on the reference grayscale image to generate each grayscale image after brightness alignment, including:

[0143] Using a brightness histogram equalization method, the brightness of the grayscale image (Gray Image) is aligned based on the reference grayscale image to generate a brightness-aligned grayscale image (Gray Image); or

[0144] The exposure parameter matching method is used to perform brightness alignment on the grayscale image based on the reference grayscale image to generate a grayscale image after brightness alignment.

[0145] Among them, the implementation process of using the brightness histogram equalization method includes: first, obtaining the brightness histogram of the grayscale image, that is, obtaining the brightness histogram of the grayscale image of the long exposure image, the medium exposure image, and the short exposure image respectively. Secondly, based on the brightness histogram of the reference grayscale image, the brightness histogram of the other grayscale images is histogram equalized to obtain the equalized brightness histogram corresponding to the other grayscale images. Finally, based on the brightness histogram of the reference grayscale image and the brightness histogram corresponding to the other grayscale images after equalization, the brightness histogram of the grayscale image after brightness alignment is obtained. Then, the brightness histogram is converted into a grayscale image to obtain the brightness aligned grayscale image (Gray Image).

[0146] Combine Figure 13 FIG. 1 is a schematic diagram of an embodiment of a process for equalizing the brightness histogram of a grayscale image. Assuming that the reference grayscale image is a grayscale image of a long exposure image, then Figure 13 The upper middle row shows the brightness histograms of the grayscale images of the long exposure image, medium exposure image, and short exposure image, respectively. Figure 13The lower middle row shows the brightness histograms of the grayscale images of the medium exposure image and the short exposure image, respectively. Since the reference grayscale image is the grayscale image of the long exposure image, the brightness histogram of the grayscale image of the long exposure image does not change during the equalization process.

[0147] An exposure parameter matching method can also be used to perform brightness alignment on the grayscale image (Gray Image) based on the reference grayscale image to generate a grayscale image (Gray Image) after brightness alignment. Specifically, first, the exposure parameters when shooting the long exposure image, the medium exposure image, and the short exposure image are obtained. For example, the exposure parameters include shutter time (Shutter) and gain (SensorGain), etc. Of course, this application does not limit this. Secondly, if it is still assumed that the reference grayscale image is the grayscale image of the long exposure image, the brightness alignment coefficient BrightAlign of the medium exposure image and the brightness alignment coefficient BrightAlign of the short exposure image can be calculated based on the following formula, for example:

[0148]

[0149]

[0150] Finally, you can multiply the pixel value of each pixel in the mid-exposure image by BrightAlign 中曝 , we can get the grayscale image of the medium exposure image after brightness alignment. Similarly, we can multiply the pixel value of each pixel in the short exposure image by BrightAlign 短曝 , we can get the grayscale image of the short exposure image after brightness alignment.

[0151] In embodiments of the present application, when performing brightness alignment on grayscale images based on a reference grayscale image to generate brightness-aligned grayscale images, various methods can be used. For example, brightness histogram equalization or exposure parameter matching can be used to perform brightness alignment on grayscale images based on the reference grayscale image to generate brightness-aligned grayscale images. Brightness histogram equalization can be used to enhance the local contrast of the grayscale image without affecting the overall contrast, thereby achieving a better distribution of brightness across the brightness histogram. Exposure parameter matching, on the other hand, allows for global brightness alignment of the grayscale images.

[0152] In one embodiment, Figure 14 As shown, step 1240, based on the second image pyramid and a second Gaussian curve corresponding to the gray image, calculates a second weight pyramid corresponding to the second image pyramid, including:

[0153] Step 1242 : Calculate a pixel difference pyramid between the second image pyramid and a target second image pyramid; the target second image pyramid is any one of the second image pyramids; the pixel difference pyramid includes n layers of pixel difference images, and the pixel difference image of the kth layer is the pixel difference image between the second reference image of the kth layer in the second image pyramid and the second reference image of the kth layer in the target second image pyramid.

[0154] First, a target second image pyramid is determined from the second image pyramids of the long-exposure image, the medium-exposure image, and the short-exposure image. The target second image pyramid is any second image pyramid from each of the second image pyramids. Next, a pixel difference pyramid is calculated between the second image pyramid and the target second image pyramid. For example, assuming the medium-exposure image's second image pyramid is used as the target second image pyramid, a pixel difference pyramid is calculated between the long-exposure image's second image pyramid and the medium-exposure image's second image pyramid; and a pixel difference pyramid is calculated between the short-exposure image's second image pyramid and the medium-exposure image's second image pyramid. The pixel difference pyramid between the medium-exposure image's second image pyramid and itself is zero (a completely black image).

[0155] Specifically, when calculating the pixel difference pyramid between the second image pyramid and the target second image pyramid, a pixel difference image is calculated between the second reference image at each layer in the second image pyramid and the target second reference image at the same layer in the target second image pyramid. Then, a pixel difference pyramid is obtained based on the pixel difference images at each layer. When calculating the pixel difference image, the position difference between each pixel in the second reference image and the target second reference image at the same layer is calculated. The pixel difference pyramid, like the second image pyramid, also includes n layers of pixel difference images, and the pixel difference image at the kth layer is a pixel difference image (Diffmap) between the second reference image at the kth layer in the second image pyramid and the second reference image at the kth layer in the target second image pyramid.

[0156] Step 1244 : Obtain a Gaussian curve corresponding to the pixel difference pyramid, and use the Gaussian curve corresponding to the pixel difference pyramid as a second Gaussian curve corresponding to the grayscale image.

[0157] Specifically, the following formula can be used to generate the second Gaussian curve:

[0158]

[0159] Where σ is the variance parameter, and offset is the threshold parameter / offset parameter. Both parameters are set empirically based on the distribution of motion regions in the image pyramid. The threshold parameter / offset parameter refers to the position difference / offset threshold between pixels in the same second reference image layer in different second image pyramids. diff is the pixel value in the pixel difference map at each layer in the pixel difference pyramid.

[0160] Similarly, based on the second Gaussian curve, we can get a lookup table (Lut). Assuming that the grayscale image of the long exposure image is an 8-bit image, the lookup table records the output y = Weight after inputting any integer in x∈[0,255] into the above formula (1-5). deghost The mapping relationship between them. That is, the Lut (Look-Up-Table) records the mapping relationship between 256 pairs of (x, y).

[0161] After the Gaussian curve corresponding to the pixel difference pyramid is obtained through the above formula, the second Gaussian curve corresponding to the grayscale image is obtained.

[0162] Step 1246 , for each pixel difference image in the pixel difference image pyramid, calculate a second weight image corresponding to the pixel difference image based on the pixel difference image combined with a second Gaussian curve corresponding to the grayscale image;

[0163] Step 1248 : Generate a second weight pyramid corresponding to the second image pyramid based on the second weight image corresponding to the pixel difference images of each layer.

[0164] By inputting the pixel values ​​in each pixel difference map layer of the pixel difference pyramid into the formula of the second Gaussian curve, a second weight corresponding to the pixel value in the pixel difference image can be obtained for each pixel difference map layer. Then, based on the second weights corresponding to the pixel values ​​in the pixel difference image, a second weight image corresponding to the pixel difference image can be obtained.

[0165] The second weight image is then normalized to generate a normalized second weight image. Based on the normalized second weight images from the first layer to the nth layer, a second weight pyramid corresponding to the second image pyramid is generated. In this way, second weight pyramids corresponding to the second image pyramids for the long-exposure image, the medium-exposure image, and the short-exposure image are obtained.

[0166] In this embodiment of the present application, a target second image pyramid is first determined from the second image pyramids of the long-exposure image, the medium-exposure image, and the short-exposure image. A pixel difference pyramid is then calculated between the second image pyramid and the target second image pyramid. Different Gaussian curves are set for different pixel difference pyramids as the second Gaussian curves corresponding to the grayscale image. Next, for each pixel difference image layer in the pixel difference image pyramid, a second weight image corresponding to the pixel difference image is calculated based on the pixel difference image combined with the second Gaussian curve corresponding to the grayscale image. Finally, a second weight pyramid corresponding to the second image pyramid is generated based on the second weight images corresponding to each pixel difference image layer.

[0167] Because different Gaussian curves are set for different pixel difference pyramids as the second Gaussian curves corresponding to the grayscale image, different second weights can be extracted from the pixel difference images at each layer of the pixel difference image pyramid based on the Gaussian curves. For example, if pixel difference information exceeds a certain threshold, a higher weight is assigned to that pixel difference information. Consequently, ghosting can be effectively removed using different second weights, ultimately achieving a better fusion effect on the target image.

[0168] In the previous embodiment, the second weight pyramid is reconstructed to generate a second fusion weight map (exposure_map) of the intermediate image (Middle Image) corresponding to the grayscale image (GrayImage), including:

[0169] Upsampling and accumulating the second weight images of each layer in the second weight pyramid, starting from the second weight image of the nth layer, until the second weight image of the first layer, to generate a second fused weight map corresponding to the second weight pyramid;

[0170] The second fused weight map corresponding to the second weight pyramid is used as the second fused weight map (exposure_map) of the intermediate image (Middle Image) corresponding to the grayscale image (GrayImage).

[0171] Specifically, the second weight pyramids corresponding to the grayscale images (GrayImage) of the long-exposure image, the medium-exposure image, and the short-exposure image are reconstructed to generate the second fused weight maps (exposure_map) of the intermediate images (MiddleImage) corresponding to the grayscale images (GrayImage). Specifically, the second fused weight map (exposure_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage) of the long-exposure image is generated; the second fused weight map (exposure_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage) of the medium-exposure image is generated; and the second fused weight map (exposure_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage) of the short-exposure image is generated.

[0172] like Figure 15 As shown, the second fusion weight map (exposure_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage) of the long exposure image is used as an example for explanation. Figure 15 On the left is a pixel difference pyramid between the second image pyramid of the long exposure image and the second image pyramid of the medium exposure image. Figure 15 The middle part is the second weight pyramid corresponding to the long exposure image, and the second weight pyramid includes n layers of second weight images. Among them, the second weight image of the L1 layer is the bottom of the pyramid, and the second weight image of the Ln layer is the top of the pyramid. The second step is to upsample and accumulate the second weight image of the Ln layer in sequence until the second weight image of the 1st layer is reached, and a second fused weight map corresponding to the second weight pyramid is generated. It can be understood as: starting from k=n, the following operations are performed iteratively: the second weight images of each layer in the second weight pyramid, starting from the second weight image of the kth layer, are upsampled in sequence to generate a new second weight image; the new second weight image is superimposed with the second weight image of the k-1th layer to generate a new second weight image of the k-1th layer; until k=1, a second fused weight map (exposure_map) corresponding to the second weight pyramid is generated; 1≤k≤n, and k is a positive integer.

[0173] Specifically, the second weight image of the Ln layer is upsampled to generate a new weight image with the same resolution as the second weight image of the Ln-1 layer; the new weight image is then accumulated with the second weight image of the Ln-1 layer, and the accumulated result is used as the new second weight image of the Ln-1 layer; then, the new second weight image of the Ln-1 layer is upsampled to generate a new weight image with the same resolution as the second weight image of the Ln-2 layer; the new weight image is accumulated with the second weight image of the Ln-2 layer, and the accumulated result is used as the new second weight image of the Ln-2 layer; the above operations are repeated until the second weight image of the first layer is obtained, and a second fused weight map corresponding to the second weight pyramid is generated.

[0174] At this time, a frame of the second fusion weight map is output corresponding to a second weight pyramid, that is, a frame of the second fusion weight map is output for the second weight pyramid corresponding to the long exposure image; a frame of the second fusion weight map is output for the second weight pyramid corresponding to the medium exposure image; and a frame of the second fusion weight map is output for the second weight pyramid corresponding to the short exposure image.

[0175] In the second step, the second fused weight map corresponding to the second weight pyramid is used as the second fused weight map (exposure_map) for the intermediate image (Middle Image) corresponding to the grayscale image (GrayImage). Specifically, the second fused weight map outputted by the second weight pyramid for the long-exposure image is used as the second fused weight map (exposure_map) for the intermediate image (MiddleImage) corresponding to the long-exposure image; the second fused weight map outputted by the second weight pyramid for the medium-exposure image is used as the second fused weight map (exposure_map) for the medium-exposure image; and the second fused weight map outputted by the second weight pyramid for the short-exposure image is used as the second fused weight map (exposure_map) for the short-exposure image. Finally, three second fused weight maps (exposure_map) are generated for the three types of images with different exposure parameters: the long-exposure image, the medium-exposure image, and the short-exposure image.

[0176] In an embodiment of the present application, after generating the second weight pyramid corresponding to the grayscale images (Gray Image) of the long exposure image, the medium exposure image, and the short exposure image, since the second weight pyramid contains multiple layers of second weight images, it is necessary to process the multiple layers of second weight images to generate a frame of second fused weight map. Here, the second weight images of each layer in the second weight pyramid are upsampled and accumulated starting from the second weight image of the nth layer to generate the second fused weight map corresponding to the second weight pyramid. By upsampling, the resolution of the accumulated second weight images can be guaranteed to be consistent; by accumulating, the weight information of the multiple layers of second weight images can be retained in the second fused weight map that is finally output. Ultimately, the accuracy and comprehensiveness of the weight information in the second fused weight map are improved.

[0177] In the previous embodiment, weighted fusion processing is performed on multiple frames of intermediate images based on the fusion weight maps of each intermediate image to obtain a target image, including:

[0178] A new fusion weight map (fusion_map) is obtained by multiplying the first fusion weight map (exposure_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage) and the second fusion weight map (deghost_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage);

[0179] Based on the new fusion weight map (fusion_map) of each intermediate image (Middle Image), weighted fusion processing is performed on multiple frames of intermediate images (Middle Image) to obtain the target image.

[0180] Specifically, when multiplying the first fusion weight map (exposure_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage) and the second fusion weight map (deghost_map) of the intermediate image (Middle Image) corresponding to the grayscale image (GrayImage) to obtain a new fusion weight map (fusion_map), the following formula can be used for multiplication:

[0181] Weight fusion =Weight expo ×Weight deghost Formula (1-9)

[0182] Among them, Weight expoRepresents the first weight in the first fusion weight map (exposure_map) of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image), Weight deghost Represents the second weight in the second fusion weight map (deghost_map) of the intermediate image (Middle Image) corresponding to the grayscale image (GrayImage), Weight fution Represents the fusion weights in the new fusion weight map (fusion_map).

[0183] Combine Figure 16 , which is a schematic diagram of multiplying the first fusion weight map (exposure_map) and the second fusion weight map (deghost_map) to obtain a new fusion weight map (fusion_map) in one embodiment.

[0184] Then, based on the new fusion weight map (fusion_map) of each intermediate image (Middle Image), multiple frames of intermediate images (Middle Image) can be weightedly fused to obtain the target image. That is, based on the new fusion weight map (fusion_map) of the intermediate image (Middle Image) of the long exposure image, the intermediate image of the long exposure image is multiplied to obtain the first result; based on the new fusion weight map (fusion_map) of the intermediate image (Middle Image) of the medium exposure image, the intermediate image of the medium exposure image is multiplied to obtain the second result; based on the new fusion weight map (fusion_map) of the intermediate image (Middle Image) of the short exposure image, the intermediate image of the short exposure image is multiplied to obtain the third result; and the first result, second result, and third result are summed to obtain the target image.

[0185] In an embodiment of the present application, a first fusion weight map (exposure_map) and a second fusion weight map (deghost_map) of multiple intermediate images are calculated based on multiple frames of original images. The first fusion weight map (exposure_map) and the second fusion weight map (deghost_map) are then multiplied together to obtain a new fusion weight map (fusion_map). Subsequently, image fusion is performed on the multiple intermediate images (Middle Image) based on the new fusion weight map to obtain a target image with a high dynamic range that contains more image information. This improves the image quality of the synthesized high dynamic range image.

[0186] In the previous embodiment, step 220, obtaining multiple frames of original images captured under different exposure parameters, includes:

[0187] Acquire multiple frames of original RAW images captured under different exposure parameters; the different exposure parameters include at least two sets of parameters selected from the group consisting of normal exposure parameters, overexposure parameters, and underexposure parameters;

[0188] Align multiple frames of original RAW images to generate multiple aligned RAW images;

[0189] The RAW images after multiple frames of alignment are preprocessed to generate multiple frames of original images; the preprocessing includes at least one of black level correction, lens shading correction, white balance correction and demosaicing.

[0190] Combine Figure 17 FIG. 1 is a schematic diagram illustrating generating multiple frames of raw images in one embodiment. First, an electronic device captures multiple frames of raw RAW images using a CMOS sensor under different exposure parameters. CMOS stands for Complementary Metal Oxide Semiconductor. Assume that the bit width of the raw RAW image is 10 bits. The different exposure parameters include three groups of exposure parameters: normal exposure parameters, overexposure parameters, and underexposure parameters. Of course, the different exposure parameters may also include at least two of the three groups. For example, the different exposure parameters may include normal exposure parameters and overexposure parameters, but this is not a limitation in this application. Accordingly, the multiple frames of raw images may include at least two frames of raw images, each of which is a raw RAW image. A RAW image, or RAW file, is a file that records the original information of the image sensor and also records certain shooting parameters (e.g., ISO data, shutter speed, aperture value, white balance parameters, etc.) during image capture. RAW files can preserve the most original image information to the greatest extent possible, facilitating subsequent accurate image processing.

[0191] like Figure 17As shown, the multiple frames of original RAW images include Long RAW images (long exposure RAW images), Middle RAW images (medium exposure RAW images), and Short RAW images (short exposure RAW images). Here, the number of Long RAW images is not limited to one frame, and can also be multiple frames (for example, 3 frames). Similarly, the number of Middle RAW images is not limited to one frame, and the number of Short RAW images is not limited to one frame. If some or all of the number of Long RAW images, Middle RAW images, and Short RAW images is not one frame, then multiple frames of Long RAW images are synthesized separately to generate one frame of synthesized Long RAW image; multiple frames of Middle RAW images are synthesized separately to generate one frame of synthesized Middle RAW image; multiple frames of Short RAW images are synthesized separately to generate one frame of synthesized Short RAW image. Among them, the use of multi-frame synthesis can achieve the effect of image denoising.

[0192] Next, multiple frames of original RAW images are aligned to generate aligned RAW images. Here, if the number of Long RAW images, Middle RAW images, and Short RAW images is one frame, the Long RAW images, Middle RAW images, and Short RAW images are directly aligned to generate aligned Long RAW images, Middle RAW images, and Short RAW images. If the number of Long RAW images, Middle RAW images, and Short RAW images is not one frame, the synthesized Long RAW image, the synthesized Middle RAW image, and the synthesized Short RAW image are aligned to generate aligned Long RAW images, Middle RAW images, and Short RAW images. For example, if there are three Long RAW images, one Middle RAW image, and one Short RAW image, the synthesized Long RAW image, one Middle RAW image, and one Short RAW image are aligned.

[0193] Finally, the RAW images after multi-frame alignment are preprocessed to generate multi-frame original images; the preprocessing includes at least one of black level correction (BLC), lens shading correction (LSC), white balance correction (WB) and demosaicing. Figure 17As shown in the figure, the aligned Long RAW image, Middle RAW image, and Short RAW image are sequentially subjected to black level correction, lens shading correction, white balance correction, and demosaicing to generate Long RGB image, Middle RGB image, and Short RGB image. These Long RGB image, Middle RGB image, and Short RGB image are then multi-frame raw images, and the resulting multi-frame raw images also have a bit width of 10 bits.

[0194] Perform image fusion on multiple 10-bit original images based on the fusion weight map (fusion_map) to output a 16-bit RGB image. Then perform tone mapping on the 16-bit RGB image to output a 10-bit RGB image.

[0195] In an embodiment of the present application, when acquiring multiple frames of raw images captured under different exposure parameters, the multiple frames of raw images captured under different exposure parameters are first acquired; secondly, the multiple frames of raw images are aligned to generate multiple aligned raw images; and finally, the multiple aligned raw images are preprocessed to generate multiple raw images; the preprocessing includes at least one of black level correction, lens shading correction, white balance correction, and demosaicing. By aligning and preprocessing the multiple frames of raw images, the image quality of the resulting multiple frames of raw images (RGB images) is improved.

[0196] In the aforementioned embodiment, it is described that the fusion weight map includes a first fusion weight map (exposure_map). In this embodiment, it is further described that the fusion weight map also includes a third fusion weight map (deghost_map). Figure 18 As shown, an image fusion method is provided, which also includes:

[0197] Step 1820 : For each aligned RAW image in the multiple frames of aligned RAW images, convert the aligned RAW image into a grayscale image.

[0198] When converting the aligned RAW images in the multiple aligned RAW images into grayscale images, the aligned RAW images can be converted into grayscale images by using methods such as averaging and weighted averaging, which are not limited in this application. The bit width of the grayscale image obtained here is consistent with the bit width of the aligned RAW image. Therefore, in order to reduce the amount of calculation, the grayscale image can be right-shifted to obtain a grayscale image with a smaller bit width. For example, a 10-bit aligned RAW image is converted into an 8-bit grayscale image.

[0199] Step 1840: Create a third image pyramid corresponding to the grayscale image.

[0200] For motion scenes, multiple frames of original images captured at different exposure parameters are acquired. Since these frames capture motion and contain moving objects, and the positions of these moving objects vary across the original images, ghosting caused by these moving objects may appear in the multiple frames. Therefore, if these frames capture motion, when fusing the multiple intermediate images based on fusion weights, it is necessary to fuse the multiple intermediate images based not only on the first fusion weight map (exposure_map) but also on the third fusion weight map (deghost_map) to obtain the final target image. This third fusion weight map (deghost_map) is also considered another deghosting weight map, primarily used to remove ghosting from images.

[0201] In order to accurately calculate the motion area, it is necessary to align the brightness of the input grayscale image. Then, a third image pyramid is created based on the brightness-aligned grayscale image. The process of creating the third image pyramid for the brightness-aligned grayscale image is similar to Figure 5 The process of creating a first image pyramid corresponding to the grayscale image directly based on the grayscale image is the same as that shown in , and will not be repeated here. The third image pyramid includes n layers of third reference images, where the third reference image in the first layer is the original grayscale image after brightness alignment.

[0202] Step 1860 : Calculate a third weight pyramid corresponding to the third image pyramid based on the third image pyramid and the third Gaussian curve corresponding to the grayscale image.

[0203] After obtaining the third image pyramid of the corresponding brightness-aligned grayscale image for the long, medium, and short exposure images, a third weighted pyramid corresponding to the third image pyramid is calculated based on the third image pyramid combined with the third Gaussian curve corresponding to the grayscale image.

[0204] Specifically, first, a third Gaussian curve corresponding to the grayscale image is obtained. This third Gaussian curve can be empirically determined based on the distribution of motion regions in the third image pyramids of the long, medium, and short exposure images. Thus, the third Gaussian curves corresponding to the grayscale images of the long, medium, and short exposure images are obtained.

[0205] Then, for the third image pyramid of the long-exposure image, a third Gaussian curve corresponding to the long-exposure image's grayscale image (Gray Image) is obtained. Based on the third image pyramid of the long-exposure image and the third Gaussian curve corresponding to the long-exposure image's grayscale image (Gray Image), a third weight pyramid corresponding to the long-exposure image's third image pyramid is calculated. Similarly, a third weight pyramid corresponding to the third image pyramid of the medium-exposure image and a third weight pyramid corresponding to the third image pyramid of the short-exposure image are calculated. Here, the third weight pyramid can also be referred to as another deghosting weight pyramid.

[0206] Step 1880: reconstruct the third weight pyramid to generate a third fusion weight map of the intermediate image corresponding to the grayscale image.

[0207] Based on the third reference image of the first layer through the third reference image of the nth layer in the third image pyramid, a third weight pyramid is obtained, including the third weight image of the first layer through the third weight image of the nth layer, that is, a third weight pyramid is obtained. The third weight pyramid includes weight information for each layer of the third reference image obtained by downsampling the grayscale image (GrayImage). Therefore, when it is necessary to obtain a third fused weight map (deghost_map) of an intermediate image (Middle Image) corresponding to the grayscale image (GrayImage), it is necessary to upsample and accumulate the third weight images of each layer in the third weight pyramid to obtain a frame of the third fused weight map (deghost_map) having the same resolution as the grayscale image (GrayImage).

[0208] In an embodiment of the present application, for each aligned RAW image in the multiple aligned RAW image frames, the aligned RAW image is converted into a grayscale image (Gray Image). A third image pyramid corresponding to the grayscale image (GrayImage) is created, and a third weight pyramid corresponding to the third image pyramid is calculated based on the third image pyramid combined with a third Gaussian curve corresponding to the grayscale image (GrayImage). The third weight pyramid is reconstructed to generate a third fused weight map (deghost_map) of an intermediate image (Middle Image) corresponding to the grayscale image (GrayImage).

[0209] Because the third weight pyramid is also a deghosting weight pyramid, and deghosting can only be performed after the grayscale image is brightness aligned, the calculation process of the third weight pyramid can be performed in the RAW domain. The generated third fusion weight map (deghost_map) is passed to be multiplied with the first fusion weight map (exposure_map) to obtain a new fusion weight map (fusion_map). Finally, based on the new fusion weight map (fusion_map), a weighted fusion process is performed on multiple intermediate images (Middle Image) to obtain the target image. Due to the linear advantage of the RAW domain, the accuracy of image alignment will be improved, which in turn can improve the accuracy of the subsequently calculated third fusion weight map (deghost_map).

[0210] In the previous embodiment, a third fusion weight map (deghost_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage) is generated. Then, based on the fusion weight maps of each intermediate image (MiddleImage), a weighted fusion process is performed on multiple frames of intermediate images (MiddleImage) to obtain a target image, including:

[0211] Multiplying the first fusion weight map (exposure_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage) and the third fusion weight map (deghost_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage) to obtain a new fusion weight map (fusion_map) of the intermediate image (MiddleImage);

[0212] Based on the new fusion weight map (fusion_map) of each intermediate image (Middle Image), weighted fusion processing is performed on multiple frames of intermediate images (Middle Image) to obtain the target image.

[0213] Specifically, when multiplying the first fusion weight map (exposure_map) of the intermediate image (MiddleImage) corresponding to the grayscale image (GrayImage) and the third fusion weight map (deghost_map) of the intermediate image (Middle Image) corresponding to the grayscale image (GrayImage) to obtain a new fusion weight map (fusion_map), formula (1-10) can be used for multiplication:

[0214] Weight fusion =Weight expo ×Weight deghost Formula(1-10)

[0215] Among them, Weight expo Represents the first weight in the first fusion weight map (exposure_map) of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image), Weight deghost Represents the third weight in the third fusion weight map (deghost_map) of the intermediate image (Middle Image) corresponding to the grayscale image (GrayImage), Weight fution Represents the fusion weights in the new fusion weight map (fusion_map).

[0216] Then, based on the new fusion weight map (fusion_map) of each intermediate image (MiddleImage), multiple frames of intermediate images (MiddleImage) can be weightedly fused to obtain the target image. That is, based on the new fusion weight map (fusion_map) of the intermediate image (MiddleImage) of the long exposure image, the intermediate image of the long exposure image is multiplied to obtain the first result; based on the new fusion weight map (fusion_map) of the intermediate image (MiddleImage) of the medium exposure image, the intermediate image of the medium exposure image is multiplied to obtain the second result; based on the new fusion weight map (fusion_map) of the intermediate image (MiddleImage) of the short exposure image, the intermediate image of the short exposure image is multiplied to obtain the third result; and the first result, second result, and third result are summed to obtain the target image.

[0217] In an embodiment of the present application, a first fusion weight map (exposure_map) and a third fusion weight map (deghost_map) of multiple intermediate images are calculated based on multiple frames of original images. The first fusion weight map (exposure_map) and the third fusion weight map (deghost_map) are then multiplied to obtain a new fusion weight map (fusion_map). Subsequently, image fusion is performed on the multiple intermediate images (Middle Image) based on the new fusion weight map to obtain a target image with a high dynamic range that contains more image information. This improves the image quality of the synthesized high dynamic range image.

[0218] In a specific embodiment, Figure 19 As shown, an image fusion method is provided, comprising:

[0219] Step 1902 , input three frames of 10-bit RGB images captured under normal exposure parameters, overexposure parameters, and underexposure parameters respectively;

[0220] In step 1904, a mapping table may be searched to stretch the three 10-bit RGB images to obtain three 16-bit RGB images.

[0221] Step 1906: Convert the three frames of 10-bit RGB images into 8-bit grayscale images.

[0222] Step 1908: Create a first image pyramid based on three frames of 8-bit grayscale images.

[0223] Step 1910: Calculate a first weight pyramid corresponding to the first image pyramid based on the first image pyramid and a first Gaussian curve corresponding to the gray image.

[0224] Step 1912: Reconstruct the first weight pyramid to generate a first fusion weight map (exposure_map) of the 16-bit RGB image.

[0225] Step 1914: Using a luminance histogram equalization method, the grayscale image (Gray Image) is brightness-aligned based on the reference grayscale image to generate a brightness-aligned grayscale image (Gray Image). The reference grayscale image is first determined from the three 8-bit grayscale images.

[0226] Step 1916, creating a second image pyramid based on the grayscale image after brightness alignment;

[0227] Step 1918: Calculate a second weight pyramid corresponding to the second image pyramid based on the second image pyramid and the second Gaussian curve corresponding to the grayscale image.

[0228] Step 1920: reconstruct the second weight pyramid to generate a second fused weight map (exposure_map) of the 16-bit RGB image.

[0229] Step 1922: multiply the first fusion weight map (exposure_map) and the second fusion weight map (deghost_map) to obtain a new fusion weight map (fusion_map);

[0230] Step 1924 : Perform weighted fusion processing on the three 16-bit RGB images based on the new fusion weight map (fusion_map) corresponding to the three 16-bit RGB images to obtain a 16-bit target image.

[0231] In an embodiment of the present application, multiple frames of original images taken under different exposure parameters are obtained, and the multiple frames of original images are stretched to obtain multiple frames of intermediate images (Middle Image); wherein the bit width of the intermediate image (Middle Image) is greater than the bit width of the original image. The fusion weights of the multiple frames of intermediate images (Middle Image) are calculated based on the multiple frames of original images, and the multiple frames of intermediate images (Middle Image) are fused based on the fusion weights to obtain a target image. Since the multiple frames of original images taken under different exposure parameters can capture image details of different dynamic ranges respectively, and the multiple frames of original images are stretched, the bit width of the obtained multiple frames of intermediate images (Middle Image) is larger and can contain more image information. Therefore, the fusion weights of the multiple frames of intermediate images (Middle Image) are calculated based on the multiple frames of original images, and the multiple frames of intermediate images (Middle Image) are fused based on the fusion weights to obtain a target image with a high dynamic range containing more image information. In addition, the image quality of the synthesized high dynamic range image is improved.

[0232] It should be understood that, although the various steps in the above flow chart are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the above flow chart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0233] In one embodiment, Figure 20 As shown, an image fusion device 2000 is provided, the device comprising:

[0234] The original image acquisition module 2020 is used to acquire multiple frames of original images captured under different exposure parameters;

[0235] The intermediate image acquisition module 2040 is used to stretch multiple frames of original images to obtain multiple frames of intermediate images (Middle Image); the bit width of the intermediate image (Middle Image) is greater than the bit width of the original image;

[0236] The image fusion module 2060 is used to calculate the fusion weights of multiple frames of intermediate images (Middle Image) according to the multiple frames of original images, and perform image fusion on the multiple frames of intermediate images (Middle Image) based on the fusion weights to obtain a target image.

[0237] In one embodiment, Figure 21 As shown, the image fusion module 2060 includes:

[0238] A grayscale image conversion unit 2062 is used to convert each original image in the multiple frames of original images into a grayscale image.

[0239] The fusion weight map calculation unit 2064 is used to calculate the fusion weight map of the intermediate image (Middle Image) corresponding to the gray image (Gray Image) according to the gray image (Gray Image);

[0240] The weighted fusion unit 2066 is used to perform weighted fusion processing on multiple frames of intermediate images (Middle Image) based on the fusion weight map of each intermediate image (Middle Image) to obtain a target image.

[0241] In one embodiment, the fusion weight map calculation unit 2066 is further configured to calculate a fusion weight map of an intermediate image corresponding to the gray image according to an image pyramid corresponding to the gray image and a weight pyramid corresponding to the image pyramid.

[0242] In one embodiment, Figure 22 As shown, the fusion weight map includes a first fusion weight map, and the fusion weight map calculation unit 2066 includes:

[0243] A first image pyramid creating subunit 2066a is configured to create a first image pyramid corresponding to a gray image.

[0244] A first weight pyramid calculation subunit 2066b is configured to calculate a first weight pyramid corresponding to the first image pyramid based on the first image pyramid and a first Gaussian curve corresponding to the grayscale image (GrayImage);

[0245] The first fusion weight map calculation subunit 2066c is used to reconstruct the first weight pyramid to generate a first fusion weight map of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image).

[0246] In one embodiment, the first image pyramid creation subunit is further configured to sequentially scale the grayscale image (Gray Image) according to a preset scaling ratio to generate a first image pyramid corresponding to the grayscale image (Gray Image); the first image pyramid includes n layers of first reference images, wherein the first reference image of the first layer is the original grayscale image (Gray Image).

[0247] In one embodiment, the first weight pyramid calculation subunit 2066b is further used to obtain a first Gaussian curve corresponding to the grayscale image (GrayImage); calculate a first weight image corresponding to each first reference image based on each layer of the first reference image in the first image pyramid in combination with the first Gaussian curve; and generate a first weight pyramid corresponding to the first image pyramid based on the first weight image corresponding to each first reference image in the first image pyramid.

[0248] In one embodiment, the first fused weight map calculation subunit 2066c is further used to upsample and accumulate the first weight images of each layer in the first weight pyramid, starting from the first weight image of the nth layer, until the first weight image of the 1st layer, to generate a first fused weight map corresponding to the first weight pyramid; and use the first fused weight map corresponding to the first weight pyramid as the first fused weight map of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image).

[0249] In one embodiment, the first fused weight map calculation subunit 2066c is further configured to iteratively perform the following operations starting from k=n: up-sample the first weight images of each layer in the first weight pyramid, starting from the first weight image of the kth layer, to generate a new first weight image; superimpose the new first weight image with the first weight image of the k-1th layer to generate a new first weight image of the k-1th layer; until k=1, a first fused weight map corresponding to the first weight pyramid is generated; 1≤k≤n, and k is a positive integer.

[0250] In one embodiment, the fused weight map further includes a second fused weight map, and the fused weight map calculation unit 2064 includes:

[0251] A second image pyramid creating subunit, configured to create a second image pyramid corresponding to a gray image;

[0252] a second weight pyramid calculation subunit, configured to calculate a second weight pyramid corresponding to the second image pyramid based on the second image pyramid and a second Gaussian curve corresponding to the grayscale image (GrayImage);

[0253] The second fusion weight map calculation subunit is used to reconstruct the second weight pyramid and generate a second fusion weight map of the intermediate image (Middle Image) corresponding to the gray image (Gray Image).

[0254] In one embodiment, the second image pyramid creation subunit is further configured to perform brightness alignment on a grayscale image based on a reference grayscale image to generate a brightness-aligned grayscale image. The reference grayscale image is a grayscale image corresponding to a first original image, and the first original image is an original image corresponding to an overexposure parameter. The brightness-aligned grayscale image is sequentially scaled according to a preset scaling ratio to generate a second image pyramid corresponding to the brightness-aligned grayscale image. The second image pyramid includes n layers of second reference images, wherein the second reference image in the first layer is the brightness-aligned original grayscale image.

[0255] In one embodiment, the second image pyramid creation subunit is further configured to perform brightness alignment on the grayscale image (Gray Image) based on the reference grayscale image using a brightness histogram equalization method to generate a brightness-aligned grayscale image (GrayImage); or to perform brightness alignment on the grayscale image (Gray Image) based on the reference grayscale image using an exposure parameter matching method to generate a brightness-aligned grayscale image (Gray Image).

[0256] In one embodiment, the second weight pyramid calculation subunit is further configured to calculate a pixel difference pyramid between the second image pyramid and a target second image pyramid; the target second image pyramid is any second image pyramid in each second image pyramid; the pixel difference pyramid includes n layers of pixel difference images, the pixel difference image of the kth layer being the pixel difference image between the second reference image of the kth layer in the second image pyramid and the second reference image of the kth layer in the target second image pyramid; obtain a Gaussian curve corresponding to the pixel difference pyramid, and use the Gaussian curve corresponding to the pixel difference pyramid as a second Gaussian curve corresponding to the grayscale image; calculate a second weight image corresponding to the pixel difference image of each layer in the pixel difference image pyramid based on the pixel difference image combined with the second Gaussian curve corresponding to the grayscale image; and generate a second weight pyramid corresponding to the second image pyramid based on the second weight images corresponding to each layer of the pixel difference image.

[0257] In one embodiment, the second fused weight map calculation subunit is further used to upsample and accumulate the second weight images of each layer in the second weight pyramid, starting from the second weight image of the nth layer, until the second weight image of the first layer, to generate a second fused weight map corresponding to the second weight pyramid; and use the second fused weight map corresponding to the second weight pyramid as the second fused weight map of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image).

[0258] In one embodiment, the second fused weight map calculation subunit is further used to iteratively perform the following operations starting from k=n: up-sample the second weight images of each layer in the second weight pyramid, starting from the second weight image of the kth layer, to generate a new second weight image; superimpose the new second weight image with the second weight image of the k-1th layer to generate a new second weight image of the k-1th layer; until k=1, a second fused weight map corresponding to the second weight pyramid is generated; 1≤k≤n, and k is a positive integer.

[0259] In one embodiment, the weighted fusion unit 2066 is further used to multiply a first fusion weight map of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image) and a second fusion weight map of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image) to obtain a new fusion weight map; and perform weighted fusion processing on multiple frames of intermediate images (Middle Image) based on the new fusion weight maps of each intermediate image (Middle Image) to obtain a target image.

[0260] In one embodiment, the original image acquisition module 2020 is also used to acquire multiple frames of original RAW images taken under different exposure parameters; the different exposure parameters include at least two groups of parameters among normal exposure parameters, overexposure parameters and underexposure parameters; align the multiple frames of original RAW images to generate multiple frames of aligned RAW images; preprocess the multiple frames of aligned RAW images to generate multiple frames of original images; the preprocessing includes at least one of black level correction, lens shading correction, white balance correction and demosaicing.

[0261] In one embodiment, the fused weight map further includes a third fused weight map, and the fused weight map calculation unit 2064 further includes:

[0262] The third fusion weight map calculation subunit is used to convert each aligned RAW image in the multiple frames of aligned RAW images into a grayscale image (Gray Image); create a third image pyramid corresponding to the grayscale image (Gray Image); calculate a third weight pyramid corresponding to the third image pyramid based on the third image pyramid combined with a third Gaussian curve corresponding to the grayscale image (Gray Image); and reconstruct the third weight pyramid to generate a third fusion weight map of an intermediate image (Middle Image) corresponding to the grayscale image (Gray Image).

[0263] In one embodiment, the weighted fusion unit 2066 is further used to multiply the first fusion weight map of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image) and the third fusion weight map of the intermediate image (Middle Image) corresponding to the grayscale image (Gray Image) to obtain a new fusion weight map of the intermediate image (Middle Image); and perform weighted fusion processing on multiple frames of intermediate images (Middle Image) based on the new fusion weight maps of each intermediate image (Middle Image) to obtain a target image.

[0264] The division of the modules in the above image fusion device is only for illustration. In other embodiments, the image fusion device may be divided into different modules as needed to complete all or part of the functions of the above image fusion device.

[0265] The specific definition of the image fusion device can be found in the definition of the image fusion method above and will not be repeated here. The various modules in the above-mentioned image fusion device can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0266] Figure 23 Schematic diagram of the internal structure of an electronic device in one embodiment. The electronic device can be any terminal device such as a mobile phone, tablet computer, laptop computer, desktop computer, PDA (Personal Digital Assistant), POS (Point of Sales), vehicle-mounted computer, wearable device, etc. The electronic device includes a processor and a memory connected via a system bus. The processor may include one or more processing units. The processor may be a CPU (Central Processing Unit) or a DSP (Digital Signal Processing), etc. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program can be executed by the processor to implement an image fusion method provided in each of the following embodiments. The internal memory provides a cached operating environment for the operating system computer program in the non-volatile storage medium.

[0267] The various modules in the image fusion apparatus provided in the embodiments of the present application may be implemented in the form of a computer program. The computer program may be run on an electronic device. The program modules comprising the computer program may be stored in a memory of the electronic device. When executed by a processor, the computer program implements the steps of the method described in the embodiments of the present application.

[0268] Embodiments of the present application also provide a computer-readable storage medium. One or more non-volatile computer-readable storage media containing computer-executable instructions, when executed by one or more processors, cause the processors to perform the steps of the image fusion method. Embodiments of the present application also provide a computer program product containing instructions, which, when executed on a computer, causes the computer to perform the image fusion method.

[0269] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0270] As used herein, any reference to memory, storage, database, or other medium may include non-volatile and / or volatile memory. Non-volatile memory may include ROM (Read-Only Memory), PROM (Programmable Read-only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-only Memory), or flash memory. Volatile memory may include RAM (Random Access Memory), which serves as an external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), SDRAM (Synchronous Dynamic Random Access Memory), Double Data Rate DDR SDRAM (Double Data Rate Synchronous Dynamic Random Access memory), ESDRAM (Enhanced Synchronous Dynamic Random Access memory), SLDRAM (Sync Link Dynamic Random Access Memory), RDRAM (Rambus Dynamic Random Access Memory), and DRDRAM (Direct Rambus Dynamic Random Access Memory).

[0271] The above embodiments merely illustrate several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. An image fusion method, characterized in that: The method comprises: Acquire multiple frames of original images captured under different exposure parameters; Stretching the multiple frames of original images to obtain multiple frames of intermediate images, wherein the bit width of the intermediate images is greater than the bit width of the original images; For each original image in the plurality of frames of original images, convert the original image into a grayscale image; Calculating a fusion weight map of an intermediate image corresponding to the grayscale image according to an image pyramid corresponding to the grayscale image and a weight pyramid corresponding to the image pyramid; The multiple frames of intermediate images are subjected to weighted fusion processing based on the fusion weight map of each intermediate image to obtain a target image.

2. The method according to claim 1, characterized in that The fusion weight map includes a first fusion weight map, and calculating the fusion weight map of the intermediate image corresponding to the grayscale image according to the image pyramid corresponding to the grayscale image and the weight pyramid corresponding to the image pyramid includes: Creating a first image pyramid corresponding to the grayscale image; Calculating a first weight pyramid corresponding to the first image pyramid according to the first image pyramid and a first Gaussian curve corresponding to the grayscale image; The first weight pyramid is reconstructed to generate a first fusion weight map of an intermediate image corresponding to the grayscale image.

3. The method according to claim 2, characterized in that The creating a first image pyramid corresponding to the grayscale image includes: The grayscale image is scaled in sequence according to a preset scaling ratio to generate a first image pyramid corresponding to the grayscale image; the first image pyramid includes n layers of first reference images, wherein the first reference image of the first layer is the original grayscale image.

4. The method according to claim 2, characterized in that The calculating a first weight pyramid corresponding to the first image pyramid according to the first image pyramid and a first Gaussian curve corresponding to the grayscale image includes: Obtaining a first Gaussian curve corresponding to the grayscale image; Calculating a first weighted image corresponding to each first reference image according to the first reference images of each layer in the first image pyramid and the first Gaussian curve; A first weight pyramid corresponding to the first image pyramid is generated based on the first weight images corresponding to the first reference images in the first image pyramid.

5. The method according to claim 2, characterized in that The reconstructing the first weight pyramid to generate a first fusion weight map of the intermediate image corresponding to the grayscale image includes: Upsampling and accumulating the first weight images of each layer in the first weight pyramid, starting from the first weight image of the nth layer upwards until the first weight image of the first layer, to generate a first fused weight map corresponding to the first weight pyramid; A first fusion weight map corresponding to the first weight pyramid is used as a first fusion weight map of an intermediate image corresponding to the grayscale image.

6. The method according to claim 5, characterized in that The method of upsampling and accumulating the first weight images of each layer in the first weight pyramid, starting from the first weight image of the nth layer and continuing upward until the first weight image of the first layer is reached, to generate a first fused weight map corresponding to the first weight pyramid, includes: Starting from k=n, the following operations are performed iteratively: the first weight images of each layer in the first weight pyramid are up-sampled in sequence, starting from the first weight image of the kth layer, to generate a new first weight image; the new first weight image is superimposed with the first weight image of the k-1th layer to generate a new first weight image of the k-1th layer; until k=1, a first fused weight map corresponding to the first weight pyramid is generated; 1≤k≤n, and k is a positive integer.

7. The method according to claim 2, characterized in that The fusion weight map further includes a second fusion weight map, and calculating the fusion weight map of the intermediate image corresponding to the grayscale image based on the image pyramid corresponding to the grayscale image and the weight pyramid corresponding to the image pyramid includes: Creating a second image pyramid corresponding to the grayscale image; Calculating a second weight pyramid corresponding to the second image pyramid according to the second image pyramid and a second Gaussian curve corresponding to the grayscale image; The second weight pyramid is reconstructed to generate a second fusion weight map of the intermediate image corresponding to the grayscale image.

8. The method according to claim 7, characterized in that The creating a second image pyramid corresponding to the grayscale image includes: Performing brightness alignment on the grayscale image based on a reference grayscale image to generate a brightness-aligned grayscale image; the reference grayscale image is a grayscale image corresponding to the first original image, and the first original image is an original image corresponding to the overexposure parameter; The brightness-aligned grayscale image is sequentially scaled according to a preset scaling ratio to generate a second image pyramid corresponding to the brightness-aligned grayscale image; the second image pyramid includes n layers of second reference images, wherein the second reference image of the first layer is the original image of the brightness-aligned grayscale image.

9. The method according to claim 8, characterized in that The step of performing brightness alignment on the grayscale images based on the reference grayscale image to generate brightness aligned grayscale images includes: Using a brightness histogram equalization method, aligning the brightness of the grayscale image based on a reference grayscale image to generate a brightness-aligned grayscale image; or An exposure parameter matching method is adopted to perform brightness alignment on the grayscale image based on a reference grayscale image to generate a brightness-aligned grayscale image.

10. The method according to claim 7, characterized in that The calculating a second weight pyramid corresponding to the second image pyramid according to the second image pyramid and a second Gaussian curve corresponding to the grayscale image includes: calculating a pixel difference pyramid between the second image pyramid and a target second image pyramid; the target second image pyramid being any one of the second image pyramids; the pixel difference pyramid comprising n layers of pixel difference images, the pixel difference image at the kth layer being the pixel difference image between the second reference image at the kth layer in the second image pyramid and the second reference image at the kth layer in the target second image pyramid; Obtaining a Gaussian curve corresponding to the pixel difference pyramid, and using the Gaussian curve corresponding to the pixel difference pyramid as a second Gaussian curve corresponding to the grayscale image; For each pixel difference image in each layer of the pixel difference image pyramid, calculating a second weight image corresponding to the pixel difference image according to the pixel difference image and a second Gaussian curve corresponding to the grayscale image; A second weight pyramid corresponding to the second image pyramid is generated based on the second weight image corresponding to the pixel difference images of each layer.

11. The method according to claim 7, characterized in that The reconstructing the second weight pyramid to generate a second fusion weight map of the intermediate image corresponding to the grayscale image includes: Upsampling and accumulating the second weight images of each layer in the second weight pyramid, starting from the second weight image of the nth layer upwards until the second weight image of the first layer, to generate a second fused weight map corresponding to the second weight pyramid; The second fused weight map corresponding to the second weight pyramid is used as the second fused weight map of the intermediate image corresponding to the grayscale image.

12. The method according to claim 11, characterized in that The method of upsampling and accumulating the second weight images of each layer in the second weight pyramid, starting from the second weight image of the nth layer and ending at the second weight image of the first layer, to generate a second fused weight map corresponding to the second weight pyramid includes: Starting from k=n, the following operations are performed iteratively: the second weight images of each layer in the second weight pyramid are up-sampled in sequence, starting from the second weight image of the kth layer, to generate a new second weight image; the new second weight image is superimposed with the second weight image of the k-1th layer to generate a new second weight image of the k-1th layer; until k=1, a second fused weight map corresponding to the second weight pyramid is generated; 1≤k≤n, and k is a positive integer.

13. The method according to claim 7, characterized in that The step of performing weighted fusion processing on the multiple frames of intermediate images based on the fusion weight map of each intermediate image to obtain a target image includes: Obtaining a new fusion weight map based on multiplying a first fusion weight map of the intermediate image corresponding to the grayscale image and a second fusion weight map of the intermediate image corresponding to the grayscale image; The multiple frames of intermediate images are subjected to weighted fusion processing based on the new fusion weight map of each intermediate image to obtain a target image.

14. The method according to claim 2, characterized in that The acquiring of multiple frames of original images captured under different exposure parameters includes: Acquire multiple frames of original RAW images captured under different exposure parameters; the different exposure parameters include at least two sets of parameters selected from the group consisting of normal exposure parameters, overexposure parameters, and underexposure parameters; Aligning the multiple frames of original RAW images to generate multiple aligned RAW images; The aligned RAW images are preprocessed to generate multiple frames of original images; the preprocessing includes at least one of black level correction, lens shading correction, white balance correction and demosaicing.

15. The method according to claim 14, characterized in that The fused weight map further includes a third fused weight map, and the method further includes: For each aligned RAW image in the multiple frames of aligned RAW images, convert the aligned RAW image into a grayscale image; Creating a third image pyramid corresponding to the grayscale image; Calculating a third weight pyramid corresponding to the third image pyramid according to the third image pyramid and a third Gaussian curve corresponding to the grayscale image; The third weight pyramid is reconstructed to generate a third fusion weight map of the intermediate image corresponding to the grayscale image.

16. The method according to claim 15, characterized in that The step of performing weighted fusion processing on the multiple frames of intermediate images based on the fusion weight map of each intermediate image to obtain a target image includes: obtaining a new fusion weight map of the intermediate image by multiplying a first fusion weight map of the intermediate image corresponding to the grayscale image and a third fusion weight map of the intermediate image corresponding to the grayscale image; The multiple frames of intermediate images are subjected to weighted fusion processing based on the new fusion weight map of each intermediate image to obtain a target image.

17. An image fusion device, characterized in that: The device comprises: The original image acquisition module is used to acquire multiple frames of original images shot under different exposure parameters; an intermediate image acquisition module, configured to perform stretching processing on the multiple frames of original images to obtain multiple frames of intermediate images; the bit width of the intermediate images is greater than the bit width of the original images; The image fusion module is configured to convert each original image in the multiple frames of original images into a grayscale image, calculate a fusion weight map of an intermediate image corresponding to the grayscale image based on an image pyramid corresponding to the grayscale image and a weight pyramid corresponding to the image pyramid, and perform weighted fusion processing on the multiple frames of intermediate images based on the fusion weight map of each intermediate image to obtain a target image.

18. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image fusion method according to any one of claims 1 to 16.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the image fusion method according to any one of claims 1 to 16 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the image fusion method according to any one of claims 1 to 16 are implemented.

Images