Image fusion method and device, storage medium and electronic device

Abstract

The application relates to the technical field of image processing, and provides an image fusion method and device, a storage medium and an electronic device. The image fusion method comprises the following steps: acquiring a plurality of first-type images, the exposure compensation values of the first-type images are all different, and the exposure compensation values of at least two of the first-type images are non-positive numbers; calculating a first-type weight image between each two adjacent first-type images; calculating a highlight image corresponding to each first-type image with a non-positive exposure compensation value; calculating a real highlight image containing a highlight region and / or a white region according to each two adjacent highlight images; correcting a weight value in the corresponding first-type weight image by using the real highlight image; and performing weighted fusion on the plurality of first-type images by using at least the corrected first-type weight image, so as to obtain a final fusion image. The method can effectively improve the problem of grayness in the high-luminance region or the white region of the fusion image, and improve the quality of the fusion image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an image fusion method and apparatus, a storage medium, and an electronic device. Background Technology

[0002] Currently, image fusion algorithms are commonly used in mobile phones to improve the quality of certain photos, such as night scene photos. These algorithms use multiple frames of images captured at different exposure compensation values ​​to generate the final fused image. However, fused images obtained based on existing algorithms generally suffer from problems such as white areas displaying normally while highlight areas appear grayish, or vice versa, which seriously affects the user's photography experience. Summary of the Invention

[0003] The purpose of this application is to provide an image fusion method, apparatus, storage medium, and electronic device to improve the above-mentioned technical problems.

[0004] To achieve the above objectives, this application provides the following technical solution:

[0005] In a first aspect, embodiments of this application provide an image fusion method, comprising: acquiring multiple frames of first-type images, wherein the exposure compensation values ​​of each frame of the first-type images are different and at least two frames of the first-type images have non-positive exposure compensation values; calculating a first-type weight image between every two adjacent frames of the first-type images, wherein the pixel values ​​in the first-type weight image represent the weight values ​​of the images during weighted fusion; calculating a highlight image corresponding to each frame of the first-type images with non-positive exposure compensation values, wherein the highlight image contains a high-brightness region in the corresponding first-type image; calculating a real highlight image containing a highlight region and / or a white region based on every two adjacent highlight images; correcting the pixel values ​​in the corresponding first-type weight image using the highlight region and / or white region in the real highlight image to obtain a corrected first-type weight image; wherein different correction strategies are adopted for the pixel values ​​located in the highlight region and the white region in the first-type weight image; and weighted fusion of the multiple frames of the first-type images using at least the corrected first-type weight image to obtain a final fused image.

[0006] The inventors discovered that highlight areas and white areas in an image often have similar pixel values. According to the current fusion weight calculation method, the weight values ​​calculated for these two types of areas are also similar. Thus, when performing image fusion, these two types of areas will be subject to similar fusion strategies. However, in an ideal situation, since highlight areas and white areas have different characteristics, the fusion strategies used for these two types of areas should be different. As a result, the current approach easily leads to the problem that white areas are displayed normally in the fused image, while highlight areas appear grayish, or vice versa. In other words, it cannot balance the display effects of both.

[0007] In the above method, a real highlight image containing highlight and / or white regions is used to correct the pixel values ​​located in the highlight and white regions of the first-type weighted image, and different correction strategies are applied to the pixel values ​​in the highlight and white regions. This results in a significant difference between the pixel values ​​in the highlight and white regions in the corrected first-type weighted image. Furthermore, since the pixel values ​​in the first-type weighted image represent the weight values ​​during weighted fusion, using the corrected first-type weighted image to perform weighted fusion on multiple frames of the first-type image allows for a significant difference in the fusion strategies in the highlight and white regions (the fusion strategy depends on the fusion weights). Therefore, this helps to improve the problem of grayish highlights or white regions in the fused image, thus improving the quality of the fused image.

[0008] In one implementation of the first aspect, the highlight region is a high-brightness region commonly contained in two adjacent highlight images, and the white region is a high-brightness region contained in the bright frame highlight image but not in the dark frame highlight image; wherein, the bright frame highlight image refers to the frame with a larger exposure compensation value of the corresponding first type image among two adjacent highlight images, and the dark frame highlight image refers to the frame with a smaller exposure compensation value of the corresponding first type image among two adjacent highlight images.

[0009] The inventors discovered that for two adjacent frames of Type I images with non-positive exposure compensation values, the highlight areas are relatively bright in both frames, while the white areas only have high brightness in frames with larger exposure compensation values, and their brightness is significantly reduced in frames with smaller exposure compensation values. Based on this principle, by calculating the highlight image (containing the bright areas in Type I images) corresponding to each frame of Type I images with non-positive exposure compensation values, and by comparing the bright areas in the highlight images of two adjacent frames in a certain sense, the true highlight areas (areas with relatively high brightness in both adjacent highlight images) and / or white areas (areas with relatively high brightness only in bright frame highlight images and relatively low brightness in dark frame highlight images) in Type I images can be determined. Furthermore, the pixel values ​​in the Type I weighted images can be specifically corrected based on the highlight areas and / or white areas.

[0010] In one implementation of the first aspect, the highlight image is a binary image, where pixels with a first value belong to a high-brightness region, and pixels with a second value do not belong to a high-brightness region; calculating a true highlight image containing a highlight region based on every two adjacent highlight images includes: determining a connected region in the bright frame highlight image composed of pixels with the first value; determining the centroid position of the connected region in the dark frame highlight image composed of pixels with the first value; for each centroid position, if it is located within a connected region in the bright frame highlight image, then the region in the true highlight image corresponding to that connected region in the bright frame highlight image is determined as the highlight region.

[0011] Since the centroid of a connected region can represent the center of the region to some extent, if the centroid of a connected region in a dark frame highlight image falls within a connected region in a bright frame highlight image, it means that these two connected regions in the dark and bright frame highlight images constitute a common high-brightness region, i.e., a highlight region. In this way, the highlight region in the real highlight image can be quickly and accurately determined.

[0012] In one implementation of the first aspect, the highlight image is a binary image, where pixels with a first value belong to a high-brightness region, and pixels with a second value do not belong to a high-brightness region. Determining a true highlight image containing a highlight region based on every two adjacent highlight images includes: determining a connected region in the bright frame highlight image composed of pixels with the first value; determining a connected region in the dark frame highlight image composed of pixels with the first value; for each connected region in the dark frame highlight image, if there is a connected region in the bright frame highlight image whose overlap with the dark frame highlight image exceeds a first threshold, then the region in the true highlight image corresponding to that connected region in the bright frame highlight image is determined as a highlight region.

[0013] If a connected region in the bright frame highlight image and a connected region in the dark frame highlight image have a high degree of overlap (greater than a first threshold), it indicates that these two connected regions in the dark frame highlight image and the bright frame highlight image are a common high-brightness region, i.e., a highlight region. In this way, the highlight region in the real highlight image can be quickly and accurately determined.

[0014] In one implementation of the first aspect, the highlight image is a binary image, where pixels with a first value belong to high-brightness regions, and pixels with a second value do not belong to high-brightness regions. Calculating a true highlight image containing white regions based on every two adjacent highlight images includes: determining connected regions in the bright highlight image formed by pixels with the first value; determining the centroid position of the connected regions in the dark highlight image formed by pixels with the first value; and for each connected region in the bright highlight image, if any centroid position in the dark highlight image is located outside it, then the region in the true highlight image corresponding to that connected region in the bright highlight image is determined as a white region.

[0015] Since the centroid of a connected region can represent the center of the region to some extent, if a connected region in a bright frame highlight image does not contain the centroid of any connected region in a dark frame highlight image, it indicates that the connected region is merely a bright area contained within the bright frame highlight image, and not a bright area contained within the dark frame highlight image, i.e., a white area. In this way, white areas in the actual highlight image can be quickly and accurately determined.

[0016] In one implementation of the first aspect, the highlight image is a binary image, where pixels with a first value belong to high-brightness regions, and pixels with a second value do not belong to high-brightness regions. The calculation of a true highlight image containing white regions based on every two adjacent highlight images includes: determining connected regions in the bright highlight image formed by pixels with the first value; determining connected regions in the dark highlight image formed by pixels with the first value; for each connected region in the bright highlight image, if the overlap between the connected region and the dark highlight image does not exceed a second threshold, then the region in the true highlight image corresponding to that connected region in the bright highlight image is determined as a white region.

[0017] If the overlap between a connected region in the bright frame highlight image and any connected region in the dark frame highlight image is not high (not exceeding the second threshold), it indicates that the connected region is merely a high-brightness area contained in the bright frame highlight image, and not a high-brightness area contained in the dark frame highlight image, i.e., a white area. In this way, the white area in the real highlight image can be quickly and accurately determined.

[0018] In one implementation of the first aspect, each frame of the first type of weighted image is used to fuse two frames of images to be fused, wherein the images to be fused are the first type of images, or, at least two frames of the first type of images are weighted and fused to obtain a fused image, wherein the pixel values ​​in the first type of weighted image represent the weight values ​​of the dark frame images to be fused during weighted fusion, and the dark frame images to be fused are the frames with smaller exposure compensation values ​​among the two frames to be fused; the step of using the highlight areas and / or white areas in the real highlight image to correct the pixel values ​​in the corresponding first type of weighted image includes: increasing the pixel values ​​in the highlight areas of the first type of weighted image, and / or decreasing the pixel values ​​in the white areas of the first type of weighted image.

[0019] If the pixel values ​​in the first type of weight image represent the weight values ​​of the dark frame image to be fused during weighted fusion, then increasing the pixel values ​​in the highlight region of the first type of weight image can increase the proportion of the dark frame image to be fused in the highlight region of the fusion result. Since the dark frame image to be fused contains more detailed information in the highlight region, this move is beneficial to improving the image quality of the fused image in the highlight region.

[0020] Reducing the pixel values ​​located in the white area of ​​the first type of weight image can reduce the proportion of the dark frame image to be fused in the white area of ​​the fusion result (relatively, it increases the proportion of the bright frame image to be fused in the white area of ​​the fused image). Since the bright frame image to be fused contains more detailed information in the white area, this move helps to improve the image quality of the fused image in the white area.

[0021] In one implementation of the first aspect, increasing the pixel value located in the highlight region of the first type of weighted image, or decreasing the pixel value located in the white region of the first type of weighted image, includes: increasing the pixel value located in the highlight region of the first type of weighted image and decreasing the pixel value located outside the highlight region of the first type of weighted image, or decreasing the pixel value located in the white region of the first type of weighted image and increasing the pixel value located outside the white region of the first type of weighted image.

[0022] If the pixel values ​​in the first type of weight image represent the weight values ​​of the dark frame image to be fused during weighted fusion, then reducing the pixel values ​​located outside the highlight areas in the first type of weight image will necessarily also reduce the pixel values ​​in the white areas (the white areas are outside the highlight areas). This reduces the proportion of the dark frame image to be fused in the white areas of the fusion result, which is beneficial to improving the image quality of the fused image in the white areas. Furthermore, since the pixel values ​​located in the highlight areas of the first type of weight image are also increased, it ensures that the correction strategies applied to the pixel values ​​located in the highlight areas and the white areas of the first type of weight image are different (increasing the former and decreasing the latter).

[0023] Increasing the pixel values ​​outside the white areas in the first type of weight image inevitably increases the pixel values ​​in the highlight areas (which are outside the white areas). This increases the proportion of the dark frame image to be fused in the highlight areas of the fused image, thus improving the image quality of the fused image in the highlight areas. Furthermore, since the pixel values ​​within the white areas of the first type of weight image are also decreased, it ensures that the correction strategies for the pixel values ​​in the highlight and white areas of the first type of weight image are different (increasing the former and decreasing the latter).

[0024] In one implementation of the first aspect, increasing the pixel value located in the highlight region of the first type of weighted image and decreasing the pixel value located in the white region of the first type of weighted image includes: increasing the pixel value located in the highlight region of the first type of weighted image, decreasing the pixel value located in the white region of the first type of weighted image, and maintaining the pixel values ​​located outside the highlight region and the white region of the first type of weighted image.

[0025] If the pixel values ​​in the first type of weight image represent the weight values ​​of the dark frame image to be fused during weighted fusion, then increasing the pixel values ​​in the highlight region of the first type of weight image can increase the proportion of the dark frame image to be fused in the highlight region of the fusion result. Conversely, decreasing the pixel values ​​in the white region of the first type of weight image can reduce the proportion of the dark frame image to be fused in the white region of the fusion result. Furthermore, in the above implementation, the pixel values ​​in the first type of weight image outside the highlight region and the white region are kept unchanged. This is beneficial for improving the image quality of the fused image in the highlight region and the white region without reducing the image quality of the fused image in other regions. As a result, the post-processing steps for the fused image can be omitted or simplified.

[0026] In one implementation of the first aspect, after calculating a true highlight image containing highlight regions and / or white regions based on every two adjacent highlight images, and before correcting the pixel values ​​in the corresponding first-class weighted image using the highlight regions and / or white regions in the true highlight image, the method further includes: performing a smoothing filter on the true highlight image.

[0027] In the above implementation, smoothing filtering of the real highlight image helps to weaken or eliminate noise in the image, thereby improving the quality of the subsequently obtained fused image.

[0028] In one implementation of the first aspect, the exposure compensation values ​​of the multiple frames of first-type images are all non-positive numbers. The step of using at least the corrected first-type weight image to perform weighted fusion on the multiple frames of first-type images to obtain the final fused image includes: taking the first-type image with the largest or smallest corresponding exposure compensation value as the current frame of first-type images and the initial fused image, and sequentially using the corrected first-type weight image of the current frame to perform weighted fusion on the current fused image and the next frame of first-type images in the order of the gradual change of the corresponding exposure compensation values ​​to obtain a new fused image, until the next frame of first-type images is the last frame of first-type images, and obtaining the final fused image; wherein, the corrected first-type weight image of the current frame refers to the corrected first-type weight image between the current frame of first-type images and the next frame of first-type images.

[0029] In the fusion process described above, the exposure compensation value of the images to be fused changes gradually rather than abruptly, thus the final fusion result has high image quality.

[0030] In one implementation of the first aspect, at least one of the multiple frames of first-class images has a positive exposure compensation value. The step of weighted fusion of the multiple frames of first-class images using at least the modified first-class weighted image to obtain a final fused image includes: weighted fusion of the multiple frames of first-class images using the following two types of images to obtain a final fused image: the first-class weighted image between every two adjacent frames of first-class images where at least one of the exposure compensation values ​​is positive; and the modified first-class weighted image between every two adjacent frames of first-class images where both of the exposure compensation values ​​are non-positive.

[0031] The above implementation method uses not only images with non-positive exposure compensation values ​​but also images with positive exposure compensation values ​​in the process of obtaining the fused image. Because it uses images with more exposure compensation values, it helps improve the quality of the fused image. For example, in some darker scenes, images with positive exposure compensation values ​​may contain more valuable detail information compared to images with non-positive exposure compensation values.

[0032] In one implementation of the first aspect, the exposure compensation value of one frame in the multi-frame first-type image is 0.

[0033] In an image with an exposure compensation value of 0, most areas contain rich details. Therefore, for the purpose of image fusion, when acquiring multiple first-class images, one image with an exposure compensation value of 0 can be acquired.

[0034] Secondly, embodiments of this application provide an image fusion method, comprising: acquiring multiple frames of first-type images, wherein the exposure compensation values ​​of each frame of first-type images are different and at least two of the frames of first-type images have non-positive exposure compensation values; taking the frame of first-type images with the largest or smallest corresponding exposure compensation value as the current frame of first-type images and the initial fusion image, and sequentially performing the following fusion steps according to the gradual change of the corresponding exposure compensation values ​​to obtain a new fusion image, until the next frame of first-type images is the last frame of first-type images, to obtain the final fusion image, wherein the fusion steps include: calculating a second-type weight image between the current fusion image and the next frame of first-type images, wherein the pixel values ​​in the second-type weight image represent the weight values ​​of the images during weighted fusion; if at least one of the exposure compensation values ​​of the current frame of first-type images and the next frame of first-type images is positive, then utilizing the second-type weight image... The weighted image performs a weighted fusion of the current fused image and the next frame's first-type image to obtain a new fused image. If the exposure compensation values ​​of both the current frame's first-type image and the next frame's first-type image are non-positive, the following operations are performed: Calculate the highlight images corresponding to the current fused image and the next frame's first-type image respectively. The two obtained highlight images respectively contain the high-brightness regions in the corresponding current fused image and the next frame's first-type image. Calculate a true highlight image containing highlight regions and / or white regions based on the two highlight images. Correct the pixel values ​​in the corresponding second-type weighted image using the highlight regions and / or white regions in the true highlight image to obtain a corrected second-type weighted image. Perform a weighted fusion of the current fused image and the next frame's first-type image using the corrected second-type weighted image to obtain a new fused image.

[0035] The method provided in the second aspect is similar to the method provided in the first aspect in principle and beneficial effects, except that they differ in the way the images are fused. For the method provided in the second aspect, since the source of the calculation for the second type of weight image and the object used for fusion are the same (both the current fused image and the first type image of the next frame), a higher quality fused image may be obtained in the end.

[0036] Thirdly, embodiments of this application provide an image fusion apparatus, comprising: a first image acquisition module, configured to acquire multiple frames of first-type images, wherein the exposure compensation values ​​of each frame of the first-type images are different and at least two frames of the first-type images have non-positive exposure compensation values; a weighted image calculation module, configured to calculate a first-type weighted image between every two adjacent frames of the first-type images, wherein the pixel values ​​in the first-type weighted image represent the weight values ​​of the images during weighted fusion; and a highlight image calculation module, configured to calculate a highlight image corresponding to each frame of the first-type images with non-positive exposure compensation values, wherein the highlight image contains a high-brightness region in the corresponding first-type image; and a real-world image. A highlight image calculation module is used to calculate a real highlight image containing highlight regions and / or white regions based on every two adjacent highlight images; a weighted image correction module is used to correct the pixel values ​​in the corresponding first-type weighted image using the highlight regions and / or white regions in the real highlight image to obtain a corrected first-type weighted image; wherein, different correction strategies are adopted for the pixel values ​​located in the corresponding highlight regions and corresponding white regions in the first-type weighted image; a first image fusion module is used to perform weighted fusion on the multiple frames of first-type images using at least the corrected first-type weighted image to obtain a final fused image.

[0037] Fourthly, embodiments of this application provide an image fusion apparatus, comprising: a second image acquisition module, configured to acquire multiple frames of first-type images, wherein the exposure compensation values ​​of each frame of the first-type images are different and at least two of the frames of the first-type images have non-positive exposure compensation values; a second image fusion module, configured to take the frame of the first-type image with the largest or smallest corresponding exposure compensation value as the current frame of the first-type image and the initial fusion image, and sequentially execute the following fusion steps according to the gradual change of the corresponding exposure compensation values ​​to obtain a new fusion image, until the next frame of the first-type image is the last frame of the first-type image, to obtain the final fusion image, wherein the fusion steps include: calculating a second-type weight image between the current fusion image and the next frame of the first-type image, wherein the pixel values ​​in the second-type weight image represent the weight values ​​of the images during weighted fusion; if the exposure compensation values ​​of the current frame of the first-type image and the next frame of the first-type image are different, the second image fusion module is configured to acquire multiple frames of first-type images, wherein the exposure compensation values ​​of each frame of the first-type image are different and at least two of the frames of the first-type images have non-positive exposure compensation values; and a second image fusion module, configured to take the frame of the first-type image with the largest or smallest corresponding exposure compensation value as the current frame of the first-type image and the next frame of the first-type image, and sequentially execute the following fusion steps to obtain a new fusion image, until the next frame of the first-type image is the last frame of the first-type image, to obtain the final fusion image, wherein the fusion steps include: calculating a second-type weight image between the current fusion image and the next frame of the first-type image, wherein the pixel values ​​in the second-type weight image represent the weight values ​​of the images during weighted fusion; and calculating a second-type weight image between the current frame of the first-type image and the next frame of the first-type image, wherein the pixel values ​​in the second-type weight image represent the weight values ​​of the images during weighted fusion If at least one of the values ​​is positive, the current fused image and the next frame's first-class image are weighted and fused using the second-class weighted image to obtain a new fused image. If the exposure compensation values ​​of the current frame's first-class image and the next frame's first-class image are both non-positive, the following operations are performed: calculate the highlight images corresponding to the current fused image and the next frame's first-class image respectively, and the two obtained highlight images respectively contain the high-brightness regions in the corresponding current fused image and the next frame's first-class image; calculate a true highlight image containing highlight regions and / or white regions based on the two highlight images; correct the pixel values ​​in the corresponding second-class weighted image using the highlight regions and / or white regions in the true highlight image to obtain a corrected second-class weighted image; and perform weighted fusion of the current fused image and the next frame's first-class image using the corrected second-class weighted image to obtain a new fused image.

[0038] Fifthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided by the first aspect, the second aspect, or any possible implementation of both aspects.

[0039] Sixthly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores computer program instructions, and the computer program instructions are read and executed by the processor to perform the method provided by the first aspect, the second aspect, or any possible implementation of both aspects. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 The image fusion method provided in this application is shown before and after applying it to the same scene, resulting in fused images.

[0042] Figure 2 The flowchart of the first image fusion method provided in the embodiments of this application is shown;

[0043] Figure 3 The flowchart of the second image fusion method provided in the embodiments of this application is shown;

[0044] Figure 4 The structure of the first image fusion apparatus provided in this application embodiment is shown;

[0045] Figure 5 The structure of the second image fusion apparatus provided in this application embodiment is shown;

[0046] Figure 6 The structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0047] Existing image fusion algorithms use multiple frames of images acquired at different exposure compensation values ​​to calculate fusion weights and then use these weights to calculate the final fused image. However, the resulting fused image often exhibits issues where white areas appear normal while highlight areas appear grayish, or vice versa, making it difficult to balance the display effects of both types of areas.

[0048] After a long period of research, the inventors discovered that the cause of this problem was:

[0049] Existing image fusion algorithms often employ different fusion strategies to process different regions in an image. For example, highlight regions are fused more with pixel information from images with negative exposure compensation values, while non-highlight regions are fused more with pixel information from images with zero exposure compensation values. This allows both highlight and non-highlight regions to retain relatively rich details in the fused image.

[0050] However, since highlight and white areas in an image often have similar pixel values, the calculated weights for these two types of regions are also quite similar using current fusion weight calculation methods. This means that during image fusion, similar fusion strategies (which depend on the fusion weights) will be applied to both types of regions, resulting in poor quality of the fused image. For example, to make white areas appear normal, more pixel information from images with zero exposure compensation needs to be fused, which will cause highlight areas in the fused image to appear grayish; similarly, to make highlight areas appear normal, more pixel information from images with negative exposure compensation needs to be fused, which will also cause white areas in the fused image to appear grayish.

[0051] To address the aforementioned issues, the image fusion method provided in this application improves the graying of either the highlight or white areas in the fused image by differentiating the fusion weights of the highlight and white areas in the image. The detailed steps of this method will be described later.

[0052] Figure 1 The images shown are fused images obtained before and after applying the image fusion method provided in this application to the same scene. The left image is the fused image obtained using an existing image fusion method, and the right image is the fused image obtained using the image fusion method provided in this application. It is easy to see that in the fused image on the left, the person's white shirt (bottom black frame) is displayed normally, while the illuminated billboard (top black frame) and the lighting scene inside the building (middle black frame) appear grayish. In the fused image on the right, the person's white shirt is displayed normally, as are the illuminated billboard and the lighting scene inside the building; the grayish appearance is significantly improved.

[0053] It should be noted that, apart from the method itself, the above analysis of the reasons for the graying of highlight areas or white areas in the image is also a conclusion obtained by the inventor during the research process, rather than a result that already exists in the prior art. Therefore, it should also be regarded as a contribution made by the inventor to this invention.

[0054] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. It should be noted that similar reference numerals and letters in the following drawings indicate similar items; therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0055] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0056] The terms “first,” “second,” etc., are used only to distinguish one entity or operation from another, and should not be construed as indicating or implying relative importance, nor as requiring or implying any such actual relationship or order between these entities or operations.

[0057] Figure 2 The flowchart of a first image fusion method provided in an embodiment of this application is shown. This method can be, but is not limited to, performed by an electronic device. Figure 6 One possible structure of the electronic device is shown below, for details in the following section. Figure 6 The explanation. (Refer to...) Figure 2 The method includes:

[0058] Step S110: Obtain multiple frames of the first type of image.

[0059] Before introducing step S110 in detail, let's first introduce the concepts of exposure compensation and exposure compensation value.

[0060] Exposure compensation is a method of exposure control: by intentionally changing the exposure parameters (such as aperture and shutter speed) automatically calculated by the camera, the amount of exposure of the camera changes, thereby making the captured photo brighter or darker.

[0061] Exposure compensation value can be considered a quantitative representation of the degree of exposure compensation. The range of exposure compensation value can be an interval centered at 0, such as [-2,2], [-3,3], etc. An exposure compensation value of 0 indicates that the exposure is automatically determined by the camera for shooting. A positive exposure compensation value (relative to an exposure compensation value of 0) indicates that the exposure is increased for shooting, and a negative exposure compensation value (relative to an exposure compensation value of 0) indicates that the exposure is decreased for shooting. The larger the absolute value of the exposure compensation value, the greater the increase or decrease in exposure. In some existing devices, the exposure compensation value is taken at intervals. For example, in [-2,2], with intervals of 1, we can get 5 exposure compensation values: ev-2, ev-1, ev0, ev+1, and ev+2. The difference between adjacent exposure compensation values ​​is 1 ev, which means that the exposure is doubled. For example, ev+1 is twice the exposure of ev0. In some implementations, 1 / 2 or 1 / 3 can also be used as the interval between adjacent exposure compensation values.

[0062] In step S110, the exposure compensation values ​​of the multiple frames of the first type of images are all different, and at least two of the first type of images have non-positive exposure compensation values. Of course, it is also possible that some of the first type of images have positive exposure compensation values. The first type of images can be the original images captured by the camera, or images obtained after preprocessing the original images captured by the camera. Understandably, since the multiple frames of the first type of images are used for image fusion, these images should be captured in the same scene and can be captured continuously within a short period of time.

[0063] For example, two frames of Type I images under ev-1 and ev-2 can be acquired; or three frames of Type I images under ev0, ev-1, and ev-2 can be acquired; or three frames of Type I images under ev+1 / 2, ev0, and ev-1 / 2 can be acquired. The inventors discovered that in images with an exposure compensation value of 0, most areas contain rich detail information. Therefore, for image fusion purposes, when acquiring multiple frames of Type I images, one frame with an exposure compensation value of 0 can be acquired. In the subsequent description of the image fusion method steps, the three frames of Type I images under ev0, ev-1, and ev-2 acquired in step S110 will be used as an example, which can be denoted as I1, I2, and I3 respectively.

[0064] As mentioned above, the first type of image in step S110 can be a pre-processed image. Possible pre-processing operations include one or more of the following: image fusion, image registration, etc. Image fusion in pre-processing refers to the fusion of multiple frames under the same exposure compensation value, mainly for noise reduction. However, the image fusion method proposed in this application is a fusion method across exposure compensation values, and the two should not be confused. Image registration in pre-processing refers to aligning different images through a certain transformation (after alignment, identical objects in different images can overlap). Registered images perform better when performing fusion and other operations.

[0065] For example, if the camera captures 5 frames, 1 frame, and 1 frame of raw images at ev0, ev-1, and ev-2 respectively, the preprocessing can be performed as follows:

[0066] (1) Image registration under the same exposure compensation value, that is, registering 5 original images under ev0;

[0067] (2) Image fusion under the same exposure compensation value, that is, the 5 frames of images under ev0 obtained in (1) are fused, and after fusion, there is one frame of image under ev0, ev-1 and ev-2 respectively.

[0068] (3) Image registration across exposure compensation values, that is, registering the three frames of images obtained in (2). After registration, there is one frame of image under ev0, ev-1 and ev-2, which are the three first-class images to be obtained in step S110.

[0069] Step S120: Calculate the first class weight image between every two adjacent first class images.

[0070] In this context, two frames of the first type of image are adjacent if their corresponding exposure compensation values ​​are adjacent. For example, I1 and I2 are adjacent, I2 and I3 are adjacent, but I1 and I3 are not adjacent.

[0071] The pixel values ​​in the first type of weighted image represent the weight values ​​used in the weighted fusion of the images. Let's call the image to be weighted and fused (see step S160) the image to be fused. Each frame of the first type of weighted image can be used to fuse two frames of images to be fused. Note that since the first type of weighted image is calculated from two adjacent frames of first type images, its pixel values, in their original sense, represent the weight values ​​used in the weighted fusion of these two frames of first type images. However, in subsequent steps, the first type of weighted image is not necessarily used to fuse the two adjacent frames of first type images from which it was calculated. That is, the two frames to be fused mentioned above are not necessarily two adjacent frames of first type images. As can be seen from step S160 below, the image to be fused can be a first type image, or it can be a fused image obtained by weighted fusion of at least two frames of first type images. This will not be elaborated on here.

[0072] The size of the first type of weighted image is the same as that of the image to be fused, and also the same as that of the first type of image. The pixel value at any pixel position x in the image is denoted as ω(x). ω(x) represents the weight value used when weighting the pixel values ​​at pixel position x in the two images to be fused. Since the weighted fusion of the two images involves two weight values, ω(x) can represent only one of the weight values, and the other weight value can be calculated based on ω(x). For example, in some implementations, the sum of the weight values ​​at the same pixel position in the two images to be fused is a fixed value (hereinafter referred to as the complementarity of the fusion weights), so knowing one weight value ω(x) allows the calculation of the other weight value.

[0073] ω(x) has a certain range of values, such as [0,255], [0,1], etc. Taking [0,255] as an example, when the pixel values ​​of two frames to be fused are weighted and fused, if one of the weight values ​​is ω(x), then the weight value of the other can be 255-ω(x), that is, the sum of the weight values ​​of the two frames to be fused at the same pixel position x is 255.

[0074] The calculation method for ω(x) can refer to existing techniques, and a formula for calculating ω(x) is also given below. Clearly, since x can represent any pixel position in the first-class weighted image, the first-class weighted image can be calculated using this formula:

[0075]

[0076] Where C(x) represents the contrast at pixel position x, S(x) represents the saturation at pixel position x, E(x) represents the exposure at pixel position x, and ω C ω S ω EThese are three constants, used to adjust the proportions of contrast, saturation, and exposure in the weight values. It should be noted that contrast, saturation, and exposure can be calculated from any frame in any two adjacent Class I images. In other words, calculating ω(x) according to the above formula only requires using one frame from two adjacent Class I images. The specific formulas for calculating C(x), S(x), and E(x) are as follows:

[0077]

[0078] in

[0079] Where μ e =0.5, σ e =0.2

[0080] Where I represents the first type of image involved in the weight calculation, R, G, and B represent the three color channels of I, and I(x) represents the grayscale value of the corresponding grayscale image at pixel position x. j μ(x) represents the color value of I at pixel position x in the j channel, and μ(x) represents the average of the three color values ​​of I at pixel position x. e and σ e These represent the mean and variance of the exposure, respectively, and can be constants as needed, such as 0.5 and 0.2 above.

[0081] For example, in step S120, a first-class weight image can be calculated based on I1 and I2 (note that, as mentioned above, some algorithms may need to use both I1 and I2 to calculate the first-class weight image, while other algorithms may only need to use one of I1 and I2 to calculate the first-class weight image, but for simplicity, it is still described here as calculating the first-class weight image based on I1 and I2), which can be denoted as m1. A first-class weight image can be calculated based on I2 and I3, which can be denoted as m2.

[0082] Step S130: Calculate the highlight image corresponding to each frame of the first type of image where the exposure compensation value is not positive.

[0083] The highlight image contains the high-brightness regions in the corresponding first-class image. In some implementations, the highlight image can be a binary image, where pixels can only take a first value (e.g., 255) or a second value (e.g., 0). A pixel taking the first value indicates that it belongs to a high-brightness region, while a pixel taking the second value indicates that it does not belong to a high-brightness region. Thus, the location of the high-brightness region in the highlight image can be determined by the position of the pixel taking the first value.

[0084] To obtain the corresponding highlight image, a binarization threshold can be set for the first type of image. For each pixel in the first type of image, if its value is greater than the threshold, the corresponding pixel value in the highlight image is set to the first value; otherwise, it is set to the second value. The brightness of a pixel and its pixel value are positively correlated; the larger the pixel value, the brighter the pixel. If the first type of image is an RGB image, it can be converted to a grayscale image before comparing its pixel values ​​with the threshold. The threshold set for different first type of images can be the same or different. For example, in one implementation, the threshold can decrease as the exposure compensation value of the first type of image decreases.

[0085] It is understandable that as long as the location of the high-brightness area can be recorded in the highlight image, it does not necessarily have to be a binary image. However, for the sake of simplicity, the following text will mainly take the case where the highlight image is a binary image as an example.

[0086] For example, for I1, I2, and I3, in step S130, the corresponding highlight images of the three frames can be calculated, which may be denoted as h1, h2, and h3. Step S140: Calculate a true highlight image containing highlight areas and / or white areas based on every two adjacent highlight images.

[0087] In this context, two adjacent highlight images refer to their corresponding first-class images being adjacent, and the adjacency of first-class images has already been defined above. For example, highlight images h1 and h2 are adjacent, and highlight images h2 and h3 are adjacent.

[0088] As described in step S130, the highlight image contains the corresponding high-brightness area in the first type of image. The highlight area and white area in step S140 are conceptually different from the high-brightness area. The highlight area and white area correspond to the highlight object and white object in the shooting scene, respectively, while the high-brightness area is simply an area formed by the aggregation of relatively bright pixels in the first type of image. However, since the brightness of highlight objects and white objects reflected in the image is usually higher than that of surrounding objects, it can be considered that the high-brightness area contains both highlight and white areas. Of course, if there are no highlight objects or white objects in the real scene, the high-brightness area may only contain either the highlight area or the white area.

[0089] Since the exposure compensation values ​​of each frame of the first type of image are different, the number and range of high-brightness areas in each frame of the highlight image may be different. By utilizing this difference between different frames of highlight images, and combining the characteristics of the highlight area and the white area itself, it is possible to further determine which high-brightness areas belong to the highlight area and which high-brightness areas belong to the white area.

[0090] For example, the inventors discovered the following principle: For at least two frames of a Type I image where all exposure compensation values ​​are non-positive, the true highlight areas in the scene are relatively bright in all frames, while the true white areas in the scene only have high brightness in frames with larger exposure compensation values, and their brightness decreases significantly in frames with smaller exposure compensation values, with the brightness decreasing rapidly as the exposure compensation value decreases. For example, the highlight areas are relatively bright in I1, I2, and I3, while the white areas are relatively bright in I1, relatively low in I2, and even lower in I3.

[0091] Based on this principle, the bright areas shared by two adjacent highlight images can be identified as the highlight areas in the shooting scene, while the bright areas contained in the bright highlight image but not in the dark highlight image can be identified as the white areas in the shooting scene. Specifically, a bright highlight image refers to the frame with the larger exposure compensation value of the corresponding first-type image among two adjacent highlight images, and a dark highlight image refers to the frame with the smaller exposure compensation value of the corresponding first-type image among two adjacent highlight images. For example, for two highlight images h1 and h2, h1 is the bright highlight image, and h2 is the dark highlight image.

[0092] It should be noted that the above calculations for highlight and white areas are based on two adjacent highlight images. Since the exposure compensation values ​​of two adjacent highlight images do not vary much, this calculation method is highly feasible. However, if the highlight and white areas are calculated on more highlight images, it may span multiple exposure compensation values, resulting in lower accuracy and less practical value.

[0093] The true highlight image in step S140 is an image containing highlight areas and / or white areas, serving as a carrier of the location information of the highlight areas and / or white areas. Whether the true highlight image should contain location information of the highlight areas or the white areas depends on the requirements: for example, if step S150 requires correction of pixel values ​​in the highlight areas, then the true highlight image calculated in step S140 must at least contain the location information of the highlight areas; if step S150 requires correction of pixel values ​​in the white areas, then the true highlight image calculated in step S140 must at least contain the location information of the white areas; if step S150 requires correction of pixel values ​​in both highlight and white areas, then the true highlight image calculated in step S140 must at least contain the location information of both highlight and white areas.

[0094] If a true highlight image contains only highlight areas, it can be a binary image. For example, pixels in the image can only take either a first value (e.g., 255) or a second value (e.g., 0). A pixel taking the first value indicates that it belongs to a highlight area, and a pixel taking the second value indicates that it does not belong to a highlight area. Similarly, if a true highlight image contains only white areas, it can also be a binary image. However, if a true highlight image contains both highlight and white areas, it can be a ternary image, or take a more complex form (see examples below).

[0095] Since the highlight images corresponding to the first type of images with non-positive exposure compensation values ​​have been obtained in step S130, and combined with the above definitions of highlight and white areas, the true highlight image can be calculated (at least one frame can be calculated, because it contains at least two first type of images with non-positive exposure compensation values). For example, a true highlight image, denoted as rh1, can be calculated based on highlight images h1 and h2; a true highlight image, denoted as rh2, can be calculated based on highlight images h2 and h3.

[0096] If the highlight image is a binary image (pixel values ​​are either the first or second value), and the true highlight image only contains highlight regions, the following are two methods for calculating the true highlight image corresponding to two adjacent highlight images:

[0097] Method A

[0098] Step a1: Determine the connected region in the bright frame specular image composed of pixels with the first value.

[0099] Step a2: Determine the centroid position of the connected region formed by the pixels with the first value in the dark frame highlight image.

[0100] In step a2, the connected regions formed by pixels with the first value in the dark frame highlight image can be determined first, and then the centroid can be calculated for each connected region. In steps a1 and a2, since the highlight images are all binarized images, it is easy to calculate the connected regions in the image, and each connected region represents a high-brightness region in the corresponding first type of image.

[0101] Step a3: For each centroid position obtained in step a2, determine whether it is located within a connected region in the bright frame specular image. If it is located within a connected region in the bright frame specular image, then determine a specular region in the real specular image corresponding to the connected region in the bright frame specular image.

[0102] Since the centroid of a connected region can represent the center of the region to some extent, if the centroid of a connected region in a dark frame highlight image falls within a connected region in a bright frame highlight image, it means that these two connected regions in the dark and bright frame highlight images constitute a common high-brightness region, i.e., a highlight region. In this way, the highlight region in the real highlight image can be quickly and accurately determined.

[0103] The highlight region can be either the region belonging to the bright frame highlight image in these two connected regions, or the region belonging to the dark frame highlight image in these two connected regions, or the union of these two regions. However, in step a3, the region belonging to the bright frame highlight image in these two connected regions was chosen.

[0104] Note that, strictly speaking, highlight regions are recorded in the actual highlight image, not in the bright frame highlight image. Therefore, the statement above that highlight regions can be taken from connected regions in the bright frame highlight image should be understood as taking connected regions in the bright frame highlight image as highlight regions, and their position information being recorded in the actual highlight image. Similar points will not be repeated later.

[0105] If the real highlight image is represented by a binary image, after the highlight area is determined, the pixels in the highlight area can be set to the first value in the real highlight image. After all the highlight areas are determined, the pixels in the real highlight image that do not belong to the highlight area can be set to the second value. In this way, the values ​​of all pixels in the real highlight image have been set.

[0106] Optionally, the true highlight image can be a separately created frame image, or the bright frame highlight image can be directly used as the true highlight image, since the bright frame highlight image is no longer used after step S140. Assuming the latter approach is adopted, if a connected region in the bright frame highlight image is determined to be a highlight region in step a3, the pixel values ​​in that region of the bright frame highlight image can be kept unchanged. After all highlight regions are determined, for those connected regions in the bright frame highlight image that do not belong to highlight regions, the pixels therein can be set to the second value. In this way, the bright frame highlight image is transformed into a true highlight image.

[0107] Method B

[0108] Step b1: Determine the connected region in the bright frame specular image composed of pixels with the first value.

[0109] Step b2: Determine the connected regions in the dark frame highlight image composed of pixels with the first value.

[0110] In steps b1 and b2, since the highlight images are all binarized images, it is easy to calculate the connected regions in the images. Each connected region represents a high-brightness region in the corresponding first-class image.

[0111] Step b3: For each connected region in the dark frame highlight image, determine whether there is a connected region in all connected regions of the bright frame highlight image whose overlap with it exceeds the first threshold. If there is a connected region in the bright frame highlight image whose overlap with it exceeds the first threshold, then determine a highlight region in the real highlight image corresponding to the connected region in the bright frame highlight image.

[0112] The degree of overlap in step b3 can be defined in different ways: for example, in one definition, the degree of overlap refers to the area ratio of the overlapping part of the connected regions in the connected regions of the dark frame highlight image; for another definition, the degree of overlap can be defined as the intersection-over-union (IoU) ratio of two connected regions.

[0113] If a connected region in the bright frame highlight image and a connected region in the dark frame highlight image have a high degree of overlap (greater than a first threshold), it indicates that these two connected regions in the dark frame highlight image and the bright frame highlight image are a common high-brightness region, i.e., a highlight region. In this way, the highlight region in the real highlight image can be quickly and accurately determined.

[0114] The highlight region can be either the region belonging to the bright frame highlight image in these two connected regions, or the region belonging to the dark frame highlight image in these two connected regions, or the union of these two regions. However, in step b3, the region belonging to the bright frame highlight image in these two connected regions was chosen.

[0115] For how to obtain a true highlight image after the highlight area has been determined, please refer to Method A, which will not be repeated here.

[0116] It is understandable that there are other ways to determine the highlight region in the real highlight image. For example, first, determine the centroid of the connected region formed by pixels with a first value in the bright frame highlight image; then, determine the connected region formed by pixels with the first value in the dark frame highlight image; finally, for each obtained centroid position, determine whether it is located within a connected region in the dark frame highlight image. If it is located within a connected region in the dark frame highlight image, then the region in the real highlight image corresponding to the connected region in the bright frame highlight image with that centroid is considered a highlight region. It is not difficult to see that this calculation method is similar to method A, but the inventors found that in the bright frame highlight image, the range of the high brightness region is generally large, or different high brightness regions are prone to overlap in the bright frame highlight image, which may lead to the calculated centroid position being inaccurate. Therefore, method A is more effective in calculating the real highlight image.

[0117] If the highlight image is a binary image (pixel values ​​are either the first or second value), and the true highlight image contains only white areas, the following are two methods for calculating the true highlight image corresponding to two adjacent highlight images:

[0118] Method C

[0119] Step c1: Determine the connected region in the bright frame specular image composed of pixels with the first value.

[0120] Step c2: Determine the centroid position of the connected region formed by the pixels with the first value in the dark frame highlight image.

[0121] Step c3: For each connected region in the bright frame specular image, determine whether the centroid position obtained in step a2 is located within it. If no centroid position is located within it, then define a white region in the real specular image corresponding to the connected region in the bright frame specular image.

[0122] Since the centroid of a connected region can represent the center of the region to some extent, if a connected region in a bright frame highlight image does not contain the centroid of any connected region in a dark frame highlight image, it indicates that the connected region is merely a bright area contained within the bright frame highlight image, and not a bright area contained within the dark frame highlight image, i.e., a white area. In this way, white areas in the actual highlight image can be quickly and accurately determined.

[0123] There are other methods for selecting the white area; please refer to Method A for selecting the highlight area. As for how to further obtain a realistic highlight image after determining the white area, you can also refer to Method A, which will not be repeated here.

[0124] Method D

[0125] Step d1: Determine the connected region in the bright frame specular image composed of pixels with the first value.

[0126] Step d2: Determine the connected regions in the dark frame highlight image composed of pixels with the first value.

[0127] Step d3: For each connected region in the bright frame highlight image, determine whether there is a connected region in the dark frame highlight image whose overlap with it exceeds the second threshold. If there is no connected region in the dark frame highlight image whose overlap with it exceeds the second threshold, then the region in the real highlight image corresponding to the connected region in the bright frame highlight image is determined as a white region.

[0128] If the overlap between a connected region in the bright frame highlight image and any connected region in the dark frame highlight image is not high (not exceeding the second threshold), it indicates that the connected region is merely a high-brightness area contained in the bright frame highlight image, and not a high-brightness area contained in the dark frame highlight image, i.e., a white area. In this way, the white area in the real highlight image can be quickly and accurately determined.

[0129] There are other methods for selecting the white area; please refer to Method A for selecting the highlight area. As for how to further obtain a realistic highlight image after determining the white area, you can also refer to Method A, which will not be repeated here.

[0130] If the highlight image is a binary image, and the true highlight image contains both highlight and white regions, then one of methods A and B can be chosen to determine the highlight region, and one of methods C and D can be chosen to determine the white region. However, it should be noted that repeated steps do not need to be executed multiple times. For example, if methods A and C are selected, since a1, a2 and c1, c2 are the same, they do not need to be repeated. Optionally, if highlight and white regions are considered complementary—that is, a bright region is either a highlight region or a white region—then all bright regions that are considered highlight regions can be determined first (e.g., in a bright frame highlight image), and then all remaining bright regions can be treated as white regions. Alternatively, all bright regions that are considered white regions can be determined first (e.g., in a bright frame highlight image), and then all remaining bright regions can be treated as highlight regions.

[0131] Step S150: Correct the pixel values ​​in the corresponding first-class weight image using the highlight area and / or white area in the real highlight image to obtain the corrected first-class weight image.

[0132] Based on the preceding steps, for two adjacent frames of Type I images where both exposure compensation values ​​are non-positive, a Type I weighted image and a true highlight image can be calculated. Therefore, each true highlight image corresponds to a Type I weighted image. However, conversely, a Type I weighted image may not correspond to a true highlight image. If a Type I weighted image does not have a corresponding true highlight image, then pixel value correction is unnecessary. However, the Type I weighted image in step S150 refers to the Type I weighted image that requires pixel value correction. For example, if the Type I weighted images are m1 and m2, and the corresponding true highlight images are rh1 and rh2, then by correcting m1 using rh1, we can obtain the corrected m1, denoted as nm1. By correcting m2 using rh2, we can obtain the corrected m2, denoted as nm2.

[0133] In particular, different correction strategies are adopted for the pixel values ​​located in the highlight area and the white area in the first type of weighted image (referring to the highlight area and the white area in the corresponding real highlight image), so that the pixel values ​​in the highlight area and the white area in the corrected first type of weighted image have obvious differences (if no correction is made, the pixel values ​​located in the highlight area and the white area in the first type of weighted image may be quite similar).

[0134] For example, increasing the pixel values ​​in the highlight region of the first-class weighted image and decreasing the pixel values ​​in the white region of the first-class weighted image are two different correction strategies; similarly, increasing the pixel values ​​in the highlight region of the first-class weighted image and keeping the pixel values ​​in the white region of the first-class weighted image unchanged are also two different correction strategies.

[0135] It should be noted that maintaining the pixel value unchanged is also a correction strategy, meaning that the correction strategy does not necessarily lead to a change in the pixel value in the first type of weighted image.

[0136] In some implementations, the correction strategy can be formulated for one of the following purposes:

[0137] Objective 1: To increase the proportion of the dark frame image to be fused in the fusion result by increasing the weight value of the first type of weighted image in the highlight region compared with the original image.

[0138] As mentioned in step S120, the two frames used for fusion of the first-class weighted image in each frame can be called the images to be fused. The dark frame to be fused mentioned in Objective 1 refers to the frame with the smaller exposure compensation value among the two frames to be fused. Correspondingly, the bright frame to be fused can be defined as the frame with the larger exposure compensation value among the two frames to be fused. For example, for two frames to be fused under ev0 and ev-1, the one under ev0 is the bright frame to be fused, and the one under ev-1 is the dark frame to be fused.

[0139] The weighted fusion process of two images to be fused can be represented by the following fusion formula:

[0140] f = m × I dark +(1-m)×I bright

[0141] Where f represents the fusion result, m represents the first type of weighted image (this formula applies to both before and after correction), and I dark I represents the dark frame image to be fused. bright This represents the image to be fused from the bright frame, where "1" indicates a matrix of all 1s. Alternatively, this formula can also be viewed as a pixel-wise formula, i.e., for f, m, I... dark I bright The formula holds true for any corresponding pixel position in the image. It should be noted that the first type of weight image used in this formula is normalized, that is, the range of pixel values ​​in the image has been mapped to [0,1]. For simplicity, the normalization of the first type of weight image will not be specifically mentioned in the following text.

[0142] According to the above fusion formula, the proportion of a certain frame of the image to be fused in the fusion result is directly determined by the fusion weights used when weighting the fused images of that frame. For example, in the fusion formula above, I dark The proportion of I in the fusion result is directly determined by m. bright The proportion in the fusion result is directly determined by (1-m). Therefore, according to the above fusion formula, if we want to improve I... dark The proportion in the fusion result can be increased by increasing the pixel values ​​in m (the pixel values ​​in the first type of weighted image represent the weight values). If the goal is to improve I... bright The proportion in the fusion result can reduce the pixel value in m.

[0143] However, it should be noted that although the pixel values ​​in the first type of weight image are used as the fusion weights of the dark frame to be fused in this fusion formula, it is possible that in some implementations, the pixel values ​​in the first type of weight image are used as the fusion weights of the bright frame to be fused. Therefore, increasing the pixel values ​​in the first type of weight image does not necessarily increase the proportion of the dark frame to be fused in the fusion result in any implementation. It may be necessary to decrease the pixel values ​​in the first type of weight image to increase the proportion of the dark frame to be fused in the fusion result. The reason is that after the fusion weight of the bright frame to be fused is reduced, the fusion weight of the dark frame to be fused will inevitably increase due to the complementarity of the fusion weights.

[0144] Objective 1 can be achieved by correcting the pixel values ​​of the first type of weighted image in the highlight region. After pixel value correction, the proportion of the dark frame image to be fused in the highlight region of the fusion result is increased. Since the exposure compensation value of the dark frame fusion image is smaller, it contains more detail information in the highlight region, thus helping to improve the problem of grayish highlight region in the fusion result. As for whether the pixel values ​​of the first type of weighted image outside the highlight region should be corrected, Objective 1 does not restrict this.

[0145] Objective 2: To reduce the proportion of dark frames in the fusion result of the first type of weighted image in the corresponding white area compared to the original image.

[0146] Objective 2 can be achieved by correcting the pixel values ​​of the first type of weight image in the white region. After pixel value correction, the proportion of the dark frame image to be merged in the white region of the fusion result is reduced. According to the complementarity of the fusion weights, this also means that the proportion of the bright frame image to be merged is increased. Since the exposure compensation value of the bright frame image to be merged is larger, it contains more detail information in the white region, thus helping to improve the problem of grayish white regions in the fusion result. As for whether the pixel values ​​of the first type of weight image outside the white region should be corrected, Objective 2 does not restrict this.

[0147] Objective 3: Includes both Objective 1 and Objective 2.

[0148] Objective 3 can be achieved by correcting the pixel values ​​of the first type of weighted image in the highlight and white regions. After pixel value correction, the proportion of dark frames to be merged in the highlight regions of the fusion result is increased, while the proportion of bright frames to be merged in the white regions of the fusion result is increased. This helps to improve the problem of graying in both highlight and white regions in the fusion result. Objective 3 does not restrict whether the pixel values ​​of the first type of weighted image outside the corresponding highlight and white regions should be corrected.

[0149] The following section will introduce the correction strategies that can be adopted to achieve one or more of the above objectives. In the introduction, we will take the pixel values ​​in the first type of weighted image as the weight values ​​of the dark frame to be fused image during weighted fusion as an example (as in the previous fusion formula).

[0150] Strategy 1: Increase the pixel values ​​located in the highlight area of ​​the first type of weighted image, and keep the pixel values ​​located outside the highlight area of ​​the first type of weighted image unchanged.

[0151] Strategy 1 actually includes two sub-strategies. Sub-strategy 1 increases the pixel values ​​located in the highlight area of ​​the first type of weighted image, and sub-strategy 2 keeps the pixel values ​​outside the highlight area of ​​the first type of weighted image unchanged. Sub-strategy 1 is applied to the highlight area, and sub-strategy 2 is applied to the area outside the highlight area (including the white area). Therefore, it satisfies the requirement of applying different correction strategies to the highlight area and the white area, and is thus a feasible strategy.

[0152] In Strategy 1, increasing the pixel values ​​located within the corresponding highlight regions in the first type of weighted image can increase the proportion of the dark frame image to be fused within the highlight regions in the fusion result, thus achieving objective 1. It can be understood that to achieve Strategy 1, the real highlight image calculated in step S140 must at least contain highlight regions.

[0153] Strategy 1 does not limit the rules used to increase the pixel value in the highlight area. For example, the original pixel value can be multiplied by a scaling factor greater than 1, the original pixel value can be added with a certain value, the original pixel value can be directly set as the upper limit of the pixel value range, and so on.

[0154] It should be noted that the statement in Strategy 1 about increasing the pixel values ​​within the corresponding highlight regions of the first-class weighted image does not mean that every pixel value within the highlight region will increase. It only means that, overall, the pixel values ​​within the highlight region increase. For example, if the range of pixel values ​​in the first-class weighted image is [0, 255], and a certain pixel value originally had a value of 255, then it cannot be increased further. When other strategies are introduced later, any mention of increasing or decreasing should be understood in this overall sense and will not be specifically explained further.

[0155] Strategy 2: Increase the pixel values ​​located in the highlight area of ​​the first-class weighted image, and decrease the pixel values ​​located outside the highlight area of ​​the first-class weighted image.

[0156] Strategy 2 actually includes two sub-strategies. Sub-strategy 1 increases the pixel values ​​in the highlight region of the first type of weighted image, and sub-strategy 2 decreases the pixel values ​​outside the highlight region of the first type of weighted image. Sub-strategy 1 is applied to the highlight region, and sub-strategy 2 is applied to the region outside the highlight region (including the white region). Therefore, it satisfies the requirement of applying different correction strategies to the highlight region and the white region, and is thus a feasible strategy.

[0157] In Strategy 2, increasing the pixel values ​​within the corresponding highlight regions of the first-class weight image increases the proportion of the dark frame image to be fused within the highlight regions of the fusion result. Conversely, decreasing the pixel values ​​outside the corresponding highlight regions of the first-class weight image necessarily decreases the pixel values ​​within the white regions, thus reducing the proportion of the dark frame image to be fused within the white regions of the fusion result, thereby achieving objective 3. Since objective 3 includes both objective 1 and objective 2, strategy 2 can be used even if only objective 1 or objective 2 is desired. It can be understood that to achieve strategy 2, the real highlight image calculated in step S140 must contain at least highlight regions.

[0158] Strategy 2 does not specify which rules to use to increase pixel values ​​within the highlight region and decrease pixel values ​​outside the highlight region. The following example uses the real highlight image rh1 to correct the first-class weight image m1, resulting in the corrected first-class weight image nm1. The following formula illustrates a weight correction rule that conforms to Strategy 2:

[0159] nm1=ω×rh1+(1-ω)×m1

[0160] Where ω is a preset coefficient with a value range of (0,1), the true specular image rh1 is a binary image, the pixel value in the specular region of rh1 is 255, and the pixel value outside the specular region of rh1 is 0, and the pixel value range in the first type weighted image m1 is [0,255]. If the specular region is denoted as G, rh1(G) represents the pixel value of rh1 in G (at any pixel position), m1(G) represents the pixel value of m1 in G, and nm1(G) represents the corrected pixel value of m1 in G, then we have:

[0161] nm1(G)=ω×(rh1(G)-m1(G))+m1(G)

[0162] =ω×(255-m1(G))+m1(G)

[0163] Since m1(G)≤255, ω×(255-m1(G))≥0, therefore nm1(G)≥m1(G). This means that after correction according to this rule, the pixel value of m1 within the highlight region increases overall. Similarly, the pixel value of m1 outside the highlight region decreases overall.

[0164] In Strategy 2, the pixel values ​​in non-highlight, non-white regions of the first type of weighted image are also reduced, which may lead to a decrease in the image quality of the fusion result in these regions. This problem can be improved through post-processing.

[0165] Strategy 3: Reduce the pixel values ​​located within the white area in the first-class weighted image, and keep the pixel values ​​located outside the white area in the first-class weighted image unchanged.

[0166] Strategy 3 actually includes two sub-strategies. Sub-strategy 1 reduces the pixel values ​​located in the white area of ​​the first type of weight image, and sub-strategy 2 keeps the pixel values ​​located outside the white area of ​​the first type of weight image unchanged. Sub-strategy 1 is applied to the white area, and sub-strategy 2 is applied to the area outside the white area (including the highlight area). Therefore, it satisfies the requirement of applying different correction strategies to the highlight area and the white area, and is thus a feasible strategy.

[0167] In strategy 3, reducing the pixel values ​​located within the white area in the first type of weight image can reduce the proportion of the dark frame image to be fused within the white area in the fusion result, thus achieving objective 2. It can be understood that to achieve strategy 3, the true highlight image calculated in step S140 must contain at least a white area.

[0168] Strategy 3 does not limit the rules used to reduce pixel values ​​in the white area. For example, the original pixel value can be multiplied by a scaling factor less than 1, the original pixel value can be subtracted by a certain value, the original weight can be directly set as the lower limit of the pixel value, and so on.

[0169] Strategy 4: Decrease the pixel values ​​in the first-class weighted image that are located within the corresponding white area, and increase the pixel values ​​in the first-class weighted image that are located outside the corresponding white area.

[0170] Strategy 4 actually includes two sub-strategies. Sub-strategy 1 reduces the pixel values ​​in the white area of ​​the first type of weighted image, and sub-strategy 2 increases the pixel values ​​in the white area of ​​the first type of weighted image. Sub-strategy 1 is applied to the white area, and sub-strategy 2 is applied to the area outside the white area (including the highlight area). Therefore, it satisfies the requirement of applying different correction strategies to the highlight area and the white area, and is thus a feasible strategy.

[0171] In strategy 4, reducing the pixel values ​​located within the white areas of the first type of weight image reduces the proportion of the dark frame image to be fused within the white areas of the fusion result. Conversely, increasing the pixel values ​​located outside the white areas of the first type of weight image inevitably increases the pixel values ​​within the highlight areas, thereby increasing the proportion of the dark frame image to be fused within the highlight areas of the fusion result. This achieves objective 3. Since objective 3 includes objectives 1 and 2, strategy 4 can be used even if only objective 1 or objective 2 is desired. It can be understood that to achieve strategy 4, the true highlight image calculated in step S140 must contain at least white areas.

[0172] Strategy 4 does not specify which rules to use to decrease pixel values ​​within the white area or increase pixel values ​​outside the white area.

[0173] In Strategy 4, the pixel values ​​in non-highlight, non-white regions of the first type of weighted image are also increased, which may lead to a decrease in the image quality of the fusion result in these regions. This problem can be improved through post-processing.

[0174] Strategy 5: Increase the pixel values ​​in the highlight areas of the first type of weighted image, decrease the pixel values ​​in the white areas of the first type of weighted image, and keep the pixel values ​​in the highlight areas and the corresponding white areas of the first type of weighted image unchanged.

[0175] Strategy 5 actually includes three sub-strategies. Sub-strategy 1 increases the pixel values ​​located in the highlight region of the first type of weight image, sub-strategy 2 decreases the pixel values ​​located in the white region of the first type of weight image, and sub-strategy 3 keeps the pixel values ​​outside the highlight and white regions of the first type of weight image unchanged. Since sub-strategy 1 is applied to the highlight region, sub-strategy 2 is applied to the white region. Therefore, it satisfies the requirement of applying different correction strategies to the highlight and white regions, and is thus a feasible strategy.

[0176] In strategy 5, increasing the pixel values ​​located in the highlight regions of the first type of weight image can increase the proportion of the dark frame image to be fused in the highlight regions of the fusion result, while decreasing the pixel values ​​located in the white regions of the first type of weight image can decrease the proportion of the dark frame image to be fused in the white regions of the fusion result, thus achieving objective 3. Since objective 3 includes objective 1 and objective 2, strategy 5 can be used even if only objective 1 or objective 2 is desired. It can be understood that to achieve strategy 5, the real highlight image calculated in step S140 must contain at least highlight regions and white regions.

[0177] Strategy 5 does not specify which rules to use to increase pixel values ​​in highlight areas and decrease pixel values ​​in white areas.

[0178] The following example uses the real specular image rh1 to correct the first-class weight image m1, resulting in the corrected first-class weight image nm1. The following formula illustrates a weight correction rule that conforms to Strategy 5:

[0179] nm1=ω×rh1+(1-ω)×m1

[0180] Wherein, ω is a preset coefficient, with a value range of (0,1). In the highlight region of the real highlight image rh1, the pixel value is 255; in the white region of rh1, the pixel value is 0; and in the non-highlight, non-white region of rh1, the pixel value is the pixel value at the corresponding pixel position in the weighted image m1 (rh1 is calculated in this way in step S140). Referring to the analysis method of the above formula in Strategy 2, it can be seen that this weight correction meets the requirements of Strategy 5.

[0181] It should be understood that the correction strategies are not limited to the above five types: For example, while increasing the pixel values ​​located in the highlight region of the first type of weighted image, there is no restriction on how the pixel values ​​located outside the highlight region of the first type of weighted image are corrected, as long as the requirement of adopting different correction strategies for the highlight region and the white region is met; for another example, while decreasing the pixel values ​​located in the white region of the first type of weighted image, there is no restriction on how the pixel values ​​located outside the white region of the first type of weighted image are corrected, as long as the requirement of adopting different correction strategies for the highlight region and the white region is met; for yet another example, while increasing the pixel values ​​located in the highlight region of the first type of weighted image and decreasing the pixel values ​​located in the white region of the first type of weighted image, there is no restriction on how the pixel values ​​located outside the white region of the first type of weighted image are corrected.

[0182] As mentioned earlier, sub-strategies 1 and 2 in strategy 5 can improve the image quality of the fusion result in the highlight and white areas. Sub-strategy 3 maintains the pixel values ​​outside the highlight and white areas in the first type of weight image, so that the pixel value correction will not reduce the image quality in these areas. This allows the post-processing steps for the fusion result to be omitted or simplified.

[0183] In some implementations, after performing step S140 and before performing step S150, the real highlight image can be smoothed and filtered first to weaken or eliminate noise in the image, thereby improving the quality of the subsequently obtained fused image. When correcting pixel values ​​in step S150, the filtered real highlight image should be used. It should be noted that smoothing filtering may, to some extent, change the shape of the highlight area and / or white area in the real highlight image.

[0184] For example, assuming that in step S150, according to strategy 2, the first type of weight image m1 is corrected using the real specular image rh1 to obtain the corrected first type of weight image nm1, then the following weight correction formula can be used:

[0185] nm1=ω×Gau(rh1)+(1-ω)×m1

[0186] Wherein, Gau represents performing Gaussian filtering on rh1 obtained in step S140.

[0187] Step S160: Use at least the corrected first-class weighted image to perform weighted fusion on multiple frames of first-class images to obtain the final fused image.

[0188] First, consider the case where the exposure compensation values ​​of the multiple first-class images in step S110 are all non-positive. By step S150, the corrected first-class weight images between two adjacent first-class images have been calculated. There are multiple methods to fuse the multiple first-class images. The following example illustrates an iterative fusion method:

[0189] Step x1: Take the first frame of the first type image with the largest or smallest exposure compensation value as the current first frame of the first type image and the initial fused image.

[0190] Step x1 is the initialization step for iterative fusion. For example, I1 can be used as the first type image of the current frame and the initial fused image, in which case the fusion order is the order in which the exposure compensation value gradually decreases. Alternatively, I3 can be used as the first type image of the current frame and the initial fused image, in which case the fusion order is the order in which the exposure compensation value gradually increases. The following mainly uses the order in which the exposure compensation value gradually decreases as an example.

[0191] Step x2: Use the corrected first-class weighted image of the current frame to perform weighted fusion of the current fused image and the first-class image of the next frame to obtain a new fused image.

[0192] Steps x2 and x3 are iterative steps. The next frame of the first-class image is relative to the current frame of the first-class image. Which frame is the next frame depends on the fusion order: if the exposure compensation values ​​decrease gradually, the next frame of the first-class image is the one adjacent to the current frame of the first-class image with a smaller corresponding exposure compensation value; if the exposure compensation values ​​increase gradually, the next frame of the first-class image is the one adjacent to the current frame of the first-class image with a larger corresponding exposure compensation value. For example, if the current frame of the first-class image is I1, then the next frame of the first-class image is I2.

[0193] The corrected first-class weight image of the current frame refers to the corrected first-class weight image between the first-class image of the current frame and the first-class image of the next frame. For example, the corrected first-class weight image between I1 and I2 is nm1.

[0194] The weighted fusion in step x2 can be performed using the fusion formula given earlier:

[0195] f i+1 =nm i ×I i+1 +(1-nm i )×fi

[0196] Where i represents the round number of the iterative fusion, i can start from 1, and f i This is the current fused image (f1 = I1), f i+1 This is the new fused image obtained after this round of fusion, I i+1 The next frame is a first-class image, nm i isI i (Current frame, first type image) and I i+1 The corrected first-class weighted image between them. i+1 Corresponding to I in the previous formula dark , and f i Corresponding to I in the previous formula bright For example, for the fusion between I1 (i.e., f1) and I2, the above formula can be specified as:

[0197] f2=nm1×I2+(1-nm1)×f1

[0198] The above fusion formula explains the image to be fused defined earlier. This image could be a first-class image, or the fusion result of at least two first-class images. Additionally, it should be noted that for f... i (When it is not equal to I1), strictly speaking, there is no precise exposure compensation value because it has already undergone image fusion, but it can be assumed that its exposure compensation value will not be greater than I. i The exposure compensation value (if blended in order of increasing exposure compensation value), or at least not less than I. i The exposure compensation value (if in order of decreasing exposure compensation value).

[0199] Step x3: Determine whether the first-class image in the next frame is the same as the first-class image in the last frame.

[0200] If the judgment result is yes, the iteration ends and the fused image obtained at this time is the final fused image; if the judgment result is no, jump to step x2 to continue execution, but the first fused image of the current frame should be updated to the first fused image of the next frame, and the current fused image should be updated to the newly obtained fused image of this iteration, because the next iteration has already begun.

[0201] For example, in the first iteration, f2 is obtained by fusing I1 (i.e. f1) and I2 using nm1. Since I2 is not the last frame of the first type image, the second iteration can continue to be executed. f2 and I3 are fused using nm2 to obtain f3. Since I3 is the last frame of the first type image, the iteration ends and f3 is the final fused image.

[0202] For the weighted fusion method given in steps x1 to x3, since the exposure compensation value of the image to be fused changes gradually rather than abruptly during the fusion process, the final fusion result has high image quality.

[0203] Considering the case in step S110 where at least one frame of the multiple first-type images has a positive exposure compensation value (of course, the condition that at least two frames have non-positive exposure compensation values ​​is still satisfied), after step S150 is completed, two types of first-type weight images are obtained: a corrected first-type weight image between two adjacent first-type images where both exposure compensation values ​​are non-positive (calculated in step S150), and a first-type weight image between two adjacent first-type images where at least one exposure compensation value is positive (calculated in step S120 and not corrected in subsequent steps). In step S160, these two images are used to fuse the multiple first-type images, still considering an iterative fusion method:

[0204] Step y1: Take the first frame of the first type image with the largest or smallest exposure compensation value as the current first type image and the initial fused image.

[0205] Step y2: If at least one of the exposure compensation values ​​of the current frame's first-class image and the next frame's first-class image is positive, then the current fused image and the next frame's first-class image are weighted and fused using the current frame's first-class weighted image to obtain a new fused image; if the exposure compensation values ​​of the current frame's first-class image and the next frame's first-class image are both non-positive, then the current fused image and the next frame's first-class image are weighted and fused using the current frame's corrected first-class weighted image to obtain a new fused image.

[0206] Among them, the first-class weighted image of the current frame refers to the first-class weighted image between the first-class image of the current frame and the first-class image of the next frame; the corrected first-class weighted image of the current frame refers to the corrected first-class weighted image between the first-class image of the current frame and the first-class image of the next frame.

[0207] Step y3: Determine whether the first-class image in the next frame is the same as the first-class image in the last frame.

[0208] If the judgment result is yes, the iteration ends and the fused image obtained at this time is the final fused image; if the judgment result is no, jump to step x2 to continue execution, but the first fused image of the current frame should be updated to the first fused image of the next frame, and the current fused image should be updated to the newly obtained fused image of this iteration, because the next iteration has already begun.

[0209] It is easy to see that steps y1 to y3 are similar to steps x1 to x3, except that the calculation of the new fused image in step y2 is divided into two cases. The details of each step can be found in the previous explanation and will not be repeated here.

[0210] For example, in step S110, in addition to I1, I2, and I3, a first-class image at ev+2 and a first-class image at ev+1 are also acquired, denoted as K2 and K1 respectively. In step S120, in addition to calculating m1 based on I1 and I2 and m2 based on I2 and I3, a first-class weighted image n1 is calculated based on K1 and I1, and a first-class weighted image n2 is calculated based on K2 and K1. In step S150, m1 is corrected to nm1, and m2 is corrected to nm2. In step S160, if the fusion order is the order in which the exposure compensation value gradually decreases, the fusion process is as follows: first, based on K2 and K1, g1 is obtained by fusing with n2; then, based on g1 and I1, g2 is obtained by fusing with n1; then, based on g2 and I2, g3 is obtained by fusing with nm1; finally, based on g3 and I3, g4 is obtained by fusing with nm2. g4 is the final fused image.

[0211] Using both non-positive and positive exposure compensation values ​​of the first class images for image fusion can improve the quality of the fused image. For example, in some dark scenes, the first class image with positive exposure compensation values ​​may contain more valuable details than the first class image with non-positive exposure compensation values.

[0212] The "at least" mentioned in step 160 takes into account the case where there are first-class images with positive exposure compensation values. In this case, image fusion uses not only the corrected first image but also a portion of the uncorrected first-class images.

[0213] It should be noted that the final fused image mentioned in step S160 only represents the final result output by the image fusion method provided in this application embodiment, and is not necessarily the final fused image presented to the user. Several post-processing operations may be performed on it before it is presented to the user.

[0214] In summary, the image fusion method provided in this application uses a real highlight image containing highlight and / or white regions to correct the pixel values ​​located in the highlight and white regions of the first type of weighted image, and applies different correction strategies to the pixel values ​​in the highlight and white regions. This results in a significant difference between the pixel values ​​in the highlight and white regions in the corrected first type of weighted image. Furthermore, since the pixel values ​​in the first type of weighted image represent the weight values ​​during weighted fusion, using the corrected first type of weighted image to perform weighted fusion on multiple frames of the first type of image allows for a significant difference in the fusion strategy between the highlight and white regions (the fusion strategy, such as whether to fuse more dark or bright frames, depends on the fusion weight). Therefore, this method helps to improve the problem of grayish highlights or white regions in the fused image and enhances the quality of the fused image.

[0215] Furthermore, in some implementations, by calculating the highlight image (containing the high-brightness area in the first type of image) corresponding to each frame of the first type of image with a non-positive exposure compensation value, and by performing a comparative calculation on the highlight images of two adjacent frames in a certain sense, the true highlight area (the area with relatively high brightness in both adjacent highlight images) and / or white area (the area with relatively high brightness only in the highlight image of the bright frame and relatively low brightness in the highlight image of the dark frame) in the first type of image can be determined. Then, the first type of weighted image can be specifically modified based on the highlight area and / or white area, so that the highlight area and white area have differentiated fusion weights in the modified first type of weighted image.

[0216] For example, the correction strategy for the first type of weighted image can be designed to tend to fuse more pixel information from the first type of image with a smaller exposure compensation value in the highlight area, thereby increasing the detail of the fused image in the highlight area and preventing it from appearing grayish; and / or, it can be designed to tend to fuse more pixel information from the first type of image with a larger exposure compensation value in the white area, thereby increasing the detail of the fused image in the white area and preventing it from appearing grayish.

[0217] Figure 3 The flowchart of a second image fusion method provided in an embodiment of this application is illustrated. This method can be, but is not limited to, performed by an electronic device. Figure 6 One possible structure of the electronic device is shown below, for details in the following section. Figure 6 The explanation. (Refer to...) Figure 3 The method includes:

[0218] Step S200: Obtain multiple frames of the first type of image.

[0219] Among them, the exposure compensation values ​​of each frame of the first type of image are different, and the exposure compensation values ​​of at least two frames of the first type of image are non-positive numbers.

[0220] Step S210: Take the first frame of the first type image with the largest or smallest exposure compensation value as the current first frame of the first type image and the initial fused image.

[0221] Step S210 is the initialization step for iterative fusion, while steps S220 to S290 are iterative steps. The choice of which frame of the first type of image to use as the current frame and the initial fused image in step S210 depends on the desired fusion order. If the fusion order is one where the exposure compensation value gradually decreases, then step S210 should select the frame of the first type of image with the largest exposure compensation value. If the fusion order is one where the exposure compensation value gradually increases, then step S210 should select the frame of the first type of image with the smallest exposure compensation value.

[0222] Step S220: Calculate the second-class weight image between the current fused image and the first-class image of the next frame.

[0223] In this context, the next frame of the first-class image is relative to the current frame of the first-class image. Which frame is the "next frame" depends on the fusion order: if the exposure compensation values ​​decrease gradually, the next frame of the first-class image is the one adjacent to the current frame with the smallest corresponding exposure compensation value; if the exposure compensation values ​​increase gradually, the next frame of the first-class image is the one adjacent to the current frame with the largest corresponding exposure compensation value. The pixel values ​​in the second-class weighted image represent the weight values ​​used in the weighted fusion process. It's important to note that the second-class weighted image is calculated based on one first-class image and one fused image, while the first-class image is calculated based on two first-class images.

[0224] Step S230: Determine whether the exposure compensation values ​​of the first type image in the current frame and the first type image in the next frame are both non-positive numbers.

[0225] If all numbers are non-positive, proceed to step S250. If at least one number is positive, proceed to step S240. However, whether you proceed to step S240 or step S250, the ultimate goal is to calculate a new fused image; only the calculation process is different.

[0226] Step S240: Use the second type of weighted image to perform weighted fusion of the current fused image and the first type of image in the next frame to obtain a new fused image.

[0227] The weighted fusion can be performed using the fusion formula given in step S160, which will not be repeated here.

[0228] Step S250: Calculate the highlight images corresponding to the current fused image and the first type image in the next frame, respectively.

[0229] Step S250 will calculate two frames of highlight images, each containing the high-brightness region in the current fused image and the next frame of the first type image, respectively.

[0230] Step S260: Calculate a true highlight image containing highlight areas and / or white areas based on the two obtained highlight images.

[0231] Among them, the highlight area is the high-brightness area that is shared by both highlight images, and the white area is the high-brightness area that is included in the bright frame highlight image but not in the dark frame highlight image.

[0232] Step S270: Correct the weight values ​​in the corresponding second-type weight image using the highlight area and / or white area in the real highlight image to obtain the corrected second-type weight image.

[0233] Specifically, different correction strategies are applied to the pixel values ​​located in the highlight and white regions of the second type of weighted image. For example, the correction strategy can be formulated to achieve the following objectives: to increase the proportion of the dark frame image to be fused in the fusion result by adjusting the weight value of the second type of weighted image in the highlight region compared to the original image, and / or to decrease the proportion of the dark frame image to be fused in the fusion result by adjusting the weight value of the second type of weighted image in the white region compared to the original image.

[0234] Step S280: Use the corrected second-class weighted image to perform weighted fusion of the current fused image and the first-class image of the next frame to obtain a new fused image.

[0235] The weighted fusion can be performed using the fusion formula given in step S160, which will not be repeated here.

[0236] Step S290: Determine whether the first type image in the next frame is the same as the first type image in the last frame.

[0237] If the judgment result is yes, the iteration ends and the fused image obtained at this time is the final fused image; if the judgment result is no, the process jumps to step S220 to continue execution, but the first fused image of the current frame should be updated to the first fused image of the next frame, and the current fused image should be updated to the fused image newly obtained in this round of iteration, because the next round of iteration has already begun.

[0238] The implementation details of steps S200 to S290 have been mostly mentioned when introducing steps S110 to S160, and can be referred to the previous content, so they will not be repeated here.

[0239] For example, when fusing I1, I2, and I3, if the fusing order is the order in which the exposure compensation value gradually decreases, in the first iteration, the second type of weight image is first calculated based on I1 (i.e., the initial fused image l1) and I2, denoted as w1. Then, the highlight images corresponding to I1 and I2 are calculated, denoted as p1 and p2. Then, the real highlight image rp1 is calculated based on p1 and p2, and w1 is corrected using rp1 to obtain the corrected second type of weight image, denoted as nw1. Finally, I1 and I2 are fused using nw1 to obtain the fused image l2. Since I2 is not the last frame of the first type image, the second iteration can continue. First, calculate the second type weight image based on l2 and I3, denoted as w2. Then, calculate the specular images corresponding to l2 and I3, denoted as q2 and p3. Next, calculate the true specular image rp2 based on q2 and p3, and use rp2 to correct w2 to obtain the corrected second type weight image, denoted as nw2. Finally, use nw2 to fuse l2 and I3 to obtain the fused image l3. Since I3 is the last frame of the first type image, the iteration ends, and l3 is the final fused image.

[0240] For example, when fusing K2, K1, I1, I2, and I3, if the fusing order is one where the exposure compensation values ​​gradually decrease, in the first iteration, a second-class weighted image, denoted as w1, is first calculated based on K2 (i.e., the initial fused image l1) and K1. Then, w1 is used to fuse K2 and K1 to obtain the fused image l2. Since K1 is not the last frame of the first-class image, the second iteration can continue. First, a second-class weighted image, denoted as w2, is calculated based on l2 and I1. Then, w2 is used to fuse l2 and I1 to obtain the fused image l3. Since I1 is not the last frame of the first type image, the third iteration can continue. First, calculate the second type weight image based on l3 and I2, denoted as w3. Then calculate the specular images corresponding to l3 and I2, denoted as p1 and p2. Then calculate the real specular image rp1 based on p1 and p2, and use rp1 to correct w3 to obtain the corrected second type weight image, denoted as nw1. Finally, use nw1 to fuse l3 and I2 to obtain the fused image l4. Since I2 is not the last frame of the first type image, the fourth iteration can continue. First, calculate the second type weight image based on l4 and I3, denoted as w4. Then, calculate the specular images corresponding to l4 and I3, denoted as q2 and p3. Next, calculate the true specular image rp2 based on q2 and p3, and use rp2 to correct w4 to obtain the corrected second type weight image, denoted as nw4. Finally, use nw4 to fuse l4 and I3 to obtain the fused image l5. Since I3 is the last frame of the first type image, the iteration ends, and l5 is the final fused image.

[0241] Figure 3In the image fusion method, since the source of the second type of weighted image and the object it is used for fusion are the same (both are the current fused image and the first type of image in the next frame), for example, in the first example above, w2 is calculated based on l2 and I3, and nw2 is also used for weighted fusion of l2 and I3, so it may obtain a fused image of better quality.

[0242] Figure 4 A functional block diagram of the image fusion apparatus 300 provided in an embodiment of this application is shown. (Refer to...) Figure 4 The image fusion device 300 includes:

[0243] The first image acquisition module 310 is used to acquire multiple frames of first-class images, each frame of the first-class images having a different exposure compensation value, and at least two of the first-class images having a non-positive exposure compensation value.

[0244] The weighted image calculation module 320 is used to calculate the first type of weighted image between every two adjacent frames of the first type of images. The pixel values ​​in the first type of weighted image represent the weight values ​​of the images when performing weighted fusion.

[0245] The highlight image calculation module 330 is used to calculate the highlight image corresponding to each frame of the first type of image with a non-positive exposure compensation value, wherein the highlight image contains the high brightness area in the corresponding first type of image.

[0246] The true specular image calculation module 340 is used to calculate a true specular image containing a specular region and / or a white region based on every two adjacent specular images;

[0247] The weighted image correction module 350 is used to correct the pixel values ​​in the corresponding first type weighted image by using the highlight area and / or white area in the real highlight image, so as to obtain the corrected first type weighted image.

[0248] The first image fusion module 360 ​​is used to perform weighted fusion of the multiple frames of first-class images using at least the corrected first-class weighted images to obtain the final fused image.

[0249] In one implementation of the image fusion device 300, the highlight region is a high-brightness region commonly contained in two adjacent highlight images, and the white region is a high-brightness region contained in the bright frame highlight image but not in the dark frame highlight image; wherein, the bright frame highlight image refers to the frame with a larger exposure compensation value of the corresponding first type image among two adjacent highlight images, and the dark frame highlight image refers to the frame with a smaller exposure compensation value of the corresponding first type image among two adjacent highlight images.

[0250] In one implementation of the image fusion device 300, the highlight image is a binary image, where pixels with a first value belong to a high-brightness region, and pixels with a second value do not belong to a high-brightness region. The real highlight image calculation module 340 calculates a real highlight image containing a highlight region based on every two adjacent highlight images, including: determining a connected region in the bright frame highlight image composed of pixels with the first value; determining the centroid position of the connected region in the dark frame highlight image composed of pixels with the first value; for each centroid position, if it is located within a connected region in the bright frame highlight image, then the region in the real highlight image corresponding to that connected region in the bright frame highlight image is determined as the highlight region.

[0251] In one implementation of the image fusion device 300, the highlight image is a binary image, where pixels with a first value belong to high-brightness regions, and pixels with a second value do not belong to high-brightness regions. The real highlight image calculation module 340 determines a real highlight image containing highlight regions based on every two adjacent highlight images, including: determining connected regions in the bright frame highlight image composed of pixels with the first value; determining connected regions in the dark frame highlight image composed of pixels with the first value; for each connected region in the dark frame highlight image, if there is a connected region in the bright frame highlight image whose overlap with it exceeds a first threshold, then the region in the real highlight image corresponding to that connected region in the bright frame highlight image is determined as a highlight region.

[0252] In one implementation of the image fusion device 300, the highlight image is a binary image, where pixels with a first value belong to high-brightness regions, and pixels with a second value do not belong to high-brightness regions. The real highlight image calculation module 340 calculates a real highlight image containing a white region based on every two adjacent highlight images, including: determining a connected region in the bright frame highlight image composed of pixels with the first value; determining the centroid position of the connected region in the dark frame highlight image composed of pixels with the first value; for each connected region in the bright frame highlight image, if any centroid position in the dark frame highlight image is located outside it, then the region in the real highlight image corresponding to the connected region in the bright frame highlight image is determined as a white region.

[0253] In one implementation of the image fusion device 300, the highlight image is a binary image, where pixels with a first value belong to high-brightness regions, and pixels with a second value do not belong to high-brightness regions. The real highlight image calculation module 340 calculates a real highlight image containing white regions based on every two adjacent highlight images, including: determining connected regions in the bright frame highlight image composed of pixels with the first value; determining connected regions in the dark frame highlight image composed of pixels with the first value; for each connected region in the bright frame highlight image, if the overlap between the connected region in the dark frame highlight image and the connected region does not exceed a second threshold, then the region in the real highlight image corresponding to the connected region in the bright frame highlight image is determined as a white region.

[0254] In one implementation of the image fusion device 300, each frame of the first type of weighted image is used to fuse two frames of images to be fused. The images to be fused are the first type of images, or the fused image obtained by weighted fusion of at least two frames of the first type of images. The pixel values ​​in the first type of weighted image represent the weight values ​​of the dark frame images to be fused during weighted fusion. The dark frame images to be fused are the frames with smaller exposure compensation values ​​among the two frames to be fused. The weighted image correction module 350 corrects the pixel values ​​in the first type of weighted image using the highlight area and / or white area in the real highlight image, including: increasing the pixel values ​​in the highlight area of ​​the first type of weighted image, and / or decreasing the pixel values ​​in the corresponding white area of ​​the first type of weighted image.

[0255] In one implementation of the image fusion device 300, the weighted image correction module 350 increases the pixel value located in the highlight region of the first type of weighted image, or decreases the pixel value located in the white region of the first type of weighted image, including: increasing the pixel value located in the highlight region of the first type of weighted image and decreasing the pixel value located outside the highlight region of the first type of weighted image, or decreasing the pixel value located in the white region of the first type of weighted image and increasing the pixel value located outside the white region of the first type of weighted image.

[0256] In one implementation of the image fusion device 300, the weighted image correction module 350 increases the pixel values ​​located in the highlight region of the first type of weighted image and decreases the pixel values ​​located in the white region of the first type of weighted image, including: increasing the pixel values ​​located in the highlight region of the first type of weighted image, decreasing the pixel values ​​located in the white region of the first type of weighted image, and maintaining the pixel values ​​located outside the highlight region and the white region of the first type of weighted image.

[0257] In one implementation of the image fusion device 300, the device further includes an image filtering module, which performs smoothing filtering on the real highlight image after the real highlight image calculation module 340 calculates a real highlight image containing highlight regions and / or white regions based on every two adjacent highlight images, and before the weighted image correction module 350 corrects the pixel values ​​in the corresponding first type of weighted image using the highlight regions and / or white regions in the real highlight image.

[0258] In one implementation of the image fusion device 300, the exposure compensation values ​​of the multiple frames of first-type images are all non-positive numbers. The first image fusion module 360 ​​uses at least the corrected first-type weighted image to perform weighted fusion on the multiple frames of first-type images to obtain the final fused image. This includes: taking the first-type image with the largest or smallest corresponding exposure compensation value as the current frame of first-type images and the initial fused image, and sequentially using the corrected first-type weighted image of the current frame to perform weighted fusion on the current fused image and the next frame of first-type images in the order of the gradual change of the corresponding exposure compensation values ​​to obtain a new fused image, until the next frame of first-type images is the last frame of first-type images, and obtaining the final fused image; wherein, the corrected first-type weighted image of the current frame refers to the corrected first-type weighted image between the current frame of first-type images and the next frame of first-type images.

[0259] In one implementation of the image fusion device 300, at least one of the multiple first-class images has a positive exposure compensation value. The first image fusion module 360 ​​uses at least the corrected first-class weighted image to perform weighted fusion on the multiple first-class images to obtain the final fused image. This includes: using the following two types of images to perform weighted fusion on the multiple first-class images to obtain the final fused image: the first-class weighted image between every two adjacent first-class images where at least one of the exposure compensation values ​​is positive; and the corrected first-class weighted image between every two adjacent first-class images where both of the exposure compensation values ​​are non-positive.

[0260] In one implementation of the image fusion device 300, the exposure compensation value of one frame in the multi-frame first-type images is 0.

[0261] The image fusion apparatus 300 provided in this application embodiment has been described in the foregoing method embodiment in terms of its implementation principle and the resulting technical effects. For the sake of brevity, any parts not mentioned in the apparatus embodiment can be referred to the corresponding content in the method embodiment.

[0262] Figure 5 A functional block diagram of an image fusion apparatus 400 provided in an embodiment of this application is shown. (Refer to...) Figure 5 The image fusion device 400 includes:

[0263] The second image acquisition module 410 is used to acquire multiple frames of first-class images, the exposure compensation values ​​of each frame of first-class images, and the exposure compensation values ​​of at least two frames of first-class images are non-positive numbers.

[0264] The second image fusion module 420 is used to take the first-type image frame with the largest or smallest exposure compensation value as the current first-type image and the initial fusion image, and sequentially perform the following fusion steps according to the gradual change of the corresponding exposure compensation values ​​to obtain a new fusion image, until the next first-type image is the last first-type image, to obtain the final fusion image. The fusion steps include: calculating a second-type weight image between the current fusion image and the next first-type image, where the pixel values ​​in the second-type weight image represent the weight values ​​of the images during weighted fusion; if at least one of the exposure compensation values ​​of the current first-type image and the next first-type image is positive, then the second-type weight image is used to perform weighted fusion of the current fusion image and the next first-type image to obtain a new fusion image; if the current first-type image has the largest or smallest exposure compensation value, then the next first-type image is weighted and fused using the second-type weight image to obtain a new fusion image; if the current first-type image has the largest or smallest exposure compensation value, then the next first-type image is weighted and fused using the second-type weight image to obtain a new fusion image. If the exposure compensation values ​​of the first-class image and the next frame's first-class image are both non-positive, then the following operations are performed: Calculate the highlight images corresponding to the current fused image and the next frame's first-class image, respectively. The two obtained highlight images contain the high-brightness regions of the corresponding current fused image and the next frame's first-class image, respectively. Calculate a true highlight image containing highlight regions and / or white regions based on the two highlight images. Correct the pixel values ​​in the corresponding second-class weighted image using the highlight regions and / or white regions in the true highlight image to obtain a corrected second-class weighted image. Different correction strategies are applied to the pixel values ​​located in the highlight regions and white regions in the second-class weighted image. Use the corrected second-class weighted image to perform weighted fusion of the current fused image and the next frame's first-class image to obtain a new fused image.

[0265] The image fusion device 400 provided in this application embodiment has been described in the foregoing method embodiment in terms of its implementation principle and the resulting technical effects. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the method embodiment.

[0266] Figure 6 This illustration shows a possible structure of the electronic device 500 provided in an embodiment of this application. (Refer to...) Figure 6 The electronic device 500 includes a processor 510, a memory 520, and a communication interface 530. These components are interconnected and communicate with each other via a communication bus 540 and / or other forms of connection mechanism (not shown).

[0267] The memory 520 includes one or more (only one is shown in the figure), which may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The processor 510 and other possible components may access the memory 520 to read and / or write data therein.

[0268] Processor 510 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 510 can be a general-purpose processor, including a Central Processing Unit (CPU), a Microcontroller Unit (MCU), a Network Processor (NP), or other conventional processors; it can also be a special-purpose processor, including a Neural-network Processing Unit (NPU), a Graphics Processing Unit (GPU), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Furthermore, when there are multiple processors 510, some can be general-purpose processors and others can be special-purpose processors.

[0269] Communication interface 530 includes one or more (only one is shown in the figure) that can be used to communicate directly or indirectly with other devices to exchange data. Communication interface 530 may include interfaces for wired and / or wireless communication.

[0270] One or more computer program instructions may be stored in the memory 520, and the processor 510 may read and run these computer program instructions to implement the image fusion method provided in the embodiments of this application.

[0271] Understandable. Figure 6 The structure shown is for illustrative purposes only; the electronic device 500 may also include more than [other components]. Figure 6 The more or fewer components shown, or having the same Figure 6 The different configurations shown. Figure 6 The components shown can be implemented using hardware, software, or a combination thereof. Electronic device 500 may be a physical device, such as a mobile phone, camera, camcorder, wearable device, tablet computer, PC, laptop computer, server, etc., or it may be a virtual device, such as a virtual machine, virtualization container, etc. Furthermore, electronic device 500 is not limited to a single device; it can also be a combination of multiple devices or a cluster of a large number of devices.

[0272] This application also provides a computer-readable storage medium storing computer program instructions. These instructions are read and executed by a computer processor to perform the ghost detection method and / or image fusion method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 6 The memory 520 in the electronic device 500.

[0273] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image fusion method, characterized in that, include: Acquire multiple frames of Class I images, each frame of Class I images having a different exposure compensation value, and at least two of the Class I images having a non-positive exposure compensation value; Calculate the first type weight image between every two adjacent first type images, where the pixel values ​​in the first type weight image represent the weight values ​​of the images during weighted fusion. Calculate the highlight image corresponding to each frame of the first type of image with a non-positive exposure compensation value, wherein the highlight image contains the high-brightness area in the corresponding first type of image; A true highlight image containing highlight regions and / or white regions is calculated based on every two adjacent highlight images; wherein, the highlight region is the high-brightness region commonly contained in two adjacent highlight images, and the white region is the high-brightness region contained in the bright frame highlight image but not in the dark frame highlight image; the bright frame highlight image refers to the frame with the larger exposure compensation value of the corresponding first type image among two adjacent highlight images, and the dark frame highlight image refers to the frame with the smaller exposure compensation value of the corresponding first type image among two adjacent highlight images; The pixel values ​​in the corresponding first-class weighted image are corrected by using the highlight area and / or white area in the real highlight image to obtain the corrected first-class weighted image; wherein, different correction strategies are adopted for the pixel values ​​located in the highlight area and the white area in the first-class weighted image; The first-class images of the multiple frames are weighted and fused using at least the modified first-class weighted images to obtain the final fused image.

2. The image fusion method according to claim 1, characterized in that, The highlight image is a binary image. Pixels with the first value in the image belong to the high brightness region, while pixels with the second value do not belong to the high brightness region. Calculate a true specular image containing the specular region based on every two adjacent specular images, including: Determine the connected region in the bright frame highlight image composed of pixels that take the first value; Determine the centroid position of the connected region formed by pixels taking the first value in the dark frame highlight image; For each centroid location, if it is located within a connected region in the bright frame specular image, then the region in the real specular image corresponding to that connected region in the bright frame specular image is determined as the specular region.

3. The image fusion method according to claim 1, characterized in that, The highlight image is a binary image. Pixels with the first value in the image belong to the high brightness region, while pixels with the second value do not belong to the high brightness region. Determine a true highlight image containing the highlight region based on every two adjacent highlight images, including: Determine the connected region in the bright frame highlight image composed of pixels that take the first value; Determine the connected region in the dark frame highlight image composed of pixels that take the first value; For each connected region in the dark frame highlight image, if there is a connected region in the bright frame highlight image whose overlap with it exceeds a first threshold, then the region in the real highlight image corresponding to that connected region in the bright frame highlight image is determined as the highlight region.

4. The image fusion method according to any one of claims 1-3, characterized in that, The highlight image is a binary image. Pixels with the first value in the image belong to the high brightness region, while pixels with the second value do not belong to the high brightness region. Calculate a true highlight image containing white areas based on every two adjacent highlight images, including: Determine the connected region in the bright frame highlight image composed of pixels that take the first value; Determine the centroid position of the connected region formed by pixels taking the first value in the dark frame highlight image; For each connected region in the bright frame highlight image, if any centroid position in the dark frame highlight image is located outside of it, then the region in the real highlight image corresponding to that connected region in the bright frame highlight image is determined as a white region.

5. The image fusion method according to any one of claims 1-3, characterized in that, The highlight image is a binary image. Pixels with the first value in the image belong to the high brightness region, while pixels with the second value do not belong to the high brightness region. Calculate a true highlight image containing white areas based on every two adjacent highlight images, including: Determine the connected region in the bright frame highlight image composed of pixels that take the first value; Determine the connected region in the dark frame highlight image composed of pixels that take the first value; For each connected region in the bright frame highlight image, if the overlap between it and any connected region in the dark frame highlight image does not exceed the second threshold, then the region in the real highlight image corresponding to that connected region in the bright frame highlight image is determined as a white region.

6. The image fusion method according to any one of claims 1-3, characterized in that, Each frame of the first type of weighted image is used to fuse two frames of images to be fused. The images to be fused are the first type of images, or the fused image obtained by weighted fusion of at least two frames of the first type of images. The pixel values ​​in the first type of weighted image represent the weight values ​​of the dark frame images to be fused when performing weighted fusion. The dark frame images to be fused are the frames with smaller exposure compensation values ​​among the two images to be fused. The step of correcting the pixel values ​​in the corresponding first-class weighted image using the highlight region and / or white region in the real highlight image includes: Increase the pixel values ​​located in the highlight region of the first type of weighted image, and / or decrease the pixel values ​​located in the white region of the first type of weighted image.

7. The image fusion method according to claim 6, characterized in that, Increasing the pixel values ​​located in the highlight regions of the first type of weighted image, or decreasing the pixel values ​​located in the white regions of the first type of weighted image, includes: Increase the pixel values ​​located in the highlight region of the first type of weighted image and decrease the pixel values ​​located outside the highlight region of the first type of weighted image, or decrease the pixel values ​​located in the white region of the first type of weighted image and increase the pixel values ​​located outside the white region of the first type of weighted image.

8. The image fusion method according to claim 6, characterized in that, Increasing the pixel values ​​located in the highlight region of the first type of weighted image, and decreasing the pixel values ​​located in the white region of the first type of weighted image, includes: Increase the pixel values ​​located in the highlight region of the first type of weighted image, and decrease the pixel values ​​located in the white region of the first type of weighted image, and maintain the pixel values ​​located outside the highlight region and the white region of the first type of weighted image.

9. The image fusion method according to any one of claims 1-3, characterized in that, The exposure compensation values ​​of the multiple frames of the first type of images are all non-positive numbers. The step of weighted fusion of the multiple frames of the first type of images using at least the corrected first type of weighted image to obtain the final fused image includes: The first-class image with the largest or smallest exposure compensation value is used as the current first-class image and the initial fused image. According to the gradual change of the corresponding exposure compensation value, the current fused image and the next first-class image are weighted and fused sequentially using the corrected first-class weight image of the current frame to obtain a new fused image. The final fused image is obtained when the next first-class image is the last first-class image. The corrected first-class weight image of the current frame refers to the corrected first-class weight image between the first-class image of the current frame and the first-class image of the next frame.

10. The image fusion method according to any one of claims 1-3, characterized in that, At least one of the multiple first-class images has a positive exposure compensation value, and the step of weighted fusion of the multiple first-class images using at least the corrected first-class weighted image to obtain the final fused image includes: The first-class images of the multiple frames are weighted and fused using the following two types of images to obtain the final fused image: The first type weighted image between every two frames of the first type of images that are adjacent and at least one of them has a positive exposure compensation value; The corrected first-class weighted image between every two adjacent first-class images where the exposure compensation values ​​are all non-positive.

11. The image fusion method according to any one of claims 1-3, characterized in that, The exposure compensation value of one frame in the multi-frame first-class image is 0.

12. An image fusion method, characterized in that, include: Acquire multiple frames of Class I images, each frame of Class I images having a different exposure compensation value, and at least two of the Class I images having a non-positive exposure compensation value; The frame with the largest or smallest exposure compensation value is used as the current frame of the first type image and the initial fused image. Following the gradual change in exposure compensation values, the following fusion steps are executed sequentially to obtain a new fused image, until the next frame of the first type image is the last frame of the first type image, at which point the final fused image is obtained. The fusion steps include: Calculate the second type weight image between the current fused image and the first type image of the next frame. The pixel values ​​in the second type weight image represent the weight values ​​of the images when performing weighted fusion. If at least one of the exposure compensation values ​​of the current frame's first-class image and the next frame's first-class image is positive, then the second-class weighted image is used to perform weighted fusion of the current fused image and the next frame's first-class image to obtain a new fused image. If the exposure compensation values ​​for both the current frame and the next frame of the first type of image are non-positive, then perform the following operations: Calculate the highlight images corresponding to the current fused image and the next frame of the first type image respectively. The two obtained highlight images contain the high brightness areas in the current fused image and the next frame of the first type image respectively. Calculate a true highlight image containing highlight regions and / or white regions based on the two highlight images. The highlight regions are the high-brightness regions shared by the two adjacent highlight images, and the white regions are the high-brightness regions contained in the bright highlight image but not in the dark highlight image. The bright highlight image refers to the frame with the larger exposure compensation value of the corresponding first type of image among the two adjacent highlight images, and the dark highlight image refers to the frame with the smaller exposure compensation value of the corresponding first type of image among the two adjacent highlight images. The pixel values ​​in the corresponding second-type weighted image are corrected by using the highlight area and / or white area in the real highlight image to obtain the corrected second-type weighted image; wherein, different correction strategies are adopted for the pixel values ​​located in the highlight area and the white area in the second-type weighted image; The current fused image and the next frame's first-class image are weighted and fused using the corrected second-class weighted image to obtain a new fused image.

13. An image fusion apparatus, characterized in that, include: The first image acquisition module is used to acquire multiple frames of first-class images, each frame of the first-class images having a different exposure compensation value, and at least two of the first-class images having a non-positive exposure compensation value. The weighted image calculation module is used to calculate the first type of weighted image between every two adjacent first type images. The pixel values ​​in the first type of weighted image represent the weight values ​​of the images when performing weighted fusion. The highlight image calculation module is used to calculate the highlight image corresponding to each frame of the first type of image with a non-positive exposure compensation value, wherein the highlight image contains the high brightness area in the corresponding first type of image. The true highlight image calculation module is used to calculate a true highlight image containing highlight regions and / or white regions based on every two adjacent highlight images. The highlight regions are the high-brightness regions shared by the two adjacent highlight images, and the white regions are high-brightness regions contained in the bright frame highlight image but not in the dark frame highlight image. The bright frame highlight image refers to the frame with the larger exposure compensation value of the corresponding first-type image among two adjacent highlight images, and the dark frame highlight image refers to the frame with the smaller exposure compensation value of the corresponding first-type image among two adjacent highlight images. The weighted image correction module is used to correct the pixel values ​​in the corresponding first type of weighted image using the highlight area and / or white area in the real highlight image to obtain the corrected first type of weighted image; wherein, different correction strategies are adopted for the pixel values ​​located in the corresponding highlight area and the corresponding white area in the first type of weighted image. The first image fusion module is used to perform weighted fusion of the multiple frames of first-class images using at least the corrected first-class weighted images to obtain the final fused image.

14. An image fusion apparatus, characterized in that, include: The second image acquisition module is used to acquire multiple frames of first-class images, each frame of the first-class images having a different exposure compensation value, and at least two of the first-class images having a non-positive exposure compensation value. The second image fusion module is used to take the frame of the first type image with the largest or smallest exposure compensation value as the current frame of the first type image and the initial fusion image, and sequentially perform the following fusion steps according to the gradual change of the corresponding exposure compensation values ​​to obtain a new fusion image, until the next frame of the first type image is the last frame of the first type image, to obtain the final fusion image. The fusion steps include: Calculate the second type weight image between the current fused image and the first type image of the next frame. The pixel values ​​in the second type weight image represent the weight values ​​of the images when performing weighted fusion. If at least one of the exposure compensation values ​​of the current frame's first-class image and the next frame's first-class image is positive, then the second-class weighted image is used to perform weighted fusion of the current fused image and the next frame's first-class image to obtain a new fused image. If the exposure compensation values ​​for both the current frame and the next frame of the first type of image are non-positive, then perform the following operations: Calculate the highlight images corresponding to the current fused image and the next frame of the first type image respectively. The two obtained highlight images contain the high brightness areas in the current fused image and the next frame of the first type image respectively. Calculate a true highlight image containing highlight regions and / or white regions based on the two highlight images. The highlight regions are the high-brightness regions shared by the two adjacent highlight images, and the white regions are the high-brightness regions contained in the bright highlight image but not in the dark highlight image. The bright highlight image refers to the frame with the larger exposure compensation value of the corresponding first type of image among the two adjacent highlight images, and the dark highlight image refers to the frame with the smaller exposure compensation value of the corresponding first type of image among the two adjacent highlight images. The pixel values ​​in the corresponding second-type weighted image are corrected by using the highlight area and / or white area in the real highlight image to obtain the corrected second-type weighted image; wherein, different correction strategies are adopted for the pixel values ​​located in the highlight area and the white area in the second-type weighted image; The current fused image and the next frame's first-class image are weighted and fused using the corrected second-class weighted image to obtain a new fused image.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when read and executed by a processor, perform the method as described in any one of claims 1-12.

16. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer program instructions, which are read and executed by the processor to perform the method according to any one of claims 1-12.

Images