A ghost-removing high dynamic image generation method, device and storage medium

Abstract

The application provides a high dynamic image generation method and device for removing ghosting and a storage medium. The method comprises the following steps: performing histogram matching on a first underexposed frame and a first overexposed frame respectively with a first normal frame to obtain a difference area; performing ghosting area detection on the difference area to obtain a motion area; and selecting a reference frame from the first normal frame, the first underexposed frame and the first overexposed frame based on information maximization, and compensating the motion area. The application adopts MIE-based histogram matching, can be deployed on a low-end platform, can avoid detail loss caused by matching of images with more overexposure or more underexposure, and adaptively selects a reference frame to compensate the ghosting area, so that the ghosting removal effect is better.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus and storage medium for generating high dynamic range images with ghosting removal. Background Technology

[0002] With the development of computer and multimedia technologies, people's demands for digital image quality are increasing. Research on high dynamic range (HDR) imaging technology can greatly promote the development of digital image technology towards greater information content and better visual effects. Currently, HDR imaging technology has wide applications in consumer electronics, remote sensing, security monitoring, and digital television. General image generation and display devices have very small dynamic ranges, which cannot meet the requirements of HDR imaging technology. High-end HDR image generation and display devices are also expensive, making them unsuitable for widespread use in consumer electronics and consumer products. Currently, a more economical solution is to capture multiple frames of low dynamic range images of the same scene with different exposures. These images contain different dynamic ranges from the real scene. Image processing algorithms then combine these multiple frames into a single high dynamic range image, allowing it to be displayed completely on general display devices. However, this method is only applicable to multi-frame images captured in static scenes. In real-world scenes, images of the same scene with different exposures are taken at different times, inevitably including moving objects such as pedestrians and vehicles. Directly fusing these images will result in the superposition of moving pixels in the final image, manifesting as ghosting and greatly affecting image quality.

[0003] For ghosting removal, a common approach is the inter-frame difference method. This involves first matching the brightness of two images to be as consistent as possible, with histogram matching being the most common method. Then, the difference is calculated, and ghosting regions are identified by comparing the differences in grayscale values ​​between the two images. This method requires a high degree of accuracy in matching the brightness of the two images. Histogram matching can alter the dynamic range of the image, resulting in the loss of image details, and pixel-by-pixel comparisons can introduce significant noise. For example, CN110619652B discloses an image registration ghosting removal method based on optical flow mapping and duplicate region detection. It first uses optical flow registration to identify suspected ghosting regions, and then sets a threshold based on the pixel value differences between non-reference frames and reference frames to obtain the ghosting regions. Essentially, this is also an inter-frame difference method.

[0004] Another method is the gradient method, but comparing gradients pixel by pixel can introduce significant noise. Another approach is optical flow, which identifies the motion region by finding the optical flow between two images. However, optical flow methods also have high brightness requirements, and dense optical flow is computationally expensive and unsuitable for real-time applications, while sparse optical flow may fail to completely locate the motion region. In recent years, many deep learning-based methods for generating high dynamic range images with ghosting removal have emerged and achieved good results, such as the high dynamic range imaging and ghosting removal method based on generative adversarial networks proposed in CN115018733A. However, deep learning-based methods have high computational complexity and demanding hardware requirements for deployment, making them unsuitable for low-performance platforms. Summary of the Invention

[0005] The purpose of this invention is to provide a method, apparatus and storage medium for generating high dynamic range images with ghosting removal, which can be deployed on low-end platforms and has a better ghosting removal effect.

[0006] This invention provides a high dynamic range image generation method for removing ghosting, comprising: acquiring an original image, the original image including a first normal frame, a first underexposed frame, and a first overexposed frame, wherein the first normal frame has a different exposure level than the first underexposed frame and the first overexposed frame; performing histogram matching between the first underexposed frame and the first overexposed frame and the first normal frame respectively to obtain difference regions; performing ghosting region detection on the difference regions to obtain motion regions; and selecting a reference frame from the first normal frame, the first underexposed frame, and the first overexposed frame based on maximizing information content to compensate for the motion regions.

[0007] Further, the step of performing histogram matching between the first underexposed frame and the first overexposed frame and the first normal frame respectively to obtain the difference region includes: performing MIE histogram matching between the first underexposed frame and the first normal frame to obtain the second underexposed frame and the second normal frame; performing MIE histogram matching between the first normal frame and the first overexposed frame to obtain the third normal frame and the second overexposed frame; performing structure-based SSIM ghost detection on the second underexposed frame and the second normal frame to obtain the first mask image of the ghost region; and performing structure-based SSIM ghost detection on the third normal frame and the second overexposed frame to obtain the second mask image of the ghost region.

[0008] Furthermore, the step of performing ghost region detection on the difference region to obtain the motion region includes: performing a union of the first mask image and the second mask image to obtain a third mask image of the merged ghost region; processing the third mask image of the ghost region to remove noise regions with a noise threshold less than the first threshold to obtain a fourth mask image of the ghost region.

[0009] Furthermore, the step of selecting compensation frames for ghosting regions based on maximizing information content and using different weights for fusion of different ghosting compensation frames includes: selecting reference frames for ghosting region compensation from the first underexposed frame, the first normal frame, and the first overexposed frame based on maximizing information content; and performing region compensation on the fourth mask image according to the reference frame to obtain repaired video frames.

[0010] Furthermore, the step of performing MIE histogram matching between the first underexposed frame and the first normal frame to obtain the second underexposed frame and the second normal frame includes: calculating the cumulative histogram of the first underexposed frame and the cumulative histogram of the first normal frame; calculating the median histogram of the first underexposed frame and the first normal frame based on the cumulative histogram of the first underexposed frame and the cumulative histogram of the first normal frame, wherein the median histogram is the harmonic mean of the cumulative histogram of the first underexposed frame and the cumulative histogram of the first normal frame; performing histogram matching between the first underexposed frame and the first normal frame to obtain the second underexposed frame and the second normal frame, wherein the second underexposed frame and the second normal frame have the same median histogram distribution.

[0011] Furthermore, the step of performing structure-based SSIM ghost detection on the second underexposed frame and the second normal frame to obtain a first mask image of the ghost region includes: obtaining the SSIM values ​​of a first pixel in the second underexposed frame and a second pixel in the second normal frame, wherein the second pixel corresponds to the first pixel; comparing the SSIM values ​​with a second threshold to obtain a comparison result; and obtaining the first mask image based on the comparison result.

[0012] Furthermore, the step of obtaining the SSIM value of the first pixel of the second underexposed frame and the second pixel of the second normal frame includes: obtaining the SSIM value based on the structure of the first pixel of the second underexposed frame and the second pixel of the second normal frame.

[0013] Furthermore, the step of selecting a reference frame based on maximizing information content and compensating for the motion region includes one of the following methods: calculating the pyramid fusion weights of the first underexposed frame, the first normal frame, and the first overexposed frame respectively, and taking the frame with the highest pyramid fusion weight in the fourth ghost region as the reference frame; calculating the pyramid fusion weights of the first underexposed frame, the first normal frame, and the first overexposed frame respectively, counting the number of pixels in the fourth ghost region whose fusion weight is less than a third threshold, and taking the frame with the fewest pixels as the reference frame for compensating for the ghost region, wherein pixels with a fusion weight less than the third threshold represent pixels with very little information content, and the information content is less than a certain threshold; counting the number of pixels in the first underexposed frame, the first normal frame, and the first overexposed frame in the fourth ghost region whose pixel values ​​are less than the fourth threshold or greater than the fifth threshold respectively, and taking the frame with the fewest pixels as the reference frame for compensating for the ghost region, wherein pixel values ​​less than the fourth threshold represent extremely dark pixel values, and pixel values ​​greater than the fifth threshold represent overexposed pixel values.

[0014] Further, the step of compensating the motion region includes: calculating the pyramid fusion weight images W1, W2, and W3 for the first underexposed frame, the first normal frame, and the first overexposed frame respectively, and obtaining the pyramid fusion weight m4 of the fourth mask image based on W1, W2, and W3; if the reference frame for ghost region compensation is the first underexposed frame, then: brighten the first underexposed frame to obtain the third underexposed frame; obtain the fusion weight W14 of the first underexposed frame based on W1 and m4; obtain the fusion weight W15 of the first normal frame based on W2 and m4; obtain the fusion weight W16 of the first overexposed frame based on W3 and m4; obtain the fusion weight W17 of the third underexposed frame based on m4; if ghost region compensation... If the reference frame is the first overexposed frame, then: perform histogram matching from the first overexposed frame to the first normal frame to obtain the third overexposed frame; obtain the fusion weight W24 of the first underexposed frame according to W1 and m4; obtain the fusion weight W25 of the first normal frame according to W2 and m4; obtain the fusion weight W26 of the first overexposed frame according to W3 and m4; obtain the fusion weight W27 of the third overexposed frame according to m4; if the reference frame for ghosting region compensation is the first normal frame: obtain the fusion weight W34 of the first underexposed frame according to W1 and m4; obtain the fusion weight W35 of the first normal frame according to W2; obtain the fusion weight W36 of the first overexposed frame according to W3 and m4.

[0015] Furthermore, W14 = W1.*(1-m4), W15 = W2.*(1-m4), W16 = W3.*(1-m4), W17 = m4; W24 = W1.*(1-m4), W25 = W2.*(1-m4), W26 = W3.*(1-m4), W27 = m4; W34 = W1.*(1-m4), W35 = W2, W36 = W3.*(1-m4).

[0016] The present invention also provides a high dynamic range image generation apparatus for removing ghosting, comprising: an original image acquisition module for acquiring an original image, the original image including a first normal frame, a first underexposed frame, and a first overexposed frame, wherein the first normal frame has a different exposure level than the first underexposed frame and the first overexposed frame; a difference region acquisition module for performing histogram matching between the first underexposed frame and the first overexposed frame and the first normal frame respectively to obtain difference regions; a motion region acquisition module for performing ghosting region detection on the difference regions to obtain motion regions; and a ghosting region compensation module for selecting reference frames from the first normal frame, the first underexposed frame, and the first overexposed frame to compensate for the motion regions based on maximizing information content, and using different weights for fusion for different reference frames.

[0017] The present invention further provides a computer-readable storage medium comprising computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in the first aspect and any possible design thereof.

[0018] This invention employs MIE-based histogram matching, which can be deployed on low-end platforms and avoids the loss of detail caused by matching images that are significantly overexposed or underexposed. At the same time, it adaptively selects reference frames to compensate for ghosting regions, resulting in better ghosting removal. Attached Figure Description

[0019] The preferred embodiments will now be described in a clear and easy-to-understand manner, in conjunction with the accompanying drawings, to further explain the above-mentioned characteristics, technical features, advantages, and implementation methods of the present invention.

[0020] Figure 1 This is a flowchart of a high dynamic range image generation method for removing ghosting according to an example embodiment of this application;

[0021] Figure 2 yes Figure 1 A detailed flowchart of step S40 in the process;

[0022] Figure 3 yes Figure 2 A detailed flowchart of step S44 in the process;

[0023] Figure 4 yes Figure 1 A detailed flowchart of step S60 in the process;

[0024] Figure 5 yes Figure 1 A detailed flowchart of step S80 in the process;

[0025] Figure 6 yes Figure 1 Another detailed flowchart of step S80 in the process;

[0026] Figure 7 This is a block diagram of a high dynamic range image generation apparatus for removing ghosting according to another exemplary embodiment of this application;

[0027] Figure 8 yes Figure 7 A schematic diagram of the working process. Detailed Implementation

[0028] The technical solutions of this application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to limit the application.

[0029] This invention can remove ghosting from N (N ≥ 3) frames of multi-exposure images, where one frame is a reference frame and the rest are non-reference frames. The overall process is as follows: MIE histogram matching is performed on the non-reference and reference frames respectively. Then, ghosting regions are detected by structural similarity (SSIM). The union of all ghosting regions is taken to obtain the final ghosting region. Then, the frame with the most information in the ghosting region is selected as the compensation frame for the ghosting region. Different compensation frames are selected with different weights for fusion and ghosting removal.

[0030] Figure 1 A flowchart illustrating a high dynamic range image generation method for removing ghosting according to an example embodiment of this application is shown. Figure 1 As shown, the high dynamic range image generation method for removing ghosting includes the following steps:

[0031] S20. Acquire the original image, which includes a first normal frame, a first underexposed frame, and a first overexposed frame, wherein the exposure level of the first normal frame is different from that of the first underexposed frame and the first overexposed frame.

[0032] S40. Perform histogram matching between the first underexposed frame and the first overexposed frame and the first normal frame respectively, so that the image can maintain the original brightness dynamic range as much as possible and find the difference area as much as possible; when the obtained original image includes multiple underexposed frames and multiple overexposed frames, the processing method is the same.

[0033] S60. Perform ghost region detection on the difference region to obtain a more complete motion region;

[0034] S80. From the first normal frame, the first underexposed frame, and the first overexposed frame, based on maximizing information content, an adaptive reference frame is selected to compensate the motion region, so that the compensation for ghosting is more natural.

[0035] Existing technologies often use underexposed frames for ghosting region compensation, i.e., for the aforementioned reference frame selection. This is because underexposed frames have shorter shooting times and less motion blur. However, underexposed frames are darker, which may result in some loss of dark area information, and color restoration after brightening is also difficult, making it hard to restore a natural visual effect during motion compensation. Alternatively, a specific frame can be used as the reference frame for compensation. In this case, if the reference frame is underexposed or overexposed, information loss will also occur, and the compensated ghosting region will be extremely abrupt. This embodiment adaptively selects the reference frame for ghosting region compensation based on maximizing information content, resulting in less information loss and a more natural ghosting compensation. Furthermore, in this embodiment, different weights are used for fusion of different reference frames to compensate for ghosting regions, resulting in better ghosting removal.

[0036] In one implementation, see [reference needed]. Figure 2 Step S40 above, namely the step of performing histogram matching between the first underexposed frame and the first overexposed frame and the first normal frame respectively to obtain the difference region, includes:

[0037] Step S42: Perform median image equalization (MIE) histogram matching on the first underexposed frame and the first normal frame to obtain the second underexposed frame and the second normal frame; perform MIE histogram matching on the first normal frame and the first overexposed frame to obtain the third normal frame and the second overexposed frame.

[0038] Specifically, firstly, the median histogram of the two input images, the first underexposed frame and the first normal frame, is calculated. The median histogram is the harmonic mean of the histograms of the two input images, i.e.:

[0039]

[0040] Among them, H midway H1 and H2 are the median histograms, respectively, representing the cumulative histograms of the two images.

[0041] Histogram matching is performed on the input image, and the two images matched using the MIE algorithm are as follows:

[0042]

[0043]

[0044] Where u1 and u2 are two input images, and U1 and U2 represent the images matched using the MIE algorithm. U1 and U2 have the same histogram distribution H. midway .

[0045] Step S44: Perform structure-based SSIM ghost detection on the second underexposed frame and the second normal frame to obtain the first mask image of the ghost region; perform structure-based SSIM ghost detection on the third normal frame and the second overexposed frame to obtain the second mask image of the ghost region.

[0046] For more details, please refer to Figure 3 The step of performing structure-based SSIM ghost detection on the second underexposed frame and the second normal frame to obtain a first mask image of the ghost region includes:

[0047] Step S442: Obtain the SSIM values ​​of the first pixel of the second underexposed frame and the second pixel of the second normal frame, wherein the second pixel corresponds to the first pixel.

[0048] Step S444: Compare the SSIM value with the second threshold to obtain the comparison result;

[0049] Step S446: Based on the comparison result, obtain the first mask image.

[0050] Specifically, the second threshold can be, for example, 0.8. Traverse the SSIM image. If the SSIM value is greater than or equal to the threshold 1, it means that the pixel similarity at that pixel point is high. Conversely, if the SSIM value is less than the threshold 1, it means that the pixel similarity at that pixel point is not high. Set the pixels with SSIM values ​​less than the threshold 1 to 1 and the rest to 0 to obtain the first ghost region.

[0051] The steps for performing structure-based SSIM ghost detection on the third normal frame and the second overexposed frame to obtain the second mask image of the ghost region are the same as the above process and will not be repeated.

[0052] More specifically, in step S442, the step of obtaining the SSIM values ​​of the first pixel of the second underexposed frame and the second pixel of the second normal frame includes: obtaining the SSIM value based on the structure of the first pixel of the second underexposed frame and the second pixel of the second normal frame. The original SSIM mainly considers the brightness, contrast, and structure of the two images. This invention simplifies this by only considering the structure. The formula for calculating SSIM in this example is:

[0053]

[0054] Where X / Y represent two images, here representing the second underexposed frame and the second normal frame, respectively, σ X σ represents the standard deviation of pixels in image X within a pxp window at a given pixel point. Y Similarly; σ XY This represents the covariance of X and Y within the pxp window at a given pixel; C is a constant.

[0055] The applicant of this application noted that the dynamic range displayed by traditional displays is limited, differing significantly from the dynamic range perceived by the human eye. Images often lose information in underexposed and overexposed areas. High dynamic range (HDR) images expand the dynamic range and image detail covered by a single image by fusing a series of low dynamic range (LMR) images with different exposures. Existing HDR technologies typically fuse LMR images with different exposures according to different weights, extracting the better-performing parts of each to synthesize a larger dynamic range. However, in actual image capture, there are often moving objects in the scene, such as pedestrians and vehicles. Simple image fusion can result in the superposition of moving pixels in the final image, manifesting as ghosting. Existing deep learning-based methods have good ghosting removal effects, but require a large dataset that closely matches real-world applications, and deep learning-based methods are difficult to deploy on low-performance platforms. Therefore, in this embodiment of the application, a histogram matching based on MIE (Multi-Image Rendering) is employed, which can be deployed on low-end platforms and avoids the loss of detail caused by matching images with significant overexposure or underexposure.

[0056] Furthermore, existing technologies, such as the ghost-free high dynamic range imaging method based on gradient structural similarity disclosed in CN108492262B, perform SSIM on the gradient. This SSIM considers a combination of three different factors: brightness, contrast, and structure, which increases computational complexity to some extent, and the gradient has a significant impact on noise. This embodiment performs SSIM detection based on the structure of the second underexposed frame and the second normal frame, considering only the structure factor among the aforementioned three factors (brightness, contrast, and structure). More precisely, this embodiment only considers the structural similarity of pixels, which can avoid the influence of brightness and contrast to some extent and can more completely detect moving regions.

[0057] As mentioned above, in this embodiment, please refer to... Figure 4 In the aforementioned step S60, ghost region detection is performed on the difference region to obtain the motion region, which can be based on the first mask:

[0058] Step S62: Perform a union of the first mask image and the second mask image to obtain the third mask image of the merged ghost region;

[0059] Step S64: Process the third mask image of the ghost region, such as morphological operations and connected component processing, to remove noise regions with noise thresholds less than the first threshold, and obtain the fourth mask image of the ghost region.

[0060] The first mask image, the second mask image, and the third mask image can all be binary images of 0 and 1, with ghost regions being 1 and static regions being 0.

[0061] Accordingly, in this embodiment, please refer to Figure 5 In the aforementioned step S80, the step of adaptively selecting compensation frames for ghosting regions based on maximizing information content, and using different weights for fusion of different ghosting compensation frames, includes:

[0062] Step S82: From the first underexposed frame, the first normal frame, and the first overexposed frame, adaptively select a reference frame for ghosting region compensation based on maximizing information content.

[0063] Specifically, the step of selecting compensation frames for ghosting regions based on maximizing information content includes one of the following: a, b, c:

[0064] (a) Calculate the pyramid fusion weights of the first underexposed frame, the first normal frame, and the first overexposed frame respectively, and take the frame with the highest pyramid fusion weight in the fourth ghost region as the reference frame.

[0065] (b) Calculate the pyramid fusion weights of the first underexposed frame, the first normal frame, and the first overexposed frame respectively. Count the number of pixels in the fourth ghost region whose fusion weight is less than the third threshold. Take the frame with the fewest pixels as the reference frame for compensating the ghost region. Pixels whose fusion weight is less than the third threshold represent pixels with very little information. The third threshold can be, for example, 0.01.

[0066] (c) Count the number of pixels in the first underexposed frame, the first normal frame, and the first overexposed frame in the fourth ghost region whose pixel values ​​are less than the fourth threshold or greater than the fifth threshold. Take the frame with the fewest pixels as the reference frame for compensating the ghost region. Pixel values ​​less than the fourth threshold represent extremely dark pixel values, and pixel values ​​greater than the fifth threshold represent overexposed pixel values. These two types of pixels contain very little information. The fourth threshold can be, for example, 0.1, and the fifth threshold can be, for example, 0.9.

[0067] When using the three methods (a), (b), and (c) mentioned above, you can choose one of them depending on the situation.

[0068] Step S84: Based on the reference frame, perform region compensation on the fourth mask image to obtain the repaired video frame.

[0069] Existing technologies often use underexposed frames for ghosting compensation, i.e., for the aforementioned reference frame selection. This is because underexposed frames have shorter shooting times and less motion blur. However, underexposed frames are darker, which may result in some loss of dark area information, and color restoration after brightening is also difficult, making it hard to restore a natural visual effect during motion compensation. Alternatively, a specific frame can be used as the reference frame for compensation. In this case, if the reference frame is underexposed or overexposed, information loss will also occur, and the compensated ghosting area will be extremely abrupt. This implementation method adaptively selects the reference frame for ghosting compensation based on maximizing information content, resulting in less information loss and a more natural ghosting compensation.

[0070] Furthermore, in yet another embodiment, see [reference needed]. Figure 6 The step of compensating for the motion region specifically includes:

[0071] Step S86: Calculate the pyramid fusion weight images W1, W2, and W3 for the first underexposed frame, the first normal frame, and the first overexposed frame, respectively. W1, W2, and W3 are all values ​​from 0 to 1, representing the product of weights for contrast, saturation, and exposure. When W1, W2, and W3 are used for multi-frame pyramid fusion and ghosting compensation, and the reference frames are the first underexposed frame, the first overexposed frame, and the first normal frame, proceed to steps S862, S866, and S868 respectively.

[0072] Step S862: If the reference frame for ghosting region compensation is the first underexposed frame, then proceed with the following steps S861 and S863:

[0073] Step S861: Brighten the first underexposed frame to obtain the third underexposed frame. The brightening method can be gamma curve brightening, bezier curve brightening or other brightening methods.

[0074] Step S863: Based on W1 and m4, obtain the fusion weight W14 of the first underexposed frame; based on W2 and m4, obtain the fusion weight W15 of the first normal frame; based on W3 and m4, obtain the fusion weight W16 of the first overexposed frame; based on m4, obtain the fusion weight W17 of the third underexposed frame; specifically, fuse the four frames—the first underexposed frame, the first normal frame, the first overexposed frame, and the third underexposed frame—and fuse them with weights W14, W15, W16, and W17, respectively, where:

[0075] W14 = W1.*(1-m4);

[0076] W15 = W2 * (1 - m4);

[0077] W16 = W3.*(1-m4);

[0078] W17 = m4;

[0079] Step S866: If the reference frame for ghosting region compensation is the first overexposed frame, then:

[0080] Step S865: Perform histogram matching on the first overexposed frame and the first normal frame to obtain the third overexposed frame;

[0081] Step S867: Based on W1 and m4, obtain the fusion weight W24 of the first underexposed frame; based on W2 and m4, obtain the fusion weight W25 of the first normal frame; based on W3 and m4, obtain the fusion weight W26 of the first overexposed frame; based on m4, obtain the fusion weight W27 of the third overexposed frame; fuse the four frames—the first underexposed frame, the first normal frame, the first overexposed frame, and the third overexposed frame—with the fusion weights W24, W25, W26, and W27 respectively, where:

[0082] W24 = W1.*(1-m4);

[0083] W25 = W2 * (1 - m4);

[0084] W26 = W3.*(1-m4);

[0085] W27 = m4;

[0086] Step S868: If the reference frame for ghost region compensation is the first normal frame:

[0087] Step S869: Based on W1 and m4, obtain the fusion weight W34 of the first underexposed frame; based on W2, obtain the fusion weight W35 of the first normal frame; based on W3 and m4, obtain the fusion weight W36 of the first overexposed frame; fuse the three frames—the first underexposed frame, the first normal frame, and the first overexposed frame—with fusion weights of W34, W32, and W36 respectively, where:

[0088] W34 = W1.*(1-m4);

[0089] W36 = W2;

[0090] W36 = W3.*(1-m4).

[0091] Existing technologies, such as the ghost-free high dynamic range imaging method based on gradient structure similarity disclosed in CN108492262B, select reference frames for compensation in all ghost compensation regions, replacing the ghost regions of each non-reference frame with reference frames. However, if the reference frame has overexposed or underexposed areas in the ghost region, its information will be less rich than that of the non-reference frames, resulting in information loss during the final fusion. This embodiment employs different compensation methods for reference frames used in different ghost region compensations, achieving a more natural ghost compensation. Furthermore, it uses different weights for different reference frames during fusion to compensate for ghost regions, resulting in better ghost removal.

[0092] Furthermore, in another embodiment of this application, after step S80, step S90 is further included: post-processing the image after ghosting region compensation to obtain the final ghost-free high dynamic range image, wherein the post-processing may be brightening, face protection, etc.

[0093] Based on the above method, the present invention also provides a high dynamic range image generation apparatus for removing ghosting, including an original image acquisition module 20, a difference region acquisition module 40, a motion region acquisition module 60, and a ghosting region compensation module 80. The original image acquisition module 20 is used to acquire an original image, which includes a first normal frame, a first underexposed frame, and a first overexposed frame, wherein the first normal frame has a different exposure level than the first underexposed and first overexposed frames. The difference region acquisition module 40 is used to perform histogram matching between the first underexposed and first overexposed frames and the first normal frame, respectively, to obtain difference regions. The motion region acquisition module 60 is used to perform ghosting region detection on the difference regions to obtain motion regions. The ghosting region compensation module 80 is used to select reference frames from the first normal frame, first underexposed frame, and first overexposed frame based on maximizing information content to compensate for the motion regions, and to use different weights for fusion for different reference frames.

[0094] Please refer to Figure 8The high dynamic range image generation device for removing ghosting provided by the present invention operates as follows:

[0095] First, perform MIE histogram matching between the first underexposed frame and the first normal frame to obtain the second underexposed frame and the second normal frame; then perform MIE histogram matching between the first normal frame and the first overexposed frame to obtain the third normal frame and the second overexposed frame.

[0096] Next, SSIM-based ghost detection is performed on the second underexposed frame and the second normal frame to obtain the first mask image of the ghost region; SSIM-based ghost detection is performed on the third normal frame and the second overexposed frame to obtain the second mask image of the ghost region; the first mask image and the second mask image are then combined to obtain the third mask image of the merged ghost region.

[0097] Next, the third mask image is processed, such as morphological operations and connected component processing, to remove smaller noise areas and obtain the final fourth mask image of the ghosting region.

[0098] Next, a reference frame for compensating the ghosting area is selected based on maximizing information content. When the selected reference frame is an underexposed frame, the underexposed frame compensation process is entered, i.e., the aforementioned step S862; when the selected reference frame is a normal frame, the normal frame compensation process is entered, i.e., the aforementioned step S868; when the selected reference frame is an overexposed frame, the overexposed frame compensation process is entered, i.e., the aforementioned step S866.

[0099] Next, the images after removing the ghosting are merged;

[0100] Finally, post-processing is performed to obtain the final image.

[0101] The present invention also provides a computer-readable storage medium storing computer instructions that, when executed on an electronic device, cause the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiments.

[0102] In summary, this invention employs MIE-based histogram matching to preserve the original brightness dynamic range of the image as much as possible and to identify as many difference regions as possible. Then, for the MIE-resolved image, i.e., the difference regions, ghosting region detection is performed using SSIM based on structural components, which to some extent avoids the influence of brightness and contrast and finds a relatively complete motion region, which is the union of the aforementioned difference regions. Next, a reference frame for ghosting region compensation is adaptively selected based on maximizing information content, resulting in more natural ghosting compensation. At the same time, different weights are used for fusion of different reference frames to achieve better ghosting removal results.

[0103] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0104] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0105] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A ghost-removing high dynamic image generation method characterized by comprising: include: Acquire an original image, which includes a first normal frame, a first underexposed frame, and a first overexposed frame, wherein the first normal frame has a different exposure level than the first underexposed frame and the first overexposed frame; Histogram matching is performed between the first underexposed frame and the first overexposed frame and the first normal frame to obtain the difference region; Ghost region detection is performed on the discrepancy regions to obtain the motion regions; as well as The step of selecting a reference frame from the first normal frame, the first underexposed frame, and the first overexposed frame based on maximizing information content, and compensating the motion region includes one of the following methods: calculating the pyramid fusion weights of the first underexposed frame, the first normal frame, and the first overexposed frame respectively, and taking the frame with the highest pyramid fusion weight in the fourth ghost region as the reference frame. Calculate the pyramid fusion weights for the first underexposed frame, the first normal frame, and the first overexposed frame respectively. Count the number of pixels in the fourth ghost region whose fusion weight is less than the third threshold. Take the frame with the fewest pixels as the reference frame for compensating the ghost region. Pixels with a fusion weight less than the third threshold represent pixels with less information. The number of pixels in the first underexposed frame, the first normal frame, and the first overexposed frame in the fourth ghost region that have a pixel value less than the fourth threshold or greater than the fifth threshold is counted separately. The frame with the fewest pixels is taken as the reference frame for compensating the ghost region. Pixel values ​​less than the fourth threshold indicate extremely dark pixel values, and pixel values ​​greater than the fifth threshold indicate overexposed pixel values.

2. The ghost-canceled high dynamic image generating method according to claim 1, characterized by: The step of performing histogram matching between the first underexposed frame and the first overexposed frame and the first normal frame to obtain the difference region includes: Perform MIE histogram matching between the first underexposed frame and the first normal frame to obtain the second underexposed frame and the second normal frame; perform MIE histogram matching between the first normal frame and the first overexposed frame to obtain the third normal frame and the second overexposed frame. The first mask image of the ghost region is obtained by performing structure-based SSIM ghost detection on the second underexposed frame and the second normal frame; the second mask image of the ghost region is obtained by performing structure-based SSIM ghost detection on the third normal frame and the second overexposed frame.

3. The ghost-canceled high dynamic image generating method according to claim 2, characterized by: The step of performing ghost region detection on the difference region to obtain the motion region includes: The first mask image and the second mask image are combined to obtain the third mask image of the merged ghost region; The third mask image of the ghost region is processed, and noise regions with noise thresholds lower than the first threshold are removed to obtain the fourth mask image of the ghost region.

4. The ghost-canceled high dynamic image generation method according to claim 3, characterized by, The step of selecting a reference frame based on maximizing information content and compensating for the motion region includes: From the first underexposed frame, the first normal frame, and the first overexposed frame, a reference frame is selected for ghosting region compensation based on maximizing information content. Based on the reference frame, region compensation is performed on the fourth mask image to obtain the repaired video frame.

5. The ghost-canceled high dynamic image generation method according to claim 2, characterized by, The step of performing MIE histogram matching between the first underexposed frame and the first normal frame to obtain the second underexposed frame and the second normal frame includes: Calculate the cumulative histogram of the first underexposed frame and the cumulative histogram of the first normal frame. Calculate the median histogram of the first underexposed frame and the first normal frame based on the cumulative histograms of the first underexposed frame and the first normal frame. The median histogram is the harmonic mean of the cumulative histograms of the first underexposed frame and the first normal frame. Histogram matching is performed on the first underexposed frame and the first normal frame to obtain the second underexposed frame and the second normal frame. The second underexposed frame and the second normal frame have the same median histogram distribution.

6. The ghost-canceled high dynamic image generation method according to claim 2, characterized by, The step of performing structure-based SSIM ghost detection on the second underexposed frame and the second normal frame to obtain a first mask image of the ghost region includes: Obtain the SSIM values ​​of the first pixel of the second underexposed frame and the second pixel of the second normal frame, wherein the second pixel corresponds to the first pixel; The SSIM value is compared with the second threshold to obtain the comparison result; Based on the comparison results, the first mask image is obtained.

7. The ghost-canceled high dynamic image generation method according to claim 6, characterized by, The step of obtaining the SSIM value of the first pixel of the second underexposed frame and the second pixel of the second normal frame includes: obtaining the SSIM value based on the structure of the first pixel of the second underexposed frame and the second pixel of the second normal frame. 8.The ghost-removed high dynamic image generation method of claim 3, wherein: The step of compensating for the motion region includes: Calculate the pyramid fusion weight images W1, W2, and W3 for the first underexposed frame, the first normal frame, and the first overexposed frame respectively, and obtain the pyramid fusion weight m4 for the fourth mask image based on W1, W2, and W3. If the reference frame for ghosting region compensation is the first underexposed frame, then: Brighten the first underexposed frame to obtain the third underexposed frame; Based on W1 and m4, obtain the fusion weight W14 of the first underexposed frame; based on W2 and m4, obtain the fusion weight W15 of the first normal frame; based on W3 and m4, obtain the fusion weight W16 of the first overexposed frame; based on m4, obtain the fusion weight W17 of the third underexposed frame. If the reference frame for ghosting region compensation is the first overexposed frame, then: The third overexposed frame is obtained by performing histogram matching from the first overexposed frame to the first normal frame. Based on W1 and m4, obtain the fusion weight W24 of the first underexposed frame; based on W2 and m4, obtain the fusion weight W25 of the first normal frame; based on W3 and m4, obtain the fusion weight W26 of the first overexposed frame; based on m4, obtain the fusion weight W27 of the third overexposed frame. If the reference frame for ghosting region compensation is the first normal frame: Based on W1 and m4, obtain the fusion weight W34 of the first underexposed frame; based on W2, obtain the fusion weight W35 of the first normal frame; based on W3 and m4, obtain the fusion weight W36 of the first overexposed frame.

9. The high dynamic range image generation method for removing ghosting according to claim 8, characterized in that: The given values ​​are: W14 = W1.*(1 - m4), W15 = W2.*(1 - m4), W16 = W3.*(1 - m4), and W17 = m4. The given values ​​are: W24 = W1.*(1 - m4), W25 = W2.*(1 - m4), W26 = W3.*(1 - m4), and W27 = m4. The values ​​are: W34 = W1.*(1 - m4), W35 = W2, W36 = W3.*(1 - m4).

10. A high dynamic range image generation apparatus for removing ghosting, characterized in that, include: The original image acquisition module is used to acquire an original image, which includes a first normal frame, a first underexposed frame, and a first overexposed frame. The first normal frame has a different exposure level than the first underexposed frame and the first overexposed frame. The difference region acquisition module is used to perform histogram matching between the first underexposed frame and the first overexposed frame and the first normal frame respectively to obtain the difference region. The motion region acquisition module is used to perform ghost region detection on the difference region to obtain the motion region; as well as The ghost region compensation module is used to select a reference frame from the first normal frame, the first underexposed frame, and the first overexposed frame based on maximizing information content, and to compensate the motion region. The steps include one of the following: calculating the pyramid fusion weights of the first underexposed frame, the first normal frame, and the first overexposed frame respectively, and taking the frame with the highest pyramid fusion weight in the fourth ghost region as the reference frame. Calculate the pyramid fusion weights for the first underexposed frame, the first normal frame, and the first overexposed frame respectively. Count the number of pixels in the fourth ghost region whose fusion weight is less than the third threshold. Take the frame with the fewest pixels as the reference frame for compensating the ghost region. Pixels with a fusion weight less than the third threshold represent pixels with less information. The number of pixels in the first underexposed frame, the first normal frame, and the first overexposed frame in the fourth ghost region that have a pixel value less than the fourth threshold or greater than the fifth threshold is counted separately. The frame with the fewest pixels is taken as the reference frame for compensating the ghost region. Pixel values ​​less than the fourth threshold indicate extremely dark pixel values, and pixel values ​​greater than the fifth threshold indicate overexposed pixel values.

11. A computer-readable storage medium storing instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-9.

Images