A ghost-free multi-exposure image fusion method and device

Abstract

The present invention provides a ghost-free multi-exposure image fusion method, device, computer equipment and readable storage medium, which realizes the alignment of source sequence images to reference images by combining brightness mapping and consistency detection, and obtains a latent image sequence of different exposure brightnesses consistent with the moving object; based on the weighted least squares filter, the present invention can combine the favorable information in the different exposure image sequences, enhance and suppress the brightness of the overly dark and overly bright areas in the image, thereby retaining the details at different levels of brightness, and finally obtain a ghost-free image with vivid colors and rich information, effectively enhancing the visual quality and dynamic range of the fused image. The present invention can meet the needs of multi-exposure fusion in different dynamic scenes, and can effectively solve the ghost problem introduced by the moving object or the imaging device jitter in the imaging scene.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a ghost-free multi-exposure image fusion method, device, computer equipment and readable storage medium. Background Art

[0002] The brightness span in natural scenes is usually very large, ranging from starlight at night to dazzling sunlight, covering 9 orders of magnitude. Due to the constraints of the dynamic range of imaging equipment, it is difficult to record the detailed information of different brightness levels in the scene through a single exposure imaging; the high dynamic range image obtained by the multi-exposure image fusion (MEF) method can contain more detailed information and better match the response of the human eye to real scenes. It has broad application prospects and important practical significance in many fields. At present, most MEF methods assume that the source images are ideally aligned, but in actual imaging scenes, there are inevitably moving objects with different motion laws and camera jitter. There is a time difference in the image acquisition process. If the multi-exposure image sequence of the acquired dynamic scene is directly fused, it will cause the fusion result to appear blurry, ghosting or translucent areas, which seriously affects the imaging quality and becomes a challenging problem in the current multi-exposure image fusion technology.

[0003] In this regard, many studies have explored methods to remove ghosts and improve the quality of multi-exposure image fusion from different perspectives, which usually differ in two aspects: how to detect ghosts and how to eliminate them. Current methods can be mainly divided into two categories: moving object removal and moving object selection and registration. Moving object removal methods remove all moving objects in the scene by estimating the static background, without selecting a reference image. Most of the image scene is static, and only a small part contains moving objects. This type of method performs a consistency check on each pixel, and the moving objects are modeled as outliers and removed to obtain artifact-free high-dynamic images. However, this type of method is not robust when the scene changes frequently, and the effect is better for fast-moving objects. For slow-moving objects, it still cannot remove artifacts well. Moving object selection and registration methods retain moving objects in the output fusion results, restore or reconstruct pixels affected by motion by finding the local correspondence between the reference image and the sequence image, and spatially align the source sequence images. However, this type of method relies on the registration accuracy, and there are still problems of ghosting or color distortion in some areas to varying degrees, and the efficiency is low. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a ghost-free multi-exposure image fusion method, device, computer equipment and readable storage medium, which can meet the needs of multi-exposure fusion in different dynamic scenes and can effectively solve the ghost problem introduced by moving objects or imaging device jitter in the imaging scene.

[0005] In a first aspect, an embodiment of the present invention provides a method for fusion of ghost-free multi-exposure images, comprising:

[0006] Acquire a multi-exposure image sequence, wherein the multi-exposure image sequence is composed of images with multiple levels of exposure brightness;

[0007] Determine a reference image from the multi-exposure image sequence, wherein the reference image is an image with a medium exposure brightness;

[0008] For an overexposed image in the multi-exposure image sequence, determining a latent image for fusion based on a brightness mapping model;

[0009] For an underexposed image in a multi-exposure image sequence, determining the latent image for fusion based on a motion consistency principle;

[0010] Performing filtering on the latent image to obtain a low-frequency component and a high-frequency component of the latent image;

[0011] Constructing a mixed weight relationship corresponding to the low-frequency component and the high-frequency component and performing exposure moderation evaluation;

[0012] The low-frequency component and the high-frequency component are weightedly fused to obtain a target fused image.

[0013] As an optional solution, determining a reference image from the multi-exposure image sequence, wherein the reference image is an image with a medium exposure brightness, includes:

[0014] Arrange each image in the multi-exposure image sequence in ascending order according to the overall brightness value, and select the middle value I ref1 is the first reference image;

[0015] The overall exposure level of the brightness components of all images is counted to obtain the exposure prior information of the image, and the total number of pixels whose pixel values ​​fall within the target interval is calculated. The image with the largest total value is the second reference image I ref2 .

[0016] Compare the first reference image I ref1 With the second reference image I ref2 , when the sorting difference is greater than 2, select the first reference image I ref1 is the final reference image I ref , otherwise select the second reference image I ref2 is the reference image I ref , for images with a ranking value lower than the reference image I ref The image is configured as an underexposed image, and the ranking value is higher than the reference image I ref The image is configured as an overexposed image.

[0017] As an optional solution, for the overexposed image in the multi-exposure image sequence, determining a latent image for fusion based on a pre-trained brightness mapping model includes:

[0018] A mapping of the brightness relationship between the image and the reference image is established in the image of the multi-exposure image sequence to obtain a brightness mapping model, and the brightness of the reference image is mapped to the same brightness range as the overexposed image to obtain a latent image for fusion.

[0019] As an optional solution, for the underexposed image in the multi-exposure image sequence, determining the latent image for fusion based on the motion consistency principle includes:

[0020] A set of color image blocks are extracted from the same spatial position of the multi-exposure image sequence using a moving window with a fixed stride, and the structure vector S of the reference image is determined. ref and the image structure vector S of the multi-exposure image sequence k The inner product between the structure vector S ref and the image structure vector S k To judge the consistency of the motion of objects between them;

[0021] Pre-configure the structural consistency judgment threshold T ρ , detect the motion inconsistent pixels and motion consistent pixels between images;

[0022] The k-th exposure image is mapped to the brightness level of the reference image to obtain a latent image, and the absolute value of the average brightness difference between the image block of the corresponding latent image and the reference image at the same spatial position is determined, and the average brightness difference threshold T is pre-configured. u , determine other motion inconsistent pixels.

[0023] The final motion consistency detection result is obtained by multiplication processing, and the pixels of the image are retained at consistent pixel positions; at inconsistent pixel positions, the reference image is used to map the image pixels within the image brightness range.

[0024] Abnormal pixel judgment conditions are designed to constrain the latent image pixel values ​​and solve the problem of brightness saturation and color distortion in some local areas of the latent image.

[0025] As an optional solution, performing weighted processing on the latent image to obtain the low-frequency component and the high-frequency component of the latent image includes:

[0026] Determine the weighted sum of the RGB three color channels of the latent image to obtain a brightness image of each latent image

[0027] The latent image is estimated based on a weighted least squares filter to obtain low-frequency information of latent images with different exposures, and the low-frequency information corresponds to the filtered low-frequency image.

[0028] When the low-frequency component image of the input latent image is obtained According to the input latent image of each channel Get high-frequency component images on the red, green, and blue channels

[0029] As an optional solution, the step of constructing a mixed weight relationship corresponding to the low-frequency component and the high-frequency component and performing exposure moderation evaluation includes:

[0030] Constructing a weight function of the low-frequency component of the image based on the local average brightness and the global average brightness to evaluate the exposure suitability;

[0031] Mean filtering is performed based on the latent image, and a weight function of the high-frequency component of the image is determined to evaluate exposure suitability.

[0032] As an optional solution, performing weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fused image includes:

[0033] The high-frequency component and the low-frequency component of the latent image are multiplied by corresponding weights respectively, and weighted sum is performed. When fusion is performed, the RGB color channels are jointly processed to obtain a target fused image.

[0034] In a second aspect, an embodiment of the present invention provides a ghost-free multi-exposure image fusion device, comprising:

[0035] An acquisition component, used for acquiring a multi-exposure image sequence, wherein the multi-exposure image sequence is composed of images with multiple levels of exposure brightness;

[0036] A determination component, used to determine a reference image from the multi-exposure image sequence, wherein the reference image is an image with a medium exposure brightness;

[0037] The determining unit is further configured to determine, for an overexposed image in the multi-exposure image sequence, a latent image for fusion based on a pre-trained brightness mapping model;

[0038] The determining unit is further used to determine the latent image for fusion based on the motion consistency principle for the underexposed image in the multi-exposure image sequence;

[0039] The determining unit is further used to perform filtering processing on the latent image to obtain a low-frequency component and a high-frequency component of the latent image;

[0040] The determination unit is further used to construct a mixed weight relationship corresponding to the low-frequency component and the high-frequency component and perform exposure moderation evaluation;

[0041] The fusion component is used to perform weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fused image.

[0042] In a third aspect, an embodiment of the present invention provides a computer device, including:

[0043] at least one processor; and

[0044] a memory communicatively connected to the at least one processor; wherein,

[0045] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above-mentioned ghost-free multi-exposure image fusion method.

[0046] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the above-mentioned ghost-free multi-exposure image fusion method.

[0047] In the embodiment of the present invention, a method, device, computer equipment and readable storage medium for fusion of multiple exposure images without ghosting are provided, which realizes the alignment of source sequence images to reference images by combining brightness mapping and consistency detection, and obtains a latent image sequence of different exposure brightnesses consistent with the moving object; based on the weighted least squares filter, the present invention can combine the favorable information in the different exposure image sequences, enhance and suppress the brightness of the too dark and too bright areas in the image, thereby retaining the details at different brightness levels, and finally obtain a ghost-free image with vivid colors and rich information, effectively enhancing the visual quality and dynamic range of the fused image. The present invention can meet the needs of multiple exposure fusion in different dynamic scenes, and can effectively solve the ghost problem introduced by the imaging scene due to the jitter of the moving object or the imaging device. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a method for fusion of multi-exposure images without ghosting is provided in an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a latent image before and after abnormal pixel constraint is performed in a ghost-free multi-exposure image fusion method according to an embodiment of the present invention;

[0050] Figure 3a Schematic diagram of image sequences under different exposures of dynamic scenes;

[0051] Figure 3bA fusion result diagram of a ghost-free multi-exposure image fusion method is provided in an embodiment of the present invention;

[0052] Figure 4 A structural block diagram of a ghost-free multi-exposure image fusion device is provided in an embodiment of the present invention;

[0053] Figure 5 A structural block diagram of a computer device is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0055] The terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0056] Combination Figure 1 As shown, an embodiment of the present invention provides a method for fusion of multi-exposure images without ghosting, including:

[0057] S101, obtaining a multi-exposure image sequence, wherein the multi-exposure image sequence is composed of images with multiple levels of exposure brightness. Input the multi-exposure image sequence I k (k=1,…,K), K is the total number of input images, and all image pixel values ​​are normalized to the range of [0,1].

[0058] S102: Determine a reference image from the multi-exposure image sequence, where the reference image is an image with a medium exposure brightness.

[0059] S103: For the overexposed image in the multi-exposure image sequence, determine a latent image for fusion based on a pre-trained brightness mapping model.

[0060] S104: For underexposed images in the multi-exposure image sequence, determine the latent image for fusion based on the motion consistency principle.

[0061] S105 , performing filtering processing on the latent image to obtain a low-frequency component and a high-frequency component of the latent image.

[0062] S106: construct a mixed weight relationship corresponding to the low-frequency component and the high-frequency component and perform exposure moderation evaluation.

[0063] S107: Perform weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fused image.

[0064] In step S102, determining a reference image from the multi-exposure image sequence, wherein the reference image is an image with a medium exposure brightness, includes:

[0065] Arrange each image in the multi-exposure image sequence in ascending order according to the overall brightness value, and select the middle value I ref1 is the first reference image;

[0066] The overall exposure level of the brightness components of all images is counted to obtain the exposure prior information of the image, and the total number of pixels whose pixel values ​​fall within the target interval is calculated. The image with the largest total value is the second reference image I ref2 .

[0067] Compare the first reference image I ref1 With the second reference image I ref2 , when the sorting difference is greater than 2, select the first reference image I ref1 is the final reference image I ref , otherwise select the second reference image I ref2 is the reference image I ref , for images with a ranking value lower than the reference image I ref The image is configured as an underexposed image, and the ranking value is higher than the reference image I ref The image is configured as an overexposed image.

[0068] In step S103, for the overexposed image in the multi-exposure image sequence, determining a latent image for fusion based on a pre-trained brightness mapping model includes:

[0069] A mapping of the brightness relationship between the image and the reference image is established in the image of the multi-exposure image sequence to obtain a brightness mapping model, and the brightness of the reference image is mapped to the same brightness range as the overexposed image to obtain a latent image for fusion.

[0070] In step S104, for the underexposed image in the multi-exposure image sequence, determining the latent image for fusion based on the motion consistency principle includes:

[0071] A set of color image blocks are extracted from the same spatial position of the multi-exposure image sequence using a moving window with a fixed stride, and the structure vector S of the reference image is determined. ref and the image structure vector S of the multi-exposure image sequence k The inner product between the structure vector S ref and the image structure vector S k To judge the consistency of the motion of objects between them;

[0072] Pre-configure the structural consistency judgment threshold T ρ , detect the motion inconsistent pixels and motion consistent pixels between images;

[0073] The k-th exposure image is mapped to the brightness level of the reference image to obtain a latent image, and the absolute value of the average brightness difference between the image block of the corresponding latent image and the reference image at the same spatial position is determined, and the average brightness difference threshold T is pre-configured. u , determine other motion inconsistent pixels.

[0074] The final motion consistency detection result is obtained by multiplication processing, and the pixels of the image are retained at consistent pixel positions; at inconsistent pixel positions, the reference image is used to map the image pixels within the image brightness range.

[0075] Abnormal pixel judgment conditions are designed to constrain the latent image pixel values ​​and solve the problem of brightness saturation and color distortion in some local areas of the latent image.

[0076] In step S105, filtering the latent image to obtain a low-frequency component and a high-frequency component of the latent image includes:

[0077] Determine the weighted sum of the RGB three color channels of the latent image to obtain a brightness image of each latent image

[0078] The latent image is estimated based on a weighted least squares filter to obtain low-frequency information of latent images with different exposures, and the low-frequency information corresponds to the filtered low-frequency image.

[0079] When the low-frequency component image of the input latent image is obtained According to the input latent image of each channel Get high-frequency component images on the red, green, and blue channels

[0080] In step S106, the step of constructing a mixed weight relationship corresponding to the low-frequency component and the high-frequency component and performing exposure moderation evaluation includes:

[0081] Constructing a weight function of the low-frequency component of the image based on the local average brightness and the global average brightness to evaluate the exposure suitability;

[0082] Mean filtering is performed based on the latent image, and a weight function of the high-frequency component of the image is determined to evaluate exposure suitability.

[0083] In step S107, the low-frequency component and the high-frequency component are weightedly fused to obtain a target fused image, including:

[0084] The high-frequency component and the low-frequency component of the latent image are multiplied by corresponding weights respectively, and weighted sum is performed. When fusion is performed, the RGB color channels are jointly processed to obtain a target fused image.

[0085] The embodiment of the present invention also provides a ghost-free multi-exposure image fusion method, which combines brightness mapping and consistency detection to realize the alignment of the source sequence image to the reference image, and obtains a latent image sequence of different exposure brightnesses that is consistent with the moving object; based on the weighted least squares filter, the present invention can combine the favorable information in the different exposure image sequences to enhance and suppress the brightness of the overly dark and overly bright areas in the image, thereby retaining the details at different levels of brightness, and finally obtain a ghost-free image with vivid colors and rich information, effectively enhancing the visual quality and dynamic range of the fused image. The present invention can meet the needs of multi-exposure fusion in different dynamic scenes, and can effectively solve the ghost problem introduced by the moving object or the imaging device jitter in the imaging scene.

[0086] An embodiment of the present invention also provides a ghost-free multi-exposure image fusion method, comprising:

[0087] S201, input multi-exposure source image sequence I k (k=1,…,K), K is the total number of input images, and all source image pixel values ​​are normalized to the range of [0,1].

[0088] S202, the exposure quality of the reference image affects the clarity, area detection and correction of the moving target in the imaging scene, and further affects the visual effect of the multi-exposure fusion image. In order to select a reference image with appropriate exposure, a strategy combining intermediate value judgment with exposure information prior is adopted for judgment.

[0089] The detailed process of step S202 is as follows:

[0090] S2021, sort the input source image sequence in ascending order according to the brightness value, and select the middle value I ref1 For the reference image:

[0091]

[0092] S2022, counting the overall exposure level of all input source image sequences to obtain image exposure prior information, and calculating the total number of pixels in each image whose pixel value falls within the interval [0.01, 0.09]. The image with the largest total value is the reference image I. ref2 .

[0093] S2023, Comparison I ref1 with I ref2 If the difference between the two rankings is greater than 2, it means that the overall brightness level of the image sequence is too high or too low, so I is selected. ref1 is the final reference image I ref Otherwise, select I ref2 is the final reference image I ref , to ensure that the reference image has the maximum number of pixel values ​​with appropriate exposure. The source images with ranking values ​​lower than the reference image are set as underexposed images, and the source images with ranking values ​​higher than the reference image are set as overexposed images.

[0094] S203: For the overexposed image, a brightness mapping relationship is established between the source sequence image and the reference image to obtain a latent image for fusion. A brightness mapping relationship is established between images of different exposure levels to obtain a brightness mapping model:

[0095] I k =τ(I ref )

[0096] Where τ(I) = g -1 (kg(I)) is the brightness mapping function, I is the image brightness; g is the camera response function; k is the exposure coefficient, which represents the radiation relationship between the two images. The brightness mapping relationship model between the reference image and the remaining overexposed images can be established using the image brightness histogram; according to the brightness mapping model, the brightness of the reference image can be mapped to the same brightness range as the overexposed image, thereby obtaining the latent image for the final fusion.

[0097] S204. For underexposed images whose brightness values ​​are lower than those of reference images, the present invention utilizes the directional information of the image structure vector to realize motion consistency detection; introduces a structure consistency threshold and a brightness difference threshold to maximize the retention of consistent pixels and correct inconsistent pixels; adds abnormal pixel point judgment conditions to constrain and solve the problem of abnormal brightness in local areas of latent images.

[0098] The detailed process of step S204 is as follows:

[0099] S2041, using the directional information of the image structure vector to realize the motion consistency analysis and detection between images. First, a moving window with a fixed step is used to extract the motion consistency from the multi-exposure image sequence I k Extract a group of color image blocks {g k}={g k |1≤k≤K}, g k It is a CWH-dimensional vector (C is 3 channels, W and H are the width and height of the image block), and the structure vector S of the source image block is calculated. k , calculate the structure vector S of the reference image block ref In motion region detection, the motion consistency detection problem between the reference image and the source sequence image is converted into the directional consistency analysis problem of the structure vector between the reference image and the source sequence image. Calculate the reference image block structure vector S ref and the source image block structure vector S k The inner product between:

[0100]

[0101] In the formula, u gk Yes k The average brightness, u gref Yes ref The average brightness, ρ k ∈[-1,1], if ρ k The larger the value, the higher the S ref With S k The higher the similarity, the better. Construct S by brightness normalization k , which can ensure that the algorithm is more robust to changes in image exposure and contrast; because in areas with insufficient exposure, the contrast is weak and the visibility is low, the structure vector S k When scaled to unit length, the noise structure will be included, so the ∈ parameter close to 0 is introduced to ensure that the consistency ratio ρ is guaranteed regardless of the noise structure. k The value of is close to 1.

[0102] S2042: In order to detect more motion inconsistent pixels, a structural consistency threshold T is introduced. ρ Determine the consistency of motion:

[0103]

[0104] Through the binary image It is possible to observe pixels with inconsistent motion throughout the image sequence.

[0105] S2043: To maximize the detection and correction of inconsistent pixels and reduce fusion ghosting, increase the brightness difference judgment condition. Calculate the underexposed image I k With reference image I ref The brightness mapping function between the kth exposure image is used to map the kth exposure image to the reference image exposure level to obtain an image with the same brightness level. The absolute value of the average brightness difference between the image with the same brightness level and the image block of the reference image at the same spatial position is calculated, and the average brightness difference threshold T is set.u Make a judgment:

[0106]

[0107] In the formula, u' gk It is the average brightness of image blocks in an image of equal brightness level obtained by mapping the underexposed image to the reference image.

[0108] Step 4-4: The final motion consistency detection result is:

[0109]

[0110] For B k = 1, indicating that the source sequence image and the reference image move in the same way. In this case, the image block at the same position in the source sequence image is retained. k = 0, indicating that the source sequence image is inconsistent with the reference image in motion. It is necessary to correct the inconsistent pixels in combination with the brightness mapping function to obtain the latent image Q of the reference image at the lower exposure image brightness level. k .

[0111] S2045, combined with Figure 2 The figure shows the latent image before and after the underexposed image is constrained. In some latent images, local areas may experience brightness saturation distortion, resulting in missing or abnormal pixel colors. To address this, we add an abnormal pixel point determination condition to constrain it. If the pixel value at a certain position in the latent image is close to saturation, the corresponding pixel value is copied from the source sequence image to solve the problem of pixel saturation in the latent image:

[0112]

[0113] In the formula, Q k,1 , Q k,2 With Q k,3 The three color channels representing the latent image are shown in the figure below. Figure 2 Through the above processing, a latent image without color anomalies can be obtained by mapping the reference image to the low-exposure source image.

[0114] S205 , using a weighted least squares filter to estimate low-frequency information of latent images with different exposures, and then calculating high-frequency information of the latent images.

[0115] The detailed process of step S205 is as follows:

[0116] S2051, calculate the weighted sum of the RGB three channels of the input latent image to obtain the brightness map of each latent image

[0117] S2052, weighted least squares filter (weighted least squares, WLS) is an edge-preserving filter that can smooth other places outside the edge while maximally approaching the image The present invention estimates the low-frequency information of different exposure latent images based on the WLS filter. Given an input image Calculate the corresponding filtered low-frequency component image It can be calculated by the following formula:

[0118]

[0119]

[0120]

[0121] In the formula, the first term Ensure the similarity between the input image and the output image. The smaller the distance between the two, the higher the similarity between the filtered image and the input latent image. The second term achieves the degree of smoothness by taking partial derivatives, where λ is the regularization factor used to maintain the balance between the two sub-formulas. The larger the λ value, the smoother the output image. x and w y is the smoothing factor; V k For the input image The logarithmic form of The exponent α represents the sensitivity to the gradient of the input image, and ∈ is a constant close to 0.

[0122] S2053, once the low-frequency component image of the input latent image is obtained According to the input latent image Q on each color channel k , we can get the high-frequency component image on any red, green, or blue channel

[0123]

[0124] S206 , constructing weight functions of low- and high-frequency components of each level of exposure latent image based on the local average brightness and the global average brightness of the latent image of that level, and performing exposure appropriateness evaluation.

[0125] The detailed process of this step is as follows:

[0126] S2061, calculate the weight of the low-frequency component of the image, and construct a weight function of the component based on the local average brightness and the global average brightness to evaluate the exposure suitability. The weight function of the low-frequency component of the k-th level input latent image is calculated as follows:

[0127]

[0128] In the formula, the low frequency component It is used as a local brightness exposure feature. The global average brightness of the k-th level input image is calculated based on the total brightness of the entire image and the image size. σ L and σ G is the Gaussian standard deviation, which is set to 0.5 and 0.2 respectively; the first item retains the well-exposed local structure, and the second item encourages the consistency of the spatial brightness of the entire image. By applying these two factors, a fused image with better visual effect can be produced.

[0129] S2062, calculating the weight of the high-frequency component of the image. The weight function of the high-frequency component of the k-th level input latent image is calculated as follows:

[0130]

[0131] In the formula, is the latent image brightness map after mean filtering, σ H is the Gaussian standard deviation.

[0132] S207, multiply the low-frequency component and the high-frequency component of the input latent image by the corresponding mixing weights, and perform weighted summation. In order to retain more color information, the image color information is implicitly retained during calculation, and the RGB three color channels are jointly processed during fusion to obtain a final ghost-free and color-rich fused image.

[0133]

[0134] Figure 3a Schematic diagram of image sequence under different exposures of dynamic scenes. Figure 3b This is a fusion result diagram after applying the method provided by the present invention. It can be seen that the fused image has no ghosting and has richer details after integrating the light and dark level information of images with different exposures.

[0135] The ghost-free multi-exposure image fusion method provided in the embodiment of the present invention does not have many complex parameter settings and does not rely on the prior knowledge of the camera. By selecting the best reference image, effective scene features can be retained, the corresponding relationship between the source sequence image and the reference image is analyzed, and the over-exposed image and the under-exposed image are respectively subjected to brightness mapping and structural consistency detection, so as to effectively remove the ghost caused by the change of the motion area, and obtain the aligned latent image sequence of different exposures. The low-frequency components of the latent images of different exposures are estimated by using a weighted least squares filter, and then the corresponding high-frequency components are obtained, which overcomes the information distortion problem caused by the inaccurate estimation of the high and low-frequency components of the image; finally, the high and low-frequency components are weightedly fused using different mixing weights to obtain the final fusion result. Compared with the prior art, the present invention can better maintain the clear details of the bright and dark areas, and can reflect the integrity of the information of different levels of the real scene, and effectively improve the visual quality and dynamic range of the fused image; the present invention can not only handle the different amplitudes of motion of different moving objects and even the motion of large movements, but also is suitable for the slight shaking of the imaging device, with very few artifacts and stable performance. The method can provide technical support for obtaining high-dynamic imaging of different dynamic scenes.

[0136] Combination Figure 4 As shown, accordingly, an embodiment of the present invention further provides a ghost-free multi-exposure image fusion device, comprising:

[0137] An acquisition component 401 is used to acquire a multi-exposure image sequence, wherein the multi-exposure image sequence is composed of images with multiple levels of exposure brightness;

[0138] A determination component 402 is used to determine a reference image from the multi-exposure image sequence, wherein the reference image is an image with a medium exposure brightness;

[0139] The determining unit 402 is further configured to determine a latent image for fusion based on a pre-trained brightness mapping model for an overexposed image in the multi-exposure image sequence;

[0140] The determining unit 402 is further configured to determine the latent image for fusion based on the motion consistency principle for the underexposed image in the multi-exposure image sequence;

[0141] The determining unit 402 is further configured to perform weighted processing on the latent image to obtain a low-frequency component and a high-frequency component of the latent image;

[0142] The determining unit 402 is further used to construct a mixed weight relationship corresponding to the low-frequency component and the high-frequency component and perform exposure moderation evaluation;

[0143] The fusion component 403 is used to perform weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fused image.

[0144] Accordingly, according to an embodiment of the present invention, the present invention also provides a computer device, a readable storage medium and a computer program product.

[0145] Figure 5 FIG. 1 is a schematic diagram of the structure of a computer device 12 provided in an embodiment of the present invention. Figure 5 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 5 The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0146] like Figure 5 As shown, computer device 12 is in the form of a general-purpose computing device. Computer device 12 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0147] Components of computer device 12 may include, but are not limited to, one or more processors or processing units 16 , a system memory 28 , and a bus 18 that connects various system components including system memory 28 and processing unit 16 .

[0148] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0149] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0150] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 not shown, usually called a "hard drive"). Although Figure 5 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present invention.

[0151] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0152] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the computer device 12, and / or may communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0153] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the image fusion method provided in the embodiment of the present invention.

[0154] An embodiment of the present invention also provides a non-transitory computer-readable storage medium storing computer instructions, on which a computer program is stored, wherein when the program is executed by a processor, the image fusion method provided by all the inventive embodiments of the present application.

[0155] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, an apparatus, or a device.

[0156] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0157] The program code included in the computer-readable medium can be transmitted with any appropriate medium, including but not limited to wireless, electric wire, optical cable, RF, etc., or any suitable combination of the above. The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or their combinations, and the programming language includes object-oriented programming languages ​​such as Java, Smalltalk, C++, and also includes conventional procedural programming languages-such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect to the Internet).

[0158] An embodiment of the present invention further provides a computer program product, including a computer program, which implements the above-mentioned image fusion method when executed by a processor.

[0159] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0160] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A ghost-free multi-exposure image fusion method, characterized in that: include: Acquire a multi-exposure image sequence, wherein the multi-exposure image sequence is composed of images with multiple levels of exposure brightness; Arrange each image in the multi-exposure image sequence in ascending order according to the overall brightness value, and select the middle value of the order I ref1 is the first reference image; The overall exposure level of the brightness components of all images is counted to obtain the exposure prior information of the image, and the total number of pixels whose pixel values ​​fall within the target interval is calculated. The image with the largest total value is the second reference image. I ref2 ; Compare to first reference image I ref1 With the second reference image I ref2 , when the sorting difference is greater than 2, select the first reference image I ref1 The final reference image I ref , otherwise select the second reference image I ref2 For reference image I ref , for images with a ranking value lower than the reference image I ref The image is configured as an underexposed image, and the ranking value is higher than the reference image. I ref The image is configured as an overexposed image; Determine a reference image from the multi-exposure image sequence, wherein the reference image is an image with a medium exposure brightness; For an overexposed image in the multi-exposure image sequence, determining a latent image for fusion based on a brightness mapping model; Establishing a mapping of brightness relationship between the image in the multi-exposure image sequence and the reference image to obtain a brightness mapping model, mapping the brightness of the reference image to the same brightness range as the over-exposure image to obtain a latent image for fusion; For an underexposed image in a multi-exposure image sequence, determining the latent image for fusion based on a motion consistency principle; A set of color image blocks are extracted from the same spatial position of the multi-exposure image sequence using a moving window with a fixed stride to determine the structure vector of the reference image. S ref and the image structure vector of the multi-exposure image sequence S k The inner product between the structure vector S ref and the image structure vector S k To judge the consistency of the motion of objects between them; Pre-configure structural consistency judgment threshold T ρ , detect the motion inconsistent pixels and motion consistent pixels between images; The first k The sub-exposure image is mapped to the brightness level of the reference image to obtain a latent image, and the absolute value of the average brightness difference between the image block at the same spatial position of the corresponding latent image and the reference image is determined, and the average brightness difference threshold is pre-configured. T u , determine other motion inconsistent pixels; The final motion consistency detection result is obtained by multiplication processing, and the pixel points of the image are retained at the consistent pixel point positions; at the inconsistent pixel point positions, the reference image is used to map the image pixel points within the brightness range of the image; constraining pixel values ​​of the latent image by using abnormal pixel determination; Performing filtering on the latent image to obtain a low-frequency component and a high-frequency component of the latent image; Determine the weighted sum of the RGB three color channels of the latent image to obtain a brightness image of each latent image ; The latent image is estimated based on a weighted least squares filter to obtain low-frequency information of latent images with different exposures, and the low-frequency information corresponds to the filtered low-frequency image. ; When the low-frequency component image of the input latent image is obtained , according to the input latent image of each channel , get the high-frequency component images on the red, green, and blue channels ; Constructing a mixed weight relationship corresponding to the low-frequency component and the high-frequency component and performing exposure moderation evaluation; Performing mean filtering based on the latent image, determining a weight function of a high-frequency component of the image to evaluate exposure suitability; No. k The weight function of the low frequency component of the input latent image The calculation is as follows: ； Among them, the image size of the low-frequency component image is , σ L and σ G is the Gaussian standard deviation, set to 0.5 and 0.2 respectively; No. k The weight function of the high frequency component of the input latent image The calculation is as follows: ； in, is the latent image brightness map after mean filtering, σ H is the Gaussian standard deviation; The low-frequency component and the high-frequency component are weightedly fused to obtain the target fused image F: 。 2. The ghost-free multi-exposure image fusion method according to claim 1, characterized in that: The step of constructing a mixed weight relationship corresponding to the low-frequency component and the high-frequency component and performing exposure moderation evaluation includes: A weight function of the low-frequency component of the image is constructed based on the local average brightness and the global average brightness to evaluate the exposure suitability.

3. The ghost-free multi-exposure image fusion method according to claim 2, characterized in that: The step of performing weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fused image includes: The high-frequency component and the low-frequency component of the latent image are multiplied by corresponding weights respectively, and weighted sum is performed. When fusion is performed, the RGB color channels are jointly processed to obtain a target fused image.

4. A ghost-free multi-exposure image fusion device based on the ghost-free multi-exposure image fusion method according to any one of claims 1 to 3, characterized in that: include: An acquisition component, used for acquiring a multi-exposure image sequence, wherein the multi-exposure image sequence is composed of images with multiple levels of exposure brightness; A determination component, used to determine a reference image from the multi-exposure image sequence, wherein the reference image is an image with a medium exposure brightness; The determining component is further used to determine a latent image for fusion based on a pre-trained brightness mapping model for an overexposed image in the multi-exposure image sequence; The determining component is also used to determine the latent image for fusion based on the motion consistency principle for the underexposed image in the multi-exposure image sequence; The determining component is also used to perform weighted processing on the latent image to obtain a low-frequency component and a high-frequency component of the latent image; The determination component is also used to construct a mixed weight relationship corresponding to the low-frequency component and the high-frequency component and perform exposure moderation evaluation; The fusion component is used to perform weighted fusion processing on the low-frequency component and the high-frequency component to obtain a target fused image.

5. A computer device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the ghost-free multi-exposure image fusion method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the ghost-free multi-exposure image fusion method according to any one of claims 1 to 3.

Images