A single-exposure high-dynamic fusion imaging method based on color narrow-band filtering

By using narrowband filters and image fusion network model SFD-MEF before imaging, the dynamic range compression problem of traditional imaging technology in large-light ratio scenarios is solved, and efficient image fusion within a single exposure time is achieved, and dynamic processing is processed by achieving high-efficiency image fusion within a single exposure time. The scene and has excellent image quality in extreme environments.

CN118250573BInactive Publication Date: 2024-11-12NANJING UNIV OF SCI & TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410658444.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-27
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional imaging technology appears weak when dealing with large-light ratio scenes, and the dynamic range of the image is severely compressed. The existing multi-exposure image fusion technology has poor fusion effect in extreme environments and cannot handle dynamic scenes.

Method used

Using a single-exposure high-dynamic fusion imaging method based on color narrowband filters, three-color light with different quantum efficiency before imaging is obtained through narrowband filters, and image enhancement of three exposure sequences is used to use the image fusion network model SFD-MEF. and fusion, combining deconstructive encoding and deep learning technology, build a step-type encoder-decoder structure and multi-scale feature integration unit, and build a loss function to obtain complementary information between source image sequences.

Benefits of technology

Obtaining multiple different response pictures within a single exposure time can handle dynamic scenes and have excellent image fusion quality in extreme environments, significantly improving the dynamic range of extreme exposure scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118250573B_ABST
    Figure CN118250573B_ABST
Patent Text Reader

Abstract

The present invention discloses a single-exposure high-dynamic fusion imaging method based on color narrow-band filtering, comprising the following steps: 1. Using a narrow-band filter to obtain three-color light with different quantum efficiencies entering a camera before imaging, and obtaining a single-exposure image of the three-color light; 2. Extracting an image sequence of three exposure responses from the single-exposure image obtained in step 1 according to the pixel arrangement characteristics of the camera's sensor; 3. Performing image enhancement on the image sequences of the three exposure responses respectively; 4. Performing image fusion on the enhanced image sequences of the three exposure responses using an image fusion network model SFD‑MEF. The single-exposure high-dynamic fusion imaging method based on color narrow-band filtering of the present invention can obtain multiple images with different responses within a single exposure time by adding a narrow-band filter of a specific wavelength band, and using an image fusion network model for fusion, so that dynamic scenes can be processed and excellent image fusion quality can be achieved in extreme environments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a single-exposure high-dynamic fusion imaging method based on color narrow-band filtering, and belongs to the technical field of computer image processing and optical imaging. Background Art

[0002] With the rapid development of digital photography technology, we are able to capture more realistic and vivid visual scenes. However, traditional imaging technology is subject to physical limitations such as sensor performance and often lacks the ability to cope with scenes with large light ratios. Such scenes always present the characteristics of extremely strong light and extremely weak light coexisting, resulting in loss of image quality and severe compression of the image dynamic range.

[0003] As a very cost-effective solution, multi-exposure image fusion has attracted great attention in the field of high dynamic range imaging. Formally speaking, the multi-exposure image fusion method aims to recombine a given sequence of low dynamic range (LDR) images at different exposures into a high dynamic range (HDR) image with rich information and visual authenticity. Specifically, multiple images of the same scene with different exposures are taken to capture details in different brightness ranges. Usually, an image may be overexposed (loss of bright details) or underexposed (loss of dark details), and the multi-exposure image fusion technology can retain more details by synthesizing these images with different exposures into one image. However, the shooting of multiple data makes this technology unable to handle dynamic scenes.

[0004] Multi-exposure image fusion technology aims to integrate the information of multiple images to generate a composite image with richer features. This process can be performed at the pixel level, feature level or decision level, and can also be implemented in the spatial domain or transform domain. Traditional image fusion methods involve weighted averaging, wavelet transform, PCA-based methods, sparse representation and compressed sensing. For example, someone in the prior art proposed a multi-exposure fusion method based on image patches, which first extracts patches from the source image and then decomposes these patches into three conceptually independent components: signal intensity, signal structure and average intensity. These components are then processed according to the patch intensity and exposure to generate color image patches, and finally these image patches are put back into the fused image. However, there are limitations. In recent years, image fusion methods based on deep learning have emerged, such as convolutional neural networks, generative adversarial networks and autoencoders, which have been widely studied for image fusion tasks. For example, someone in the prior art also proposed DPE-MEF, which emphasizes the importance of details and colors of the source image. The detail enhancement module extracts the information hidden in the dark part and the information truncated in the bright part. After deeply mining the lost information of the source image, the color enhancement module corrects the original color display performance. However, the fusion effect is poor in extreme environments and cannot handle dynamic scenes.

[0005] Therefore, a single-exposure high-dynamic fusion imaging method based on color narrow-band filtering is needed to solve the above problems. Summary of the invention

[0006] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a single-exposure high-dynamic fusion imaging method based on color narrow-band filtering.

[0007] A single-exposure high-dynamic fusion imaging method based on color narrow-band filtering comprises the following steps:

[0008] Step 1: Use narrow-band filters to obtain three-color light with different quantum efficiencies before imaging to enter the camera and obtain a single exposure image of the three-color light;

[0009] Step 2: extracting three exposure response image sequences from the single exposure image obtained in step 1 according to the pixel arrangement characteristics of the camera sensor;

[0010] Step 3: Perform image enhancement on the image sequences of the three exposure responses respectively;

[0011] Step 4: Use the image fusion network model SFD-MEF to fuse the enhanced image sequences of the three exposure responses.

[0012] Furthermore, the sensor of the camera in step 1 adopts a Bayer array pixel arrangement, wherein the Bayer array arrangement feature is RGGB.

[0013] Furthermore, the single exposure image is in RAW format.

[0014] Furthermore, in step 1, the three-color light is red light, green light and blue light.

[0015] Furthermore, in step 2, a deconstruction coding method is used to extract an image sequence of three exposure responses from the single exposure image of three-color light obtained in step 1.

[0016] The deconstruction encoding method is to decompose and encode the original image with different quantum efficiency responses of the RGB three channels pixel by pixel in the order of pixel arrangement.

[0017] Furthermore, in step 3, the following formula is used to perform image enhancement on the image sequences of the three exposure responses respectively:

[0018]

[0019] In the formula, is the pixel brightness, To enhance the curve, is the training parameter.

[0020] Furthermore, the image fusion network model SFD-MEF in step 4 includes an encoder and a decoder, and the encoder and the decoder are connected through two convolutional layers;

[0021] The encoder comprises a first aliasing residual attention module RMA, a first downsampling module, a second aliasing residual attention module RMA, a second downsampling module, a third aliasing residual attention module RMA and a third downsampling module, wherein the first aliasing residual attention module RMA, the first downsampling module, the second aliasing residual attention module RMA, the second downsampling module, the third aliasing residual attention module RMA and the third downsampling module are connected in sequence;

[0022] The decoder includes a first upsampling module, a first multi-scale feature integration unit MFIM, a second upsampling module, a second multi-scale feature integration unit MFIM, a third upsampling module, a third multi-scale feature integration unit MFIM and an SFD-MEF output, and the first upsampling module, the first multi-scale feature integration unit MFIM, the second upsampling module, the second multi-scale feature integration unit MFIM, the third upsampling module, the third multi-scale feature integration unit MFIM and the SFD-MEF output are connected in sequence.

[0023] The image fusion network model SFD-MEF is a stepped encoder-decoder structure that uses the aliasing residual attention module RMA for feature extraction and the multi-scale feature integration unit MFIM for context feature enhancement.

[0024] Furthermore, the first aliased residual attention module RMA, the second aliased residual attention module RMA and the third aliased residual attention module RMA are all aliased residual attention modules RMA, and the aliased residual attention module RMA includes a first feature map input, a first two-dimensional convolution module Conv2D, a normalization module, a LeakyRelu activation function, a second two-dimensional convolution module Conv2D, a channel attention module and a spatial attention module. The first feature map input is processed by the first two-dimensional convolution module Conv2D to obtain a second feature map and a third feature map. The second feature map is normalized by the normalization module and connected to the third feature map. The LeakyRelu activation function and the second two-dimensional convolution module Conv2D connected in series are input and added to the first feature map input. The addition result is multiplied by the channel attention module to obtain a channel attention result. The channel attention result is multiplied by the spatial attention module to obtain an RMA output result.

[0025] Furthermore, the first multi-scale feature integration unit MFIM, the second multi-scale feature integration unit MFIM and the third multi-scale feature integration unit MFIM are all multi-scale feature integration units MFIM, and the multi-scale feature integration unit MFIM includes multiple fourth feature map inputs, a two-dimensional convolution module Conv2D and an aliasing residual attention module RMA. The multiple fourth feature maps are connected after the spatial dimension is increased, and then the results are input into the two-dimensional convolution module Conv2D and the aliasing residual attention module RMA connected in series to obtain the MFIM output result.

[0026] Furthermore, the loss function of the image fusion network model SFD-MEF is It is expressed by the following formula:

[0027] ,

[0028] In the formula, To fuse the images, These are the decomposition diagrams of three different exposure responses. represents the Sobel gradient operator, represents the mean square error calculation, represents the similarity function, Represents the maximum value selected pixel by pixel, represents pixel-by-pixel averaging, represents the structural similarity loss function, represents the gradient loss function, represents the mean square error loss function.

[0029] The loss function is constructed based on the structural similarity, gradient texture and pixel intensity of the image. The final loss function It can fully obtain the complementary information between source image sequences.

[0030] Principle of the invention: Taking advantage of the different response efficiencies of sensors to light in different bands, a narrow-band filter of a specific band is installed to eliminate high-intensity light and obtain equivalent high and low exposure image data by taking advantage of the different imaging efficiencies of RGB three-color in the same narrow band. Multiple pictures with different responses can be obtained within a single exposure time without the need for long-term multiple exposures. While ensuring that single-frame data improves sensor efficiency, the image fusion network model is used for fusion. The result has a huge improvement in the dynamic range of extreme exposure scenes and has a fast processing speed.

[0031] Beneficial effect: The single-exposure high-dynamic fusion imaging method based on color narrow-band filtering of the present invention can obtain multiple different response images within a single exposure time by adding a narrow-band filter of a specific band, and use the image fusion network model for fusion. It can handle dynamic scenes and has excellent image fusion quality in extreme environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is the camera quantum efficiency curve;

[0033] Figure 2 The Bayer array of the color camera sensor in the present invention;

[0034] Figure 3 Decomposition encoding is performed on the single exposure source image to extract the decomposed three-exposure response image in the present invention;

[0035] Figure 4 It is an unsupervised learning exposure compensation model in the present invention;

[0036] Figure 5 The single exposure decomposition multi-exposure response image fusion network model SFD-MEF in the present invention and the aliasing residual attention module and multi-scale feature integration unit contained therein;

[0037] Figure 6 This is the result of the image fusion network model SFD-MEF in the present invention on the fusion test of multiple exposure public datasets;

[0038] Figure 7 This is the result of the fusion test of the image fusion network model SFD-MEF in the present invention on the self-built single-exposure decomposition three-exposure response image dataset. DETAILED DESCRIPTION

[0039] The preferred embodiments of the present invention will be described below in conjunction with the accompanying drawings to more clearly and completely illustrate the technical solutions of the present invention.

[0040] The single-exposure high-dynamic fusion imaging method based on color narrow-band filtering of the present invention comprises the following steps:

[0041] Step 1: Use filters to obtain three colors of light with different quantum efficiencies before imaging to enter the camera;

[0042] like Figure 1As shown, in step one, we use a narrow-band filter in the visible light band to filter the incident light entering the camera in the front section of imaging. When visible light passes through the lens and is incident on the image point and absorbed by the photosensitive area, there is a certain probability of exciting electrons. This process is called photoelectric conversion. The probability of photons exciting electrons is also called quantum efficiency. The quantum efficiency of light of different wavelengths is different. The present invention uses a 620nm narrow-band filter with a half-wave bandwidth of 10nm. When the incident visible light passes through the narrow-band filter, there are three kinds of light with different quantum efficiencies in the narrow-band spectral range that is not filtered out, specifically, the R channel with high quantum efficiency, the B channel with low quantum efficiency, and the G channel with a quantum efficiency in the middle. The length, width and height of the final device are 45mm*45mm*80mm, which can capture images efficiently and clearly.

[0043] Step 2: Based on the pixel arrangement characteristics of the camera sensor, a decomposition-type three-exposure response image sequence is extracted from the obtained single-exposure image using a deconstruction coding method; in the present invention, the deconstruction coding method is to decompose and encode the original image of different quantum efficiency responses of the RGB three channels pixel by pixel in the pixel arrangement order.

[0044] like Figure 2 As shown, in step 2, the color camera adopts the Bayer array arrangement, and the Bayer array arrangement method is RGGB. The RGB three-color filter in the color camera sensor selects the color of the pixel. Specifically, each pixel is a combination of three color components of red, green, and blue, which can express a point in the RGB space. It only presents one color due to a single color filter. As described in step 1, we obtained three kinds of light with different quantum efficiencies, which were recorded in the pixels on the corresponding color channels after passing through the Bayer filter. The R channel pixel value is the largest, the G channel pixel value is the second, and the B channel pixel value is the smallest. More directly, it is manifested as different brightness of the RGB channels. The pixel array of the color camera is arranged on the sensor in the RGGB color pattern. Therefore, in the initial stage of imaging, the number of RGB three-color pixels is unbalanced, and all pixels are initially divided into one-quarter R channel pixels, one-quarter B channel pixels, and one-half G channel pixels. As Figure 3As shown in the figure, the incident light is filtered by the filter, and then color selected by the sensor Bayer filter, and then the single exposure image in RAW format is decomposed and encoded. The RAW format contains the light intensity collected by the camera sensor during exposure, which is a data packet. We deconstruct and encode the data packet to extract each pixel information. Specifically, in the original image matrix, the red channel pixels are staggered in odd rows and odd columns, the green channel pixels are distributed in odd rows and even columns and even rows and odd columns, and the blue channel pixels are staggered in even rows and even columns. The position information is used to decompose and extract each channel pixel value from the original image matrix one by one, and then the pixels of the same color channel are spliced ​​and converted into a grayscale image. The three decomposed images obtained at this time are the three exposure response images obtained after exposure decomposition extraction.

[0045] Step 3: Perform image enhancement on different exposure response image sequences;

[0046] In step three, image enhancement is performed on the three exposure response images to perform adaptive exposure compensation. Figure 4 As shown in the figure, an unsupervised learning exposure compensation model is used for exposure compensation, and a deep neural network is used to estimate the specific brightness curve of the fitted image. The image brightness adjustment problem is converted into a nonlinear curve mapping problem, and the formula is as follows:

[0047] (1)

[0048] (2)

[0049] in, is the pixel brightness, To enhance the curve, is a trainable parameter.

[0050] The range and scope of the curve are in the range [0,1] and the curve is simple and differentiable. The curve is monotonic, ensuring that the relative size relationship of pixels remains unchanged after enhancement. Then, using Figure 4 The image enhancement network shown performs image enhancement on images of different exposure response sequences.

[0051] Step 4: Perform image fusion on the enhanced image sequences with different exposure responses.

[0052] To further obtain a single exposure HDR fusion image. Figure 5As shown, the present invention designs a deep learning image fusion network model SFD-MEF based on single exposure decomposition multi-exposure images obtained by color narrowband filtering. The network mainly predicts the exposure weight map in the source image sequence, and finally obtains a high dynamic fusion result by multiplying the weight map with the corresponding pixels of the source image. The designed SFD-MEF main framework is a multi-scale connected hierarchical encoder-decoder, which can not only meet the fusion requirements of single frame decomposition images in extreme exposure scenes, but also adapt to the fusion tasks of other static sequence data. More specifically, we cross-mix the spatial domain attention mechanism and the channel domain attention mechanism in the residual module to improve the information feature extraction ability of the exposure weight map while avoiding the risk of information loss in each feature domain. In addition, we construct a multi-scale feature integration unit to capture and integrate exposure weight feature maps of different sizes. In the decoding stage, the present invention enhances the network's ability to express complex exposure environments by continuously integrating the underlying detail information and high-level semantic information of the source image sequence, and the merged association between features at different levels helps the model obtain a better continuous representation of the exposure weight map. Finally, we construct a loss function based on the structural similarity, gradient texture and pixel intensity of the image:

[0053] (3)

[0054] (4)

[0055] (5)

[0056] (6)

[0057] in, To fuse the images, These are the decomposition diagrams of three different exposure responses. represents the Sobel gradient operator, represents the mean square error calculation, Represents the maximum value selected pixel by pixel, represents the pixel-by-pixel average. Finally, our It can fully obtain the complementary information between source image sequences.

[0058] Extensive experiments have demonstrated the practicality of the single-frame multi-exposure image fusion method of the present invention. Figure 6 The fusion effect of the present invention on a public multi-exposure dataset is shown. Figure 7The fusion effect of the present invention on decomposing multi-exposure response image data in an actual large light ratio scene is shown. Table 1 and Table 2 show the quantitative data of the present invention under multiple evaluation indicators, where red is the highest value, blue is the second highest value, and green is the third highest value. From the qualitative actual image display effect and the quantitative evaluation index data, it can be seen that the image fusion model designed by the present invention has excellent image fusion quality.

[0059] Table 1 Quantitative display of evaluation indicators of SFD-MEF model 1

[0060]

[0061] Table 2 Quantitative display of evaluation indicators of SFD-MEF model 2

[0062] .

Claims

1. A single-exposure high-dynamic fusion imaging method based on color narrow-band filtering, characterized in that: The following steps are involved: Step 1: Use narrow-band filters to obtain three-color light with different quantum efficiencies before imaging to enter the camera and obtain a single exposure image of the three-color light; Step 2: extracting three exposure response image sequences from the single exposure image obtained in step 1 according to the pixel arrangement characteristics of the camera sensor; Step 3: Perform image enhancement on the image sequences of the three exposure responses respectively; Step 4: Use the image fusion network model SFD-MEF to fuse the enhanced image sequences of the three exposure responses; The image fusion network model SFD-MEF in step 4 includes an encoder and a decoder, and the encoder and the decoder are connected through two convolutional layers; The encoder comprises a first aliasing residual attention module RMA, a first downsampling module, a second aliasing residual attention module RMA, a second downsampling module, a third aliasing residual attention module RMA and a third downsampling module, wherein the first aliasing residual attention module RMA, the first downsampling module, the second aliasing residual attention module RMA, the second downsampling module, the third aliasing residual attention module RMA and the third downsampling module are connected in sequence; The decoder comprises a first upsampling module, a first multi-scale feature integration unit MFIM, a second upsampling module, a second multi-scale feature integration unit MFIM, a third upsampling module, a third multi-scale feature integration unit MFIM and an SFD-MEF output, wherein the first upsampling module, the first multi-scale feature integration unit MFIM, the second upsampling module, the second multi-scale feature integration unit MFIM, the third upsampling module, the third multi-scale feature integration unit MFIM and the SFD-MEF output are connected in sequence; The first aliased residual attention module RMA, the second aliased residual attention module RMA and the third aliased residual attention module RMA are all aliased residual attention modules RMA, and the aliased residual attention module RMA includes a first feature map input, a first two-dimensional convolution module Conv2D, a normalization module, a LeakyRelu activation function, a second two-dimensional convolution module Conv2D, a channel attention module and a spatial attention module. The first feature map input is processed by the first two-dimensional convolution module Conv2D to obtain a second feature map and a third feature map. The second feature map is normalized by the normalization module and connected to the third feature map. The LeakyRelu activation function and the second two-dimensional convolution module Conv2D connected in series are input and added to the first feature map input. The addition result is multiplied by the channel attention module to obtain a channel attention result. The channel attention result is multiplied by the spatial attention module to obtain an RMA output result. The first multi-scale feature integration unit MFIM, the second multi-scale feature integration unit MFIM and the third multi-scale feature integration unit MFIM are all multi-scale feature integration units MFIM, and the multi-scale feature integration unit MFIM includes a plurality of fourth feature map inputs, a two-dimensional convolution module Conv2D and an aliasing residual attention module RMA, and the plurality of fourth feature maps are connected after increasing the spatial dimension, and then the result is input into the two-dimensional convolution module Conv2D and the aliasing residual attention module RMA connected in series to obtain the MFIM output result; The loss function of the image fusion network model SFD-MEF It is expressed by the following formula: , In the formula, To fuse the images, These are the decomposition diagrams of three different exposure responses. represents the Sobel gradient operator, represents the mean square error calculation, represents the similarity function, Represents the maximum value selected pixel by pixel, represents pixel-by-pixel averaging, represents the structural similarity loss function, represents the gradient loss function, represents the mean square error loss function.

2. The single-exposure high-dynamic fusion imaging method based on color narrow-band filtering according to claim 1, characterized in that: The sensor of the camera described in step 1 adopts a Bayer array pixel arrangement.

3. The single-exposure high-dynamic fusion imaging method based on color narrow-band filtering according to claim 1, characterized in that: The single exposure image is in RAW format.

4. The single-exposure high-dynamic fusion imaging method based on color narrow-band filtering according to claim 1, characterized in that: In step 1, the three-color light is red light, green light and blue light.

5. The single-exposure high-dynamic fusion imaging method based on color narrow-band filtering according to claim 1, characterized in that: In step 2, a deconstruction coding method is used to extract an image sequence of three exposure responses from the single exposure image of three-color light obtained in step 1.

6. The single-exposure high-dynamic fusion imaging method based on color narrow-band filtering according to claim 1, characterized in that: In step 3, the following formula is used to perform image enhancement on the image sequences of the three exposure responses: In the formula, is the pixel brightness, To enhance the curve, is the training parameter.