Dual-exposure image fusion method based on NSST transform

Abstract

The present application relates to the technical field of image fusion, and more particularly to a double-exposure image fusion method based on NSST transformation, comprising: setting camera exposure parameters mainly with integration time and secondarily with gain, and taking an underexposed image and an overexposed image; respectively pre-processing the two images to obtain pre-processed underexposed and overexposed images; performing NSST transformation on the underexposed and overexposed images to obtain low-frequency subbands and high-frequency subbands; fusing the detail layer using the absolute value maximum method and the approximate layer using the saliency map weighting method; and inversely transforming to obtain a fused image. The present application solves the information loss problem existing in the down-sampling process of traditional fusion methods, and finally realizes stable fusion effect and complete detail preservation of double-exposure image fusion.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image fusion, and particularly relates to a double-exposure image fusion method based on NSST transformation. BACKGROUND

[0002] In recent years, high dynamic range imaging technology has been widely applied in many fields. This technology can be realized through hardware and software. The professional high dynamic acquisition and display device based on hardware is very expensive, and the feasibility is greatly limited. The method based on software depends on the traditional high dynamic imaging technology, and involves the estimation of camera response function and tone mapping. Compared with the traditional method, the multi-exposure image fusion technology is a more efficient and economical strategy. The multi-exposure image fusion effect is closely related to the number of images involved in the fusion. More input images help to realize the smoother transition of brightness information, and will not cause local brightness mutation of the fused image. However, in practical application, it is not easy to obtain multiple images with different exposure degrees under the same scene, especially when there is a moving target in the scene, which will cause ghosting phenomenon in the fused image.

[0003] At present, the mainstream multi-exposure image fusion method usually needs to collect multiple images, usually more than two, and the fusion effect depends on the brightness distribution level of the source image sequence. For input image sequences with few source images and large exposure difference, the fusion effect of the image is unstable. In addition, the mainstream fusion method usually adopts the fusion framework based on pyramid decomposition or wavelet transform. These methods all have down-sampling process in the transformation process, which leads to information loss and affects the fusion effect of the image. Moreover, the wavelet transform only has a limited number of directions, and cannot well represent the edge direction information in the image. SUMMARY

[0004] Therefore, the application aims to provide a double-exposure image fusion method based on NSST transformation, so as to solve the problems of unstable fusion effect depending on the brightness distribution level of the source image sequence, and information loss in the down-sampling process.

[0005] To achieve the above-mentioned purpose, the technical scheme of the application is as follows:

[0006] A double-exposure image fusion method based on NSST transformation, comprising the following steps:

[0007] S1, acquiring multiple images by setting the exposure parameters of the camera, and selecting underexposed images and overexposed images from the multiple images; The exposure parameters are mainly adjusted by integral time, and gain adjustment is auxiliary;

[0008] ​S2, for underexposed images and overexposed images Preprocessing was performed separately to obtain the preprocessed underexposed image. and overexposed images ;

[0009] S3. Process the pre-processed underexposed image and overexposed images Perform NSST transformations separately to obtain The NSST transform decomposition scale is set to 2, the number of directions in the first decomposition layer is 4, and the number of directions in the second decomposition layer is 8, thus obtaining the low-frequency subband corresponding to each preprocessed image. and high-frequency subband ,in, Represented as the low-frequency sub-band of the nth input image, This is represented as the high-frequency sub-bands of the nth input image at different scales and directions;

[0010] S4. The high-frequency subband obtained in step S3 The high-frequency subband is obtained by fusing the components using the method of taking the largest absolute value; the low-frequency subband is then processed. A saliency map-based weighted method is used for fusion to obtain the fused low-frequency subband. ;

[0011] S5. Perform an inverse NSST transform on the fused high-frequency subband and low-frequency subband obtained in step S4 to obtain the final fused image. .

[0012] Furthermore, in step S1, the specific method for acquiring multiple images by setting the camera's exposure parameters is as follows: control the camera to acquire multiple images of the same high dynamic range scene, thereby acquiring a series of images with different exposure conditions.

[0013] Furthermore, in step S2, the underexposed image... The AINDANE method was used for low-light enhancement preprocessing to improve overexposed images. First, inversion processing is performed, and then high brightness suppression preprocessing is performed using the AINDANE method.

[0014] Furthermore, in step S3, the NSST employs a non-downsampling tower filter.

[0015] Furthermore, in step S4, the high-frequency sub-band... The fusion process employs the method of taking the largest absolute value, specifically involving selecting pre-processed underexposed images from high-frequency sub-band components at the same decomposition scale and in the same direction. and overexposed images The value with the largest absolute value at the component is taken as the fusion result, and is specifically expressed as:

[0016] ,

[0017] ;

[0018] wherein, is expressed as the fusion result of the th high-frequency sub-band component at scale 1, k is expressed as the fusion result of the th high-frequency sub-band component at scale 2; the high-frequency sub-band components at the same scale are combined together to obtain the fused high-frequency sub-band at the scale. k k

[0019] Further, in step S4, the low-frequency sub-band is fused by using a method based on saliency map weighting, and specifically includes the following steps:

[0020] S41, a saliency detection algorithm based on frequency tuning is adopted, and a Gaussian difference filter is used to perform saliency detection on the preprocessed underexposed image and the preprocessed overexposed image respectively to obtain corresponding saliency maps and , and the expression of the Gaussian difference filter is:

[0021] ,

[0022] wherein, is expressed as a Gaussian filter, (x y) is expressed as a spatial position coordinate of a pixel point, and is expressed as a standard deviation parameter of the Gaussian filter;

[0023] S42, the saliency maps and are normalized to obtain normalized weight maps , and the expression is:

[0024]

[0025] wherein, is expressed as a normalized weight map corresponding to the th image, n is expressed as a saliency map gray value of the th image. n ​​​​​

[0026] S43, multiplying the normalized weight map respectively with the pre-processed underexposed image , the pre-processed overexposed image corresponding low-frequency subband to obtain a fused low-frequency subband , the expression is:

[0027] .

[0028] Compared with the prior art, the present application can achieve the following beneficial effects:

[0029] By setting the exposure parameters mainly in integral time and secondarily in gain, the noise interference caused by gain is reduced when obtaining the underexposed image and the overexposed image, ensuring that the two source images respectively completely retain the details of the dark part and the bright part of the scene, and providing high-quality basic data for subsequent fusion; then by performing NSST transformation on the pre-processed underexposed image and the overexposed image, the loss of image detail information caused by down-sampling is prevented; at the same time, the high-frequency subband and the low-frequency subband obtained by NSST transformation are respectively subjected to differential fusion by using the "absolute value taking maximum method" and the "significant map based weighting method", so as to fully retain the detail features and overall brightness information, thereby solving the information loss problem existing in the down-sampling process of the traditional fusion method, and finally realizing stable fusion effect and complete detail retention of the double-exposure image fusion. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which form a part of the present application, are intended to provide further understanding of the present application and are incorporated herein for illustrative purposes. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0031] Figure 1 The flowchart of the double-exposure image fusion method based on NSST transformation provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not constitute a limitation on the present application. In different embodiments, similar elements are associated with similar element labels. In the following embodiments, many details are described in order to make the present application better understood. However, those skilled in the art can easily recognize that some features can be omitted in different cases, or can be replaced by other elements, materials or methods. In some cases, some operations related to the present application are not shown or described in the specification in order to avoid the core part of the present application being overwhelmed by too much description, and it is not necessary for those skilled in the art to describe these related operations in detail according to the description in the specification and general technical knowledge in the art.

[0033] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other to form various embodiments without conflict. Meanwhile, each step or action in the method description can be sequentially adjusted or adjusted in a manner that can be easily seen by those skilled in the art. Therefore, the various sequences in the specification and drawings are only for the purpose of clearly describing a certain embodiment, and do not mean a necessary sequence, unless otherwise stated that a certain sequence must be followed.

[0034] In the description of the present application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" and the like can explicitly or implicitly include one or more features. In the description of the present application, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0035] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0036] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0037] As shown in the figure, the present embodiment provides a dual-exposure image fusion method based on NSST transform, including the following steps: Figure 1

[0038] S1, control the camera to capture multiple images of the same high dynamic scene by setting the exposure parameters of the camera, obtain a series of images with different exposure conditions, and select underexposed images and overexposed images from them; wherein the exposure parameters are mainly adjusted by integral time, and gain adjustment is auxiliary.

[0039] Specifically, the camera is controlled to maintain a relatively fixed position with the same high dynamic scene, so as to avoid scene misplacement of the multiple images due to camera displacement, and to ensure the spatial consistency of the image content in the subsequent preprocessing and fusion process. When setting the exposure parameters of the camera, the integral time is adjusted to control the exposure amount of the image: for the same high dynamic scene, overexposed images are captured by prolonging the integral time, so as to ensure that the information of dark areas in the scene is fully captured and the core details of bright areas are not completely lost; underexposed images are captured by shortening the integral time, so as to ensure that the information of bright areas in the scene does not overflow and the key profile of dark areas is identifiable. At the same time, gain adjustment is used as an auxiliary means, and only when the integral time adjustment cannot meet the basic exposure requirement, the gain is adjusted slightly, and the gain adjustment range is strictly controlled, so as to avoid introducing a large amount of noise due to excessive gain, which will interfere with the detail information in the subsequent preprocessing and fusion process, and to ensure that the underexposed images can clearly retain the dark detail features of the scene, and the overexposed images can effectively retain the bright detail features of the scene, so as to provide high-quality source image data for the subsequent preprocessing and fusion of dual-exposure images.

[0040] S2, preprocess the underexposed images and the overexposed images respectively, to obtain preprocessed underexposed images and overexposed images​ .

[0041] Further, for the underexposed image Low-light enhancement preprocessing is performed by using the AINDANE method, and the overexposed image First, reverse processing is performed, and then high-light suppression preprocessing is performed by using the AINDANE method.

[0042] Specifically, for the underexposed image Low-light enhancement preprocessing of the AINDANE method: The AINDANE (adaptive neighborhood nonlinear enhancement) method fuses brightness compression, contrast enhancement, and gamma correction. In the preprocessing process, first, the brightness compression mechanism is used to limit the overall brightness transition of the underexposed image to avoid distortion caused by sudden brightness rise in local areas; then, the contrast enhancement mechanism is used to perform adaptive neighborhood processing on the dark part of the image to strengthen the gray difference between the dark details and the surrounding areas, so that the dark details are clearly highlighted from the low brightness background; finally, the gamma correction mechanism is used to adjust the image gray curve to optimize the overall gray distribution, so that the brightness of the processed image meets the habit of human visual perception, and finally the preprocessed underexposed image is obtained, which retains the dark details completely and has natural overall visual effect.

[0043] Preprocessing of the overexposed image : First, reverse processing is performed to inversely map the brightness value of the overexposed image, so that the truncated bright details in the original overexposed area are converted into a form that can be highlighted by enhancement processing; then, the AINDANE method is used for high-light suppression preprocessing of the reversed overexposed image: through the brightness compression mechanism of the AINDANE method, the brightness rise of the original high-light area corresponding to the original overexposed area after reversal is specifically suppressed to avoid the recovery of the original overexposed area to too high brightness; at the same time, the contrast enhancement mechanism is used to retain the bright details in the original overexposed image that are not truncated, and the gamma correction mechanism is used to adjust the gray balance of the reversed image, and finally the preprocessed overexposed image is obtained, which retains the original bright details completely and has overall brightness adaptation, providing a consistent basis for subsequent NSST transformation and fusion operations.

[0044] S3, the preprocessed underexposed image and the overexposed image are respectively subjected to NSST transformation to obtain , the decomposition scale of the NSST transformation is set to 2, the number of directions of the first layer decomposition is set to 4, and the number of directions of the second layer decomposition is set to 8, to obtain the detail layer image and the approximation layer image corresponding to each preprocessed image, wherein represents the low-frequency subband of the n input image, denotes the first n high frequency subbands of the input image at different scales and orientations.

[0045] Further, in step S3, the NSST employs a non-subsampled pyramid filter.

[0046] After the pre-processed underexposed image and the pre-processed overexposed image is performed, a non-subsampled pyramid filter is employed to replace the Laplacian pyramid commonly used in traditional multi-scale decomposition, and multi-scale decomposition of the image is achieved through non-subsampled operation, which fundamentally avoids the loss of image detail information caused by the traditional down-sampling process, prevents the pseudo-Gibbs effect in the subsequent image reconstruction stage, and ensures the complete preservation of image high-frequency details and low-frequency energy information. In the specific transformation execution process, the decomposition scale of the NSST transformation is fixed at 2, the number of directions of the first layer decomposition is set to 4, and the number of directions of the second layer decomposition is set to 8. Through multi-directional localization processing, the transformation can accurately capture the edge, texture and other detail features of the pre-processed image in different directions, and adapt to the complex structure information in the image. After the transformation is completed, for each pre-processed image n =1 corresponds to the pre-processed underexposed image , n =2 corresponds to the pre-processed overexposed image ), the high frequency subband and the low frequency subband data: the low frequency subband mainly carries the overall brightness information and core energy of the image, reflects the macroscopic content and basic gray distribution of the image, and constitutes the approximate layer image corresponding to the pre-processed image; the high frequency subband is the direction number corresponding to each decomposition direction) specially carries the high frequency information such as edge profile and texture details of the image, integrates all high frequency subbands of the same pre-processed image, i.e. constitutes the detail layer image corresponding to the image, and finally provides clear and accurate processing objects for the subsequent differentiated fusion rules for the detail layer and the approximate layer.

[0047] S4, the high frequency subbands obtained in step S3 are fused by using the absolute value taking maximum method to obtain the fused high frequency subbands; the low frequency subbands are fused by using the method based on saliency map weighting to obtain the fused low frequency subbands.

[0048] The high frequency subbands ​The fusion process employs the method of taking the largest absolute value, specifically involving selecting pre-processed underexposed images from high-frequency sub-band components at the same decomposition scale and in the same direction. and overexposed images The value with the largest absolute value at this component is taken as the fusion result, specifically expressed as follows:

[0049] ;

[0050] ;

[0051] in, Represented as the first on scale 1 k The fusion result of each high-frequency subband component Represented as the 2nd scale k The fusion result of high-frequency sub-band components; combining components of the same scale k The high-frequency subband components are combined together to obtain the fused high-frequency subband at that scale.

[0052] For low-frequency subband The fusion is performed using a saliency map-based weighted method, which includes the following steps:

[0053] S41. A frequency-tuned saliency detection algorithm is used to process the pre-processed underexposed image using a Gaussian difference filter. and pre-processed overexposed image Significance tests were performed separately to obtain the corresponding saliency maps. and The expression for the Gaussian difference filter is:

[0054] ,

[0055] in, Represented as a Gaussian filter, (x y) Represented as the spatial coordinates of a pixel. and It is expressed as the standard deviation parameter of the Gaussian filter.

[0056] S42, Saliency Map and After normalization, the normalized weight graph is obtained, expressed as:

[0057] ,

[0058] in, This is represented as a normalized weighted graph after normalization.

[0059] S43. Normalize the weighted graph respectively with the pre-processed under-exposed image , the pre-processed over-exposed image corresponding low-frequency sub-band are multiplied to obtain a fused low-frequency sub-band , and the expression is:

[0060] .

[0061] In this way, the fused high-frequency sub-band completely retains the sharpest edges, textures and other high-frequency detail information in the two input images, and the fused low-frequency sub-band adaptively integrates the overall brightness and contour features of the two images according to human visual saliency, avoiding information loss caused by insufficient or excessive exposure of a single image, and strengthening key visual information through targeted fusion rules. The above fused high-frequency sub-band and low-frequency sub-band are subjected to NSST inverse transformation, and a high dynamic range fused image can be reconstructed. The image has both the dark details not lost in the under-exposed image and the complete bright information in the over-exposed image, with smooth brightness transition, rich details and no information loss, effectively solving the problem of fusion effect depending on multiple input images and information loss caused by downsampling in the prior art, and stable and high-quality high dynamic scene imaging can be achieved through only two exposure images.

[0062] S5, performing NSST inverse transformation on the fused high-frequency sub-band and low-frequency sub-band obtained in step S4 to obtain a final fused image .

[0063] Specifically, in step S5, the input data to be inverse transformed is first determined, i.e., the "fused high-frequency sub-band" and "fused low-frequency sub-band" obtained by fusion in step S4, wherein the fused high-frequency sub-band is composed of the detail components of each scale and each direction fused in step S4 j representing the decomposition scale, k representing the direction number under the corresponding scale) are integrated, and the fused low-frequency sub-band is the low-frequency component fused in step S4 .

[0064] Subsequently, NSST inverse transformation is performed according to parameters (decomposition scale is 2, first layer decomposition direction number is 4, and second layer decomposition direction number is 8) completely consistent with the NSST transformation in step S3, to ensure that the multi-scale and multi-direction reconstruction logic of the inverse transformation process matches the decomposition logic in the early stage, avoiding image reconstruction distortion caused by inconsistent parameters. Through NSST inverse transformation, the fused high-frequency sub-band detail components and the fused low-frequency sub-band (the ​The low-frequency component of the fused image carrying the overall brightness and contour information of the fused image) is subjected to frequency domain information reorganization and spatial domain recovery to finally generate a final fused image which can completely retain the dark and bright details of the double-exposure source image and has smooth brightness transition .

[0065] Through the above technical solution, by setting the exposure parameters mainly based on the integration time and supplemented by the gain, the noise interference caused by the gain is reduced when obtaining the underexposed image and the overexposed image, and it is ensured that the two source images respectively completely retain the dark and bright details of the scene, thereby providing high-quality basic data for subsequent fusion; then, the preprocessed underexposed image and overexposed image are subjected to NSST transformation with clear parameters, thereby preventing the loss of image detail information caused by down-sampling; at the same time, the high-frequency sub-band and the low-frequency sub-band obtained through the NSST transformation are subjected to differential fusion respectively by using the "absolute value taking the larger one" method and the "weighting method based on the saliency map", thereby fully retaining the detail features and overall brightness information, so as to solve the information loss problem existing in the down-sampling process of the traditional fusion method, and finally realize the double-exposure image fusion with stable fusion effect and complete detail retention.

[0066] Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

[0067] The specific embodiments of the present application do not constitute a limitation on the scope of protection of the present application. Any various other corresponding changes and modifications made according to the technical concept of the present application should be included in the scope of protection of the claims of the present application.

Claims

1. A double-exposure image fusion method based on NSST transform, characterized in that, Includes the following steps: S1. Acquire multiple images by setting the camera's exposure parameters, and select the underexposed image from them. and overexposed images The exposure parameters primarily adjust the integration time, with gain adjustment as a secondary measure; underexposed images. Clearly preserves details in the shadows of the scene and overexposed images. Effectively preserves details in the bright areas of the scene; S2, for underexposed images The AINDANE method was used for low-light enhancement preprocessing to improve overexposed images. First, the image is inverted, then the AINDANE method is used for high-brightness suppression preprocessing to obtain the preprocessed underexposed image. and overexposed images ; S3. Process the pre-processed underexposed image and overexposed images Perform NSST transformations separately to obtain The NSST transform decomposition scale is set to 2, the number of directions in the first decomposition layer is 4, and the number of directions in the second decomposition layer is 8, thus obtaining the low-frequency subband corresponding to each preprocessed image. and high-frequency subband ,in, Represented as the low-frequency sub-band of the nth input image, This is represented as the high-frequency sub-bands of the nth input image at different scales and directions; S4. The high-frequency subband obtained in step S3 The high-frequency subband is obtained by fusing the components using the method of taking the largest absolute value; the low-frequency subband is then processed. A saliency map-based weighted method is used for fusion to obtain the fused low-frequency subband. ; S5. Perform an inverse NSST transform on the fused high-frequency subband and low-frequency subband obtained in step S4 to obtain the final fused image. .

2. The double-exposure image fusion method based on NSST transform according to claim 1, characterized in that: In step S1, the specific method for acquiring multiple images by setting the camera's exposure parameters is as follows: control the camera to acquire multiple images of the same high dynamic range scene, thereby obtaining a series of images with different exposure conditions.

3. The double-exposure image fusion method based on NSST transform according to claim 1, characterized in that: In step S3, the NSST employs a non-downsampling tower filter.

4. The double-exposure image fusion method based on NSST transform according to claim 1, characterized in that: In step S4, the high-frequency sub-band is... The fusion process employs the method of taking the largest absolute value, specifically involving selecting pre-processed underexposed images from high-frequency sub-band components at the same decomposition scale and in the same direction. and overexposed images The value with the largest absolute value at this component is taken as the fusion result, specifically expressed as follows: ； ； in, Represented as the first on scale 1 k The fusion result of each high-frequency subband component Represented as the 2nd scale k The fusion result of high-frequency sub-band components; combining components of the same scale k The high-frequency subband components are combined together to obtain the fused high-frequency subband at that scale.

5. The double-exposure image fusion method based on NSST transform according to claim 1, characterized in that: In step S4, the low-frequency sub-band is... The fusion is performed using a saliency map-based weighted method, which includes the following steps: S41. A frequency-tuned saliency detection algorithm is used to process the pre-processed underexposed image using a Gaussian difference filter. and pre-processed overexposed image Significance tests were performed separately to obtain the corresponding saliency maps. and The expression for the Gaussian difference filter is: ， in, Represented as a Gaussian filter, (x y) Represented as the spatial coordinates of a pixel. and This is expressed as the standard deviation parameter of the Gaussian filter; S42, Saliency Map and Normalization is performed to obtain the normalized weight map. The expression is: ， in, Represented as the first n The weighted image corresponding to each image after normalization. Represented as the first n The saliency grayscale values ​​of the image; S43. Normalize the weighted graph Compared with the pre-processed underexposed image Pre-processed overexposed image Corresponding low-frequency sub-band Multiply to obtain the fused low-frequency subband. The expression is: 。

Images