An image fusion method and system based on domain transform filter and sparse representation

Abstract

This invention discloses an image fusion method and system based on domain transform filter and sparse representation. The method includes: acquiring corresponding visible light and infrared images; decomposing the visible light and infrared images respectively based on a low-pass filter; calculating fusion high-frequency subband coefficients based on the high-frequency subband coefficients of the visible light and infrared images according to a sparse representation strategy; calculating fusion low-frequency subband coefficients based on the low-frequency subband coefficients of the visible light and infrared images according to the low-frequency subband coefficients of the visible light and infrared images according to the low-frequency fusion rules of the domain transform filter; and superimposing the fusion high-frequency subband coefficients and fusion low-frequency subband coefficients to obtain a fused image. The system includes: an acquisition module, a decomposition module, a first fusion module, a second fusion module, and a coefficient superposition module. By using this invention, the focus information in the source image can be effectively preserved without introducing erroneous information. This invention can be widely applied in the field of image fusion.

Description

Technical Field

[0001] This invention relates to the field of image fusion, and in particular to an image fusion method and system based on domain transform filters and sparse representation. Background Technology

[0002] Due to differences in imaging mechanisms, the information contained in a single image acquired by an image sensor often fails to fully reflect the characteristics of a scene. To provide a comprehensive interpretation of a scene, a feasible approach is to synthesize images from different scenes, leading to the development of multi-source image fusion technology. Infrared and visible light image fusion is an important branch of this technology. Infrared imaging sensors can capture the thermal radiation emitted by objects, enabling the detection of hot objects in darkness or adverse weather conditions. However, infrared images typically lack sufficient background information, while visible light images contain more scene details and texture information. The fusion of infrared and visible light images can integrate infrared heat sources and visible light detail information, providing a more comprehensive interpretation of the scene. In recent years, infrared and visible light image fusion has received considerable attention in fields such as military reconnaissance, resource exploration, security monitoring, and automatic target recognition.

[0003] Spatial domain methods are an effective approach for image fusion. They directly utilize pixel information, solving the fusion problem by finding the weights of image patches or pixels, and can directly determine the pixel weights of the source image based on certain participation rate calculation rules. Furthermore, in recent years, spatial domain image enhancement techniques—edge-preserving filtering algorithms—have become increasingly common in image processing. However, general filters are inefficient at edge detection and may lose some details in the source image. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention aims to provide an image fusion method and system based on domain transform filters and sparse representation, which can effectively preserve the focus information in the source image without introducing erroneous information.

[0005] The first technical solution adopted in this invention is: an image fusion method based on domain transform filter and sparse representation, comprising the following steps:

[0006] Obtain the corresponding visible light image and infrared image from the source image;

[0007] Based on the low-pass filter, the visible light image and the infrared image are decomposed to obtain the high-frequency subband coefficient of the visible light image, the low-frequency subband coefficient of the visible light image, the high-frequency subband coefficient of the infrared image, and the low-frequency subband coefficient of the infrared image.

[0008] Based on the sparse representation strategy, the fused high-frequency subband coefficient is calculated according to the high-frequency subband coefficients of the visible light image and the infrared image.

[0009] The low-frequency fusion rule based on domain transform filtering calculates the fused low-frequency subband coefficients according to the low-frequency subband coefficients of visible light images and infrared images.

[0010] The fused high-frequency subband coefficients and the fused low-frequency subband coefficients are superimposed to obtain the fused image.

[0011] Furthermore, the step of calculating the fused high-frequency subband coefficients based on the sparse representation strategy, according to the high-frequency subband coefficients of the visible light image and the infrared image, specifically includes:

[0012] Sliding window block processing is performed on the high-frequency subband coefficients of visible light images and infrared images to obtain the corresponding image block sets;

[0013] The image patches in the image patch set are stretched into column vectors to obtain the sparse coefficient vector;

[0014] The sparse coefficient vector is optimized and fused based on the OMP algorithm to obtain the fused sparse vector;

[0015] By combining the fused sparse vector with a pre-defined overcomplete dictionary, a fused column vector is obtained;

[0016] The fused column vectors are reconstructed and inserted into their corresponding original positions to obtain the fused high-frequency subband coefficients.

[0017] Furthermore, the step of reconstructing the fused column vectors and inserting them into their corresponding original positions to obtain the fused high-frequency subband coefficients is expressed by the following formula:

[0018]

[0019] In the above formula, F represents the fusion high-frequency subband coefficient. k This represents the k-th image patch after reconstruction, where K represents the number of patches.

[0020] Furthermore, it also includes pre-training a pre-defined overcomplete dictionary, with specific steps including:

[0021] Select images as training samples;

[0022] The training samples are preprocessed using multi-scale detail enhancement techniques to obtain preprocessed samples.

[0023] The preprocessed samples are processed using the sliding window technique, image patches are collected, and a training set is constructed.

[0024] Based on the K-SVD algorithm, an overcomplete dictionary is trained using the training set to obtain a pre-defined overcomplete dictionary.

[0025] Furthermore, the low-frequency fusion rule based on domain transform filtering, specifically includes the step of calculating the fused low-frequency subband coefficients according to the low-frequency subband coefficients of the visible light image and the infrared image, which includes:

[0026] The low-frequency subband coefficients of the visible light image and the infrared image are normalized to obtain the normalized image.

[0027] The normalized image is processed using a rolling guided filter to obtain the filtering result;

[0028] The filtering results are iteratively updated to obtain the iterative results.

[0029] Detect the structure of the iteration results and generate an initial decision graph;

[0030] A mean filter is introduced to evaluate the initial decision map, and a numerical value is returned based on the evaluation result to obtain a salient structure image;

[0031] An iterative domain transform filter is used to filter images with salient structures to obtain the corresponding domain transform filter output.

[0032] The low-frequency subband coefficients of the visible light image and the infrared image are fused based on the domain transform filter output to obtain the fused low-frequency subband coefficients.

[0033] Furthermore, the step of processing the normalized image using a rolling guided filter to obtain the filtering result is expressed by the following formula:

[0034]

[0035] In the above formula, G A (m) represents the low-frequency subband coefficient of visible light. Gaussian filtering at the center pixel m Indicates normalization, S represents the set of neighboring pixels at position m, n represents the neighborhood pixels, and the structural scale parameter. Defined as the minimum standard deviation of the Gaussian kernel.

[0036] Furthermore, the step of fusing the low-frequency subband coefficients of the visible light image and the infrared image based on the domain transform filter output to obtain the fused low-frequency subband coefficients is expressed by the following formula:

[0037]

[0038] In the above formula, L represents the fusion low-frequency subband coefficient. T This represents the output of the domain transform filter after T iterations. This represents the low-frequency subband coefficient of visible light. This represents the low-frequency subband coefficient of infrared light.

[0039] The second technical solution adopted in this invention is: an image fusion system based on domain transform filter and sparse representation, comprising:

[0040] The acquisition module is used to acquire the corresponding visible light image and infrared image based on the source image;

[0041] The decomposition module decomposes the visible light image and the infrared image based on a low-pass filter to obtain the high-frequency subband coefficients of the visible light image, the low-frequency subband coefficients of the visible light image, the high-frequency subband coefficients of the infrared image, and the low-frequency subband coefficients of the infrared image.

[0042] The first fusion module, based on a sparse representation strategy, calculates the fused high-frequency subband coefficients according to the high-frequency subband coefficients of the visible light image and the infrared image.

[0043] The second fusion module, based on the low-frequency fusion rule of domain transform filtering, calculates the fused low-frequency subband coefficients according to the low-frequency subband coefficients of the visible light image and the infrared image.

[0044] The coefficient overlay module is used to overlay the fused high-frequency subband coefficients and the fused low-frequency subband coefficients to obtain a fused image.

[0045] The beneficial effects of the method and system of the present invention are as follows: The present invention uses a low-pass filter to decompose the image, fuses the high-frequency image using a sparse representation method, and then uses an improved domain transform filter to transfer small-scale details from the source image to the fused output. The present invention narrows the gap between sparse domain-based and spatial domain-based methods, and effectively preserves the focus information in the multi-focus source image without introducing erroneous information. Attached Figure Description

[0046] Figure 1 This is a flowchart of the steps of an image fusion method based on domain transform filter and sparse representation according to the present invention;

[0047] Figure 2 This is a schematic diagram of the data flow in the method of the present invention;

[0048] Figure 3 This is a block diagram of an image fusion system based on domain transform filter and sparse representation according to the present invention. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0050] like Figure 1 As shown, this invention provides an image fusion method based on domain transform filters and sparse representation, which includes the following steps:

[0051] S1. Obtain the corresponding visible light image and infrared image based on the source image;

[0052] S2. Based on the low-pass filter, the visible light image and the infrared image are decomposed to obtain the high-frequency subband coefficient of the visible light image, the low-frequency subband coefficient of the visible light image, the high-frequency subband coefficient of the infrared image, and the low-frequency subband coefficient of the infrared image.

[0053] This also includes pre-training on a pre-defined overcomplete dictionary:

[0054] First, eight images are randomly selected as samples for training the dictionary. Multi-scale detail enhancement techniques are then used to preprocess the samples. The low-frequency and high-frequency information layers of the image are obtained by solving the low-pass filter optimization function composed of the following discrete gradient operators:

[0055]

[0056] Where h c h represents the c-th source image. b It is h c The low-frequency components, g p =[-1,1] and g q =[-1,1] T These are the gradient operators for the vertical and horizontal directions, respectively, with * indicating a convolution operation. β is the regularization parameter controlling the low-pass filter.

[0057] Then, the low-pass filter is used to perform multiple decompositions on the image, extracting high-frequency information at different levels, and finally merging them into the source image.

[0058]

[0059]

[0060] It is an image enhanced with multi-scale details. Representative image h c The high-frequency information of the lth layer.

[0061] We then selected the images after multi-scale detail enhancement as training data, used the sliding window technique to collect 8×8 patches in the images, constructed the final training set, and used the K-SVD algorithm to obtain the overcomplete dictionary D.

[0062] S3. Based on the sparse representation strategy, calculate the fused high-frequency subband coefficient according to the high-frequency subband coefficients of the visible light image and the infrared image.

[0063] S3.1 Perform sliding window block extraction on the high-frequency subband coefficients of the visible light image and the high-frequency subband coefficients of the infrared image to obtain the corresponding image block set;

[0064] S3.2. Pull the image patches in the image patch set into column vectors to obtain the sparse coefficient vector;

[0065] S3.3. Based on the OMP algorithm, the sparse coefficient vector is optimized and fused to obtain the fused sparse vector;

[0066] S3.4. Combine the fused sparse vector with the pre-defined overcomplete dictionary to obtain the fused column vector;

[0067] S3.5 Reconstruct the fused column vectors and insert them into their corresponding original positions to obtain the fused high-frequency subband coefficients.

[0068] Specifically, the sparse representation stage can be roughly divided into three steps: sliding window block extraction and block vectorization, sparse coding, and sparse vector fusion.

[0069] In the first stage, we first process each source image into blocks, taking eight 8×8 patches from top to bottom and left to right. Then, we normalize each patch, and finally stretch each image patch into a corresponding column vector. k and n represent the number of image patches and the number of source images, respectively. Representative image I d,l The column vector of the k-th layer image block.

[0070] In the sparse coding stage, we use the OMP algorithm to process the sparse coefficient vector of high-frequency images. Optimize the solution.

[0071]

[0072] in y is a sparse coefficient vector

[0073] In the sparse vector fusion stage, we first use the "maximum absolute value" fusion rule to obtain the fused sparse vector.

[0074] By combining D and The column vector of the k-th layer image patch of the fused image can be obtained.

[0075]

[0076] All column vectors Reconstructing image patch F k Then each F k By inserting it into its corresponding original position, the fused high-frequency image is finally obtained.

[0077]

[0078] S4. Low-frequency fusion rules based on domain transform filtering: calculate the fusion low-frequency subband coefficients based on the low-frequency subband coefficients of the visible light image and the infrared image.

[0079] S4.1 Normalize the low-frequency subband coefficients of the visible light image and the infrared image to obtain the normalized image;

[0080] S4.2. The normalized image is processed using a rolling guided filter to obtain the filtering result;

[0081] S4.3. Iterate and update the filtering results to obtain the iterative results;

[0082] S4.4 Detect the structure of the iteration results and generate an initial decision graph;

[0083] S4.5. Introduce a mean filter to judge the initial decision map, and return the numerical value according to the judgment result to obtain the salient structure image;

[0084] S4.6. An iterative domain transform filter is used to filter images with significant structures to obtain the corresponding domain transform filter output.

[0085] S4.7. Based on the domain transform filter output, fuse the low-frequency subband coefficients of the visible light image and the low-frequency subband coefficients of the infrared image to obtain the fused low-frequency subband coefficients.

[0086] Specifically, based on the characteristics of the low-frequency subband coefficients of visible-domain infrared images, we designed a low-frequency fusion rule based on domain transform filtering, which enables the fused low-frequency image to effectively and quickly retain the main structure of the source image, as follows:

[0087] For low-frequency images of visible and infrared light and First, their intensities are normalized to the range [0,1]. Assume each low-frequency image I has an intensity in the range [0,255]. b If there are 8 bits, then I b Normalized to:

[0088]

[0089] S is a normalized image in the range [0,1]. It should be noted that the low-frequency image and high-frequency image mentioned in this embodiment are no different from the low-frequency subband coefficient and the high-frequency self-band coefficient.

[0090] We then used a rolling guided filter to process low-frequency images of visible and infrared light. and Image processing and The Gaussian filter at the center pixel m can be expressed as:

[0091]

[0092]

[0093] in For normalization, S is the set of neighboring pixels at position m, and n represents the neighborhood pixels, the structural scale parameter. It can be defined as the minimum standard deviation of the Gaussian kernel, G. A and G B These are images and The output result.

[0094] We then iteratively update the image G. and Initially set to G A and G B This is the output of the Gaussian filter. Let K... N+1 The result of the Nth iteration is represented as follows:

[0095]

[0096]

[0097] Used for normalization, S is the set of neighboring pixels of m. and We control the domain weights and range weights separately, where m and n represent the center pixel and the neighborhood pixel, respectively. Then, we use the absolute value approximation of the gradient magnitude to detect the structure of the iteration result K.

[0098]

[0099] Where x and y are spatial coordinates, their numerical differences are:

[0100]

[0101] By comparing K A and K BBased on its size, we can obtain a decision diagram:

[0102] D = W A -W B

[0103] where W A and W B are the sizes of images K A and K B respectively.

[0104] Then, a mean filter is introduced to process image D:

[0105]

[0106] The saliency structure L A is a binary matrix:

[0107]

[0108] where, if the corresponding element is positive, the function step(·) returns 1 for an element of L A , otherwise it returns 0. Finally, an iterative domain transform filter is used to filter the saliency structure image L A :

[0109]

[0110] where t represents the number of iterations. After the domain transform filter iterates T times, the output is denoted by L T . Finally, the fused low-frequency component is obtained through the following formula:

[0111]

[0112] represents the low-frequency fusion component, and represent the low-frequency components of the visible light image and the infrared image respectively.

[0113] S5. Superimpose the fused high-frequency subband coefficients and the fused low-frequency subband coefficients to obtain the fused image.

[0114] Furthermore, as a preferred embodiment of this method, the model of sparse representation in step S3 is defined as follows:

[0115] For a signal y = (y1, y2,... y n ), y ∈ R e , the basic assumption of SR theory is that y can be approximately represented as a set of basic signals in a redundant dictionary D ∈ R e×f (e < f) A linear combination of these, the signal y can be expressed as:

[0116]

[0117] Where α=(α1,α2,…α) N ) is an unknown sparse coefficient vector, d i It is an atom of D. Since the dictionary is redundant, the vector α is not unique. Therefore, the SR model is proposed as a method to determine the solution vector α with the fewest non-zero vectors. This method is expressed mathematically as follows:

[0118]

[0119] Where ||·||0 denotes the norm for calculating the number of non-zero terms, and ε>0 is a tolerance. The above formula is solved using the greedy approximation method OMP.

[0120] As a preferred embodiment of this method, the right side of step S4 is as follows:

[0121] Given an input image I and a guidance image T, the output of the bilateral filter for pixel p is:

[0122]

[0123] Where Ω p A sliding window centered on pixel p. For space scaling parameters, U is the range scaling parameter. p For the normalization term:

[0124]

[0125] Distance metrics can be derived from Defined by the power term in:

[0126]

[0127] Then convert and rewrite as follows:

[0128]

[0129] Transformed signal δ p The derivative is defined as:

[0130]

[0131] Where δ' p δ p The derivative of T' p It is T p The derivative of . Then define a distance δ in the transformation domain. pq :

[0132]

[0133] Because this domain transform filter has two input images I p and T p And it contains two parameters σ s and σ r Ultimately, a single output L is generated. p It can generally be defined as:

[0134] L p =DTF(l p ,T p ,σ s ,σ r )

[0135] like Figure 3 As shown, an image fusion system based on domain transform filters and sparse representation includes:

[0136] The acquisition module is used to acquire the corresponding visible light image and infrared image based on the source image;

[0137] The decomposition module decomposes the visible light image and the infrared image based on a low-pass filter to obtain the high-frequency subband coefficients of the visible light image, the low-frequency subband coefficients of the visible light image, the high-frequency subband coefficients of the infrared image, and the low-frequency subband coefficients of the infrared image.

[0138] The first fusion module, based on a sparse representation strategy, calculates the fused high-frequency subband coefficients according to the high-frequency subband coefficients of the visible light image and the infrared image.

[0139] The second fusion module, based on the low-frequency fusion rule of domain transform filtering, calculates the fused low-frequency subband coefficients according to the low-frequency subband coefficients of the visible light image and the infrared image.

[0140] The coefficient overlay module is used to overlay the fused high-frequency subband coefficients and the fused low-frequency subband coefficients to obtain a fused image.

[0141] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0142] An image fusion device based on domain transform filter and sparse representation:

[0143] At least one processor;

[0144] At least one memory for storing at least one program;

[0145] When the at least one program is executed by the at least one processor, the at least one processor implements an image fusion method based on a domain transform filter and sparse representation as described above.

[0146] The content of the above method embodiments is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0147] A storage medium storing processor-executable instructions, characterized in that: the processor-executable instructions, when executed by the processor, are used to implement the image fusion method based on domain transform filter and sparse representation as described above.

[0148] The content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0149] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An image fusion method based on domain transform filter and sparse representation, characterized in that, Includes the following steps: Obtain the corresponding visible light image and infrared image from the source image; Based on the low-pass filter, the visible light image and the infrared image are decomposed to obtain the high-frequency subband coefficient of the visible light image, the low-frequency subband coefficient of the visible light image, the high-frequency subband coefficient of the infrared image, and the low-frequency subband coefficient of the infrared image. Based on the sparse representation strategy, the fused high-frequency subband coefficient is calculated according to the high-frequency subband coefficients of the visible light image and the infrared image. The low-frequency fusion rule based on domain transform filtering calculates the fused low-frequency subband coefficients according to the low-frequency subband coefficients of visible light images and infrared images. The fused high-frequency subband coefficients and the fused low-frequency subband coefficients are superimposed to obtain the fused image; The low-frequency fusion rule based on domain transform filtering, specifically includes the step of calculating the fused low-frequency subband coefficients according to the low-frequency subband coefficients of the visible light image and the infrared image: The low-frequency subband coefficients of the visible light image and the infrared image are normalized to obtain the normalized image. The normalized image is processed using a rolling guided filter to obtain the filtering result; The filtering results are iteratively updated to obtain the iterative results. Detect the structure of the iteration results and generate an initial decision graph; A mean filter is introduced to evaluate the initial decision map, and a numerical value is returned based on the evaluation result to obtain a salient structure image; An iterative domain transform filter is used to filter images with salient structures to obtain the corresponding domain transform filter output. The low-frequency subband coefficients of the visible light image and the infrared image are fused based on the domain transform filter output to obtain the fused low-frequency subband coefficients.

2. The image fusion method based on domain transform filter and sparse representation according to claim 1, characterized in that, The step of calculating the fused high-frequency subband coefficients based on the sparse representation strategy, according to the high-frequency subband coefficients of the visible light image and the infrared image, specifically includes: Sliding window block processing is performed on the high-frequency subband coefficients of visible light images and infrared images to obtain the corresponding image block sets; The image patches in the image patch set are stretched into column vectors to obtain the sparse coefficient vector; The sparse coefficient vector is optimized and fused based on the OMP algorithm to obtain the fused sparse vector; By combining the fused sparse vector with a pre-defined overcomplete dictionary, a fused column vector is obtained; The fused column vectors are reconstructed and inserted into their corresponding original positions to obtain the fused high-frequency subband coefficients.

3. The image fusion method based on domain transform filter and sparse representation according to claim 2, characterized in that, The step of reconstructing the fused column vector and inserting it into the corresponding original positions to obtain the fused high-frequency subband coefficients is expressed by the following formula: In the above formula, Indicates the fusion high-frequency subband coefficients. This represents the k-th image patch after reconstruction, where K represents the number of patches.

4. The image fusion method based on domain transform filter and sparse representation according to claim 2, characterized in that, It also includes pre-training a pre-defined overcomplete dictionary, with specific steps including: Select images as training samples; The training samples are preprocessed using multi-scale detail enhancement techniques to obtain preprocessed samples. The preprocessed samples are processed using the sliding window technique, image patches are collected, and a training set is constructed. Based on the K-SVD algorithm, an overcomplete dictionary is trained using the training set to obtain a pre-defined overcomplete dictionary.

5. The image fusion method based on domain transform filter and sparse representation according to claim 1, characterized in that, The step of processing the normalized image using a rolling guided filter to obtain the filtered result is expressed by the following formula: In the above formula, Represents the low-frequency subband coefficient of visible light Gaussian filtering at the center pixel m Indicates normalization. express The set of adjacent pixels, Represents neighborhood pixels, structural scale parameters Defined as the minimum standard deviation of the Gaussian kernel.

6. The image fusion method based on domain transform filter and sparse representation according to claim 1, characterized in that, The step of fusing the low-frequency subband coefficients of the visible light image and the infrared image based on the domain transform filter output to obtain the fused low-frequency subband coefficients is expressed by the following formula: In the above formula, This represents the fusion low-frequency subband coefficient. This represents the output of the domain transform filter after T iterations. This represents the low-frequency subband coefficient of visible light. This represents the low-frequency subband coefficient of infrared light.

7. An image fusion system based on domain transform filter and sparse representation, characterized in that, An image fusion method based on domain transform filter and sparse representation as described in claim 1 includes: The acquisition module is used to acquire the corresponding visible light image and infrared image based on the source image; The decomposition module decomposes the visible light image and the infrared image based on a low-pass filter to obtain the high-frequency subband coefficients of the visible light image, the low-frequency subband coefficients of the visible light image, the high-frequency subband coefficients of the infrared image, and the low-frequency subband coefficients of the infrared image. The first fusion module, based on a sparse representation strategy, calculates the fused high-frequency subband coefficients according to the high-frequency subband coefficients of the visible light image and the infrared image. The second fusion module, based on the low-frequency fusion rule of domain transform filtering, calculates the fused low-frequency subband coefficients according to the low-frequency subband coefficients of the visible light image and the infrared image. The coefficient overlay module is used to overlay the fused high-frequency subband coefficients and the fused low-frequency subband coefficients to obtain a fused image.

Images