Image fusion method and system based on multi-scale transformation and sparse low-rank representation

Abstract

The invention discloses an image fusion method and system based on multi-scale transformation and sparse low-rank representation, and is applied to the technical field of information fusion, and the method comprises the steps: collecting a visible light image and an infrared image of an object, carrying out the multi-scale transformation, and decomposing the visible light image and the infrared image into a high-frequency image and a low-frequency image; fusing the high-frequency images of the visible light image and the infrared image by adopting a maximum absolute value fusion criterion to obtain a fused high-frequency image; fusing the low-frequency images of the visible light image and the infrared image by adopting a sparse low-rank representation method to obtain a fused low-frequency image; and performing inverse multi-scale transformation on the fused high-frequency image and the fused low-frequency image, and calculating to obtain a visible light and infrared image fusion result. In this way, image fusion is realized by adopting a multi-scale transformation and sparse low-rank representation method, so that more image information is reserved, and a better image fusion effect is realized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of information fusion, and in particular to an image fusion method and system based on multi-scale transformation and sparse low-rank representation. Background Art

[0002] Due to the increasing demand in transportation, security, military and other fields, a single image sensor is increasingly difficult to adapt to various emerging application scenarios. In this case, it is promising to collect information of specific scenes based on multiple image sensors and fuse images from different sensors. Since the fused image contains more detailed information, multi-sensor and image fusion technology can meet more complex needs than a single sensor. Infrared and visible light image sensors are widely used and have strong complementarity.

[0003] Visible light images usually have high spatial resolution and considerable light and dark details and contrast, but they are easily affected by adverse conditions, such as low light, fog and other bad weather. Infrared images can resist these interferences, but usually have low resolution and poor texture. Therefore, the combination of infrared and visible light sensors can adapt to a wider range of fields. At present, scholars have proposed many solutions for the fusion of infrared and visible light, including spatial domain fusion methods, transform domain fusion methods, artificial intelligence fusion methods, etc. However, the application environment of visible light and infrared image fusion is becoming more and more complex, and the fusion effect requirements are becoming higher and higher. The existing methods are difficult to meet the actual needs in terms of fusion quality.

[0004] Therefore, there is an urgent need for a technical solution that can effectively realize the fusion of visible light and infrared images. Summary of the invention

[0005] The present disclosure provides an image fusion method and system based on multi-scale transformation and sparse low-rank representation. By utilizing multi-scale transformation and adopting sparse low-rank to represent the low-frequency part and then performing image fusion, it at least solves the technical problem of low fusion quality in complex environments during the image fusion process in the prior art.

[0006] According to a first aspect of the present disclosure, there is provided an image fusion method based on multi-scale transformation and sparse low-rank representation, comprising the following steps:

[0007] Collecting visible light images and infrared images of the object, and performing multi-scale transformation to decompose the visible light and infrared images into high-frequency and low-frequency images;

[0008] The high-frequency images of the visible light image and the infrared image are fused by adopting a maximum absolute value fusion criterion to obtain a fused high-frequency image;

[0009] Using a sparse low-rank representation method to fuse the low-frequency images of the visible light image and the infrared image to obtain a fused low-frequency image;

[0010] The fused high-frequency image and the fused low-frequency image are subjected to inverse multi-scale transformation to calculate and obtain a fusion result of the visible light and infrared image.

[0011] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the low-frequency image is specifically:

[0012] L i =Down(w*I)

[0013] Among them, i represents the i-th decomposition, L i is the low-frequency part obtained by the i-th decomposition, Down represents downsampling, w is the convolution kernel, * represents convolution, and I represents infrared and visible light images.

[0014] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the high-frequency image is specifically:

[0015] H i =L i-1 -w*Up(L i ),i=1,2,...,M

[0016] Among them, H i is the high-frequency part obtained by the i-th decomposition, Up represents upsampling, and M is the total number of decompositions.

[0017] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the process of fusing the high-frequency images of the visible light image and the infrared image using the maximum absolute value fusion criterion to obtain the fused high-frequency image is specifically as follows:

[0018]

[0019] in, and is the high-frequency image obtained by the i-th decomposition of the visible light and infrared images, represents the fused high-frequency image, || is the absolute value operation, and the process is to take the absolute value of each element in the matrix.

[0020] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the process of fusing the low-frequency images of the visible light image and the infrared image using the sparse low-rank representation method to obtain the fused low-frequency image is:

[0021] A sparse representation dictionary is obtained by training using the K-SVD method, a sparse low-rank representation coefficient and a sparse low-rank representation error are defined, and a sparse low-rank representation model is constructed based on the sparse representation dictionary, the sparse low-rank representation coefficient and the sparse low-rank representation error;

[0022] The sparse low-rank representation model is solved based on the augmented Lagrangian function method and the maximum L1 norm criterion, and the sparse low-rank representation coefficients are calculated;

[0023] The product of the sparse low-rank representation coefficient and the sparse representation dictionary is calculated to obtain a fused low-frequency image.

[0024] According to the above aspects and any possible implementation, an implementation is further provided, wherein the sparse low-rank representation model is specifically:

[0025]

[0026] Among them, L M is the low-frequency image obtained by the last decomposition of infrared and visible light, D is the sparse representation dictionary, a is the sparse low-rank representation coefficient, E is the sparse low-rank representation error, α and λ are constant coefficients, ||||1, |||| * and|||| 2,1 They represent L1 norm, nuclear norm, and L21 norm respectively.

[0027] According to the above aspects and any possible implementation, an implementation is further provided, wherein the process of solving the sparse low-rank representation model based on the augmented Lagrangian function method and using the maximum L1 norm criterion to calculate the sparse low-rank representation coefficient is:

[0028]

[0029] Among them, a in and a vi are the sparse low-rank representation coefficients of infrared and visible light images, respectively, f is the sparse low-rank representation coefficient after fusion.

[0030] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the process of calculating the fused low-frequency image according to the sparse low-rank representation coefficient and the sparse representation dictionary is:

[0031] L f =Da f

[0032] Among them, L f is the fused low-frequency image.

[0033] According to the above aspects and any possible implementation, an implementation is further provided, wherein the process of performing inverse multi-scale transformation on the fused high-frequency image and the fused low-frequency image to calculate the visible light and infrared image fusion result is:

[0034]

[0035] in, is the fused image after the i-th inverse multi-scale transformation. When i=1, That is the fused low-frequency image L f , is the fused high-frequency image.

[0036] According to a second aspect of the present disclosure, there is provided an image fusion system based on multi-scale transformation and sparse low-rank representation, which is used to implement the image fusion method based on multi-scale transformation and sparse low-rank representation as in the first aspect, including: an image acquisition module, a high-frequency image fusion module, a low-frequency image fusion module and an image fusion module;

[0037] The image acquisition module is used to acquire visible light images and infrared images of objects, and perform multi-scale transformation to decompose them into high-frequency and low-frequency images;

[0038] The high-frequency image fusion module is used to fuse the high-frequency images of the visible light and infrared images by adopting the maximum absolute value fusion criterion to obtain a fused high-frequency image;

[0039] The low-frequency image fusion module is used to fuse the low-frequency images of the visible light and infrared images by using a sparse low-rank representation method to obtain a fused low-frequency image;

[0040] The image fusion module is used to perform inverse multi-scale transformation on the fused high-frequency image and the low-frequency image, and calculate the fusion result of the visible light and infrared images.

[0041] Compared with the prior art, the present invention has the following technical effects:

[0042] The present invention discloses a method and system for fusion of infrared and visible light images based on multi-scale transformation and sparse low-rank representation. Multi-scale transformation is used to decompose infrared and visible light images into low-frequency and high-frequency parts. The high-frequency part is fused using the maximum absolute value criterion. The low-frequency part is first sparsely represented with a low rank, and then fused using the maximum L1 norm criterion. Finally, an inverse multi-scale transformation is performed to obtain a fused image. After the multi-scale transformation, the present invention does not directly fuse the low-frequency part, but first performs a sparse low-rank representation and then performs fusion. This operation can further perform low-frequency image fusion in a subtle feature space, and can retain more image information, thereby effectively improving the quality of image fusion.

[0043] It should be understood that the contents described in the summary of the invention are not intended to limit the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, among which:

[0045] Figure 1 A schematic diagram of a process of an image fusion method based on multi-scale transformation and sparse low-rank representation according to an embodiment of the present disclosure is shown;

[0046] Figure 2 A schematic diagram of the structure of an image fusion system based on multi-scale transformation and sparse low-rank representation according to an embodiment of the present disclosure is shown;

[0047] Figure 3 A schematic diagram of visible light and infrared images to be fused according to an embodiment of an image fusion method based on multi-scale transformation and sparse low-rank representation according to an embodiment of the present disclosure is shown;

[0048] Figure 4 A comparison diagram of image fusion results of an image fusion method based on multi-scale transformation and sparse low-rank representation according to an embodiment of the present disclosure and ADF, CBF, RP_SR, CNN and ResNet methods is shown;

[0049] Figure 5 A schematic diagram showing the evaluation of fusion of 21 images in an image fusion method embodiment based on multi-scale transformation and sparse low-rank representation according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] Reference Figure 1As shown, this embodiment provides an image fusion method based on multi-scale transformation and sparse low-rank representation, including the following steps:

[0053] S101, collecting visible light images and infrared images of the object, and performing multi-scale transformation to decompose them into high-frequency and low-frequency images.

[0054] In this embodiment, an industrial camera and an infrared camera are used to collect a visible light image and a corresponding infrared image of the same object respectively. After the images are obtained, a multi-scale transformation is performed to decompose the infrared image and the visible light image into high-frequency and low-frequency parts, specifically:

[0055] L i =Down(w*I) (1)

[0056] Among them, i represents the i-th decomposition, L i is the low-frequency part obtained by the i-th decomposition, Down means downsampling, w is the convolution kernel, * means convolution, and I means infrared or visible light image.

[0057] The high frequency parts of infrared and visible light images are calculated as follows:

[0058] H i =L i-1 -w*Up(L i ),i=1,2,...,M (2)

[0059] Among them, H i is the high frequency part obtained by the i-th decomposition, Up represents upsampling, and M is the total number of decompositions. When i = 0, L i-1 That is the input infrared or visible light image.

[0060] S102 , using a maximum absolute value fusion criterion to fuse the high-frequency images of the visible light image and the infrared image to obtain a fused high-frequency image.

[0061] In this embodiment, the process of performing image fusion based on the maximum absolute value fusion criterion is as follows:

[0062]

[0063] in, and is the high-frequency image obtained by the i-th decomposition of the visible light and infrared images, represents the fused high-frequency image, || is the absolute value operation, and the process is to take the absolute value of each element in the matrix.

[0064] During the fusion process, and The pixel with the larger absolute value is taken as the fusion result.

[0065] S103, using a sparse low-rank representation method to fuse the low-frequency images of the visible light image and the infrared image to obtain a fused low-frequency image.

[0066] In this embodiment, by performing sparse low-rank representation on the low-frequency parts of the infrared and visible light images, sparse low-rank coefficients are obtained, and the maximum L1 norm criterion is adopted to complete the fusion of the low-frequency parts of the infrared and visible light images. The specific process is:

[0067] The model of sparse low-rank representation is as follows:

[0068]

[0069] Among them, L M is the low-frequency image obtained by the last decomposition of infrared or visible light, D is the sparse representation dictionary, a is the sparse low-rank representation coefficient, E is the sparse low-rank representation error, which is affected by noise, α and λ are constant coefficients, ||||1, |||| * and|||| 2,1 They represent L1 norm, nuclear norm, and L21 norm respectively. The L1 norm is used to optimize the sparsity of a, the nuclear norm is used to ensure the low rank characteristics of a, and the L21 norm is used to optimize the reconstruction error of sparse low rank representation.

[0070] The dictionary D in this embodiment is pre-trained by the KSVD method, and then equation (4) can be solved by the augmented Lagrangian function method. The sparse low-rank representation coefficients are obtained by solving the equation (4) model, and the maximum L1 norm criterion is adopted. The specific fusion method is as follows:

[0071]

[0072] Among them, a in and a vi are the sparse low-rank representation coefficients of infrared and visible light images, respectively, f is the sparse low-rank representation coefficient after fusion. Then the fused low-frequency image is obtained by the following formula:

[0073] L f =Da f (6)

[0074] S104, performing an inverse multi-scale transformation on the fused high-frequency image and the low-frequency image, and calculating and obtaining a fusion result of the visible light and infrared images.

[0075] In this embodiment, the specific calculation method of the inverse multi-scale transformation is:

[0076]

[0077] in, is the fused image after the i-th inverse multi-scale transformation. When i=1, That is the fused low-frequency image L f , is the fused high-frequency image.

[0078] After performing M inverse transformations, the fused image can be obtained.

[0079] like Figure 2 As shown, this embodiment also provides an image fusion system based on multi-scale transformation and sparse low-rank representation, including: an image acquisition module 1, a high-frequency image fusion module 2, a low-frequency image fusion module 3 and an image fusion module 4;

[0080] The image acquisition module 1 is used to acquire visible light images and infrared images of objects, and perform multi-scale transformation to decompose them into high-frequency and low-frequency images;

[0081] The high-frequency image fusion module 2 is used to fuse the high-frequency images of the visible light and infrared images by adopting the maximum absolute value fusion criterion to obtain a fused high-frequency image;

[0082] The low-frequency image fusion module 3 is used to fuse the low-frequency images of the visible light and infrared images by using a sparse low-rank representation method to obtain a fused low-frequency image;

[0083] The image fusion module 4 is used to perform inverse multi-scale transformation on the fused high-frequency image and the low-frequency image, and calculate the fusion result of the visible light and infrared images.

[0084] Example

[0085] This embodiment is a visible light and infrared image fusion experiment carried out by the present invention on the public dataset VIFB. Figure 3 The infrared image to be fused is shown in Figure 1. The image resolution is 512×384, where: Figure 3 (a) is a visible light image. Figure 3(b) is an infrared image. At the same time, this embodiment will be compared with anisotropic diffusion-based image fusion (ADF), cross bilateral filter fusion method (CBF), low-pass pyramid and sparse representation (RP_SR), convolutional neural network (CNN) and ResNet to verify the effectiveness of the present invention. Among them, the first two methods belong to spatial domain fusion methods, the third method belongs to transform domain fusion methods, and the last two methods belong to artificial intelligence methods. Figure 4 (a), (b), (c), (d), (e), and (f) are the fusion result diagrams of ADF, CBF, RP_SR, CNN, ResNet and the present invention respectively. It can be seen that the fusion effect of the present invention on street lamps, vehicles, sheds, etc. is significantly better than that of other methods.

[0086] Further comparative verification is given:

[0087] In order to compare the image effects of the embodiments, five evaluation indicators, including cross entropy (CE), mutual information (MI), edge based similarity measurement (QAB / F), Chen-Varshney metric (QCV) and standard deviation (SD), were fused for evaluation, among which the smaller the CE and QCV, the better, and the larger the others, the better.

[0088] Table 1 shows the average evaluation results of all 21 visible light and infrared images in the VIFB dataset for fused images, where "↓" means the smaller the better. It can be seen that the method of the present invention is superior to other methods in all four evaluation indicators. In the SD evaluation standard, the result of the present invention is second only to CNN and is very close to that of CNN. Therefore, it can be considered that the present invention has significantly improved the image fusion quality compared with other methods.

[0089] Table 1 Evaluation results of different image fusion methods on the VIFB dataset

[0090]

[0091]

[0092] To further analyze the image fusion results, Figure 5 The fusion evaluation results of various methods on each image are plotted. Figure 5 (a), (b), (c), (d), and (e) are the evaluation results of CE, MI, QAB / F, QCV, and SD, respectively. It can be seen from the results that in the first four evaluation indicators, the performance of the present invention on most image pairs is better than that of other methods. In SD, some images have the best evaluation indicators, and the evaluation results of other images are second only to CNN, and the gap is small. Therefore, it can be seen that the image fusion quality of the present invention method is significantly improved compared with other methods.

[0093] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present disclosure is not limited by the order of the actions described, because according to the present disclosure, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present disclosure.

[0094] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0095] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image fusion method based on multi-scale transformation and sparse low-rank representation, characterized in that: The following steps are involved: Collecting visible light images and infrared images of the object, and performing multi-scale transformation to decompose the visible light and infrared images into high-frequency and low-frequency images; The high-frequency images of the visible light image and the infrared image are fused by adopting a maximum absolute value fusion criterion to obtain a fused high-frequency image; Using a sparse low-rank representation method to fuse the low-frequency images of the visible light image and the infrared image to obtain a fused low-frequency image; The fused high-frequency image and the fused low-frequency image are subjected to inverse multi-scale transformation to calculate and obtain a fusion result of the visible light and infrared image.

2. The image fusion method based on multi-scale transformation and sparse low-rank representation according to claim 1, characterized in that: The low-frequency image is specifically: L i =Down(w*I) Among them, i represents the i-th decomposition, L i is the low-frequency part obtained by the i-th decomposition, Down represents downsampling, w is the convolution kernel, * represents convolution, and I represents infrared and visible light images.

3. The image fusion method based on multi-scale transformation and sparse low-rank representation according to claim 1, characterized in that: The high-frequency image is specifically: H i =L i-1 -w*Up(L i ),i=1,2,...,M Among them, H i is the high-frequency part obtained by the i-th decomposition, Up represents upsampling, and M is the total number of decompositions.

4. The image fusion method based on multi-scale transformation and sparse low-rank representation according to claim 1, characterized in that: The process of fusing the high-frequency images of the visible light image and the infrared image using the maximum absolute value fusion criterion to obtain the fused high-frequency image is specifically as follows: in, and is the high-frequency image obtained by the i-th decomposition of the visible light and infrared images, represents the fused high-frequency image, || is the absolute value operation, and the process is to take the absolute value of each element in the matrix.

5. The image fusion method based on multi-scale transformation and sparse low-rank representation according to claim 1, characterized in that: The process of fusing the low-frequency images of the visible light and infrared images using the sparse low-rank representation method to obtain the fused low-frequency image is as follows: A sparse representation dictionary is obtained by training using the K-SVD method, a sparse low-rank representation coefficient and a sparse low-rank representation error are defined, and a sparse low-rank representation model is constructed based on the sparse representation dictionary, the sparse low-rank representation coefficient and the sparse low-rank representation error; The sparse low-rank representation model is solved based on the augmented Lagrangian function method and the maximum L1 norm criterion, and the sparse low-rank representation coefficients are calculated; A fused low-frequency image is obtained by calculation according to the sparse low-rank representation coefficients and the sparse representation dictionary.

6. The image fusion method based on multi-scale transformation and sparse low-rank representation according to claim 5, characterized in that: The sparse low-rank representation model is specifically: Among them, L M is the low-frequency image obtained by the last decomposition of infrared and visible light, D is the sparse representation dictionary, a is the sparse low-rank representation coefficient, E is the sparse low-rank representation error, α and λ are constant coefficients, ||||1, |||| * and|||| 2,1 They represent L1 norm, nuclear norm, and L21 norm respectively.

7. The image fusion method based on multi-scale transformation and sparse low-rank representation according to claim 5, characterized in that: The process of solving the sparse low-rank representation model based on the augmented Lagrangian function method and adopting the maximum L1 norm criterion to calculate the sparse low-rank representation coefficients is: Among them, a in and a vi are the sparse low-rank representation coefficients of infrared and visible light images, respectively, f is the sparse low-rank representation coefficient after fusion.

8. The image fusion method based on multi-scale transformation and sparse low-rank representation according to claim 7, characterized in that: The process of calculating the fused low-frequency image according to the sparse low-rank representation coefficient and the sparse representation dictionary is: L f =From f Among them, L f is the fused low-frequency image.

9. The image fusion method based on multi-scale transformation and sparse low-rank representation according to claim 1, characterized in that: The process of performing inverse multi-scale transformation on the fused high-frequency image and the fused low-frequency image to calculate the visible light and infrared image fusion result is as follows: in, is the fused image after the i-th inverse multi-scale transformation. When i=1, That is the fused low-frequency image L f , is the fused high-frequency image.

10. An image fusion system based on multi-scale transformation and sparse low-rank representation, used to implement the image fusion method based on multi-scale transformation and sparse low-rank representation as claimed in any one of claims 1 to 9, characterized in that: include: An image acquisition module (1), a high-frequency image fusion module (2), a low-frequency image fusion module (3) and an image fusion module (4); The image acquisition module (1) is used to acquire visible light images and infrared images of objects, and perform multi-scale transformation to decompose them into high-frequency and low-frequency images; The high-frequency image fusion module (2) is used to fuse the high-frequency images of the visible light image and the infrared image using a maximum absolute value fusion criterion to obtain a fused high-frequency image; The low-frequency image fusion module (3) is used to fuse the low-frequency images of the visible light and infrared images using a sparse low-rank representation method to obtain a fused low-frequency image; The image fusion module (4) is used to perform inverse multi-scale transformation on the fused high-frequency image and the low-frequency image, and calculate and obtain the visible light and infrared image fusion results.

Images