Infrared and visible light image fusion method and system based on low-rank sparse representation

Through sparse consistency low-rank representation decomposition and deep neural network feature extraction, the problem of uncaptured spatial consistency relationship in infrared and visible light images is solved, efficient image fusion is achieved, and fusion effect and efficiency are improved.

CN114648475BActive Publication Date: 2024-11-26TAISHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210246475.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-14
Publication Date
2024-11-26
Estimated Expiration
2042-03-14

AI Technical Summary

Technical Problem

The existing infrared and visible image fusion method based on potential low rank representation fails to effectively capture the spatial consistency relationship between images, resulting in poor fusion effect.

Method used

The potential low-rank representation decomposition model of sparse consistency is used to decompose infrared and visible light images, combine deep neural networks for feature extraction and fusion, and use alternating direction multiplier method and L2,1 norm regularization constraints to synchronize the low image. rank consensus is represented and fused through weighted sum and maximum selection strategies.

Benefits of technology

The clarity and hierarchy of the fusion image are improved, prominent areas in the scene are highlighted, and algorithm operation efficiency and fusion effect are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114648475B_ABST
    Figure CN114648475B_ABST
Patent Text Reader

Abstract

The present invention provides an infrared and visible light image fusion method based on low-rank sparse representation, the method comprising: taking infrared and visible light images as different task inputs and decomposing them into a significant part and a basic part by a sparse consistent potential low-rank representation decomposition module; using a deep neural network model to further mine the decomposed basic part to extract more effective fusion features; in the image fusion module, applying different fusion strategies to the basic part and the significant part to obtain a final fused image. The present invention also provides an infrared and visible light image fusion system based on low-rank sparse representation. The final fused image obtained by the present invention utilizes the complementary information of multi-modal sensor image data, so that the final fused image obtained has high clarity and distinct layers, and the significant areas and targets in the scene are relatively more prominent.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a method and system for fusion of infrared and visible light images based on low-rank sparse representation. Background Art

[0002] Image fusion is an image enhancement technology that aims to combine images from different sensors to generate fused images with complementary information and significant visual effects, so that it has better target salient information and comprehensive detailed background information. Therefore, it can provide reliable and information-rich images for advanced visual tasks including target detection, computer-aided diagnosis, semantic segmentation, etc.

[0003] Generally, image fusion methods can be divided into three categories: (1) methods based on multi-scale decomposition; (2) methods based on sparse representation; and (3) methods based on deep learning. The multi-scale decomposition method is the most widely studied and applied method in the field of image fusion. It can decompose the image into low-frequency sub-bands and high-frequency sub-bands, and then perform weighted fusion. However, since they are not translationally invariant, they are prone to pseudo-Gibbs phenomenon. The key to sparse representation methods is how to construct redundant dictionaries, and dictionary learning is extremely time-consuming. In deep learning-based fusion methods, the deep features of the source image are mainly used to generate the fused image, but image decomposition is rarely paid attention to.

[0004] In order to obtain a fused image with rich image information and good visual effects, the latent low-rank representation (LatLRR) method has been widely used in the field of infrared and visible light image fusion because it can effectively decompose the salient part and the basic part of the image. The algorithm based on LatLRR focuses on decomposing the image row, and uses the average and summation strategy to solve the basic part and the salient part of the decomposition to obtain the final fused image. At the same time, due to the rise of deep learning, the method combining LatLRR with deep neural network has also achieved great success.

[0005] However, the existing LatLRR-based image fusion methods do not take into account that infrared and visible light images are obtained by different sensors at the same location. When performing image fusion, the infrared and visible light images are decomposed separately, ignoring the spatial consistency relationship between them, resulting in the inability to effectively capture more feature information. Therefore, how to obtain an image with better fusion effect is a major challenge in the field of infrared and visible light image fusion. Summary of the invention

[0006] In view of the above technical problems, the technical solution adopted by the present invention is:

[0007] The present invention provides a method for fusion of infrared and visible light images based on low-rank sparse representation.

[0008] An embodiment of the present invention provides a method for fusion of infrared and visible light images based on low-rank sparse representation, comprising the following steps:

[0009] S10, using the sparse consistent latent low-rank representation decomposition model to decompose the infrared image and the visible light image respectively;

[0010] The sparse consistent potential low-rank representation decomposition model satisfies the following conditions:

[0011] (1)

[0012] (2)stX m =X m Z m +L m X m +E m ;

[0013] Among them, X m represents the source image, Z m Represents the low-rank matrix of the image, L m Represents the saliency matrix of the image, E m represents the noise part of the image, m=1, 2, representing infrared image and visible light image respectively; ||Z m || * Represents Z m The nuclear norm, ||L m || * Indicates L m The nuclear norm of ||Z|| 2,1 It means that the low-rank matrix of the infrared image and the low-rank matrix of the visible light image are simultaneously L 2,1 Norm regularization constraint; ||E m || 2,1 Indicates that the noise part of the infrared image and the noise part of the visible light are L 2,1 norm regularization constraint; γ is the balance noise parameter, α is the regularization balance coefficient;

[0014] S20, using the alternating direction multiplier method to solve the sparse consistent potential low-rank representation decomposition model, respectively obtain the basic part F of the infrared image a1 and the significant part F b1 And the basic part of the visible light image F a2 and the significant part F b2 ;

[0015] S30, using the deep neural network model to obtain F a1and F a2 Perform deep feature extraction and fusion to obtain the basic part fusion image Fa;

[0016] S40, F b1 and F b2 Fusion is performed to obtain the salient part fusion image F b ;

[0017] S50, Fa and F b Fusion is performed to obtain an infrared and visible light fused image.

[0018] Another embodiment of the present invention provides an infrared and visible light image fusion system based on low-rank sparse representation, comprising: an image decomposition module, a feature extraction module and an image fusion module;

[0019] The image decomposition module is used to decompose the infrared image and the visible light image respectively by using the sparse consistent potential low-rank representation decomposition model, and solve the sparse consistent potential low-rank representation decomposition model by using the alternating direction multiplier method to obtain the basic part F of the infrared image respectively. a1 and the significant part F b1 And the basic part of the visible light image F a2 and the significant part F b2 ;

[0020] The sparse consistent potential low-rank representation decomposition model satisfies the following conditions:

[0021]

[0022] xT m =X m Z m +L m X m +E m ;

[0023] Among them, X m represents the source image, Z m Represents the low-rank matrix of the image, L m Represents the saliency matrix of the image, E m represents the noise part of the image, m=1, 2, representing infrared image and visible light image respectively; ||Z m || * Represents Z m The nuclear norm, ||L m || * Indicates L m The nuclear norm of ||Z|| 2,1 It means that the low-rank matrix of the infrared image and the low-rank matrix of the visible light image are simultaneously L 2,1Norm regularization constraint; ||E m || 2,1 Indicates that L is applied to the noise part of the infrared image and the noise part of the visible light respectively. 2,1 norm regularization constraint; γ is the balance noise parameter, α is the regularization balance coefficient;

[0024] The feature extraction module is used to use the deep neural network model to obtain the F a1 and F a2 Perform deep feature extraction and fusion to obtain the basic part fusion image Fa;

[0025] The image fusion module is used to transform F b1 and F b2 Fusion is performed to obtain the salient part fusion image F b , and Fa and F b Fusion is performed to obtain an infrared and visible light fused image.

[0026] The infrared and visible light image fusion method and system based on low-rank sparse representation provided by the embodiment of the present invention firstly decomposes the infrared and visible light images as different task inputs by the sparse consistent potential low-rank representation decomposition module. In the decomposition process, the L2,1 norm is used to constrain the rank to keep it sparsely consistent, and the low-rank consensus representation of the infrared and visible light images is obtained synchronously; secondly, the deep neural network model is used to further mine the basic part to extract more effective fusion features; finally, in the image fusion module, different fusion strategies are applied to the basic part and the significant part, wherein the significant part uses a weighted summation fusion strategy, and the basic part is fused by the maximum selection strategy, and the final basic part and the significant part are summed to obtain the final fused image. In other words, the final fused image obtained by the present invention utilizes the complementary information of the multimodal sensor image data, so that the final fused image obtained has high clarity and distinct layers, and the significant areas and targets in the scene are relatively more prominent. In addition, the synchronous low-rank decomposition of multimodal images can improve the operation efficiency of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic diagram of a flow chart of a method for fusion of infrared and visible light images based on low-rank sparse representation provided in an embodiment of the present invention;

[0029] Figure 2a to Figure 2c This is a fused image diagram obtained by using the infrared and visible light image fusion method based on low-rank sparse representation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0031] Figure 1 The following is a flow chart of a method for fusion of infrared and visible light images based on low-rank sparse representation provided by an embodiment of the present invention. Figure 1 As shown, an infrared and visible light image fusion method based on low-rank sparse representation provided by an embodiment of the present invention may include the following steps:

[0032] S10, the infrared image and the visible light image are decomposed respectively using the sparse consistent latent low-rank representation decomposition model.

[0033] In this embodiment of the present invention, the sparse consistent potential low-rank representation decomposition model satisfies the following conditions:

[0034] (1)

[0035] (2)stX m =X m Z m +L m X m +E m ;

[0036] Among them, X m represents the source image, Z m Represents the low-rank matrix of the image, L m Represents the saliency matrix of the image, E m represents the noise part of the image, m=1, 2, representing infrared image and visible light image respectively; ||Z m || * Represents Z m The nuclear norm, ||L m || * Indicates L m The nuclear norm of ||Z|| 2,1 It means that the low-rank matrix of the infrared image and the low-rank matrix of the visible light image are simultaneously L 2,1 Norm regularization constraint; ||E m ||2,1 Indicates that L is applied to the noise part of the infrared image and the noise part of the visible light respectively. 2,1 norm regularization constraint; γ is the balanced noise parameter, used to balance the impact of noise. α is the regularization balance coefficient, used to constrain Z m and L m γ and α may be empirical values. In an exemplary embodiment, the value ranges of γ and α are both [0.001, 0.01, 0.1, 1, 10, 100]. Preferably, γ and α are both equal to 100.

[0037] Through the above conditions (1) and (2), the basis part X of each image can be obtained m Z m , significant part L m X m and the noise part E m .

[0038] In an embodiment of the present invention, the potential low-rank representation method is improved. Through the improved Scc-LatLRR method, the common information between different modal data is fully utilized, and the sparse consistency is partially decomposed to decompose the dual-rank (Z, L) part of the input image. The present invention imposes sparse consistency constraints on the matrix Z in the dual-rank, because for the matrix Z, more spatial consistency information between matrices can be obtained through constraints. The reason for not imposing constraints on L is that when the matrix Z obtains certain spatial consistency information, because the amount of information of the image itself is limited, the constraint method of Scc-LatLRR is X=XZ+LX+E, so the amount of information of the significant part will increase, and in image fusion, the significant part is the key part.

[0039] S20, using the alternating direction multiplier method to solve the sparse consistent potential low-rank representation decomposition model, respectively obtain the basic part F of the infrared image a1 and the significant part F b1 And the basic part of the visible light image F a2 and the significant part F b2 .

[0040] In the model training stage, the present invention uses the Alternating Direction Method of Multipliers (ADMM) to optimize the potential low-rank representation decomposition model of sparse consistency. Specifically, by constructing an augmented Lagrangian function, the large global problem can be decomposed into multiple smaller and easier to solve local sub-problems, and the large global function solution can be obtained by coordinating the solutions of the sub-problems. The specific solution process may include:

[0041] S201, obtaining an augmented Lagrangian function model defined by the following conditions (3) and (4):

[0042]

[0043] xT m =X m S m +K m X m +E m , Z m =J m , Z m =S m , L m =K m (4)

[0044] S202, iteratively update each variable according to the following conditions (5) to (11) until the convergence condition is met:

[0045]

[0046]

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] Wherein, k is the number of iterations, the value of k ranges from 1 to N, N is the total number of iterations, ρ is the iteration step, preferably, ρ = 1.1. k is the balance factor at the kth iteration. The initial value u is set to 0.0001 and increases slowly with the iteration until u max =10 10 Then stop. is the augmented Lagrange multiplier at the kth iteration; I is the identity matrix, X m ′ represents X m The transposed matrix of G is a matrix formed by pulling the low-rank matrix obtained from a single image into rows and connecting them together. n represents the matrix dimension.

[0053]

[0054]

[0055] R is Z k+1 Each 1*n in 2The n*n matrix is ​​reconstructed from the row vector of dimension. Specifically, after L 2,1 Norm regularization constraints to obtain Z k+1 After that, Reconstruct Z k+1 The low-rank matrix of the row vector is reconstructed into a matrix of dimension n*n, which is

[0056] In an embodiment of the present invention, The closed-form solution of can be decomposed using SVD, and the closed-form solution obtained is

[0057] Similarly, The closed-form solution of The above condition (8) can be obtained through the following process:

[0058]

[0059] right Taking the partial derivative and setting it equal to 0, we get:

[0060]

[0061] Similarly, the above condition (9) can be solved by The partial derivatives are solved to obtain

[0062]

[0063] S203, update the augmented Lagrange multiplier according to the following conditions (12)-(15):

[0064]

[0065]

[0066]

[0067]

[0068] S304: Update the balance factor according to the following condition (16):

[0069] u k+1 =min(ρu k ,u max ) (16).

[0070] In the embodiment of the present invention, preferably, u max =10 10 .

[0071] Furthermore, in an embodiment of the present invention, the convergence condition satisfies:

[0072]

[0073]

[0074]

[0075] ε is a preset value, preferably, ε = 10 -6 .

[0076] Furthermore, in the embodiment of the present invention, Z m , J m , S m , L m , K m 、E m , W m 、M m , Y m and V m The initial value of is set to 0.

[0077] S30, using the deep neural network model to obtain F a1 and F a2 Perform deep feature extraction and fusion to obtain the basic part fusion image Fa.

[0078] The basis part decomposed by the LatLRR method is similar to the smoothed version of the source image, which contains the most effective image information. In order to better integrate it, the present invention introduces a deep neural network model for further feature mining.

[0079] In an illustrative embodiment of the present invention, the deep neural network model may be a convolutional neural network VGG model. The convolution group can extract the features of the image through convolution operations, and as the number of convolution groups increases, more abstract features can be extracted. Since the size of the feature map output by the 5th convolution of the VGG19 convolutional neural network is too different from the source image detail content map, the output of the first four convolution groups is selected as the basis for constructing the weight map in the present invention, that is, the first four convolution groups are mainly selected to fuse the basic parts of the infrared image and the visible light image.

[0080] Further, S30 may include:

[0081] S301, F a1 and F a2 Input into the convolutional neural network VGG model.

[0082] In an embodiment of the present invention, the convolutional neural network VGG model may be a trained model. The training of the convolutional neural network VGG model may adopt existing technologies.

[0083] S302, obtaining a multi-channel feature map output by convolution group i in the convolutional neural network VGG model The value of i ranges from 1 to 4, and k is the number of channels of the output feature map of convolution group i;

[0084] S302: Obtain a single channel feature map of convolution group i according to the following conditions (17) and (18): and

[0085]

[0086]

[0087] and Respectively and The above conditions (17) and (18) are solved by using the L1 norm and Compress to obtain a single channel feature map and S303, according to the following conditions (19) and (20): and Perform smoothing to obtain a single-channel feature map after smoothing and

[0088]

[0089]

[0090] Here, r is the set area size, preferably, r=1.

[0091] The above conditions (19) and (20) are used to perform regional mean operation, that is, to smooth the single-channel feature map, which can make the fused image more natural.

[0092] S304: Obtain a first infrared weighted map according to the following conditions (21) and (22): and the first visible light weight map

[0093]

[0094]

[0095] The weights obtained by the above conditions (21) and (22) are the initial normalized weights.

[0096] S305: Obtain a second infrared weighted map according to the following conditions (23) and (24): and the second visible light weight map

[0097]

[0098]

[0099] Since the pooling layer of the convolutional neural network is a data downsampling operation, the image size will become smaller during the continuous iteration process. In order to obtain a consistent image fusion size, it is necessary to use upsampling to adjust the weight mapping. The size of , that is, through the above conditions (23) and (24), the image has the same size as the original part of the decomposed image.

[0100] S306, obtaining a fused image obtained by convolution group i Four fused images are obtained.

[0101] S307, according to the following conditions (25) to (27), obtain and

[0102]

[0103]

[0104]

[0105] in, Represents the fused image The amount of source image information contained, that is, the size of the index value, represents the amount of source image information in the fused image, and is used to measure the information correlation between the source image and the fused image. Represent the fused images The amount of information contained in the infrared image and the visible light image. P(X1), P(X2) and Represent the infrared source image X1, the visible light source image X2 and the fused image respectively Normalized marginal histogram of . and Represent the infrared source image X1, the visible light source image X2 and the fused image respectively Normalized joint grayscale histogram of .

[0106] S308, obtain And MI max The corresponding fused image is taken as Fa.

[0107] A large MI value indicates that the fused image of the source image will contain more source image information and the fusion performance will be improved. The MI value can be used to obtain the final basic part Fa according to the maximum selection principle.

[0108] In this embodiment, a convolutional neural network model is used to perform deep feature extraction on the basic parts of the two images decomposed in the previous step. Since the basic part information contains the consistency information of the two images, further mining it can enhance the fusion effect of the basic part image to a certain extent.

[0109] In another embodiment of the present invention, the deep neural network model may adopt a residual network (ResNet 50) model.

[0110] S40, F b1 and F b2 Fusion is performed to obtain the salient part fusion image F b .

[0111] Those skilled in the art will b1 and F b2 Fusion is performed to obtain the salient part fusion image F b It may be a prior art. For example, in an exemplary embodiment, F b =F b1 +F b2 In another exemplary embodiment, F b =k1*F b1 +k2*F b2 , k1 and k2 are F b1 and F b2 The weight coefficient of .

[0112] S50, Fa and F b Fusion is performed to obtain an infrared and visible light fused image.

[0113] In the embodiment of the present invention, a weighted summation strategy can be used for image fusion, that is, F = F a +F b .

[0114] [Example]

[0115] In the embodiment of the present invention, the infrared and visible light fusion dataset uses TNO, and its link is https: / / figshare.com / articles / dataset / TNO_Image_Fusion_Dataset / 1008029. According to the image quality and the frequency of appearance in the paper, 21 pairs of infrared and visible light images are selected as the original images. The present invention uses the evaluation indicators CE, EN, MI, FMIdct and Qabf based on information theory, the human visual perception evaluation index VIF based on information theory, the evaluation index SCD based on source and generated images, and the human visual sensitivity evaluation index Qcb for qualitative evaluation. Except for CE, a larger value usually means a better fusion effect.

[0116] All experiments were conducted in Matlab 2018a and Windows 10 system, Intel Core i5-8400 CPU and 32-GB RAM. Figure 2a and Figure 2b Taking the image shown as an example (corresponding to Figure 14 in Table 1, with a size of 768×576), the running time of the processing using the existing LatLRR method and the improved LatLRR method (Scc-LatLRR method) of the present invention can be shown in Table 1. It can be seen from Table 1 that in the existing LatLRR method, the running time is 164.263553 seconds, while the running time of the Scc-LatLRR method of the present invention is 64.060377 seconds, saving nearly three times the running time. It can be seen from the average running time that the method of the present invention can achieve higher running efficiency. The fused image obtained by the method of the present invention can be shown as follows Figure 2c The evaluation values ​​of the images processed by the LatLRR method and the improved Scc-LatLRR method of the present invention can be shown in Table 2. Referring to Table 2, it can be seen that the evaluation indicators of the images obtained by the present invention are better than those of the LatLRR method.

[0117] Table 1: Dataset image running time

[0118]

[0119]

[0120] Table 2: Evaluation index values

[0121] method EN MI VIF Qabf FMIdct SCD QUR CE LatLRR 6.19917 12.39833 0.30237 0.34063 0.30742 1.60412 0.48911 1.50908 SccLatLRR 6.76293 13.52587 0.89563 0.48190 0.40875 1.69510 0.51472 1.30792

[0122] The application scenarios of the infrared and visible light image fusion method based on low-rank sparse representation provided by the embodiment of the present invention may include military reconnaissance, medical diagnosis, artificial intelligence, etc. Taking military reconnaissance as an example, the fusion method proposed by the present invention is used to fuse infrared and visible light images, which can provide combat personnel with a true and reliable basis for judging targets, thereby improving combat efficiency.

[0123] Another embodiment of the present invention further provides an infrared and visible light image fusion system based on low-rank sparse representation, comprising: an image decomposition module, a feature extraction module and an image fusion module;

[0124] The image decomposition module is used to decompose the infrared image and the visible light image respectively by using the sparse consistent potential low-rank representation decomposition model, and solve the sparse consistent potential low-rank representation decomposition model by using the alternating direction multiplier method to obtain the basic part F of the infrared image respectively. a1 and the significant part F b1 And the basic part of the visible light image F a2 and the significant part F b2 ;

[0125] The sparse consistent potential low-rank representation decomposition model satisfies the following conditions:

[0126]

[0127] xT m =X m Z m +L m X m +E m ;

[0128] Among them, X m represents the source image, Z m Represents the low-rank matrix of the image, L m Represents the saliency matrix of the image, E m represents the noise part of the image, m=1, 2, representing infrared image and visible light image respectively; ||Z m || * Represents Z m The nuclear norm, ||L m || * Indicates L m The nuclear norm of ||Z|| 2,1 It means that the low-rank matrix of the infrared image and the low-rank matrix of the visible light image are simultaneously L 2,1 Norm regularization constraint; ||E m || 2,1 Indicates that L is applied to the noise part of the infrared image and the noise part of the visible light respectively.2,1 norm regularization constraint; γ is the balance noise parameter, α is the regularization balance coefficient;

[0129] The feature extraction module is used to use the deep neural network model to obtain the F a1 and F a2 Perform deep feature extraction and fusion to obtain the basic part fusion image Fa;

[0130] The image fusion module is used to transform F b1 and F b2 Fusion is performed to obtain the salient part fusion image F b , and Fa and F b Fusion is performed to obtain an infrared and visible light fused image.

[0131] Furthermore, the image decomposition module is specifically used to perform the following operations:

[0132] S1, obtain the augmented Lagrangian function model defined by the following conditions:

[0133]

[0134] xT m =X m S m +K m X m +E m , Z m =J m , Z m =S m , L m =K m

[0135] S2, iteratively update each variable according to the following conditions until the convergence condition is met:

[0136]

[0137]

[0138]

[0139]

[0140]

[0141]

[0142]

[0143] Where k is the number of iterations, the value of k ranges from 1 to N, N is the total number of iterations, ρ is the iteration step length, u k is the balance factor at the kth iteration, is the augmented Lagrange multiplier at the kth iteration; I is the identity matrix, X m ′ represents X m The transposed matrix of

[0144]

[0145]

[0146] R is Z k+1 Each 1*n in 2 The n*n matrix is ​​reconstructed from the row vector of dimension .

[0147] S3, update the augmented Lagrange multiplier according to the following conditions:

[0148]

[0149]

[0150]

[0151]

[0152] S4, update the balance factor according to the following conditions:

[0153] u k+1 =min(ρu k , μ max )

[0154] Furthermore, the convergence condition satisfies:

[0155]

[0156]

[0157]

[0158] ε is the default value.

[0159] Furthermore, the deep neural network model may be a convolutional neural network VGG model.

[0160] Furthermore, the image fusion module is specifically used to perform the following operations:

[0161] S11, F a1 and F a2 Input into the convolutional neural network VGG model;

[0162] S12, obtain the multi-channel feature map output by convolution group i in the convolutional neural network VGG model The value of i ranges from 1 to 4, and k is the number of channels of the output feature map of convolution group i;

[0163] S13, obtain the single channel feature map of convolution group i according to the following conditions and

[0164]

[0165]

[0166] and Respectively and The value at position (x, y);

[0167] S14, according to the following conditions and Perform smoothing to obtain a single-channel feature map after smoothing and

[0168]

[0169]

[0170] Among them, r is the set area size;

[0171] S15, obtaining a first infrared weighted map according to the following conditions and the first visible light weight map

[0172]

[0173]

[0174] S16, obtaining a second infrared weighted image according to the following conditions and the second visible light weight map

[0175] c∈{0,1,...(2 i-1 -1)}

[0176] c∈{0,1,...(2 i-1 -1)}

[0177] S17, obtain the fused image obtained by convolution group i Four fused images are obtained.

[0178] S18, obtain according to the following conditions and

[0179]

[0180]

[0181]

[0182] in, Represents the fused image The amount of source image information contained in Represent the fused images The amount of information contained in the infrared image and the visible light image; P(X1), P(X2) and Represent the infrared source image X1, the visible light source image X2 and the fused image respectively Normalized marginal histogram of ; and Represent the infrared source image X1, the visible light source image X2 and the fused image respectively Normalized joint grayscale histogram of ;

[0183] S19, obtain And MI max The corresponding fused image is taken as Fa.

[0184] In summary, the infrared and visible light image fusion method and system based on low-rank sparse representation provided by the embodiments of the present invention have at least the following advantages:

[0185] (1) By utilizing the complementary information of multimodal sensor image data, a latent low-rank representation multimodal image fusion framework based on sparse consistency constraints is developed, and deep feature information is deeply mined through VGG to obtain better image fusion effect. Different from the previous method of using LatLRR to decompose infrared and visible light images separately, the method proposed in this invention can better utilize the two image modalities through synchronous low-rank decomposition, which helps to obtain the common parts of the basic parts of different sensor images in the same scene.

[0186] (2) Considering that the basic part in the potential low-rank representation decomposition is similar to a smoothed version of the source image and has more common information between images, in order to obtain effective fusion information, a feature extraction algorithm based on the VGG neural network is selected, which can realize the deep extraction of two heterogeneous modal features.

[0187] (3) Synchronous low-rank decomposition of multimodal images can improve image processing efficiency.

[0188] Although some specific embodiments of the present invention have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It should also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for fusion of infrared and visible light images based on low-rank sparse representation, characterized in that: The following steps are involved: S10, using the sparse consistent latent low-rank representation decomposition model to decompose the infrared image and the visible light image respectively; The sparse consistent potential low-rank representation decomposition model satisfies the following conditions: (1) (2)s.t.X m =X m Z m +L m X m +E m ; Among them, X m represents the source image, Z m Represents the low-rank matrix of the image, L m Represents the saliency matrix of the image, E m represents the noise part of the image, m = 1, 2, representing infrared image and visible light image respectively; ‖Z m ‖ * Represents Z m The nuclear norm, ‖L m ‖ * Indicates L m The nuclear norm of 2,1 It means that the low-rank matrix of the infrared image and the low-rank matrix of the visible light image are simultaneously L 2,1 Norm regularization constraint; ‖E m ‖ 2,1 Indicates that L is applied to the noise part of the infrared image and the noise part of the visible light respectively. 2,1 norm regularization constraint; γ is the balance noise parameter, α is the regularization balance coefficient; S20, using the alternating direction multiplier method to solve the sparse consistent potential low-rank representation decomposition model, respectively obtain the basic part F of the infrared image a1 and the significant part F b1 And the basic part of the visible light image F a2 and the significant part F b2 ; S30, using the deep neural network model to obtain F a1 and F a2 Perform deep feature extraction and fusion to obtain the basic part fusion image Fa; S40, F b1 and F b2 Fusion is performed to obtain the salient part fusion image F b ; S50, Fa and F b Fusion is performed to obtain an infrared and visible light fusion image; The S20 further includes: S201, obtaining an augmented Lagrangian function model defined by the following conditions (3) and (4): s.t.X m =X m S m +K m X m +E m ,Z m =J m ,Z m =S m ,L m =K m (4) S202, iteratively update each variable according to the following conditions (5) to (11) until the convergence condition is met: Where k is the number of iterations, the value of k ranges from 1 to N, N is the total number of iterations, ρ is the iteration step length, u k is the balance factor at the kth iteration, is the augmented Lagrange multiplier at the kth iteration; I is the identity matrix, X m ′ represents X m The transposed matrix of R is Z k+1 Each 1*n in 2 The n*n matrix is ​​reconstructed from the row vector of dimension; S203, update the augmented Lagrange multiplier according to the following conditions (12)-(15): S204, update the balance factor according to the following condition (16): you k+1 =min(ρu k ,m max ) (16).

2. The method according to claim 1, characterized in that The convergence condition is satisfied: ε is the default value.

3. The method according to claim 1, characterized in that In S30, the deep neural network model is a convolutional neural network VGG model.

4. The method according to claim 3, characterized in that S30 further includes: S301, F a1 and F a2 Input into the convolutional neural network VGG model; S302, obtaining a multi-channel feature map output by convolution group i in the convolutional neural network VGG model The value of i ranges from 1 to 4, and k is the number of channels of the output feature map of convolution group i; S302: Obtain a single channel feature map of convolution group i according to the following conditions (17) and (18): and and Respectively and The value at position (x, y); S303, according to the following conditions (19) and (20): and Perform smoothing to obtain a single-channel feature map after smoothing and Among them, r is the set area size; S304: Obtain a first infrared weighted map according to the following conditions (21) and (22): and the first visible light weight map S305: Obtain a second infrared weighted map according to the following conditions (23) and (24): and the second visible light weight map S306, obtaining a fused image obtained by convolution group i Get 4 fused images; S307, according to the following conditions (25) to (27), obtain and in, Represents the fused image The amount of source image information contained in Represent the fused images The amount of information contained in the infrared image and the visible light image; P(X1), P(X2) and Represent the infrared source image X1, the visible light source image X2 and the fused image respectively Normalized marginal histogram of ; and Represent the infrared source image X1, the visible light source image X2 and the fused image respectively Normalized joint grayscale histogram of ; S308, obtain And MI max The corresponding fused image is taken as Fa.

5. An infrared and visible light image fusion system based on low-rank sparse representation, characterized in that: include: Image decomposition module, feature extraction module and image fusion module; The image decomposition module is used to decompose the infrared image and the visible light image respectively by using the sparse consistent potential low-rank representation decomposition model, and solve the sparse consistent potential low-rank representation decomposition model by using the alternating direction multiplier method to obtain the basic part F of the infrared image respectively. a1 and the significant part F b1 And the basic part of the visible light image F a2 and the significant part F b2 ; The sparse consistent potential low-rank representation decomposition model satisfies the following conditions: s.t.X m =X m Z m +L m X m +E m ; Among them, X m represents the source image, Z m Represents the low-rank matrix of the image, L m Represents the saliency matrix of the image, E m represents the noise part of the image, m = 1, 2, representing infrared image and visible light image respectively; ‖Z m ‖ * Represents Z m The nuclear norm, ‖L m ‖ * Indicates L m The nuclear norm of 2,1 It means that the low-rank matrix of the infrared image and the low-rank matrix of the visible light image are simultaneously L 2,1 Norm regularization constraint; ‖E m ‖ 2,1 Indicates that L is applied to the noise part of the infrared image and the noise part of the visible light respectively. 2,1 norm regularization constraint; γ is the balance noise parameter, α is the regularization balance coefficient; The feature extraction module is used to use the deep neural network model to obtain the F a1 and F a2 Perform deep feature extraction and fusion to obtain the basic part fusion image Fa; The image fusion module is used to transform F b1 and F b2 Fusion is performed to obtain the salient part fusion image F b , and Fa and F b Fusion is performed to obtain an infrared and visible light fusion image; The image decomposition module is specifically used to perform the following operations: S201, obtaining an augmented Lagrangian function model defined by the following conditions (3) and (4): s.t.X m =X m S m +K m X m +E m ,Z m =J m ,Z m =S m ,L m =K m (4) S202, iteratively update each variable according to the following conditions (5) to (11) until the convergence condition is met: Where k is the number of iterations, the value of k ranges from 1 to N, N is the total number of iterations, ρ is the iteration step length, u k is the balance factor at the kth iteration, is the augmented Lagrange multiplier at the kth iteration; I is the identity matrix, X m ′ represents X m The transposed matrix of R is Z k+1 Each 1*n in 2 The n*n matrix is ​​reconstructed from the row vector of dimension; S203, update the augmented Lagrange multiplier according to the following conditions (12)-(15): S204, update the balance factor according to the following condition (16): you k+1 =min(ρu k ,m max ) (16).